[gpfsug-discuss] Tuning AFM for high throughput/high IO over _really_ long distances

[Loic Tortay]

On 09/11/2016 21:53, Olaf Weiser wrote:
> let's say you have a RRT of 180 ms
> what you then need is your theoretical link speed  - let's say 10 Gbit/s ...
> easily let's take 1 GB/s
>
> this means, you socket must be capable to take your bandwidth (data stream)
> during the "first" 180ms because it will take at least this time to get back the
> first ACKs .. .
> so 1 GB / s x 0,180 s = 1024 MB/s x 0,180 s ==>> 185 MB   this means, you have
> to allow the operating system to accept socketsizes in that range...
>
> set something like this - but increase these values to 185 MB
> sysctl -w net.ipv4.tcp_rmem="12194304 12194304 12194304"
> sysctl -w net.ipv4.tcp_wmem="12194304 12194304 12194304"
> sysctl -w net.ipv4.tcp_mem="12194304 12194304 12194304"
> sysctl -w net.core.rmem_max=12194304
> sysctl -w net.core.wmem_max=12194304
> sysctl -w net.core.rmem_default=12194304
> sysctl -w net.core.wmem_default=12194304
> sysctl -w net.core.optmem_max=12194304
>
Hello,
In my opinion, some of these changes are, at best, misguided.
For instance, the unit for "tcp_mem" is not bytes but pages. It's also 
not a parameter for buffers but a parameter influencing global kernel 
memory management for TCP (source: Linux kernel documentation/source).
Or setting the maximum TCP ancillary data buffer size ("optmem_max") to 
a very large value when, as far a I know/saw when testing AFM w/ NFS, 
there is no ancillary data used.
Setting the min, default and max to the same value for the buffers is 
also, in my opinion, highly debatable (do you really want, for instance, 
each and every SSH connection to have 185 MB TCP buffers? -- 185 MB 
being the value suggested above).
I have seen the same suggestions in the AFM documentation, and in my 
opinion, along with the unhelpful "nfsPrefetchStrategy" recommandation 
("it's critical: set it to at least 5 to 10", OK but how do I chose 
between 5 to 10 or should I use 42?, what's the unit?, what are the 
criteria?), these do not contribute to give a good understanding of the 
configuration (let alone "optimization") required for AFM over NFS.

I must add that, in my opinion, I have "enough" experience with setting 
these "sysctl" parameters of NFS "tuning" (so I'm not overwhelmed by the 
complexity or whatever), to think something is really not right in that 
part of the AFM documentation.


Loïc.
-- 
|     Loïc Tortay <tortay at cc.in2p3.fr>  -  IN2P3 Computing Centre      |

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2931 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20161110/bf94eccd/attachment.bin>