[gpfsug-discuss] AFM - prefetch of many small files - tuning - storage latency required to increase max socket buffer size ...

[Billich Heinrich Rainer (PSI)]

Hello,

A while ago I asked the list for advice on how to tune AFM to speed-up the prefetch of small files (~1MB). In the meantime, we got some results which I want to share.

We had to increase the maximum socket buffer sizes to very high values of 40-80MB. Consider that we use IP over Infiniband and the bandwidth-delay-product is about 5MB (1-10us latency). 

How do we explain this? The reads on the nfs server have a latency of about 20ms. This is physics of disks. Hence a single client can get up to 50 requests/s. Each request is 1MB. To get 1GB/s we need 20 clients in parallel. At all times we have about 20 requests pending. Looks like the server does allocate the socket buffer space before it asks for the data. Hence it allocates/blocks about 20MB at all times. Surprisingly it’s storage latency and not network latency that required us to increase the max. socket buffer size. 

For large files prefetch works and reduces the latency of reads drastically and no special tuning is required.

We did test with kernel-nfs and gpfs  4.2.3 on  RHEL7. Whether ganesha shows a similar pattern would be interesting to know. Once we fixed the nfs issues afm did show a nice parallel prefetch up to ~1GB/s with 1MB sized files without any tuning. Still much below the 4GB/s measured with iperf between the two nodes ….

Kind regards,

Heiner

--
Paul Scherrer Institut
Science IT
Heiner Billich
WHGA 106
CH 5232  Villigen PSI
056 310 36 02
https://www.psi.ch