[gpfsug-discuss] Tuning: single client, single thread, small files - native Scale vs NFS

[Michael Diederich]

All NFS IO requires syncing.
The client does send explicit fsync (commit). If the NFS server does not 
sync, a server fail will cause data loss!
(for small files <1M it really does not matter if it is sync on write or 
sync on close/explicit commit)


while that may be ok for a "git pull" or similar, in general it violates 
the NFS spec.

The client can decide to cache, and usually NFSv4 does less caching (for 
better consistency)

So the observed factor 100 is realistic. 
Latencies will make matters worse, so the FS should be tuned for very 
small random IO (small blocksize - small subblock-size will not help)

If you were to put the Linux kernel NFS server into the picture, it will 
behave very much the same - although Ganesha could be a bit more efficient 
(by some percent - certainly less then 200%). 


But hey - this is a GPFS cluster not some NAS box.
Run "git pull" on tthe GPFS client. Enjoy the 1800 files/sec (or more). 
Modify the files on your XY client mounting over NFS. Use a wrapper script 
to automatically  have your AD or LDAP user id SSH into the cluster to 
perform it.

Michael

Mit freundlichen Grüßen / with best regards 
 
Michael Diederich
IBM Systems Group 
Spectrum Scale
Software Development
Contact Information
IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp 
Sitz der Gesellschaft: Böblingen 
Registergericht: Amtsgericht Stuttgart, HRB 243294 

mail:
fon:
address:
michael.diederich at de.ibm.com 
+49-7034-274-4062
Am Weiher 24
D-65451 Kelsterbach





From:   valdis.kletnieks at vt.edu
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc:     Silvana De Gyves <silvanad at mx1.ibm.com>, Jay Vaddi 
<jayvaddi at us.ibm.com>, Michael Diederich <diederich at de.ibm.com>
Date:   10/16/2018 02:42 AM
Subject:        Re: [gpfsug-discuss] Tuning: single client, single thread, 
small files - native Scale vs NFS
Sent by:        Valdis Kletnieks <valdis at vt.edu>



On Mon, 15 Oct 2018 18:34:50 -0400, "Kumaran Rajaram" said:

> 1. >>When writing to GPFS directly I'm able to write ~1800 files / 
second  in a test setup. 
> >>This is roughly the same on the protocol nodes (NSD client), as well 
as 
> on the ESS IO nodes (NSD server). 
>
> 2. >> When writing to the NFS export on the protocol node itself (to 
avoid 
> any network effects) I'm only able to write ~230 files / second.

> IMHO #2, writing to the NFS export on the protocol node should be same 
as #1.
> Protocol node is also a NSD client and when you write from a protocol 
node, it
> will use the NSD protocol to write to the ESS IO nodes. In #1, you cite 
seeing
> ~1800 files from protocol node and in #2 you cite seeing ~230 file/sec 
which
> seem to contradict each other. 

I think he means this:

1) ssh nsd_server
2) cd /gpfs/filesystem/testarea
3) (whomp out 1800 files/sec)
4) mount -t nfs localhost:/gpfs/filesystem/testarea /mnt/test
5) cd /mnt/test
6) Watch the same test struggle to hit 230.

Indicating the issue is going from NFS to GPFS

(For what it's worth, we've had issues with Ganesha as well...)
[attachment "att4z9wh.dat" deleted by Michael Diederich/Germany/IBM] 



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20181016/73a0ef45/attachment.htm>