[gpfsug-discuss] GPFS and both Samba and NFS

[Jonathan Buzzard]

On Mon, 2013-12-16 at 07:31 -0800, Sven Oehme wrote:

[SNIP]

> a few words of clarification, on the controversial point, compile ctdb
> yes/no. 
> while its technically correct, that you don't have to recompile ctdb
> and it has no code that links/depends in any form on GPFS (samba
> does), you still have to compile one if the one shipped with your
> distro is out of date and while i didn't explicitly wrote it this way,
> this is what i intended to say. 
> 

Fair enough, but what you actually said is not remotely close to that.


> the version RH ships is very old and since has gotten many fixes which
> i highly recommend people to use unless the distros will pickup more
> recent version like RHEL7 will do. i probably wouldn't recommend a
> daily git build, but rather the packaged versions that are hosted on
> ftp.samba.org like :
> https://ftp.samba.org/pub/ctdb/packages/redhat/RHEL6/ 
> 
> so the proposed order of things would be to install gpfs, pull the src
> package of ctdb, compile and install ctdb and the devel packages, 

As I pointed out that is a pointless exercise. Just install the
packages.

> then pull a recent samba src package , install all the dependencies
> and build samba on this same host with gpfs and ctdb packages already
> installed. the resulting rpm's should contain the proper code to
> continue. 

I would in the strongest possible terms recommend using a VM for
compiling packages. Anyone compiling and maintaining packages on the
production or even for that matter a test GPFS cluster deserves sacking.

> after you got your versions compiled, installed and the basics of ctdb
> running, you should use the following smb.conf as a starting point : 
> 
> [global] 
>         netbios name = cluster 
>         fileid:mapping = fsname 
>         gpfs:sharemodes = yes 
>         gpfs:leases = yes 
>         gpfs:dfreequota = yes 
>         gpfs:hsm = yes 
>         syncops:onmeta = no 
>         kernel oplocks = no 
>         level2 oplocks = yes 
>         notify:inotify = no 
>         vfs objects = shadow_copy2 syncops gpfs fileid 
>         shadow:snapdir = .snapshots 
>         shadow:fixinodes = yes 
>         shadow:snapdirseverywhere = yes 
>         shadow:sort = desc 
>         wide links = no 
>         async smb echo handler = yes 
>         smbd:backgroundqueue = False 
>         use sendfile = no 
>         strict locking = yes 
>         posix locking = yes 
>         force unknown acl user = yes 
>         nfs4:mode = simple 
>         nfs4:chown = yes 
>         nfs4:acedup = merge 
>         nfs4:sidmap = /etc/samba/sidmap.tdb 
>         gpfs:winattr = yes 
>         store dos attributes = yes 
>         map readonly = yes 
>         map archive = yes 
>         map system = yes 
>         map hidden = yes 
>         ea support = yes 
>         dmapi support = no 
>         unix extensions = no 
>         socket options = TCP_NODELAY SO_KEEPALIVE TCP_KEEPCNT=4
> TCP_KEEPIDLE=240 TCP_KEEPINTVL=15 
>         strict allocate = yes 
>         gpfs:prealloc = yes 
> 

Nothing new there for those in the know. That said some of those options
might not be wanted.

> 
> if you don't configure using the registry you need to maintain the
> smb.conf file on all your nodes and i am not diving into how to setup
> the registry, but for the people who care, michael adam's presentation
> is a good starting point
> http://www.samba.org/~obnox/presentations/linux-kongress-2008/lk2008-obnox.pdf 
> 
> also depending on your authentication/idmap setup , there are multiple
> changes/additions that needs to be made. 
> 
> on the gpfs side you should set the following config parameters : 
> 
> cifsBypassShareLocksOnRename=yes 
> syncSambaMetadataOps=yes 
> cifsBypassTraversalChecking=yes 
> allowSambaCaseInsensitiveLookup=no 
> 

The cifsBypassTraversalChecking=yes is very very much a site based
choice. You need to understand what it does because it could create a
security nightmare if you just randomly enable it. On the other hand it
could be a life saver if you are moving from an existing Windows server.

Basically it enables Windows security schematics rather than Posix based
ones. That is you will no longer need access to all the parent folders
to a file in order to be able to access the file. So long as you have
permissions on the file you are good to go.

> your filesystem should have the following settings configured : 
> 
>  -D                 nfs4                     File locking semantics in
> effect 
>  -k                 nfs4                     ACL semantics in effect 

Really I would have expected -k samba :-)

>  -o                 nfssync                  Additional mount options 
>  --fastea           Yes                      Fast external attributes
> enabled? 
>  -S                 relatime                 Suppress atime mount
> option 
> 
> the -S is a performance optimization, but you need to check if your
> backup/av or other software can deal with it, it essentially reduces
> the frequency of atime updates to once every 24 hours which is the new
> default on nfs mounts and others as well. 

How does -S option impact on doing policy based tiering? I would
approach that with caution if you are doing any tiering or HSM.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.