[gpfsug-discuss] GPFS and both Samba and NFS
Sven Oehme
oehmes at us.ibm.com
Mon Dec 16 15:31:53 GMT 2013
Jez,
the other replies to my email where kind of unexpected as i was indeed
intending to help, i will continue to provide details and try to answer
serious questions and comments.
also as you point out there is no secret master plan to spread
misinformation, the topic is simply complicated and there are reasons that
are hard to explain why there is no full official support outside the
current shipping IBM products and i will not dive into discussions around
it.
a few words of clarification, on the controversial point, compile ctdb
yes/no.
while its technically correct, that you don't have to recompile ctdb and
it has no code that links/depends in any form on GPFS (samba does), you
still have to compile one if the one shipped with your distro is out of
date and while i didn't explicitly wrote it this way, this is what i
intended to say.
the version RH ships is very old and since has gotten many fixes which i
highly recommend people to use unless the distros will pickup more recent
version like RHEL7 will do. i probably wouldn't recommend a daily git
build, but rather the packaged versions that are hosted on ftp.samba.org
like : https://ftp.samba.org/pub/ctdb/packages/redhat/RHEL6/
so the proposed order of things would be to install gpfs, pull the src
package of ctdb, compile and install ctdb and the devel packages, then
pull a recent samba src package , install all the dependencies and build
samba on this same host with gpfs and ctdb packages already installed. the
resulting rpm's should contain the proper code to continue.
after you got your versions compiled, installed and the basics of ctdb
running, you should use the following smb.conf as a starting point :
[global]
netbios name = cluster
fileid:mapping = fsname
gpfs:sharemodes = yes
gpfs:leases = yes
gpfs:dfreequota = yes
gpfs:hsm = yes
syncops:onmeta = no
kernel oplocks = no
level2 oplocks = yes
notify:inotify = no
vfs objects = shadow_copy2 syncops gpfs fileid
shadow:snapdir = .snapshots
shadow:fixinodes = yes
shadow:snapdirseverywhere = yes
shadow:sort = desc
wide links = no
async smb echo handler = yes
smbd:backgroundqueue = False
use sendfile = no
strict locking = yes
posix locking = yes
force unknown acl user = yes
nfs4:mode = simple
nfs4:chown = yes
nfs4:acedup = merge
nfs4:sidmap = /etc/samba/sidmap.tdb
gpfs:winattr = yes
store dos attributes = yes
map readonly = yes
map archive = yes
map system = yes
map hidden = yes
ea support = yes
dmapi support = no
unix extensions = no
socket options = TCP_NODELAY SO_KEEPALIVE TCP_KEEPCNT=4
TCP_KEEPIDLE=240 TCP_KEEPINTVL=15
strict allocate = yes
gpfs:prealloc = yes
if you don't configure using the registry you need to maintain the
smb.conf file on all your nodes and i am not diving into how to setup the
registry, but for the people who care, michael adam's presentation is a
good starting point
http://www.samba.org/~obnox/presentations/linux-kongress-2008/lk2008-obnox.pdf
also depending on your authentication/idmap setup , there are multiple
changes/additions that needs to be made.
on the gpfs side you should set the following config parameters :
cifsBypassShareLocksOnRename=yes
syncSambaMetadataOps=yes
cifsBypassTraversalChecking=yes
allowSambaCaseInsensitiveLookup=no
your filesystem should have the following settings configured :
-D nfs4 File locking semantics in
effect
-k nfs4 ACL semantics in effect
-o nfssync Additional mount options
--fastea Yes Fast external attributes
enabled?
-S relatime Suppress atime mount option
the -S is a performance optimization, but you need to check if your
backup/av or other software can deal with it, it essentially reduces the
frequency of atime updates to once every 24 hours which is the new default
on nfs mounts and others as well.
a lot of the configuration parameters above are also above and beyond
locking and data integrity, they are for better windows compatibility and
should be checked if applicable for your environment.
i would also recommend to run on GPFS 3.5 TL3 or newer to get the proper
GPFS level of fixes for this type of configurations.
i would like to repeat that i don't write this email to encourage people
to all go start installing/ configuring samba on top of GPFS as i pointed
out that you are kind on your own unless you can read source code and/or
have somebody who does and is able to help as soon as you run into a
problem.
the main point of sharing this information is to clarify a lot of
misinformation out there and provide the people who already have setups
that are incorrect the information to at least not run into data
corruption issues do to wrong configuration.
Sven
From: Chair <chair at gpfsug.org>
To: gpfsug-discuss at gpfsug.org
Date: 12/16/2013 03:31 AM
Subject: Re: [gpfsug-discuss] GPFS and both Samba and NFS
Sent by: gpfsug-discuss-bounces at gpfsug.org
Allo
Just jumping in here a minute:
> It is unworthy of an IBM employee to spread such inaccurate
misinformation.
Whilst this may be inaccurate - I very, very, much doubt that IBM or
their employees have a secret master plan to spread misinformation (!)
In the spirit of this group, let's work together to technically look at
such issues.
Sven, if that is the case, perhaps you could crib the lines of code /
show your methodology that supports your views / experience.
Regards,
Jez
--
UG Chair
On 16/12/13 11:21, Jonathan Buzzard wrote:
> On Fri, 2013-12-13 at 10:14 -0800, Sven Oehme wrote:
>
> [SNIP]
>
>> the only way to get something working (don't get confused with
>> officially Supported) is to recompile the CTDB src packages AND the
>> Samba src packages on a node that has GPFS already installed. also the
>> inclusion of CTDB into Samba will not address this, its just a more
>> convenient packaging.
>>
>> Only if the build happens on such a node things like the vfs modules
>> for GPFS are build and included in the package.
>>
> That is a factually inaccurate statement. There is nothing in CTDB that
> is GPFS specific. Trust me I have examined the code closely to determine
> if this is the case. So unless this has changed recently you are flat
> out wrong.
>
> Consequently there is no requirement whatsoever to rebuild CTDB to get
> the vfs_gpfs module. In addition there is also no requirement to
> actually have GPFS installed to build the vfs_gpfs module either. What
> you need to have is the GPFS GPL header files and nothing else. As it is
> a loadable VFS module linking takes place at load time not compile time.
>
> It is unworthy of an IBM employee to spread such inaccurate
> misinformation.
>
> [SNIP]
>
>> said all this the binaries alone are only part of the Solution, after
>> you have the correct packages, you need to properly configuration the
>> system and setting all the right options (on GPFS as well as on CTDB
>> and smbd.conf), which unfortunate are very System configuration
>> specific, as otherwise you still can end up with data corruption if
>> not set right.
> Indeed. However I know not only what those options are, but also what
> they do despite IBM's refusal to tell us anything about them.
>
> I would also point out that there are sites that where running Samba on
> top of GPFS for many years before IBM began offering their
> SONAS/Storwize Unifed products.
>
>
> JAB.
>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20131216/cc2e7d62/attachment.htm>
More information about the gpfsug-discuss
mailing list