[gpfsug-discuss] Strange timestamp behaviour on NFS via CES

Thu Sep 21 13:09:29 BST 2017

Since I solved this old issue a long time ago, I'd thought I'd come back and report the solution in case someone else encounters similar problems in the future.

Original problem reported by users:

Copying files between folders on NFS exports from a CES server gave random timestamps on the files.  Also, apart from the initial reported problem, there where issues where users sometimes couldn't change or delete files that they where owners of.

Background:

We have a Active Directory with RFC2307 posix attributes populated, and use the built in Winbind-based AD authentication with RFC2307 ID mapping of our Spectrum Scale CES protocol servers.

All our Linux clients and servers are also AD integrated, using Nslcd and nss-pam-ldapd.

Trigger:

If a user was part of a AD group with a mixed case name, and this group gave access to a folder, and the NFS mount was done using NFSv4, the behavior in my original post occurred when copying or changing files in that folder.

Cause:

Active Directory handle LDAP-requests case insensitive, but results are returned with case retained.

Winbind and SSSD-AD converts groups and usernames to lower case. Nslcd retains case.

We run NFS with managed GIDs. Managed GIDs in NFSv3 seems to be handled case insensitive, or to ignore the actual group name after it has resolved the GID-number of the group, while NFSv4 seems to handle group names case sensitive and check the actual group name for certain operations even if the GID-number matches.

Don't fully understand the mechanism behind why certain file operations would work but others not, but in essence a user would be part of a group called "UserGroup" with GID-number 1234 in AD and on the client, but would be part of a group called "usergroup" with GID-number 1234 on the CES server.

Any operation that's authorized on the GID-number, or a case insensitive lookup of the group name, would work. Any operation authorized by a case sensitive group lookup would fail.

Three different workarounds where found to work:

1. Rename groups and users to lower case in AD

2. Change from Nslcd to either SSSD or Winbind on the clients

3. Change from NFSv4 to NFSv3 when mounting NFS

Remember to clear ID-mapping caches.

Regards,

Andreas

___________________________________
[https://mail.google.com/mail/u/0/?ui=2&ik=b0a6f02971&view=att&th=14618fab2daf0e10&attid=0.1.1&disp=emb&zw&atsh=1]<https://www.maxlab.lu.se>

Andreas Mattsson
System Engineer

MAX IV Laboratory
Lund University
Tel: +46-706-649544<tel:%2B46-706-649544>
E-mail: andreas.mattsson at maxlab.lu.se<mailto:andreas.mattsson at maxlab.lu.se>
<mailto:daniel.liikamaa at maxlab.lu.se>
________________________________
Från: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> för Stephen Ulmer <ulmer at ulmer.org>
Skickat: den 3 februari 2017 14:35:21
Till: gpfsug main discussion list
Ämne: Re: [gpfsug-discuss] Strange timestamp behaviour on NFS via CES

Does the cp actually complete? As in, does it copy all of the blocks?  What’s the exit code?

A cp’d file should have  “new” metadata. That is, it should have it’s own dates, owners, etc. (not necessarily copied from the source file).

I ran ‘strace cp foo1 foo2’, and it was pretty instructive, maybe that would get you more info. On CentOS strace is in it’s own package, YMMV.

--
Stephen

On Feb 3, 2017, at 8:19 AM, Andreas Mattsson <andreas.mattsson at maxiv.lu.se<mailto:andreas.mattsson at maxiv.lu.se>> wrote:

That works.

’touch test100’

Feb 3 14:16 test100

‘cp test100 test101’

Feb 3 14:16 test100
Apr 21 2027 test101

‘touch –r test100 test101’

Feb 3 14:16 test100
Feb 3 14:16 test101

/Andreas

That’s a cool one. :)

What if you use the "random date" file as a time reference to touch another file (like, 'touch -r file02 file03’)?

--
Stephen

On Feb 3, 2017, at 7:46 AM, Andreas Mattsson <andreas.mattsson at maxiv.lu.se<mailto:andreas.mattsson at maxiv.lu.se>> wrote:

I’m having some really strange timestamp behaviour when doing file operations on NFS mounts shared via CES on spectrum scale 4.2.1.1
The NFS clients are up to date Centos and Debian machines.
All Scale servers and NFS clients have correct date and time via NTP.

Creating a file, for instance ‘touch file00’, gives correct timestamp.
Moving the file, ‘mv file00 file01’, gives correct timestamp
Copying the file, ‘cp file01 file02’, gives a random timestamp anywhere in time, for instance Oct 12 2095 or Feb 29 1976 or something similar.

This is only via NFS. Copying the file via a native gpfs-mount or via SMB gives a correct timestamp.
Doing the same operation over NFS to other NFS-servers works correct, it is only when operating on the NFS-share from the Spectrum Scale CES the issue occurs.

Have anyone seen this before?

Regards,
Andreas Mattsson
_____________________________________________
<image001.png>

Andreas Mattsson
Systems Engineer

MAX IV Laboratory
Lund University
P.O. Box 118, SE-221 00 Lund, Sweden
Visiting address: Fotongatan 2, 225 94 Lund
Mobile: +46 706 64 95 44
andreas.mattsson at maxiv.se<mailto:andreas.mattsson at maxiv.se>
www.maxiv.se<http://www.maxiv.se/>

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170921/fbe5e837/attachment.htm>