[gpfsug-discuss] CES-NFS: UID and GID resolution
Wilson, Neil
neil.wilson at metoffice.gov.uk
Mon Jun 18 17:05:35 BST 2018
I think it’s caused by the ID mapping not being configured properly.
Found this on the redhat knowledge base.
Environment
* Red Hat Enterprise Linux 5
* Red Hat Enterprise Linux 6
* Red Hat Enterprise Linux 7
* NFSv4 share being exported from an NFSv4 capable NFS server
Issue
* From the client, the mounted NFSv4 share has ownership for all files and directories listed as nobody:nobody instead of the actual user that owns them on the NFSv4 server, or who created the new file and directory.
* Seeing nobody:nobody permissions on nfsv4 shares on the nfs client. Also seeing the following error in /var/log/messages:
* How to configure Idmapping for NFSv4
Raw<https://access.redhat.com/solutions/33455>
nss_getpwnam: name 'root at example.com' does not map into domain 'localdomain'
Resolution
* Modify the /etc/idmapd.conf with the proper domain (FQDN), on both the client and server. In this example, the proper domain is "example.com" so the "Domain =" directive within /etc/idmapd.conf should be modified to read:
Raw<https://access.redhat.com/solutions/33455>
Domain = example.com
* Note:
* If using a NetApp Filer, the NFS.V4.ID.DOMAIN parameter must be set to match the "Domain =" parameter on the client.
* If using a Solaris machine as the NFS server, the NFSMAPID_DOMAIN value in /etc/default/nfs must match the RHEL clients Domain.
* On Red Hat Enterprise Linux 6.2 and older, to put the changes into effect restart the rpcidmapd service and remount the NFSv4 filesystem :
Raw<https://access.redhat.com/solutions/33455>
# service rpcidmapd restart
# mount -o remount /nfs/mnt/point
NOTE: It is only necessary to restart rpc.idmapd service on systems where rpc.idmapd is actually performing the id mapping. On RHEL 6.3 and newer NFS CLIENTS, the maps are stored in the kernel keyring and the id mapping itself is performed by the /sbin/nfsidmap program. On older NFS CLIENTS (RHEL 6.2 and older) as well as on all NFS SERVERS running RHEL, the id mapping is performed by rpc.idmapd.
* Ensure the client and server have matching UID's and GID's. It is a common misconception that the UID's and GID's can differ when using NFSv4. The sole purpose of id mapping is to map an id to a name and vice-versa. ID mapping is not intended as some sort of replacement for managing id's.
* On Red Hat Enterprise Linux 6.3 and higher, if the above settings have been applied and UID/GID's are matched on server and client and users are still being mapped to nobody:nobody than a clearing of the idmapd cache may be required:
Raw<https://access.redhat.com/solutions/33455>
# nfsidmap -c
NOTE: The above command is only necessary on systems that use the keyring-based id mapper, i.e. NFS CLIENTS running RHEL 6.3 and higher. On RHEL 6.2 and older NFS CLIENTS as well as all NFS SERVERS running RHEL, the cache should be cleared out when rpc.idmapd is restarted.
* Another check, see if the passwd:, shadow: and group: settings are set correctly in the /etc/nsswitch.conf file on both Server and Client.
Disabling idmapping
NOTE: In order to properly disable idmapping, it must be disabled on both the NFS client and NFS server.
- By default, RHEL6.3 and newer NFS clients and servers disable idmapping when utilizing the AUTH_SYS/UNIX authentication flavor by enabling the following booleans:
Raw<https://access.redhat.com/solutions/33455>
NFS client
# echo 'Y' > /sys/module/nfs/parameters/nfs4_disable_idmapping
NFS server
# echo 'Y' > /sys/module/nfsd/parameters/nfs4_disable_idmapping
* If using a NetApp filer, the options nfs.v4.id.allow_numerics on command can be used to disable idmapping. More information can be found here<https://library.netapp.com/ecmdocs/ECMP1196993/html/GUID-24367A9F-E17B-4725-ADC1-02D86F56F78E.html>.
* With this boolean enabled, NFS clients will instead send numeric UID/GID numbers in outgoing attribute calls and NFS servers will send numeric UID/GID numbers in outgoing attribute replies.
· If NFS clients sending numeric UID/GID values in a SETATTR call receive an NFS4ERR_BADOWNER reply from the NFS server clients will re-enable idmapping and send user at domain strings for that specific mount from that point forward.
· We can make the option nfs4_disable_idmapping persistent across reboot.
· After the above value has been changed, for the setting to take effect for any NFS server export mounted on the NFS client, you must unmount all NFS mount points for the given NFS server, and then re-mount them. If you have auto mounts stop all processes accessing the mounts and allow the automount daemon to unmount them. Once all NFS mount points are gone to the desired NFS server, remount the NFS mount points and the new setting should be in place. If this is too problematic, you may want to schedule a reboot of the NFS client.
· To verify the setting has been changed properly, you can look inside the /proc/self/mountstats file 'caps' line, which contains a hex value of 2 bytes (16 bits). This is the line that shows the NFS server's "capabilities", and the most significant bit #15 is the one which represents whether idmapping is disabled or not (the NFS_CAP_UIDGID_NOMAP bit - see the Root Cause section)
Raw<https://access.redhat.com/solutions/33455>
# cat /sys/module/nfs/parameters/nfs4_disable_idmapping
Y
# umount /mnt
# mount rhel6u6-node2:/exports/nfs4 /mnt
# grep -A 5 rhel6u6-node2 /proc/self/mountstats | egrep '(rhel6u6-node2|caps:)'
device rhel6u6-node2:/exports/nfs4 mounted on /mnt with fstype nfs4 statvers=1.0
caps: caps=0xffff,wtmult=512,dtsize=32768,bsize=0,namlen=255
^
* Example of nfs4_disable_idmapping = 'N'
Raw<https://access.redhat.com/solutions/33455>
[root at rhel6u3-node1 ~]# echo N > /sys/module/nfs/parameters/nfs4_disable_idmapping
[root at rhel6u3-node1 ~]# umount /mnt
[root at rhel6u3-node1 ~]# mount rhel6u6-node2:/exports/nfs4 /mnt
[root at rhel6u3-node1 ~]# grep -A 5 rhel6u6-node2 /proc/self/mountstats | egrep '(rhel6u6-node2|caps:)'
device rhel6u6-node2:/exports/nfs4 mounted on /mnt with fstype nfs4 statvers=1.0
caps: caps=0x7fff,wtmult=512,dtsize=32768,bsize=0,namlen=255
^
NOTE: To force ONLY numeric IDs to be used on the client, add RPCIDMAPDARGS="-C" to the etc/sysconfig/nfs file and restart the rpcidmapd service. See man rpc.idmapd for more information.
NOTE: This option can only be used with AUTH_SYS/UNIX authentication flavors, if you wish to use something like Kerberos, idmapping must be used.
Root Cause
* NFSv4 utilizes ID mapping to ensure permissions are set properly on exported shares, if the domains of the client and server do not match then the permissions are mapped to nobody:nobody.
NFS_CAP_UIDGID_NOMAP bit
* The nfs4_disable_idmapping is a module parameter which is read only one time, at the point at which the kernel sets up the data structure that represents an NFS server. Once it is read, a flag is set in the nfs_server structure NFS_CAP_UIDGID_NOMAP.
Raw<https://access.redhat.com/solutions/33455>
#define NFS_CAP_UIDGID_NOMAP (1U << 15)
static int nfs4_init_server(struct nfs_server *server,
const struct nfs_parsed_mount_data *data)
{
struct rpc_timeout timeparms;
int error;
dprintk("--> nfs4_init_server()\n");
nfs_init_timeout_values(&timeparms, data->nfs_server.protocol,
data->timeo, data->retrans);
/* Initialise the client representation from the mount data */
server->flags = data->flags;
server->caps |= NFS_CAP_ATOMIC_OPEN|NFS_CAP_CHANGE_ATTR|NFS_CAP_POSIX_LOCK;
if (!(data->flags & NFS_MOUNT_NORDIRPLUS))
server->caps |= NFS_CAP_READDIRPLUS;
server->options = data->options;
/* Get a client record */
error = nfs4_set_client(server,
data->nfs_server.hostname,
(const struct sockaddr *)&data->nfs_server.address,
data->nfs_server.addrlen,
data->client_address,
data->auth_flavors[0],
data->nfs_server.protocol,
&timeparms,
data->minorversion);
if (error < 0)
goto error;
/*
* Don't use NFS uid/gid mapping if we're using AUTH_SYS or lower
* authentication.
*/
if (nfs4_disable_idmapping && data->auth_flavors[0] == RPC_AUTH_UNIX) <--- set a flag based on the module parameter
server->caps |= NFS_CAP_UIDGID_NOMAP; <-------------------------- flag set
if (data->rsize)
server->rsize = nfs_block_size(data->rsize, NULL);
if (data->wsize)
server->wsize = nfs_block_size(data->wsize, NULL);
server->acregmin = data->acregmin * HZ;
server->acregmax = data->acregmax * HZ;
server->acdirmin = data->acdirmin * HZ;
server->acdirmax = data->acdirmax * HZ;
server->port = data->nfs_server.port;
error = nfs_init_server_rpcclient(server, &timeparms, data->auth_flavors[0]);
error:
/* Done */
dprintk("<-- nfs4_init_server() = %d\n", error);
return error;
}
* This flag is later checked when deciding whether to use numeric uid or gids or to use idmapping.
Raw<https://access.redhat.com/solutions/33455>
int nfs_map_uid_to_name(const struct nfs_server *server, __u32 uid, char *buf, size_t buflen)
{
struct idmap *idmap = server->nfs_client->cl_idmap;
int ret = -EINVAL;
if (!(server->caps & NFS_CAP_UIDGID_NOMAP)) { <------------ CHECK FLAG, DECIDE whether to call idmapper
ret = nfs_idmap_lookup_name(uid, "user", buf, buflen);
if (ret < 0)
ret = nfs_idmap_name(idmap, &idmap->idmap_user_hash, uid, buf);
}
if (ret < 0)
ret = nfs_map_numeric_to_string(uid, buf, buflen);
return ret;
}
int nfs_map_gid_to_group(const struct nfs_server *server, __u32 gid, char *buf, size_t buflen)
{
struct idmap *idmap = server->nfs_client->cl_idmap;
int ret = -EINVAL;
if (!(server->caps & NFS_CAP_UIDGID_NOMAP)) { <------------ CHECK FLAG, DECIDE whether to call idmapper
ret = nfs_idmap_lookup_name(gid, "group", buf, buflen);
if (ret < 0)
ret = nfs_idmap_name(idmap, &idmap->idmap_group_hash, gid, buf);
}
if (ret < 0)
ret = nfs_map_numeric_to_string(gid, buf, buflen);
return ret;
}
"fs/nfs/idmap.c" 872L, 21804C
* For more information on NFSv4 ID mapping in Red Hat Enterprise Linux, see https://access.redhat.com/articles/2252881
Diagnostic Steps
* Debugging/verbosity can be enabled by editing /etc/sysconfig/nfs:
Raw<https://access.redhat.com/solutions/33455>
RPCIDMAPDARGS="-vvv"
* The following output is shown in /var/log/messages when the mount has been completed and the system shows nobody:nobody as user and group permissions on directories and files:
Raw<https://access.redhat.com/solutions/33455>
Jun 3 20:22:08 node1 rpc.idmapd[1874]: nss_getpwnam: name 'root at example.com' does not map into domain 'localdomain'
Jun 3 20:25:44 node1 rpc.idmapd[1874]: nss_getpwnam: name 'root at example.com' does not map into domain 'localdomain'
* Collect a tcpdump of the mount attempt:
Raw<https://access.redhat.com/solutions/33455>
# tcpdump -s0 -i {INTERFACE} host {NFS.SERVER.IP} -w /tmp/{casenumber}-$(hostname)-$(date +"%Y-%m-%d-%H-%M-%S").pcap &
* If a TCP packet capture has been obtained, check for a nfs.nfsstat4 packet that has returned a non-zero response equivalent to 10039 (NFSV4ERR_BADOWNER).
* From the NFSv4 RFC:
Raw<https://access.redhat.com/solutions/33455>
NFS4ERR_BADOWNER = 10039,/* owner translation bad */
NFS4ERR_BADOWNER An owner, owner_group, or ACL attribute value
can not be translated to local representation.
Hope this helps.
Neil Wilson Senior IT Practitioner
Storage, Virtualisation and Mainframe Team IT Services
Met Office FitzRoy Road Exeter Devon EX1 3PB United Kingdom
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Oesterlin, Robert
Sent: 18 June 2018 16:54
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] CES-NFS: UID and GID resolution
Can anyone tell me why I’m not seeing the correct UID/GID resolution from CES? Configured with LDAP authentication, and this appears to work correctly.
On my CES cluster (V4):
[robert_oesterlin at unv-oester robert_oesterlin]$ ls -l
total 3
-rw-r--r-- 1 nobody nobody 15 Jun 18 11:40 junk1
-rw-r--r-- 1 nobody nobody 4 Oct 9 2016 junk.file
-rw-r--r-- 1 nobody nobody 1 May 24 10:44 test1
On my CNFS cluster (V3)
[root at unv-oester2 robert_oesterlin]# ls -l
total 0
-rw-r--r-- 1 robert_oesterlin users 15 Jun 18 11:40 junk1
-rw-r--r-- 1 robert_oesterlin users 4 Oct 9 2016 junk.file
-rw-r--r-- 1 robert_oesterlin users 1 May 24 10:44 test1
Bob Oesterlin
Sr Principal Storage Engineer, Nuance
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180618/340f7e4d/attachment.htm>
More information about the gpfsug-discuss
mailing list