[gpfsug-discuss] CES-NFS: UID and GID resolution
Chetan R Kulkarni
chetkulk at in.ibm.com
Mon Jun 18 17:20:29 BST 2018
Please make sure NFSv4 ID Mapping value matches on client and server
(e.g. test.com; may vary on your setup).
server:
mmnfs config change IDMAPD_DOMAIN=test.com
client:
e.g. RHEL NFS client; set Domain attribute in /etc/idmapd.conf file
and restart idmap service.
# egrep ^Domain /etc/idmapd.conf
Domain = test.com
# service nfs-idmap restart
reference Link:
https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/b1ladm_authconsidfornfsv4access.htm
Thanks,
Chetan.
From: "Wilson, Neil" <neil.wilson at metoffice.gov.uk>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: 06/18/2018 09:35 PM
Subject: Re: [gpfsug-discuss] CES-NFS: UID and GID resolution
Sent by: gpfsug-discuss-bounces at spectrumscale.org
I think it’s caused by the ID mapping not being configured properly.
Found this on the redhat knowledge base.
Environment
Red Hat Enterprise Linux 5
Red Hat Enterprise Linux 6
Red Hat Enterprise Linux 7
NFSv4 share being exported from an NFSv4 capable NFS server
Issue
From the client, the mounted NFSv4 share has ownership for all files
and directories listed as nobody:nobody instead of the actual user
that owns them on the NFSv4 server, or who created the new file and
directory.
Seeing nobody:nobody permissions on nfsv4 shares on the nfs client.
Also seeing the following error in /var/log/messages:
How to configure Idmapping for NFSv4
Raw
nss_getpwnam: name 'root at example.com' does not map into domain
'localdomain'
Resolution
Modify the /etc/idmapd.conf with the proper domain (FQDN), on both
the client and server. In this example, the proper domain is
"example.com" so the "Domain =" directive within /etc/idmapd.conf
should be modified to read:
Raw
Domain = example.com
Note:
If using a NetApp Filer, the NFS.V4.ID.DOMAIN parameter must be
set to match the "Domain =" parameter on the client.
If using a Solaris machine as the NFS server, the
NFSMAPID_DOMAIN value in /etc/default/nfs must match the RHEL
clients Domain.
On Red Hat Enterprise Linux 6.2 and older, to put the changes into
effect restart the rpcidmapd service and remount the NFSv4
filesystem :
Raw
# service rpcidmapd restart
# mount -o remount /nfs/mnt/point
NOTE: It is only necessary to restart rpc.idmapd service on systems where
rpc.idmapd is actually performing the id mapping. On RHEL 6.3 and newer NFS
CLIENTS, the maps are stored in the kernel keyring and the id mapping
itself is performed by the /sbin/nfsidmap program. On older NFS CLIENTS
(RHEL 6.2 and older) as well as on all NFS SERVERS running RHEL, the id
mapping is performed by rpc.idmapd.
Ensure the client and server have matching UID's and GID's. It is a
common misconception that the UID's and GID's can differ when using
NFSv4. The sole purpose of id mapping is to map an id to a name and
vice-versa. ID mapping is not intended as some sort of replacement
for managing id's.
On Red Hat Enterprise Linux 6.3 and higher, if the above settings
have been applied and UID/GID's are matched on server and client and
users are still being mapped to nobody:nobody than a clearing of the
idmapd cache may be required:
Raw
# nfsidmap -c
NOTE: The above command is only necessary on systems that use the
keyring-based id mapper, i.e. NFS CLIENTS running RHEL 6.3 and higher. On
RHEL 6.2 and older NFS CLIENTS as well as all NFS SERVERS running RHEL, the
cache should be cleared out when rpc.idmapd is restarted.
Another check, see if the passwd:, shadow: and group: settings are
set correctly in the /etc/nsswitch.conf file on both Server and
Client.
Disabling idmapping
NOTE: In order to properly disable idmapping, it must be disabled on both
the NFS client and NFS server.
- By default, RHEL6.3 and newer NFS clients and servers disable idmapping
when utilizing the AUTH_SYS/UNIX authentication flavor by enabling the
following booleans:
Raw
NFS client
# echo 'Y' > /sys/module/nfs/parameters/nfs4_disable_idmapping
NFS server
# echo 'Y' > /sys/module/nfsd/parameters/nfs4_disable_idmapping
If using a NetApp filer, the options nfs.v4.id.allow_numerics on
command can be used to disable idmapping. More information can be
found here.
With this boolean enabled, NFS clients will instead send numeric
UID/GID numbers in outgoing attribute calls and NFS servers will send
numeric UID/GID numbers in outgoing attribute replies.
· If NFS clients sending numeric UID/GID values in a SETATTR
call receive an NFS4ERR_BADOWNER reply from the NFS server clients
will re-enable idmapping and send user at domain strings for that
specific mount from that point forward.
· We can make the option nfs4_disable_idmapping persistent
across reboot.
· After the above value has been changed, for the setting to
take effect for any NFS server export mounted on the NFS client, you
must unmount all NFS mount points for the given NFS server, and then
re-mount them. If you have auto mounts stop all processes accessing
the mounts and allow the automount daemon to unmount them. Once all
NFS mount points are gone to the desired NFS server, remount the NFS
mount points and the new setting should be in place. If this is too
problematic, you may want to schedule a reboot of the NFS client.
· To verify the setting has been changed properly, you can
look inside the /proc/self/mountstats file 'caps' line, which
contains a hex value of 2 bytes (16 bits). This is the line that
shows the NFS server's "capabilities", and the most significant bit
#15 is the one which represents whether idmapping is disabled or not
(the NFS_CAP_UIDGID_NOMAP bit - see the Root Cause section)
Raw
# cat /sys/module/nfs/parameters/nfs4_disable_idmapping
Y
# umount /mnt
# mount rhel6u6-node2:/exports/nfs4 /mnt
# grep -A 5 rhel6u6-node2 /proc/self/mountstats | egrep '(rhel6u6-node2|
caps:)'
device rhel6u6-node2:/exports/nfs4 mounted on /mnt with fstype nfs4
statvers=1.0
caps: caps=0xffff,wtmult=512,dtsize=32768,bsize=0,namlen=255
^
Example of nfs4_disable_idmapping = 'N'
Raw
[root at rhel6u3-node1 ~]# echo N
> /sys/module/nfs/parameters/nfs4_disable_idmapping
[root at rhel6u3-node1 ~]# umount /mnt
[root at rhel6u3-node1 ~]# mount rhel6u6-node2:/exports/nfs4 /mnt
[root at rhel6u3-node1 ~]# grep -A 5 rhel6u6-node2 /proc/self/mountstats |
egrep '(rhel6u6-node2|caps:)'
device rhel6u6-node2:/exports/nfs4 mounted on /mnt with fstype nfs4
statvers=1.0
caps: caps=0x7fff,wtmult=512,dtsize=32768,bsize=0,namlen=255
^
NOTE: To force ONLY numeric IDs to be used on the client, add
RPCIDMAPDARGS="-C" to the etc/sysconfig/nfs file and restart the rpcidmapd
service. See man rpc.idmapd for more information.
NOTE: This option can only be used with AUTH_SYS/UNIX authentication
flavors, if you wish to use something like Kerberos, idmapping must be
used.
Root Cause
NFSv4 utilizes ID mapping to ensure permissions are set properly on
exported shares, if the domains of the client and server do not match
then the permissions are mapped to nobody:nobody.
NFS_CAP_UIDGID_NOMAP bit
The nfs4_disable_idmapping is a module parameter which is read only
one time, at the point at which the kernel sets up the data structure
that represents an NFS server. Once it is read, a flag is set in the
nfs_server structure NFS_CAP_UIDGID_NOMAP.
Raw
#define NFS_CAP_UIDGID_NOMAP (1U << 15)
static int nfs4_init_server(struct nfs_server *server,
const struct nfs_parsed_mount_data *data)
{
struct rpc_timeout timeparms;
int error;
dprintk("--> nfs4_init_server()\n");
nfs_init_timeout_values(&timeparms, data->nfs_server.protocol,
data->timeo, data->retrans);
/* Initialise the client representation from the mount data */
server->flags = data->flags;
server->caps |= NFS_CAP_ATOMIC_OPEN|NFS_CAP_CHANGE_ATTR|
NFS_CAP_POSIX_LOCK;
if (!(data->flags & NFS_MOUNT_NORDIRPLUS))
server->caps |= NFS_CAP_READDIRPLUS;
server->options = data->options;
/* Get a client record */
error = nfs4_set_client(server,
data->nfs_server.hostname,
(const struct sockaddr *)&data->nfs_server.address,
data->nfs_server.addrlen,
data->client_address,
data->auth_flavors[0],
data->nfs_server.protocol,
&timeparms,
data->minorversion);
if (error < 0)
goto error;
/*
* Don't use NFS uid/gid mapping if we're using AUTH_SYS or lower
* authentication.
*/
if (nfs4_disable_idmapping && data->auth_flavors[0] ==
RPC_AUTH_UNIX) <--- set a flag based on the module parameter
server->caps |= NFS_CAP_UIDGID_NOMAP;
<-------------------------- flag set
if (data->rsize)
server->rsize = nfs_block_size(data->rsize, NULL);
if (data->wsize)
server->wsize = nfs_block_size(data->wsize, NULL);
server->acregmin = data->acregmin * HZ;
server->acregmax = data->acregmax * HZ;
server->acdirmin = data->acdirmin * HZ;
server->acdirmax = data->acdirmax * HZ;
server->port = data->nfs_server.port;
error = nfs_init_server_rpcclient(server, &timeparms, data->
auth_flavors[0]);
error:
/* Done */
dprintk("<-- nfs4_init_server() = %d\n", error);
return error;
}
This flag is later checked when deciding whether to use numeric uid
or gids or to use idmapping.
Raw
int nfs_map_uid_to_name(const struct nfs_server *server, __u32 uid, char
*buf, size_t buflen)
{
struct idmap *idmap = server->nfs_client->cl_idmap;
int ret = -EINVAL;
if (!(server->caps & NFS_CAP_UIDGID_NOMAP)) { <------------ CHECK
FLAG, DECIDE whether to call idmapper
ret = nfs_idmap_lookup_name(uid, "user", buf, buflen);
if (ret < 0)
ret = nfs_idmap_name(idmap, &idmap->
idmap_user_hash, uid, buf);
}
if (ret < 0)
ret = nfs_map_numeric_to_string(uid, buf, buflen);
return ret;
}
int nfs_map_gid_to_group(const struct nfs_server *server, __u32 gid, char
*buf, size_t buflen)
{
struct idmap *idmap = server->nfs_client->cl_idmap;
int ret = -EINVAL;
if (!(server->caps & NFS_CAP_UIDGID_NOMAP)) { <------------ CHECK
FLAG, DECIDE whether to call idmapper
ret = nfs_idmap_lookup_name(gid, "group", buf, buflen);
if (ret < 0)
ret = nfs_idmap_name(idmap, &idmap->
idmap_group_hash, gid, buf);
}
if (ret < 0)
ret = nfs_map_numeric_to_string(gid, buf, buflen);
return ret;
}
"fs/nfs/idmap.c" 872L, 21804C
For more information on NFSv4 ID mapping in Red Hat Enterprise Linux,
see https://access.redhat.com/articles/2252881
Diagnostic Steps
Debugging/verbosity can be enabled by editing /etc/sysconfig/nfs:
Raw
RPCIDMAPDARGS="-vvv"
The following output is shown in /var/log/messages when the mount has
been completed and the system shows nobody:nobody as user and group
permissions on directories and files:
Raw
Jun 3 20:22:08 node1 rpc.idmapd[1874]: nss_getpwnam: name
'root at example.com' does not map into domain 'localdomain'
Jun 3 20:25:44 node1 rpc.idmapd[1874]: nss_getpwnam: name
'root at example.com' does not map into domain 'localdomain'
Collect a tcpdump of the mount attempt:
Raw
# tcpdump -s0 -i {INTERFACE} host {NFS.SERVER.IP} -w /tmp/{casenumber}-$
(hostname)-$(date +"%Y-%m-%d-%H-%M-%S").pcap &
If a TCP packet capture has been obtained, check for a nfs.nfsstat4
packet that has returned a non-zero response equivalent to 10039
(NFSV4ERR_BADOWNER).
From the NFSv4 RFC:
Raw
NFS4ERR_BADOWNER = 10039,/* owner translation bad */
NFS4ERR_BADOWNER An owner, owner_group, or ACL attribute value
can not be translated to local representation.
Hope this helps.
Neil Wilson Senior IT Practitioner
Storage, Virtualisation and Mainframe Team IT Services
Met Office FitzRoy Road Exeter Devon EX1 3PB United Kingdom
From: gpfsug-discuss-bounces at spectrumscale.org
<gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Oesterlin, Robert
Sent: 18 June 2018 16:54
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] CES-NFS: UID and GID resolution
Can anyone tell me why I’m not seeing the correct UID/GID resolution from
CES? Configured with LDAP authentication, and this appears to work
correctly.
On my CES cluster (V4):
[robert_oesterlin at unv-oester robert_oesterlin]$ ls -l
total 3
-rw-r--r-- 1 nobody nobody 15 Jun 18 11:40 junk1
-rw-r--r-- 1 nobody nobody 4 Oct 9 2016 junk.file
-rw-r--r-- 1 nobody nobody 1 May 24 10:44 test1
On my CNFS cluster (V3)
[root at unv-oester2 robert_oesterlin]# ls -l
total 0
-rw-r--r-- 1 robert_oesterlin users 15 Jun 18 11:40 junk1
-rw-r--r-- 1 robert_oesterlin users 4 Oct 9 2016 junk.file
-rw-r--r-- 1 robert_oesterlin users 1 May 24 10:44 test1
Bob Oesterlin
Sr Principal Storage Engineer, Nuance
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180618/370c4a59/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180618/370c4a59/attachment.gif>
More information about the gpfsug-discuss
mailing list