[gpfsug-discuss] Nested NFSv4 Exports

Fri Oct 26 12:18:20 BST 2018

Hi Malhal,

thanks for the input.
I did already run Ganesha in debug mode, maybe this snippet I saved from that time might be helpful:

2018-10-16 13:21:11 : epoch 00080017 : ces-001.desy.de : ganesha.nfsd-119406[work-162] export_check_access :EXPORT :M_DBG :Check for address 192.168.142.92 for export id 3 fullpath /gpfs/exfel/d/proc
2018-10-16 13:21:11 : epoch 00080017 : ces-001.desy.de : ganesha.nfsd-119406[work-162] client_match :EXPORT :M_DBG :Match 0x941550, type = HOSTIF_CLIENT, options 0x42302050
2018-10-16 13:21:11 : epoch 00080017 : ces-001.desy.de : ganesha.nfsd-119406[work-162] LogClientListEntry :EXPORT :M_DBG :  0x941550 HOSTIF_CLIENT: 192.168.8.32 (root_squash   , R-r-, 34-, ---, TCP, ----, M
anage_Gids   , -- Deleg, anon_uid=    -2, anon_gid=    -2, sys)
2018-10-16 13:21:11 : epoch 00080017 : ces-001.desy.de : ganesha.nfsd-119406[work-162] client_match :EXPORT :M_DBG :Match 0x940c90, type = HOSTIF_CLIENT, options 0x42302050
2018-10-16 13:21:11 : epoch 00080017 : ces-001.desy.de : ganesha.nfsd-119406[work-162] LogClientListEntry :EXPORT :M_DBG :  0x940c90 HOSTIF_CLIENT: 192.168.8.33 (root_squash   , R-r-, 34-, ---, TCP, ----, M
anage_Gids   , -- Deleg, anon_uid=    -2, anon_gid=    -2, sys)
2018-10-16 13:21:11 : epoch 00080017 : ces-001.desy.de : ganesha.nfsd-119406[work-162] export_check_access :EXPORT :M_DBG :EXPORT          (              ,     ,    ,               ,               , -- Dele
g,                ,                )
2018-10-16 13:21:11 : epoch 00080017 : ces-001.desy.de : ganesha.nfsd-119406[work-162] export_check_access :EXPORT :M_DBG :EXPORT_DEFAULTS (root_squash   , ----, 34-, ---, TCP, ----, No Manage_Gids,        
 , anon_uid=    -2, anon_gid=    -2, sys)
2018-10-16 13:21:11 : epoch 00080017 : ces-001.desy.de : ganesha.nfsd-119406[work-162] export_check_access :EXPORT :M_DBG :default options (root_squash   , ----, 34-, UDP, TCP, ----, No Manage_Gids, -- Dele
g, anon_uid=    -2, anon_gid=    -2, none, sys)
2018-10-16 13:21:11 : epoch 00080017 : ces-001.desy.de : ganesha.nfsd-119406[work-162] export_check_access :EXPORT :M_DBG :Final options   (root_squash   , ----, 34-, ---, TCP, ----, No Manage_Gids, -- Dele
g, anon_uid=    -2, anon_gid=    -2, sys)
2018-10-16 13:21:11 : epoch 00080017 : ces-001.desy.de : ganesha.nfsd-119406[work-162] nfs4_export_check_access :NFS4 :INFO :NFS4: INFO: Access not allowed on Export_Id 3 /gpfs/exfel/d/proc for client ::fff
f:192.168.142.92
2018-10-16 13:21:11 : epoch 00080017 : ces-001.desy.de : ganesha.nfsd-119406[work-162] nfs4_op_lookup :EXPORT :DEBUG :NFS4ERR_ACCESS Hiding Export_Id 3 Path /gpfs/exfel/d/proc with NFS4ERR_NOENT

192.168.142.92 would be the client2 from my pseudo example, /gpfs/exfel/d/proc resembles /gpfs/filesystem1/directory1
Ganesha never checks anything for /gpfs/filesystem1/directory1/sub-directory1...or rather a subdir of /gpfs/exfel/d/proc

Is this what you meant by looking at the real export object?
If you think this is a bug, I would open a case in order to get this analyzed.

mmnfs does not show me any pseudo options, I think this has been included in 5.0.2.

Regards,
Stefan

----- Original Message -----
> From: "Malahal R Naineni" <mnaineni at in.ibm.com>
> To: gpfsug-discuss at spectrumscale.org
> Cc: gpfsug-discuss at spectrumscale.org
> Sent: Friday, October 26, 2018 7:09:45 AM
> Subject: Re: [gpfsug-discuss] Nested NFSv4 Exports

>>> /gpfs/filesystem1/directory1/sub-directory1 is exported to client2 as
>>> read-write.
>>> client2 is not included in the export for /gpfs/filesystem1/directory1.
>>> Mounting /gpfs/filesystem1/directory1/sub-directory1 on client2 does not work
>>> and results in a permission denied
> Any NFSv4 implementation needs to traverse the pseudo path for being able to
> mount an export. One would expect "client2" to traverse over
> /gpfs/filesystem1/directory1/ but not list its content/other files. I strongly
> think this is a bug in Ganesha implementation, it is probably looking at the
> real-export object than the pseudo-object for permission checking.
> One option is to change the Pseudo file system layout. For example,
> "/gpfs/client2" as "Pseudo" option for export with path "
> /gpfs/filesystem1/directory1/sub-directory1". This is directly not possible
> with Spectrum CLI command mmnfs unless you are using the latest and greatest
> ("mmnfs export add" usage would show if it supports Pseudo option). Of course,
> you can manually do it (using CCR) as Ganesha itself allows it.
> Yes, NFSv3 has no pseudo traversal, it should work.
> Regards, Malahal.
> 
> 
> ----- Original message -----
> From: "Dietrich, Stefan" <stefan.dietrich at desy.de>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug-discuss at spectrumscale.org
> Cc:
> Subject: [gpfsug-discuss] Nested NFSv4 Exports
> Date: Thu, Oct 25, 2018 5:52 PM
> Hi,
> 
> I am currently fiddling around with some nested NFSv4 exports and the differing
> behaviour to NFSv3.
> The environment is a GPFS 5.0.1 with enabled CES, so Ganesha is used as the NFS
> server.
> 
> Given the following (pseudo) directory structure:
> 
> /gpfs/filesystem1/directory1
> /gpfs/filesystem1/directory1/sub-directory1
> /gpfs/filesystem1/directory1/sub-directory2
> 
> Now to the exports:
> /gpfs/filesystem1/directory1 is exported to client1 as read-only.
> /gpfs/filesystem1/directory1/sub-directory1 is exported to client2 as
> read-write.
> 
> client2 is not included in the export for /gpfs/filesystem1/directory1.
> 
> Mounting /gpfs/filesystem1/directory1 on client1 works as expected.
> Mounting /gpfs/filesystem1/directory1/sub-directory1 on client2 does not work
> and results in a permission denied.
> If I change the protocol from NFSv4 to NFSv3, it works.
> 
> There is a section about nested NFS exports in the mmnfs doc:
> Creating nested exports (such as /path/to/folder and /path/to/folder/subfolder)
> is strongly discouraged since this might lead to serious issues in data
> consistency. Be very cautious when creating and using nested exports.
> If there is a need to have nested exports (such as /path/to/folder and
> /path/to/folder/inside/subfolder), NFSv4 client that mounts the parent
> (/path/to/folder) export will not be able to see the child export subtree
> (/path/to/folder/inside/subfolder) unless the same client is explicitly allowed
> to access the child export as well. This is okay as long as the client uses
> only NFSv4 mounts.
> 
> The Linux kernel NFS server and other NFSv4 servers do not show this behaviour.
> Is there a way to bypass this with CES/Ganesha? Or is the only solution to add
> client2 to /gpfs/filesystem1/directory1?
> 
> Regards,
> Stefan
> 
> --
> ------------------------------------------------------------------------
> Stefan Dietrich Deutsches Elektronen-Synchrotron (IT-Systems)
> Ein Forschungszentrum der Helmholtz-Gemeinschaft
> Notkestr. 85
> phone: +49-40-8998-4696 22607 Hamburg
> e-mail: stefan.dietrich at desy.de Germany
> ------------------------------------------------------------------------
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> [ http://gpfsug.org/mailman/listinfo/gpfsug-discuss |
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss ]
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss