[gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file

Bryan Banister bbanister at jumptrading.com
Thu Jan 18 20:12:15 GMT 2018


I should have also stated that the iostat results also report data going through the device-mapper-multipath devices as desired,
-B

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Bryan Banister
Sent: Thursday, January 18, 2018 2:09 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file

Note: External Email
-------------------------------------------------

Great!  So this is just for selecting devices and according to my `mmlsnsd -X` output it's using the correct ones, so I can probably return to ignoring this parameter!
-Bryan

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Eric Ross
Sent: Thursday, January 18, 2018 2:01 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file

Note: External Email
-------------------------------------------------

Bryan,

While waiting on the "official" word as to what each setting does
differently,  I remembered there was a brief explanation of the
differences (at least of dmm vs generic) in the
/var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit
(https://github.com/finley/GPFS-Goodies) I use to use when I was at
IBM.

//snip

dmm vs. generic is used by GPFS to prioritize internal order of
searching through available disks, then later GPFS discards other
disk device names that it finds that match as the same NSD device
by a different path.  For this reason, dmm vs. generic is an
important distinction if you are not explicitly producing the
entire and exclusive set of disks that GPFS should use, as output
from this script (nsddevices) _and_ exiting this script with a
"return 0". -Brian Finley

//end snip

If I read that correctly, it would indicate GPFS uses those additional
labels (at least dmm vs generic) as a mechanism to determine which
device files to prefer when scanning a system and finding the same NSD
available via different devices (i.e. /dev/mapper/foo vs
/dev/sdq,/dev/sdx).  By associating dmm with the dm-multipathed
device, I guess it would just ignore the /dev/sd${foo} devices when it
scans them.  Also, the difference only seems to matter if you're not
explicitly creating a list a Brian F. indicates.  If you simply
generate the list and exit (via return 0), GPFS wouldn't continue
scanning, and then find the associated /dev/sd${foo} devices to begin
with, and therefore wouldn't need a label like dmm to prioritize it
over say a generic device.


- Eric

On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister
<bbanister at jumptrading.com> wrote:
> Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the
> Devtype to the “correct” setting, for example:
>
>
>
> if [[ $osName = Linux ]]
>
> then
>
>   : # Add function to discover disks in the Linux environment.
>
>   ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}'
>
>   ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}'
>
>   ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}'
>
> fi
>
>
>
> My question is what is the correct setting?
>
>
>
> And what does GPFS do differently with each setting?
>
>
>
> Thanks,
>
> -Bryan
>
>
>
> From: gpfsug-discuss-bounces at spectrumscale.org
> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R.
> Lang
> Sent: Thursday, January 18, 2018 9:20 AM
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting,
> nsddevices file
>
>
>
> Note: External Email
>
> ________________________________
>
> So is there a way to change it if it’s set incorrectly?
>
>
>
> jeff
>
>
>
> From: gpfsug-discuss-bounces at spectrumscale.org
> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty
> Sent: Wednesday, January 17, 2018 6:28 PM
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting,
> nsddevices file
>
>
>
>
>
> Run a mmlsnsd -X   I suspect you will see that GPFS is using one of the
> /dev/sd* "generic" paths to the LUN,   not the /dev/mapper/ path.   In our
> case the device is setup as dmm
>
>
>
> [root at service5 ~]# mmlsnsd -X
>
>  Disk name    NSD volume ID      Device         Devtype  Node name
> Remarks
> ---------------------------------------------------------------------------------------------------
>  volume1      0972B6CD587CD8E0   /dev/dm-0      dmm
> service5.pok.stglabs.ibm.com server node
>  volume1      0972B6CD587CD8E0   /dev/dm-0      dmm
> service6.pok.stglabs.ibm.com server node
>  volume2      0972B6CE587CD8E4   /dev/dm-4      dmm
> service5.pok.stglabs.ibm.com server node
>  volume2      0972B6CE587CD8E4   /dev/dm-3      dmm
> service6.pok.stglabs.ibm.com server node
>  volume3      0972B6CD587CD8E7   /dev/dm-1      dmm
> service5.pok.stglabs.ibm.com server node
>  volume3      0972B6CD587CD8E7   /dev/dm-2      dmm
> service6.pok.stglabs.ibm.com server node
>  volume4      0972B6CE587CF625   /dev/dm-3      dmm
> service5.pok.stglabs.ibm.com server node
>  volume4      0972B6CE587CF625   /dev/dm-4      dmm
> service6.pok.stglabs.ibm.com server node
>
> [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK
> %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com:::::
> [root at service5 ~]#
>
>
>
> If you run an tspreparedisk -s  it will show you all of the paths.
>
>
>
> [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0
> 0972B6CD587CD8E0 /dev/sda generic
> 0972B6CD587CD8E0 /dev/sdk generic
> 0972B6CD587CD8E0 /dev/sdu generic
> 0972B6CD587CD8E0 /dev/sdah generic
> 0972B6CD587CD8E0 /dev/dm-0 dmm
> [root at service5 ~]#
>
>
>
> Jim
>
>
>
>
>
>
>
> Jim
>
> On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister
> <bbanister at jumptrading.com> wrote:
>
>
>
>
>
> Hi all,
>
>
>
> We are reviewing some of our configurations and were not sure what to make
> of the NSD Device Types that GPFS uses and what, if anything, do they change
> about how GPFS accesses/recovers/manages/etc the underlying storage based on
> this setting.
>
>
>
> The documentation doesn’t say much about it other than to consult the
> /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this
> section:
>
>
>
> # Known disk types currently are:
>
> #
>
> #   powerdisk  - EMC power path disk
>
> #   vpath      - IBM virtual path disk
>
> #   dmm        - Device-Mapper Multipath (DMM)
>
> #   dlmfdrv    - Hitachi dlm
>
> #   hdisk      - AIX hard disk
>
> #   lv         - AIX logical volume.  Historical usage only.
>
> #                Not allowed as a new device to mmcrnsd.
>
> #   gpt        - GPFS partition on Windows disk
>
> #   generic    - Device having no unique failover or multipathing
>
> #                characteristic (predominantly Linux devices).
>
> #   dasd       - DASD device (for Linux on z Systems)
>
>
>
> We have our storage under Linux Device-Mapper Multipath control (two device
> paths to all storage, active/passive) and are accessible under /dev/mapper,
> but the NSD types are current set to ‘generic’ not ‘dmm’.  This is
> configured in the /var/mmfs/etc/nsddevices file:
>
>
>
> if [[ $osName = Linux ]]
>
> then
>
>   : # Add function to discover disks in the Linux environment.
>
>   ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}'
>
>   ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}'
>
>   ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}'
>
> fi
>
>
>
> Can somebody from IBM explain what the correct setting should be and what
> differences GPFS does with ‘generic’ vs. ‘dmm’ vs. others?
>
>
>
> Thanks in advance!
>
> -Bryan
>
>
>
> ________________________________
>
>
> Note: This email is for the confidential use of the named addressee(s) only
> and may contain proprietary, confidential or privileged information. If you
> are not the intended recipient, you are hereby notified that any review,
> dissemination or copying of this email is strictly prohibited, and to please
> notify the sender immediately and destroy this email and any attachments.
> Email transmission cannot be guaranteed to be secure or error-free. The
> Company, therefore, does not make any guarantees as to the completeness or
> accuracy of this email or any attachments. This email is for informational
> purposes only and does not constitute a recommendation, offer, request or
> solicitation of any kind to buy, sell, subscribe, redeem or perform any type
> of transaction of a financial product.
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> ________________________________
>
> Note: This email is for the confidential use of the named addressee(s) only
> and may contain proprietary, confidential or privileged information. If you
> are not the intended recipient, you are hereby notified that any review,
> dissemination or copying of this email is strictly prohibited, and to please
> notify the sender immediately and destroy this email and any attachments.
> Email transmission cannot be guaranteed to be secure or error-free. The
> Company, therefore, does not make any guarantees as to the completeness or
> accuracy of this email or any attachments. This email is for informational
> purposes only and does not constitute a recommendation, offer, request or
> solicitation of any kind to buy, sell, subscribe, redeem or perform any type
> of transaction of a financial product.
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.


More information about the gpfsug-discuss mailing list