[gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file
Jon Bernard
jonbernard at gmail.com
Fri Jan 19 15:57:11 GMT 2018
A few years ago, Yuri noted that
"dmm" has a special meaning to GPFS code. SCSI3 PR code keys off this disk
type.
https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014924464#77777777-0000-0000-0000-000014924464
https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#scsi3
Jon
On Thu, Jan 18, 2018 at 3:22 PM, Eric Ross <er.a.ross at gmail.com> wrote:
> I would say that's what it *appears* to be doing. The sys admin in me
> would still want some "official" word from IBM, as I wouldn't my
> assumption to be the cause of a meltdown at 16.50 on a Friday when
> everyone is heading out for the weekend ;)
>
> Cheers,
> -Eric
>
> On Thu, Jan 18, 2018 at 2:09 PM, Bryan Banister
> <bbanister at jumptrading.com> wrote:
> > Great! So this is just for selecting devices and according to my
> `mmlsnsd -X` output it's using the correct ones, so I can probably return
> to ignoring this parameter!
> > -Bryan
> >
> > -----Original Message-----
> > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-
> bounces at spectrumscale.org] On Behalf Of Eric Ross
> > Sent: Thursday, January 18, 2018 2:01 PM
> > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting,
> nsddevices file
> >
> > Note: External Email
> > -------------------------------------------------
> >
> > Bryan,
> >
> > While waiting on the "official" word as to what each setting does
> > differently, I remembered there was a brief explanation of the
> > differences (at least of dmm vs generic) in the
> > /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit
> > (https://github.com/finley/GPFS-Goodies) I use to use when I was at
> > IBM.
> >
> > //snip
> >
> > dmm vs. generic is used by GPFS to prioritize internal order of
> > searching through available disks, then later GPFS discards other
> > disk device names that it finds that match as the same NSD device
> > by a different path. For this reason, dmm vs. generic is an
> > important distinction if you are not explicitly producing the
> > entire and exclusive set of disks that GPFS should use, as output
> > from this script (nsddevices) _and_ exiting this script with a
> > "return 0". -Brian Finley
> >
> > //end snip
> >
> > If I read that correctly, it would indicate GPFS uses those additional
> > labels (at least dmm vs generic) as a mechanism to determine which
> > device files to prefer when scanning a system and finding the same NSD
> > available via different devices (i.e. /dev/mapper/foo vs
> > /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed
> > device, I guess it would just ignore the /dev/sd${foo} devices when it
> > scans them. Also, the difference only seems to matter if you're not
> > explicitly creating a list a Brian F. indicates. If you simply
> > generate the list and exit (via return 0), GPFS wouldn't continue
> > scanning, and then find the associated /dev/sd${foo} devices to begin
> > with, and therefore wouldn't need a label like dmm to prioritize it
> > over say a generic device.
> >
> >
> > - Eric
> >
> > On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister
> > <bbanister at jumptrading.com> wrote:
> >> Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the
> >> Devtype to the “correct” setting, for example:
> >>
> >>
> >>
> >> if [[ $osName = Linux ]]
> >>
> >> then
> >>
> >> : # Add function to discover disks in the Linux environment.
> >>
> >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}'
> >>
> >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}'
> >>
> >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}'
> >>
> >> fi
> >>
> >>
> >>
> >> My question is what is the correct setting?
> >>
> >>
> >>
> >> And what does GPFS do differently with each setting?
> >>
> >>
> >>
> >> Thanks,
> >>
> >> -Bryan
> >>
> >>
> >>
> >> From: gpfsug-discuss-bounces at spectrumscale.org
> >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey
> R.
> >> Lang
> >> Sent: Thursday, January 18, 2018 9:20 AM
> >> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting,
> >> nsddevices file
> >>
> >>
> >>
> >> Note: External Email
> >>
> >> ________________________________
> >>
> >> So is there a way to change it if it’s set incorrectly?
> >>
> >>
> >>
> >> jeff
> >>
> >>
> >>
> >> From: gpfsug-discuss-bounces at spectrumscale.org
> >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim
> Doherty
> >> Sent: Wednesday, January 17, 2018 6:28 PM
> >> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting,
> >> nsddevices file
> >>
> >>
> >>
> >>
> >>
> >> Run a mmlsnsd -X I suspect you will see that GPFS is using one of the
> >> /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In
> our
> >> case the device is setup as dmm
> >>
> >>
> >>
> >> [root at service5 ~]# mmlsnsd -X
> >>
> >> Disk name NSD volume ID Device Devtype Node name
> >> Remarks
> >> ------------------------------------------------------------
> ---------------------------------------
> >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm
> >> service5.pok.stglabs.ibm.com server node
> >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm
> >> service6.pok.stglabs.ibm.com server node
> >> volume2 0972B6CE587CD8E4 /dev/dm-4 dmm
> >> service5.pok.stglabs.ibm.com server node
> >> volume2 0972B6CE587CD8E4 /dev/dm-3 dmm
> >> service6.pok.stglabs.ibm.com server node
> >> volume3 0972B6CD587CD8E7 /dev/dm-1 dmm
> >> service5.pok.stglabs.ibm.com server node
> >> volume3 0972B6CD587CD8E7 /dev/dm-2 dmm
> >> service6.pok.stglabs.ibm.com server node
> >> volume4 0972B6CE587CF625 /dev/dm-3 dmm
> >> service5.pok.stglabs.ibm.com server node
> >> volume4 0972B6CE587CF625 /dev/dm-4 dmm
> >> service6.pok.stglabs.ibm.com server node
> >>
> >> [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK
> >> %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:
> 0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.
> pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:
> service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com:::::
> >> [root at service5 ~]#
> >>
> >>
> >>
> >> If you run an tspreparedisk -s it will show you all of the paths.
> >>
> >>
> >>
> >> [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0
> >> 0972B6CD587CD8E0 /dev/sda generic
> >> 0972B6CD587CD8E0 /dev/sdk generic
> >> 0972B6CD587CD8E0 /dev/sdu generic
> >> 0972B6CD587CD8E0 /dev/sdah generic
> >> 0972B6CD587CD8E0 /dev/dm-0 dmm
> >> [root at service5 ~]#
> >>
> >>
> >>
> >> Jim
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> Jim
> >>
> >> On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister
> >> <bbanister at jumptrading.com> wrote:
> >>
> >>
> >>
> >>
> >>
> >> Hi all,
> >>
> >>
> >>
> >> We are reviewing some of our configurations and were not sure what to
> make
> >> of the NSD Device Types that GPFS uses and what, if anything, do they
> change
> >> about how GPFS accesses/recovers/manages/etc the underlying storage
> based on
> >> this setting.
> >>
> >>
> >>
> >> The documentation doesn’t say much about it other than to consult the
> >> /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this
> >> section:
> >>
> >>
> >>
> >> # Known disk types currently are:
> >>
> >> #
> >>
> >> # powerdisk - EMC power path disk
> >>
> >> # vpath - IBM virtual path disk
> >>
> >> # dmm - Device-Mapper Multipath (DMM)
> >>
> >> # dlmfdrv - Hitachi dlm
> >>
> >> # hdisk - AIX hard disk
> >>
> >> # lv - AIX logical volume. Historical usage only.
> >>
> >> # Not allowed as a new device to mmcrnsd.
> >>
> >> # gpt - GPFS partition on Windows disk
> >>
> >> # generic - Device having no unique failover or multipathing
> >>
> >> # characteristic (predominantly Linux devices).
> >>
> >> # dasd - DASD device (for Linux on z Systems)
> >>
> >>
> >>
> >> We have our storage under Linux Device-Mapper Multipath control (two
> device
> >> paths to all storage, active/passive) and are accessible under
> /dev/mapper,
> >> but the NSD types are current set to ‘generic’ not ‘dmm’. This is
> >> configured in the /var/mmfs/etc/nsddevices file:
> >>
> >>
> >>
> >> if [[ $osName = Linux ]]
> >>
> >> then
> >>
> >> : # Add function to discover disks in the Linux environment.
> >>
> >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}'
> >>
> >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}'
> >>
> >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}'
> >>
> >> fi
> >>
> >>
> >>
> >> Can somebody from IBM explain what the correct setting should be and
> what
> >> differences GPFS does with ‘generic’ vs. ‘dmm’ vs. others?
> >>
> >>
> >>
> >> Thanks in advance!
> >>
> >> -Bryan
> >>
> >>
> >>
> >> ________________________________
> >>
> >>
> >> Note: This email is for the confidential use of the named addressee(s)
> only
> >> and may contain proprietary, confidential or privileged information. If
> you
> >> are not the intended recipient, you are hereby notified that any review,
> >> dissemination or copying of this email is strictly prohibited, and to
> please
> >> notify the sender immediately and destroy this email and any
> attachments.
> >> Email transmission cannot be guaranteed to be secure or error-free. The
> >> Company, therefore, does not make any guarantees as to the completeness
> or
> >> accuracy of this email or any attachments. This email is for
> informational
> >> purposes only and does not constitute a recommendation, offer, request
> or
> >> solicitation of any kind to buy, sell, subscribe, redeem or perform any
> type
> >> of transaction of a financial product.
> >>
> >> _______________________________________________
> >> gpfsug-discuss mailing list
> >> gpfsug-discuss at spectrumscale.org
> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >>
> >>
> >> ________________________________
> >>
> >> Note: This email is for the confidential use of the named addressee(s)
> only
> >> and may contain proprietary, confidential or privileged information. If
> you
> >> are not the intended recipient, you are hereby notified that any review,
> >> dissemination or copying of this email is strictly prohibited, and to
> please
> >> notify the sender immediately and destroy this email and any
> attachments.
> >> Email transmission cannot be guaranteed to be secure or error-free. The
> >> Company, therefore, does not make any guarantees as to the completeness
> or
> >> accuracy of this email or any attachments. This email is for
> informational
> >> purposes only and does not constitute a recommendation, offer, request
> or
> >> solicitation of any kind to buy, sell, subscribe, redeem or perform any
> type
> >> of transaction of a financial product.
> >>
> >> _______________________________________________
> >> gpfsug-discuss mailing list
> >> gpfsug-discuss at spectrumscale.org
> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >>
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
> > ________________________________
> >
> > Note: This email is for the confidential use of the named addressee(s)
> only and may contain proprietary, confidential or privileged information.
> If you are not the intended recipient, you are hereby notified that any
> review, dissemination or copying of this email is strictly prohibited, and
> to please notify the sender immediately and destroy this email and any
> attachments. Email transmission cannot be guaranteed to be secure or
> error-free. The Company, therefore, does not make any guarantees as to the
> completeness or accuracy of this email or any attachments. This email is
> for informational purposes only and does not constitute a recommendation,
> offer, request or solicitation of any kind to buy, sell, subscribe, redeem
> or perform any type of transaction of a financial product.
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180119/0377446f/attachment.htm>
More information about the gpfsug-discuss
mailing list