[gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file

Thu Jan 18 16:32:58 GMT 2018

What does the /var/mmfs/gen/mmsdrfs file show?

[root at service5 ~]# grep SG_DISK /var/mmfs/gen/mmsdrfs | cut -f 15 -d ':'
dmm
dmm
dmm
dmm

    On Thursday, January 18, 2018, 10:49:32 AM EST, Bryan Banister <bbanister at jumptrading.com> wrote:  

 #yiv4648160129 #yiv4648160129 -- _filtered #yiv4648160129 {panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv4648160129 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv4648160129 {font-family:Verdana;panose-1:2 11 6 4 3 5 4 4 2 4;} _filtered #yiv4648160129 {}#yiv4648160129 #yiv4648160129 p.yiv4648160129MsoNormal, #yiv4648160129 li.yiv4648160129MsoNormal, #yiv4648160129 div.yiv4648160129MsoNormal {margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv4648160129 a:link, #yiv4648160129 span.yiv4648160129MsoHyperlink {color:blue;text-decoration:underline;}#yiv4648160129 a:visited, #yiv4648160129 span.yiv4648160129MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv4648160129 p.yiv4648160129msonormal, #yiv4648160129 li.yiv4648160129msonormal, #yiv4648160129 div.yiv4648160129msonormal {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv4648160129 p.yiv4648160129msochpdefault, #yiv4648160129 li.yiv4648160129msochpdefault, #yiv4648160129 div.yiv4648160129msochpdefault {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv4648160129 span.yiv4648160129msohyperlink {}#yiv4648160129 span.yiv4648160129msohyperlinkfollowed {}#yiv4648160129 span.yiv4648160129emailstyle17 {}#yiv4648160129 p.yiv4648160129msonormal1, #yiv4648160129 li.yiv4648160129msonormal1, #yiv4648160129 div.yiv4648160129msonormal1 {margin:0in;margin-bottom:.0001pt;font-size:11.0pt;}#yiv4648160129 span.yiv4648160129msohyperlink1 {color:#0563C1;text-decoration:underline;}#yiv4648160129 span.yiv4648160129msohyperlinkfollowed1 {color:#954F72;text-decoration:underline;}#yiv4648160129 span.yiv4648160129emailstyle171 {color:windowtext;}#yiv4648160129 p.yiv4648160129msochpdefault1, #yiv4648160129 li.yiv4648160129msochpdefault1, #yiv4648160129 div.yiv4648160129msochpdefault1 {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv4648160129 span.yiv4648160129EmailStyle28 {color:#1F497D;}#yiv4648160129 .yiv4648160129MsoChpDefault {font-size:10.0pt;} _filtered #yiv4648160129 {margin:1.0in 1.0in 1.0in 1.0in;}#yiv4648160129 div.yiv4648160129WordSection1 {}#yiv4648160129 
Hi Jim,

Our NSDs are reporting the /dev/mapper/ devices in the `mmlsnsd -X` output, but due to the nsddevices file configuration they are all configured as ‘generic’ Devtype.

-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org]On Behalf Of Jim Doherty
Sent: Wednesday, January 17, 2018 6:28 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file

Note: External Email

Run a mmlsnsd -X   I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN,   not the /dev/mapper/ path.   In our case the device is setup as dmm 

[root at service5 ~]# mmlsnsd -X

 Disk name    NSD volume ID      Device         Devtype  Node name                Remarks          
---------------------------------------------------------------------------------------------------
 volume1      0972B6CD587CD8E0   /dev/dm-0      dmm      service5.pok.stglabs.ibm.com server node
 volume1      0972B6CD587CD8E0   /dev/dm-0      dmm      service6.pok.stglabs.ibm.com server node
 volume2      0972B6CE587CD8E4   /dev/dm-4      dmm      service5.pok.stglabs.ibm.com server node
 volume2      0972B6CE587CD8E4   /dev/dm-3      dmm      service6.pok.stglabs.ibm.com server node
 volume3      0972B6CD587CD8E7   /dev/dm-1      dmm      service5.pok.stglabs.ibm.com server node
 volume3      0972B6CD587CD8E7   /dev/dm-2      dmm      service6.pok.stglabs.ibm.com server node
 volume4      0972B6CE587CF625   /dev/dm-3      dmm      service5.pok.stglabs.ibm.com server node
 volume4      0972B6CE587CF625   /dev/dm-4      dmm      service6.pok.stglabs.ibm.com server node

[root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK
%%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com:::::
[root at service5 ~]# 

If you run an tspreparedisk -s  it will show you all of the paths.

[root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0
0972B6CD587CD8E0 /dev/sda generic  
0972B6CD587CD8E0 /dev/sdk generic  
0972B6CD587CD8E0 /dev/sdu generic  
0972B6CD587CD8E0 /dev/sdah generic  
0972B6CD587CD8E0 /dev/dm-0 dmm  
[root at service5 ~]# 

Jim

Jim

On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister <bbanister at jumptrading.com> wrote:

Hi all,

We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting.

The documentation doesn’t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section:

# Known disk types currently are:

#

#   powerdisk  - EMC power path disk

#   vpath      - IBM virtual path disk

#   dmm        - Device-Mapper Multipath (DMM)

#   dlmfdrv    - Hitachi dlm

#   hdisk      - AIX hard disk

#   lv         - AIX logical volume.  Historical usage only.

#                Not allowed as a new device to mmcrnsd.

#   gpt        - GPFS partition on Windows disk

#   generic    - Device having no unique failover or multipathing

#                characteristic (predominantly Linux devices).

#   dasd       - DASD device (for Linux on z Systems)

We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ‘generic’ not ‘dmm’.  This is configured in the /var/mmfs/etc/nsddevices file:

if [[ $osName = Linux ]]

then

  : # Add function to discover disks in the Linux environment.

  ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}'

  ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 "generic"}'

  ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}'

fi

Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ‘generic’ vs. ‘dmm’ vs. others?

Thanks in advance!

-Bryan

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180118/240241c5/attachment.htm>