[gpfsug-discuss] How to use RHEL 7 mdadm NVMe devices with Spectrum Scale 4.2.3.10?

IBM Spectrum Scale scale at us.ibm.com
Fri Nov 16 12:31:57 GMT 2018


Note, RHEL 7.6 is not yet a supported platform for Spectrum Scale so you 
may want to use RHEL 7.5 or wait for RHEL 7.6 to be supported. 

Using "generic" for the device type should be the proper option here.

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
. 

If your query concerns a potential software error in Spectrum Scale (GPFS) 
and you have an IBM software maintenance contract please contact 
1-800-237-5511 in the United States or your local IBM Service Center in 
other countries. 

The forum is informally monitored as time permits and should not be used 
for priority messages to the Spectrum Scale (GPFS) team.



From:   "Uwe Falke" <UWEFALKE at de.ibm.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   11/16/2018 04:35 AM
Subject:        Re: [gpfsug-discuss] How to use RHEL 7 mdadm NVMe devices 
with    Spectrum        Scale 4.2.3.10?
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Hi, 

Having mentioned nsddevices, I do not know how Scale treats different 
device types differently, so generic would be a fine choice unless 
development tells you differently.
Currently known device types are listed in the comments of the script 
/usr/lpp/mmfs/bin/mmdevdiscover

 
Mit freundlichen Grüßen / Kind regards

 
Dr. Uwe Falke
 
IT Specialist
High Performance Computing Services / Integrated Technology Services / 
Data Center Services
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland
Rathausstr. 7
09111 Chemnitz
Phone: +49 371 6978 2165
Mobile: +49 175 575 2877
E-Mail: uwefalke at de.ibm.com
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland Business & Technology Services GmbH / Geschäftsführung: 
Thomas Wolter, Sven Schooß
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, 
HRB 17122 




From:   Uwe Falke/Germany/IBM
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   16/11/2018 10:19
Subject:        Re: [gpfsug-discuss] How to use RHEL 7 mdadm NVMe devices 
with Spectrum   Scale 4.2.3.10?


Hi Lance, 
you might need to use 
/var/mmfs/etc/nsddevices
to tell GPFS about these devices (template in 
/usr/lpp/mmfs/samples/nsddevices.sample)


 
Mit freundlichen Grüßen / Kind regards

 
Dr. Uwe Falke
 
IT Specialist
High Performance Computing Services / Integrated Technology Services / 
Data Center Services
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland
Rathausstr. 7
09111 Chemnitz
Phone: +49 371 6978 2165
Mobile: +49 175 575 2877
E-Mail: uwefalke at de.ibm.com
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland Business & Technology Services GmbH / Geschäftsführung: 
Thomas Wolter, Sven Schooß
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, 
HRB 17122 





From:   Lance Nakata <LNakata at SLAC.STANFORD.EDU>
To:     gpfsug-discuss at spectrumscale.org
Cc:     "Jon L. Bergman" <jonl at SLAC.STANFORD.EDU>
Date:   16/11/2018 04:14
Subject:        [gpfsug-discuss] How to use RHEL 7 mdadm NVMe devices with 

Spectrum        Scale 4.2.3.10?
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



We have a Dell R740xd with 24 x 1TB NVMe SSDs in the internal slots. Since 

PERC
RAID cards don't see these devices, we are using mdadm software RAID to 
build
NSDs.  We took 12 NVMe SSDs and used mdadm to create a 10 + 1 + 1 hot 
spare RAID
5 stripe named /dev/md101.  We took the other 12 NVMe SSDs and created a 
similar
/dev/md102.

mmcrnsd worked without errors.  The problem is that Spectrum Scale does 
not see
the /dev/md10x devices as proper NSDs; the Device and Devtype columns are 
blank:

host2:~> sudo mmlsnsd -X

 Disk name    NSD volume ID      Device         Devtype  Node name 
Remarks
---------------------------------------------------------------------------------------------------
 nsd0001      864FD12858A36E79   /dev/sdb       generic 
host1.slac.stanford.edu server node
 nsd0002      864FD12858A36E7A   /dev/sdc       generic 
host1.slac.stanford.edu server node
 nsd0021      864FD1285956B0A7   /dev/sdd       generic 
host1.slac.stanford.edu server node
 nsd0251a     864FD1545BD0CCDF   /dev/dm-9      dmm 
host2.slac.stanford.edu server node
 nsd0251b     864FD1545BD0CCE0   /dev/dm-11     dmm 
host2.slac.stanford.edu server node
 nsd0252a     864FD1545BD0CCE1   /dev/dm-10     dmm 
host2.slac.stanford.edu server node
 nsd0252b     864FD1545BD0CCE2   /dev/dm-8      dmm 
host2.slac.stanford.edu server node
 nsd02nvme1   864FD1545BEC5D72   -              - host2.slac.stanford.edu 
(not found) server node
 nsd02nvme2   864FD1545BEC5D73   -              - host2.slac.stanford.edu 
(not found) server node

I know we can access the internal NVMe devices by their individual 
/dev/nvmeXX
paths, but non-ESS-based Spectrum Scale does not have built-in RAID
functionality.  Hence, the only option in that scenario is replication, 
which is
expensive and won't give us enough usable space.

Software Environment:
RHEL 7.6 with kernel 3.10.0-862.14.4.el7.x86_64
Spectrum Scale 4.2.3.10

Spectrum Scale Support has implied we can't use mdadm for NVMe devices. Is 

that
really true?  Does anyone use an mdadm-based NVMe config?  If so, did you 
have
to do some kind of customization to get it working?

Thank you,

Lance Nakata
SLAC National Accelerator Laboratory
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss








_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20181116/64612a58/attachment.htm>


More information about the gpfsug-discuss mailing list