[gpfsug-discuss] verbs status not working in 5.0.2

Bryan Banister bbanister at jumptrading.com
Tue Jun 11 20:37:52 BST 2019 

This has been brocket for a long time... we too were checking that `mmfsadm test verbs status` reported that RDMA is working.  We don't want nodes that are not using RDMA running in the cluster.

We have decided to just look for the log entry like this:
test_gpfs_rdma_active() {
    [[ "$(grep -c "VERBS RDMA started" /var/adm/ras/mmfs.log.latest)" == "1" ]]
}

Hope that helps,
-Bryan

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Ryan Novosielski
Sent: Tuesday, June 11, 2019 2:33 PM
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] verbs status not working in 5.0.2

[EXTERNAL EMAIL]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

This is not a change I like much either, though can obviously adapt to it. We have used "mmfsadm test verbs status" to confirm that RDMA is working by NHC (https://github.com/mej/nhc) on our compute nodes, and just for a quick check on the command line. Yes, there are the usual caveats, and yes the information is available another way, but a) it's the removal of a convenience that I'm quite sure that -- caveats aside
- -- is not dangerous (it runs every 5 minutes on our system) b) it doesn't match the usage printed out on the command line and c) any other methods are quite a bit more information that then has to be parsed (perhaps also not as light a touch, but I don't know the code), and d) there doesn't seem to be a way now that works on both GPFS V4 and V5 (I confirmed that mmfsadm saferdump verbs | grep verbsRdmaStarted does not on V4).

You'd also mentioned we really shouldn't be using mmfsadm regularly.
Is there a way to get this information out of mmdiag if that is the supported command? Is there a way to do this that works for both V4 and V5?

Philosophy of using mmfsadm aside though, we aren't supposed to rely on syntax for these commands remaining the same, but aren't we supposed to be able to rely on commands not falsely reporting syntax in their own usage message? I'd think at the very least, that's a bug in the "usage" text.

On 12/19/18 5:35 AM, Tomer Perry wrote:
> Hi,
>
> So, with all the usual disclaimers... mmfsadm saferdump verbs is not 
> enough? or even mmfsadm saferdump verbs | grep VerbsRdmaStarted
>
> Regards,
>
> Tomer Perry Scalable I/O Development (Spectrum Scale) email:
> tomp at il.ibm.com 1 Azrieli Center, Tel Aviv 67021, Israel Global
> Tel:    +1 720 3422758 Israel Tel:      +972 3 9188625 Mobile:
> +972 52 2554625
>
>
>
>
> From:        "Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch> To:
> gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date:        19/12/2018 12:22 Subject:        Re: [gpfsug-discuss]
> verbs status not working in 5.0.2 Sent by:
> gpfsug-discuss-bounces at spectrumscale.org
> ----------------------------------------------------------------------
- --
>
>
>
>
> I'd like just one line that says "RDMA ON" or "RMDA OFF" (as was 
> reported more or less by mmfsadm).
>
> I can get info about RMDA using mmdiag, but is much more output to 
> parse (e.g. by a nagios script or just a human eye). Ok, never mind, I 
> understand your explanation and it is not definitely a big issue... it 
> was, above all, a curiosity to understand if the command was modified 
> to get the same behavior as before, but in a different way.
>
> Cheers,
>
> Alvise
>
> ----------------------------------------------------------------------
- --
>
>  *From:* gpfsug-discuss-bounces at spectrumscale.org
> [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Tomer Perry 
> [TOMP at il.ibm.com]* Sent:* Wednesday, December 19, 2018 11:05
> AM* To:* gpfsug main discussion list* Subject:* Re:
> [gpfsug-discuss] verbs status not working in 5.0.2
>
> Changed means it provides some functions/information in a different 
> way. So, I guess the question is what information do you need? ( and 
> "officially" why isn't mmdiag good enough - what is missing. As you 
> probably know, mmfsadm might cause crashes and deadlock from time to 
> time, this is why we're trying to provide "safe ways" to get the 
> required information).
>
>
> Regards,
>
> Tomer Perry Scalable I/O Development (Spectrum Scale) email:
> tomp at il.ibm.com 1 Azrieli Center, Tel Aviv 67021, Israel Global
> Tel:    +1 720 3422758 Israel Tel:      +972 3 9188625 Mobile:
> +972 52 2554625
>
>
>
>
> From:        "Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch> To:
> gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date:        19/12/2018 11:53 Subject:        Re: [gpfsug-discuss]
> verbs status not working in 5.0.2 Sent by:
> gpfsug-discuss-bounces at spectrumscale.org
> ----------------------------------------------------------------------
- --
>
>
>
>
> Hi Tomer, "changed" makes me suppose that it is still possible, but in 
> a different way... am I correct ? if yes, what it is ?
>
> thanks,
>
> Alvise
>
> ----------------------------------------------------------------------
- --
>
>
*
> From:* gpfsug-discuss-bounces at spectrumscale.org
> [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Tomer Perry 
> [TOMP at il.ibm.com]* Sent:* Wednesday, December 19, 2018 10:47
> AM* To:* gpfsug main discussion list* Subject:* Re:
> [gpfsug-discuss] verbs status not working in 5.0.2
>
> Hi,
>
> Yes, as part of the RDMA enhancements in 5.0.X much of the hidden test 
> commands were changed. And since mmfsadm is not externalized none of 
> them is documented ( and the help messages are not consistent as 
> well).
>
> Regards,
>
> Tomer Perry Scalable I/O Development (Spectrum Scale) email:
> tomp at il.ibm.com 1 Azrieli Center, Tel Aviv 67021, Israel Global
> Tel:    +1 720 3422758 Israel Tel:      +972 3 9188625 Mobile:
> +972 52 2554625
>
>
>
>
> From:        Simon Thompson <S.J.Thompson at bham.ac.uk> To:
> gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date:        19/12/2018 11:29 Subject:        Re: [gpfsug-discuss]
> verbs status not working in 5.0.2 Sent by:
> gpfsug-discuss-bounces at spectrumscale.org
> ----------------------------------------------------------------------
- --
>
>
>
>
> Hmm interesting …
>
> # mmfsadm test verbs usage: {udapl | verbs} { status | skipio | 
> noskipio | dump | maxRpcsOut | maxReplysOut | maxRdmasOut }
>
> # mmfsadm test verbs status usage: {udapl | verbs} { status | skipio | 
> noskipio | dump | maxRpcsOut | maxReplysOut | maxRdmasOut
> | config | conn | conndetails | stats | resetstats | ibcntreset |
> ibcntr | ia | pz | psp | evd | lmr | break | qps | inject op cnt err | 
> breakqperr | qperridx idx | breakidx idx}
>
> mmfsadm test verbs config still works though (which includes 
> RdmaStarted flag)
>
> Simon*
>
> From: *<gpfsug-discuss-bounces at spectrumscale.org> on behalf of 
> "alvise.dorigo at psi.ch" <alvise.dorigo at psi.ch>* Reply-To:
> *"gpfsug-discuss at spectrumscale.org"
> <gpfsug-discuss at spectrumscale.org>* Date: *Wednesday, 19 December
> 2018 at 08:51* To: *"gpfsug-discuss at spectrumscale.org"
> <gpfsug-discuss at spectrumscale.org>* Subject: *[gpfsug-discuss] verbs 
> status not working in 5.0.2
>
> Hi, in GPFS 5.0.2 I cannot run anymore "mmfsadm test verbs
> status":
>
> [root at sf-dss-1 ~]# mmdiag --version ; mmfsadm test verbs status
>
> === mmdiag: version === Current GPFS build: "4.2.3.7 ". Built on Feb 
> 15 2018 at 11:38:38 Running 62 days 11 hours 24 minutes 35 secs, pid 
> 7510 VERBS RDMA status: started
>
> [root at sf-export-2 ~]# mmdiag --version ; mmfsadm test verbs status
>
> === mmdiag: version === Current GPFS build: "5.0.2.1 ". Built on Oct 
> 24 2018 at 21:23:46 Running 10 minutes 24 secs, pid 3570 usage:
> {udapl | verbs} { status | skipio | noskipio | dump | maxRpcsOut | 
> maxReplysOut | maxRdmasOut | config | conn | conndetails | stats | 
> resetstats | ibcntreset | ibcntr | ia | pz | psp | evd | lmr | break | 
> qps | inject op cnt err | breakqperr | qperridx idx | breakidx idx}
>
>
> Is it a known problem or am I doing something wrong ?
>
> Thanks,
>
> Alvise_______________________________________________
> gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org_ 
> __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>
>
> _______________________________________________ gpfsug-discuss mailing 
> list gpfsug-discuss at spectrumscale.org_ 
> __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>
>
> _______________________________________________ gpfsug-discuss mailing 
> list gpfsug-discuss at spectrumscale.org 
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
>
> _______________________________________________ gpfsug-discuss mailing 
> list gpfsug-discuss at spectrumscale.org 
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>

- --
 ____
 || \\UTGERS,     |----------------------*O*------------------------
 ||_// the State  |    Ryan Novosielski - novosirj at rutgers.edu
 || \\ University | Sr. Technologist - 973/972.0922 ~*~ RBHS Campus
 ||  \\    of NJ  | Office of Advanced Res. Comp. - MSB C630, Newark
      `'
-----BEGIN PGP SIGNATURE-----

iF0EARECAB0WIQST3OUUqPn4dxGCSm6Zv6Bp0RyxvgUCXQAB1AAKCRCZv6Bp0Ryx
vhPDAKCZFKcsFcbNk8MBZvfr6Oz8C3+C5wCgvwXwHwX0S6SKI7NoRTszLPR2n/E=
=Qxja
-----END PGP SIGNATURE-----
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

More information about the gpfsug-discuss mailing list