[gpfsug-discuss] wait for permission to append to log

Sven Oehme oehmes at us.ibm.com
Tue Oct 14 17:22:41 BST 2014


your GSS code version is very backlevel. 

can you please send me the output of mmlsrecoverygroup RGNAME -L --pdisk 
as well as mmlsconfig and mmlsfs all

thx. Sven

------------------------------------------
Sven Oehme 
Scalable Storage Research 
email: oehmes at us.ibm.com 
Phone: +1 (408) 824-8904 
IBM Almaden Research Lab 
------------------------------------------



From:   Salvatore Di Nardo <sdinardo at ebi.ac.uk>
To:     gpfsug-discuss at gpfsug.org
Date:   10/14/2014 08:23 AM
Subject:        Re: [gpfsug-discuss] wait for permission to append to log
Sent by:        gpfsug-discuss-bounces at gpfsug.org




On 14/10/14 15:51, Sven Oehme wrote:
it means there is contention on inserting data into the fast write log on 
the GSS Node, which could be config or workload related 
what GSS code version are you running 
[root at ebi5-251 ~]# mmdiag --version

=== mmdiag: version ===
Current GPFS build: "3.5.0-11 efix1 (888041)".
Built on Jul  9 2013 at 18:03:32
Running 6 days 2 hours 10 minutes 35 secs



and how are the nodes connected with each other (Ethernet or IB) ? 
ethernet. they use the same bonding (4x10Gb/s) where the data is passing. 
We don't have admin dedicated network

[root at gss03a ~]# mmlscluster 

GPFS cluster information
========================
  GPFS cluster name:         GSS.ebi.ac.uk
  GPFS cluster id:           17987981184946329605
  GPFS UID domain:           GSS.ebi.ac.uk
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    gss01a.ebi.ac.uk
  Secondary server:  gss02b.ebi.ac.uk

 Node  Daemon node name    IP address  Admin node name     Designation
-----------------------------------------------------------------------
   1   gss01a.ebi.ac.uk    10.7.28.2   gss01a.ebi.ac.uk    quorum-manager
   2   gss01b.ebi.ac.uk    10.7.28.3   gss01b.ebi.ac.uk    quorum-manager
   3   gss02a.ebi.ac.uk    10.7.28.67  gss02a.ebi.ac.uk    quorum-manager
   4   gss02b.ebi.ac.uk    10.7.28.66  gss02b.ebi.ac.uk    quorum-manager
   5   gss03a.ebi.ac.uk    10.7.28.34  gss03a.ebi.ac.uk    quorum-manager
   6   gss03b.ebi.ac.uk    10.7.28.35  gss03b.ebi.ac.uk    quorum-manager


Note: The 3 node "pairs" (gss01, gss02 and gss03)  are in different subnet 
because of datacenter constraints ( They are not physically in the same 
row, and due to network constraints was not possible to put them in the 
same subnet). The packets are routed, but should not be a problem as there 
is 160Gb/s bandwidth between them.

Regards,
Salvatore



------------------------------------------
Sven Oehme 
Scalable Storage Research 
email: oehmes at us.ibm.com 
Phone: +1 (408) 824-8904 
IBM Almaden Research Lab 
------------------------------------------ 



From:        Salvatore Di Nardo <sdinardo at ebi.ac.uk> 
To:        gpfsug main discussion list <gpfsug-discuss at gpfsug.org> 
Date:        10/14/2014 07:40 AM 
Subject:        [gpfsug-discuss] wait for permission to append to log 
Sent by:        gpfsug-discuss-bounces at gpfsug.org 



hello all,
could someone explain me the meaning of those waiters?

gss02b.ebi.ac.uk:  0x7F21EA8541B0 waiting 0.122786709 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA5F4EC0 waiting 0.122770807 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA9BD1A0 waiting 0.122115115 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA32FF30 waiting 0.121371877 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA6A1BA0 waiting 0.119322600 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA2E4330 waiting 0.118216774 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA72E930 waiting 0.117961594 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA6539C0 waiting 0.116133122 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA3D3490 waiting 0.116103642 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA85A060 waiting 0.115137978 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA4C84A0 waiting 0.115046631 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA229310 waiting 0.114498225 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA2AB630 waiting 0.113035120 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA83D9E0 waiting 0.112934666 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA736DC0 waiting 0.112834203 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA3A2C20 waiting 0.111498004 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA3B2250 waiting 0.111309423 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EAABDF10 waiting 0.110939219 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EAA27A00 waiting 0.110025022 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EAA8D6A0 waiting 0.109176110 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA2B3AC0 waiting 0.109025355 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA2080D0 waiting 0.108702893 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA3AC3A0 waiting 0.107691494 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EAB460E0 waiting 0.106003854 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA2093C0 waiting 0.105781682 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA6FBAE0 waiting 0.105696084 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA573E90 waiting 0.105182795 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA4191E0 waiting 0.104335963 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA82AAE0 waiting 0.104079258 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA538BB0 waiting 0.103798658 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EAA15DF0 waiting 0.102778144 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA57C320 waiting 0.100503136 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA802700 waiting 0.100499392 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EAA5F410 waiting 0.100489143 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA861200 waiting 0.100351636 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA54BAB0 waiting 0.099615942 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EAAAFBD0 waiting 0.099477387 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA657290 waiting 0.099123599 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA2BD240 waiting 0.099074074 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA205AF0 waiting 0.097532291 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA477CE0 waiting 0.097311417 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA2F9810 waiting 0.096209425 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA463AF0 waiting 0.096143868 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA8B2CB0 waiting 0.094143517 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA7D1E90 waiting 0.093156759 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EAB473D0 waiting 0.093154775 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EAB03C60 waiting 0.092952495 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk:  0x7F21EA8766E0 waiting 0.092908405 seconds, 
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750) 
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'

Does it means that the vdisk logs are struggling?

Regards,
Salvatore

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20141014/a578f87a/attachment.htm>


More information about the gpfsug-discuss mailing list