[gpfsug-discuss] wait for permission to append to log
Sven Oehme
oehmes at us.ibm.com
Tue Oct 14 17:22:41 BST 2014
your GSS code version is very backlevel.
can you please send me the output of mmlsrecoverygroup RGNAME -L --pdisk
as well as mmlsconfig and mmlsfs all
thx. Sven
------------------------------------------
Sven Oehme
Scalable Storage Research
email: oehmes at us.ibm.com
Phone: +1 (408) 824-8904
IBM Almaden Research Lab
------------------------------------------
From: Salvatore Di Nardo <sdinardo at ebi.ac.uk>
To: gpfsug-discuss at gpfsug.org
Date: 10/14/2014 08:23 AM
Subject: Re: [gpfsug-discuss] wait for permission to append to log
Sent by: gpfsug-discuss-bounces at gpfsug.org
On 14/10/14 15:51, Sven Oehme wrote:
it means there is contention on inserting data into the fast write log on
the GSS Node, which could be config or workload related
what GSS code version are you running
[root at ebi5-251 ~]# mmdiag --version
=== mmdiag: version ===
Current GPFS build: "3.5.0-11 efix1 (888041)".
Built on Jul 9 2013 at 18:03:32
Running 6 days 2 hours 10 minutes 35 secs
and how are the nodes connected with each other (Ethernet or IB) ?
ethernet. they use the same bonding (4x10Gb/s) where the data is passing.
We don't have admin dedicated network
[root at gss03a ~]# mmlscluster
GPFS cluster information
========================
GPFS cluster name: GSS.ebi.ac.uk
GPFS cluster id: 17987981184946329605
GPFS UID domain: GSS.ebi.ac.uk
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
GPFS cluster configuration servers:
-----------------------------------
Primary server: gss01a.ebi.ac.uk
Secondary server: gss02b.ebi.ac.uk
Node Daemon node name IP address Admin node name Designation
-----------------------------------------------------------------------
1 gss01a.ebi.ac.uk 10.7.28.2 gss01a.ebi.ac.uk quorum-manager
2 gss01b.ebi.ac.uk 10.7.28.3 gss01b.ebi.ac.uk quorum-manager
3 gss02a.ebi.ac.uk 10.7.28.67 gss02a.ebi.ac.uk quorum-manager
4 gss02b.ebi.ac.uk 10.7.28.66 gss02b.ebi.ac.uk quorum-manager
5 gss03a.ebi.ac.uk 10.7.28.34 gss03a.ebi.ac.uk quorum-manager
6 gss03b.ebi.ac.uk 10.7.28.35 gss03b.ebi.ac.uk quorum-manager
Note: The 3 node "pairs" (gss01, gss02 and gss03) are in different subnet
because of datacenter constraints ( They are not physically in the same
row, and due to network constraints was not possible to put them in the
same subnet). The packets are routed, but should not be a problem as there
is 160Gb/s bandwidth between them.
Regards,
Salvatore
------------------------------------------
Sven Oehme
Scalable Storage Research
email: oehmes at us.ibm.com
Phone: +1 (408) 824-8904
IBM Almaden Research Lab
------------------------------------------
From: Salvatore Di Nardo <sdinardo at ebi.ac.uk>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Date: 10/14/2014 07:40 AM
Subject: [gpfsug-discuss] wait for permission to append to log
Sent by: gpfsug-discuss-bounces at gpfsug.org
hello all,
could someone explain me the meaning of those waiters?
gss02b.ebi.ac.uk: 0x7F21EA8541B0 waiting 0.122786709 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA5F4EC0 waiting 0.122770807 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA9BD1A0 waiting 0.122115115 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA32FF30 waiting 0.121371877 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA6A1BA0 waiting 0.119322600 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA2E4330 waiting 0.118216774 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA72E930 waiting 0.117961594 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA6539C0 waiting 0.116133122 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA3D3490 waiting 0.116103642 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA85A060 waiting 0.115137978 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA4C84A0 waiting 0.115046631 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA229310 waiting 0.114498225 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA2AB630 waiting 0.113035120 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA83D9E0 waiting 0.112934666 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA736DC0 waiting 0.112834203 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA3A2C20 waiting 0.111498004 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA3B2250 waiting 0.111309423 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EAABDF10 waiting 0.110939219 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EAA27A00 waiting 0.110025022 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EAA8D6A0 waiting 0.109176110 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA2B3AC0 waiting 0.109025355 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA2080D0 waiting 0.108702893 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA3AC3A0 waiting 0.107691494 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EAB460E0 waiting 0.106003854 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA2093C0 waiting 0.105781682 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA6FBAE0 waiting 0.105696084 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA573E90 waiting 0.105182795 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA4191E0 waiting 0.104335963 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA82AAE0 waiting 0.104079258 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA538BB0 waiting 0.103798658 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EAA15DF0 waiting 0.102778144 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA57C320 waiting 0.100503136 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA802700 waiting 0.100499392 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EAA5F410 waiting 0.100489143 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA861200 waiting 0.100351636 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA54BAB0 waiting 0.099615942 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EAAAFBD0 waiting 0.099477387 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA657290 waiting 0.099123599 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA2BD240 waiting 0.099074074 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA205AF0 waiting 0.097532291 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA477CE0 waiting 0.097311417 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA2F9810 waiting 0.096209425 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA463AF0 waiting 0.096143868 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA8B2CB0 waiting 0.094143517 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA7D1E90 waiting 0.093156759 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EAB473D0 waiting 0.093154775 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EAB03C60 waiting 0.092952495 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
gss02b.ebi.ac.uk: 0x7F21EA8766E0 waiting 0.092908405 seconds,
NSDThread: on ThCond 0x7F2114005750 (0x7F2114005750)
(VdiskLogAppendCondvar), reason 'wait for permission to append to log'
Does it means that the vdisk logs are struggling?
Regards,
Salvatore
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20141014/a578f87a/attachment.htm>
More information about the gpfsug-discuss
mailing list