[gpfsug-discuss] High I/O wait times

[Frederick Stock]

Are you seeing similar values for all the nodes or just some of them?  One 
possible issue is how the NSD queues are configured on the NSD servers. 
You can see this with the output of "mmfsadm dump nsd".  There are queues 
for LARGE IOs (greater than 64K) and queues for SMALL IOs (64K or less). 
Check the highest pending values to see if many IOs are queueing.  There 
are a couple of options to fix this but rather than explain them I suggest 
you look for information about NSD queueing on the developerWorks site. 
There has been information posted there that should prove helpful.

Fred
__________________________________________________
Fred Stock | IBM Pittsburgh Lab | 720-430-8821
stockf at us.ibm.com



From:   "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   07/03/2018 03:49 PM
Subject:        [gpfsug-discuss] High I/O wait times
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Hi all, 

We are experiencing some high I/O wait times (5 - 20 seconds!) on some of 
our NSDs as reported by “mmdiag —iohist" and are struggling to understand 
why.  One of the confusing things is that, while certain NSDs tend to show 
the problem more than others, the problem is not consistent … i.e. the 
problem tends to move around from NSD to NSD (and storage array to storage 
array) whenever we check … which is sometimes just a few minutes apart.

In the past when I have seen “mmdiag —iohist” report high wait times like 
this it has *always* been hardware related.  In our environment, the most 
common cause has been a battery backup unit on a storage array controller 
going bad and the storage array switching to write straight to disk.  But 
that’s *not* happening this time. 

Is there anything within GPFS / outside of a hardware issue that I should 
be looking for??  Thanks!

—
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and 
Education
Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180703/d349e318/attachment.htm>