[gpfsug-discuss] gpfs performance monitoring
Orlando Richards
orlando.richards at ed.ac.uk
Thu Sep 4 15:14:02 BST 2014
On 04/09/14 15:07, Salvatore Di Nardo wrote:
>
> On 04/09/14 14:54, Orlando Richards wrote:
>>
>>
>> On 04/09/14 14:32, Salvatore Di Nardo wrote:
>>> Sorry to bother you again but dstat have some issues with the plugin:
>>>
>>> [root at gss01a util]# dstat --gpfs
>>> /usr/bin/dstat:1672: DeprecationWarning: os.popen3 is
>>> deprecated. Use the subprocess module.
>>> pipes[cmd] = os.popen3(cmd, 't', 0)
>>> Module dstat_gpfs failed to load. (global name 'select' is not
>>> defined)
>>> None of the stats you selected are available.
>>>
>>> I found this solution , but involve dstat recompile....
>>>
>>> https://github.com/dagwieers/dstat/issues/44
>>>
>>> Are you aware about any easier solution (we use RHEL6.3) ?
>>>
>>
>> This worked for me the other day on a dev box I was poking at:
>>
>> # rm /usr/share/dstat/dstat_gpfsops*
>>
>> # cp /usr/lpp/mmfs/samples/util/dstat_gpfsops.py.dstat.0.7
>> /usr/share/dstat/dstat_gpfsops.py
>>
>> # dstat --gpfsops
>> /usr/bin/dstat:1672: DeprecationWarning: os.popen3 is deprecated. Use
>> the subprocess module.
>> pipes[cmd] = os.popen3(cmd, 't', 0)
>> ---------------------------gpfs-vfs-ops--------------------------#-----------------------------gpfs-disk-i/o-----------------------------
>>
>> cr del op/cl rd wr trunc fsync looku gattr sattr other
>> mb_rd mb_wr pref wrbeh steal clean sync revok logwr logda oth_r oth_w
>> 0 0 0 0 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0 0 0 0
>>
>> ...
>>
>
> NICE!! The only problem is that the box seems lacking those python scripts:
>
> ls /usr/lpp/mmfs/samples/util/
> makefile README tsbackup tsbackup.C tsbackup.h tsfindinode
> tsfindinode.c tsgetusage tsgetusage.c tsinode tsinode.c
> tslistall tsreaddir tsreaddir.c tstimes tstimes.c
>
It came from the gpfs.base rpm:
# rpm -qf /usr/lpp/mmfs/samples/util/dstat_gpfsops.py.dstat.0.7
gpfs.base-3.5.0-13.x86_64
> Do you mind sending me those py files? They should be 3 as i see e gpfs
> options: gpfs, gpfs-ops, gpfsops (dunno what are the differences )
>
Only the gpfsops.py is included in the bundle - one for dstat 0.6 and
one for dstat 0.7.
I've attached it to this mail as well (it seems to be GPL'd).
> Regards,
> Salvatore
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
--
--
Dr Orlando Richards
Research Facilities (ECDF) Systems Leader
Information Services
IT Infrastructure Division
Tel: 0131 650 4994
skype: orlando.richards
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
-------------- next part --------------
#
# Copyright (C) 2009, 2010 IBM Corporation
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
#
global string, select, os, re, fnmatch
import string, select, os, re, fnmatch
# Dstat class to display selected gpfs performance counters returned by the
# mmpmon "vfs_s", "ioc_s", "vio_s", "vflush_s", and "lroc_s" commands.
#
# The set of counters displayed can be customized via environment variables:
#
# DSTAT_GPFS_WHAT
#
# Selects which of the five mmpmon commands to display.
# It is a comma separated list of any of the following:
# "vfs": show mmpmon "vfs_s" counters
# "ioc": show mmpmon "ioc_s" counters related to NSD client I/O
# "nsd": show mmpmon "ioc_s" counters related to NSD server I/O
# "vio": show mmpmon "vio_s" counters
# "vflush": show mmpmon "vflush_s" counters
# "lroc": show mmpmon "lroc_s" counters
# "all": equivalent to specifying all of the above
#
# Example:
#
# DSTAT_GPFS_WHAT=vfs,lroc dstat -M gpfsops
#
# will display counters for mmpmon "vfs_s" and "lroc" commands.
#
# The default setting is "vfs,ioc", i.e., by default only "vfs_s" and NSD
# client related "ioc_s" counters are displayed.
#
# DSTAT_GPFS_VFS
# DSTAT_GPFS_IOC
# DSTAT_GPFS_VIO
# DSTAT_GPFS_VFLUSH
# DSTAT_GPFS_LROC
#
# Allow finer grain control over exactly which values will be displayed for
# each of the five mmpmon commands. Each variable is a comma separated list
# of counter names with optional column header string.
#
# Example:
#
# export DSTAT_GPFS_VFS='create, remove, rd/wr=read+write'
# export DSTAT_GPFS_IOC='sync*, logwrap*, oth_rd=*_rd, oth_wr=*_wr'
# dstat -M gpfsops
#
# Under "vfs-ops" this will display three columns, showing creates, deletes
# (removes), and a third column labelled "rd/wr" with a combined count of
# read and write operations.
# Under "disk-i/o" it will display four columns, showing all disk I/Os
# initiated by sync, and log wrap, plus two columns labeled "oth_rd" and
# "oth_wr" showing counts of all other disk reads and disk writes,
# respectively.
#
# Note: setting one of these environment variables overrides the
# corrosponding setting in DSTAT_GPFS_WHAT. For example, setting
# DSTAT_GPFS_VFS="" will omit all "vfs_s" counters regardless of whether
# "vfs" appears in DSTAT_GPFS_WHAT or not.
#
# Counter sets are specified as a comma-separated list of entries of one
# of the following forms
#
# counter
# label = counter
# label = counter1 + counter2 + ...
#
# If no label is specified, the name of the counter is used as the column
# header (truncated to 5 characters).
# Counter names may contain shell-style wildcards. For example, the
# pattern "sync*" matches the two ioc_s counters "sync_rd" and "sync_wr" and
# therefore produce a column containing the combined count of disk reads and
# disk writes initiated by sync. If a counter appears in or matches a name
# pattern in more than one entry, it is included only in the count under the
# first entry in which it appears. For example, adding an entry "other = *"
# at the end of the list will add a column labeled "other" that shows the
# sum of all counter values *not* included in any of the previous columns.
#
# DSTAT_GPFS_LIST=1 dstat -M gpfsops
#
# This will show all available counter names and the default definition
# for which sets of counter values are displayed.
#
# An alternative to setting environment variables is to create a file
# ~/.dstat_gpfs_rc
# with python statements that sets any of the following variables
# vfs_wanted: equivalent to setting DSTAT_GPFS_VFS
# ioc_wanted: equivalent to setting DSTAT_GPFS_IOC
# vio_wanted: equivalent to setting DSTAT_GPFS_VIO
# vflush_wanted: equivalent to setting DSTAT_GPFS_VFLUSH
# lroc_wanted: equivalent to setting DSTAT_GPFS_LROC
#
# For example, the following ~/.dstat_gpfs_rc file will produce the same
# result as the environment variables in the example above:
#
# vfs_wanted = 'create, remove, rd/wr=read+write'
# ioc_wanted = 'sync*, logwrap*, oth_rd=*_rd, oth_wr=*_wr'
#
# See also the default vfs_wanted, ioc_wanted, and vio_wanted settings in
# the dstat_gpfsops __init__ method below.
class dstat_plugin(dstat):
def __init__(self):
# list of all stats counters returned by mmpmon "vfs_s", "ioc_s", "vio_s", "vflush_s", and "lroc_s"
# always ignore the first few chars like : io_s _io_s_ _n_ 172.31.136.2 _nn_ mgmt001st001 _rc_ 0 _t_ 1322526286 _tu_ 415518
vfs_keys = ('_access_', '_close_', '_create_', '_fclear_', '_fsync_', '_fsync_range_', '_ftrunc_', '_getattr_',
'_link_', '_lockctl_', '_lookup_', '_map_lloff_', '_mkdir_', '_mknod_', '_open_', '_read_', '_write_',
'_mmapRead_', '_mmapWrite_', '_aioRead_', '_aioWrite_','_readdir_', '_readlink_', '_readpage_', '_remove_', '_rename_', '_rmdir_',
'_setacl_', '_setattr_', '_symlink_', '_unmap_', '_writepage_', '_tsfattr_', '_tsfsattr_', '_flock_',
'_setxattr_', '_getxattr_', '_listxattr_', '_removexattr_', '_encode_fh_', '_decode_fh_', '_get_dentry_',
'_get_parent_', '_mount_', '_statfs_', '_sync_', '_vget_')
ioc_keys = ('_other_rd_', '_other_wr_','_mb_rd_', '_mb_wr_', '_steal_rd_', '_steal_wr_', '_cleaner_rd_', '_cleaner_wr_',
'_sync_rd_', '_sync_wr_', '_logwrap_rd_', '_logwrap_wr_', '_revoke_rd_', '_revoke_wr_',
'_prefetch_rd_', '_prefetch_wr_', '_logdata_rd_', '_logdata_wr_', '_nsdworker_rd_', '_nsdworker_wr_','_nsdlocal_rd_','_nsdlocal_wr_',
'_vdisk_rd_','_vdisk_wr_', '_pdisk_rd_','_pdisk_wr_', '_logtip_rd_', '_logtip_wr_')
vio_keys = ('_r_', '_sw_', '_mw_', '_pfw_', '_ftw_', '_fuw_', '_fpw_', '_m_', '_s_', '_l_', '_rgd_', '_meta_')
vflush_keys = ('_ndt_', '_ngdb_', '_nfwlmb_', '_nfipt_', '_nfwwt_', '_ahwm_', '_susp_', '_uwrttf_', '_fftc_', '_nalth_', '_nasth_', '_nsigth_', '_ntgtth_')
lroc_keys = ('_Inode_s_', '_Inode_sf_', '_Inode_smb_', '_Inode_r_', '_Inode_rf_', '_Inode_rmb_', '_Inode_i_', '_Inode_imb_',
'_Directory_s_', '_Directory_sf_', '_Directory_smb_', '_Directory_r_', '_Directory_rf_', '_Directory_rmb_', '_Directory_i_', '_Directory_imb_',
'_Data_s_', '_Data_sf_', '_Data_smb_', '_Data_r_', '_Data_rf_', '_Data_rmb_', '_Data_i_', '_Data_imb_',
'_agt_i_', '_agt_i_rm_', '_agt_i_rM_', '_agt_i_ra_', '_agt_r_', '_agt_r_rm_', '_agt_r_rM_', '_agt_r_ra_',
'_ssd_w_', '_ssd_w_p_', '_ssd_w_rm_', '_ssd_w_rM_', '_ssd_w_ra_', '_ssd_r_', '_ssd_r_p_', '_ssd_r_rm_', '_ssd_r_rM_', '_ssd_r_ra_')
# Default counters to display for each mmpmon category
vfs_wanted = '''cr = create + mkdir + link + symlink,
del = remove + rmdir,
op/cl = open + close + map_lloff + unmap,
rd = read + readdir + readlink + mmapRead + readpage + aioRead + aioWrite,
wr = write + mmapWrite + writepage,
trunc = ftrunc + fclear,
fsync = fsync + fsync_range,
lookup,
gattr = access + getattr + getxattr + getacl,
sattr = setattr + setxattr + setacl,
other = *
'''
ioc_wanted1 = '''mb_rd, mb_wr, pref=prefetch_rd, wrbeh=prefetch_wr,
steal*, cleaner*, sync*, revoke*, logwrap*, logdata*,
oth_rd = other_rd, oth_wr = other_wr
'''
ioc_wanted2 = '''rns_r=nsdworker_rd, rns_w=nsdworker_wr,
lns_r=nsdlocal_rd, lns_w=nsdlocal_wr,
vd_r=vdisk_rd, vd_w=vdisk_wr, pd_r=pdisk_rd, pd_w=pdisk_wr,
'''
vio_wanted = '''ClRead=r, ClShWr=sw, ClMdWr=mw, ClPFTWr=pfw, ClFTWr=ftw,
FlUpWr=fuw, FlPFTWr=fpw, Migrte=m, Scrub=s, LgWr=l,
RGDsc=rgd, Meta=meta
'''
vflush_wanted = '''DiTrk = ndt,
DiBuf = ngdb,
FwLog = nfwlmb,
FinPr = nfipt,
WraTh = nfwwt,
HiWMa = ahwm,
Suspd = susp,
WrThF = uwrttf,
Force = fftc,
TrgTh = ntgtth,
other = nalth + nasth + nsigth
'''
lroc_wanted = '''StorS = Inode_s + Directory_s + Data_s,
StorF = Inode_sf + Directory_sf + Data_sf,
FetcS = Inode_r + Directory_r + Data_r,
FetcF = Inode_rf + Directory_rf + Data_rf,
InVAL = Inode_i + Directory_i + Data_i
'''
# Coarse counter selection via DSTAT_GPFS_WHAT
if 'DSTAT_GPFS_WHAT' in os.environ:
what_wanted = os.environ['DSTAT_GPFS_WHAT'].split(',')
else:
what_wanted = [ 'vfs', 'ioc' ]
# If ".dstat_gpfs_rc" exists in user's home directory, run it.
# Otherwise, use DSTAT_GPFS_WHAT for counter selection and look for other
# DSTAT_GPFS_XXX environment variables for additional customization.
userprofile = os.path.join(os.environ['HOME'], '.dstat_gpfs_rc')
if os.path.exists(userprofile):
ioc_wanted = ioc_wanted1 + ioc_wanted2
exec file(userprofile)
else:
if 'all' not in what_wanted:
if 'vfs' not in what_wanted:
vfs_wanted = ''
if 'ioc' not in what_wanted:
ioc_wanted1 = ''
if 'nsd' not in what_wanted:
ioc_wanted2 = ''
if 'vio' not in what_wanted:
vio_wanted = ''
if 'vflush' not in what_wanted:
vflush_wanted = ''
if 'lroc' not in what_wanted:
lroc_wanted = ''
ioc_wanted = ioc_wanted1 + ioc_wanted2
# Fine grain counter cusomization via DSTAT_GPFS_XXX
if 'DSTAT_GPFS_VFS' in os.environ:
vfs_wanted = os.environ['DSTAT_GPFS_VFS']
if 'DSTAT_GPFS_IOC' in os.environ:
ioc_wanted = os.environ['DSTAT_GPFS_IOC']
if 'DSTAT_GPFS_VIO' in os.environ:
vio_wanted = os.environ['DSTAT_GPFS_VIO']
if 'DSTAT_GPFS_VFLUSH' in os.environ:
vflush_wanted = os.environ['DSTAT_GPFS_VFLUSH']
if 'DSTAT_GPFS_LROC' in os.environ:
lroc_wanted = os.environ['DSTAT_GPFS_LROC']
self.debug = 0
vars1, nick1, keymap1 = self.make_keymap(vfs_keys, vfs_wanted, 'gpfs-vfs-')
vars2, nick2, keymap2 = self.make_keymap(ioc_keys, ioc_wanted, 'gpfs-io-')
vars3, nick3, keymap3 = self.make_keymap(vio_keys, vio_wanted, 'gpfs-vio-')
vars4, nick4, keymap4 = self.make_keymap(vflush_keys, vflush_wanted, 'gpfs-vflush-')
vars5, nick5, keymap5 = self.make_keymap(lroc_keys, lroc_wanted, 'gpfs-lroc-')
if 'DSTAT_GPFS_LIST' in os.environ or self.debug:
self.show_keymap('vfs_s', 'DSTAT_GPFS_VFS', vfs_keys, vfs_wanted, vars1, keymap1, 'gpfs-vfs-')
self.show_keymap('ioc_s', 'DSTAT_GPFS_IOC', ioc_keys, ioc_wanted, vars2, keymap2, 'gpfs-io-')
self.show_keymap('vio_s', 'DSTAT_GPFS_VIO', vio_keys, vio_wanted, vars3, keymap3, 'gpfs-vio-')
self.show_keymap('vflush_stat', 'DSTAT_GPFS_VFLUSH', vflush_keys, vflush_wanted, vars4, keymap4, 'gpfs-vflush-')
self.show_keymap('lroc_s', 'DSTAT_GPFS_LROC', lroc_keys, lroc_wanted, vars5, keymap5, 'gpfs-lroc-')
print
self.vars = vars1 + vars2 + vars3 + vars4 + vars5
self.varsrate = vars1 + vars2 + vars3 + vars5
self.varsconst = vars4
self.nick = nick1 + nick2 + nick3 + nick4 + nick5
self.vfs_keymap = keymap1
self.ioc_keymap = keymap2
self.vio_keymap = keymap3
self.vflush_keymap = keymap4
self.lroc_keymap = keymap5
names = []
self.addtitle(names, 'gpfs vfs ops', len(vars1))
self.addtitle(names, 'gpfs disk i/o', len(vars2))
self.addtitle(names, 'gpfs vio', len(vars3))
self.addtitle(names, 'gpfs vflush', len(vars4))
self.addtitle(names, 'gpfs lroc', len(vars5))
self.name = '#'.join(names)
self.type = 'd'
self.width = 5
self.scale = 1000
def make_keymap(self, keys, wanted, prefix):
'''Parse the list of counter values to be displayd
"keys" is the list of all available counters
"wanted" is a string of the form "name1 = key1 + key2 + ..., name2 = key3 + key4 ..."
Returns a list of all names found, e.g. ['name1', 'name2', ...], and a dictionary that
maps counters to names, e.g., { 'key1': 'name1', 'key2': 'name1', 'key3': 'name2', ... },
'''
vars = []
nick = []
kmap = {}
## print re.split(r'\s*,\s*', wanted.strip())
for n in re.split(r'\s*,\s*', wanted.strip()):
l = re.split(r'\s*=\s*', n, 2)
if len(l) == 2:
v = l[0]
kl = re.split(r'\s*\+\s*', l[1])
elif l[0]:
v = l[0].strip('*')
kl = l
else:
continue
nick.append(v[0:5])
v = prefix + v.replace('/', '-')
vars.append(v)
for s in kl:
for k in keys:
if fnmatch.fnmatch(k.strip('_'), s) and k not in kmap:
kmap[k] = v
return vars, nick, kmap
def show_keymap(self, label, envname, keys, wanted, vars, kmap, prefix):
'show available counter names and current counter set definition'
linewd = 100
print '\nAvailable counters for "%s":' % label
mlen = max([len(k.strip('_')) for k in keys])
ncols = linewd // (mlen + 1)
nrows = (len(keys) + ncols - 1) // ncols
for r in range(nrows):
print ' ',
for c in range(ncols):
i = c *nrows + r
if not i < len(keys):
break
print keys[i].strip('_').ljust(mlen),
print
print '\nCurrent counter set selection:'
print "\n%s='%s'\n" % (envname, re.sub(r'\s+', '', wanted).strip().replace(',', ', '))
if not vars:
return
mlen = 5
for v in vars:
if v.startswith(prefix):
s = v[len(prefix):]
else:
s = v
n = ' %s = ' % s[0:mlen].rjust(mlen)
kl = [ k.strip('_') for k in keys if kmap.get(k) == v ]
i = 0
while i < len(kl):
slen = len(n) + 3 + len(kl[i])
j = i + 1
while j < len(kl) and slen + 3 + len(kl[j]) < linewd:
slen += 3 + len(kl[j])
j += 1
print n + ' + '.join(kl[i:j])
i = j
n = ' %s + ' % ''.rjust(mlen)
def addtitle(self, names, name, ncols):
'pad title given by "name" with minus signs to span "ncols" columns'
if ncols == 1:
names.append(name.split()[-1].center(6*ncols - 1))
elif ncols > 1:
names.append(name.center(6*ncols - 1))
def check(self):
'start mmpmon command'
if os.access('/usr/lpp/mmfs/bin/mmpmon', os.X_OK):
try:
self.stdin, self.stdout, self.stderr = dpopen('/usr/lpp/mmfs/bin/mmpmon -p -s')
self.stdin.write('reset\n')
readpipe(self.stdout)
except IOError:
raise Exception, 'Cannot interface with gpfs mmpmon binary'
return True
raise Exception, 'Needs GPFS mmpmon binary'
def extract_vfs(self):
'collect "vfs_s" counter values'
self.stdin.write('vfs_s\n')
l = []
for line in readpipe(self.stdout):
if not line: continue
l += line.split()
for i in range(11, len(l), 3):
try:
self.set2[self.vfs_keymap[l[i]]] += long(l[i+1])
except KeyError:
pass
def extract_ioc(self):
'collect "ioc_s" counter values'
self.stdin.write('ioc_s\n')
l = []
for line in readpipe(self.stdout):
if not line: continue
l += line.split()
for i in range(11, len(l), 3):
try:
self.set2[self.ioc_keymap[l[i]+'rd_']] += long(l[i+1])
except KeyError:
pass
try:
self.set2[self.ioc_keymap[l[i]+'wr_']] += long(l[i+2])
except KeyError:
pass
def extract_vio(self):
'collect "vio_s" counter values'
self.stdin.write('vio_s\n')
l = []
for line in readpipe(self.stdout):
if not line: continue
l += line.split()
for i in range(19, len(l), 2):
try:
if l[i] in self.vio_keymap:
self.set2[self.vio_keymap[l[i]]] += long(l[i+1])
except KeyError:
pass
def extract_vflush(self):
'collect "vflush_stat" counter values'
self.stdin.write('vflush_stat\n')
l = []
for line in readpipe(self.stdout):
if not line: continue
l += line.split()
for i in range(11, len(l), 2):
try:
if l[i] in self.vflush_keymap:
self.set2[self.vflush_keymap[l[i]]] += long(l[i+1])
except KeyError:
pass
def extract_lroc(self):
'collect "lroc_s" counter values'
self.stdin.write('lroc_s\n')
l = []
for line in readpipe(self.stdout):
if not line: continue
l += line.split()
for i in range(11, len(l), 2):
try:
if l[i] in self.lroc_keymap:
self.set2[self.lroc_keymap[l[i]]] += long(l[i+1])
except KeyError:
pass
def extract(self):
try:
for name in self.vars:
self.set2[name] = 0
self.extract_ioc()
self.extract_vfs()
self.extract_vio()
self.extract_vflush()
self.extract_lroc()
for name in self.varsrate:
self.val[name] = (self.set2[name] - self.set1[name]) * 1.0 / elapsed
for name in self.varsconst:
self.val[name] = self.set2[name]
except IOError, e:
for name in self.vars: self.val[name] = -1
## print 'dstat_gpfs: lost pipe to mmpmon,', e
except Exception, e:
for name in self.vars: self.val[name] = -1
print 'dstat_gpfs: exception', e
if self.debug >= 0:
self.debug -= 1
if step == op.delay:
self.set1.update(self.set2)
More information about the gpfsug-discuss
mailing list