[gpfsug-discuss] mmfsck runs very slowly on a small filesystem
Mengxing Cheng
mxcheng at uchicago.edu
Thu Mar 9 21:24:21 GMT 2017
Eric, thank you very much for replying. Here is the memory configuration and current usage. Note that mmfsck is not running now. The two gss servers have the same 256GB memory and the service node has 128GB.
1. service node:
total used free shared buffers cached
Mem: 125 58 66 0 0 4
-/+ buffers/cache: 53 71
Swap: 7 0 7
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12990 root 0 -20 71.0g 43g 885m S 7.6 34.4 9306:00 mmfsd
2. gss nodes:
====================================
total used free shared buff/cache available
Mem: 251 210 37 0 4 36
Swap: 3 0 3
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
36770 root 0 -20 0.216t 0.192t 2.667g S 48.9 78.1 75684:09 /usr/lpp/mmfs/bin/mmfsd
The gss nodes' memory usage is so high because their pagepool is set to 192GB while the service node has 16GB pagepool.
Mengxing
---
Mengxing Cheng, Ph.D.
HPC System Administrator
Research Computing Center
The University of Chicago
5607 S. Drexel Ave.
Chicago, IL 60637
email: mxcheng at uchicago.edu
phone: (773) 702-4070
________________________________
From: Eric Sperley [esperle at us.ibm.com]
Sent: Thursday, March 09, 2017 3:13 PM
To: Mengxing Cheng
Cc: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem
Mengxing,
It is nice meeting you.
I have seen a situation where the amount of RAM on a node can affect mmfsck times. Do all the nodes have the same amount of RAM, or does the slow running node have less RAM?
Best Regards, Eric
[cid:3__=8FBB0A4DDFE7DC3F8f9e8a93df938690918c8FB@] Eric Sperley
SDI Architect
"Carpe Diem" IBM Systems
esperle at us.ibm.com
+15033088721
[Inactive hide details for Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system]Mengxing Cheng ---03/09/2017 11:24:02 AM---Dear all, My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago
From: Mengxing Cheng <mxcheng at uchicago.edu>
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Date: 03/09/2017 11:24 AM
Subject: [gpfsug-discuss] mmfsck runs very slowly on a small filesystem
Sent by: gpfsug-discuss-bounces at spectrumscale.org
________________________________
Dear all,
My name is Mengxing Cheng and I am a HPC system administrator at the University of Chicago. We have a GSS26 running gss2.5.10.3-3b and gpfs-4.2.0.3.
Recently, we run mmfsck on a relatively small filesystem with 14TB block and 73863102 inodes but it was unusually slow so as to not be able to finish in 48 hours. In contrast, mmfsck run on a filesystem with the same size and inodes but sitting on a traditional IBM DS3512 storage took only 2 hours to complete.
In particular, the mmfsck run in parallel using 3 nodes within the GSS storage cluster, we notice that one gss storage server scans inodes much slower than the other gss storage server and the quorum service node.
Has anyone experience the same mmfsck performance issue?
Could anyone make recommendation to troubleshoot and improve mmfsck performance?
Thank you!
Mengxing
---
Mengxing Cheng, Ph.D.
HPC System Administrator
Research Computing Center
The University of Chicago
5607 S. Drexel Ave.
Chicago, IL 60637
email: mxcheng at uchicago.edu
phone: (773) 702-4070
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170309/6ad55d6b/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1C803747.gif
Type: image/gif
Size: 481 bytes
Desc: 1C803747.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170309/6ad55d6b/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: ecblank.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170309/6ad55d6b/attachment-0001.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1C166252.gif
Type: image/gif
Size: 2322 bytes
Desc: 1C166252.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170309/6ad55d6b/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: graycol.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170309/6ad55d6b/attachment-0003.gif>
More information about the gpfsug-discuss
mailing list