[gpfsug-discuss] Thousands of CLOSE_WAIT connections

[Iban Cabrillo]

Dear, 

We have reinstall recently from gpfs 3.5 to SpectrumScale 4.2.3-6 version redhat 7. 

We are running two nsd servers and a a gui, there is no firewall on gpfs network, and selinux is disable, I have checked changing the manager and cluster manager node between server with the same result, server 01 always increase the CLOSE_WAIT : 

Node Daemon node name IP address Admin node name Designation 
-------------------------------------------------------------------------------- 
1 gpfs01.ifca.es 10.10.0.111 gpfs01.ifca.es quorum-manager-perfmon 
2 gpfs02.ifca.es 10.10.0.112 gpfs02.ifca.es quorum-manager-perfmon 
3 gpfsgui.ifca.es 10.10.0.60 gpfsgui.ifca.es quorum-perfmon 
....... 

Installation and configuration works fine, but now we see that one of the servers do not close the mmfsd connections and this growing for ever while the othe nsd servers is always in the same range: 

[root at gpfs01 ~]# netstat -putana | grep 1191 | wc -l 
19701 
[root at gpfs01 ~]# netstat -putana | grep 1191 | grep CLOSE_WAIT| wc -l 
19528 
.... 
[root at gpfs02 ~]# netstat -putana | grep 1191 | wc -l 
215 
[root at gpfs02 ~]# netstat -putana | grep 1191 | grep CLOSE_WAIT| wc -l 
0 

this is causing that gpfs01 do not answer to cluster commands 

NSD are balance between server (same size): 
[root at gpfs02 ~]# mmlsnsd 

File system Disk name NSD servers 
--------------------------------------------------------------------------- 
gpfs nsd1 gpfs01,gpfs02 
gpfs nsd2 gpfs01,gpfs02 
gpfs nsd3 gpfs02,gpfs01 
gpfs nsd4 gpfs02,gpfs01 
..... 

proccess seems to be similar in both servers, only mmccr is running on server 1 and not in 2 

gpfs01 
####### 
root 9169 1 0 feb07 ? 22:27:54 python /usr/lpp/mmfs/bin/mmsysmon.py 
root 11533 6154 0 13:41 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmsdrquery sdrq_fs_info all 
root 11713 1 0 13:41 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 
root 12367 11533 0 13:43 ? 00:00:00 /usr/lpp/mmfs/bin/mmccr vget mmRunningCommand 
root 12641 6162 0 13:44 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmsdrquery sdrq_nsd_info sdrq_nsd_name:sdrq_fs_name:sdrq_storage_pool 
root 12668 12641 0 13:44 ? 00:00:00 /usr/lpp/mmfs/bin/mmccr fget -c 835 mmsdrfs /var/mmfs/gen/mmsdrfs.12641 
root 12950 11713 0 13:44 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 
root 12959 9169 13 13:44 ? 00:00:00 /usr/lpp/mmfs/bin/mmccr check -Y -e 
root 12968 3150 0 13:45 pts/3 00:00:00 grep --color=auto mm 
root 19620 26468 38 jun14 ? 11:28:36 /usr/lpp/mmfs/bin/mmfsd 
root 19701 2 0 jun14 ? 00:00:00 [mmkproc] 
root 19702 2 0 jun14 ? 00:00:00 [mmkproc] 
root 19703 2 0 jun14 ? 00:00:00 [mmkproc] 
root 26468 1 0 jun05 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/runmmfs 


[root at gpfs02 ~]# ps -feA | grep mm 
root 5074 1 0 feb07 ? 01:00:34 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 
root 5128 31456 28 jun14 ? 06:18:07 /usr/lpp/mmfs/bin/mmfsd 
root 5255 2 0 jun14 ? 00:00:00 [mmkproc] 
root 5256 2 0 jun14 ? 00:00:00 [mmkproc] 
root 5257 2 0 jun14 ? 00:00:00 [mmkproc] 
root 15196 5074 0 13:47 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 
root 15265 13117 0 13:47 pts/0 00:00:00 grep --color=auto mm 
root 31456 1 0 jun05 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/runmmfs 

Any idea will be appreciated. 
Regards, I 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180615/ea6b08ab/attachment.htm>