[gpfsug-discuss] Spectrum Scale Slow to create directories

[Peter Childs]

This is a curious issue which I'm trying to get to the bottom of.

We currently have two Spectrum Scale file systems, both are running GPFS 4.2.1-1 some of the servers have been upgraded to 4.2.1-2.

The older one which was upgraded from GPFS 3.5 works find create a directory is always fast and no issue.

The new one, which has nice new SSD for metadata and hence should be faster. can take up to 30 seconds to create a directory but usually takes less than a second, The longer directory creates usually happen on busy nodes that have not used the new storage in a while. (Its new so we've not moved much of the data over yet) But it can also happen randomly anywhere, including from the NSD servers them selves. (times of 3-4 seconds from the NSD servers have been seen, on a single directory create)

We've been pointed at the network and suggested we check all network settings, and its been suggested to build an admin network, but I'm not sure I entirely understand why and how this would help. Its a mixed 1G/10G network with the NSD servers connected at 40G with an MTU of 9000. 

However as I say, the older filesystem is fine, and it does not matter if the nodes are connected to the old GPFS cluster or the new one, (although the delay is worst on the old gpfs cluster), So I'm really playing spot the difference. and the network is not really an obvious difference.

Its been suggested to look at a trace when it occurs but as its difficult to recreate collecting one is difficult. 

Any ideas would be most helpful.

Thanks



Peter Childs
ITS Research Infrastructure
Queen Mary, University of London