[gpfsug-discuss] Thoughts on GPFS on IB & MTU sizes

[Saula, Oluwasijibomi]

Hi Folks,


As this is my first post to the group, let me start by saying I applaud the commentary from the user group as it has been a resource to those of us watching from the sidelines.


That said, we have a GPFS layered on IPoIB, and recently, we started having some issues on our IB FDR fabric which manifested when GPFS began sending persistent expel messages to particular nodes.


Shortly after, we embarked on a tuning exercise using IBM tuning recommendations<https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Welcome%20to%20High%20Performance%20Computing%20%28HPC%29%20Central/page/Linux%20System%20Tuning%20Recommendations> but this page is quite old and we've run into some snags, specifically with setting 4k MTUs using mlx4_core/mlx4_en module options.


While setting 4k MTUs as the guide recommends is our general inclination, I'd like to solicit some advice as to whether 4k MTUs are a good idea and any hitch-free steps to accomplishing this. I'm getting some conflicting remarks from Mellanox support asking why we'd want to use 4k MTUs with Unreliable Datagram mode.


Also, any pointers to best practices or resources for network configurations for heavy I/O clusters would be much appreciated.


Thanks,

Siji Saula
HPC System Administrator
Center for Computationally Assisted Science & Technology
NORTH DAKOTA STATE UNIVERSITY


<https://www.ndsu.edu/alphaindex/buildings/Building::395>Research 2 Building<https://www.ndsu.edu/alphaindex/buildings/Building::396><https://www.ndsu.edu/alphaindex/buildings/Building::395> – Room 220B
Dept 4100, PO Box 6050  / Fargo, ND 58108-6050
p:701.231.7749
www.ccast.ndsu.edu<file://composeviewinternalloadurl/www.ccast.ndsu.edu> | www.ndsu.edu<file://composeviewinternalloadurl/www.ndsu.edu>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180308/0f2fc16f/attachment.htm>