[gpfsug-discuss] Thoughts on GPFS on IB & MTU sizes

[Wei Guo]

Hi, Saula,


Can the expelled node and expelling node ping each other?
We expanded our gpfs IB network from /24 to /20 but some clients still used /24, they cannot talk to the added new clients using /20 and expelled the new clients persistently.
Changing the netmask all to /20 works out.


FYI.

Wei Guo
HPC Administartor
UT Southwestern Medical Center
wei1.guo at utsouthwestern.edu

________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of gpfsug-discuss-request at spectrumscale.org <gpfsug-discuss-request at spectrumscale.org>
Sent: Thursday, March 8, 2018 11:37 AM
To: gpfsug-discuss at spectrumscale.org
Subject: gpfsug-discuss Digest, Vol 74, Issue 17

Send gpfsug-discuss mailing list submissions to
        gpfsug-discuss at spectrumscale.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
        gpfsug-discuss-request at spectrumscale.org

You can reach the person managing the list at
        gpfsug-discuss-owner at spectrumscale.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. Thoughts on GPFS on IB & MTU sizes (Saula, Oluwasijibomi)
   2. Re: wondering about outage free protocols upgrades
      (Christof Schmitt)


----------------------------------------------------------------------

Message: 1
Date: Thu, 8 Mar 2018 15:06:03 +0000
From: "Saula, Oluwasijibomi" <oluwasijibomi.saula at ndsu.edu>
To: "gpfsug-discuss at spectrumscale.org"
        <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Thoughts on GPFS on IB & MTU sizes
Message-ID:
        <CY4PR08MB2854FF1706F7B6C59D687BE998D80 at CY4PR08MB2854.namprd08.prod.outlook.com>

Content-Type: text/plain; charset="windows-1252"

Hi Folks,


As this is my first post to the group, let me start by saying I applaud the commentary from the user group as it has been a resource to those of us watching from the sidelines.


That said, we have a GPFS layered on IPoIB, and recently, we started having some issues on our IB FDR fabric which manifested when GPFS began sending persistent expel messages to particular nodes.


Shortly after, we embarked on a tuning exercise using IBM tuning recommendations<https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Welcome%20to%20High%20Performance%20Computing%20%28HPC%29%20Central/page/Linux%20System%20Tuning%20Recommendations> but this page is quite old and we've run into some snags, specifically with setting 4k MTUs using mlx4_core/mlx4_en module options.


While setting 4k MTUs as the guide recommends is our general inclination, I'd like to solicit some advice as to whether 4k MTUs are a good idea and any hitch-free steps to accomplishing this. I'm getting some conflicting remarks from Mellanox support asking why we'd want to use 4k MTUs with Unreliable Datagram mode.


Also, any pointers to best practices or resources for network configurations for heavy I/O clusters would be much appreciated.


Thanks,

Siji Saula
HPC System Administrator
Center for Computationally Assisted Science & Technology
NORTH DAKOTA STATE UNIVERSITY


<https://www.ndsu.edu/alphaindex/buildings/Building::395>Research 2 Building<https://www.ndsu.edu/alphaindex/buildings/Building::396><https://www.ndsu.edu/alphaindex/buildings/Building::395> ? Room 220B
Dept 4100, PO Box 6050  / Fargo, ND 58108-6050
p:701.231.7749
www.ccast.ndsu.edu<file://composeviewinternalloadurl/www.ccast.ndsu.edu> | www.ndsu.edu<file://composeviewinternalloadurl/www.ndsu.edu>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20180308/0f2fc16f/attachment-0001.html>

------------------------------

Message: 2
Date: Thu, 8 Mar 2018 17:37:12 +0000
From: "Christof Schmitt" <christof.schmitt at us.ibm.com>
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] wondering about outage free protocols
        upgrades
Message-ID:
        <OF84AA7F39.23BFBFAF-ON0025824A.005E7D99-0025824A.0060CA39 at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20180308/89483e8a/attachment.html>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 74, Issue 17
**********************************************

________________________________

UT Southwestern


Medical Center



The future of medicine, today.