[gpfsug-discuss] mmchconfig subnets

[Skylar Thompson]

My understanding is that GPFS uses the network configuration on each node
to determine netmask. The subnets option can be applied to specific nodes
or groups of nodes with "mmchconfig subnets=... -N <node-list>", so what
you're doing is specificy the preferred subnets for GPFS node
communication, just for that list of nodes.

For instance, we have four GPFS clusters, with three subnets:

* eichler-cluster, eichler-cluster2 (10.130.0.0/16)
* grc-cluster (10.200.0.0/16)
* gs-cluster (10.110.0.0/16)

And one data transfer system weasel that is a member of gs-cluster, but
provides transfer services to all the clusters, and has an IP address on each subnet
to avoid a bunch of network cross-talk. Its subnets setting looks like this:

[weasel]
subnets 10.130.0.0/eichler-cluster*.grid.gs.washington.edu 10.200.0.0/grc-cluster.grid.gs.washington.edu 10.110.0.0/gs-cluster.grid.gs.washington.edu

Of course, there's some policy routing too to keep replies on the right
interface as well, but that's the extent of the GPFS configuration.

On Mon, Jun 25, 2018 at 04:20:35PM -0700, Eric Horst wrote:
> Hi, I'm hoping somebody has insights into how the subnets option actually
> works. I've read the docs a dozen times and I want to make sure I
> understand before I take my production cluster down to make the changes.
> 
> On the current cluster the daemon addresses are on a gpfs private network
> and the admin addresses are on a public network. I'm changing so both
> daemon and admin are public and the subnets option is used to utilize the
> private network. This is to facilitate remote mounts to an independent
> cluster.
> 
> The confusing factor in my case, not covered in the docs, is that the gpfs
> private network is subnetted and static routes are used to reach them. That
> is, there are three private networks, one for each datacenter and the
> cluster nodes daemon interfaces are spread between the three.
> 
> 172.16.141.32/27
> 172.16.141.24/29
> 172.16.141.128/27
> 
> A router connects these three networks but are otherwise 100% private.
> 
> For my mmchconfig subnets command should I use this?
> 
> mmchconfig subnets="172.16.141.24 172.16.141.32 172.16.141.128"
> 
> Where I get confused is that I'm trying to reason through how Spectrum
> Scale is utilizing the subnets setting to decide if this will have the
> desired result on my cluster. If I change the node addresses to their
> public addresses, ie the private addresses are not explicitly configured in
> Scale, then how are the private addresses discovered? Does each node use
> the subnets option to identify that it has a private address and then
> dynamically shares that with the cluster?
> 
> Thanks in advance for your clarifying comments.
> 
> -Eric
> 
> --
> 
> Eric Horst
> University of Washington

> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-- 
-- Skylar Thompson (skylar2 at u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine