[gpfsug-discuss] Steps for gracefully handling bandwidth reduction during network maintenance

Luis Bolinches luis.bolinches at fi.ibm.com
Mon Jun 17 17:54:04 BST 2019


Hi

Writing from phone so excuse the typos.

Assuming you have a system pool (metadata) and some other pool/s you can
set limits on maintenance class that you done already and on other class
that would affect all the other ops. You can add those per node or
nodeclass that can be matched to what part/s of network you are working
with.

Changes are online and immediate. And you can measure normal load just by
having QoS activated and looking into the values for few days.

Hope makes some sense the above.

--
Cheers

> On 17 Jun 2019, at 19.48, Christopher Black <cblack at nygenome.org> wrote:
>
> The man page indicates that maxMBpS can be used to "artificially limit
how much I/O one node can put on all of the disk servers", but it might not
be the best choice. Man page also says maxmbps is in the class of
mmchconfig changes take place immediately.
> We've only ever used QoS for throttling maint operations (restripes, etc)
and are unfamiliar with how to best use it to throttle client load.
>
> Best,
> Chris
>
> On 6/17/19, 12:40 PM, "gpfsug-discuss-bounces at spectrumscale.org on behalf
of Skylar Thompson" <gpfsug-discuss-bounces at spectrumscale.org on behalf of
skylar2 at uw.edu> wrote:
>
>    IIRC, maxMBpS isn't really a limit, but more of a hint for how GPFS
should
>    use its in-memory buffers for read prefetches and dirty writes.
>
>>    On Mon, Jun 17, 2019 at 09:31:38AM -0700, Alex Chekholko wrote:
>> Hi Chris,
>>
>> I think the next thing to double-check is when the maxMBpS change takes
>> effect.  You may need to restart the nsds.  Otherwise I think your plan
is
>> sound.
>>
>> Regards,
>> Alex
>>
>>
>> On Mon, Jun 17, 2019 at 9:24 AM Christopher Black <cblack at nygenome.org>
>> wrote:
>>
>>> Our network team sometimes needs to take down sections of our network
for
>>> maintenance. Our systems have dual paths thru pairs of switches, but
often
>>> the maintenance will take down one of the two paths leaving all our nsd
>>> servers with half bandwidth.
>>>
>>> Some of our systems are transmitting at a higher rate than can be
handled
>>> by half network (2x40Gb hosts with tx of 50Gb+).
>>>
>>> What can we do to gracefully handle network maintenance reducing
bandwidth
>>> in half?
>>>
>>> Should we set maxMBpS for affected nodes to a lower value? (default on
our
>>> ess appears to be maxMBpS = 30000, would I reduce this to ~4000 for
32Gbps?)
>>>
>>> Any other ideas or comments?
>>>
>>> Our hope is that metadata operations are not affected much and users
just
>>> see jobs and processes read or write at a slower rate.
>>>
>>>
>>>
>>> Best,
>>>
>>> Chris
>>> ------------------------------
>>> This message is for the recipient???s use only, and may contain
>>> confidential, privileged or protected information. Any unauthorized use
or
>>> dissemination of this communication is prohibited. If you received this
>>> message in error, please immediately notify the sender and destroy all
>>> copies of this message. The recipient should check this email and any
>>> attachments for the presence of viruses, as we accept no liability for
any
>>> damage caused by any virus transmitted by this email.
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>>
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=2ioq3oT4gzOlIvyQRqkdZF0GWKv1APEBmstC48AyVdo&s=fvxPTdT1cVT7av_-vR5-3wVgjIzEpUP8OY8vGx0i5kc&e=

>>>
>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>>
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=2ioq3oT4gzOlIvyQRqkdZF0GWKv1APEBmstC48AyVdo&s=fvxPTdT1cVT7av_-vR5-3wVgjIzEpUP8OY8vGx0i5kc&e=

>
>
>    --
>    -- Skylar Thompson (skylar2 at u.washington.edu)
>    -- Genome Sciences Department, System Administrator
>    -- Foege Building S046, (206)-685-7354
>    -- University of Washington School of Medicine
>    _______________________________________________
>    gpfsug-discuss mailing list
>    gpfsug-discuss at spectrumscale.org
>
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=2ioq3oT4gzOlIvyQRqkdZF0GWKv1APEBmstC48AyVdo&s=fvxPTdT1cVT7av_-vR5-3wVgjIzEpUP8OY8vGx0i5kc&e=

>
>
> ________________________________
>
> This message is for the recipient’s use only, and may contain
confidential, privileged or protected information. Any unauthorized use or
dissemination of this communication is prohibited. If you received this
message in error, please immediately notify the sender and destroy all
copies of this message. The recipient should check this email and any
attachments for the presence of viruses, as we accept no liability for any
damage caused by any virus transmitted by this email.
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=zyyij5eDMGGtTC00mplr-3aAR3dbStZGhwocBYKIyUg&s=dlSFGfd_CW47EaNE-5X9tMCkmqZ8WayaLCGI1sTzpkA&e=

>
Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3 
Registered in Finland

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190617/ce2b2431/attachment.htm>


More information about the gpfsug-discuss mailing list