[gpfsug-discuss] Change to default for verbsRdmaMinBytes?

[Sven Oehme]

Hi,

you still need both of them, but they are both on the list to be removed,
the first is already integrated for the next major release, the 2nd we
still work on.

Sven

On Wed, Sep 6, 2017 at 4:55 AM Kenneth Waegeman <kenneth.waegeman at ugent.be>
wrote:

> Hi Sven,
>
> I see two parameters that we have set to non-default values that are not
> in your list of options still to configure.
> verbsRdmasPerConnection (256) and
> socketMaxListenConnections (1024)
>
> I remember we had to set socketMaxListenConnections because our cluster
> consist of +550 nodes.
>
> Are these settings still needed, or is this also tackled in the code?
>
> Thank you!!
>
> Cheers,
> Kenneth
>
>
>
> On 02/09/17 00:42, Sven Oehme wrote:
>
> Hi Ed,
>
> yes the defaults for that have changed for customers who had not
> overridden the default settings. the reason we did this was that many
> systems in the field including all ESS systems that come pre-tuned where
> manually changed to 8k from the 16k default due to better performance that
> was confirmed in multiple customer engagements and tests with various
> settings , therefore we change the default to what it should be in the
> field so people are not bothered to set it anymore (simplification) or get
> benefits by changing the default to provides better performance.
> all this happened when we did the communication code overhaul that did
> lead to significant (think factors) of improved RPC performance for RDMA
> and VERBS workloads.
> there is another round of significant enhancements coming soon , that will
> make even more parameters either obsolete or change some of the defaults
> for better out of the box performance.
> i see that we should probably enhance the communication of this changes,
> not that i think this will have any negative effect compared to what your
> performance was with the old setting i am actually pretty confident that
> you get better performance with the new code, but by setting parameters
> back to default on most 'manual tuned' probably makes your system even
> faster.
> if you have a Scale Client on 4.2.3+ you really shouldn't have anything
> set beside maxfilestocache, pagepool, workerthreads and potential prefetch
> , if you are a protocol node, this and settings specific to an  export
> (e.g. SMB, NFS set some special settings) , pretty much everything else
> these days should be set to default so the code can pick the correct
> parameters., if its not and you get better performance by manual tweaking
> something i like to hear about it.
> on the communication side in the next release will eliminate another set
> of parameters that are now 'auto set' and we plan to work on NSD next.
> i presented various slides about the communication and simplicity changes
> in various forums, latest public non NDA slides i presented are here -->
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf
>
> hope this helps .
>
> Sven
>
>
>
> On Fri, Sep 1, 2017 at 1:56 PM Edward Wahl < <ewahl at osc.edu>ewahl at osc.edu>
> wrote:
>
>> Howdy.   Just noticed this change to min RDMA packet size and I don't
>> seem to
>> see it in any patch notes.  Maybe I just skipped the one where this
>> changed?
>>
>>  mmlsconfig verbsRdmaMinBytes
>> verbsRdmaMinBytes 16384
>>
>> (in case someone thinks we changed it)
>>
>> [root at proj-nsd01 ~]# mmlsconfig |grep verbs
>> verbsRdma enable
>> verbsRdma disable
>> verbsRdmasPerConnection 14
>> verbsRdmasPerNode 1024
>> verbsPorts mlx5_3/1
>> verbsPorts mlx4_0
>> verbsPorts mlx5_0
>> verbsPorts mlx5_0 mlx5_1
>> verbsPorts mlx4_1/1
>> verbsPorts mlx4_1/2
>>
>>
>> Oddly I also see this in config, though I've seen these kinds of things
>> before.
>> mmdiag --config |grep verbsRdmaMinBytes
>>    verbsRdmaMinBytes 8192
>>
>> We're on a recent efix.
>> Current GPFS build: "4.2.2.3 efix21 (1028007)".
>>
>> --
>>
>> Ed Wahl
>> Ohio Supercomputer Center
>> 614-292-9302 <%28614%29%20292-9302>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170906/e1f559d1/attachment.htm>