[gpfsug-discuss] mixed verbsRdmaSend

[Stijn De Weirdt]

hi all,

what is the expected behaviour of a mixed verbsRdmaSend setup: some
nodes enabled, most disabled.

we have some nodes that have a very high iops workload, but most of the
cluster of 500+ nodes do not have such usecase.
we enabled verbsRdmaSend on the managers/quorum nodes (<10) and on the
few (<10) clients with this workload, but not on the others (500+). it
seems to work out fine, but is this acceptable as config? (the docs
mention that enabling verbsrdamSend on a> 100 nodes might lead to errors).


the nodes use ipoib as ip network, and running with verbsRdmaSend
disabled on all nodes leads to unstable cluster (TX errors (<1 error in
1M packets) on some clients leading to gpfs expel nodes etc).
(we still need to open a case wil mellanox to investigate further)

many thanks,

stijn