[gpfsug-discuss] Anybody running GPFS over iSCSI?
Jonathan Buzzard
jonathan.buzzard at strath.ac.uk
Mon Dec 17 00:21:57 GMT 2018
On 13/12/2018 20:54, Buterbaugh, Kevin L wrote:
[SNIP]
>
> Two things that I am already aware of are: 1) use jumbo frames, and 2)
> run iSCSI over it’s own private network. Other things I should be aware
> of?!?
>
Yes, don't do it. Really do not do it unless you have datacenter
Ethernet switches and adapters. Those are the ones required for FCoE.
Basically unless you have per channel pause on your Ethernet fabric then
performance will at some point all go to shit.
So what happens is your NSD makes a whole bunch of requests to read
blocks off the storage array. Requests are small, response is not. The
response can overwhelms the Ethernet channel at which point performance
falls through the floor. Now you might be lucky not to see this,
especially if you have say have 10Gbps links from the storage and 40Gbps
links to the NSD servers, but you are taking a gamble. Also the more
storage arrays you have the more likely you are to see the problem.
To fix this you have two options. The first is datacenter Ethernet with
per channel pause. This option is expensive, probably in the same ball
park as fibre channel. At least it was last time I looked, though this
was some time ago now.
The second option is dedicated links between the storage array and the
NSD server. That is the cable goes directly between the storage array
and the NSD server with no switches involved. This option is a
maintenance nightmare.
At he site where I did this, we had to go option two because I need to
make it work, We ended up ripping it all out are replacing with FC.
Personally I would see what price you can get DSS storage for, or use
SAS arrays.
Note iSCSI can in theory work, it's just the issue with GPFS scattering
stuff to the winds over multiple storage arrays so your ethernet channel
gets swamped and standard ethernet pauses all the upstream traffic. The
vast majority of iSCSI use cases don't see this effect.
There is a reason that to run FC over ethernet they had to turn ethernet
lossless.
JAB.
--
Jonathan A. Buzzard Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
More information about the gpfsug-discuss
mailing list