[gpfsug-discuss] GPFS architecture choice: large servers or directly-attached clients?

[Eric Horst]

GPFS NSD servers (the ones with the disks attached) do not do any
caching. There is no benefit to configuring the NSD servers with
significant amounts of memory and increasing pagepool will not provide
caching. NSD servers with pagepool in the single digit GB is plenty.
The NSD servers for our 4000 core cluster have 12GB RAM and pagepool
of 4GB. The 500 clients have pagepool of 2GB. This is some info from
the GPFS wiki regarding NSD servers:

"Assuming no applications or Filesystem Manager services are running
on the NSD servers, the pagepool is only used transiently by the NSD
worker threads to gather data from client nodes and write the data to
disk. The NSD server does not cache any of the data. Each NSD worker
just needs one pagepool buffer per operation, and the buffer can be
potentially as large as the largest filesystem blocksize that the
disks belong to. With the default NSD configuration, there will be 3
NSD worker threads per LUN (nsdThreadsPerDisk) that the node services.
So the amount of memory needed in the pagepool will be
3*#LUNS*maxBlockSize. The target amount of space in the pagepool for
NSD workers is controlled by nsdBufSpace which defaults to 30%. So the
pagepool should be large enough so that 30% of it has enough buffers."

-Eric

On Mon, Mar 11, 2013 at 12:26 PM,  <mark.bergman at uphs.upenn.edu> wrote:
> [1] Large GPFS servers
>         About 5 GPFS servers with significant RAM. Each GPFS server would
>         be connected to storage via an 8Gb/s fibre SAN (multiple paths)
>         to storage arrays.
>
>         Each GPFS server would provide NSDs via 10Gb/s and 1Gb/s (for legacy
>         servers) ethernet to GPFS clients (computational compute nodes).
>
>                 Questions:
>
>                         Since the GPFS clients would not be SAN attached
>                         with direct access to block storage, and many
>                         clients (~50) will access similar data (and the
>                         same directories) for many jobs, it seems like it
>                         would make sense to do a lot of caching on the
>                         GPFS servers. Multiple clients would benefit by
>                         reading from the same cached data on the servers.
>
>                         I'm thinking of sizing caches to handle 1~2GB
>                         per core in the compute nodes, divided by the
>                         number of GPFS servers.  This would mean caching
>                         (maxFilesToCache, pagepool, maxStatCache) on the
>                         GPFS servers of about 200GB+ on each GPFS server.
>
>                         Is there any way to configure GPFS so that the
>                         GPFS servers can do a large amount of caching
>                         without requiring the same resources on the
>                         GPFS clients?
>
>                         Is there any way to configure the GPFS clients
>                         so that their RAM can be used primarily for
>                         computational jobs?