[gpfsug-discuss] mmap performance against Spectrum Scale

[Ray Coetzee]

Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2.
(Without the meltdown patch)

Hey Bryan, I remember that quote from Yuri, that's why I hoped some "magic"
may have been done since then.

Other attempts to improve performance I've tried include:

   - Using LROC to have a larger chance of a cache hit (Unfortunately the
   entire dataset is multiple TB)
   - Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for this
   purpose (Job runs halved in duration but nowhere near what NFS can give)
   - Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, and
   some others with little improvement.

For those interested, as a performance comparison. The same job when run on
an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe
scratch filesystem and over 60min on spindle based filesystem.

Kind regards

Ray Coetzee
Email: coetzee.ray at gmail.com


On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister <bbanister at jumptrading.com>
wrote:

> You could put all of your data onto SSDs in a RAID1 configuration so that
> you don’t have insane read-modify-write penalties on writes (RAID1) and
> avoid horrible seek thrashing that spinning rust requires (SSD random
> access medium) for your 4K I/O operations.
>
>
>
> One of my favorite Yuri quotes, “The mmap code is like asbestos… best not
> to touch it”.  He gave many reasons why mmap operations on a distributed
> file system is incredibly hard and not recommended.
>
> -Bryan
>
>
>
> *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-
> bounces at spectrumscale.org] *On Behalf Of *Sven Oehme
> *Sent:* Friday, January 12, 2018 8:45 AM
> *To:* coetzee.ray at gmail.com; gpfsug main discussion list <
> gpfsug-discuss at spectrumscale.org>
> *Subject:* Re: [gpfsug-discuss] mmap performance against Spectrum Scale
>
>
>
> *Note: External Email*
> ------------------------------
>
> what version of Scale are you using right now ?
>
>
>
> On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee <coetzee.ray at gmail.com> wrote:
>
> I'd like to ask the group of their experiences in improving the
> performance of applications that use mmap calls against files on Spectrum
> Scale.
>
>
>
> Besides using an NFS export from CES instead of a native GPFS mount, or
> precaching the dataset into the pagepool, what other approaches are there
> to offset the performance hit of the 4K IO size?
>
>
>
> Kind regards
>
> Ray Coetzee
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> ------------------------------
>
> Note: This email is for the confidential use of the named addressee(s)
> only and may contain proprietary, confidential or privileged information.
> If you are not the intended recipient, you are hereby notified that any
> review, dissemination or copying of this email is strictly prohibited, and
> to please notify the sender immediately and destroy this email and any
> attachments. Email transmission cannot be guaranteed to be secure or
> error-free. The Company, therefore, does not make any guarantees as to the
> completeness or accuracy of this email or any attachments. This email is
> for informational purposes only and does not constitute a recommendation,
> offer, request or solicitation of any kind to buy, sell, subscribe, redeem
> or perform any type of transaction of a financial product.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180112/d0354588/attachment.htm>