[gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30)
Aaron Knister
aaron.s.knister at nasa.gov
Tue Aug 22 17:45:00 BST 2017
(I'm slowly catching up on a backlog of e-mail, sorry for the delayed
reply).
Thanks, Sven. I recognize the complexity and appreciate your explanation.
In my mind I had envisioned either the block integrity information being
stored as a new metadata structure or stored leveraging T10-DIX/DIF
(perhaps configurable on a per-pool basis) to pass the checksums down to
the RAID controller.
I would quite like to run GNR as software on generic hardware and in
fact voted, along with 26 other customers, on an RFE
(https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95090)
requesting this but the request was declined. I think customers spoke
pretty loudly there and IBM gave it the kibosh.
-Aaron
On 06/30/2017 02:25 PM, Sven Oehme wrote:
>
> end-to-end data integrity is very important and the reason it hasn't
> been done in Scale is not because its not important, its because its
> very hard to do without impacting performance in a very dramatic way.
>
> imagine your raid controller blocksize is 1mb and your filesystem
> blocksize is 1MB . if your application does a 1 MB write this ends up
> being a perfect full block , full track de-stage to your raid layer
> and everything works fine and fast. as soon as you add checksum
> support you need to add data somehow into this, means your 1MB is no
> longer 1 MB but 1 MB+checksum.
>
> to store this additional data you have multiple options, inline ,
> outside the data block or some combination ,the net is either you need
> to do more physical i/o's to different places to get both the data and
> the corresponding checksum or your per block on disc structure becomes
> bigger than than what your application reads/or writes, both put
> massive burden on the Storage layer as e.g. a 1 MB write will now,
> even the blocks are all aligned from the application down to the raid
> layer, cause a read/modify/write on the raid layer as the data is
> bigger than the physical track size.
>
> so to get end-to-end checksum in Scale outside of ESS the best way is
> to get GNR as SW to run on generic HW, this is what people should vote
> for as RFE if they need that functionality. beside end-to-end
> checksums you get read/write cache and acceleration , fast rebuild and
> many other goodies as a added bonus.
>
> Sven
>
>
> On Fri, Jun 30, 2017 at 10:53 AM Aaron Knister
> <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
>
> In fact the answer was quite literally "no":
>
> https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84523
> (the RFE was declined and the answer was that the "function is already
> available in GNR environments").
>
> Regarding GNR, see this RFE request
> https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95090
> requesting the use of GNR outside of an ESS/GSS environment. It's
> interesting to note this is the highest voted Public RFE for GPFS
> that I
> can see, at least. It too was declined.
>
> -Aaron
>
> On 6/30/17 1:41 PM, Aaron Knister wrote:
> > Thanks Olaf, that's good to know (and is kind of what I
> suspected). I've
> > requested a number of times this capability for those of us who
> can't
> > use or aren't using GNR and the answer is effectively "no". This
> > response is curious to me because I'm sure IBM doesn't believe
> that data
> > integrity is only important and of value to customers who
> purchase their
> > hardware *and* software.
> >
> > -Aaron
> >
> > On Fri, Jun 30, 2017 at 1:37 PM, Olaf Weiser
> <olaf.weiser at de.ibm.com <mailto:olaf.weiser at de.ibm.com>
> > <mailto:olaf.weiser at de.ibm.com <mailto:olaf.weiser at de.ibm.com>>>
> wrote:
> >
> > yes.. in case of GNR (GPFS native raid) .. we do end-to-end
> > check-summing ... client --> server --> downToDisk
> > GNR writes down a chksum to disk (to all pdisks /all "raid"
> segments
> > ) so that dropped writes can be detected as well as miss-done
> > writes (bit flips..)
> >
> >
> >
> > From: Aaron Knister <aaron.s.knister at nasa.gov
> <mailto:aaron.s.knister at nasa.gov>
> > <mailto:aaron.s.knister at nasa.gov
> <mailto:aaron.s.knister at nasa.gov>>>
> > To: gpfsug main discussion list
> <gpfsug-discuss at spectrumscale.org
> <mailto:gpfsug-discuss at spectrumscale.org>
> > <mailto:gpfsug-discuss at spectrumscale.org
> <mailto:gpfsug-discuss at spectrumscale.org>>>
> > Date: 06/30/2017 07:15 PM
> > Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS):
> > RDMA-enabled network adapter failure on the NSD server may
> result in
> > file IO error (2017.06.30)
> > Sent by: gpfsug-discuss-bounces at spectrumscale.org
> <mailto:gpfsug-discuss-bounces at spectrumscale.org>
> > <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
> >
> ------------------------------------------------------------------------
> >
> >
> >
> > I'm curious to know why this doesn't affect GSS/ESS? Is it a
> feature of
> > the additional check-summing done on those platforms?
> >
> >
> > -------- Forwarded Message --------
> > Subject: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network
> > adapter
> > failure on the NSD server may result in file IO error
> (2017.06.30)
> > Date: Fri, 30 Jun 2017 14:19:02 +0000
> > From: IBM My Notifications
> > <mynotify at stg.events.ihost.com
> <mailto:mynotify at stg.events.ihost.com>
> <mailto:mynotify at stg.events.ihost.com
> <mailto:mynotify at stg.events.ihost.com>>>
> > To: aaron.s.knister at nasa.gov
> <mailto:aaron.s.knister at nasa.gov> <mailto:aaron.s.knister at nasa.gov
> <mailto:aaron.s.knister at nasa.gov>>
> >
> >
> >
> >
> > My Notifications for Storage - 30 Jun 2017
> >
> > Dear Subscriber (aaron.s.knister at nasa.gov
> <mailto:aaron.s.knister at nasa.gov>
> > <mailto:aaron.s.knister at nasa.gov
> <mailto:aaron.s.knister at nasa.gov>>),
> >
> > Here are your updates from IBM My Notifications.
> >
> > Your support Notifications display in English by default.
> Machine
> > translation based on your IBM profile
> > language setting is added if you specify this option in My
> defaults
> > within My Notifications.
> > (Note: Not all languages are available at this time, and the
> English
> > version always takes precedence
> > over the machine translated version.)
> >
> >
> ------------------------------------------------------------------------------
> > 1. IBM Spectrum Scale
> >
> > - TITLE: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter
> > failure
> > on the NSD server may result in file IO error
> > - URL:
> >
> http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E
> >
> <http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E>
> > - ABSTRACT: IBM has identified an issue with all IBM GPFS
> and IBM
> > Spectrum Scale versions where the NSD server is enabled to
> use RDMA for
> > file IO and the storage used in your GPFS cluster accessed
> via NSD
> > servers (not fully SAN accessible) includes anything other
> than IBM
> > Elastic Storage Server (ESS) or GPFS Storage Server (GSS);
> under these
> > conditions, when the RDMA-enabled network adapter fails, the
> issue may
> > result in undetected data corruption for file write or read
> operations.
> >
> >
> ------------------------------------------------------------------------------
> > Manage your My Notifications subscriptions, or send
> questions and
> > comments.
> > - Subscribe or Unsubscribe -
> > https://www.ibm.com/support/mynotifications
> > <https://www.ibm.com/support/mynotifications>
> > - Feedback -
> >
> https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html
> >
> <https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html>
> >
> > - Follow us on Twitter - https://twitter.com/IBMStorageSupt
> > <https://twitter.com/IBMStorageSupt>
> >
> >
> >
> > To ensure proper delivery please add
> mynotify at stg.events.ihost.com <mailto:mynotify at stg.events.ihost.com>
> > <mailto:mynotify at stg.events.ihost.com
> <mailto:mynotify at stg.events.ihost.com>> to
> > your address book.
> > You received this email because you are subscribed to IBM My
> > Notifications as:
> > aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>
> <mailto:aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>>
> >
> > Please do not reply to this message as it is generated by an
> automated
> > service machine.
> >
> > (C) International Business Machines Corporation 2017. All rights
> > reserved.
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> <http://spectrumscale.org> <http://spectrumscale.org>
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> > <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> >
> >
> >
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> <http://spectrumscale.org> <http://spectrumscale.org>
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> > <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> >
> >
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776 <tel:%28301%29%20286-2776>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170822/d3a0aecb/attachment.htm>
More information about the gpfsug-discuss
mailing list