[gpfsug-discuss] mmfsd write behavior

[Aaron Knister]

Hi Sven,

Just wondering if you've had any additional thoughts/conversations about 
this.

-Aaron

On 9/8/17 5:21 PM, Sven Oehme wrote:
> Hi,
> 
> the code assumption is that the underlying device has no volatile write 
> cache, i was absolute sure we have that somewhere in the FAQ, but i 
> couldn't find it, so i will talk to somebody to correct this.
> if i understand 
> https://www.kernel.org/doc/Documentation/block/writeback_cache_control.txt correct 
> one could enforce this by setting REQ_FUA, but thats not explicitly set 
> today, at least i can't see it. i will discuss this with one of our devs 
> who owns this code and come back.
> 
> sven
> 
> 
> On Thu, Sep 7, 2017 at 8:05 PM Aaron Knister <aaron.s.knister at nasa.gov 
> <mailto:aaron.s.knister at nasa.gov>> wrote:
> 
>     Thanks Sven. I didn't think GPFS itself was caching anything on that
>     layer, but it's my understanding that O_DIRECT isn't sufficient to force
>     I/O to be flushed (e.g. the device itself might have a volatile caching
>     layer). Take someone using ZFS zvol's as NSDs. I can write() all day log
>     to that zvol (even with O_DIRECT) but there is absolutely no guarantee
>     those writes have been committed to stable storage and aren't just
>     sitting in RAM until an fsync() occurs (or some other bio function that
>     causes a flush). I also don't believe writing to a SATA drive with
>     O_DIRECT will force cache flushes of the drive's writeback cache..
>     although I just tested that one and it seems to actually trigger a scsi
>     cache sync. Interesting.
> 
>     -Aaron
> 
>     On 9/7/17 10:55 PM, Sven Oehme wrote:
>      > I am not sure what exactly you are looking for but all
>     blockdevices are
>      > opened with O_DIRECT , we never cache anything on this layer .
>      >
>      >
>      > On Thu, Sep 7, 2017, 7:11 PM Aaron Knister
>     <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>
>      > <mailto:aaron.s.knister at nasa.gov
>     <mailto:aaron.s.knister at nasa.gov>>> wrote:
>      >
>      >     Hi Everyone,
>      >
>      >     This is something that's come up in the past and has recently
>     resurfaced
>      >     with a project I've been working on, and that is-- it seems
>     to me as
>      >     though mmfsd never attempts to flush the cache of the block
>     devices its
>      >     writing to (looking at blktrace output seems to confirm
>     this). Is this
>      >     actually the case? I've looked at the gpl headers for linux
>     and I don't
>      >     see any sign of blkdev_fsync, blkdev_issue_flush, WRITE_FLUSH, or
>      >     REQ_FLUSH. I'm sure there's other ways to trigger this
>     behavior that
>      >     GPFS may very well be using that I've missed. That's why I'm
>     asking :)
>      >
>      >     I figure with FPO being pushed as an HDFS replacement using
>     commodity
>      >     drives this feature has *got* to be in the code somewhere.
>      >
>      >     -Aaron
>      >
>      >     --
>      >     Aaron Knister
>      >     NASA Center for Climate Simulation (Code 606.2)
>      >     Goddard Space Flight Center
>      > (301) 286-2776 <tel:(301)%20286-2776>
>      >     _______________________________________________
>      >     gpfsug-discuss mailing list
>      >     gpfsug-discuss at spectrumscale.org
>     <http://spectrumscale.org> <http://spectrumscale.org>
>      > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>      >
>      >
>      >
>      > _______________________________________________
>      > gpfsug-discuss mailing list
>      > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>      > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>      >
> 
>     --
>     Aaron Knister
>     NASA Center for Climate Simulation (Code 606.2)
>     Goddard Space Flight Center
>     (301) 286-2776 <tel:(301)%20286-2776>
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776