[gpfsug-discuss] LROC
Sven Oehme
oehmes at us.ibm.com
Wed Dec 21 15:39:16 GMT 2016
close, but not 100% :-)
LROC only needs a StatCache Object for files that don't have a Full
OpenFile (maxFilestoCache) Object and you still want to be able to hold
Metadata and/or Data in LROC.
e.g. you can have a OpenFile instance that has Data blocks in LROC, but no
Metadata (as everything is in the OpenFile Object itself), then you don't
need a maxStatCache Object for this one.
but you would need a StatCache object if we have to throw this file
metadata or data out of the FileCache and/or Pagepool as we would otherwise
loose all references to that file in LROC. the MaxStat Object is the most
compact form to hold only references to the real data.
if its still unclear we might have to do a small writeup in form of a paper
with a diagram to better explain it, but that would take a while due to a
lot of other work ahead of that :-)
sven
------------------------------------------
Sven Oehme
Scalable Storage Research
email: oehmes at us.ibm.com
Phone: +1 (408) 824-8904
IBM Almaden Research Lab
------------------------------------------
From: Stephen Ulmer <ulmer at ulmer.org>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: 12/21/2016 04:17 PM
Subject: Re: [gpfsug-discuss] LROC
Sent by: gpfsug-discuss-bounces at spectrumscale.org
Sven,
I’ve read this several times, and it will help me to re-state it. Please
tell me if this is not what you meant:
You often see even common operations (like ls) blow out the StatCache, and
things are inefficient when the StatCache is in use but constantly overrun.
Because of this, you normally recommend disabling the StatCache with
maxStatCache=0, and instead spend the memory normally used for StatCache on
the FileCache.
In the case of LROC, there *must* be a StatCache entry for every file that
is held in the LROC. In this case, we want to set maxStatCache at least as
large as the number of files whose data or metadata we’d like to be in the
LROC.
Close?
--
Stephen
On Dec 21, 2016, at 6:57 AM, Sven Oehme <oehmes at gmail.com> wrote:
its not the only place used, but we see that most calls for
attributes even from simplest ls requests are beyond what the
StatCache provides, therefore my advice is always to disable
maxStatCache by setting it to 0 and raise the maxFilestoCache limit
to a higher than default as the memory is better spent there than
wasted on StatCache, there is also waste by moving back and forth
between StatCache and FileCache if you constantly need more that what
the FileCache provides, so raising it and reduce StatCache to zero
eliminates this overhead (even its just a few cpu cycles).
on LROC its essential as a LROC device can only keep data or Metadata
for files it wants to hold any references if it has a StatCache
object available, this means if your StatCache is set to 10000 and
lets say you have 100000 files you want to cache in LROC this would
never work as we throw the oldest out of LROC as soon as we try to
cache nr 10001 as we have to reuse a StatCache Object to keep the
reference to the data or metadata block stored in LROC .
Sven
On Wed, Dec 21, 2016 at 12:48 PM Peter Childs <p.childs at qmul.ac.uk>
wrote:
So your saying maxStatCache should be raised on LROC enabled nodes
only as its the only place under Linux its used and should be set
low on non-LROC enabled nodes.
Fine just good to know, nice and easy now with nodeclasses....
Peter Childs
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org <
gpfsug-discuss-bounces at spectrumscale.org> on behalf of Sven Oehme <
oehmes at gmail.com>
Sent: Wednesday, December 21, 2016 11:37:46 AM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] LROC
StatCache is not useful on Linux, that hasn't changed if you don't
use LROC on the same node. LROC uses the compact object (StatCache)
to store its pointer to the full file Object which is stored on the
LROC device. so on a call for attributes that are not in the
StatCache the object gets recalled from LROC and converted back
into a full File Object, which is why you still need to have a
reasonable maxFiles setting even you use LROC as you otherwise
constantly move file infos in and out of LROC and put the device
under heavy load.
sven
On Wed, Dec 21, 2016 at 12:29 PM Peter Childs <p.childs at qmul.ac.uk
<mailto:p.childs at qmul.ac.uk>> wrote:
My understanding was the maxStatCache was only used on AIX and
should be set low on Linux, as raising it did't help and wasted
resources. Are we saying that LROC now uses it and setting it low
if you raise maxFilesToCache under linux is no longer the advice.
Peter Childs
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:
gpfsug-discuss-bounces at spectrumscale.org> <
gpfsug-discuss-bounces at spectrumscale.org<mailto:
gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Sven Oehme
<oehmes at gmail.com<mailto:oehmes at gmail.com>>
Sent: Wednesday, December 21, 2016 9:23:16 AM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] LROC
Lroc only needs a StatCache object as it 'compacts' a full open
File object (maxFilesToCache) to a StatCache Object when it moves
the content to the LROC device.
therefore the only thing you really need to increase is
maxStatCache on the LROC node, but you still need maxFiles Objects,
so leave that untouched and just increas maxStat
Olaf's comment is important you need to make sure your manager
nodes have enough memory to hold tokens for all the objects you
want to cache, but if the memory is there and you have enough its
well worth spend a lot of memory on it and bump maxStatCache to a
high number. i have tested maxStatCache up to 16 million at some
point per node, but if nodes with this large amount of inodes crash
or you try to shut them down you have some delays , therefore i
suggest you stay within a 1 or 2 million per node and see how well
it does and also if you get a significant gain.
i did help Bob to setup some monitoring for it so he can actually
get comparable stats, i suggest you setup Zimon and enable the Lroc
sensors to have real stats too , so you can see what benefits you
get.
Sven
On Tue, Dec 20, 2016 at 8:13 PM Matt Weil <mweil at wustl.edu<mailto:
mweil at wustl.edu><mailto:mweil at wustl.edu<mailto:mweil at wustl.edu>>>
wrote:
as many as possible and both
have maxFilesToCache 128000
and maxStatCache 40000
do these effect what sits on the LROC as well? Are those to small?
1million seemed excessive.
On 12/20/16 11:03 AM, Sven Oehme wrote:
how much files do you want to cache ?
and do you only want to cache metadata or also data associated to
the files ?
sven
On Tue, Dec 20, 2016 at 5:35 PM Matt Weil <mweil at wustl.edu<mailto:
mweil at wustl.edu><mailto:mweil at wustl.edu<mailto:mweil at wustl.edu>>>
wrote:
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20
(GPFS)/page/Flash%20Storage
<
https://www.ibm.com/developerworks/community/wikis/home?lang=en#%21/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Flash%20Storage
>
Hello all,
Are there any tuning recommendations to get these to cache more
metadata?
Thanks
Matt
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org><
http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org><
http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org><
http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20161221/572c5523/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20161221/572c5523/attachment-0002.gif>
More information about the gpfsug-discuss
mailing list