[gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat"
Marc A Kaplan
makaplan at us.ibm.com
Mon Jun 27 15:49:27 BST 2016
I'd like to see comments and/or questions and/or suggestions from any
others who have implemented, deployed or seriously planned using FILE_HEAT
and/or GROUP POOL.
We implemented this a few years ago. Since we haven't had much feedback,
it's either perfect or ... ? ;-)
Alex, my reading of your story is that your hardware does not match your
requirements, capacity, speeds, feeds.
One new possibly helpful improvement is that SS release 4.2 includes QOSio
which allows one to control the IOPs consumed by migration.
So you can control the performance "hit" of running mmapplypolicy. You
can even control it on-the-fly without killing a long running job
-- for example you can throttle it back to 100iops during prime-time day
shift operations
-- marc of GPFS.
From: Alex Chekholko <chekh at stanford.edu>
To: gpfsug-discuss at spectrumscale.org
Date: 06/24/2016 02:10 PM
Subject: Re: [gpfsug-discuss] Blog on "Automatic tiering of object
data based on object heat"
Sent by: gpfsug-discuss-bounces at spectrumscale.org
On 06/21/2016 10:25 PM, Gaurang Tapase wrote:
> Hello,
>
> Based on inputs from some of the Spectrum Scale users, sharing a blog
> link for "Automatic tiering of object data based on object heat" by
Smita.
>
https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/
>
Thanks for the blog post. I wanted to post my experiences:
Note "sata0" is really NL-SAS.
[root at scg-gs0 ~]# df -h|grep gsfs
/dev/gsfs0 4.4P 3.1P 1.4P 70% /srv/gsfs0
[root at scg-gs0 ~]# df -i|grep gsfs
/dev/gsfs0 314572800 234237984 80334816 75% /srv/gsfs0
[root at scg-gs0 ~]# mmlspool gsfs0
Storage pools in file system at '/srv/gsfs0':
Name Id BlkSize Data Meta Total Data in (KB) Free
Data in (KB) Total Meta in (KB) Free Meta in (KB)
system 0 256 KB yes yes 1887436800
118287616 ( 6%) 1887436800 328248832 ( 17%)
sas0 65537 256 KB yes no 259795189760
59495012864 ( 23%) 0 0 ( 0%)
sata0 65538 256 KB yes no 4523875631104
1405369561600 ( 31%) 0 0 ( 0%)
[root at scg-gs0 ~]# mmlspolicy gsfs0 -L
RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch'
RULE 'default' SET POOL 'sas0'
[root at scg-gs0 ~]# cat policy_gpool.txt
rule 'grpdef' GROUP POOL 'gpool' IS 'system' LIMIT(80) THEN 'sas0'
LIMIT(80) THEN 'sata0'
rule 'repack' MIGRATE FROM POOL 'gpool' TO POOL 'gpool' WEIGHT(FILE_HEAT)
[root at scg-gs0 ~]# mmlsconfig|grep -i heat
fileHeatPeriodMinutes 1440
fileHeatLossPercent 1
The first thing to know is that the MIGRATE rules only run when you run
'mmapplypolicy', even is you have the migrate policy loaded. So you
have to decide when/how to do that.
The filesystem scan takes a while; here are the parameters that let a
full scan of our filesystem complete in under 10mins (8 I/O servers,
~300M inodes):
mmapplypolicy gsfs0 -P policy.txt -N
scg-gs0,scg-gs1,scg-gs2,scg-gs3,scg-gs4,scg-gs5,scg-gs6,scg-gs7 -L 2 -g
/srv/gsfs0/admin_stuff/ -I yes -B 500 -A 61 -a 4
The issues we have are:
1) the pools are not sized correctly. We have 3 tiers: 1.8TB SSD, 259TB
10K SAS, 4523 TB 7.2K NL-SAS (0.03% ssd, 5.4% “fast”, 94.57% “slow”)
2) the client I/O can max out the IOPS in the "slow" pool and in the
"fast" pool
The original idea was to have all client I/O go to the 10K pool, then
migrate least-used data to slow pool. Unfortunately our fast pool fills
up too quickly and then GPFS can't empty it quickly enough into the slow
pool. So I end up switching the placement policy to put files into the
slow pool. And then emptying the fast pool into the slow pool doesn't
work too well because the slow pool is already maxed out on IOPS just
handling the workload. So moving the data from fast to slow takes weeks
and impacts performance.
The original intention was:
[root at scg-gs0 ~]# cat policy_placement.txt
RULE 'faster01' SET POOL 'system' WHERE FILESET_NAME LIKE 'home.%'
RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch'
RULE 'default' SET POOL 'sas0'
I would recommend that your total workload fits in both capacity and
performance into your "primary" aka "fast" pool.
I also just learned recently of the LIMIT parameter for placement, e.g.
RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch'
RULE 'default' SET POOL 'sas0' LIMIT (95)
RULE 'spillover' SET POOL 'sata0'
So that would help with the pool not filling, but won't help with not
having enough I/O for migration+clients.
Regards,
--
Alex Chekholko chekh at stanford.edu
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160627/74674eac/attachment-0001.htm>
More information about the gpfsug-discuss
mailing list