[gpfsug-discuss] pool block allocation algorithm
Peter Serocka
peserocka at gmail.com
Sat Jan 13 17:00:49 GMT 2018
Within reasonable capacity limits it would also expect
to direct incoming data to disks that are best “available”
from a current performance perspective — like doing least
IOPS, having lowest latency and shortest filled queue length.
You new NSDs, filled only with recent data, might quickly have
become the most busy units before reaching capacity balance,
simply because recent data tends to be more active than older stuff.
Makes sense?
— Peter
> On 2018 Jan 13 Sat, at 17:18, Aaron Knister <aaron.s.knister at nasa.gov> wrote:
>
> Thanks Everyone! I whipped up a script to dump the block layout of a
> file and then join that with mmdf information. As part of my exploration
> I wrote one 2GB file to each of this particular filesystem's 4 data
> pools last night (using "touch $file; mmchattr $file -P $pool; dd
> of=$file") and have attached a dump of the layout/nsd information for
> each file/pool. The fields for the output are:
>
> diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks,
> freePct, freeKbFragments, freeKbFragmentsPct
>
>
> Here's the highlight from pool1:
>
> 36 264 d13_06_006 23437934592 1213 4548935680 (19%)
> 83304320 (0%)
> 59 74 d10_41_025 23437934592 1011 6993759232 (30%)
> 58642816 (0%)
>
> For this file (And anecdotally what I've seen looking at NSD I/O data
> for other files written to this pool) the pattern of more blocks being
> allocated to the NSDs that are ~20% free vs the NSDs that are 30% free
> seems to be fairly consistent.
>
>
> Looking at a snippet of pool2:
> 101 238 d15_15_011 23437934592 1415 2008394752 (9%)
> 181699328 (1%)
> 102 235 d15_16_012 23437934592 1415 2009153536 (9%)
> 182165312 (1%)
> 116 248 d11_42_026 23437934592 1011 4146111488 (18%)
> 134941504 (1%)
> 117 249 d13_42_026 23437934592 1213 4147710976 (18%)
> 135203776 (1%)
>
> there are slightly more blocks allocated in general on the NSDs with
> twice the amount of free space, but it doesn't seem to be a significant
> amount relative to the delta in free space. The pattern from pool1
> certainly doesn't hold true here.
>
> Pool4 isn't very interesting because all of the NSDs are well balanced
> in terms of free space (all 16% free).
>
> Pool3, however, *is* particularly interesting. Here's a snippet:
>
> 106 222 d15_24_016 23437934592 1415 2957561856 (13%)
> 37436768 (0%)
> 107 222 d15_25_017 23437934592 1415 2957537280 (13%)
> 37353984 (0%)
> 108 222 d15_26_018 23437934592 1415 2957539328 (13%)
> 37335872 (0%)
> 125 222 d11_44_028 23437934592 1011 13297235968 (57%)
> 20505568 (0%)
> 126 222 d12_44_028 23437934592 1213 13296712704 (57%)
> 20632768 (0%)
> 127 222 d12_45_029 23437934592 1213 13297404928 (57%)
> 20557408 (0%)
>
> GPFS consistently allocated the same number of blocks to disks with ~4x
> the free space than it did the other disks in the pool.
>
> Suffice it to say-- I'm *very* confused :)
>
> -Aaron
>
> On 1/13/18 8:18 AM, Daniel Kidger wrote:
>> Aaron,
>>
>> Also are your new NSDs the same size as your existing ones?
>> i.e. the NSDs that are at a higher %age full might have more free blocks
>> than the other NSDs?
>> Daniel
>>
>>
>> IBM Storage Professional Badge
>> <https://www.youracclaim.com/user/danel-kidger>
>>
>>
>> *Dr Daniel Kidger*
>> IBM Technical Sales Specialist
>> Software Defined Solution Sales
>>
>> +44-(0)7818 522 266
>> daniel.kidger at uk.ibm.com
>>
>>
>>
>>
>>
>> ----- Original message -----
>> From: Jan-Frode Myklebust <janfrode at tanso.net>
>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>> Cc:
>> Subject: Re: [gpfsug-discuss] pool block allocation algorithm
>> Date: Sat, Jan 13, 2018 9:25 AM
>>
>> Don’t have documentation/whitepaper, but as I recall, it will first
>> allocate round-robin over failureGroup, then round-robin over
>> nsdServers, and then round-robin over volumes. So if these new NSDs
>> are defined as different failureGroup from the old disks, that might
>> explain it..
>>
>>
>> -jf
>> lør. 13. jan. 2018 kl. 00:15 skrev Aaron Knister
>> <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>>:
>>
>> Apologies if this has been covered elsewhere (I couldn't find it
>> if it
>> has). I'm curious how GPFS decides where to allocate new blocks.
>>
>> We've got a filesystem that we added some NSDs to a while back
>> and it
>> hurt there for a little while because it appeared as though GPFS was
>> choosing to allocate new blocks much more frequently on the
>> ~100% free
>> LUNs than the existing LUNs (I can't recall how free they were
>> at the
>> time). Looking at it now, though, it seems GPFS is doing the
>> opposite.
>> There's now a ~10% difference between the LUNs added and the
>> existing
>> LUNs (20% free vs 30% free) and GPFS is choosing to allocate new
>> writes
>> at a ratio of about 3:1 on the disks with *fewer* free blocks
>> than on
>> the disks with more free blocks. That's completely inconsistent with
>> what we saw when we initially added the disks which makes me
>> wonder how
>> GPFS is choosing to allocate new blocks (other than the obvious bits
>> about failure group, and replication factor). Could someone
>> explain (or
>> point me at a whitepaper) what factors GPFS uses when allocating
>> blocks,
>> particularly as it pertains to choosing one NSD over another
>> within the
>> same failure group.
>>
>> Thanks!
>>
>> -Aaron
>>
>> --
>> Aaron Knister
>> NASA Center for Climate Simulation (Code 606.2)
>> Goddard Space Flight Center
>> (301) 286-2776
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=QYsXVDOdNRcII7FPtAbCXEyYJzNSd_UXq8bmreALKxs&e=>
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe-GTf8EwJ6AkZQiTsRQZ73UH20&e=>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe-GTf8EwJ6AkZQiTsRQZ73UH20&e=
>>
>>
>> Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with number
>> 741598.
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>>
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
-------------- next part --------------
1 221 d12_01_001 23437934592 1213 4550865920 (19%) 83734976 (0%)
2 266 d12_02_002 23437934592 1213 4550282240 (19%) 83712000 (0%)
3 265 d12_03_003 23437934592 1213 4550419456 (19%) 83986432 (0%)
4 265 d12_04_004 23437934592 1213 4550207488 (19%) 83382624 (0%)
5 266 d12_05_005 23437934592 1213 4551135232 (19%) 83820832 (0%)
6 264 d12_06_006 23437934592 1213 4549501952 (19%) 83537984 (0%)
31 256 d13_01_001 23437934592 1213 4550247424 (19%) 83557184 (0%)
32 266 d13_02_002 23437934592 1213 4548240384 (19%) 83289888 (0%)
33 264 d13_03_003 23437934592 1213 4548679680 (19%) 83851360 (0%)
34 266 d13_04_004 23437934592 1213 4551052288 (19%) 83524320 (0%)
35 265 d13_05_005 23437934592 1213 4550079488 (19%) 83578624 (0%)
36 264 d13_06_006 23437934592 1213 4548935680 (19%) 83304320 (0%)
59 74 d10_41_025 23437934592 1011 6993759232 (30%) 58642816 (0%)
61 216 d14_01_001 23437934592 1415 4575688704 (20%) 83600032 (0%)
62 266 d14_02_002 23437934592 1415 4574487552 (20%) 83376960 (0%)
63 265 d14_03_003 23437934592 1415 4574819328 (20%) 83378240 (0%)
64 266 d14_04_004 23437934592 1415 4575392768 (20%) 83162080 (0%)
65 264 d14_05_005 23437934592 1415 4576001024 (20%) 83447968 (0%)
66 265 d14_06_006 23437934592 1415 4575699968 (20%) 83043040 (0%)
85 72 d11_41_025 23437934592 1011 6991225856 (30%) 58576768 (0%)
86 66 d12_41_025 23437934592 1213 6992339968 (30%) 58402688 (0%)
87 71 d12_42_026 23437934592 1213 6992146432 (30%) 58284032 (0%)
88 67 d13_41_025 23437934592 1213 6993167360 (30%) 58134624 (0%)
89 75 d14_41_025 23437934592 1415 6992531456 (30%) 58169600 (0%)
90 71 d14_42_026 23437934592 1415 6993435648 (30%) 58234720 (0%)
91 265 d15_01_001 23437934592 1415 4575676416 (20%) 83394144 (0%)
92 264 d15_02_002 23437934592 1415 4576221184 (20%) 82999168 (0%)
93 264 d15_03_003 23437934592 1415 4576231424 (20%) 83179680 (0%)
94 266 d15_04_004 23437934592 1415 4577104896 (20%) 83445088 (0%)
95 265 d15_05_005 23437934592 1415 4576627712 (20%) 82928064 (0%)
96 259 d15_06_006 23437934592 1415 4576740352 (20%) 83355776 (0%)
115 73 d15_41_025 23437934592 1415 6990783488 (30%) 58595296 (0%)
145 266 d10_01_001 23437934592 1011 4588489728 (20%) 83100448 (0%)
146 266 d10_02_002 23437934592 1011 4587276288 (20%) 83239488 (0%)
147 265 d10_03_003 23437934592 1011 4586872832 (20%) 83024000 (0%)
148 264 d10_04_004 23437934592 1011 4585822208 (20%) 82897920 (0%)
149 264 d10_05_005 23437934592 1011 4586415104 (20%) 83277056 (0%)
150 266 d10_06_006 23437934592 1011 4587424768 (20%) 83076544 (0%)
151 265 d11_01_001 23437934592 1011 4586209280 (20%) 83178368 (0%)
152 265 d11_02_002 23437934592 1011 4587372544 (20%) 83091232 (0%)
153 264 d11_03_003 23437934592 1011 4587196416 (20%) 83147040 (0%)
154 264 d11_04_004 23437934592 1011 4586633216 (20%) 83049024 (0%)
155 265 d11_05_005 23437934592 1011 4586576896 (20%) 83188768 (0%)
156 264 d11_06_006 23437934592 1011 4587261952 (20%) 83393952 (0%)
-------------- next part --------------
7 238 d12_11_007 23437934592 1213 2005351424 (9%) 184092352 (1%)
8 237 d12_12_008 23437934592 1213 2005532672 (9%) 183893472 (1%)
9 243 d12_13_009 23437934592 1213 2004583424 (9%) 183000704 (1%)
10 239 d12_14_010 23437934592 1213 2004461568 (9%) 182958048 (1%)
11 228 d12_15_011 23437934592 1213 2004750336 (9%) 183160064 (1%)
12 240 d12_16_012 23437934592 1213 2008772608 (9%) 183263936 (1%)
37 228 d13_11_007 23437934592 1213 2006091776 (9%) 183041024 (1%)
38 230 d13_12_008 23437934592 1213 2003761152 (9%) 182625024 (1%)
39 229 d13_13_009 23437934592 1213 2004123648 (9%) 182907552 (1%)
40 208 d13_14_010 23437934592 1213 2006372352 (9%) 182797376 (1%)
41 233 d13_15_011 23437934592 1213 2003358720 (9%) 182322880 (1%)
42 232 d13_16_012 23437934592 1213 2004774912 (9%) 182729792 (1%)
67 234 d14_11_007 23437934592 1415 2006553600 (9%) 182736800 (1%)
68 249 d14_12_008 23437934592 1415 2010278912 (9%) 182304832 (1%)
69 226 d14_13_009 23437934592 1415 2008749056 (9%) 182854464 (1%)
70 237 d14_14_010 23437934592 1415 2009348096 (9%) 182257024 (1%)
71 249 d14_15_011 23437934592 1415 2008017920 (9%) 182217152 (1%)
72 233 d14_16_012 23437934592 1415 2008158208 (9%) 182239680 (1%)
97 237 d15_11_007 23437934592 1415 2007257088 (9%) 181898304 (1%)
98 233 d15_12_008 23437934592 1415 2008309760 (9%) 182000928 (1%)
99 229 d15_13_009 23437934592 1415 2008755200 (9%) 182380544 (1%)
100 224 d15_14_010 23437934592 1415 2008460288 (9%) 181691616 (1%)
101 238 d15_15_011 23437934592 1415 2008394752 (9%) 181699328 (1%)
102 235 d15_16_012 23437934592 1415 2009153536 (9%) 182165312 (1%)
116 248 d11_42_026 23437934592 1011 4146111488 (18%) 134941504 (1%)
117 249 d13_42_026 23437934592 1213 4147710976 (18%) 135203776 (1%)
118 248 d15_42_026 23437934592 1415 4147094528 (18%) 134748320 (1%)
119 249 d10_43_027 23437934592 1011 4148652032 (18%) 135124288 (1%)
120 247 d11_43_027 23437934592 1011 4147335168 (18%) 134808256 (1%)
121 250 d12_43_027 23437934592 1213 4147177472 (18%) 134812096 (1%)
122 248 d13_43_027 23437934592 1213 4147270656 (18%) 134516192 (1%)
123 248 d14_43_027 23437934592 1415 4148153344 (18%) 134928896 (1%)
124 248 d15_43_027 23437934592 1415 4149500928 (18%) 134818880 (1%)
157 247 d10_11_007 23437934592 1011 2008516608 (9%) 182482592 (1%)
159 248 d10_13_009 23437934592 1011 2009745408 (9%) 182393184 (1%)
161 233 d10_15_011 23437934592 1011 2007539712 (9%) 182169856 (1%)
163 234 d11_11_007 23437934592 1011 2009283584 (9%) 182173824 (1%)
164 244 d11_12_008 23437934592 1011 2009091072 (9%) 182099200 (1%)
165 248 d11_13_009 23437934592 1011 2008827904 (9%) 182029632 (1%)
166 231 d11_14_010 23437934592 1011 2010264576 (9%) 181675424 (1%)
167 235 d11_15_011 23437934592 1011 2010696704 (9%) 181854304 (1%)
168 236 d11_16_012 23437934592 1011 2008183808 (9%) 182236800 (1%)
-------------- next part --------------
13 222 d12_21_013 23437934592 1213 2957768704 (13%) 37317824 (0%)
14 223 d12_22_014 23437934592 1213 2957680640 (13%) 37608576 (0%)
15 223 d12_23_015 23437934592 1213 2958575616 (13%) 37380672 (0%)
16 223 d12_24_016 23437934592 1213 2957406208 (13%) 37403648 (0%)
17 223 d12_25_017 23437934592 1213 2957883392 (13%) 37558016 (0%)
18 222 d12_26_018 23437934592 1213 2957815808 (13%) 37556416 (0%)
43 222 d13_21_013 23437934592 1213 2957149184 (13%) 37745280 (0%)
44 222 d13_22_014 23437934592 1213 2957812736 (13%) 37520736 (0%)
45 222 d13_23_015 23437934592 1213 2957990912 (13%) 37341952 (0%)
46 222 d13_24_016 23437934592 1213 2958204928 (13%) 37477536 (0%)
47 222 d13_25_017 23437934592 1213 2958315520 (13%) 37507072 (0%)
48 222 d13_26_018 23437934592 1213 2958086144 (13%) 37407648 (0%)
73 222 d14_21_013 23437934592 1415 2957812736 (13%) 37631488 (0%)
74 223 d14_22_014 23437934592 1415 2959673344 (13%) 37427680 (0%)
75 223 d14_23_015 23437934592 1415 2957627392 (13%) 37400992 (0%)
76 223 d14_24_016 23437934592 1415 2957408256 (13%) 37235776 (0%)
77 222 d14_25_017 23437934592 1415 2957407232 (13%) 37499200 (0%)
78 222 d14_26_018 23437934592 1415 2956661760 (13%) 37633696 (0%)
103 222 d15_21_013 23437934592 1415 2957427712 (13%) 37629664 (0%)
104 222 d15_22_014 23437934592 1415 2957506560 (13%) 37571968 (0%)
105 222 d15_23_015 23437934592 1415 2958114816 (13%) 37422400 (0%)
106 222 d15_24_016 23437934592 1415 2957561856 (13%) 37436768 (0%)
107 222 d15_25_017 23437934592 1415 2957537280 (13%) 37353984 (0%)
108 222 d15_26_018 23437934592 1415 2957539328 (13%) 37335872 (0%)
125 222 d11_44_028 23437934592 1011 13297235968 (57%) 20505568 (0%)
126 222 d12_44_028 23437934592 1213 13296712704 (57%) 20632768 (0%)
127 222 d12_45_029 23437934592 1213 13297404928 (57%) 20557408 (0%)
128 222 d13_44_028 23437934592 1213 13297271808 (57%) 20502304 (0%)
129 222 d14_44_028 23437934592 1415 13294722048 (57%) 20569824 (0%)
130 222 d14_45_029 23437934592 1415 13296071680 (57%) 20667104 (0%)
131 222 d15_44_028 23437934592 1415 13297007616 (57%) 20429536 (0%)
132 223 d10_44_028 23437934592 1011 13309036544 (57%) 20299552 (0%)
133 223 d10_45_029 23437934592 1011 13309213696 (57%) 20464576 (0%)
169 223 d10_21_013 23437934592 1011 2797943808 (12%) 36881600 (0%)
170 222 d10_22_014 23437934592 1011 2798030848 (12%) 37152256 (0%)
171 222 d10_23_015 23437934592 1011 2798152704 (12%) 36959840 (0%)
172 222 d10_24_016 23437934592 1011 2798002176 (12%) 36954976 (0%)
173 222 d10_25_017 23437934592 1011 2798524416 (12%) 36737120 (0%)
174 222 d10_26_018 23437934592 1011 2797768704 (12%) 36888096 (0%)
175 222 d11_21_013 23437934592 1011 2958944256 (13%) 37504288 (0%)
176 222 d11_22_014 23437934592 1011 2957225984 (13%) 37575648 (0%)
177 222 d11_23_015 23437934592 1011 2958100480 (13%) 37546336 (0%)
178 222 d11_24_016 23437934592 1011 2958453760 (13%) 37578784 (0%)
179 222 d11_25_017 23437934592 1011 2958053376 (13%) 37331808 (0%)
180 222 d11_26_018 23437934592 1011 2958290944 (13%) 37445824 (0%)
-------------- next part --------------
19 223 d12_31_019 23437934592 1213 3803873280 (16%) 117281408 (1%)
20 222 d12_32_020 23437934592 1213 3779237888 (16%) 116652576 (0%)
21 222 d12_33_021 23437934592 1213 3779206144 (16%) 116820288 (0%)
22 222 d12_34_022 23437934592 1213 3776828416 (16%) 116108032 (0%)
23 222 d12_35_023 23437934592 1213 3776950272 (16%) 116238752 (0%)
24 222 d12_36_024 23437934592 1213 3776524288 (16%) 116599744 (0%)
49 222 d13_31_019 23437934592 1213 3778293760 (16%) 116365600 (0%)
50 222 d13_32_020 23437934592 1213 3778381824 (16%) 115960448 (0%)
51 222 d13_33_021 23437934592 1213 3777104896 (16%) 116225984 (0%)
52 222 d13_34_022 23437934592 1213 3775632384 (16%) 116496576 (0%)
53 222 d13_35_023 23437934592 1213 3776899072 (16%) 116397120 (0%)
54 222 d13_36_024 23437934592 1213 3776140288 (16%) 116400000 (0%)
79 222 d14_31_019 23437934592 1415 3844638720 (16%) 113531904 (0%)
80 222 d14_32_020 23437934592 1415 3817139200 (16%) 113600928 (0%)
81 222 d14_33_021 23437934592 1415 3815447552 (16%) 113358272 (0%)
82 222 d14_34_022 23437934592 1415 3814561792 (16%) 113348160 (0%)
83 222 d14_35_023 23437934592 1415 3815387136 (16%) 113582432 (0%)
84 222 d14_36_024 23437934592 1415 3815303168 (16%) 112914816 (0%)
109 222 d15_31_019 23437934592 1415 3814614016 (16%) 113298240 (0%)
110 222 d15_32_020 23437934592 1415 3814936576 (16%) 113053664 (0%)
111 222 d15_33_021 23437934592 1415 3815122944 (16%) 113438656 (0%)
112 222 d15_34_022 23437934592 1415 3813474304 (16%) 113481216 (0%)
113 222 d15_35_023 23437934592 1415 3812912128 (16%) 113618624 (0%)
114 222 d15_36_024 23437934592 1415 3813067776 (16%) 113282848 (0%)
134 223 d11_45_029 23437934592 1011 3683404800 (16%) 100860000 (0%)
135 222 d13_45_029 23437934592 1213 3682211840 (16%) 101101824 (0%)
136 223 d15_45_029 23437934592 1415 3682046976 (16%) 101375328 (0%)
137 222 d10_46_030 23437934592 1011 3684398080 (16%) 100946912 (0%)
138 222 d11_46_030 23437934592 1011 3683755008 (16%) 101295200 (0%)
139 223 d12_46_030 23437934592 1213 3682649088 (16%) 100540832 (0%)
140 223 d13_46_030 23437934592 1213 3684602880 (16%) 100489376 (0%)
141 223 d14_46_030 23437934592 1415 3681666048 (16%) 100854112 (0%)
142 223 d15_46_030 23437934592 1415 3685487616 (16%) 100715744 (0%)
181 222 d10_31_019 23437934592 1011 3816681472 (16%) 113389344 (0%)
182 222 d10_32_020 23437934592 1011 3816535040 (16%) 113652672 (0%)
183 222 d10_33_021 23437934592 1011 3817772032 (16%) 113124224 (0%)
184 222 d10_34_022 23437934592 1011 3817069568 (16%) 113225504 (0%)
185 222 d10_35_023 23437934592 1011 3816517632 (16%) 113163488 (0%)
186 222 d10_36_024 23437934592 1011 3815605248 (16%) 113475648 (0%)
187 222 d11_31_019 23437934592 1011 3815664640 (16%) 113100992 (0%)
188 222 d11_32_020 23437934592 1011 3815985152 (16%) 113325504 (0%)
189 222 d11_33_021 23437934592 1011 3815148544 (16%) 113328480 (0%)
190 223 d11_34_022 23437934592 1011 3814780928 (16%) 113457440 (0%)
191 223 d11_35_023 23437934592 1011 3815219200 (16%) 113065504 (0%)
192 223 d11_36_024 23437934592 1011 3815383040 (16%) 113170592 (0%)
-------------- next part --------------
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
More information about the gpfsug-discuss
mailing list