[gpfsug-discuss] Blocksize - file size distribution
Buterbaugh, Kevin L
Kevin.Buterbaugh at Vanderbilt.Edu
Wed Sep 28 14:45:14 BST 2016
Greg,
Not saying this is the right way to go, but I rolled my own. I wrote a very simple Perl script that essentially does the Perl equivalent of a find on my GPFS filesystems, then stat’s files and directories and writes the output to a text file. I run that one overnight or on the weekends. Takes about 6 hours to run across our 3 GPFS filesystems with metadata on SSDs.
Then I’ve written a couple of different Perl scripts to analyze that data in different ways (the idea being that collecting the data is “expensive” … but once you’ve got it it’s cheap to analyze it in different ways). But the one I’ve been using for this project just breaks down the number of files and directories by size and age and produces a table. Rather than try to describe this, here’s sample output:
For input file: gpfsFileInfo_20160915.txt
<1 day | <1 wk | <1 mo | <2 mo | <3 mo | <4 mo | <5 mo | <6 mo | <1 yr | >1 year | Total Files
<1 KB 29538 111364 458260 634398 150199 305715 4388443 93733 966618 3499535 10637803
<2 KB 9875 20580 119414 295167 35961 67761 80462 33688 269595 851641 1784144
<4 KB 9212 45282 168678 496796 27771 23887 105135 23161 259242 1163327 2322491
<8 KB 4992 29284 105836 303349 28341 20346 246430 28394 216061 1148459 2131492
<16 KB 3987 18391 92492 218639 20513 19698 675097 30976 190691 851533 2122017
<32 KB 4844 12479 50235 265222 24830 18789 1058433 18030 196729 1066287 2715878
<64 KB 6358 24259 29474 222134 17493 10744 1381445 11358 240528 1123540 3067333
<128 KB 6531 59107 206269 186213 71823 114235 1008724 36722 186357 845921 2721902
<256 KB 1995 17638 19355 436611 8505 7554 3582738 7519 249510 744885 5076310
<512 KB 20645 12401 24700 111463 5659 22132 1121269 10774 273010 725155 2327208
<1 MB 2681 6482 37447 58459 6998 14945 305108 5857 160360 386152 984489
<4 MB 4554 84551 23320 100407 6818 32833 129758 22774 210935 528458 1144408
<1 GB 56652 33538 99667 87778 24313 68372 118928 42554 251528 916493 1699823
<10 GB 1245 2482 4524 3184 1043 1794 2733 1694 8731 20462 47892
<100 GB 47 230 470 265 92 198 172 122 1276 2061 4933
>100 GB 2 3 12 1 14 4 5 1 37 165 244
Total TB: 6.49 13.22 30.56 18.00 10.22 15.69 19.87 12.48 73.47 187.44
Grand Total: 387.46 TB
Everything other than the total space lines at the bottom are counts of number of files meeting that criteria. I’ve got another variation on the same script that we used when we were trying to determine how many old files we have and therefore how much data was an excellent candidate for moving to a slower, less expensive “capacity” pool.
I’m not sure how useful my tools would be to others … I’m certainly not a professional programmer by any stretch of the imagination (and yes, I do hear those of you who are saying, “Yeah, he’s barely a professional SysAdmin!” <grin>). But others of you have been so helpful to me … I’d like to try in some small way to help someone else.
Kevin
On Sep 28, 2016, at 5:56 AM, Oesterlin, Robert <Robert.Oesterlin at nuance.com<mailto:Robert.Oesterlin at nuance.com>> wrote:
/usr/lpp/mmfs/samples/debugtools/filehist
Look at the README in that directory.
Bob Oesterlin
Sr Storage Engineer, Nuance HPC Grid
From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Greg.Lehmann at csiro.au<mailto:Greg.Lehmann at csiro.au>" <Greg.Lehmann at csiro.au<mailto:Greg.Lehmann at csiro.au>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, September 28, 2016 at 2:40 AM
To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Blocksize
I am wondering what people use to produce a file size distribution report for their filesystems. Has everyone rolled their own or is there some goto app to use.
Cheers,
Greg
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: Wednesday, 28 September 2016 7:21 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Blocksize
Hi All,
Again, my thanks to all who responded to my last post. Let me begin by stating something I unintentionally omitted in my last post … we already use SSDs for our metadata.
Which leads me to yet another question … of my three filesystems, two (/home and /scratch) are much older (created in 2010) and therefore currently have a 512 byte inode size. /data is newer and has a 4K inode size. Now if I combine /scratch and /data into one filesystem with a 4K inode size, the amount of space used by all the inodes coming over from /scratch is going to grow by a factor of eight unless I’m horribly confused. And I would assume I need to count the amount of space taken up by allocated inodes, not just used inodes.
Therefore … how much space my metadata takes up just grew significantly in importance since: 1) metadata is on very expensive enterprise class, vendor certified SSDs, 2) we use RAID 1 mirrors of those SSDs, and 3) we have metadata replication set to two. Some of the information presented by Sven and Yuri seems to contradict each other in regards to how much space inodes take up … or I’m misunderstanding one or both of them! Leaving aside replication, if I use a 256K block size for my metadata and I use 4K inodes, are those inodes going to take up 4K each or are they going to take up 8K each (1/32nd of a 256K block)?
By the way, I do have a file size / file age spreadsheet for each of my filesystems (which I would be willing to share with interested parties) and while I was not surprised to learn that I have over 10 million sub-1K files on /home, I was a bit surprised to find that I have almost as many sub-1K files on /scratch (and a few million more on /data). So there’s a huge potential win in having those files in the inode on SSD as opposed to on spinning disk, but there’s also a huge potential $$$ cost.
Thanks again … I hope others are gaining useful information from this thread. I sure am!
Kevin
On Sep 27, 2016, at 1:26 PM, Yuri L Volobuev <volobuev at us.ibm.com<mailto:volobuev at us.ibm.com>> wrote:
> 1) Let’s assume that our overarching goal in configuring the block
> size for metadata is performance from the user perspective … i.e.
> how fast is an “ls -l” on my directory? Space savings aren’t
> important, and how long policy scans or other “administrative” type
> tasks take is not nearly as important as that directory listing.
> Does that change the recommended metadata block size?
The performance challenges for the "ls -l" scenario are quite different from the policy scan scenario, so the same rules do not necessarily apply.
During "ls -l" the code has to read inodes one by one (there's some prefetching going on, to take the edge off for the actual 'ls' thread, but prefetching is still one inode at a time). Metadata block size doesn't really come into the picture in this case, but inode size can be important -- depending on the storage performance characteristics. Does the storage you use for metadata exhibit a meaningfully different latency for 4K random reads vs 512 byte random reads? In my personal experience, on any modern storage device the difference is non-existent; in fact many devices (like all flash-based storage) use 4K native physical block size, and merely emulate 512 byte "sectors", so there's no way to read less than 4K. So from the inode read latency point of view 4K vs 512B is most likely a wash, but then 4K inodes can help improve performance of other operations, e.g. readdir of a small directory which fits entirely into the inode. If you use xattrs (e.g. as a side effect of using HSM), 4K inodes definitely help, but allowing xattrs to be stored in the inode.
Policy scans reads inodes in full blocks, and there both metadata block size and inode size matter. Larger blocks could improve the inode read performance, while larger inodes mean that the same number of blocks hold fewer inodes and thus more blocks need to be read. So the policy inode scan performance is benefited by larger metadata block size and smaller inodes. However, policy scans also have to perform a directory traversal step, and that step tends to dominate the runtime of the overall run, and using larger inodes actually helps to speed up traversal of smaller directories. So whether larger inodes help or hurt the policy scan performance depends, yet again, on your file system composition. Overall, I believe that with all angles considered, larger inodes help with performance, and that was one of the considerations for making 4K inodes the default in V4.2+ versions.
> 2) Let’s assume we have 3 filesystems, /home, /scratch (traditional
> HPC use for those two) and /data (project space). Our storage
> arrays are 24-bay units with two 8+2P RAID 6 LUNs, one RAID 1
> mirror, and two hot spare drives. The RAID 1 mirrors are for /home,
> the RAID 6 LUNs are for /scratch or /data. /home has tons of small
> files - so small that a 64K block size is currently used. /scratch
> and /data have a mixture, but a 1 MB block size is the “sweet spot” there.
>
> If you could “start all over” with the same hardware being the only
> restriction, would you:
>
> a) merge /scratch and /data into one filesystem but keep /home
> separate since the LUN sizes are so very different, or
> b) merge all three into one filesystem and use storage pools so that
> /home is just a separate pool within the one filesystem? And if you
> chose this option would you assign different block sizes to the pools?
It's not possible to have different block sizes for different data pools. We are very aware that many people would like to be able to do just that, but this is counter to where the product is going. Supporting different block sizes for different pools is actually pretty hard: it's tricky to describe a large file that has some blocks in poolA and some in poolB where poolB has a different block size (perhaps during a migration) with the existing inode/indirect block format where each disk address pointer addresses a block of fixed size. With some effort, and some changes to how block addressing works, it would be possible to implement the support for this. However, as I mentioned in another post in this thread, we don't really want to glorify manual block size selection any further, we want to move away from it, by addressing the reasons that drive different block size selection today (like disk space utilization and performance).
I'd recommend calculating a file size distribution histogram for your file systems. You may, for example, discover that 80% of the small files you have in /home would fit into 4K inodes, and then the storage space efficiency gains for the remaining 20% don't justify the complexity of managing an extra file system with a small block size. We don't recommend using block sizes smaller than 256K, because smaller block size is not good for disk space allocation code efficiency. It's a quadratic dependency: with a smaller block size, one block worth of the block allocation map covers that much less disk space, because each bit in the map covers fewer disk sectors, and fewer bits fit into a block. This means having to create a lot more block allocation map segments than what is needed for an ample level of parallelism. This hurts performance of many block allocation-related operations.
I don't see a reason for /scratch and /data to be separate file systems, aside from perhaps failure domain considerations.
yuri
> On Sep 26, 2016, at 2:29 PM, Yuri L Volobuev <volobuev at us.ibm.com<mailto:volobuev at us.ibm.com>> wrote:
>
> I would put the net summary this way: in GPFS, the "Goldilocks zone"
> for metadata block size is 256K - 1M. If one plans to create a new
> file system using GPFS V4.2+, 1M is a sound choice.
>
> In an ideal world, block size choice shouldn't really be a choice.
> It's a low-level implementation detail that one day should go the
> way of the manual ignition timing adjustment -- something that used
> to be necessary in the olden days, and something that select
> enthusiasts like to tweak to this day, but something that's
> irrelevant for the overwhelming majority of the folks who just want
> the engine to run. There's work being done in that general direction
> in GPFS, but we aren't there yet.
>
> yuri
>
> <graycol.gif>Stephen Ulmer ---09/26/2016 12:02:25 PM---Now I’ve got
> anther question… which I’ll let bake for a while. Okay, to (poorly) summarize:
>
> From: Stephen Ulmer <ulmer at ulmer.org<mailto:ulmer at ulmer.org>>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>,
> Date: 09/26/2016 12:02 PM
> Subject: Re: [gpfsug-discuss] Blocksize
> Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
>
>
>
> Now I’ve got anther question… which I’ll let bake for a while.
>
> Okay, to (poorly) summarize:
> There are items OTHER THAN INODES stored as metadata in GPFS.
> These items have a VARIETY OF SIZES, but are packed in such a way
> that we should just not worry about wasted space unless we pick a
> LARGE metadata block size — or if we don’t pick a “reasonable”
> metadata block size after picking a “large” file system block size
> that applies to both.
> Performance is hard, and the gain from calculating exactly the best
> metadata block size is much smaller than performance gains attained
> through code optimization.
> If we were to try and calculate the appropriate metadata block size
> we would likely be wrong anyway, since none of us get our data at
> the idealized physics shop that sells massless rulers and
> frictionless pulleys.
> We should probably all use a metadata block size around 1MB. Nobody
> has said this outright, but it’s been the example as the “good” size
> at least three times in this thread.
> Under no circumstances should we do what many of us would have done
> and pick 128K, which made sense based on all of our previous
> education that is no longer applicable.
>
> Did I miss anything? :)
>
> Liberty,
>
> --
> Stephen
>
> On Sep 26, 2016, at 2:18 PM, Yuri L Volobuev <volobuev at us.ibm.com<mailto:volobuev at us.ibm.com>> wrote:
> It's important to understand the differences between different
> metadata types, in particular where it comes to space allocation.
>
> System metadata files (inode file, inode and block allocation maps,
> ACL file, fileset metadata file, EA file in older versions) are
> allocated at well-defined moments (file system format, new storage
> pool creation in the case of block allocation map, etc), and those
> contain multiple records packed into a single block. From the block
> allocator point of view, the individual metadata record size is
> invisible, only larger blocks get actually allocated, and space
> usage efficiency generally isn't an issue.
>
> For user metadata (indirect blocks, directory blocks, EA overflow
> blocks) the situation is different. Those get allocated as the need
> arises, generally one at a time. So the size of an individual
> metadata structure matters, a lot. The smallest unit of allocation
> in GPFS is a subblock (1/32nd of a block). If an IB or a directory
> block is smaller than a subblock, the unused space in the subblock
> is wasted. So if one chooses to use, say, 16 MiB block size for
> metadata, the smallest unit of space that can be allocated is 512
> KiB. If one chooses 1 MiB block size, the smallest allocation unit
> is 32 KiB. IBs are generally 16 KiB or 32 KiB in size (32 KiB with
> any reasonable data block size); directory blocks used to be limited
> to 32 KiB, but in the current code can be as large as 256 KiB. As
> one can observe, using 16 MiB metadata block size would lead to a
> considerable amount of wasted space for IBs and large directories
> (small directories can live in inodes). On the other hand, with 1
> MiB block size, there'll be no wasted metadata space. Does any of
> this actually make a practical difference? That depends on the file
> system composition, namely the number of IBs (which is a function of
> the number of large files) and larger directories. Calculating this
> scientifically can be pretty involved, and really should be the job
> of a tool that ought to exist, but doesn't (yet). A more practical
> approach is doing a ballpark estimate using local file counts and
> typical fractions of large files and directories, using statistics
> available from published papers.
>
> The performance implications of a given metadata block size choice
> is a subject of nearly infinite depth, and this question ultimately
> can only be answered by doing experiments with a specific workload
> on specific hardware. The metadata space utilization efficiency is
> something that can be answered conclusively though.
>
> yuri
>
> <graycol.gif>"Buterbaugh, Kevin L" ---09/24/2016 07:19:09 AM---Hi
> Sven, I am confused by your statement that the metadata block size
> should be 1 MB and am very int
>
> From: "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at vanderbilt.edu>>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>,
> Date: 09/24/2016 07:19 AM
> Subject: Re: [gpfsug-discuss] Blocksize
> Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
>
>
>
>
> Hi Sven,
>
> I am confused by your statement that the metadata block size should
> be 1 MB and am very interested in learning the rationale behind this
> as I am currently looking at all aspects of our current GPFS
> configuration and the possibility of making major changes.
>
> If you have a filesystem with only metadataOnly disks in the system
> pool and the default size of an inode is 4K (which we would do,
> since we have recently discovered that even on our scratch
> filesystem we have a bazillion files that are 4K or smaller and
> could therefore have their data stored in the inode, right?), then
> why would you set the metadata block size to anything larger than
> 128K when a sub-block is 1/32nd of a block? I.e., with a 1 MB block
> size for metadata wouldn’t you be wasting a massive amount of space?
>
> What am I missing / confused about there?
>
> Oh, and here’s a related question … let’s just say I have the above
> configuration … my system pool is metadata only and is on SSD’s.
> Then I have two other dataOnly pools that are spinning disk. One is
> for “regular” access and the other is the “capacity” pool … i.e. a
> pool of slower storage where we move files with large access times.
> I have a policy that says something like “move all files with an
> access time > 6 months to the capacity pool.” Of those bazillion
> files less than 4K in size that are fitting in the inode currently,
> probably half a bazillion (<grin>) of them would be subject to that
> rule. Will they get moved to the spinning disk capacity pool or will
> they stay in the inode??
>
> Thanks! This is a very timely and interesting discussion for me as well...
>
> Kevin
> On Sep 23, 2016, at 4:35 PM, Sven Oehme <oehmes at us.ibm.com<mailto:oehmes at us.ibm.com>> wrote:
> your metadata block size these days should be 1 MB and there are
> only very few workloads for which you should run with a filesystem
> blocksize below 1 MB. so if you don't know exactly what to pick, 1
> MB is a good starting point.
> the general rule still applies that your filesystem blocksize
> (metadata or data pool) should match your raid controller (or GNR
> vdisk) stripe size of the particular pool.
>
> so if you use a 128k strip size(defaut in many midrange storage
> controllers) in a 8+2p raid array, your stripe or track size is 1 MB
> and therefore the blocksize of this pool should be 1 MB. i see many
> customers in the field using 1MB or even smaller blocksize on RAID
> stripes of 2 MB or above and your performance will be significant
> impacted by that.
>
> Sven
>
> ------------------------------------------
> Sven Oehme
> Scalable Storage Research
> email: oehmes at us.ibm.com<mailto:oehmes at us.ibm.com>
> Phone: +1 (408) 824-8904
> IBM Almaden Research Lab
> ------------------------------------------
>
> <graycol.gif>Stephen Ulmer ---09/23/2016 12:16:34 PM---Not to be too
> pedantic, but I believe the the subblock size is 1/32 of the block
> size (which strengt
>
> From: Stephen Ulmer <ulmer at ulmer.org<mailto:ulmer at ulmer.org>>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
> Date: 09/23/2016 12:16 PM
> Subject: Re: [gpfsug-discuss] Blocksize
> Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
>
>
>
>
> Not to be too pedantic, but I believe the the subblock size is 1/32
> of the block size (which strengthens Luis’s arguments below).
>
> I thought the the original question was NOT about inode size, but
> about metadata block size. You can specify that the system pool have
> a different block size from the rest of the filesystem, providing
> that it ONLY holds metadata (—metadata-block-size option to mmcrfs).
>
> So with 4K inodes (which should be used for all new filesystems
> without some counter-indication), I would think that we’d want to
> use a metadata block size of 4K*32=128K. This is independent of the
> regular block size, which you can calculate based on the workload if
> you’re lucky.
>
> There could be a great reason NOT to use 128K metadata block size,
> but I don’t know what it is. I’d be happy to be corrected about this
> if it’s out of whack.
>
> --
> Stephen
> On Sep 22, 2016, at 3:37 PM, Luis Bolinches <luis.bolinches at fi.ibm.com<mailto:luis.bolinches at fi.ibm.com>> wrote:
>
> Hi
>
> My 2 cents.
>
> Leave at least 4K inodes, then you get massive improvement on small
> files (less 3.5K minus whatever you use on xattr)
>
> About blocksize for data, unless you have actual data that suggest
> that you will actually benefit from smaller than 1MB block, leave
> there. GPFS uses sublocks where 1/16th of the BS can be allocated to
> different files, so the "waste" is much less than you think on 1MB
> and you get the throughput and less structures of much more data blocks.
>
> No warranty at all but I try to do this when the BS talk comes in:
> (might need some clean up it could not be last note but you get the idea)
>
> POSIX
> find . -type f -name '*' -exec ls -l {} \; > find_ls_files.out
> GPFS
> cd /usr/lpp/mmfs/samples/ilm
> gcc mmfindUtil_processOutputFile.c -o mmfindUtil_processOutputFile
> ./mmfind /gpfs/shared -ls -type f > find_ls_files.out
> CONVERT to CSV
>
> POSIX
> cat find_ls_files.out | awk '{print $5","}' > find_ls_files.out.csv
> GPFS
> cat find_ls_files.out | awk '{print $7","}' > find_ls_files.out.csv
> LOAD in octave
>
> FILESIZE = int32 (dlmread ("find_ls_files.out.csv", ","));
> Clean the second column (OPTIONAL as the next clean up will do the same)
>
> FILESIZE(:,[2]) = [];
> If we are on 4K aligment we need to clean the files that go to
> inodes (WELL not exactly ... extended attributes! so maybe use a
> lower number!)
>
> FILESIZE(FILESIZE<=3584) =[];
> If we are not we need to clean the 0 size files
>
> FILESIZE(FILESIZE==0) =[];
> Median
>
> FILESIZEMEDIAN = int32 (median (FILESIZE))
> Mean
>
> FILESIZEMEAN = int32 (mean (FILESIZE))
> Variance
>
> int32 (var (FILESIZE))
> iqr interquartile range, i.e., the difference between the upper and
> lower quartile, of the input data.
>
> int32 (iqr (FILESIZE))
> Standard deviation
>
>
> For some FS with lots of files you might need a rather powerful
> machine to run the calculations on octave, I never hit anything
> could not manage on a 64GB RAM Power box. Most of the times it is
> enough with my laptop.
>
>
>
> --
> Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations
>
> Luis Bolinches
> Lab Services
> http://www-03.ibm.com/systems/services/labservices/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D03.ibm.com_systems_services_labservices_&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=FZK2diT_8zDTwgFdqdGpVFmBDtR2_l2SXiOZ7ZtOJlw&e=>
>
> IBM Laajalahdentie 23 (main Entrance) Helsinki, 00330 Finland
> Phone: +358 503112585
>
> "If you continually give you will continually have." Anonymous
>
>
> ----- Original message -----
> From: Stef Coene <stef.coene at docum.org<mailto:stef.coene at docum.org>>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
> Cc:
> Subject: Re: [gpfsug-discuss] Blocksize
> Date: Thu, Sep 22, 2016 10:30 PM
>
> On 09/22/2016 09:07 PM, J. Eric Wonderley wrote:
> > It defaults to 4k:
> > mmlsfs testbs8M -i
> > flag value description
> > ------------------- ------------------------
> > -----------------------------------
> > -i 4096 Inode size in bytes
> >
> > I think you can make as small as 512b. Gpfs will store very small
> > files in the inode.
> >
> > Typically you want your average file size to be your blocksize and your
> > filesystem has one blocksize and one inodesize.
>
> The files are not small, but around 20 MB on average.
> So I calculated with IBM that a 1 MB or 2 MB block size is best.
>
> But I'm not sure if it's better to use a smaller block size for the
> metadata.
>
> The file system is not that large (400 TB) and will hold backup data
> from CommVault.
>
>
> Stef
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=1fmUrBFyMxUXUHqah3E7V_sMrg7zbWBghNQWwAJPc9c&e=>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=CEf2AMIDRUD_XeAA8cqtX6bFc7rnTTOgq7-mx5_dl44&e=>
>
>
>
> Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
> Oy IBM Finland Ab
> PL 265, 00101 Helsinki, Finland
> Business ID, Y-tunnus: 0195876-3
> Registered in Finland
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=1fmUrBFyMxUXUHqah3E7V_sMrg7zbWBghNQWwAJPc9c&e=>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=CEf2AMIDRUD_XeAA8cqtX6bFc7rnTTOgq7-mx5_dl44&e=>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=1fmUrBFyMxUXUHqah3E7V_sMrg7zbWBghNQWwAJPc9c&e=>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=CEf2AMIDRUD_XeAA8cqtX6bFc7rnTTOgq7-mx5_dl44&e=>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=1fmUrBFyMxUXUHqah3E7V_sMrg7zbWBghNQWwAJPc9c&e=>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=CEf2AMIDRUD_XeAA8cqtX6bFc7rnTTOgq7-mx5_dl44&e=>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=1fmUrBFyMxUXUHqah3E7V_sMrg7zbWBghNQWwAJPc9c&e=>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=CEf2AMIDRUD_XeAA8cqtX6bFc7rnTTOgq7-mx5_dl44&e=>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=1fmUrBFyMxUXUHqah3E7V_sMrg7zbWBghNQWwAJPc9c&e=>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=CEf2AMIDRUD_XeAA8cqtX6bFc7rnTTOgq7-mx5_dl44&e=>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=1fmUrBFyMxUXUHqah3E7V_sMrg7zbWBghNQWwAJPc9c&e=>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=CEf2AMIDRUD_XeAA8cqtX6bFc7rnTTOgq7-mx5_dl44&e=>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=1fmUrBFyMxUXUHqah3E7V_sMrg7zbWBghNQWwAJPc9c&e=>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=CEf2AMIDRUD_XeAA8cqtX6bFc7rnTTOgq7-mx5_dl44&e=>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=1fmUrBFyMxUXUHqah3E7V_sMrg7zbWBghNQWwAJPc9c&e=>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=CEf2AMIDRUD_XeAA8cqtX6bFc7rnTTOgq7-mx5_dl44&e=>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=1fmUrBFyMxUXUHqah3E7V_sMrg7zbWBghNQWwAJPc9c&e=>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMGaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Tpe4SZ_1FVcZKlLQ9tq9V7fIAidT34vMVKPHS97tiSc&s=CEf2AMIDRUD_XeAA8cqtX6bFc7rnTTOgq7-mx5_dl44&e=>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160928/eca0635b/attachment.htm>
More information about the gpfsug-discuss
mailing list