[gpfsug-discuss] Follow-up: migrating billions of files
Venkateswara R Puvvada
vpuvvada at in.ibm.com
Thu Mar 7 13:52:12 GMT 2019
AFM based migration provides near-zero downtime and supports migrating
EAs/ACLs including immutability attributes (if home is Scale/ESS). I would
recommend starting migration in read-only mode, prefetch most of the data
and convert the fileset to local-updates (if backup is not needed during
the migration) or independent-writer mode before moving the applications
to the AFM cache filesets. AFM now supports (from 5.0.2) directory level
prefetch with many performance improvements and does not require
list-files to be specified.
~Venkat (vpuvvada at in.ibm.com)
From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: 03/06/2019 06:14 PM
Subject: [gpfsug-discuss] Follow-up: migrating billions of files
Sent by: gpfsug-discuss-bounces at spectrumscale.org
Some of you had questions to my original post. More information:
Source:
- Files are straight GPFS/Posix - no extended NFSV4 ACLs
- A solution that requires $’s to be spent on software (ie, Aspera) isn’t
a very viable option
- Both source and target clusters are in the same DC
- Source is stand-alone NSD servers (bonded 10g-E) and 8gb FC SAN storage
- Approx 40 file systems, a few large ones with 300M-400M files each,
others smaller
- no independent file sets
- migration must pose minimal disruption to existing users
Target architecture is a small number of file systems (2-3) on ESS with
independent filesets
- Target (ESS) will have multiple 40gb-E links on each NSD server (GS4)
My current thinking is AFM with a pre-populate of the file space and
switch the clients over to have them pull data they need (most of the data
is older and less active) and them let AFM populate the rest in the
background.
Bob Oesterlin
Sr Principal Storage Engineer, Nuance
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=YkRmc5bZTZ4O8u_y9PwCjhzuvVXZmhm-_SNQzKhDt0g&s=DUBqVmYz6ycQjkr-PZk4r5hndMIB1-FVzan1CCzlxRg&e=
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190307/b7503301/attachment.htm>
More information about the gpfsug-discuss
mailing list