[gpfsug-discuss] GPFS cpoying huge batches of files from one stgpool to another- how do you do it?

[Jez Tucker]

Here's a quick script to use GNU parallel:

We saturated 8 Gb FC with this.


#!/bin/sh

DATADIR=/mnt/gpfs/srcfldr
DESTDIR=/mnt/gpfs/destinationfldr

while read LINE; do

  PROJ=$(echo $LINE | awk '{ print $1; }');
  DESTFLDR=$(echo $LINE | awk '{ print $2; }');

  echo "$PROJ -> $DEST";
  mkdir -p "$DESTDIR/$DESTFLDR";

  find $PROJ/ | parallel  cp --parents -puv "{}" "$DESTDIR/$DESTFLDR/";
  rsync -av $PROJ "$DESTDIR/$DESTFLDR/" 2>&1 > RESTORE_LOGS/$PROJ.restore.log;

done < restore.my.projectlist


This assumes restore.nsd01.projectlist contains something such as:

<data folder> <destinationfldr subdir>
<data folder> <destinationfldr  subdir>
...



> -----Original Message-----
> From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-
> bounces at gpfsug.org] On Behalf Of Orlando Richards
> Sent: 20 March 2012 09:09
> To: gpfsug-discuss at gpfsug.org
> Subject: Re: [gpfsug-discuss] GPFS cpoying huge batches of files from one
> stgpool to another- how do you do it?
> 
> We don't do this often, so when we do we tend to do a "noddy"
> parallelisation of rsync - manually divvy the folders up and spawn
> multiple threads.
> 
> We might have a requirement soon(ish) to do this on a much larger scale
> though, with virtually no interruption to service - so I'm very keen to
> see what how the AFM solution looks, since this should allow us to
> present a continual and single view into the filesystem, whilst
> migrating it all in the background to the new filesystem, with just a
> brief wobble whilst we flip between the old and new views.
> 
> 
> On 20/03/12 00:47, Jez Tucker wrote:
> > In answer to my own question ...
> >
> > http://www.gnu.org/software/parallel
> >
> >
> http://www.gnu.org/software/parallel/man.html#example__parallelizing_r
> sync
> >
> > Or
> >
> > http://code.google.com/p/parallel-ssh/ (parallel versions of rsync etc).
> >
> > One for the bookmarks... hopefully you'll find it useful.
> >
> > Jez
> >
> > *From:*gpfsug-discuss-bounces at gpfsug.org
> > [mailto:gpfsug-discuss-bounces at gpfsug.org] *On Behalf Of *Jez Tucker
> > *Sent:* 19 March 2012 23:36
> > *To:* gpfsug main discussion list
> > *Subject:* [gpfsug-discuss] GPFS cpoying huge batches of files from one
> > stgpool to another- how do you do it?
> >
> > Hello
> >
> > Just wondering how other people go about copying loads of files in a
> > many, many deep directory path from one file system to another. Assume
> > filenames are full UNICODE and can contain almost any character.
> >
> > r to use.
> >
> >
> > Rushes Postproduction Limited, 66 Old Compton Street, London W1D 4UH
> > tel: +44 (0)20 7437 8676
> > web: http://www.rushes.co.uk
> > The information contained in this e-mail is confidential and may be
> > subject to legal privilege. If you are not the intended recipient, you
> > must not use, copy, distribute or disclose the e-mail or any part of its
> > contents or take any action in reliance on it. If you have received this
> > e-mail in error, please e-mail the sender by replying to this message.
> > All reasonable precautions have been taken to ensure no viruses are
> > present in this e-mail. Rushes Postproduction Limited cannot accept
> > responsibility for loss or damage arising from the use of this e-mail or
> > attachments and recommend that you subject these to your virus checking
> > procedures prior to use.
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at gpfsug.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> 
> --
>              --
>     Dr Orlando Richards
>    Information Services
> IT Infrastructure Division
>         Unix Section
>      Tel: 0131 650 4994
> 
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Rushes Postproduction Limited, 66 Old Compton Street, London W1D 4UH
tel: +44 (0)20 7437 8676
web: http://www.rushes.co.uk
The information contained in this e-mail is confidential and may be subject to legal privilege. If you are not the intended recipient, you must not use, copy, distribute or disclose the e-mail or any part of its contents or take any action in reliance on it. If you have received this e-mail in error, please e-mail the sender by replying to this message. All reasonable precautions have been taken to ensure no viruses are present in this e-mail. Rushes Postproduction Limited cannot accept responsibility for loss or damage arising from the use of this e-mail or attachments and recommend that you subject these to your virus checking procedures prior to use.