[gpfsug-discuss] GPFS, LTFS/EE and data-in-inode?

[valdis.kletnieks at vt.edu]

On Tue, 25 Jul 2017 10:02:14 +0100, Jonathan Buzzard said:

> I would be tempted to zip up the directories and move them ziped ;-)

Not an option, unless you want to come here and re-write the researcher's
tracking systems that knows where they archived a given run, and teach it
"Except now it's in a .tar.gz in that directory, or perhaps one or two
directories higher up, under some name".  Yes, researchers do that.  And
as the joke goes:  "What's the difference between a tenured professor and
a terrorist?" "You can negotiate with a terrorist..."

Plus remember that most of these directories are currently scattered across
multiple tapes, which means "zip up a directory" may mean reading as many as 10
to 20 tapes just to get the directory on disk so you can zip it up.  As it is,
I had to write code that recall and processes all the files on tape 1,
*wherever they are in the file system*, free them from the source disk, recall
and process all the files on tape 2, repeat until tape 3,857. (And due to
funding issues 5 years ago which turned into a "who paid for what tapes" food
fight, most of the tapes ended up with files from entirely different file
systems on them, going into different filesets on the destination).

(And in fact, the migration is currently hosed up because a researcher *is*
doing pretty much that - recalling all the files from one directory, then
the next, then the next, to get files they need urgently for a deliverable
but haven't been moved to the new system.  So rather than having 12 LTO-5
drives to multistream the tape recalls, I've got 12 recalls fighting for one
drive while the researcher's processing is hogging the other 11, due to the
way the previous system prioritizes in-line opens of files versus bulk recalls)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170725/a95d7f73/attachment.sig>