[gpfsug-discuss] *New* IBM Spectrum Protect Whitepaper "Petascale Data Protection"
Lukas Hejtmanek
xhejtman at ics.muni.cz
Wed Aug 31 19:39:26 BST 2016
On Tue, Aug 30, 2016 at 04:24:59PM -0700, Sven Oehme wrote:
> so lets start with some simple questions.
>
> when you say mmbackup takes ages, what version of gpfs code are you running
> ?
that was GPFS 3.5.0-8. The mmapplypolicy took over 2 hours but that was the
least problem. We developed our own set of backups scripts around mmbackup to
address these issues:
1) while mmbackup is running, you cannot run another instance on the same file
system.
2) mmbackup can be very slow, but not mmbackup itself but consecutive dsmc
selective, sorry for being misleading, but mainly due to the large number of
files to be backed up
3) related to the previous, mmbackup scripts seem to be executing a 'grep' cmd
for every input file to check whether it has entry in dmsc output log. well
guess what happens if you have millions of files at the input and several
gigabytes in dsmc outpu log... In our case, the grep storm took several
*weeks*.
4) very surprisingly, some of the files were not backed up at all. We cannot
find why but dsmc incremental found some old files that were not covered by
mmbackup backups. Maybe because the mmbackup process was not gracefully
terminated in some cases (node crash) and so on.
> how do you execute the mmbackup command ? exact parameters would be useful
> .
/usr/lpp/mmfs/bin/mmbackup tape_tape -t incremental -v -N fe1 -P ${POLICY_FILE}
--tsm-servers SERVER1 -g /gpfs/clusterbase/tmp/ -s /tmp -m 4 -B 9999999999999 -L 0
we had external exec script that split files from policy into chunks that were
run in parallel.
> what HW are you using for the metadata disks ?
4x SSD
> how much capacity (df -h) and how many inodes (df -i) do you have in the
> filesystem you try to backup ?
df -h
/dev/tape_tape 1.5P 745T 711T 52% /exports/tape_tape
df -hi
/dev/tape_tape 1.3G 98M 1.2G 8% /exports/tape_tape
(98M inodes used)
mmdf tape_tape
disk disk size failure holds holds free KB free KB
name in KB group metadata data in full blocks in fragments
--------------- ------------- -------- -------- ----- -------------------- -------------------
Disks in storage pool: system (Maximum disk size allowed is 175 TB)
nsd_t1_5 23437934592 1 No Yes 7342735360 ( 31%) 133872128 ( 1%)
nsd_t1_6 23437934592 1 No Yes 7341166592 ( 31%) 133918784 ( 1%)
nsd_t1b_2 23437934592 1 No Yes 7343919104 ( 31%) 134165056 ( 1%)
nsd_t1b_3 23437934592 1 No Yes 7341283328 ( 31%) 133986560 ( 1%)
nsd_ssd_4 770703360 2 Yes No 692172800 ( 90%) 15981952 ( 2%)
nsd_ssd_3 770703360 2 Yes No 692252672 ( 90%) 15921856 ( 2%)
nsd_ssd_2 770703360 2 Yes No 692189184 ( 90%) 15928832 ( 2%)
nsd_ssd_1 770703360 2 Yes No 692197376 ( 90%) 16013248 ( 2%)
------------- -------------------- -------------------
(pool total) 96834551808 32137916416 ( 33%) 599788416 ( 1%)
Disks in storage pool: maid (Maximum disk size allowed is 466 TB)
nsd8_t2_12 31249989632 1 No Yes 13167828992 ( 42%) 36282048 ( 0%)
nsd8_t2_13 31249989632 1 No Yes 13166729216 ( 42%) 36131072 ( 0%)
nsd8_t2_14 31249989632 1 No Yes 13166886912 ( 42%) 36371072 ( 0%)
nsd8_t2_15 31249989632 1 No Yes 13168209920 ( 42%) 36681728 ( 0%)
nsd8_t2_16 31249989632 1 No Yes 13165176832 ( 42%) 36279488 ( 0%)
nsd8_t2_17 31249989632 1 No Yes 13159870464 ( 42%) 36002560 ( 0%)
nsd8_t2_46 31249989632 1 No Yes 29624694784 ( 95%) 81600 ( 0%)
nsd8_t2_45 31249989632 1 No Yes 29623111680 ( 95%) 77184 ( 0%)
nsd8_t2_44 31249989632 1 No Yes 29621467136 ( 95%) 61440 ( 0%)
nsd8_t2_43 31249989632 1 No Yes 29622964224 ( 95%) 64640 ( 0%)
nsd8_t2_18 31249989632 1 No Yes 13166675968 ( 42%) 36147648 ( 0%)
nsd8_t2_19 31249989632 1 No Yes 13164529664 ( 42%) 36225216 ( 0%)
nsd8_t2_20 31249989632 1 No Yes 13165223936 ( 42%) 36242368 ( 0%)
nsd8_t2_21 31249989632 1 No Yes 13167353856 ( 42%) 36007744 ( 0%)
nsd8_t2_31 31249989632 1 No Yes 13116979200 ( 42%) 14155200 ( 0%)
nsd8_t2_32 31249989632 1 No Yes 13115633664 ( 42%) 14243840 ( 0%)
nsd8_t2_33 31249989632 1 No Yes 13115830272 ( 42%) 14235392 ( 0%)
nsd8_t2_34 31249989632 1 No Yes 13119727616 ( 42%) 14500608 ( 0%)
nsd8_t2_35 31249989632 1 No Yes 13116925952 ( 42%) 14304192 ( 0%)
nsd8_t2_0 31249989632 1 No Yes 13145503744 ( 42%) 99222016 ( 0%)
nsd8_t2_36 31249989632 1 No Yes 13119858688 ( 42%) 14054784 ( 0%)
nsd8_t2_37 31249989632 1 No Yes 13114101760 ( 42%) 14200704 ( 0%)
nsd8_t2_38 31249989632 1 No Yes 13116483584 ( 42%) 14174720 ( 0%)
nsd8_t2_39 31249989632 1 No Yes 13121257472 ( 42%) 14094720 ( 0%)
nsd8_t2_40 31249989632 1 No Yes 29622908928 ( 95%) 84352 ( 0%)
nsd8_t2_1 31249989632 1 No Yes 13146089472 ( 42%) 99566784 ( 0%)
nsd8_t2_2 31249989632 1 No Yes 13146208256 ( 42%) 99128960 ( 0%)
nsd8_t2_3 31249989632 1 No Yes 13146890240 ( 42%) 99766720 ( 0%)
nsd8_t2_4 31249989632 1 No Yes 13145143296 ( 42%) 98992576 ( 0%)
nsd8_t2_5 31249989632 1 No Yes 13135876096 ( 42%) 99555008 ( 0%)
nsd8_t2_6 31249989632 1 No Yes 13142831104 ( 42%) 99728064 ( 0%)
nsd8_t2_7 31249989632 1 No Yes 13140283392 ( 42%) 99412480 ( 0%)
nsd8_t2_8 31249989632 1 No Yes 13143470080 ( 42%) 99653696 ( 0%)
nsd8_t2_9 31249989632 1 No Yes 13143650304 ( 42%) 99224704 ( 0%)
nsd8_t2_10 31249989632 1 No Yes 13145440256 ( 42%) 99238528 ( 0%)
nsd8_t2_11 31249989632 1 No Yes 13143201792 ( 42%) 99283008 ( 0%)
nsd8_t2_22 31249989632 1 No Yes 13171724288 ( 42%) 36040704 ( 0%)
nsd8_t2_23 31249989632 1 No Yes 13166782464 ( 42%) 36212416 ( 0%)
nsd8_t2_24 31249989632 1 No Yes 13167990784 ( 42%) 35842368 ( 0%)
nsd8_t2_25 31249989632 1 No Yes 13166972928 ( 42%) 36086848 ( 0%)
nsd8_t2_26 31249989632 1 No Yes 13167495168 ( 42%) 36114496 ( 0%)
nsd8_t2_27 31249989632 1 No Yes 13164419072 ( 42%) 36119680 ( 0%)
nsd8_t2_28 31249989632 1 No Yes 13167804416 ( 42%) 36088832 ( 0%)
nsd8_t2_29 31249989632 1 No Yes 13166057472 ( 42%) 36107072 ( 0%)
nsd8_t2_30 31249989632 1 No Yes 13163673600 ( 42%) 36102528 ( 0%)
nsd8_t2_41 31249989632 1 No Yes 29620840448 ( 95%) 70208 ( 0%)
nsd8_t2_42 31249989632 1 No Yes 29621110784 ( 95%) 69568 ( 0%)
------------- -------------------- -------------------
(pool total) 1468749512704 733299890176 ( 50%) 2008331584 ( 0%)
============= ==================== ===================
(data) 1562501251072 762668994560 ( 49%) 2544274112 ( 0%)
(metadata) 3082813440 2768812032 ( 90%) 63845888 ( 2%)
============= ==================== ===================
(total) 1565584064512 765437806592 ( 49%) 2608120000 ( 0%)
Inode Information
-----------------
Number of used inodes: 102026081
Number of free inodes: 72791199
Number of allocated inodes: 174817280
Maximum number of inodes: 1342177280
--
Lukáš Hejtmánek
More information about the gpfsug-discuss
mailing list