[gpfsug-discuss] mmbackup logging issue

[Ashish Thandavan]

Dear all,

We have a small GPFS cluster and a separate server running TSM and one 
of the three NSD servers backs up our GPFS filesystem to the TSM server 
using mmbackup. After a recent upgrade from v3.5 to 4.1.1, we've noticed 
that mmbackup no longer logs stuff like it used to :

...
Thu Jan 19 05:45:41 2017 mmbackup:Backing up files: 0 backed up, 870532 
expired, 2 failed.
Thu Jan 19 06:15:41 2017 mmbackup:Backing up files: 0 backed up, 870532 
expired, 3 failed.
Thu Jan 19 06:45:41 2017 mmbackup:Backing up files: 0 backed up, 870532 
expired, 3 failed.
...


instead of

...
Sat Dec  3 12:01:00 2016 mmbackup:Backing up files: 105030 backed up, 
635456 expired, 30 failed.
Sat Dec  3 12:31:00 2016 mmbackup:Backing up files: 205934 backed up, 
635456 expired, 57 failed.
Sat Dec  3 13:01:00 2016 mmbackup:Backing up files: 321702 backed up, 
635456 expired, 169 failed.
...

like it used to pre-upgrade.

I am therefore unable to see how far long it has got, and indeed if it 
completed successfully, as this is what it logs at the end of a job :

...
Tue Jan 17 18:07:31 2017 mmbackup:Completed policy backup run with 0 
policy errors, 10012 files failed, 0 severe errors, returning rc=9.
Tue Jan 17 18:07:31 2017 mmbackup:Policy for backup returned 9 Highest 
TSM error 12
mmbackup: TSM Summary Information:
     Total number of objects inspected:     20617273
     Total number of objects backed up:     0
     Total number of objects updated:     0
     Total number of objects rebound:     0
     Total number of objects deleted:     0
     Total number of objects expired:     1
     Total number of objects failed:     10012
     Total number of objects encrypted:     0
     Total number of bytes inspected:     3821624716861
     Total number of bytes transferred:     3712040943672
Tue Jan 17 18:07:31 2017 mmbackup:Audit files /cs/mmbackup.audit.gpfs* 
contain 0 failed paths but there were 10012 failures.
Cannot reconcile shadow database.
Unable to compensate for all TSM errors in new shadow database.
   Preserving previous shadow database.
   Run next mmbackup with -q to synchronize shadow database.  exit 12

If it helps, the mmbackup job is kicked off with the following options :
  /usr/lpp/mmfs/bin/mmbackup gpfs -n 8 -t full -B 20000 -L 1 
--tsm-servers gpfs_weekly_stanza -N glossop1a | /usr/bin/tee 
/var/log/mmbackup/gpfs_weekly/backup_log.`date +%Y%m%d_%H_%M`

(The excerpts above are from the backup_log.<datestamp> file.)

Our NSD servers are running GPFS 4.1.1-11, TSM is at 7.1.1.100 and the 
File system version is 12.06 (3.4.0.3). Has anyone else seen this 
behaviour with mmbackup and if so, found a fix?

Thanks,

Regards,
Ash

-- 
-------------------------
Ashish Thandavan

UNIX Support Computing Officer
Department of Computer Science
University of Oxford
Wolfson Building
Parks Road
Oxford OX1 3QD

Phone: 01865 610733
Email: ashish.thandavan at cs.ox.ac.uk