[gpfsug-discuss] GPFS snapshot cron job

bergman at panix.com bergman at panix.com
Wed Feb 6 21:28:30 GMT 2013



In the message dated: Wed, 06 Feb 2013 13:38:56 -0500,
The pithy ruminations from Stuart Barkley on 
<[gpfsug-discuss] GPFS snapshot cron job> were:
=> I'm new on this list.  It looks like it can be useful for exchanging
=> GPFS experiences.
=> 
=> We have been running GPFS for a couple of years now on one cluster and
=> are in process of bringing it up on a couple of other clusters.
=> 
=> One thing we would like, but have not had time to do is automatic
=> snapshots similar to what NetApp does.  For our purposes a cron job
=> that ran every 4 hours that creates a new snapshot and removes older
=> snapshots would be sufficient.  The slightly hard task is correctly
=> removing the older snapshots.
=> 
=> Does anyone have such a cron script they can share?

Yes.

I've attached the script that we run from cron.

Our goal was to keep a decaying set of snapshots over a fairly long
time period, so that users would be able to recover from "rm", while
not using excessive space.

Snapshots are named with timestamp, making it slightly easier to understand
what data they contain and the remove the older ones.

The cron job runs every 15 minutes on every GPFS server node, but checks
if it is executing on the node that is the manager for the specified
filesystem to avoid concurrency issues.

The script will avoid making a snapshot if there isn't sufficient disk space.

Our config file to manage snapshots is:
------------ CUT HERE -- CUT HERE --------------
case $1 in
        home)
                intervals=(1 4 24 48)   # hour number of each interval
                counts=(4 4 4 4)        # max number of snapshots to keep per each interval
                MINFREE=5       # minimum free disk space, in percent
                ;;

        shared)
                intervals=(1 4 48)      # hour number of each interval
                counts=(4 2 2)  # max number of snapshots to keep per each interval
                MINFREE=20      # minimum free disk space, in percent
                ;;
esac
------------ CUT HERE -- CUT HERE --------------

For the "home" filesystem, this says:

	keep 4 snapshots in the most recent hourly interval (every 15 minutes)
	keep 4 snapshots made in the most recent 4 hr interval (1 for each hour)
	keep 4 snapshots made in the most recent 24 hr interval (1 each 6hrs)
	keep 4 snapshots made in the most recent 48 hr interval (1 each 12 hrs)

For the "shared" filesystem, the configuration says:
	keep 4 snapshots in the most recent hourly interval (every 15 minutes)
	keep 2 snapshots made in the most recent 4 hr interval (1 each 2 hours)
	keep 2 snapshots made in the most recent 48 hr interval (1 each 24 hrs)

Those intervals "overlap", so there are a lot of recent snapshots, and fewer
older ones. Each time a snapshot is made, older snapshots may be removed.

So, at 5:01 PM on Thursday, there may be snapshots of the "home" filesystem from:

	17:00 Thursday   ---+-- 4 in the last hour
	16:45 Thursday      |
	16:30 Thursday      |
	16:15 Thursday   -- +

	16:00 Thursday   ---+-- 4 in the last 4 hours, including
	15:00 Thursday      |   the 5:00PM Thursday snapshot
	14:00 Thursday   ---+

	11:00 Thursday   ---+-- 4 in the last 24 hours, including
	05:00 Thursday      |	17:00 Thursday
	23:00 Wednesday  ---+

	17:00 Wednesday  ---+-- 4 @ 12-hr intervals in the last 48 hours,
	05:00 Wednesday  ---+   including 17:00 & 05:00 Thursday


Suggestions and patches are welcome.

=> 
=> Or did I miss something in GPFS that handles automatic snapshots?

We have seen periodic slowdowns when snapshots are running, but nothing to the
extent described by Jonathan Buzzard.

Mark

=> 
=> Thanks,
=> Stuart Barkley
=> -- 
=> I've never been lost; I was once bewildered for three days, but never lost!
=>                                         --  Daniel Boone
-------------- next part --------------
#! /bin/bash
#$Id: snapshotter 858 2012-01-31 19:24:11Z$
# Manage snapshots of GPFS volumes
#
# Desgined to be called from cron at :15 intervals

# 

##################################################################
# Defaults, may be overridden by /usr/local/etc/snappshotter.conf
# or file specified by "-c"
CONF=/usr/local/etc/snappshotter.conf	# config file, supersceded by "-c" option
MINFREE=10	# minimum free space, in percent.
#
# Series of intervals and counts. Intervals expressed as the end-point in hours.
# count = number of snapshots to keep per-interval
##############
#  time
#  ====
#  :00-59	keep snapshots at 15 minute intervals; ceiling of interval = 1hr
#  01-03:59	keep snapshots at 1hr interval; ceiling of interval = 4hr
#  04-23:59	keep snapshots at 6hr intervals; ceiling of interval = 24hr
#  24-47:59	keep snapshots at 12hr intervals; ceiling of interval = 48hr
intervals=(1 4 24 48)	# hour number of each interval
counts=(4 4 4 4)	# max number of snapshots to keep per each interval 
# Note that the snapshots in interval (N+1) must be on a time interval
# that corresponds to the snapshots kept in interval N. 
#
#  :00-59	keep snapshots divisible by 1/4hr:	 00:00, 00:15, 00:30, 00:45, 01:00, 01:15 ...
#  01-04:59	keep snapshots divisible by 4/4hr:	 00:00, 01:00, 02:00, 03:00 ...
#  05-23:59	keep snapshots divisible by 24/4hr:	 00:00, 06:00, 12:00, 18:00
#  24-48:59	keep snapshots divisible by 48/4hr:  00:00, 12:00
#
#
##################################################################

TESTING="no"

MMDF=/usr/lpp/mmfs/bin/mmdf
MMCRSNAPSHOT=/usr/lpp/mmfs/bin/mmcrsnapshot
MMLSSNAPSHOT=/usr/lpp/mmfs/bin/mmlssnapshot
MMDELSNAPSHOT=/usr/lpp/mmfs/bin/mmdelsnapshot
LOGGER="logger -p user.alert -t snapshotter"
PATH="${PATH}:/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/usr/local/sbin" # for access to 'ip' command, GPFS commands

now=`date '+%Y_%m_%d_%H:%M'`
nowsecs=`echo $now | sed -e "s/_\([^_]*\)$/ \1/" -e "s/_/\//g"`
nowsecs=`date --date "$nowsecs" "+%s"`

secsINhr=$((60 * 60))

#####################################################################
usage()
{
	cat - << E-O-USAGE 1>&2
$0 -- manage GPFS snapshots
    
Create new GPFS snapshots and remove old snapshots.

Options:

    -f filesystem       required -- name of filesystem to snapshot

    -t testing          test mode, report what would
                        be done but perform no action

    -d "datestamp"      test mode only; used supplied
                        date stamp as if it was the
                        current time.

    -c configfile       use supplied configuration file
                        in place of default:
                            $CONF

     -L					show license statement

    In test mode, the input data, in the same format as
    produced by "mmlssnap" must be supplied. This can be
    done on STDIN, as:
      $0 -t -f home -d "\`date --date "Dec 7 23:45"\`" < mmlssnap.data 
    or
      $0 -t -f home -d "\`date --date "now +4hours"\`" < mmlssnap.data 


E-O-USAGE
	echo 1>&2
	echo $1 1>&2
	exit 1
}
#####################################################################

license()
{
	cat - << E-O-LICENSE
  Section of Biomedical Image Analysis
  Department of Radiology
  University of Pennsylvania
  3600 Market Street, Suite 380
  Philadelphia, PA 19104

  Web:   http://www.rad.upenn.edu/sbia/
  Email: sbia-software at uphs.upenn.edu



SBIA Contribution and Software License Agreement ("Agreement")
==============================================================

  Version 1.0 (June 9, 2011)

  This Agreement covers contributions to and downloads from Software maintained by
  the Section of Biomedical Image Analysis, Department of Radiology at the University
  of Pennsylvania ("SBIA"). Part A of this Agreement applies to contributions of
  software and/or data to the Software (including making revisions of or additions
  to code and/or data already in this Software). Part B of this Agreement applies to
  downloads of software and/or data from SBIA. Part C of this Agreement applies to
  all transactions with SBIA. If you distribute Software (as defined below) downloaded
  from SBIA, all of the paragraphs of Part B of this Agreement must be included with
  and apply to such Software.

  Your contribution of software and/or data to SBIA (including prior to the date
  of the first publication of this Agreement, each a "Contribution") and/or
  downloading, copying, modifying, displaying, distributing or use of any software
  and/or data from SBIA (collectively, the "Software") constitutes acceptance of
  all of the terms and conditions of this Agreement. If you do not agree to such
  terms and conditions, you have no right to contribute your Contribution, or to
  download, copy, modify, display, distribute or use the Software.


  PART A. CONTRIBUTION AGREEMENT - LICENSE TO SBIA WITH RIGHT TO SUBLICENSE ("CONTRIBUTION AGREEMENT").
  -----------------------------------------------------------------------------------------------------

  1. As used in this Contribution Agreement, "you" means the individual contributing
     the Contribution to the Software maintained by SBIA and the institution or
     entity which employs or is otherwise affiliated with such individual in
     connection with such Contribution.

  2. This Contribution Agreement applies to all Contributions made to the Software
     maintained by SBIA, including without limitation Contributions made prior to
     the date of first publication of this Agreement. If at any time you make a
     Contribution to the Software, you represent that (i) you are legally authorized
     and entitled to make such Contribution and to grant all licenses granted in this
     Contribution Agreement with respect to such Contribution; (ii) if your
     Contribution includes any patient data, all such data is de-identified in
     accordance with U.S. confidentiality and security laws and requirements,
     including but not limited to the Health Insurance Portability and Accountability
     Act (HIPAA) and its regulations, and your disclosure of such data for the purposes
     contemplated by this Agreement is properly authorized and in compliance with all
     applicable laws and regulations; and (iii) you have preserved in the Contribution
     all applicable attributions, copyright notices and licenses for any third party
     software or data included in the Contribution.

  3. Except for the licenses granted in this Agreement, you reserve all right,
     title and interest in your Contribution.

  4. You hereby grant to SBIA, with the right to sublicense, a perpetual, worldwide,
     non-exclusive, no charge, royalty-free, irrevocable license to use, reproduce,
     make derivative works of, display and distribute the Contribution. If your
     Contribution is protected by patent, you hereby grant to SBIA, with the right
     to sublicense, a perpetual, worldwide, non-exclusive, no-charge, royalty-free,
     irrevocable license under your interest in patent rights covering the Contribution,
     to make, have made, use, sell and otherwise transfer your Contribution, alone
     or in combination with any other code.

  5. You acknowledge and agree that SBIA may incorporate your Contribution into
     the Software and may make the Software available to members of the public
     on an open source basis under terms substantially in accordance with the
     Software License set forth in Part B of this Agreement. You further acknowledge
     and agree that SBIA shall have no liability arising in connection with claims
     resulting from your breach of any of the terms of this Agreement.

  6. YOU WARRANT THAT TO THE BEST OF YOUR KNOWLEDGE YOUR CONTRIBUTION DOES NOT
     CONTAIN ANY CODE THAT REQUIRES OR PRESCRIBES AN "OPEN SOURCE LICENSE" FOR
     DERIVATIVE WORKS (by way of non-limiting example, the GNU General Public
     License or other so-called "reciprocal" license that requires any derived
     work to be licensed under the GNU General Public License or other
     "open source license").


  PART B. DOWNLOADING AGREEMENT - LICENSE FROM SBIA WITH RIGHT TO SUBLICENSE ("SOFTWARE LICENSE").
  ------------------------------------------------------------------------------------------------

  1. As used in this Software License, "you" means the individual downloading and/or
     using, reproducing, modifying, displaying and/or distributing the Software and
     the institution or entity which employs or is otherwise affiliated with such
     individual in connection therewith. The Section of Biomedical Image Analysis,
     Department of Radiology at the Universiy of Pennsylvania ("SBIA") hereby grants
     you, with right to sublicense, with respect to SBIA's rights in the software,
     and data, if any, which is the subject of this Software License (collectively,
     the "Software"), a royalty-free, non-exclusive license to use, reproduce, make
     derivative works of, display and distribute the Software, provided that:
     (a) you accept and adhere to all of the terms and conditions of this Software
     License; (b) in connection with any copy of or sublicense of all or any portion
     of the Software, all of the terms and conditions in this Software License shall
     appear in and shall apply to such copy and such sublicense, including without
     limitation all source and executable forms and on any user documentation,
     prefaced with the following words: "All or portions of this licensed product
     (such portions are the "Software") have been obtained under license from the
     Section of Biomedical Image Analysis, Department of Radiology at the University
     of Pennsylvania and are subject to the following terms and conditions:"
     (c) you preserve and maintain all applicable attributions, copyright notices
     and licenses included in or applicable to the Software; (d) modified versions
     of the Software must be clearly identified and marked as such, and must not
     be misrepresented as being the original Software; and (e) you consider making,
     but are under no obligation to make, the source code of any of your modifications
     to the Software freely available to others on an open source basis.

  2. The license granted in this Software License includes without limitation the
     right to (i) incorporate the Software into proprietary programs (subject to
     any restrictions applicable to such programs), (ii) add your own copyright
     statement to your modifications of the Software, and (iii) provide additional
     or different license terms and conditions in your sublicenses of modifications
     of the Software; provided that in each case your use, reproduction or
     distribution of such modifications otherwise complies with the conditions
     stated in this Software License.

  3. This Software License does not grant any rights with respect to third party
     software, except those rights that SBIA has been authorized by a third
     party to grant to you, and accordingly you are solely responsible for
     (i) obtaining any permissions from third parties that you need to use,
     reproduce, make derivative works of, display and distribute the Software,
     and (ii) informing your sublicensees, including without limitation your
     end-users, of their obligations to secure any such required permissions.

  4. The Software has been designed for research purposes only and has not been
     reviewed or approved by the Food and Drug Administration or by any other
     agency. YOU ACKNOWLEDGE AND AGREE THAT CLINICAL APPLICATIONS ARE NEITHER
     RECOMMENDED NOR ADVISED. Any commercialization of the Software is at the
     sole risk of the party or parties engaged in such commercialization.
     You further agree to use, reproduce, make derivative works of, display
     and distribute the Software in compliance with all applicable governmental
     laws, regulations and orders, including without limitation those relating
     to export and import control.

  5. The Software is provided "AS IS" and neither SBIA nor any contributor to
     the software (each a "Contributor") shall have any obligation to provide
     maintenance, support, updates, enhancements or modifications thereto.
     SBIA AND ALL CONTRIBUTORS SPECIFICALLY DISCLAIM ALL EXPRESS AND IMPLIED
     WARRANTIES OF ANY KIND INCLUDING, BUT NOT LIMITED TO, ANY WARRANTIES OF
     MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT.
     IN NO EVENT SHALL SBIA OR ANY CONTRIBUTOR BE LIABLE TO ANY PARTY FOR
     DIRECT, INDIRECT, SPECIAL, INCIDENTAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES
     HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY ARISING IN ANY WAY RELATED
     TO THE SOFTWARE, EVEN IF SBIA OR ANY CONTRIBUTOR HAS BEEN ADVISED OF THE
     POSSIBILITY OF SUCH DAMAGES. TO THE MAXIMUM EXTENT NOT PROHIBITED BY LAW OR
     REGULATION, YOU FURTHER ASSUME ALL LIABILITY FOR YOUR USE, REPRODUCTION,
     MAKING OF DERIVATIVE WORKS, DISPLAY, LICENSE OR DISTRIBUTION OF THE SOFTWARE
     AND AGREE TO INDEMNIFY AND HOLD HARMLESS SBIA AND ALL CONTRIBUTORS FROM
     AND AGAINST ANY AND ALL CLAIMS, SUITS, ACTIONS, DEMANDS AND JUDGMENTS ARISING
     THEREFROM.

  6. None of the names, logos or trademarks of SBIA or any of SBIA's affiliates
     or any of the Contributors, or any funding agency, may be used to endorse
     or promote products produced in whole or in part by operation of the Software
     or derived from or based on the Software without specific prior written
     permission from the applicable party.

  7. Any use, reproduction or distribution of the Software which is not in accordance
     with this Software License shall automatically revoke all rights granted to you
     under this Software License and render Paragraphs 1 and 2 of this Software
     License null and void.

  8. This Software License does not grant any rights in or to any intellectual
     property owned by SBIA or any Contributor except those rights expressly
     granted hereunder.


  PART C. MISCELLANEOUS
  ---------------------

  This Agreement shall be governed by and construed in accordance with the laws
  of The Commonwealth of Pennsylvania without regard to principles of conflicts
  of law. This Agreement shall supercede and replace any license terms that you
  may have agreed to previously with respect to Software from SBIA.
E-O-LICENSE
	exit
}

#####################################################################
# Parse the command-line
while [ "X$1" != "X" ] 
do
	case $1 in
		-L)
			license
			;;

		-t)
			TESTING="yes"
			shift
			;;

		-d)
			# Date stamp given...only valid in testing mode
			shift

			# Convert the user-supplied date to the YYYY_Mo_DD_HH:MM form,
			# throwing away the seconds
			UserDATE="$1"
			now=`date --date "$1" '+%Y_%m_%d_%H:%M'`
			nowsecs=`echo $now | sed -e "s/_\([^_]*\)$/ \1/" -e "s/_/\//g"`
			nowsecs=`date --date "$nowsecs" "+%s"`
			shift
			;;

		-c)
			shift
			CONF=$1
			if [ ! -f $CONF ] ; then
				usage "Specified configuration file ($CONF) not found"
			fi
			shift
			;;

		-f)
			shift
			filesys=$1
			shift
			;;

		*)
			usage "Unrecognized option: \"$1\""
			;;
	esac
done

############## End of command line parsing

LOCKFILE=/var/run/snapshotter.$filesys
if [ -f $LOCKFILE ] ; then
	PIDs=`cat $LOCKFILE | tr "\012" " "`
	echo "Lockfile $LOCKFILE from snapshotter process $PID exists. Will not continue." 1>&2
	$LOGGER "Lockfile $LOCKFILE from snapshotter process $PID exists. Will not continue."
	exit 1
else
	echo $$ > $LOCKFILE
	if [ $? != 0 ] ; then
		echo "Could not create lockfile $LOCKFILE for process $$. Exiting." 1>&2
		$LOGGER "Could not create lockfile $LOCKFILE for process $$"
		exit 2
	fi
fi


######## Check sanity of user-supplied values
if [ "X$filesys" = "X" ] ; then
	$LOGGER "Filesystem must be specified"
	usage "Filesystem must be specified" 
fi

if [ $TESTING = "yes" ] ; then
	# testing mode:
	#   accept faux filesystem argument
	#	accept faux datestamp as arguments
	#	read faux results from mmlssnapshot on STDIN

	# MMDF
	#
	# Do not really use mmdf executable, so that the testing can be
	# done outside a GPFS cluster Use a 2-digit random number 00 .. 99
	# from $RANDOM, but fill the variable with dummy fields so the
	# the random number corresponds to field5, where it would be in
	# the mmdf output.
	MMDF="eval echo \(total\) f1 f2 f3 f4  \(\${RANDOM: -2:2}%\) "

	MMCRSNAPSHOT="echo mmcrsnapshot"
	MMDELSNAPSHOT="echo mmdelsnapshot"
	MMLSSNAPDATA=`cat - | tr "\012" "%"`
	MMLSSNAPSHOT="eval echo \$MMLSSNAPDATA|tr '%' '\012'"
	LOGGER="echo Log message: "
else
	if [ "X$UserDATE" != "X" ] ; then
		$LOGGER "Option \"-d\" only valid in testing mode"
		usage "Option \"-d\" only valid in testing mode"
	fi

	/usr/lpp/mmfs/bin/mmlsfs $filesys -T 1> /dev/null 2>&1
	if [ $? != 0 ] ; then
		$LOGGER "Error accessing GPFS filesystem: $filesys"
		echo "Error accessing GPFS filesystem: $filesys" 1>&2
		rm -f $LOCKFILE
		exit 1
	fi

	# Check if the node where this script is running is the GPFS manager node for the
	# specified filesystem
	manager=`/usr/lpp/mmfs/bin/mmlsmgr $filesys | grep -w "^$filesys" |awk '{print $2}'`
	ip addr list | grep -qw "$manager"
	if [ $? != 0 ] ; then
		# This node is not the manager...exit
		rm -f $LOCKFILE
		exit
	fi
	MMLSSNAPSHOT="$MMLSSNAPSHOT $filesys"
fi

# It is valid for the default config file not to exist, so check if
# is there before sourcing it
if [ -f $CONF ] ; then
	. $CONF	$filesys # load variables found in $CONF, based on $filesys
fi

# Get current free space 
freenow=`$MMDF $filesys|grep '(total)' | sed -e "s/%.*//" -e "s/.*( *//"`

# Produce list of valid snapshot names (w/o header lines)
snapnames=`$MMLSSNAPSHOT |grep Valid |sed -e '$d' -e 's/ .*//'`
# get the number of existing snapshots
snapcount=($snapnames) ; snapcount=${#snapcount[*]}


###########################################################
# given a list of old snapshot names, in the form:
#	YYYY_Mo_DD_HH:MM
# fill the buckets by time. A snapshot can only go
# into one bucket!
###########################################################
for oldsnap in $snapnames
do
	oldstamp=`echo $oldsnap|sed -e "s/_\([^_]*\)$/ \1/" -e "s/_/\//g"`
	oldsecs=`date --date "$oldstamp" "+%s"`

	diff=$((nowsecs - oldsecs)) # difference in seconds between 'now' and old snapshot
	if [ $diff -lt 0 ] ; then
		# this can happen during testing...we have got a faux
		# snapshot date in the future...skip it
		continue
	fi
	
	index=0
	prevbucket=0
	filled=No

	while [ $index -lt ${#intervals[*]}  -a $filled != "Yes" ]
	do
		bucket=${intervals[$index]} # ceiling for number of hours for this bucket (1 more than the number of 
								# actual hours, ie., "7" means that the bucket can contain snapshots that are
								# at least 6:59 (hh:mm) old.
		count=${counts[$index]}		# max number of items in this bucket
		bucketinterval=$(( bucket * ( secsINhr / count ) ))	# Number of hours (in seconds) between snapshots that should be retained
					# for this bucket...convert from hrs (bucket/count) to seconds in order to deal with :15 minute intervals
					# Force the mathematical precedence to do (secsINhr / count) so that cases where count>bucket (like the first 1hr
					# that may have a count of 4 retained snapshots) doesn't result in the shell throwing away the fraction

		if [  $diff -ge $((prevbucket * secsINhr)) -a  $diff -lt $((bucket * secsINhr)) ] ; then
			# We found the correct bucket
			filled=Yes

			## printf "Checking if $oldsnap should be retained if it is multiple of $bucketinterval [ ($oldsecs %% $bucketinterval) = 0]"

			# Does the snapshot being examined fall on the interval determined above for the snapshots that should be retained?
			if [ $(( oldsecs  % bucketinterval )) = 0 ] ; then
				# The hour of the old snapshot is evenly divisible by the number of snapshots that should be
				# retained in this interval...keep it
				tokeep="$tokeep $oldsnap"
				## printf "...yes\n"
			else
				todelete="$todelete $oldsnap"
				## printf "...no\n"
			fi
			prevbucket=$bucket
		fi
		index=$((index + 1))
	done

	if [ $diff -ge $((bucket * secsINhr )) ] ; then
		filled=Yes
		# This is too old...schedule it for deletion
		$LOGGER "Scheduling old snapshot $oldsnap from $filesys for deletion"
		todelete="$todelete $oldsnap"
	fi

	# We should not get here
	if [ $filled != Yes ] ; then
		$LOGGER "Snapshot \"$oldsnap\" on $filesys does not match any intervals"
	fi
done

# Sort the lists to make reading the testing output easier
todelete=`echo $todelete | tr " " "\012" | sort -bdfu`
tokeep=`echo $tokeep | tr " " "\012" | sort -bdfu`

#############################################################
for oldsnap in $todelete
do
	if [ $TESTING = "yes" ] ; then
		# "run" $MMDELSNAPSHOT without capturing results in order to produce STDOUT in testing mode
		$MMDELSNAPSHOT $filesys $oldsnap
		# remove the entry for the snapshot scheduled for deletion
		# from MMLSSNAPDATA so that the next call to MMLSSNAPSHOT is accurate
		## echo "Removing entry for \"$oldsnap\" from \$MMLSSNAPDATA"
		MMLSSNAPDATA=`echo $MMLSSNAPDATA | sed -e "s/%$oldsnap [^%]*%/%/"`
	else
		# Run mmdelsnapshot, and capture the output to prevent verbose messages from being
		# sent as the result of each cron job. Only display the messages in case of error.
		output=`$MMDELSNAPSHOT $filesys $oldsnap 2>&1`
	fi
	if [ $? != 0 ] ; then
		printf "Error from \"$MMDELSNAPSHOT $filesys $oldsnap\": $output" 1>&2
		$LOGGER "Error removing snapshot of $filesys with label \"$oldsnap\": $output"
		rm -f $LOCKFILE
		exit 1
	else
		$LOGGER "successfully removed snapshot of $filesys with label \"$oldsnap\""
	fi
done

############# Now check for free space #######################################
# Get current free space 
freenow=`$MMDF $filesys|grep '(total)' | sed -e "s/%.*//" -e "s/.*( *//"`
# get the number of existing snapshots
snapcount=`$MMLSSNAPSHOT |grep Valid |wc -l`

while [ $freenow -le $MINFREE -a $snapcount -gt 0 ]
do
	# must delete some snapshots, from the oldest first...
	todelete=`$MMLSSNAPSHOT|grep Valid |sed -n -e 's/ .*//' -e '1p'`

	if [ $TESTING = "yes" ] ; then
		# "run" $MMDELSNAPSHOT without capturing results in order to produce STDOUT in testing mode
		$MMDELSNAPSHOT $filesys $todelete
		# remove the entry for the snapshot scheduled for deletion
		# from MMLSSNAPDATA so that the next call to MMLSSNAPSHOT is accurate and from $tokeep
		## echo "Removing entry for \"$todelete\" from \$MMLSSNAPDATA"
		MMLSSNAPDATA=`echo $MMLSSNAPDATA | sed -e "s/%$todelete [^%]*%/%/"`
		tokeep=`echo $tokeep | sed -e "s/^$todelete //" -e "s/ $todelete / /" -e "s/ $todelete$//" -e "s/^$todelete$//"`
	else
		# Run mmdelsnapshot, and capture the output to prevent verbose messages from being
		# sent as the result of each cron job. Only display the messages in case of error.
		output=`$MMDELSNAPSHOT $filesys $todelete 2>&1`
	fi
	if [ $? != 0 ] ; then
		printf "Error from \"$MMDELSNAPSHOT $filesys $todelete\": $output" 1>&2
		$LOGGER "Low disk space (${freenow}%) triggered attempt to remove snapshot of $filesys with label \"$todelete\" -- Error: $output"
		rm -f $LOCKFILE
		exit 1
	else
		$LOGGER "removed snapshot \"$todelete\" from $filesys because ${freenow}% free disk is less than ${MINFREE}%"
	fi

	# update the number of existing snapshots
	snapcount=`$MMLSSNAPSHOT |grep Valid |wc -l`
	freenow=`$MMDF $filesys|grep '(total)' | sed -e "s/%.*//" -e "s/.*( *//"`
done


if [ $snapcount = 0 -a $freenow -ge $MINFREE ] ; then
	echo "All existing snapshots removed on $filesys, but insufficient disk space to create a new snapshot: ${freenow}% free is less than ${MINFREE}%" 1>&2
	$LOGGER "All existing snapshots on $filesys removed, but insufficient disk space to create a new snapshot: ${freenow}% free is less than ${MINFREE}%"
	rm -f $LOCKFILE
	exit 1
fi

$LOGGER "Free disk space on $filesys (${freenow}%) above minimum required (${MINFREE}%) to create new snapshot"

##############################################################
if [ $TESTING = "yes" ] ; then
	# List snapshots being kept
	for oldsnap in $tokeep
	do
		echo "Keeping snapshot $oldsnap"
	done
fi
#############################################################

# Now create the current snapshot...do this after deleting snaps in order to reduce the chance of running 
# out of disk space
results=`$MMCRSNAPSHOT $filesys $now 2>&1 | tr "\012" "%"`
if [ $? != 0 ] ; then
	printf "Error from \"$MMCRSNAPSHOT $filesys $now\":\n\t" 1>&2
	echo $results | tr '%' '\012' 1>&2
	results=`echo $results | tr '%' '\012'`
	$LOGGER "Error creating snapshot of $filesys with label $now: \"$results\""
	rm -f $LOCKFILE
	exit 1
else
	$LOGGER "successfully created snapshot of $filesys with label $now"
fi
rm -f $LOCKFILE


More information about the gpfsug-discuss mailing list