×
INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!
  • Students Click Here

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

Jobs

rsync with hardlinks rotating snapshots code

rsync with hardlinks rotating snapshots code

rsync with hardlinks rotating snapshots code

(OP)
Hello everyone,

I’m curious to know if my code is working as designed? The first hourly.0 performs a full backup then every snapshot after (hourly.1 —> hourly.11 afterwards is hardlinked. Now here is the part I’m confused. During one of the runs, I’ll get a full backup performed again on hourly.1 and hourly.2 which I don’t understand. If the rotate function moves 10->11 9->10 8->9 7->8 etc... Why don’t I have just 1 full copy? On the 12th run, hourly.11 would have rotated out and been deleted according to my code? Thanks in advance.

du -sch * //
hourly.0 10gb
hourly.1 20k
hourly.2. 20k
and
so
on...
—————
rotateSnapshots()
{
if [ -d “$DST/hourly.0” ]; then
rm -fr hourly.11
for i in {11..1}
do
mv "$DST/hourly.$[${i}-1]" "$DST/hourly.${i}" 2> /dev/null
sleep 1
done
fi
}

runIncremental()
{
/opt/local/bin/rsync -a --delete --link-dest="$DST/hourly.1" "$SRC" "$DST/hourly.0"
sleep 1
}

RE: rsync with hardlinks rotating snapshots code

(OP)
additional info: this is after many runs of the script... please assist. much appreciated.

sample expected ouput:
hourly.0 10gb
hourly.1 20k
hourly.2. 20k
hourly.3 10k
hourly.4. 20k
hourly.5 20k
hourly.6 10k
hourly.7. 20k
hourly.8 20k
hourly.9 20k
hourly.10 20k
hourly.11 20k

unfortunate output I get:
hourly.0 10gb
hourly.1 10gb
hourly.2. 10gb
hourly.3 10k
hourly.4. 20k
hourly.5 20k
hourly.6 10k
hourly.7. 20k
hourly.8 20k
ourly.9 20k
hourly.10 20k
hourly.11 10k

RE: rsync with hardlinks rotating snapshots code

Not sure I fully understand, but your loop goes from 11 to 1, which means the last pass moves hourly.0 to hourly.1. That's your full backup going to hourly.1. Then, I assume this pass or next pass does a full backup again because there's no hourly.0.

But I'm making some assumptions because you just posted a snippet of code and not the whole thing.

RE: rsync with hardlinks rotating snapshots code

(OP)
I’ll post the entire code when I get home tonight.

RE: rsync with hardlinks rotating snapshots code

(OP)
Here is the code. If you run it, you'll need a partition/volume called 'Hourly' ...
It appears to be working. Just need a second opinion on it ..because it shouldn't be creating more than 1 full backup other than the first time it runs and excluding the SRC directory. After running it more than 12 times .. it appears to be working ..?
All the 0B sizes are all hardlinks below, just as expected ..

du -sch /Volumes/Hourly/snapshot/*
860K /Volumes/Hourly/snapshot/Source
860K /Volumes/Hourly/snapshot/hourly.0
0B /Volumes/Hourly/snapshot/hourly.1
0B /Volumes/Hourly/snapshot/hourly.10
0B /Volumes/Hourly/snapshot/hourly.11
0B /Volumes/Hourly/snapshot/hourly.2
0B /Volumes/Hourly/snapshot/hourly.3
0B /Volumes/Hourly/snapshot/hourly.4
0B /Volumes/Hourly/snapshot/hourly.5
0B /Volumes/Hourly/snapshot/hourly.6
0B /Volumes/Hourly/snapshot/hourly.7
0B /Volumes/Hourly/snapshot/hourly.8
0B /Volumes/Hourly/snapshot/hourly.9
1.7M total


--------------
#!/bin/bash
set -x

SRC="/Volumes/Hourly/snapshot/Source/"
DST="/Volumes/Hourly/snapshot"

consolidate()
{

/opt/local/bin/rsync -a --delete "/Users/user1/Documents" "$SRC"

}

rotateSnapshots()
{
cd "$DST"
if [ -d hourly.0 ]; then
rm -fr hourly.11
sleep 5
for i in {11..1}
do
mv "$DST/hourly.$[${i}-1]" "$DST/hourly.${i}" 2> /dev/null
sleep 1
done
fi
}

runIncremental()
{
/opt/local/bin/rsync -a --delete --link-dest="$DST/hourly.1" "$SRC" "$DST/hourly.0"
sleep 1
}

consolidate;
rotateSnapshots;
runIncremental;
exit 0

RE: rsync with hardlinks rotating snapshots code

(OP)
And in response to your question, yes, the hourly.0 moves to hourly.1 moving the first time full backup down the line until it gets to hourly.11 and then it gets deleted per the code. it should repeat again ..with full backup starting at hourly.0, am I right?

RE: rsync with hardlinks rotating snapshots code

You should post your code between code tags; [code][/code]. That makes it MUCH easier to read and debug (see below).

Looking at the code, it's not really clear what you're trying to do. Every time you run this script you do a full rsync, then rotate the files, then do an incremental. I don't understand what you're trying to do.

The -link-test parameter creates hard links, not soft links. That means a du or an ls will report the full size of the file. The hard links give an misleading idea of how much space is actually being taken up. Each directory will report the full file sizes of the linked files, but it's actually only taking up the space on disk once.

Also, you don't need the semi-colons at the end of the function calls. They do no harm, but they aren't needed.

CODE

#!/bin/bash
set -x

SRC="/Volumes/Hourly/snapshot/Source/"
DST="/Volumes/Hourly/snapshot"

consolidate()
{
/opt/local/bin/rsync -a --delete "/Users/user1/Documents" "$SRC"
}

rotateSnapshots()
{
cd "$DST"
if [ -d hourly.0 ]; then
    rm -fr hourly.11
    sleep 5
    for i in {11..1}
    do
        mv "$DST/hourly.$[${i}-1]" "$DST/hourly.${i}" 2> /dev/null
        sleep 1
    done
fi
}

runIncremental()
{
/opt/local/bin/rsync -a --delete --link-dest="$DST/hourly.1" "$SRC" "$DST/hourly.0"
sleep 1
}

consolidate;
rotateSnapshots;
runIncremental;

exit 0 

RE: rsync with hardlinks rotating snapshots code

Oh, also, why do you have sleeps sprinkled throughout the code? All that does is slow the script down, and nothing more.

If you're worried about a command not completing before the next command (i.e. rsync, rm, mv, etc), it would be better to put a sync in there. The sync command will flush the buffers to disk (or at least request that they be flushed). That will add small delays while the buffers write, but those delays are for a reason, not just because you want to slow the script down for some reason.

Example:

CODE

rotateSnapshots()
{
cd "$DST"
if [ -d hourly.0 ]; then
    rm -fr hourly.11
    sync
    for i in {11..1}
    do
        mv "$DST/hourly.$[${i}-1]" "$DST/hourly.${i}" 2> /dev/null
        sync ; sync ; sync    # Multiple syncs don't hurt anything
    done
fi
} 

Just keep in mind that the sync command affects the buffers for the whole machine, so you could end up impacting performance of other things running on that machine.




RE: rsync with hardlinks rotating snapshots code

(OP)
Thanks Sam. Appreciate your feedback. Very insightful. Always learning something new here.

RE: rsync with hardlinks rotating snapshots code

You're welcome. Glad to help.

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members! Already a Member? Login

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close