Reply
Topic Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
How to restore a nas with rsync
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2013-01-23
01:59 AM
2013-01-23
01:59 AM
How to restore a nas with rsync
Here is how I restored my ReadyNAS Duo v.1 after factory reset from an external USB drive backup.
The backup on the USB disk was prior to reset created with a Frontview rsync backup job of volume c.
It's basically just the steps 7, 8, 9 and 10 below. The rest is how to set it up and fix problems.
To minimize headache and confusion, set the recreated users and groups id:s (UIDs and GIDs) to the same as before.
Of course UIDs and GIDs can be changed later, see below for related commands.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1. Recreate the previous shared volumes and set permisions in Frontview. The permissions doesn't have to be the same as before.
2. Install "Enable Root SSH Access" add-on.
3. If the apt-get command is missing from terminal, install APT add-on for RAIDiator inside FrontView (the admin panel).
4. In terminal or Putty window login to server as root with admin or root password.
5. Re-synchronize the package index files from their sources.
6. Install the program called screen. Screen will keep your tasks/commands running even if you logout from the server.
7. Create a new terminal screen with the name rsync.
8. I restored the volumes one by one. Example for the volume named books and the external drive is connected to rear port 1. See note below for more about using rsync.
9. Detach from screen and let it work.
10. Re-login, step 4, after some time (you can usually hear when the drives are not that busy any more or check with iStat on your smartphone).
When done, repeat from step 8 for next volume.
And be patient. Rsync over USB2.0 to ReadyNAS Duo v1 goes slow!
Average transfer speeds are 2000000 bytes/sec to 3500000 bytes/sec. That's roughly 10 gigabyte/hour.
If anyone knows how to speed up this with some rsync trix, please reply.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Internet sources
http://serverkb.co.uk/wiki/SSH
https://whatbox.ca/wiki/rsync
http://www.thegeekstuff.com/2010/09/rsy ... -examples/
http://www.cyberciti.biz/tips/linux-scr ... howto.html
http://www.fprimex.com/linux/screen.html
http://sunsite.ualberta.ca/Documentatio ... creen.html
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Some useful commands when wokring with UIDs
Find files by user UID.
Find files by user name.
Change user tom UID to 10000.
List all UID and GID from 1000 to 6000 (ReadyNAS default range in /etc/login.defs)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Notes using rsync
Another method is to use copy, cp, on first run only:
If you would like to run rsync a second time or after copy command. It's recommend to exclude hidden files on second pass (OS X creates a lot of hidden files). Those hidden .* files seem to take a lot of power from rsync, at least on a Sparc machine. I like to have them copied in the first run just so that all folders looks the same as before, but they don't have to be sync'd twice. If you don't care about the hidden files from Apple, exclude them in the first run. Or stop OS X from creating them. Add this to rsync to exclude hidden files:
According to the man files -p for keeping permissions is not needed when using -a, archive. Archive -a, is the same as -rlptgoD. Arguments copied from man rsync:
These are also some useful rsync options:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Notes using screen
If you by misstake press:
To activate the terminal again.
If you try to close a split screen with Ctrl+a, X you will be asked for password. Screen uses PAM for password management, but unfortunately it's not set up when installing. So you will be stuck. To solve this, login to the server in another/new terminal window and then use this command:
More here: http://serverfault.com/questions/169832 ... en-command
Occasionally screen needs to force a detach before it can successfully reattach. This is done with:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Comparing folders and validating restoration
You can also compare dirs with rsync. This method also allows large files. Use it with verbose and dry run. If you are not paranoid about data, omit -c so rsync speeds up by not comparing checksums.
So the fastest way to compare dirs with rsync would be something like this:
Or just count the number of files in specified dirs:
The backup on the USB disk was prior to reset created with a Frontview rsync backup job of volume c.
It's basically just the steps 7, 8, 9 and 10 below. The rest is how to set it up and fix problems.
To minimize headache and confusion, set the recreated users and groups id:s (UIDs and GIDs) to the same as before.
Of course UIDs and GIDs can be changed later, see below for related commands.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1. Recreate the previous shared volumes and set permisions in Frontview. The permissions doesn't have to be the same as before.
2. Install "Enable Root SSH Access" add-on.
3. If the apt-get command is missing from terminal, install APT add-on for RAIDiator inside FrontView (the admin panel).
4. In terminal or Putty window login to server as root with admin or root password.
ssh root@[nas IP]
5. Re-synchronize the package index files from their sources.
apt-get update
6. Install the program called screen. Screen will keep your tasks/commands running even if you logout from the server.
apt-get install screen
7. Create a new terminal screen with the name rsync.
screen -S rsync
8. I restored the volumes one by one. Example for the volume named books and the external drive is connected to rear port 1. See note below for more about using rsync.
rsync -acvh /USB_HDD_1/books/ /c/books//USB_HDD_1/books/ = source, /c/books/ = destination
9. Detach from screen and let it work.
Ctrl+a, dMost screen shortkeys begins with Ctrl+a. So first press Ctrl+a, relase and press the desired command, in this case d.
10. Re-login, step 4, after some time (you can usually hear when the drives are not that busy any more or check with iStat on your smartphone).
screen -r rsync
When done, repeat from step 8 for next volume.
And be patient. Rsync over USB2.0 to ReadyNAS Duo v1 goes slow!
Average transfer speeds are 2000000 bytes/sec to 3500000 bytes/sec. That's roughly 10 gigabyte/hour.
If anyone knows how to speed up this with some rsync trix, please reply.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Internet sources
http://serverkb.co.uk/wiki/SSH
https://whatbox.ca/wiki/rsync
http://www.thegeekstuff.com/2010/09/rsy ... -examples/
http://www.cyberciti.biz/tips/linux-scr ... howto.html
http://www.fprimex.com/linux/screen.html
http://sunsite.ualberta.ca/Documentatio ... creen.html
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Some useful commands when wokring with UIDs
Find files by user UID.
find /c/ -uid 1002
Find files by user name.
find ./folder -user jerry
Change user tom UID to 10000.
usermod -u 10000 tom
List all UID and GID from 1000 to 6000 (ReadyNAS default range in /etc/login.defs)
awk -F: '$3 >= 1000 && $3 <= 60000' /etc/passwd
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Notes using rsync
Another method is to use copy, cp, on first run only:
cp -a source destAnd after that, run rsync for safety and sync changes.
If you would like to run rsync a second time or after copy command. It's recommend to exclude hidden files on second pass (OS X creates a lot of hidden files). Those hidden .* files seem to take a lot of power from rsync, at least on a Sparc machine. I like to have them copied in the first run just so that all folders looks the same as before, but they don't have to be sync'd twice. If you don't care about the hidden files from Apple, exclude them in the first run. Or stop OS X from creating them. Add this to rsync to exclude hidden files:
--exclude='.*'
According to the man files -p for keeping permissions is not needed when using -a, archive. Archive -a, is the same as -rlptgoD. Arguments copied from man rsync:
-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)
-r, --recursive recurse into directories
-l, --links copy symlinks as symlinks
-p, --perms preserve permissions
-t, --times preserve modification times
-g, --group preserve group
-o, --owner preserve owner (super-user only)
-D same as --devices --specials. --devices preserve device files (super-user only). --specials preserve special files
-c, --checksum skip based on checksum, not mod-time & size
--progress This option tells rsync to print information showing the progress of the transfer. This gives a bored user something to watch. Implies --verbose if it wasn't already specified.
These are also some useful rsync options:
-P same as --partial --progress
-h, --human-readable output numbers in a human-readable format
-u, --update skip files that are newer on the receiver
-z, --compress compress file data during the transfer
-n, --dry-run show what would have been transferred
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Notes using screen
If you by misstake press:
ctrl + s
Ctrl-s calls the software flow control method XOFF which stops the character flow to the terminal, which when you did not realise what you pressed just seems to freeze the terminal.
To activate the terminal again.
ctrl + qMore here: http://munkymorgy.blogspot.se/2008/07/s ... s-bug.html
If you try to close a split screen with Ctrl+a, X you will be asked for password. Screen uses PAM for password management, but unfortunately it's not set up when installing. So you will be stuck. To solve this, login to the server in another/new terminal window and then use this command:
echo -n "auth required pam_unix.so" > /etc/pam.d/screenYour root password should now work with Screen.
More here: http://serverfault.com/questions/169832 ... en-command
Occasionally screen needs to force a detach before it can successfully reattach. This is done with:
screen -d -r name
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Comparing folders and validating restoration
# diff -qrH -x 'DS_Store' -x '.*' /USB_HDD_1/books/ /c/books/ > diffs.txt
-q Output only whether files differ.Not that diff can't handle files over 4GB due to 32-bit limitations. So diff will report large files with "Value too large for defined data type"
-r Recursively compare any subdirectories found.
-H Assume large files and many scattered small changes.
-x PAT Exclude files that match PAT.
> Saves output to file
You can also compare dirs with rsync. This method also allows large files. Use it with verbose and dry run. If you are not paranoid about data, omit -c so rsync speeds up by not comparing checksums.
# rsync -rvvcn --exclude='.*' /USB_HDD_1/books/ /c/books/
Two -v flags will give you information on what files are being skipped and slightly more information at the end.
So the fastest way to compare dirs with rsync would be something like this:
# rsync -rvn --exclude='.*' /USB_HDD_1/books/ /c/books/
Or just count the number of files in specified dirs:
# find /c/books/ -type f | wc -l
Message 1 of 1
Labels: