Reply

How to restore a nas with rsync

eton
Apprentice

How to restore a nas with rsync

Here is how I restored my ReadyNAS Duo v.1 after factory reset from an external USB drive backup.
The backup on the USB disk was prior to reset created with a Frontview rsync backup job of volume c.
It's basically just the steps 7, 8, 9 and 10 below. The rest is how to set it up and fix problems.

To minimize headache and confusion, set the recreated users and groups id:s (UIDs and GIDs) to the same as before.
Of course UIDs and GIDs can be changed later, see below for related commands.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

1. Recreate the previous shared volumes and set permisions in Frontview. The permissions doesn't have to be the same as before.

2. Install "Enable Root SSH Access" add-on.

3. If the apt-get command is missing from terminal, install APT add-on for RAIDiator inside FrontView (the admin panel).

4. In terminal or Putty window login to server as root with admin or root password.
ssh root@[nas IP]

5. Re-synchronize the package index files from their sources.
apt-get update

6. Install the program called screen. Screen will keep your tasks/commands running even if you logout from the server.
apt-get install screen

7. Create a new terminal screen with the name rsync.
screen -S rsync

8. I restored the volumes one by one. Example for the volume named books and the external drive is connected to rear port 1. See note below for more about using rsync.
rsync -acvh /USB_HDD_1/books/ /c/books/
/USB_HDD_1/books/ = source, /c/books/ = destination

9. Detach from screen and let it work.
Ctrl+a, d
Most screen shortkeys begins with Ctrl+a. So first press Ctrl+a, relase and press the desired command, in this case d.

10. Re-login, step 4, after some time (you can usually hear when the drives are not that busy any more or check with iStat on your smartphone).
screen -r rsync


When done, repeat from step 8 for next volume.

And be patient. Rsync over USB2.0 to ReadyNAS Duo v1 goes slow!
Average transfer speeds are 2000000 bytes/sec to 3500000 bytes/sec. That's roughly 10 gigabyte/hour.
If anyone knows how to speed up this with some rsync trix, please reply.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Internet sources

http://serverkb.co.uk/wiki/SSH
https://whatbox.ca/wiki/rsync
http://www.thegeekstuff.com/2010/09/rsy ... -examples/
http://www.cyberciti.biz/tips/linux-scr ... howto.html
http://www.fprimex.com/linux/screen.html
http://sunsite.ualberta.ca/Documentatio ... creen.html

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Some useful commands when wokring with UIDs

Find files by user UID.
find /c/ -uid 1002

Find files by user name.
find ./folder -user jerry

Change user tom UID to 10000.
usermod -u 10000 tom

List all UID and GID from 1000 to 6000 (ReadyNAS default range in /etc/login.defs)
awk -F: '$3 >= 1000 && $3 <= 60000' /etc/passwd

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Notes using rsync

Another method is to use copy, cp, on first run only:
cp -a source dest
And after that, run rsync for safety and sync changes.

If you would like to run rsync a second time or after copy command. It's recommend to exclude hidden files on second pass (OS X creates a lot of hidden files). Those hidden .* files seem to take a lot of power from rsync, at least on a Sparc machine. I like to have them copied in the first run just so that all folders looks the same as before, but they don't have to be sync'd twice. If you don't care about the hidden files from Apple, exclude them in the first run. Or stop OS X from creating them. Add this to rsync to exclude hidden files:
--exclude='.*'

According to the man files -p for keeping permissions is not needed when using -a, archive. Archive -a, is the same as -rlptgoD. Arguments copied from man rsync:
-a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
-r, --recursive recurse into directories
-l, --links copy symlinks as symlinks
-p, --perms preserve permissions
-t, --times preserve modification times
-g, --group preserve group
-o, --owner preserve owner (super-user only)
-D same as --devices --specials. --devices preserve device files (super-user only). --specials preserve special files
-c, --checksum skip based on checksum, not mod-time & size
--progress This option tells rsync to print information showing the progress of the transfer. This gives a bored user something to watch. Implies --verbose if it wasn't already specified.

These are also some useful rsync options:
-P                          same as --partial --progress
-h, --human-readable output numbers in a human-readable format
-u, --update skip files that are newer on the receiver
-z, --compress compress file data during the transfer
-n, --dry-run show what would have been transferred

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Notes using screen

If you by misstake press:
ctrl + s
Ctrl-s calls the software flow control method XOFF which stops the character flow to the terminal, which when you did not realise what you pressed just seems to freeze the terminal.

To activate the terminal again.
ctrl + q
More here: http://munkymorgy.blogspot.se/2008/07/s ... s-bug.html

If you try to close a split screen with Ctrl+a, X you will be asked for password. Screen uses PAM for password management, but unfortunately it's not set up when installing. So you will be stuck. To solve this, login to the server in another/new terminal window and then use this command:
echo -n "auth    required    pam_unix.so" > /etc/pam.d/screen
Your root password should now work with Screen.
More here: http://serverfault.com/questions/169832 ... en-command

Occasionally screen needs to force a detach before it can successfully reattach. This is done with:
screen -d -r name

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Comparing folders and validating restoration

# diff -qrH -x 'DS_Store' -x '.*' /USB_HDD_1/books/ /c/books/ > diffs.txt
-q      Output only whether files differ.
-r Recursively compare any subdirectories found.
-H Assume large files and many scattered small changes.
-x PAT Exclude files that match PAT.

> Saves output to file
Not that diff can't handle files over 4GB due to 32-bit limitations. So diff will report large files with "Value too large for defined data type"

You can also compare dirs with rsync. This method also allows large files. Use it with verbose and dry run. If you are not paranoid about data, omit -c so rsync speeds up by not comparing checksums.
# rsync -rvvcn --exclude='.*' /USB_HDD_1/books/ /c/books/

Two -v flags will give you information on what files are being skipped and slightly more information at the end.

So the fastest way to compare dirs with rsync would be something like this:
# rsync -rvn --exclude='.*' /USB_HDD_1/books/ /c/books/

Or just count the number of files in specified dirs:
# find /c/books/ -type f | wc -l
Server: RN312 with 2 x Hitachi Deskstar 5K3000 2TB, backup: DB-ALU3E-6G (ext2). Router: PC Engines APU2D4 (pfSense). Switch: Cisco SLM2008. AP: UniFi nanoHD.
Message 1 of 1
Top Contributors
Discussion stats
  • 0 replies
  • 2293 views
  • 1 kudo
  • 1 in conversation
Announcements