NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
OS6 Data Recovery - How to Mount BTRFS Volumes
I recently purchased a ReadyNAS 314 and 5 Seagate ST2000VN000 hard disks (one as a cold spare). I work as a system administrator, so I've been reading up on OS6 before I entrust this new system with my data. I'm not very comfortable using BTRFS since it seems to still be a little buggy and I don't need snapshots. But since that's the direction Netgear chose to go, I'd like to give it a fair chance.
The older generation of devices had several excellent web sites with detailed data recovery instructions for situations involving hardware failures. Usually, this involved removing the hard disks and connecting them to a desktop, then booting into a version of Linux and mounting the array using the familiar ext4/LVM tools. I've been searching through the forums and Googling for an updated version, but there don't seem to be any recovery instructions for the new generation. I've seen a lot of discussion about BTRFS including some comments that make me quite concerned.
As mangrove stated in thread 419088
"It should also be said that for EXT solutions, there are restore tools available which you run from a computer. So if the unit fails and you need to get data back, you can without having to buy an identical unit. This is much harder though.
This is impossible with OS6 units as BTRFS is not well supported by restore software."
I'm sure this is paranoid of me, but before I start to trust this device with 4 Tb of data, which is time consuming and difficult to back up in multiple places, I need to know I can access the data if the proprietary hardware fails. The situation I'm thinking of is the ReadyNAS works fine for about 6 years, then the mainboard fails. It's out of warranty, there are no replacement parts and the newer models can't read the volumes due to data structure changes. The disks are fine, the data is there, but it can't be accessed without mounting the disks on a different machine.
Option 1 - Test data recovery using the current OS 6.1.6 build and a spare desktop. Set up the array in the normal way using OS6 tools, save some test data on it, then shut it down and take the disks out. Figure out how to mount the disks and access the volume by connecting them to the desktop and installing the appropriate Linux/kernel/disk tools on a separate disk. Once this is working, create a small Linux build on a thumb drive that is pre-configured to mount up the disks properly. My preferred configuration would be Flex-RAID set up in RAID6. But I'll test Flex-RAID in RAID5 and XRaid2 if I have time.
If that can be done, then I'll go ahead and use the system and just keep an updated version of the recovery thumb drive handy (updated to match each new build of OS6).
I'm not here on the forum to ask someone to do this for me. Since I happen to have a new ReadyNAS 314 with blank hard disks and a spare desktop sitting around, I'm happy to roll up my sleeves and test it myself. I'm not a Linux guru, but the command line doesn't scare me. And at this point, I'm not risking any real data and this will allow me to have my recovery solution already built and ready to go. I'll post the results here in the forum for others, since there doesn't seem to be a definitive solution out there (or if someone can point me to one that already exists? Thanks! I can try that out first and save time!)
What I'm here to ask for, since there are so many very experienced ReadyNAS Jedis, is for some background on the data structure so I can get started. What I need to know is the following:
The older generation of devices had several excellent web sites with detailed data recovery instructions for situations involving hardware failures. Usually, this involved removing the hard disks and connecting them to a desktop, then booting into a version of Linux and mounting the array using the familiar ext4/LVM tools. I've been searching through the forums and Googling for an updated version, but there don't seem to be any recovery instructions for the new generation. I've seen a lot of discussion about BTRFS including some comments that make me quite concerned.
As mangrove stated in thread 419088
"It should also be said that for EXT solutions, there are restore tools available which you run from a computer. So if the unit fails and you need to get data back, you can without having to buy an identical unit. This is much harder though.
This is impossible with OS6 units as BTRFS is not well supported by restore software."
I'm sure this is paranoid of me, but before I start to trust this device with 4 Tb of data, which is time consuming and difficult to back up in multiple places, I need to know I can access the data if the proprietary hardware fails. The situation I'm thinking of is the ReadyNAS works fine for about 6 years, then the mainboard fails. It's out of warranty, there are no replacement parts and the newer models can't read the volumes due to data structure changes. The disks are fine, the data is there, but it can't be accessed without mounting the disks on a different machine.
Option 1 - Test data recovery using the current OS 6.1.6 build and a spare desktop. Set up the array in the normal way using OS6 tools, save some test data on it, then shut it down and take the disks out. Figure out how to mount the disks and access the volume by connecting them to the desktop and installing the appropriate Linux/kernel/disk tools on a separate disk. Once this is working, create a small Linux build on a thumb drive that is pre-configured to mount up the disks properly. My preferred configuration would be Flex-RAID set up in RAID6. But I'll test Flex-RAID in RAID5 and XRaid2 if I have time.
If that can be done, then I'll go ahead and use the system and just keep an updated version of the recovery thumb drive handy (updated to match each new build of OS6).
I'm not here on the forum to ask someone to do this for me. Since I happen to have a new ReadyNAS 314 with blank hard disks and a spare desktop sitting around, I'm happy to roll up my sleeves and test it myself. I'm not a Linux guru, but the command line doesn't scare me. And at this point, I'm not risking any real data and this will allow me to have my recovery solution already built and ready to go. I'll post the results here in the forum for others, since there doesn't seem to be a definitive solution out there (or if someone can point me to one that already exists? Thanks! I can try that out first and save time!)
What I'm here to ask for, since there are so many very experienced ReadyNAS Jedis, is for some background on the data structure so I can get started. What I need to know is the following:
- 1 Which OS would be best to access the data? It appears that Debian is the core of OS6, but which build/kernel should be used?
[list=2:f5h9ejz4]2 Which tools are needed for testing/repairing/mounting the BTRFS filesystem?
[list=3:f5h9ejz4]3 A general overview of how the volumes are arranged (ie. it's not LVM anymore, so what is it?)
[list=4:f5h9ejz4]4 Specific settings to be aware of that might not be standard in the vanilla Linux configuration (ie. block sizes? other things I don't know about at all)
[list=5:f5h9ejz4]5 Gotchas or special hazards to watch out for when working with BTRFS. I'm really not familiar with it.
[list=6:f5h9ejz4]6 Which log files show success or failure and which command-line commands can test the volume and data.
This doesn't have to be super detailed or line-by-line. Just let me know the basics and I can look up the syntax and details in the man pages or online. I'm sure it'll blow up on me at least the first few times and I'll be back on this thread with the output and more questions when I get stuck. :shock:
Option 2 - Work on a way to change the file structure to the old-style EXT4/LVM so the older generation recovery process works. Yes, I understand that this is not supported and would have to be painfully redone after every OS version upgrade, but it might be a (tedious) solution.
Just a quick note on what I'm planning to do with this unit - I just need to use iSCSI to connect it to a CentOS 6 virtual server running on VMware. That server will provide all the file management, permissions and network services. I just need OS6 to provide iSCSI connectivity, basic hardware monitoring and UPS-triggered safe shutdown (I think the APC Network Shutdown 3.1 Linux software will work - I'll test that also). The primary role for the ReadyNAS will be to provide centralized backup to another NAS and various servers. Some of the servers will backup and store data from the desktops that will also be backed up to the ReadyNAS.
I know this probably sounds like belt, suspenders and a flak jacket, but data integrity is a big deal with me. I'm hoping that what I find out will be useful to other people if they have a data access problems (or system meltdown). Plus, since system administration is my day job, this is kind of a scale model of my work system and should be good training as well (score!) :D
Thanks in advance to all the ReadyNAS experts out there for your time and assistance. I know I'm asking for a lot, but I'll share what I find out in return. Please be patient - I have a little Linux experience, but mostly at the power-user level so I'm weak in some of the admin areas. (Yeah, yeah, I do Windows system administration :( - stop snickering)
Stan
33 Replies
Replies have been turned off for this discussion
- It doesn't use LVM anymore. On Linux the steps are actually less involved than for previous OSes to mount the volume.
Just need an ordinary Linux PC with a reasonably up to date kernel, and to install mdadm and btrfs-progs (if they are not already installed).
Something like:
# apt-get update
# apt-get install mdadm btrfs-tools
# mdadm --assemble --scan
# cat /proc/mdstat
# mount -t btrfs -o ro /dev/md127 /mnt
Your data volumes should be on /dev/md127, /dev/md126 etc.
/dev/md0 holds the 4GB OS partition and /dev/md1 is for swap.
Welcome to the forum! - Regarding EXT4: I have tried to make the volumes EXT4, but haven't succeeded as of yet. I can create a volume that shows up alright, but it's impossible to create iSCSI containers on that volume. There might be something in the iSCSI scripts that requires the file system to be BTRFS.
- Thanks mdgm - it looks deceptively simple. I'll give it a try and put a play-by-play on here later. If it works OK, then my fears are unfounded and I'll go ahead and start using it for real data.
Stan - If you have an array of 6 disks, do they need to be placed on the same disk device port (/dev/sdaX) as on the ReadyNAS or is mdadm smart enough to figure that out?
Since I need to do a factory reset anyway and I have 6 other disks sitting around I figure I could just move the 6 disks that are in the ReadyNAS to a PC instead of doing a backup, then once I populate the ReadyNAS with 6 new disks I could just restore from the PC using the original disks. mdgm wrote: Just need an ordinary Linux PC with a reasonably up to date kernel, and to install mdadm and btrfs-progs (if they are not already installed).
Hello mdgm, just a clarification on this matter.
Assuming I have this scenario: a RN102/104 equipped with 2 HDDs or a RN312/RN314 (maybe valid also for a RN316) equipped with 4 HDDs both configured with so called - AFAIK default, "X-RAID" (and BRTFS as factory default filesystem for data) and I want/need to recover data from them because things went badly wrong.
Question (maybe silly): Have I to pull out all involved ReadyNAS HDDs and push them in into a suitable Linux box (with enough SATA ports available) respecting exact ReadyNAS HDDs install order and exact HDDs quantity (so, as example, equipping the Linux box with HDD 1 and 2 from a RN102/312 fully filled or with HDD 1, 2, 3 and 4 from a RN104/314 fully filled) so to have, on the target Linux system, new devices sdb/sdc or sdb/sdc/sde/sdf (considering sda used by my Linux distribution for its OS)?
The question above should be also valid if I have a 3 HDDs configuration (or a different one, as example, 2 HDDs on a RN104/RN314) too. Thanks!JMehring wrote: If you have an array of 6 disks, do they need to be placed on the same disk device port (/dev/sdaX) as on the ReadyNAS or is mdadm smart enough to figure that out?
It shouldn't matter but you should keep the order the same where possible. It would be a good idea to note down the order e.g. note which serial number is for disk 1 etc.JMehring wrote:
Since I need to do a factory reset anyway and I have 6 other disks sitting around I figure I could just move the 6 disks that are in the ReadyNAS to a PC instead of doing a backup, then once I populate the ReadyNAS with 6 new disks I could just restore from the PC using the original disks.
That's not recommended. If the NAS is working it would be better to do a backup.kimera wrote:
Have I to pull out all involved ReadyNAS HDDs and push them in into a suitable Linux box (with enough SATA ports available) respecting exact ReadyNAS HDDs install order and exact HDDs quantity (so, as example, equipping the Linux box with HDD 1 and 2 from a RN102/312 fully filled or with HDD 1, 2, 3 and 4 from a RN104/314 fully filled) so to have, on the target Linux system, new devices sdb/sdc or sdb/sdc/sde/sdf (considering sda used by my Linux distribution for its OS)?
If you have four disks in the NAS and the array is fine, you would need at least three disks. If you had two disks in the array you would need at least one of them etc.
It's possible you might be able to mount the array putting the disks in e.g. USB enclosures.
I think the PC should be smart enough to tell which disks are in the array, but if not you could specify them manually.mdgm wrote: If you have four disks in the NAS and the array is fine, you would need at least three disks. If you had two disks in the array you would need at least one of them etc.
Leaving out of the picture which disk? any disk of the four disks array (in case of a ReadyNAS unit filled with four disks)? isn't the "parity" spread all over the array and so the whole array should be pulled out from ReadyNAS unit to be pushed in the Linux box or the the parity is not evaluated when rebuilding the md RAID array in Linux (and so one disk could be left behind)?- You really should connect all disks.
The parity is spread amongst the disks so if one disk is missing the volume should still mount though in a degraded state. - Thanks for the clarification!
- First OS6 Data Recovery Test
ReadyNAS 314 configuration
OS 6.1.6
Default configuration - one user created
Drives - 4 x Seagate ST2000NV000 disks - 2 Tb (on the approved hardware list)
Volume configuration - XRAID2 default - RAID 5 array for data partition (appears as MD127)
Approximately 17 Gb of data copied over - Debian Linux 7.4 AMD64 DVD ISO files (five files) verified with MD5
One user created - SMB used as the file share protocol
Recovery PC configuration
OS - Debian Linux 7.4 with all updates to 19 July 2014 - default install with Desktop, File Server and Base packages only
Added MDADM package using APT-GET (as described in the recommended directions above)
Volumes default but with separate partition for /home
OS name 3.2.0-4-amd64
Kernel 3.2.60-1+deb7u1
Motherboard Gigabyte GA-Z77X-UP5TH
i5 CPU 3570
8 Gb RAM
6 SATA ports
Port 0 - Plextor DVD drive
Port 1 - Seagate ST3808110AS - 80 Gb SATA drive for OS install
Port 2 - Drive 1 from ReadyNAS
Port 3 - Drive 2 from ReadyNAS
Port 4 - Drive 3 from ReadyNAS
Port 5 - Drive 4 from ReadyNAS
BIOS set to AHCI and "Legacy" mode (meaning non-UEFI)
Recovering the data at the command line
Followed the above guide shown above (all commands executed as root at the command line):
apt-get update (updates packages and kernel to latest online version)
apt-get install mdadm (RAID volume manager package)
mdadm --assemble --scan (finds and connects the RAID arrays as /dev/md0, /dev/md1, /dev/md127)
cat /proc/mdstat (shows the connected arrays and their properties - shows if the above command worked)
mount -t btrfs -o ro /dev/md127 /mnt (mounts the data volume at /mnt and allows access)
NOTE: I found that the correct name for the btrfs utilities is actually "btrfs-tools", but I didn't install or use any of them.
Data recovery
Once the volume was mounted, I was able to move to the /mnt/Documents folder where the test ISO files were stored. The files were moved to the folder /home/<username>/recov. The files were checked again by generating an MD5 sum and checking it against the official MD5 sum in the distro. The MD5 signature matched showing that the data was not corrupted. This was only 17 Gb of test data, but it was just a proof of concept.
Final Thoughts
This was a lot easier than I thought it would be. One important thing is to use a recent distro with kernel 3.x and BTRFS support. The 7 series of Debian Linux has BTRFS support built in and appears to be able to recognize XRAID2 RAID5 volumes and mount them without special settings. There is a package called "btrfs-tools" with testing and repair tools, but I didn't use them at all. It's easier if the motherboard has at least 5 or 6 SATA ports, but it could be done with just 4 if you would be willing to boot to a USB stick or use a net boot.
Please post here if you have questions or need more information. I'm kind of a Linux newbie, but I'll help as much as I can.
CAUTION: FlexRAID with RAID 6 DOES NOT WORK USING THIS METHOD
After the above test, I changed the configuration on the ReadyNAS to FlexRAID with all the disks in a RAID 6 array. This erased all the data, but no other configuration changes were made. I set up an SMB share named "test" and copied the exact same data to the folder and verified the MD5 sums. I pulled the drives and tried to recover the data use the exact same configuration and attempted to mount the volumes, but it didn't work using the method shown above. I'm still testing, but there might be some issues with RAID 6. FlexRAID seems to change the configuration to one RAID 1 OS volume and one RAID 6 data volume. The OS volume mounts using the above method, but the data array won't mount up. I'll do some more testing and try to fix it. If I can't get it to work, I'll update to Debian 7.6.x to see if the updated version might have better support.
If anyone has any ideas about mounting this, please let me know. The error message says that it can't identify the type of the file system and thinks it might be a swap file. Auto type doesn't work and setting the type to BTRFS fails with that error message.
More later when I have more time. Sorry for the long delay in following up on the original post, but I'm working crazy hours these days.
Stan
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
