I recently purchased a ReadyNAS 314 and 5 Seagate ST2000VN000 hard disks (one as a cold spare). I work as a system administrator, so I've been reading up on OS6 before I entrust this new system with my data. I'm not very comfortable using BTRFS since it seems to still be a little buggy and I don't need snapshots. But since that's the direction Netgear chose to go, I'd like to give it a fair chance.
The older generation of devices had several excellent web sites with detailed data recovery instructions for situations involving hardware failures. Usually, this involved removing the hard disks and connecting them to a desktop, then booting into a version of Linux and mounting the array using the familiar ext4/LVM tools. I've been searching through the forums and Googling for an updated version, but there don't seem to be any recovery instructions for the new generation. I've seen a lot of discussion about BTRFS including some comments that make me quite concerned.
As mangrove stated in thread 419088 "It should also be said that for EXT solutions, there are restore tools available which you run from a computer. So if the unit fails and you need to get data back, you can without having to buy an identical unit. This is much harder though.
This is impossible with OS6 units as BTRFS is not well supported by restore software."
I'm sure this is paranoid of me, but before I start to trust this device with 4 Tb of data, which is time consuming and difficult to back up in multiple places, I need to know I can access the data if the proprietary hardware fails. The situation I'm thinking of is the ReadyNAS works fine for about 6 years, then the mainboard fails. It's out of warranty, there are no replacement parts and the newer models can't read the volumes due to data structure changes. The disks are fine, the data is there, but it can't be accessed without mounting the disks on a different machine.
Option 1 - Test data recovery using the current OS 6.1.6 build and a spare desktop. Set up the array in the normal way using OS6 tools, save some test data on it, then shut it down and take the disks out. Figure out how to mount the disks and access the volume by connecting them to the desktop and installing the appropriate Linux/kernel/disk tools on a separate disk. Once this is working, create a small Linux build on a thumb drive that is pre-configured to mount up the disks properly. My preferred configuration would be Flex-RAID set up in RAID6. But I'll test Flex-RAID in RAID5 and XRaid2 if I have time.
If that can be done, then I'll go ahead and use the system and just keep an updated version of the recovery thumb drive handy (updated to match each new build of OS6).
I'm not here on the forum to ask someone to do this for me. Since I happen to have a new ReadyNAS 314 with blank hard disks and a spare desktop sitting around, I'm happy to roll up my sleeves and test it myself. I'm not a Linux guru, but the command line doesn't scare me. And at this point, I'm not risking any real data and this will allow me to have my recovery solution already built and ready to go. I'll post the results here in the forum for others, since there doesn't seem to be a definitive solution out there (or if someone can point me to one that already exists? Thanks! I can try that out first and save time!)
What I'm here to ask for, since there are so many very experienced ReadyNAS Jedis, is for some background on the data structure so I can get started. What I need to know is the following:
1 Which OS would be best to access the data? It appears that Debian is the core of OS6, but which build/kernel should be used? [list=2:f5h9ejz4]2 Which tools are needed for testing/repairing/mounting the BTRFS filesystem? [list=3:f5h9ejz4]3 A general overview of how the volumes are arranged (ie. it's not LVM anymore, so what is it?) [list=4:f5h9ejz4]4 Specific settings to be aware of that might not be standard in the vanilla Linux configuration (ie. block sizes? other things I don't know about at all) [list=5:f5h9ejz4]5 Gotchas or special hazards to watch out for when working with BTRFS. I'm really not familiar with it. [list=6:f5h9ejz4]6 Which log files show success or failure and which command-line commands can test the volume and data.
This doesn't have to be super detailed or line-by-line. Just let me know the basics and I can look up the syntax and details in the man pages or online. I'm sure it'll blow up on me at least the first few times and I'll be back on this thread with the output and more questions when I get stuck. :shock:
Option 2 - Work on a way to change the file structure to the old-style EXT4/LVM so the older generation recovery process works. Yes, I understand that this is not supported and would have to be painfully redone after every OS version upgrade, but it might be a (tedious) solution.
Just a quick note on what I'm planning to do with this unit - I just need to use iSCSI to connect it to a CentOS 6 virtual server running on VMware. That server will provide all the file management, permissions and network services. I just need OS6 to provide iSCSI connectivity, basic hardware monitoring and UPS-triggered safe shutdown (I think the APC Network Shutdown 3.1 Linux software will work - I'll test that also). The primary role for the ReadyNAS will be to provide centralized backup to another NAS and various servers. Some of the servers will backup and store data from the desktops that will also be backed up to the ReadyNAS.
I know this probably sounds like belt, suspenders and a flak jacket, but data integrity is a big deal with me. I'm hoping that what I find out will be useful to other people if they have a data access problems (or system meltdown). Plus, since system administration is my day job, this is kind of a scale model of my work system and should be good training as well (score!) 😄
Thanks in advance to all the ReadyNAS experts out there for your time and assistance. I know I'm asking for a lot, but I'll share what I find out in return. Please be patient - I have a little Linux experience, but mostly at the power-user level so I'm weak in some of the admin areas. (Yeah, yeah, I do Windows system administration 😞 - stop snickering)
Re: OS6 Data Recovery - How to Mount BTRFS Volumes
Regarding EXT4: I have tried to make the volumes EXT4, but haven't succeeded as of yet. I can create a volume that shows up alright, but it's impossible to create iSCSI containers on that volume. There might be something in the iSCSI scripts that requires the file system to be BTRFS.
Re: OS6 Data Recovery - How to Mount BTRFS Volumes
Thanks mdgm - it looks deceptively simple. I'll give it a try and put a play-by-play on here later. If it works OK, then my fears are unfounded and I'll go ahead and start using it for real data.
Re: OS6 Data Recovery - How to Mount BTRFS Volumes
If you have an array of 6 disks, do they need to be placed on the same disk device port (/dev/sdaX) as on the ReadyNAS or is mdadm smart enough to figure that out?
Since I need to do a factory reset anyway and I have 6 other disks sitting around I figure I could just move the 6 disks that are in the ReadyNAS to a PC instead of doing a backup, then once I populate the ReadyNAS with 6 new disks I could just restore from the PC using the original disks.
Re: OS6 Data Recovery - How to Mount BTRFS Volumes
mdgm wrote:
Just need an ordinary Linux PC with a reasonably up to date kernel, and to install mdadm and btrfs-progs (if they are not already installed).
Hello mdgm, just a clarification on this matter.
Assuming I have this scenario: a RN102/104 equipped with 2 HDDs or a RN312/RN314 (maybe valid also for a RN316) equipped with 4 HDDs both configured with so called - AFAIK default, "X-RAID" (and BRTFS as factory default filesystem for data) and I want/need to recover data from them because things went badly wrong.
Question (maybe silly): Have I to pull out all involved ReadyNAS HDDs and push them in into a suitable Linux box (with enough SATA ports available) respecting exact ReadyNAS HDDs install order and exact HDDs quantity (so, as example, equipping the Linux box with HDD 1 and 2 from a RN102/312 fully filled or with HDD 1, 2, 3 and 4 from a RN104/314 fully filled) so to have, on the target Linux system, new devices sdb/sdc or sdb/sdc/sde/sdf (considering sda used by my Linux distribution for its OS)?
The question above should be also valid if I have a 3 HDDs configuration (or a different one, as example, 2 HDDs on a RN104/RN314) too. Thanks!
Re: OS6 Data Recovery - How to Mount BTRFS Volumes
JMehring wrote:
If you have an array of 6 disks, do they need to be placed on the same disk device port (/dev/sdaX) as on the ReadyNAS or is mdadm smart enough to figure that out?
It shouldn't matter but you should keep the order the same where possible. It would be a good idea to note down the order e.g. note which serial number is for disk 1 etc.
JMehring wrote:
Since I need to do a factory reset anyway and I have 6 other disks sitting around I figure I could just move the 6 disks that are in the ReadyNAS to a PC instead of doing a backup, then once I populate the ReadyNAS with 6 new disks I could just restore from the PC using the original disks.
That's not recommended. If the NAS is working it would be better to do a backup.
kimera wrote:
Have I to pull out all involved ReadyNAS HDDs and push them in into a suitable Linux box (with enough SATA ports available) respecting exact ReadyNAS HDDs install order and exact HDDs quantity (so, as example, equipping the Linux box with HDD 1 and 2 from a RN102/312 fully filled or with HDD 1, 2, 3 and 4 from a RN104/314 fully filled) so to have, on the target Linux system, new devices sdb/sdc or sdb/sdc/sde/sdf (considering sda used by my Linux distribution for its OS)?
If you have four disks in the NAS and the array is fine, you would need at least three disks. If you had two disks in the array you would need at least one of them etc.
It's possible you might be able to mount the array putting the disks in e.g. USB enclosures.
I think the PC should be smart enough to tell which disks are in the array, but if not you could specify them manually.
Re: OS6 Data Recovery - How to Mount BTRFS Volumes
mdgm wrote:
If you have four disks in the NAS and the array is fine, you would need at least three disks. If you had two disks in the array you would need at least one of them etc.
Leaving out of the picture which disk? any disk of the four disks array (in case of a ReadyNAS unit filled with four disks)? isn't the "parity" spread all over the array and so the whole array should be pulled out from ReadyNAS unit to be pushed in the Linux box or the the parity is not evaluated when rebuilding the md RAID array in Linux (and so one disk could be left behind)?
Re: OS6 Data Recovery - How to Mount BTRFS Volumes
First OS6 Data Recovery Test
ReadyNAS 314 configuration
OS 6.1.6 Default configuration - one user created Drives - 4 x Seagate ST2000NV000 disks - 2 Tb (on the approved hardware list) Volume configuration - XRAID2 default - RAID 5 array for data partition (appears as MD127) Approximately 17 Gb of data copied over - Debian Linux 7.4 AMD64 DVD ISO files (five files) verified with MD5 One user created - SMB used as the file share protocol
Recovery PC configuration
OS - Debian Linux 7.4 with all updates to 19 July 2014 - default install with Desktop, File Server and Base packages only Added MDADM package using APT-GET (as described in the recommended directions above) Volumes default but with separate partition for /home OS name 3.2.0-4-amd64 Kernel 3.2.60-1+deb7u1 Motherboard Gigabyte GA-Z77X-UP5TH i5 CPU 3570 8 Gb RAM 6 SATA ports Port 0 - Plextor DVD drive Port 1 - Seagate ST3808110AS - 80 Gb SATA drive for OS install Port 2 - Drive 1 from ReadyNAS Port 3 - Drive 2 from ReadyNAS Port 4 - Drive 3 from ReadyNAS Port 5 - Drive 4 from ReadyNAS BIOS set to AHCI and "Legacy" mode (meaning non-UEFI)
Recovering the data at the command line
Followed the above guide shown above (all commands executed as root at the command line): apt-get update (updates packages and kernel to latest online version) apt-get install mdadm (RAID volume manager package) mdadm --assemble --scan (finds and connects the RAID arrays as /dev/md0, /dev/md1, /dev/md127) cat /proc/mdstat (shows the connected arrays and their properties - shows if the above command worked) mount -t btrfs -o ro /dev/md127 /mnt (mounts the data volume at /mnt and allows access)
NOTE: I found that the correct name for the btrfs utilities is actually "btrfs-tools", but I didn't install or use any of them.
Data recovery
Once the volume was mounted, I was able to move to the /mnt/Documents folder where the test ISO files were stored. The files were moved to the folder /home/<username>/recov. The files were checked again by generating an MD5 sum and checking it against the official MD5 sum in the distro. The MD5 signature matched showing that the data was not corrupted. This was only 17 Gb of test data, but it was just a proof of concept.
Final Thoughts
This was a lot easier than I thought it would be. One important thing is to use a recent distro with kernel 3.x and BTRFS support. The 7 series of Debian Linux has BTRFS support built in and appears to be able to recognize XRAID2 RAID5 volumes and mount them without special settings. There is a package called "btrfs-tools" with testing and repair tools, but I didn't use them at all. It's easier if the motherboard has at least 5 or 6 SATA ports, but it could be done with just 4 if you would be willing to boot to a USB stick or use a net boot.
Please post here if you have questions or need more information. I'm kind of a Linux newbie, but I'll help as much as I can.
CAUTION: FlexRAID with RAID 6 DOES NOT WORK USING THIS METHOD
After the above test, I changed the configuration on the ReadyNAS to FlexRAID with all the disks in a RAID 6 array. This erased all the data, but no other configuration changes were made. I set up an SMB share named "test" and copied the exact same data to the folder and verified the MD5 sums. I pulled the drives and tried to recover the data use the exact same configuration and attempted to mount the volumes, but it didn't work using the method shown above. I'm still testing, but there might be some issues with RAID 6. FlexRAID seems to change the configuration to one RAID 1 OS volume and one RAID 6 data volume. The OS volume mounts using the above method, but the data array won't mount up. I'll do some more testing and try to fix it. If I can't get it to work, I'll update to Debian 7.6.x to see if the updated version might have better support.
If anyone has any ideas about mounting this, please let me know. The error message says that it can't identify the type of the file system and thinks it might be a swap file. Auto type doesn't work and setting the type to BTRFS fails with that error message.
More later when I have more time. Sorry for the long delay in following up on the original post, but I'm working crazy hours these days.
Re: OS6 Data Recovery - How to Mount BTRFS Volumes
Thanks for your help on this mgdm.
First, one change from my previous test - I downloaded and clean installed Debian 7.6.0. Default install with the /home directory in a separate partition. This time, after installing mdadm (apt-get install mdadm) and btrfs-tools (apt get install btrfs-tools), it appears to auto assemble the raid array during startup without any other commands.
When I do the command cat /proc/mdstat what I see is the following:
Personalities : [raid6] [raid5] [raid4] [raid1] md125 : active (auto-read-only) raid1 sdb1[0] sdd1[2] sdc1[1] 4192192 blocks super 1.2 [4/3] [UUU_]
md126 : active (auto-read-only) raid6 sdb2[0] sdd2[2] sdc2[1] 1047936 blocks super 1.2 level 6, 64k chunk, algorithm 2 [4/3] [UUU_]
md127 : active (auto-read-only) raid6 sdb3[0] sdd3[2] sdc3[1] 3897329408 blocks super 1.2 level 6, 64k chunk, algorithm 2 [4/3] [UUU_]
unused devices : <none>
When I do the command btrfs fi label /dev/md127 I get the following:
5e276850:TESTFLX6
The name "TESTFLX6" is the name I gave to the RAID 6 data volume.
When I follow your instructions for mounting the partition, using these commands:
mdadm --assemble --scan (response) mdadm: No arrays found in config file or automatically
cat /proc/mdstat (response) As shown above
mount -t btrfs -o ro /dev/md127 /mnt (response) mount: wrong fs type, bad option, bad superblock on /dev/md127, missing codepage or helper program, or other error. In some cases useful info is found in syslog - try dmesg | tail or so
When I put in dmesg | tail I get the following:
[ 1305.427980] device label 5e276850:TESTFLX6 devid 1 transid 34 /dev/md127 [ 1305.428326] btrfs: disk space caching is enabled [ 1305.428329] BTRFS: couldn't mount because of unsupported optional features (20). [ 1305.459491] btrfs: open_ctree failed
This is where I'm stuck. Any ideas on troubleshooting this?
Re: OS6 Data Recovery - How to Mount BTRFS Volumes
I've already updated to the latest version of Debian (7.6.0) and run the updates. I'm not sure how to update any further.
Also, this has the latest btrfs-tools package in the Debian distro (from the standard respository). Would some of the unstable newer builds be better? Or newer, beta tools?
Until I can find a way to mount these disks, I can't trust this NAS. If the hardware failed, the data would be un-recoverable. Supposedly this format follows the Linux (Debian, at least) standard, but if it won't mount, then it's a hardware/proprietary software lock-in.
Are there any other outside open-source packages to troubleshoot/test/mount the array? Would another distro (Ubuntu, CentOS, etc) work better? I stuck with Debian Linux for testing since that's the core used in ReadyNAS OS6 and I figured that it would be the most compatible.
The kernel for the test recovery system (Debian Linux 3.2.0-4-amd64) is named Debian 3.2.60-1+deb7u1 x56_64. I'm not sure what version of the tools is installed.
Re: OS6 Data Recovery - How to Mount BTRFS Volumes
To begin with, I'd like to say that you are not paranoid, you just try to prevent problems before they even have a chance to happen, that's pretty uncommon (most of the time it's already too late) and that's a good thing that you do not do this while your data is at risk.
As stated, you will have hardware replacement for the duration of the warranty (5years) and may be able to buy spare parts at a cost for some time after. You would also be able to be assisted as I think the 314 has a 24/7 lifetime assistance : http://www.netgear.com/about/press-rele ... ranty.aspx
If that's not enough, you would still be able to recover from secondary and tertiary backups. Although not ideal as it may take time to recover from tapes or off-site backup having three backups failing at the same time is pretty rare and will give you time to find a solution. The more you wait, the more BTRFS gets close to get full support in every linux distribution and recovery tool, so your actual problems may solve themselves with some time. I think Fedora was one of the first to be able to use BTRFS as a root filesystem, maybe you would have better out of the box support with it as it is known to be less conservative than Debian.
Re: OS6 Data Recovery - How to Mount BTRFS Volumes
I tested the mounting procedure detailed in the previous post on the two disks Array grabbed from my ReadyNAS RN102 (assembled with two Toshiba DT01ACA100) which is running ReadyOS 6.1.9 RC8.
I used a Fedora 20 x64 XFCE Live Image not yum updated (it's exactly the 03.07.2014 respin, so quite updated yet) and below my findings:
[root@localhost ~]# mount -l|grep md127 /dev/md127 on /mnt type btrfs (ro,relatime,seclabel,space_cache) [0e35a911:data]
Few dmseg grep messages about BTRFS and md:
Jul 30 04:03:57 localhost kernel: [ 719.673597] BTRFS: device label 0e35a911:data devid 1 transid 76 /dev/md127 Jul 30 04:03:57 localhost kernel: BTRFS: device label 0e35a911:data devid 1 transid 76 /dev/md127 Jul 30 04:03:57 localhost kernel: BTRFS info (device md127): disk space caching is enabled Jul 30 04:03:57 localhost kernel: [ 719.674570] BTRFS info (device md127): disk space caching is enabled
[root@localhost Documents]# dmesg|grep md [ 56.422296] md: bind<sda3> [ 56.432502] md: bind<sdb3> [ 58.602410] md: raid1 personality registered for level 1 [ 58.603526] md/raid1:md127: active with 2 out of 2 mirrors [ 58.603650] created bitmap (8 pages) for device md127 [ 58.603862] md127: bitmap initialized from disk: read 1 pages, set 0 of 14831 bits [ 58.629099] md127: detected capacity change from 0 to 995236642816 [ 58.629884] md: bind<sda1> [ 58.632111] md: bind<sda2> [ 58.640096] md: bind<sdb2> [ 58.643333] md/raid1:md125: active with 2 out of 2 mirrors [ 58.643368] md125: detected capacity change from 0 to 536281088 [ 58.653856] md127: unknown partition table [ 58.669308] md125: unknown partition table [ 58.678096] md: bind<sdb1> [ 58.680007] md/raid1:md126: active with 2 out of 2 mirrors [ 58.680053] md126: detected capacity change from 0 to 4290772992 [ 58.686240] md126: unknown partition table [ 63.132849] systemd-journald[632]: Received request to flush runtime journal from PID 1 [ 73.919360] BTRFS: device label 0e35a911:data devid 1 transid 76 /dev/md127 [ 74.285276] Adding 523708k swap on /dev/md125. Priority:-1 extents:1 across:523708k FS [ 719.673597] BTRFS: device label 0e35a911:data devid 1 transid 76 /dev/md127 [ 719.674570] BTRFS info (device md127): disk space caching is enabled [ 719.714024] SELinux: initialized (dev md127, type btrfs), uses xattr
Status of /dev/mdx (x=127, 126 and 125) in detail:
[root@localhost ~]# mdadm --detail /dev/md127 /dev/md127: Version : 1.2 Creation Time : Wed Jul 16 11:03:50 2014 Raid Level : raid1 Array Size : 971910784 (926.89 GiB 995.24 GB) Used Dev Size : 971910784 (926.89 GiB 995.24 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Jul 30 04:02:48 2014 State : active Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0
Name : 0e35a911:data-0 UUID : 1617bc13:e5576688:64018e5b:0ca61008 Events : 1759
Number Major Minor RaidDevice State 0 8 19 0 active sync /dev/sdb3 1 8 3 1 active sync /dev/sda3 [root@localhost ~]# mdadm --detail /dev/md126 /dev/md126: Version : 1.2 Creation Time : Wed Jul 16 11:03:50 2014 Raid Level : raid1 Array Size : 4190208 (4.00 GiB 4.29 GB) Used Dev Size : 4190208 (4.00 GiB 4.29 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent
Update Time : Mon Jul 28 11:46:35 2014 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0
Name : 0e35a911:0 UUID : 7062aefd:dcbae9c7:1dd84684:e7744d65 Events : 23
Number Major Minor RaidDevice State 0 8 17 0 active sync /dev/sdb1 1 8 1 1 active sync /dev/sda1 [root@localhost ~]# mdadm --detail /dev/md125 /dev/md125: Version : 1.2 Creation Time : Wed Jul 16 11:03:50 2014 Raid Level : raid1 Array Size : 523712 (511.52 MiB 536.28 MB) Used Dev Size : 523712 (511.52 MiB 536.28 MB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent
Update Time : Mon Jul 21 10:11:11 2014 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0
Name : 0e35a911:1 UUID : 767a9300:bd522aac:366459ee:712f6896 Events : 19
Number Major Minor RaidDevice State 0 8 18 0 active sync /dev/sdb2 1 8 2 1 active sync /dev/sda2
After trying I'm now quite confident about BTRFS data recovery option using an (updated) Linux box: the only test we should do now could be to test a degraded array.
Re: OS6 Data Recovery - How to Mount BTRFS Volumes
Nice guide. Should still work on a degraded array provided only one disk is bad. Of course if using RAID-6 you could have a few bad disks and still be O.K.
Re: OS6 Data Recovery - How to Mount BTRFS Volumes
Per mdgm above, the key appears to be getting an updated kernel and set of BTRFS tools. I switched to Fedora Linux v20 x86_64 with kernel 3.15 and that works without modification.
Second OS6 Data Recovery Test
ReadyNAS 314 configuration
OS 6.1.6 Default configuration - one user created Drives - 4 x Seagate ST2000NV000 disks - 2 Tb (on the approved hardware list) Volume configuration - FlexRAID in RAID 6 array for data partition (appears as MD127) Approximately 17 Gb of data copied over - Debian Linux 7.4 AMD64 DVD ISO files (five files) verified with SHA1 One user created - SMB used as the file share protocol
Recovery PC configuration
OS - Fedora Linux 20 x86_64 - default install with customized partitioning - BTRFS selected as the format, default partitions (/home on the same partition as root) Ran "Yum update" from the command line after install - Many packages (and the kernel) updated up to 2 Aug 2014. OS name Fedora 20 Kernel 3.15.7-200.fc20.x86_64 SMP BTRFS v3.14.2 Motherboard Gigabyte GA-Z77X-UP5TH i5 CPU 3570 8 Gb RAM 6 SATA ports Port 0 - Plextor DVD drive Port 1 - Seagate ST3808110AS - 80 Gb SATA drive for OS install Port 2 - Drive 1 from ReadyNAS Port 3 - Drive 2 from ReadyNAS Port 4 - Drive 3 from ReadyNAS Port 5 - Drive 4 from ReadyNAS BIOS set to AHCI and "Legacy" mode (meaning non-UEFI)
Recovering the data at the command line
Followed the recommended commands from the earlier thread entry (all commands executed as root at the command line):
yum update (updates packages and kernel to latest online version)
*** NOTE: No need to install mdadm or btrfs-progs - since BTRFS was chosen as the install volume format, these were already in place. Running the mdadm --assemble --scan command was also not needed - the system found the array at boot up automatically.
cat /proc/mdstat (shows the connected arrays and their properties)
mount -t btrfs -o ro /dev/md127 /mnt (mounts the data volume at /mnt and allows access)
NOTE: On Fedora, the tools are actually named "btrfs-progs" and the version was "btrfs-progs 3.14.2-3.fc20.x86_64"
Data recovery
Once the volume was mounted, I was able to move to the /mnt/Documents folder where the test ISO files were stored. The files were moved to the folder /home/<username>/recov. The files were checked again by generating an SHA1 sum and checking it against the official SHA1 sum in the distro. The SHA1 signature matched showing that the data was not corrupted. This was only 17 Gb of test data, but it was just a proof of concept.
Extra Test
There was a question up-thread about whether the mounting order of the hard disks makes a difference, so I decided to test that (since everything was ready anyway).
Here's what I found:
Out of Order Test 1
Motherboard SATA 2 - ReadyNAS Disk 3 Motherboard SATA 3 - ReadyNAS Disk 1 Motherboard SATA 4 - ReadyNAS Disk 4 Motherboard SATA 5 - ReadyNAS Disk 2
Data volume appears as "md126" this time when cat /proc/mdstat is run. It is mounted the same way:
mount -t btrfs -o ro /dev/md126 /mnt
Some sample SHA1 sums verified correctly.
Out of Order Test 2
Motherboard SATA 2 - ReadyNAS Disk 4 Motherboard SATA 3 - ReadyNAS Disk 1 Motherboard SATA 4 - ReadyNAS Disk 3 Motherboard SATA 5 - ReadyNAS Disk 2
Data volume appears as "md127" this time when cat /proc/mdstat is run. Mounted the same way as above and again the SHA1 sums verified correctly.
Can someone with more knowlege please explain the components of the cat /proc/mdstat output? It would be nice to know what I'm looking at.
From the "Out of Order Test 2", I see the following:
I mounted up md125, and that appears to be the OS6 partition, with no data available. Just the OS files but this might be crucial for troubleshooting - the log files would be available for analysis. The partition md126 would not mount - not sure what that is, maybe the swap file?
Why do the md names change? Sometimes the data volume is "md127" and sometimes it's "md126".
Just out of curiousity, I thought I'd do some "less than ideal" data recovery tests.
Single Hard Disk Missing from Array Test 1
For this test, I put the drive cables back in the first test configuration:
Mb SATA 2 - RN Disk 1 Mb SATA 3 - RN Disk 2 Mb SATA 4 - RN Disk 3 Mb SATA 5 - RN Disk 4
I tested to make sure the data was available (strangely, this time the data was on "md126"). It was accessible.
Then I shut down the machine and disconnected the Disk 1 data cable (simulated failed drive) and started up again.
This time the /proc/mdstat command gives this result:
Personalities : md125 : inactive sdc1[2] (S) sdd1[3] (S) sdb1[1] (S) 12576768 blocks super 1.2
md126 : inactive sdd3[3] (S) sdc3[2] (S) sdb3[1] (S) 5845994613 blocks super 1.2
md127 : inactive sdb2[1] (S) sdd2[3] (S) sdc2[2] (S) 1572096 blocks super 1.2
unused devices: <none>
I poked around with mdadm, but other than getting a little information, I'm not sure how to rebuild this array or force it to mount.
I'm open to ideas on how to get this array to be visible and accessible with Disk 1 missing. What's the best way to force the array to be active and mount up with a disk missing? I'd like to test the scenario where a spare disk is not available, so the array needs to mount up using 3 disks.
Re: OS6 Data Recovery - How to Mount BTRFS Volumes
The package is called btrfs-progs (I think I mistakenly referred to it as something else earlier).
The 4GB partition is the OS partition, the smaller one is for swap and the large one is for your data volume. If you vertically expand your array or create additional volumes there will be additional md devices.
Re: OS6 Data Recovery - How to Mount BTRFS Volumes
Attempting to recover a RAID 6 volume with a missing hard disk
To attempt this recovery, the configuration is the same as listed in the above entry. I just physically unplugged the ReadyNAS drive number 1 (on the motherboard SATA connector 2) to simulate a completely failed drive.
I have been attempting to use mdadm to assemble the array in a degraded mode with just the remaining 3 hard disks. So far, no luck, but here's what I found.
First, the information about the array as I collected previously:
cat /proc/mdstat Personalities: md125 : inactive sdd1[3] (S) sdc1[2] (S) sdb1[1] (S) 12576768 blocks super 1.2
md126 : inactive sdb2[2] (S) sdd2[3] (S) sdc2[2] (S) 1572096 blocks super 1.2
md127 : inactive sdd3[3] (S) sdb3[1] (S) sdc3[2] (S) 5845994613 blocks super 1.2
unused devices: <none>
Here is the result of information gathering about individual drives separately:
mdadm --examine /dev/sdd3 /dev/sdd3: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : be4dd378:31e311ab:9af53399:976b5245 Name : 5e276850:TESTFLX6-0 Creation Time : Sun Jul 20 07:45:00 2014 Raid Level : raid6 Raid Devices : 4
Avail Dev Size : 3897329742 (1858.39 GiB 1995.43 GB) Array Size : 3897329408 (3716.78 GiB 3990.87 GB) Used Dev Size : 3897329408 (1858.39 GiB 1995.43 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=334 sectors State : clean Device UUID : fa36d5e1:76f3276a:e10be6e3:535a21bb
After investigating how to force the array to assemble, I tried this:
mdadm --create md127 --chunk=64 --level=raid6 --raid-devices=4 --run /dev/sdd3 /dev/sdc3 /dev/sdb3 missing --verbose mdadm: layout defaults to left-symmetric mdadm: layout defaults to left-symmetric mdadm: super1.x cannot open /dev/sdd3: Device or resource busy mdmon: ddf: Cannot use /dev/sdd3: Device or resource busy mdmon: Cannot use /dev/sdd3: It is busy mdadm: cannot open /dev/sdd3: Device or resource busy
Any ideas about getting this to assemble and mount up as a degraded array with one disk missing? :shock:
Since RAID 6 should allow an array to be available with up to 2 disks missing, the array should be able to mount with just one missing. The mdadm man information says to use the word "missing" in place of the device that is missing and use the command "--run" to force the assembly of the incomplete array. I tried several variants of the commands including the following (but all state the device is busy):
mdadm -- assemble --scan --run (FAILED - mdadm: No arrays found in config file or automatically)
The command shown in this article to eliminate the "device busy" error is "mdadm --stop /dev/mdxxx" in order to release the drives. Apparently the OS picks up the drives automatically during boot up and they're held in "busy" status until the stop command is issued.
How to Mount Degraded BTRFS Volumes - RAID 6 - One Hard Disk Missing
Just a quick reminder - the hardware and software set up is the same as the entry above. A FlexRAID RAID 6 array on 4 disks, each disk is 2 Tb and the recovery OS is Fedora Linux 20 updated to the 3.15 kernel.
Here is the exact method to mount an array with one hard disk failed (ReadyNAS drive 1 disconnected from motherboard SATA connector 2 to simulate drive failure):
cat /proc/mdstat Personalities: md125 : inactive sdd1[3] (S) sdc1[2] (S) sdb1[1] (S) 12576768 blocks super 1.2
md126 : inactive sdb2[1] (S) sdd2[3] (S) sdc2[2] (S) 1572096 blocks super 1.2
md127 : inactive sdd3[3] (S) sdb3[1] (S) sdc3[2] (S) 5845994613 blocks super 1.2
mdadm --assemble md127 --uuid=be4dd378:31e311ab:9af53399:976b5245 --run mdadm: /dev/md/md127 has been started with 3 drives (out of 4).
mount -t btrfs -o ro /dev/md127 /mnt
cd /mnt ls home test cd test ls debian-7.4.0-amd64-DVD-1.iso <snip>
It is now possible to use the cp command to copy the files from this directory (/mnt/test) but the directory name will depend on the configuration on your particular ReadyNAS. I copied the test files (5 Debian DVD ISO files - about 17 Gb) over to the home directory and then ran the SHA1SUM utility and verified them. They all passed.
Note the following:
1) I don't think this is data destructive since the volume is mounted read-only and this is just requesting an existing volume be assembled (not creating or changing anything). Maybe people with more Linux experience can explain more.
2) The metadata (ie. superblocks) may change once the volume is assembled - I don't know what that means in terms of data risk. (ie. timestamps?)
3) I ran SHA1 sums on the ISO files copied from the degraded volume and the copies verified correctly (no data corruption even from a degraded volume).
4) The mdadm utility will give an error (see above) and I had desktop popups on Fedora stating that the volume was degraded. But the data was accessible.
5) When using the "mdadm --examine" command, the device can be any of the partitions that are part of the volume you are trying to access. In this case, the md127 volume is the data volume and is made up of the 3 remaining partitions (sdd3, sdc3 and sdb3). To get the UUID to identify the array, the command "mdadm --examine /dev/sdd3 --scan" can be used, but any of the partitions can also be used. The "--scan" option will give the ARRAY information from any of the component partitions.
6) Any of the other volumes can be mounted the same way (except the swap volume). This might be useful - the OS volume could be mounted in order to check the ReadyNAS OS log files to find out what problems caused a crash.
For more details, a good page explaining the mdadm utility command line options and syntax:
http://linux.die.net/man/8/mdadm
Overall, this appears to be a workable solution for data recovery for RAID 6 with a missing (failed) hard disk.
Now for hard-core data recovery: 2 disks missing! 😄😄😄
