OS6 Data Recovery - How to Mount BTRFS Volumes

Found the answer!

After some serious googling and reading tons of related and unrelated articles, I found one that was relevant:

https://unix.stackexchange.com/questions/26489/mdadm-how-to-reassemble-raid-5-reporting-device-or-resource-busy

The command shown in this article to eliminate the "device busy" error is "mdadm --stop /dev/mdxxx" in order to release the drives. Apparently the OS picks up the drives automatically during boot up and they're held in "busy" status until the stop command is issued.

How to Mount Degraded BTRFS Volumes - RAID 6 - One Hard Disk Missing

Just a quick reminder - the hardware and software set up is the same as the entry above. A FlexRAID RAID 6 array on 4 disks, each disk is 2 Tb and the recovery OS is Fedora Linux 20 updated to the 3.15 kernel.

Here is the exact method to mount an array with one hard disk failed (ReadyNAS drive 1 disconnected from motherboard SATA connector 2 to simulate drive failure):

cat /proc/mdstat
Personalities:
md125 : inactive sdd1[3] (S) sdc1[2] (S) sdb1[1] (S)
            12576768 blocks super 1.2

md126 : inactive sdb2[1] (S) sdd2[3] (S) sdc2[2] (S)
             1572096 blocks super 1.2

md127 : inactive sdd3[3] (S) sdb3[1] (S) sdc3[2] (S)
             5845994613 blocks super 1.2

unused devices: <none>

mdadm --stop /dev/md125
mdadm: stopped /dev/md125

mdadm --stop /dev/md126
mdadm: stopped /dev/md126

mdadm --stop /dev/md127
mdadm: stopped /dev/md127

cat /proc/mdstat
Personalities:
unused devices: <none>

mdadm --examine /dev/sdd3 --scan
ARRAY /dev/md/TESTFLX6-0 metadata=1.2 UUID=be4dd378:31e311ab:9af53399:976b5245 name=5e276850:TESTFLX6-0

mdadm --assemble md127 --uuid=be4dd378:31e311ab:9af53399:976b5245 --run
mdadm: /dev/md/md127 has been started with 3 drives (out of 4).

mount -t btrfs -o ro /dev/md127 /mnt

cd /mnt
ls
home test
cd test
ls
debian-7.4.0-amd64-DVD-1.iso
<snip>

It is now possible to use the cp command to copy the files from this directory (/mnt/test) but the directory name will depend on the configuration on your particular ReadyNAS. I copied the test files (5 Debian DVD ISO files - about 17 Gb) over to the home directory and then ran the SHA1SUM utility and verified them. They all passed.

Note the following:

1) I don't think this is data destructive since the volume is mounted read-only and this is just requesting an existing volume be assembled (not creating or changing anything). Maybe people with more Linux experience can explain more.

2) The metadata (ie. superblocks) may change once the volume is assembled - I don't know what that means in terms of data risk. (ie. timestamps?)

3) I ran SHA1 sums on the ISO files copied from the degraded volume and the copies verified correctly (no data corruption even from a degraded volume).

4) The mdadm utility will give an error (see above) and I had desktop popups on Fedora stating that the volume was degraded. But the data was accessible.

5) When using the "mdadm --examine" command, the device can be any of the partitions that are part of the volume you are trying to access. In this case, the md127 volume is the data volume and is made up of the 3 remaining partitions (sdd3, sdc3 and sdb3). To get the UUID to identify the array, the command "mdadm --examine /dev/sdd3 --scan" can be used, but any of the partitions can also be used. The "--scan" option will give the ARRAY information from any of the component partitions.

6) Any of the other volumes can be mounted the same way (except the swap volume). This might be useful - the OS volume could be mounted in order to check the ReadyNAS OS log files to find out what problems caused a crash.

For more details, a good page explaining the mdadm utility command line options and syntax:

http://linux.die.net/man/8/mdadm

Overall, this appears to be a workable solution for data recovery for RAID 6 with a missing (failed) hard disk.

Now for hard-core data recovery: 2 disks missing! :D :D :D

-Stan