NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
OS6 Data Recovery - How to Mount BTRFS Volumes
I recently purchased a ReadyNAS 314 and 5 Seagate ST2000VN000 hard disks (one as a cold spare). I work as a system administrator, so I've been reading up on OS6 before I entrust this new system with my data. I'm not very comfortable using BTRFS since it seems to still be a little buggy and I don't need snapshots. But since that's the direction Netgear chose to go, I'd like to give it a fair chance.
The older generation of devices had several excellent web sites with detailed data recovery instructions for situations involving hardware failures. Usually, this involved removing the hard disks and connecting them to a desktop, then booting into a version of Linux and mounting the array using the familiar ext4/LVM tools. I've been searching through the forums and Googling for an updated version, but there don't seem to be any recovery instructions for the new generation. I've seen a lot of discussion about BTRFS including some comments that make me quite concerned.
As mangrove stated in thread 419088
"It should also be said that for EXT solutions, there are restore tools available which you run from a computer. So if the unit fails and you need to get data back, you can without having to buy an identical unit. This is much harder though.
This is impossible with OS6 units as BTRFS is not well supported by restore software."
I'm sure this is paranoid of me, but before I start to trust this device with 4 Tb of data, which is time consuming and difficult to back up in multiple places, I need to know I can access the data if the proprietary hardware fails. The situation I'm thinking of is the ReadyNAS works fine for about 6 years, then the mainboard fails. It's out of warranty, there are no replacement parts and the newer models can't read the volumes due to data structure changes. The disks are fine, the data is there, but it can't be accessed without mounting the disks on a different machine.
Option 1 - Test data recovery using the current OS 6.1.6 build and a spare desktop. Set up the array in the normal way using OS6 tools, save some test data on it, then shut it down and take the disks out. Figure out how to mount the disks and access the volume by connecting them to the desktop and installing the appropriate Linux/kernel/disk tools on a separate disk. Once this is working, create a small Linux build on a thumb drive that is pre-configured to mount up the disks properly. My preferred configuration would be Flex-RAID set up in RAID6. But I'll test Flex-RAID in RAID5 and XRaid2 if I have time.
If that can be done, then I'll go ahead and use the system and just keep an updated version of the recovery thumb drive handy (updated to match each new build of OS6).
I'm not here on the forum to ask someone to do this for me. Since I happen to have a new ReadyNAS 314 with blank hard disks and a spare desktop sitting around, I'm happy to roll up my sleeves and test it myself. I'm not a Linux guru, but the command line doesn't scare me. And at this point, I'm not risking any real data and this will allow me to have my recovery solution already built and ready to go. I'll post the results here in the forum for others, since there doesn't seem to be a definitive solution out there (or if someone can point me to one that already exists? Thanks! I can try that out first and save time!)
What I'm here to ask for, since there are so many very experienced ReadyNAS Jedis, is for some background on the data structure so I can get started. What I need to know is the following:
The older generation of devices had several excellent web sites with detailed data recovery instructions for situations involving hardware failures. Usually, this involved removing the hard disks and connecting them to a desktop, then booting into a version of Linux and mounting the array using the familiar ext4/LVM tools. I've been searching through the forums and Googling for an updated version, but there don't seem to be any recovery instructions for the new generation. I've seen a lot of discussion about BTRFS including some comments that make me quite concerned.
As mangrove stated in thread 419088
"It should also be said that for EXT solutions, there are restore tools available which you run from a computer. So if the unit fails and you need to get data back, you can without having to buy an identical unit. This is much harder though.
This is impossible with OS6 units as BTRFS is not well supported by restore software."
I'm sure this is paranoid of me, but before I start to trust this device with 4 Tb of data, which is time consuming and difficult to back up in multiple places, I need to know I can access the data if the proprietary hardware fails. The situation I'm thinking of is the ReadyNAS works fine for about 6 years, then the mainboard fails. It's out of warranty, there are no replacement parts and the newer models can't read the volumes due to data structure changes. The disks are fine, the data is there, but it can't be accessed without mounting the disks on a different machine.
Option 1 - Test data recovery using the current OS 6.1.6 build and a spare desktop. Set up the array in the normal way using OS6 tools, save some test data on it, then shut it down and take the disks out. Figure out how to mount the disks and access the volume by connecting them to the desktop and installing the appropriate Linux/kernel/disk tools on a separate disk. Once this is working, create a small Linux build on a thumb drive that is pre-configured to mount up the disks properly. My preferred configuration would be Flex-RAID set up in RAID6. But I'll test Flex-RAID in RAID5 and XRaid2 if I have time.
If that can be done, then I'll go ahead and use the system and just keep an updated version of the recovery thumb drive handy (updated to match each new build of OS6).
I'm not here on the forum to ask someone to do this for me. Since I happen to have a new ReadyNAS 314 with blank hard disks and a spare desktop sitting around, I'm happy to roll up my sleeves and test it myself. I'm not a Linux guru, but the command line doesn't scare me. And at this point, I'm not risking any real data and this will allow me to have my recovery solution already built and ready to go. I'll post the results here in the forum for others, since there doesn't seem to be a definitive solution out there (or if someone can point me to one that already exists? Thanks! I can try that out first and save time!)
What I'm here to ask for, since there are so many very experienced ReadyNAS Jedis, is for some background on the data structure so I can get started. What I need to know is the following:
- 1 Which OS would be best to access the data? It appears that Debian is the core of OS6, but which build/kernel should be used?
[list=2:f5h9ejz4]2 Which tools are needed for testing/repairing/mounting the BTRFS filesystem?
[list=3:f5h9ejz4]3 A general overview of how the volumes are arranged (ie. it's not LVM anymore, so what is it?)
[list=4:f5h9ejz4]4 Specific settings to be aware of that might not be standard in the vanilla Linux configuration (ie. block sizes? other things I don't know about at all)
[list=5:f5h9ejz4]5 Gotchas or special hazards to watch out for when working with BTRFS. I'm really not familiar with it.
[list=6:f5h9ejz4]6 Which log files show success or failure and which command-line commands can test the volume and data.
This doesn't have to be super detailed or line-by-line. Just let me know the basics and I can look up the syntax and details in the man pages or online. I'm sure it'll blow up on me at least the first few times and I'll be back on this thread with the output and more questions when I get stuck. :shock:
Option 2 - Work on a way to change the file structure to the old-style EXT4/LVM so the older generation recovery process works. Yes, I understand that this is not supported and would have to be painfully redone after every OS version upgrade, but it might be a (tedious) solution.
Just a quick note on what I'm planning to do with this unit - I just need to use iSCSI to connect it to a CentOS 6 virtual server running on VMware. That server will provide all the file management, permissions and network services. I just need OS6 to provide iSCSI connectivity, basic hardware monitoring and UPS-triggered safe shutdown (I think the APC Network Shutdown 3.1 Linux software will work - I'll test that also). The primary role for the ReadyNAS will be to provide centralized backup to another NAS and various servers. Some of the servers will backup and store data from the desktops that will also be backed up to the ReadyNAS.
I know this probably sounds like belt, suspenders and a flak jacket, but data integrity is a big deal with me. I'm hoping that what I find out will be useful to other people if they have a data access problems (or system meltdown). Plus, since system administration is my day job, this is kind of a scale model of my work system and should be good training as well (score!) :D
Thanks in advance to all the ReadyNAS experts out there for your time and assistance. I know I'm asking for a lot, but I'll share what I find out in return. Please be patient - I have a little Linux experience, but mostly at the power-user level so I'm weak in some of the admin areas. (Yeah, yeah, I do Windows system administration :( - stop snickering)
Stan
33 Replies
Replies have been turned off for this discussion
- Oops. Yes, you're right. It is btrfs-tools.
What do you see when you do a
# cat /proc/mdstat
when using RAID-6?
Are you entering the correct path.
/dev/md0 is for the OS partition
/dev/md1 is for swap
Then e.g. /dev/md127 is what the data volume is on.
If you do the flowing what do you get?:
# btrfs fi label /dev/md127 - Thanks for your help on this mgdm.
First, one change from my previous test - I downloaded and clean installed Debian 7.6.0. Default install with the /home directory in a separate partition. This time, after installing mdadm (apt-get install mdadm) and btrfs-tools (apt get install btrfs-tools), it appears to auto assemble the raid array during startup without any other commands.
When I do the command cat /proc/mdstat what I see is the following:
Personalities : [raid6] [raid5] [raid4] [raid1]
md125 : active (auto-read-only) raid1 sdb1[0] sdd1[2] sdc1[1]
4192192 blocks super 1.2 [4/3] [UUU_]
md126 : active (auto-read-only) raid6 sdb2[0] sdd2[2] sdc2[1]
1047936 blocks super 1.2 level 6, 64k chunk, algorithm 2 [4/3] [UUU_]
md127 : active (auto-read-only) raid6 sdb3[0] sdd3[2] sdc3[1]
3897329408 blocks super 1.2 level 6, 64k chunk, algorithm 2 [4/3] [UUU_]
unused devices : <none>
When I do the command btrfs fi label /dev/md127 I get the following:
5e276850:TESTFLX6
The name "TESTFLX6" is the name I gave to the RAID 6 data volume.
When I follow your instructions for mounting the partition, using these commands:
mdadm --assemble --scan
cat /proc/mdstat
mount -t btrfs -o ro /dev/md127 /mnt
I get the following:
mdadm --assemble --scan
(response) mdadm: No arrays found in config file or automatically
cat /proc/mdstat
(response) As shown above
mount -t btrfs -o ro /dev/md127 /mnt
(response) mount: wrong fs type, bad option, bad superblock on /dev/md127, missing codepage or helper program, or other error. In some cases useful info is found in syslog - try dmesg | tail or so
When I put in dmesg | tail I get the following:
[ 1305.427980] device label 5e276850:TESTFLX6 devid 1 transid 34 /dev/md127
[ 1305.428326] btrfs: disk space caching is enabled
[ 1305.428329] BTRFS: couldn't mount because of unsupported optional features (20).
[ 1305.459491] btrfs: open_ctree failed
This is where I'm stuck. Any ideas on troubleshooting this?
Stan - You may need a newer kernel and/or btrfs-tools.
Although the ReadyNAS uses kernel 3.0.x there is a lot backported from newer kernels. btrfs-tools is regularly updated.
You could try updating your Debian install to a newer kernel and also updating to the latest btrfs-tools and see if this helps.
In a data recovery situation you would want to do this before connecting the disks from the array. - I've already updated to the latest version of Debian (7.6.0) and run the updates. I'm not sure how to update any further.
Also, this has the latest btrfs-tools package in the Debian distro (from the standard respository). Would some of the unstable newer builds be better? Or newer, beta tools?
Until I can find a way to mount these disks, I can't trust this NAS. If the hardware failed, the data would be un-recoverable. Supposedly this format follows the Linux (Debian, at least) standard, but if it won't mount, then it's a hardware/proprietary software lock-in.
Are there any other outside open-source packages to troubleshoot/test/mount the array? Would another distro (Ubuntu, CentOS, etc) work better? I stuck with Debian Linux for testing since that's the core used in ReadyNAS OS6 and I figured that it would be the most compatible.
The kernel for the test recovery system (Debian Linux 3.2.0-4-amd64) is named Debian 3.2.60-1+deb7u1 x56_64. I'm not sure what version of the tools is installed.
Any further tests I can try?
Thanks,
Stan - If the NAS hardware failed under warranty you should be able to get it replaced under RMA.
That kernel version is too old. You need to update to a newer kernel or manually backport a whole lot of patches.
See https://wiki.debian.org/HowToUpgradeKernel
I would suggest trying at least kernel 3.14
You may also need to install a newer version of btrfs-tools.
What output do you get if you run
# btrfs version - To begin with, I'd like to say that you are not paranoid, you just try to prevent problems before they even have a chance to happen, that's pretty uncommon (most of the time it's already too late) and that's a good thing that you do not do this while your data is at risk.
As stated, you will have hardware replacement for the duration of the warranty (5years) and may be able to buy spare parts at a cost for some time after.
You would also be able to be assisted as I think the 314 has a 24/7 lifetime assistance : http://www.netgear.com/about/press-rele ... ranty.aspx
If that's not enough, you would still be able to recover from secondary and tertiary backups. Although not ideal as it may take time to recover from tapes or off-site backup having three backups failing at the same time is pretty rare and will give you time to find a solution. The more you wait, the more BTRFS gets close to get full support in every linux distribution and recovery tool, so your actual problems may solve themselves with some time. I think Fedora was one of the first to be able to use BTRFS as a root filesystem, maybe you would have better out of the box support with it as it is known to be less conservative than Debian. - I don't think we usually sell spare parts.
Stanman130 purchased his unit before that change came in. There is still of course online support at my.netgear.com - I tested the mounting procedure detailed in the previous post on the two disks Array grabbed from my ReadyNAS RN102 (assembled with two Toshiba DT01ACA100) which is running ReadyOS 6.1.9 RC8.
I used a Fedora 20 x64 XFCE Live Image not yum updated (it's exactly the 03.07.2014 respin, so quite updated yet) and below my findings:
Software components:
[root@localhost ~]# rpm -q btrfs-progs
btrfs-progs-3.14.2-3.fc20.x86_64
[root@localhost ~]# btrfs version
Btrfs v3.14.2
[root@localhost ~]# uname -ar
Linux localhost 3.14.9-200.fc20.x86_64 #1 SMP Thu Jun 26 21:40:51 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Connected disks status:
[root@localhost ~]# fdisk -l
Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 328E1A16-CB49-47FA-9421-D9608DCA2752
Device Start End Size Type
/dev/sda1 64 8388671 4G Linux RAID
/dev/sda2 8388672 9437247 512M Linux RAID
/dev/sda3 9437248 1953521072 927G Linux RAID
Disk /dev/sdb: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: B5EEA0F0-F7CD-4E9F-891F-2F1497D59651
Device Start End Size Type
/dev/sdb1 64 8388671 4G Linux RAID
/dev/sdb2 8388672 9437247 512M Linux RAID
/dev/sdb3 9437248 1953521072 927G Linux RAID
Disk /dev/md127: 926.9 GiB, 995236642816 bytes, 1943821568 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/md126: 4 GiB, 4290772992 bytes, 8380416 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/md125: 511.4 MiB, 536281088 bytes, 1047424 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
mdadm status (initially my array wasn't degraded, it was and it is in good shape):
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md125 : active (auto-read-only) raid1 sdb2[0] sda2[1]
523712 blocks super 1.2 [2/2] [UU]
md126 : active (auto-read-only) raid1 sdb1[0] sda1[1]
4190208 blocks super 1.2 [2/2] [UU]
md127 : active (auto-read-only) raid1 sdb3[0] sda3[1]
971910784 blocks super 1.2 [2/2] [UU]
bitmap: 0/8 pages [0KB], 65536KB chunk
unused devices: <none>
BTRFS status:
[root@localhost ~]# btrfs fi label /dev/md127
0e35a911:data
mdadm assemble resulted in this message:
[root@localhost ~]# mdadm --assemble --scan
mdadm: No arrays found in config file or automatically
Mounting /dev/md127 (data partition) looks good:
[root@localhost ~]# mount -t btrfs -o ro /dev/md127 /mnt
And here I am (as you can see my array was and is basically empty):
[root@localhost ~]# df -h|grep md127
/dev/md127 927G 3.1M 925G 1% /mnt
[root@localhost ~]# cd /mnt
[root@localhost mnt]# ls -lah
total 36K
drwxr-xr-x. 1 root root 124 Jul 28 11:36 .
drwxr-xr-x. 18 root root 4.0K Jul 30 03:54 ..
drwxrwxrwx. 1 root root 64 Jul 28 11:04 .apps
drwxrwxrwx+ 1 nobody nobody 0 Jul 16 11:43 Backup
drwxrwxrwx+ 1 nobody nobody 0 Jul 16 11:43 Documents
drwxr-xr-x. 1 98 98 10 Jul 16 11:04 home
drwxrwxrwx+ 1 nobody nobody 0 Jul 16 11:43 Music
drwxrwxrwx+ 1 nobody nobody 0 Jul 16 11:43 Pictures
drwxr-xr-x. 1 root root 36 Jul 28 11:36 .purge
drwxr-xr-x. 1 root root 68 Jul 28 11:36 ._share
drwxr-xr-x. 1 root root 0 Jul 16 11:04 .vault
drwxrwxrwx+ 1 nobody nobody 0 Jul 16 11:43 Videos
Mount point (I show the /dev/md127 only) status:
[root@localhost ~]# mount -l|grep md127
/dev/md127 on /mnt type btrfs (ro,relatime,seclabel,space_cache) [0e35a911:data]
Few dmseg grep messages about BTRFS and md:
Jul 30 04:03:57 localhost kernel: [ 719.673597] BTRFS: device label 0e35a911:data devid 1 transid 76 /dev/md127
Jul 30 04:03:57 localhost kernel: BTRFS: device label 0e35a911:data devid 1 transid 76 /dev/md127
Jul 30 04:03:57 localhost kernel: BTRFS info (device md127): disk space caching is enabled
Jul 30 04:03:57 localhost kernel: [ 719.674570] BTRFS info (device md127): disk space caching is enabled
[root@localhost Documents]# dmesg|grep md
[ 56.422296] md: bind<sda3>
[ 56.432502] md: bind<sdb3>
[ 58.602410] md: raid1 personality registered for level 1
[ 58.603526] md/raid1:md127: active with 2 out of 2 mirrors
[ 58.603650] created bitmap (8 pages) for device md127
[ 58.603862] md127: bitmap initialized from disk: read 1 pages, set 0 of 14831 bits
[ 58.629099] md127: detected capacity change from 0 to 995236642816
[ 58.629884] md: bind<sda1>
[ 58.632111] md: bind<sda2>
[ 58.640096] md: bind<sdb2>
[ 58.643333] md/raid1:md125: active with 2 out of 2 mirrors
[ 58.643368] md125: detected capacity change from 0 to 536281088
[ 58.653856] md127: unknown partition table
[ 58.669308] md125: unknown partition table
[ 58.678096] md: bind<sdb1>
[ 58.680007] md/raid1:md126: active with 2 out of 2 mirrors
[ 58.680053] md126: detected capacity change from 0 to 4290772992
[ 58.686240] md126: unknown partition table
[ 63.132849] systemd-journald[632]: Received request to flush runtime journal from PID 1
[ 73.919360] BTRFS: device label 0e35a911:data devid 1 transid 76 /dev/md127
[ 74.285276] Adding 523708k swap on /dev/md125. Priority:-1 extents:1 across:523708k FS
[ 719.673597] BTRFS: device label 0e35a911:data devid 1 transid 76 /dev/md127
[ 719.674570] BTRFS info (device md127): disk space caching is enabled
[ 719.714024] SELinux: initialized (dev md127, type btrfs), uses xattr
Status of /dev/mdx (x=127, 126 and 125) in detail:
[root@localhost ~]# mdadm --detail /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Wed Jul 16 11:03:50 2014
Raid Level : raid1
Array Size : 971910784 (926.89 GiB 995.24 GB)
Used Dev Size : 971910784 (926.89 GiB 995.24 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Jul 30 04:02:48 2014
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : 0e35a911:data-0
UUID : 1617bc13:e5576688:64018e5b:0ca61008
Events : 1759
Number Major Minor RaidDevice State
0 8 19 0 active sync /dev/sdb3
1 8 3 1 active sync /dev/sda3
[root@localhost ~]# mdadm --detail /dev/md126
/dev/md126:
Version : 1.2
Creation Time : Wed Jul 16 11:03:50 2014
Raid Level : raid1
Array Size : 4190208 (4.00 GiB 4.29 GB)
Used Dev Size : 4190208 (4.00 GiB 4.29 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Mon Jul 28 11:46:35 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : 0e35a911:0
UUID : 7062aefd:dcbae9c7:1dd84684:e7744d65
Events : 23
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 1 1 active sync /dev/sda1
[root@localhost ~]# mdadm --detail /dev/md125
/dev/md125:
Version : 1.2
Creation Time : Wed Jul 16 11:03:50 2014
Raid Level : raid1
Array Size : 523712 (511.52 MiB 536.28 MB)
Used Dev Size : 523712 (511.52 MiB 536.28 MB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Mon Jul 21 10:11:11 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : 0e35a911:1
UUID : 767a9300:bd522aac:366459ee:712f6896
Events : 19
Number Major Minor RaidDevice State
0 8 18 0 active sync /dev/sdb2
1 8 2 1 active sync /dev/sda2
After trying I'm now quite confident about BTRFS data recovery option using an (updated) Linux box: the only test we should do now could be to test a degraded array. - Nice guide. Should still work on a degraded array provided only one disk is bad. Of course if using RAID-6 you could have a few bad disks and still be O.K.
- Per mdgm above, the key appears to be getting an updated kernel and set of BTRFS tools. I switched to Fedora Linux v20 x86_64 with kernel 3.15 and that works without modification.
Second OS6 Data Recovery Test
ReadyNAS 314 configuration
OS 6.1.6
Default configuration - one user created
Drives - 4 x Seagate ST2000NV000 disks - 2 Tb (on the approved hardware list)
Volume configuration - FlexRAID in RAID 6 array for data partition (appears as MD127)
Approximately 17 Gb of data copied over - Debian Linux 7.4 AMD64 DVD ISO files (five files) verified with SHA1
One user created - SMB used as the file share protocol
Recovery PC configuration
OS - Fedora Linux 20 x86_64 - default install with customized partitioning - BTRFS
selected as the format, default partitions (/home on the same partition as root)
Ran "Yum update" from the command line after install - Many packages (and the kernel) updated up to 2 Aug 2014.
OS name Fedora 20
Kernel 3.15.7-200.fc20.x86_64 SMP
BTRFS v3.14.2
Motherboard Gigabyte GA-Z77X-UP5TH
i5 CPU 3570
8 Gb RAM
6 SATA ports
Port 0 - Plextor DVD drive
Port 1 - Seagate ST3808110AS - 80 Gb SATA drive for OS install
Port 2 - Drive 1 from ReadyNAS
Port 3 - Drive 2 from ReadyNAS
Port 4 - Drive 3 from ReadyNAS
Port 5 - Drive 4 from ReadyNAS
BIOS set to AHCI and "Legacy" mode (meaning non-UEFI)
Recovering the data at the command line
Followed the recommended commands from the earlier thread entry (all commands executed as root at the command line):
yum update (updates packages and kernel to latest online version)
*** NOTE: No need to install mdadm or btrfs-progs - since BTRFS was chosen as the install volume format, these were already in place. Running the mdadm --assemble --scan command was also not needed - the system found the array at boot up automatically.
cat /proc/mdstat (shows the connected arrays and their properties)
mount -t btrfs -o ro /dev/md127 /mnt (mounts the data volume at /mnt and allows access)
NOTE: On Fedora, the tools are actually named "btrfs-progs" and the version was "btrfs-progs 3.14.2-3.fc20.x86_64"
Data recovery
Once the volume was mounted, I was able to move to the /mnt/Documents folder where the test ISO files were stored. The files were moved to the folder /home/<username>/recov.
The files were checked again by generating an SHA1 sum and checking it against the official SHA1 sum in the distro. The SHA1 signature matched showing that the data was not corrupted. This was only 17 Gb of test data, but it was just a proof of concept.
Extra Test
There was a question up-thread about whether the mounting order of the hard disks makes a difference, so I decided to test that (since everything was ready anyway).
Here's what I found:
Out of Order Test 1
Motherboard SATA 2 - ReadyNAS Disk 3
Motherboard SATA 3 - ReadyNAS Disk 1
Motherboard SATA 4 - ReadyNAS Disk 4
Motherboard SATA 5 - ReadyNAS Disk 2
Data volume appears as "md126" this time when cat /proc/mdstat is run. It is mounted the same way:
mount -t btrfs -o ro /dev/md126 /mnt
Some sample SHA1 sums verified correctly.
Out of Order Test 2
Motherboard SATA 2 - ReadyNAS Disk 4
Motherboard SATA 3 - ReadyNAS Disk 1
Motherboard SATA 4 - ReadyNAS Disk 3
Motherboard SATA 5 - ReadyNAS Disk 2
Data volume appears as "md127" this time when cat /proc/mdstat is run. Mounted the same way as above and again the SHA1 sums verified correctly.
Can someone with more knowlege please explain the components of the cat /proc/mdstat output? It would be nice to know what I'm looking at.
From the "Out of Order Test 2", I see the following:
Personalities : [raid6] [raid5] [raid4] [raid1]
md125 : active (auto-read-only) raid1 sdd1[2] sdb1[3] sde1[1] sdc1[0]
4192192 blocks super 1.2 [4/4] [UUUU]
md126 : active (auto-read-only) raid6 sdd2[2] sde2[1] sdc2[0] sdb2[3]
1047936 blocks super 1.2 level 6, 64k chunk, algorithm 2 [4/4] [UUUU]
md127 : active (auto-read-only) raid6 sde3[1] sdd3[2] sdc3[0] sdb3[3]
3897329408 blocks super 1.2 level 6, 64k chunk, algorithm 2 [4/4] [UUUU]
unused devices: <none>
I mounted up md125, and that appears to be the OS6 partition, with no data available. Just the OS files but this might be crucial for troubleshooting - the log files would be available for analysis. The partition md126 would not mount - not sure what that is, maybe the swap file?
Why do the md names change? Sometimes the data volume is "md127" and sometimes it's "md126".
Just out of curiousity, I thought I'd do some "less than ideal" data recovery tests.
Single Hard Disk Missing from Array Test 1
For this test, I put the drive cables back in the first test configuration:
Mb SATA 2 - RN Disk 1
Mb SATA 3 - RN Disk 2
Mb SATA 4 - RN Disk 3
Mb SATA 5 - RN Disk 4
I tested to make sure the data was available (strangely, this time the data was on "md126"). It was accessible.
Then I shut down the machine and disconnected the Disk 1 data cable (simulated failed drive) and started up again.
This time the /proc/mdstat command gives this result:
Personalities :
md125 : inactive sdc1[2] (S) sdd1[3] (S) sdb1[1] (S)
12576768 blocks super 1.2
md126 : inactive sdd3[3] (S) sdc3[2] (S) sdb3[1] (S)
5845994613 blocks super 1.2
md127 : inactive sdb2[1] (S) sdd2[3] (S) sdc2[2] (S)
1572096 blocks super 1.2
unused devices: <none>
I poked around with mdadm, but other than getting a little information, I'm not sure how to rebuild this array or force it to mount.
I'm open to ideas on how to get this array to be visible and accessible with Disk 1 missing. What's the best way to force the array to be active and mount up with a disk missing? I'd like to test the scenario where a spare disk is not available, so the array needs to mount up using 3 disks.
Thanks for all the help, it's going great so far!
Stan
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
