Re: ReadyNas 6 locked up and now array is inactive

StrawDragon · ‎2022-02-20

Hello!

I have a 6 bay ReadyNas running 6.10.6.

System has been expanded from 6x 6TB - > 4x10TB and 2x6TB about a year ago running RAID5 +XRAID.

This afternoon the box just locked up, so I rebooted it.

When logging into my NAS I saw that all drives were not inactive volumes.

First I tried to reboot the device. No change.

Powered off and reseated the drives.. No Change.

I ssh into the NAS and found /devt/md0, md1, md126 (27TB) and md127 (11TB)

md126 contains /dev/sd[abcdef]3

md127 contains /dev/sd[abcd]4

So that is where I am lost on how to try and resolve this. I only had one DATA volume created in the ReadyNAS gui.

How does btrfs play in here with the two volumes? I should have a single DATA volume of nearly 40TB.

I am fairly confident in my Linux skills so any help would be awesome. I have googled the heck out of the internet for some information. If I can at least get my array online and move the data off. That would be perfect.

StephenB · ‎2022-02-21

@StrawDragon wrote:

System has been expanded from 6x 6TB - > 4x10TB and 2x6TB about a year ago running RAID5 +XRAID.

I ssh into the NAS and found /devt/md0, md1, md126 (27TB) and md127 (11TB)

md126 contains /dev/sd[abcdef]3

md127 contains /dev/sd[abcd]4

So that is where I am lost on how to try and resolve this. I only had one DATA volume created in the ReadyNAS gui.

How does btrfs play in here with the two volumes? I should have a single DATA volume of nearly 40TB.

You do have only one volume. But because of the expansion you have two RAID groups - which are concatenated together.

One is 6x6TB, the other is 4x4TB (filling the remaining space on the four 10 TB drives).

Raw space on the 6x6 TB group is 36 TB (32.7 TiB). Available storage - subtracting RAID redundancy - is 30 TB ( 27 TiB)

Raw space on the 4x6 TB group is 16 TB (14.5 TiB), and available storage is 12 TB (10.9 TiB).

So this part looks correct.

@StrawDragon wrote:

This afternoon the box just locked up, so I rebooted it.

When logging into my NAS I saw that all drives were not inactive volumes.

Do you mean that you have inactive volume (s) on the volume tab???

StrawDragon · ‎2022-02-21

Thank you.

The total usable space should be ~38TB (10TB x 4 & 6TB x 2) in RAID 5 with XRAID.

Yes it was a single volume.

StephenB · ‎2022-02-21

@StrawDragon wrote:

The total usable space should be ~38TB (10TB x 4 & 6TB x 2) in RAID 5 with XRAID.

Agreed. Which is the sum of the two RAID groups that comprise the volume - 27 + 10.9 TiB. The NAS reports space in TiB.

Once again: Do you mean that you have inactive volume (s) on the volume tab??? Your original post wasn't clear on that.

StrawDragon · ‎2022-02-21

Yes, they show as red inactive drives and the gui suggests to remove them from the raid.

I did include a picture of the volume page in my last post. (see below).

How does the system uses the two defined arrays as a single? (md126 and md127) to form a single array of 38.18TB?

StephenB · ‎2022-02-21

@StrawDragon wrote:

How does the system uses the two defined arrays as a single? (md126 and md127) to form a single array of 38.18TB?

Each array creates a virtual disk. And you can create file systems in linux that span multiple disks (concatenating the space). The file system is the volume.

Thanks for the screenshot, it is helpful. Don't remove those inactive volumes.

I suggest starting by looking at the smart stats for the disks. smartctl -x /dev/sda will show you stats for the first one, then proceed to sdb, etc. There is a lot of data (much of it extraneous). But -x adds in the error log stored on the drive, so look at that section for any recent errors. Note the time is in power-on hours, so you do need to look at the current power on hours to get some idea of how recent the errors are.

I also suggest running the smart self test with smartctl for each drive. If you want to take the time, you could also run the full test - though that will take a long time, given the number and size of the disks you have.

There is a command you could try which (if successful) will allow the volume to be remounted - with perhaps a small amount of data corruption/loss. But let's check the disk health before we move onto that.

Note there is another avenue - you could get a data recovery contract with Netgear. That is expensive, but it would give you the best odds of data recovery. https://kb.netgear.com/69/ReadyNAS-Data-Recovery-Diagnostics-Scope-of-Service

StrawDragon · ‎2022-02-21

I am not asking for a data recovery, I should be able to pay per incident for them to bring the array online if I cannot do so. There was no drive failure and smart shows 0 errors on the 6 drives.

One drive has a couple of errors but most recently was over ~1680 days ago on one of the two 6TB drives.

So I have two logical volume md126 and md127, how does it combine two virtual volumes? Does it use raid0? as parity is across each drive?

Sandshark · ‎2022-02-21

Concatenating the RAID groups is done by BTRFS, it's not RAID. It's nothing unique to ReadyNAS. XRAID is "just" some logic that makes doing that easier for those that don't care to go "under the hood" with generic Linux. The RAID groups (and their parity) are entirely independent of BTRFS, being created and maintained by mdadm.

But if BTRFS wasn't aware that the RAID groups belonged together, you'd be usually seeing multiple (inaccessible) volumes in the GUI, usually data and data-0 and/or data-1. Your display is what usually is shown when a volume has gone read-only due to an error, not one where the volume is inaccessible.

StephenB · ‎2022-02-21

@StrawDragon wrote:

So I have two logical volume md126 and md127, how does it combine two virtual volumes? Does it use raid0? as parity is across each drive?

No RAID0. The file system (BTRFS in this case) spans two virtual disks, and the file system decides what it will put on each one. md126 and md127 each have their own (independent) RAID parity. This is standard RAID-5, so the parity blocks are evenly distributed across all your physical disks.

@StrawDragon wrote:

I am not asking for a data recovery, I should be able to pay per incident for them

FYI, neither @Sandshark nor I work for Netgear. I was just presenting your options. Unfortunately, Netgear no longer offers pay-per-incident support.

@StrawDragon wrote:

There was no drive failure and smart shows 0 errors on the 6 drives.

Please download the full log zip from the logs page. Then look for btrfs errors in kernel.log. I am thinking you will see some errors like "parent transid verify failed", followed by "failed to read log tree" and "open_ctree failed".

If you don't see this, then someone should probably analyze your logs for you.

If you do see this, then note the device in the error (I think it will be md126, but it's possible it will be md127). Then try this command

btrfs rescue zero-log /dev/md12X

(using the same device that appears in the kernel.log entry).

This should give some more messages like "parent transid verify failed", and then say it is clearing the log on /dev/md12X If it does, then reboot the NAS and see if the volume mounts.