Orbi WiFi 7 RBE973
Reply

ReadyNAS RN424 | Inactive Volume + RAID Issue

dschau
Aspirant

ReadyNAS RN424 | Inactive Volume + RAID Issue

Hello!

 

I am unable to access the drives / data on my Netgear ReadyNAS with four drives in RAID5 configuration with the following error in the UI:

 

> Remove inactive volumes to use the disk. Disk #1, #2, #3, #4

 

I am a panicked dad who may have done something incredibly silly. I am using the ReadyNAS RN424 and was trying to increase the capacity since space was running low by purchasing individual 16Tb drives, letting them sync / replace the existing 8Tb drive, and then continuing to move on until all four drives were replaced.

 

However, this morning I did the following set of steps, and now my ReadyNAS is showing _all_ drives as inactive and I cannot recover the data and get the volumes to mount. Here is what I did (in full transparency, learn from my mistakes!):

 

1. Last week, I swapped a drive (slot 4) from 8Tb to 16Tb, and it successfully re-built / updated to the new drive in about ~24 hours

2. Today, I replaced the second drive (slot 3) with a 16Tb drive and it was about ~0.2% replaced (2-3 minutes) when I realized that in installing the previous drive (slot 4) I didn't add the plastic drive mount to the previous drive so it was working but "floating"

3. I (foolishly) pulled out the previous 16Tb drive (slot 4), and then re-mounted it with the plastic mount and re-inserted the drive 

 

So, at this point I have three (working) drives in position, 1, 2, and 4 and drive 3 (which should be re-building). What happened instead is drive 3 and 4 both show as needing to be replaced, so I rebooted.

 

Upon reboot, now _all_ drives show as needing to be replaced presumably because of my actions. I panicked a bit, and then swapped drive 3 back to the original drive, which as of now as not worked (I also have rebooted a few times, just to try the whole "turn it off and on again" approach that frequently works)

 

I have the logs saved in Google Drive, and can message anyone who is willing to help with the logs. The key moment is at 1033AM PST, which is when the NAS went from "degraded" to "dead." I have also attached a few screenshots from the UI and the logs. I have checked in a few files, notably sytstem.log, kernal.log, etc. but nothing illustrative yet and frankly I don't know what to look for!

 

Any and all help is sincerely appreciated. I am fine if I lose some of the content, but I had some (non-backed up, again foolish) photos and videos of my newborn that I'd love to recover without needing to re-format the drive. Finally, I have opened up a ticket with Netgear support (I even tried to pay for premium support, but cannot seem to figure out how) as well.

 

 

Thank you.

logs-dead.png

Message 1 of 6

Accepted Solutions
dschau
Aspirant

Re: ReadyNAS RN424 | Inactive Volume + RAID Issue

OK, so first and foremost,@StephenB is an absolute legend. He took time out of his day to review the logs, and write an amazingly detailed guide for how to fix it. I'll summarize the high-level points with a disclaimer... if you're reading this, perhaps your issue is different. This is how I solved my issue, and Stephen was able to look into the logs to know this was the issue.

 

1. I removed the drive in slot 3 (the original one I tried to hot swap)

2. I re-booted the device in Tech Support mode (power down device, hold reset switch, power up device, keep holding reset switch, choose tech support mode)

3. I connected to the device with telnet (telnet ip-address-here) using root:infr8ntdebug

 

Then, I ran the following commands:

 

1. rnutil chroot (I think this logs me in to root user or something?)

2. mdadm --stop /dev/md127 (stop active partition)

3. mdadm --assemble --really-force /dev/md127 /dev/sda3 /dev/sdb3 /dev/sdc3

4. btrfs device scan

5. mount -r ro /dev/md127 /data

 

At this point, I had a working /data partition (huzzah) that I could inspect with ls (e.g. ls /data) and which showed the data I'd expect, as well as additional data for the Netgear device.

 

Finally, I BACKED UP the critical data just to avoid this issue going forward -- and also set up an rclone while I was at it to keep it up-to-date -- and then I inserted the new drive into slot 3 which is now being recovered.

 

A journey, but the destination was beautiful. Thank you again to @StephenB, you are a lifesaver, and a saint. 

View solution in original post

Message 5 of 6

All Replies
dschau
Aspirant

Re: ReadyNAS RN424 | Inactive Volume + RAID Issue

Some maybe illustrative (and concerning) messages in kernal.log at that 10:33 timestamp.

 

Aug 16 10:33:46 NAS kernel: sd 8:0:0:0: [sde] tag#17 Add. Sense: Unaligned write command

Aug 16 10:33:46 NAS kernel: sd 8:0:0:0: [sde] tag#17 CDB: Read(16) 88 00 00 00 00 00 02 da 19 80 00 00 05 40 00 00
Aug 16 10:33:46 NAS kernel: blk_update_request: I/O error, dev sde, sector 47847808
 
and then a number of failures:
 
Aug 16 10:33:46 NAS kernel: sd 8:0:0:0: rejecting I/O to offline device
 
and then other errors like:
 
Aug 16 10:33:46 NAS kernel: sd 8:0:0:0: [sde] tag#18 Add. Sense: Unaligned write command
 
Model: RN424|ReadyNAS 424 – High-performance Business Data Storage - 4-Bay
Message 2 of 6
StephenB
Guru

Re: ReadyNAS RN424 | Inactive Volume + RAID Issue

I'd be happy to look at it.  Resolving it will require you to use the linux command line. I can give you guidance after I review the logs.

 

If you want me to do that, then upload the full log zip file to cloud storage, and send me a download link in a PM (private message).  Make sure the link allows anyone with the link to download.

 

Send the PM using the envelope icon in the upper right of the forum page.

 

Message 3 of 6
dschau
Aspirant

Re: ReadyNAS RN424 | Inactive Volume + RAID Issue

Just messaged you! I know at least some Linux commands and such (I've been a software engineer for ~10-years) but I am admittedly less familiar with I/O things like this. So happy to take any guidance, and deeply appreciate the help! You are a lifesaver.

Message 4 of 6
dschau
Aspirant

Re: ReadyNAS RN424 | Inactive Volume + RAID Issue

OK, so first and foremost,@StephenB is an absolute legend. He took time out of his day to review the logs, and write an amazingly detailed guide for how to fix it. I'll summarize the high-level points with a disclaimer... if you're reading this, perhaps your issue is different. This is how I solved my issue, and Stephen was able to look into the logs to know this was the issue.

 

1. I removed the drive in slot 3 (the original one I tried to hot swap)

2. I re-booted the device in Tech Support mode (power down device, hold reset switch, power up device, keep holding reset switch, choose tech support mode)

3. I connected to the device with telnet (telnet ip-address-here) using root:infr8ntdebug

 

Then, I ran the following commands:

 

1. rnutil chroot (I think this logs me in to root user or something?)

2. mdadm --stop /dev/md127 (stop active partition)

3. mdadm --assemble --really-force /dev/md127 /dev/sda3 /dev/sdb3 /dev/sdc3

4. btrfs device scan

5. mount -r ro /dev/md127 /data

 

At this point, I had a working /data partition (huzzah) that I could inspect with ls (e.g. ls /data) and which showed the data I'd expect, as well as additional data for the Netgear device.

 

Finally, I BACKED UP the critical data just to avoid this issue going forward -- and also set up an rclone while I was at it to keep it up-to-date -- and then I inserted the new drive into slot 3 which is now being recovered.

 

A journey, but the destination was beautiful. Thank you again to @StephenB, you are a lifesaver, and a saint. 

Message 5 of 6
StephenB
Guru

Re: ReadyNAS RN424 | Inactive Volume + RAID Issue

I'm glad I was able to help.

Message 6 of 6
Top Contributors
Discussion stats
  • 5 replies
  • 5517 views
  • 2 kudos
  • 2 in conversation
Announcements