× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: Ready NAS Pro Pioneer RAID 5 Recovery

Tuttle
Aspirant

Ready NAS Pro Pioneer RAID 5 Recovery

Hello everyone!

 

A couple of month ago my Ready NAS Pro Pioneer Edition - 6 disks, RAID 5 - failed me after 6 years of service, not starting up at all. Since my biggest concern was to retrieve the data I followed the guidelines in this thread: https://community.netgear.com/t5/Using-your-ReadyNAS/ReadyNAS-Pro-Data-Recovery-18789299/td-p/924886.

 

I hooked up the 6 drives (ext4) to a mainboard with 6 sata connectors (one single controller) und booted an ubuntu installation from an usb-stick. After using the commands from said thread I was able to browse and read all files from the RAID 5.

 

sudo su
apt-get install lvm2
mdadm --assemble --scan
vgscan
vgchange -ay c
mount /dev/c/c /mnt

Everything went curiously well for a non-linux user... until 40 minutes into copying the process stopped with the failure message "i/o error - file not found" (or similar). After that I was able to assemble the volume once more with "mdadm --assemble --scan", but another message prompted "4 out of 6 drives not sufficient to build array". Now the "mdadm"-command does nothing anymore, "vgscan" does not find any volumes. The HDD drives are all healthy and recognized.

 

A different system (mainboard, processor, PSU), that was displaying the array/data correctly earlier, shows the same symptoms. "mdadm" does nothing at all, just the prompt, no failure, no result.

 

My questions:

 

1. Did I possibly - due to lack of linux knowledge - corrupt the data somehow, maybe failing to mount the RAID 5 volume writeprotected/degraded? If yes, what would have been the proper course of action?

 

2. Is there any chance of fixing a possibly corrupted RAID 5 structure spanning 6 disks? And if yes... *desperate* how? I dont know linux too well, but am willing to invest a lot of research into it. I just need a helpful hint which path to follow.

 

Best regards!

 

 

Message 1 of 7

Accepted Solutions
mdgm-ntgr
NETGEAR Employee Retired

Re: Ready NAS Pro Pioneer RAID 5 Recovery

Seeing that both disks dropped out at the same time it hopefully will work.

 

However forcing an array online is not without its risks. If it doesn't work out then your array may be left in a state where you can't recover anything.

You may wish to clone your disks first. It's up to you.

There should be info on the OS partition which indicates the drive order.

View solution in original post

Message 6 of 7

All Replies
mdgm-ntgr
NETGEAR Employee Retired

Re: Ready NAS Pro Pioneer RAID 5 Recovery

When your Pro Pioneer did not start up at all do you mean to say that you had no response when you attempted to power it on? If it did say get stuck with booting on the display that would suggest a bad disk. It's also advisable to power down, remove the disks (label order) and see if you get any response with the disks removed if you get no response with the disks installed. With no disks installed you should get an error indicating that no disks are installed.

It is advisable to check the health of the disks before attempting to start the array and to check that the array is in a good state.

 

You can check the health of the disks using e.g. SeaTools for SeaGate disks or WD Data Lifeguard Diagnostics for WD disks. This can be done using a boot disk such as the Ultimate Boot CD (a free boot disk). Alternatively if you prefer you can install smartmontools on Linux and use the smartctl command.

To check the update time for the md devices you could use e.g.

 

# cat /proc/partitions
# mdadm -E /dev/sda3

# mdadm -E /dev/sdb3


Assuming that the 3rd partition is the large one. It may be sda5 if you last did a factory reset on very old firmware. Furthermore if you have multiple RAID layers there will be multiple layers to check.

 

Mounting the volume read-only is certainly preferable.

Data recovery using an ordinary Linux PC is a last resort.

Message 2 of 7
Tuttle
Aspirant

Re: Ready NAS Pro Pioneer RAID 5 Recovery

Thanks very much for the quick reply!

 

The unit did not start up at all, just the Network LEDs on the RJ45 LAN-Ports were flickering dimly, with and without drives installed. Somewhere here a user fixed his ReadyNAS Pioneer with said behaviour by changing the PSU, which did not work for me. With the new PSU still no response.

 

So i am focusing on recovering the data initiallly. The drives are 6x Hitachi 3TB Hitachi HDS5C3030ALA630, running for about 3 years. What I gathered from the system info (Windows and Linux) the drives look fine (temperature, partitions, etc.).

 

smartctl indicated no errors, selftests have been ok.

 

mdadm -E /dev/<big partition> gives the following results (magic numbers match, all states clean, all checksums correct):

 

/dev/sda3: Device Role: Active device 4, Array State: AAAAAA , Events: 68

/dev/sdb3: Device Role: Active device 5, Array State: AAAAAA , Events: 68

/dev/sdc3: Device Role: Active device 3, Array State: AAAA.. , Events: 75

/dev/sde3: Device Role: Active device 2, Array State: AAAA.. , Events: 75

/dev/sdf3: Device Role: Active device 1, Array State: AAAA.. , Events: 75

/dev/sdg3: Device Role: Active device 0, Array State: AAAA.. , Events: 75

 

(missing 2 devices on 4 of the disks) 

 

1. How do I proceed from there? Do I have to reconfigure the raid somehow? Is mdadm --assemble --verbose --force a correct approach? As mentioned I just started using linux and am a little nervous to do further damage, so try and error is out of the question.

 

2. Is the order the devices are attached to the hardware controller in any way important?

 

 

 

Message 3 of 7
mdgm-ntgr
NETGEAR Employee Retired

Re: Ready NAS Pro Pioneer RAID 5 Recovery

What are the update times for sda3 and sdb3?

 

You may wish to consider purchasing a new ReadyNAS e.g. 316 and purchasing a data recovery attempt rather than try to recover it yourself. It is very easy to get things wrong when attempting data recovery is you are unfamiliar with doing it.

Message 4 of 7
Tuttle
Aspirant

Re: Ready NAS Pro Pioneer RAID 5 Recovery

The update times

sda3: 23:21:01

sdb3: 23:21:01

sdc3: 23:49:08

sde3: 23:49:08

sdf3: 23:49:08

sdg3: 23:49:08.

 

I was copying a rather large file of 128 GB at that time, that failed to read after 40 GBs.

 

Here is the /proc/mdstat to give a picture of the setup, if that might help.

 

Personalities : [raid1] [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid10]
md2 : inactive sdc3[3](S) sde3[2](S) sdg3[0](S) sdf3[1](S) sda3[4](S) sdb3[5](S)
      17553269331 blocks super 1.2

md1 : active raid6 sde2[2] sdc2[3] sdg2[0] sdf2[1] sda2[4] sdb2[5]
      2096896 blocks super 1.2 level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]
md0 : active raid1 sde1[2] sdc1[3] sdg1[0] sdf1[1] sda1[4] sdb1[5]
      4193268 blocks super 1.2 [6/6] [UUUUUU]

 

I am aware that I might screw things up during but that is a risk ill have to take since I cant invest more than time into the recovery at the moment.

 

in this scenario, would

mdadm --asseble --force /dev/md2 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sde3 /dev/sdf3 /dev/sdg3

be the right command for an assemble? can --force destroy data, eg in the superblocks?
      

 

 

Message 5 of 7
mdgm-ntgr
NETGEAR Employee Retired

Re: Ready NAS Pro Pioneer RAID 5 Recovery

Seeing that both disks dropped out at the same time it hopefully will work.

 

However forcing an array online is not without its risks. If it doesn't work out then your array may be left in a state where you can't recover anything.

You may wish to clone your disks first. It's up to you.

There should be info on the OS partition which indicates the drive order.

Message 6 of 7
Tuttle
Aspirant

Re: Ready NAS Pro Pioneer RAID 5 Recovery

It worked! I still think i was EXTREMELY lucky and do not really have a clue what i did yet... however heres a short recap that might come in handy to someone else.

 

This basically was all done with mdgm's help, thanks a billion for the extraordinary help!

 

1. I connected all 6 drives to a system with 6 SATA channels and booted from USB-Stick with the installation image of Ubuntu, see here, choosing first boot option "Try Ubuntu..."

 

2. The "Disks" Application available from the Search icon top left gives an overview, if the drives are connected properly and their corresponding device names /dev/sda etc. The big partition on each drive is the on to look for, usually /dev/sda3, /dev/sdb3 etc...

 

(All following commands were executed as root, so I left out the "sudo" most of the times)

 

3. smartctl /dev/sda -a (available after apt-get install smartmontools) helps to check the physical status of the disks.

 

4. This thread pointed me to the commands necessary to reassemble the raid:

 

sudo su
apt-get install lvm2
mdadm --assemble --scan
vgscan
vgchange -ay c
mount /dev/c/c /mnt

 

The data from the RAID 5 should be available by then in the /mnt directory.

 

5. Unfortunately later during copying 2 drives dropped out of the array for reasons unknown. To examine the drives I used

 

mdadm -E /dev/sd[abcefg]3 (Notice that the drives are not necessarily sequentially labeled).

 

I found that two of the partitions had dropped out and were not seen by the other partitions. The small difference in the Event count (<50) and the identical dropout times seem to be a good thing...

 

/dev/sda3: Device Role: Active device 4, Array State: AAAAAA , Events: 68, Update time: 23:21:01

/dev/sdb3: Device Role: Active device 5, Array State: AAAAAA , Events: 68, Update time: 23:21:01

/dev/sdc3: Device Role: Active device 3, Array State: AAAA.. , Events: 75, Update 23:49:08

/dev/sde3: Device Role: Active device 2, Array State: AAAA.. , Events: 75, Update 23:49:08

/dev/sdf3: Device Role: Active device 1, Array State: AAAA.. , Events: 75, Update 23:49:08

/dev/sdg3: Device Role: Active device 0, Array State: AAAA.. , Events: 75, Update 23:49:08

 

(mdadm -E /dev/sd[abcefg]3 | egrep "Events|Device Role|Array State|Update Time" can help with a condensed view.)

 

6. cat /proc/mdstat indicated, that Ubuntu automatically had assembled the other partitions as raids (I guess...?).

 

Personalities : [raid1] [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid10]
md2 : inactive sdc3[3](S) sde3[2](S) sdg3[0](S) sdf3[1](S) sda3[4](S) sdb3[5](S)
      17553269331 blocks super 1.2

md1 : active raid6 sde2[2] sdc2[3] sdg2[0] sdf2[1] sda2[4] sdb2[5]
      2096896 blocks super 1.2 level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]
md0 : active raid1 sde1[2] sdc1[3] sdg1[0] sdf1[1] sda1[4] sdb1[5]
      4193268 blocks super 1.2 [6/6] [UUUUUU]

 

So I stopped them with mdadm --stop /dev/md[012].

 

(From here I switched the whoe setup to another machine, so please excuse inconsistent device letters, now they are labeled abdefg iso abcefg, sequence changed as well.)

 

7. Finally I force assemble the RAID (This is the point where one should have his disks cloned in case all goes south...).

 

mdadm --assemble --force --verbose /dev/md0 /dev/sda3 /dev/sdb3 /dev/sdd3 /dev/sde3 /dev/sdf3 /dev/sdg3


mdadm: looking for devices for /dev/md0
mdadm: /dev/sda3 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdb3 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdd3 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sde3 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdf3 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdg3 is identified as a member of /dev/md0, slot 5.
mdadm: forcing event count in /dev/sdf3(4) from 68 upto 75
mdadm: forcing event count in /dev/sdg3(5) from 68 upto 75
mdadm: clearing FAULTY flag for device 4 in /dev/md0 for /dev/sdf3
mdadm: clearing FAULTY flag for device 5 in /dev/md0 for /dev/sdg3
mdadm: Marking array /dev/md0 as 'clean'
mdadm: added /dev/sdb3 to /dev/md0 as 1
mdadm: added /dev/sdd3 to /dev/md0 as 2
mdadm: added /dev/sde3 to /dev/md0 as 3
mdadm: added /dev/sdf3 to /dev/md0 as 4
mdadm: added /dev/sdg3 to /dev/md0 as 5
mdadm: added /dev/sda3 to /dev/md0 as 0
mdadm: /dev/md0 has been started with 6 drives.

 

8. Final mount:

 

vgscan
vgchange -ay c
mount -r /dev/c/c /mnt 
(This time as read-only!)

 

Still copying... 

 

 

 

 

 

Message 7 of 7
Top Contributors
Discussion stats
  • 6 replies
  • 3340 views
  • 0 kudos
  • 2 in conversation
Announcements