NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
RN 316 - Raid 5 config, Data Volume Dead
Hello. I have a ReadyNas 316 with 6x hd running in a raid 5 configuration. The other day I had disk2 go from ONLINE to FAILED and the system entered a degraded state. This forced a resync. As it was ...
Data is all personal/unique and important enough to attempt recovery. Not the end of the world if the data is lost, but valuable enough to try. My thinking is spend a little to do an initial but safe evaluation of my drives. If it does look possible for me to proceed with data recovery, pause and assess if I continue on my own or consult with a professional based on their price estimates.
I was unable to pull the latest logs from the RN 316 admin dashboard. Although I was able to navigate around the dashboard, each time I tried to download the logs, I was presented with a generic web service error. I don’t recall the error…I want to say it was a 503 error code. The web service was not able to respond to my request to download the logs.
Question: are you familiar enough with the internals of how RN 316 stores and manages its log? My thinking is it should have its own internal storage or memory to collect logs (and not use the hdd that make up the raid array). If so, I could remove and label all drives, power up the RN and connect to it and attempt to either download the logs from the web portal or ssh in and access the logs from the terminal. What do you think? Is this worth pursuing?
I do however, have logs from a day earlier, prior to the failure of the NAS. It’s important to note from the logs that disk3 has multiple warning reports.
Yes, the resync happened on its own. A scheduled scrub was running, which caused Disk2 to “lose sync” and that triggered the resync to automatically occur. This is what lead to Disk3s reported failure and then data volume dying.
Question: since the NAS reported that disk2 was offline and in a resync state, does this imply that any data that was on this disk to preserve the data volume has been lost or become invalid? It had only been resyncing for a 3 hours of the 27 hours it estimated for completion. I’m asking b/c I want to know if I can even consider disk2 as a candidate for restoring the data from the NAS.
I don’t have experience in data recovery. I am very comfortable working in Linux environment and running command lines. I’m not experienced with mdadm or btrf so i would need to study up on those filesystems.
The total volume size was at 18TB. I do see 20+ TB drives in the market that are reasonably priced. I prefer to proceed with your suggestion for cloning each drive as an image and store them on a single hard drive before attempting the data recovery process. At least I will then have images of each drive saved and can then decide how to proceed with data recovery using the original hard drives.
Question:
- You suggested I connect each drive to a PC running Western Digitals dashboard utility to check the status of the drive. Are you referring to this https://support-en.wd.com/app/answers/detailweb/a_id/31759/~/download%2C-install%2C-test-drive-and-update-firmware-using-western-digital
- Assuming tool recognizes the connected drive from the NAS, what should I be looking for?
- I understand I can run various tests, but I’m not sure I want to run them at the risk of stressing the drives further.
Here’s my proposed plan. Please let me know what you think:
- Purchase a 20TB+ hard drive for storing images of each cloned drive. Even if data recovery is impossible, I can utilize this drive for other purposes so I’m willing to make this expense in the beginning.
- Connect disk3 (drive that was in a FAILED state) to a PC running WD dashboard.
- If the drive is recognized by the dashboard, proceed with cloning the drive.
- Question: I’m learning that ddrescue for Linux is the best tool for cloning. Assuming I get this far, I’d need to reboot the machine and open my Linux environment. Is this the right tool and env to use for cloning? Alternatively, I can setup ddrescue-gui in advance to be prepared to do a clone in a windows environment.
Thank you!
halbertn wrote:I understand I can run various tests, but I’m not sure I want to run them at the risk of stressing the drives further.
That is a risk. You could limit the test to the short self test (which normally only takes about 2 minutes). If that fails, the drive has certainly failed. Though success wouldn't mean that the drive is ok, given the brevity of the test.
You can run essentially the same test using smartctl on a linux system.
halbertn wrote:
Question: are you familiar enough with the internals of how RN 316 stores and manages its log?
Many of the files in the zip extracted from the systemd journal. There are some others - mdstat.log is a copy of \etc\mdstat. lsblk is just the output of lsblk, etc. If you can log in via ssh (or tech support mode) you can create the zip file manually with rnutil create_system_log -o </path/filename> Use root as the username when you log in.
halbertn wrote:
Question: since the NAS reported that disk2 was offline and in a resync state, does this imply that any data that was on this disk to preserve the data volume has been lost or become invalid? It had only been resyncing for a 3 hours of the 27 hours it estimated for completion. I’m asking b/c I want to know if I can even consider disk2 as a candidate for restoring the data from the NAS.
It depends on whether you were doing a lot of other writes to the volume during the resync. It the scrub/resync is all that was happening, then the data being written to the disk would have been identical to what was already on the disk. Likely there was some other activity. But I think it is reasonable to try recovery with disk 2 if you can clone it. There is no harm in trying.
halbertn wrote:
Question: I’m learning that ddrescue for Linux is the best tool for cloning. Assuming I get this far, I’d need to reboot the machine and open my Linux environment. Is this the right tool and env to use for cloning? Alternatively, I can setup ddrescue-gui in advance to be prepared to do a clone in a windows environment.
You would need to install ddrescue. In general, installing stuff on the NAS will require some changes to the apt config.
More info on that is here:
Or you could just boot the PC up using a linux live boot flash drive.
StephenB
During the scrub/resync process, there should have been 0 writes to the volume. I wasn't accessing the drive or writing any new data to it. It should have been idling with the exception of whatever operation a scrub or sync would have been doing. From your answer and based on the overall health of the drive, Disk2 could be a likely candidate for recovering the data volume.
Question:
1. If my intention is to clone an image of every drive from my NAS onto a single harddrive, which filesystem should I put on this new drive that will be storing my images? I'm assuming ext4 as it only needs to store the images and be accessible in Linux (and windows).
2. Let's assume I'm able to access each drive and clone an image of them. Doesn't this imply that if I were to re-insert these drives back into their original slots in the RN 316 enclosure, then *maybe* the data volume may just come up and and be readable?
3. Suppose I can only clone 5 of the 6 drives because 1 of the drives is failing to be recognized or fails to be cloned. Then same question as 2 but with the 5 drives.
4. Assuming 2 or more drives are unaccessible, then I can declare that recovery on my own is impossible? Maybe I consider professional help, but they are unlikely to succeed considering that the drives have failed.
5. Suppose I do have 5 or 6 images cloned, and I'm not able to access the data volume in the RN 316 enclosure using the original drives. Then I attempt to access the data volume by rebuilding the raid array using the cloned images in a linux environment as a next step?
Thank you!
PS. How are you performing an inline response to my post? I can't find the option to do that.
halbertn wrote:
PS. How are you performing an inline response to my post? I can't find the option to do that.I use the quote tool to get a full quote of the original post, and then edit it. This is a bit buggy - I do need to shift to html sometimes to clean it up. It's tedious in the beginning, but like most things gets easier/faster with practice.
halbertn wrote:
Question:
1. If my intention is to clone an image of every drive from my NAS onto a single harddrive, which filesystem should I put on this new drive that will be storing my images? I'm assuming ext4 as it only needs to store the images and be accessible in Linux (and windows).You want something that can handle a 3 TB drive image, so ext4 or ntfs could both work. ext4 is probably the right option.
halbertn wrote:
2. Let's assume I'm able to access each drive and clone an image of them. Doesn't this imply that if I were to re-insert these drives back into their original slots in the RN 316 enclosure, then *maybe* the data volume may just come up and and be readable?Likely you would also need to forcibly assemble the RAID array (since it will be out of sync). BTRFS repair might also be needed.
But if you had 6 clones (instead of images), you could put these in the NAS and do those steps in the NAS.
halbertn wrote:
3. Suppose I can only clone 5 of the 6 drives because 1 of the drives is failing to be recognized or fails to be cloned. Then same question as 2 but with the 5 drives.You'll need at least 5 drives, as RAID can reconstruct the 6th from the remaining 5. Though if there are unreadable sectors on the source drives, that reconstruction will have some errors.
halbertn wrote:
4. Assuming 2 or more drives are unaccessible, then I can declare that recovery on my own is impossible?Yes.
halbertn wrote:
Maybe I consider professional help, but they are unlikely to succeed considering that the drives have failed.Likely true. Forensic recovery tools might be able to read sectors that dd rescue can't. Their recovery software might also be able to get some things back.
halbertn wrote:
5. Suppose I do have 5 or 6 images cloned, and I'm not able to access the data volume in the RN 316 enclosure using the original drives. Then I attempt to access the data volume by rebuilding the raid array using the cloned images in a linux environment as a next step?
Yes, though you could just directly move to working with the images. Sandshark's posts explain how to mount the images as disks in a ReadyNAS VM. You can substitute a generic Linux VM for that also.
Hello StephenB
halbertn wrote:
2. Let's assume I'm able to access each drive and clone an image of them. Doesn't this imply that if I were to re-insert these drives back into their original slots in the RN 316 enclosure, then *maybe* the data volume may just come up and and be readable?Likely you would also need to forcibly assemble the RAID array (since it will be out of sync). BTRFS repair might also be needed.
But if you had 6 clones (instead of images), you could put these in the NAS and do those steps in the NAS.
Ok. I think I understand: no matter what, some repair or assembly of the raid would have to take place. Therefore, I need to perform this against the new clones (not the original hd as they won't be able to withstand the operation without risk of failure).
Question1: If I load the images of each clone using a ReadyNAS VM, then would the ReadyNAS VM be able to assembly the raid array and possibly repair the BTRFS filesystem? I assume this will require the VM to have to write to each image to perform these operations, which it should have permission to do? (For some reason, I'm thinking images are readonly, but I think I'm wrong... as long as the write permissions are set, it should be okay)
Question 2: I'm researching how to clone disks using ddrescue. I'm learning ddrescue performs multiple passes against the hd when cloning. Therefore, one recommendation is to split the clone into multiple phases:# first phase - skip bad sectors sudo ddrescue --verbose --idirect --no-scrape [source] [dest] disk.logfile# retry bad sectors up to 3 times sudo ddrescue --verbose --idirect -r3 --no-scrape [source] [dest] disk.logfileWith this last run as optional:
# last run - removing 'no-scrape' sudo ddrescue --verbose --idirect -r3 [source] [dev] disk.logfile
What do you think? My use-case may be different as I am trying to recover as much data to rebuild the raid-array after the clone operation. How would you go about performing the clone for each disk?
Thank you!
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
