NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
RN 316 - Raid 5 config, Data Volume Dead
Hello. I have a ReadyNas 316 with 6x hd running in a raid 5 configuration. The other day I had disk2 go from ONLINE to FAILED and the system entered a degraded state. This forced a resync. As it was ...
halbertn wrote:
- The raid 5 configuration was setup using netgear’s X-Raid. Does this imply rebuilding this raid array outside of netgears ReadyNAS HW is impossible? I think X-Raid is proprietary to netgear?
X-RAID is built on Linux software RAID (mdadm). All X-RAID adds is software that automatically manages expansion.
Rebuilding the array and mounting the data volume is certainly possible on a standard linux PC that has mdadm and btrfs (the file system used by the NAS) installed.
halbertn wrote:
However, it is possible that I may be able to recover disk3 despite it being marked FAILED.
Mar 26, 2024 08:17:57 PM Disk: Detected increasing reallocated sector count: [1967] on disk 3 (Internal)
This says ~2000 failed sectors on disk 3. There could easily be more that haven't been detected yet.
Plus you have these errors:
Mar 26, 2024 10:12:34 AM Volume: Disk test failed on disk in channel 5, model WDC_WD3003FZEX-00Z4SA0, serial WD-WCC130725037.
Mar 26, 2024 10:12:33 AM Volume: Disk test failed on disk in channel 3, model WDC_WD30EFRX-68EUZN0, serial WD-WMC4N1002183.
Mar 26, 2024 10:12:33 AM Volume: Disk test failed on disk in channel 2, model WDC_WD30EFRX-68EUZN0, serial WD-WMC4N0956058.
suggesting that disk 5 also is at risk.
I've helped quite a few people deal with failed volumes over the years. Honestly your odds of success aren't good.
BTW, did you download the full log zip file from the NAS before you powered it down?
halbertn wrote:
Based on my preliminary research, what I need to do is clone each drive to preserve whatever data is recoverable. If I’m lucky and can access disk 3 and clone it (along with disk 1,4,5,6) and not run into any other issues, I *should* be able to rebuild/recover the raid array as I have enough drives to do so.
When you try to recover from failing disks, the recovery can stress the disks - which can push them to complete failure. The benefit of cloning is that it can limit that additional damage.
There is a downside. RAID recovery can only help recover/repair failed sectors when it can detect which sectors have failed. Cloning hides that information. A failed sector in the source will result in garbage data on the corresponding sector of the clone, and there is no way RAID recovery software can identify that sector on the clone is bogus,
Still, in your case cloning makes sense.
There is a related path with similar plusses and minuses. That is to image the disks, and then work with the images. That can be done in a ReadyNAS VM or using RAID recovery software. If you have a large enough destination disk, it can hold multiple images.
halbertn wrote:Can I say with certainty that disk2 is useless in the context of using it for any data recovery,
It's hard to say whether disk 2 is in worse shape than disk 3 or not. The fact that the system started to resync means that the disk 2 came back on line after the degraded message.
It is possible that disk 2 has completely failed, but you can't conclude that yet.
halbertn wrote:
Cloning every drive to preserve their data would require me purchasing 6 new older hd models that are compatible with the ReadyNAS 316.
No. It would be a big mistake to try and purchase old hard drive models from the HCL
Current Seagate Ironwolf drives and WD Red Plus drives are all compatible with your NAS. So are all enterprise-clasee drives.
Avoid desktop drives and WD Red models, even if they are on the compatibility list. Most desktop drives in the 2-6 TB size range are now SMR drives that are not good choices for your NAS. Regrettably current WD Red drives are also SMR (but not Red Plus or Red Pro). These drives often have very similar model numbers to older CMR drives on the compatibility list.
Plus in recent years Netgear really didn't test all the drives they added to the HCL. Many were added based on tests of different models that Netgear believed were similar.
Bottom line here is that you can't trust the HCL, and it is best ignored.. Just get Red Plus, Seagate Ironwolf, or enterprise class disks.
halbertn wrote:
Before I make this investment, I do want to be able get some confidence or indication that I can recover the data. What's the best way to start? I'm thinking maybe I just attempt to clone disk3, then replace it with the new clone in ReadyNAS 316.
I'd start with asking yourself how much your data is worth to you. That would establish a ceiling on how much you are willing to pay to recover it.
It is also worth asking yourself what you will do if the recovery fails. If you plan to bring the NAS back on line anyway (starting fresh), then investing in new disks wouldn't be wasted money, as you need to replace the failed disks anyway.
As far as where to start - I'd connect each drive to a PC (either with a USB adapter/dock or with SATA) and test it with WD's dashboard utility. That will give you more information on how many disks are failing. Label the disks by slot number as you remove them from the NAS.
As I mentioned above I think successful recovery will be problematic even for a professional. In addition to dealing with multiple disk failures, it sounds like you have no experience with data recovery. How much experience do you have with the linux command line interface? Have you ever worked with mdadm or btrfs using those interfaces? It is very easy to do more damage (making recovery even more difficult) if you don't really know what you are doing.
halbertn wrote:
These hard drives are more than 5-6 years old and non enterprise. In hindsight, I should *not* have allowed the resync to occur and instead performed a backup of the NAS before resyncing the data. The resyncing of the raid array taxed disk3 resulting in its failure and breaking the whole system which I’m learning is quite common. At this point, I have lost 2 disks of my 6 hds, thus making recovery via the raid 5 configuration impossible.
Did you do something to trigger the resync? Or did it happen on its own???
FWIW, I think the lesson here is that RAID isn't enough to keep data safe - you definitely need a backuip plan in place to do that. Disks can fail without warning (as can a NAS or any storage device). Waiting until there's a failure to make a backup often results in waiting too long.
Data is all personal/unique and important enough to attempt recovery. Not the end of the world if the data is lost, but valuable enough to try. My thinking is spend a little to do an initial but safe evaluation of my drives. If it does look possible for me to proceed with data recovery, pause and assess if I continue on my own or consult with a professional based on their price estimates.
I was unable to pull the latest logs from the RN 316 admin dashboard. Although I was able to navigate around the dashboard, each time I tried to download the logs, I was presented with a generic web service error. I don’t recall the error…I want to say it was a 503 error code. The web service was not able to respond to my request to download the logs.
Question: are you familiar enough with the internals of how RN 316 stores and manages its log? My thinking is it should have its own internal storage or memory to collect logs (and not use the hdd that make up the raid array). If so, I could remove and label all drives, power up the RN and connect to it and attempt to either download the logs from the web portal or ssh in and access the logs from the terminal. What do you think? Is this worth pursuing?
I do however, have logs from a day earlier, prior to the failure of the NAS. It’s important to note from the logs that disk3 has multiple warning reports.
Yes, the resync happened on its own. A scheduled scrub was running, which caused Disk2 to “lose sync” and that triggered the resync to automatically occur. This is what lead to Disk3s reported failure and then data volume dying.
Question: since the NAS reported that disk2 was offline and in a resync state, does this imply that any data that was on this disk to preserve the data volume has been lost or become invalid? It had only been resyncing for a 3 hours of the 27 hours it estimated for completion. I’m asking b/c I want to know if I can even consider disk2 as a candidate for restoring the data from the NAS.
I don’t have experience in data recovery. I am very comfortable working in Linux environment and running command lines. I’m not experienced with mdadm or btrf so i would need to study up on those filesystems.
The total volume size was at 18TB. I do see 20+ TB drives in the market that are reasonably priced. I prefer to proceed with your suggestion for cloning each drive as an image and store them on a single hard drive before attempting the data recovery process. At least I will then have images of each drive saved and can then decide how to proceed with data recovery using the original hard drives.
Question:
- You suggested I connect each drive to a PC running Western Digitals dashboard utility to check the status of the drive. Are you referring to this https://support-en.wd.com/app/answers/detailweb/a_id/31759/~/download%2C-install%2C-test-drive-and-update-firmware-using-western-digital
- Assuming tool recognizes the connected drive from the NAS, what should I be looking for?
- I understand I can run various tests, but I’m not sure I want to run them at the risk of stressing the drives further.
Here’s my proposed plan. Please let me know what you think:
- Purchase a 20TB+ hard drive for storing images of each cloned drive. Even if data recovery is impossible, I can utilize this drive for other purposes so I’m willing to make this expense in the beginning.
- Connect disk3 (drive that was in a FAILED state) to a PC running WD dashboard.
- If the drive is recognized by the dashboard, proceed with cloning the drive.
- Question: I’m learning that ddrescue for Linux is the best tool for cloning. Assuming I get this far, I’d need to reboot the machine and open my Linux environment. Is this the right tool and env to use for cloning? Alternatively, I can setup ddrescue-gui in advance to be prepared to do a clone in a windows environment.
Thank you!
halbertn wrote:I understand I can run various tests, but I’m not sure I want to run them at the risk of stressing the drives further.
That is a risk. You could limit the test to the short self test (which normally only takes about 2 minutes). If that fails, the drive has certainly failed. Though success wouldn't mean that the drive is ok, given the brevity of the test.
You can run essentially the same test using smartctl on a linux system.
halbertn wrote:
Question: are you familiar enough with the internals of how RN 316 stores and manages its log?
Many of the files in the zip extracted from the systemd journal. There are some others - mdstat.log is a copy of \etc\mdstat. lsblk is just the output of lsblk, etc. If you can log in via ssh (or tech support mode) you can create the zip file manually with rnutil create_system_log -o </path/filename> Use root as the username when you log in.
halbertn wrote:
Question: since the NAS reported that disk2 was offline and in a resync state, does this imply that any data that was on this disk to preserve the data volume has been lost or become invalid? It had only been resyncing for a 3 hours of the 27 hours it estimated for completion. I’m asking b/c I want to know if I can even consider disk2 as a candidate for restoring the data from the NAS.
It depends on whether you were doing a lot of other writes to the volume during the resync. It the scrub/resync is all that was happening, then the data being written to the disk would have been identical to what was already on the disk. Likely there was some other activity. But I think it is reasonable to try recovery with disk 2 if you can clone it. There is no harm in trying.
halbertn wrote:
Question: I’m learning that ddrescue for Linux is the best tool for cloning. Assuming I get this far, I’d need to reboot the machine and open my Linux environment. Is this the right tool and env to use for cloning? Alternatively, I can setup ddrescue-gui in advance to be prepared to do a clone in a windows environment.
You would need to install ddrescue. In general, installing stuff on the NAS will require some changes to the apt config.
More info on that is here:
Or you could just boot the PC up using a linux live boot flash drive.
StephenB
During the scrub/resync process, there should have been 0 writes to the volume. I wasn't accessing the drive or writing any new data to it. It should have been idling with the exception of whatever operation a scrub or sync would have been doing. From your answer and based on the overall health of the drive, Disk2 could be a likely candidate for recovering the data volume.
Question:
1. If my intention is to clone an image of every drive from my NAS onto a single harddrive, which filesystem should I put on this new drive that will be storing my images? I'm assuming ext4 as it only needs to store the images and be accessible in Linux (and windows).
2. Let's assume I'm able to access each drive and clone an image of them. Doesn't this imply that if I were to re-insert these drives back into their original slots in the RN 316 enclosure, then *maybe* the data volume may just come up and and be readable?
3. Suppose I can only clone 5 of the 6 drives because 1 of the drives is failing to be recognized or fails to be cloned. Then same question as 2 but with the 5 drives.
4. Assuming 2 or more drives are unaccessible, then I can declare that recovery on my own is impossible? Maybe I consider professional help, but they are unlikely to succeed considering that the drives have failed.
5. Suppose I do have 5 or 6 images cloned, and I'm not able to access the data volume in the RN 316 enclosure using the original drives. Then I attempt to access the data volume by rebuilding the raid array using the cloned images in a linux environment as a next step?
Thank you!
PS. How are you performing an inline response to my post? I can't find the option to do that.
halbertn wrote:
PS. How are you performing an inline response to my post? I can't find the option to do that.I use the quote tool to get a full quote of the original post, and then edit it. This is a bit buggy - I do need to shift to html sometimes to clean it up. It's tedious in the beginning, but like most things gets easier/faster with practice.
halbertn wrote:
Question:
1. If my intention is to clone an image of every drive from my NAS onto a single harddrive, which filesystem should I put on this new drive that will be storing my images? I'm assuming ext4 as it only needs to store the images and be accessible in Linux (and windows).You want something that can handle a 3 TB drive image, so ext4 or ntfs could both work. ext4 is probably the right option.
halbertn wrote:
2. Let's assume I'm able to access each drive and clone an image of them. Doesn't this imply that if I were to re-insert these drives back into their original slots in the RN 316 enclosure, then *maybe* the data volume may just come up and and be readable?Likely you would also need to forcibly assemble the RAID array (since it will be out of sync). BTRFS repair might also be needed.
But if you had 6 clones (instead of images), you could put these in the NAS and do those steps in the NAS.
halbertn wrote:
3. Suppose I can only clone 5 of the 6 drives because 1 of the drives is failing to be recognized or fails to be cloned. Then same question as 2 but with the 5 drives.You'll need at least 5 drives, as RAID can reconstruct the 6th from the remaining 5. Though if there are unreadable sectors on the source drives, that reconstruction will have some errors.
halbertn wrote:
4. Assuming 2 or more drives are unaccessible, then I can declare that recovery on my own is impossible?Yes.
halbertn wrote:
Maybe I consider professional help, but they are unlikely to succeed considering that the drives have failed.Likely true. Forensic recovery tools might be able to read sectors that dd rescue can't. Their recovery software might also be able to get some things back.
halbertn wrote:
5. Suppose I do have 5 or 6 images cloned, and I'm not able to access the data volume in the RN 316 enclosure using the original drives. Then I attempt to access the data volume by rebuilding the raid array using the cloned images in a linux environment as a next step?
Yes, though you could just directly move to working with the images. Sandshark's posts explain how to mount the images as disks in a ReadyNAS VM. You can substitute a generic Linux VM for that also.
halbertn wrote:Question: are you familiar enough with the internals of how RN 316 stores and manages its log? My thinking is it should have its own internal storage or memory to collect logs (and not use the hdd that make up the raid array). If so, I could remove and label all drives, power up the RN and connect to it and attempt to either download the logs from the web portal or ssh in and access the logs from the terminal. What do you think? Is this worth pursuing?
Readynas store absolutely nothing on the flash that's not there from the factory or an OS update. They run from and put all data, including configuration data, on the the drives. It's a separate partition of the drives -- one that is replicated on them all in RAID1. Removing the drives effectively returns the unit to factory configuration.
If your OS partition has become read-only because of the drive errors, that may be the source of your error in creating the log .zip.
Another way to get the log zip is using SSH command. Here is what I run daily on my RD5200 converted to OS6:
rnutil create_system_log -o /data/logs/log_RD5200A_$(date +%Y%m%d%H%M%S).zipYou, of course, need to choose your own name for the file, including a directory you currently have in place of "logs". Since you have no accessible volume on which to put it, you'll have to copy it to a mounted drive (USB or network) or use rsync to get it where you can access it. And if the OS partition is read-only, you'll have to make the target directory one that's on a mounted device that's not read-only.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
