This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
Hope to Get Data & Volume: Volume Degraded --> No Volume
First, I apologize, this is a long thread, but I wanted to provide as much context and details as possible.
I've reached out to free chat support for help but was wondering if anyone has any knowledge as to if it is possible to get a healthy volume, and my data, back?
As a note of context, while having deployed, and been successfully using, the RN516 since 2014, I'm not really knowledgable in RAID other than high level understanding. Hence the questions, and complexity, I inadvertantly added, as seen below.
Additional note, I now have 4x8TB WD Red Pro from the hardware compatiblity list on order, arriving soon.
Since 2014 I had been using my RN516 happily to provide network storage for my local media (photos, music, videos) and then as I added two more drives it grew into a backup of my PC drives when I scheduled a couple of robocopy jobs to backup my PC drives to shares on the RN516.
I started it in 2014 with one volume, multiple shares, in X-Raid on a 4TB WD Red NAS drive (3.5") and eventually expanded by added two 8TB WD Red NAS drives in 2017. Along the way, through present, I have updated the ReadyNas updated. It is currently at 6.10.6.
So setup at time of reallocated sector count notification, and before failure of volume:
Current state: Bay #1-4 TB WD Red [Compatible] Bay #2-8 TB WD Red [Compatible](increasing reallocated sector count) Bay #3-8 TB WD Red [Compatible] Bay #4-EMPTY Bay #5-EMPTY Bay #6-EMPTY
Recently in the Admin UI I noticed numerous messages indicating increasing reallocated sector count on disk 2 mentioning that this often indicates an impending failure of the disk, and be prepared to replace the disk to maintain redundancy.
With that in mind I went back to Western Digital and drilled down to 8TB WD Red NAS drives. I didn't know to check the compatibility chart at the time (I know now!!). I saw the WD Red Pro and Plus, with the price difference I went for the 8TB WD Red Plus NAS Drive (bummer as I would later learn.) Didn't know it wasn't compatible at the time.
Not knowing the intricate details of X-Raid/Raid, when the drive arrived I wasn't sure if I should hot swap it with disk #2, or if I should add it separately so it could copy data off of #2 (I now know to hot swap!!!). So unfortunately I made a big mistake and put the new 8TB into Bay #4 thinking the NAS would copy the data from failing disk #2. It took approximately three days for the Syncing and Reshaping of the data to complete, and that was with a large number of reallocated sector count entries in the logs for disk #2.
Current state: Bay #1-4 TB WD Red [Compatible] Bay #2-8 TB WD Red [Compatible](increasing reallocated sector count) Bay #3-8 TB WD Red [Compatible] Bay #4-8 TB WD Red [NOT Compatible] Bay #5-EMPTY Bay #6-EMPTY
Then I hot removed #2 from the NAS:
Current state:
Bay #1-4 TB WD Red [Compatible] Bay #2-EMPTY Bay #3-8 TB WD Red [Compatible] Bay #4-8 TB WD Red [NOT Compatible] Bay #5-EMPTY Bay #6-EMPTY
At this point I received a Volume Degraded message, which frankly freaked me out. After doing some web searching without a ton of luck I then made the next big mistake hoping to remove the volume degraded message. I put #2 back into the NAS and I ordered another WD Red Plus 8TB (not compatible, but still unknown to me.)
Current state: Bay #1-4 TB WD Red [Compatible] Bay #2-8 TB WD Red [Compatible](increasing reallocated sector count) Bay #3-8 TB WD Red [Compatible] Bay #4-8 TB WD Red [NOT Compatible] Bay #5-EMPTY Bay #6-EMPTY
This led to a crazy quick Resyncing followed by nearly two days of Restoring Data being showed on the display and in the log page in UI. Unfortunately in the last day of Reshaping Data disk #1 failed changing the volume from Degraded to DEAD.
Current state: Bay #1-4 TB WD Red [Compatible] - FAILED/DEAD Bay #2-8 TB WD Red [Compatible](increasing reallocated sector count) Bay #3-8 TB WD Red [Compatible] Bay #4-8 TB WD Red [NOT Compatible] Bay #5-EMPTY Bay #6-EMPTY
Once Restoring data finished the volume was changed to DEAD. I noticed I could see the mapped shares in Windows but there was a lot of data missing. In the Admin UI I tried to restore several of the folders by rollback but the three or four rollback attempts failed nearly immediately with an error message. I suspect this may have been a mistake to try as well because it did indicate it would remove later restores. I thought though that with disks #2,#3,#4 working (#2 with increasing warnings), that it may work. Nope.
This is when I think I made another major mistake. Hoping to see the full data in the shares I then rebooted the RN516. When it came back online there were no mapped shares and in the Admin UI it said there was no volume/shares and recommended I create a volume. I did not do that. I contacted support by chat.
Working with free chat support I uploaded my current logs and in them they determined that disk #4 was a newer, non-compatible WD Red NAS drive. :(. We tried rebooting the NAS, and also removing all drives with NAS off, and restarting. but the volume did not come back. I ordered four 8TB WD Red Pro compatible disks which should arrive in the next few days.
Once the drives arrive their advice is to swap disk #2 when the power is off and hope that the NAS rebuilds the volume, with the data. I'm worried as it seems to me like #2, while having an increasing reallocated sector count, may actually hold a copy, or subset, of the data on it, and disk #3 should be fine?
Unfortunately since taking the actions with support from the chat option, when I start up the NAS I receive the following notifications in the Admin UI:
Remove inactive volumes to use the disk. Disk #1,2,3,4.
Detected increasing reallocated sector count [####] on disk 2 messages
I can understand the issues with disk #1, #2, #4, but am confused as #3 should be okay. I even feel like #2 should be okay just with warnings...
The drives have arrived, so with the configuration backed up, I've gone ahead and replaced disk #2, per chat support recommendation, with one of the new drives.
Upon regular bootup, the volume/shares aren't shown, and there is top message int he admin page that says:
Remove inactive volumes to use the disk. Disk #3,4.
I didn't think it would resync - I.suspect the support person was mostly hoping.
I went for the 8TB WD Red Plus NAS Drive (bummer as I would later learn.) Didn't know it wasn't compatible at the time.
WD Red Plus drives are compatible, even though the current model number isn't on the HCL. Netgear is quite slow to add new models. In general NAS-purposed drives (WD Red Plus, IronWolf), and enterprise class are all compatible. The exception is the current WD Red line - which all use SMR technology.
I don't know who you talked to in support, but I believe they gave you wrong information. For some years now, Netgear mods have described the HCL as a more of a guide than a true compatibility list.
In any event, there's nothing in your narrative (other than the one comment from chat) that suggests the 8 TB WD Plus drives caused any of the problems you are seeing. You began by adding a disk instead of replacing it, then later on panicked and put a failing drive back into the NAS. Then a second drive (the oldest disk) failed. So it appears to me (w/o logs of course) that the problems are all caused by issues with two of the original drives.
Bay #1-4 TB WD Red [Compatible] Bay #2-EMPTY Bay #3-8 TB WD Red [Compatible] Bay #4-8 TB WD Red [NOT Compatible] Bay #5-EMPTY Bay #6-EMPTY
At this point I received a Volume Degraded message, which frankly freaked me out. After doing some web searching without a ton of luck I then made the next big mistake hoping to remove the volume degraded message.
I put #2 back into the NAS ...
Current state: Bay #1-4 TB WD Red [Compatible] - FAILED/DEAD Bay #2-8 TB WD Red [Compatible](increasing reallocated sector count) Bay #3-8 TB WD Red [Compatible] Bay #4-8 TB WD Red [NOT Compatible] Bay #5-EMPTY Bay #6-EMPTY
Once Restoring data finished the volume was changed to DEAD.
Was the volume dead at this point? Or just disk 1?
It looks like what happened is that disk 1 failed (perhaps stressed by by multiple resyncs). If it failed during the resync of disk 2, then the volume would have failed. If it failed after the resync completed, then the volume would not have failed.
I'm thinking disk 1 failed after the resync completed, and that the volume wasn't dead. If it was dead, you would not have seen any data on it from Windows (and in fact you could). The disk errors on disk 2 could have resulted in the missing data you saw.
Unfortunately since taking the actions with support from the chat option, when I start up the NAS I receive the following notifications in the Admin UI:
Remove inactive volumes to use the disk. Disk #1,2,3,4.
Detected increasing reallocated sector count [####] on disk 2 messages
I can understand the issues with disk #1, #2, #4, but am confused as #3 should be okay. I even feel like #2 should be okay just with warnings...
Is it still showing disk 1 as dead?
RAID spreads the data evenly across all four disks. One potential problem is when data isn't written for some reason, the four disks can get out of sync. (There are counters on each disk, so this can be detected). When this happens, the NAS will give you the inactive volume message.
Whether the errors on disk 2 should be understood "warnings" depends on how many there are - and also how rapidly they are increasing. Can you let us know what the counts are?
Honestly, it depends on how much your data is worth to you. You've ended up in a bit of a mess, and it might not be easy to untangle.
One possibility here is that you will need to start over - doing a factory reset with all the new drives in place. Then gather what you can of the old data from PCs, USB drives, or whatever other sources you have.
First,I wouldn't replace disk 2 at this point.
Second, if your data is worth the cost, then you should get professional data recovery. Netgear offers a recovery service, you can see details here: https://kb.netgear.com/69/ReadyNAS-Data-Recovery-Diagnostics-Scope-of-Service Note that doing more on your own might make things worse. If professional recovery is the right path, then you should start down it right away.
If professional recovery is unaffordable, then I'd suggest that you should get a new set of logs, and ask one of the mods (perhaps @Marc_V ) to review both the original set you provided support, and also the current ones. If you have case numbers, you could maybe also give them that info, as they can review the case info that support has.
Another thing you could try is
power down
remove disk 1
reboot in READ-ONLY mode (ask if you don't know how to do that).
and see you can access any data. If you can, then power down immediately, and post back.
Re: Hope to Get Data & Volume: Volume Degraded --> No Volume
Hello Stephen,
I really appreciate your analysis and thoughtful response and am happy to hear that disk #4 may be alright. It was surprising to me that it did the sync and reshaping and showed as healthy considering chat support had indicated it is not counting as one of the disks in the RAID.
I very much wish I had posted before replacing disks. Esp. since many of the models on the compatibility list for the RN516 don't even appear to be sold anymore.
Regarding your notes about the value of the data to me, I'm less worried about the data backup of my PC as my PC is still functioning fine. I have most, but unfortunately not all, of the photos, music, videos backed up in Google Drive so I can work on a path to download that while I pursue a path to recovery if the volume cannot be rebuilt.
To answer your questions:
Was the volume dead at this point? Or just disk 1?
It looks like the volume failed and then disk #1 failed., note I've removed snapshot messages and all the increasing reallocated sector count warnings.
Feb 15, 2022 12:15:22 PM Disk: Disk in channel 2 (Internal) changed state from RESYNC to ONLINE. Feb 14, 2022 09:52:41 AM Disk: Disk in channel 1 (Internal) changed state from ONLINE to FAILED. Feb 14, 2022 09:46:10 AM Volume: Volume data health changed from Degraded to Dead. Feb 14, 2022 01:00:11 AM Volume: Volume data is Degraded. Feb 13, 2022 03:15:09 PM Volume: Resyncing started for Volume data. Feb 13, 2022 03:11:55 PM Disk: Disk Model:WDC WD80EFZX-68UW8N0 Serial:VJHAWEHX was added to Channel 2 of the head unit. Feb 13, 2022 02:36:30 PM Volume: The resync operation finished on volume data. However, the volume is still degraded. Feb 13, 2022 02:36:20 PM Disk: Disk Model:WDC WD80EFZX-68UW8N0 Serial:VJHAWEHX was removed from Channel 2 of the head unit. Feb 13, 2022 02:36:19 PM Volume: Volume data health changed from Redundant to Degraded. Feb 13, 2022 02:36:19 PM Volume: Resyncing started for Volume data. Feb 12, 2022 01:42:14 PM Volume: Volume data is resynced. Feb 09, 2022 01:51:08 PM Disk: Disk in channel 4 (Internal) changed state from RESYNC to ONLINE. Feb 09, 2022 01:48:50 PM Volume: Resyncing started for Volume data. Feb 09, 2022 01:48:37 PM Disk: Disk Model:WDC WD80EFBX-68AZZN0 Serial:VR0A0HPK was added to Channel 4 of the head unit.
Is it still showing disk 1 as dead?
Yes, but interestingingly I get two immediate inconsistent messages on boot. Note, I've been keeping the NAS off until the new disks arrive and we determine the best path forward. 01:27:11 PM 02/17/2022 Remove inactive volumes to use the disk. Disk #2,3,4. 01:27:11 PM 02/17/2022 Remove inactive volumes to use the disk. Disk #1,2,3,4.
Whether the errors on disk 2 should be understood "warnings" depends on how many there are - and also how rapidly they are increasing. Can you let us know what the counts are?
They have definitely increased significantly after putting in disk #4 then removing and reinserting disk #2.
Feb 17, 2022 01:38:34 PM Disk: Detected increasing reallocated sector count: [1921] on disk 2 (Internal) [WDC WD80EFZX-68UW8N0 VJHAWEHX] 254 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy. Feb 17, 2022 01:30:32 PM Disk: Detected increasing reallocated sector count: [1917] on disk 2 (Internal) [WDC WD80EFZX-68UW8N0 VJHAWEHX] 253 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy. Feb 17, 2022 01:28:31 PM Disk: Detected increasing reallocated sector count: [1914] on disk 2 (Internal) [WDC WD80EFZX-68UW8N0 VJHAWEHX] 252 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
I very much appreicate the tips on data recovery, and pointing me in the right direction for services as well as a note for logs. Luckily the NAS itself boots up fine and I can still download logs.
Re: Hope to Get Data & Volume: Volume Degraded --> No Volume
Hello Stephen,
I appreicate it, I went ahead and did that test removing disk #2 while powered down and then booting into Volume ReadOnly mode with no luck. The new drives should arrive today before noon so I am thinking I will try to do the replacement of disk #2 as per chat support and if it doesn't rebuild the volume I guess I will put in all of the new drives and create a new volume and shares and copy back what I can from my PC and Google Drive.
@StephenB Quick question, is there a way to save the configuration I had as far as shares, permissions, etc. So I don't have to recreate all of that again?
Re: Hope to Get Data & Volume: Volume Degraded --> No Volume
Ooh, I found Configuration Backup under System --> Settings.
Going to do that now though I'm not sure it will capture my actual configuration before the failure since currently when I go to the NAS it wants me to create a volume.
Re: Hope to Get Data & Volume: Volume Degraded --> No Volume
The drives have arrived, so with the configuration backed up, I've gone ahead and replaced disk #2, per chat support recommendation, with one of the new drives.
Upon regular bootup, the volume/shares aren't shown, and there is top message int he admin page that says:
Remove inactive volumes to use the disk. Disk #3,4.
It seems the volume is marked as inactive and is still associated with disks 3 and 4, even though those disks should be okay... the only disks that had problems were #1 and #2, with questionable note that the model for #4 wasn't on the HCL even though it built just fine and was healthy before all these problems occurred.
Re: Hope to Get Data & Volume: Volume Degraded --> No Volume
Alright, so I've tried cold swapping #1 and #2 independantly and at the same time with no luck. It is not rebuilding the volume.
Interestingly when I tried one last time to put the original #1 and #2 back in (also as a cold swap) on bootup it started a recover data process. However with #1 dead and #2 with very high and frequent increasing reallocated sector count warnings, I don't think it is worth going through the 2-3 days of recovery when likely #2 will fail anyways.
I still have no idea why #3 and #4 aren't of any help for the volume coming online since #3 had no errors and #4 is brand new, worked fine in the NAS after that initial three day syncing/reshaping data activity and the only outstanding bit with it is that it isn't in the HCL.
I'm going to toss the towel in. Put in all four new drives (replacing all the old drives) with the power off and then start up and recreate the volume and go about copying back my data from PC and Google Drive where it exists.
I'm hoping I'll be able to restart the the configuration backup I just took but since that was with the volume in a failed/dead state, I'm not sure I can count on that either.
Thank you so much for your help @StephenB . I'm bummed I did not get the volume to a working state but am okay moving on. Your time, help, and thoughtful analysis and responses have been greatly appreciated.
The drives have arrived, so with the configuration backed up, I've gone ahead and replaced disk #2, per chat support recommendation, with one of the new drives.
Upon regular bootup, the volume/shares aren't shown, and there is top message int he admin page that says:
Remove inactive volumes to use the disk. Disk #3,4.
I didn't think it would resync - I.suspect the support person was mostly hoping.
Re: Hope to Get Data & Volume: Volume Degraded --> No Volume
So I put in all four new drives and it took me to the initial page to set up everything up as it began the sync process for the new drives.
That's the good news, now the bummer.
As if to add a little salt into the fresh wound my configuration backup failed to restore, saying the volume name(s) has to be identical. I see no place to change the volume name and was never manually doing that. I think X-RAID managed that. They were data, data-0, and data-1.
So unfortunately resetting up all the shares, permissions, protocols, etc. while the sync job for the new drives takes place. 😞
Seems a little silly as I had access set up for Windows, Linux, and other devices by permissions and I hadn't reviewed the permissions and enabled protocols for each share in some time. 😞 I did recreate the same exact share names and the configuration restore still failed. It really is tied to the volume or old array.
Anyways, thanks again for the assistance. Complete restart for me which leaves a little bit of a bitter taste in my mouth after 8 years of great service and dependability from the RN516. I had choosen it after a few years of using a Buffalo Terastation and I can say the RN516 was a nice upgrade and way more pleasant to set up and use. To be fair I did make the mistake, and cause my own problems, of adding the new disk, instead of swapping with disk #2 that had warnings. I had thought that would help it get any data off of the disk with warnings, but obviously I understand now that the other drives would have been enough between them and the space reserved for data protection. Still a big bummer to not be able to recover any data despite 3 working drives (one with warnings, one with no warnings whatsoever, and one that was healthy and synced fine, worked well, but was not on HCL). I guess I'm just really bummed it couldn't even restore my share settings (permissions and protocols for each share) and cloud backup of specific paths.
In either event I'll get things set back up and most of the data back in place and I'm sure time will numb the disappointment at the end.
As if to add a little salt into the fresh wound my configuration backup failed to restore, saying the volume name(s) has to be identical. I see no place to change the volume name and was never manually doing that. I think X-RAID managed that. They were data, data-0, and data-1.
Did you choose FlexRAID or X-RAID when you did the new install?
X-RAID uses data for the volume name (data-0 and data-1 are RAID group names, not quite the same thing).
Re: Hope to Get Data & Volume: Volume Degraded --> No Volume
But when the system sees drives with parts of raid groups data-0, data-1, etc that BTRFS can't put together into a usable volume, the GUI often shows them as volumes as well, though there is nothing on them you can access.
Re: Hope to Get Data & Volume: Volume Degraded --> No Volume
Great question @StephenB , I used X-RAID. In the volumes screen I would see the existing volumes name on the left side. Additionally when I'd mouse over a drive bay on the volumes screen it would show which voume the disk was in.
@Sandshark That may explain part of the problem on the configuration restore. In either event I've got my new volume with all new drives. They are synced and I am copying all of my data back to the RN516. I have only a few days of data delta between PC and NAS. The backup to Google Drive should also include almost everything so my loss is negible, just the time, frustration, and cash outlay in new drives.
Thanks to both of you for your response, support. and help. Greatly appreciated.
