× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Hope to Get Data & Volume: Volume Degraded --> No Volume

Augestflex
Aspirant

Hope to Get Data & Volume: Volume Degraded --> No Volume

First, I apologize, this is a long thread, but I wanted to provide as much context and details as possible.

 

I've reached out to free chat support for help but was wondering if anyone has any knowledge as to if it is possible to get a healthy volume, and my data, back?

 

As a note of context, while having deployed, and been successfully using, the RN516 since 2014, I'm not really knowledgable in RAID other than high level understanding. Hence the questions, and complexity, I inadvertantly added, as seen below.

 

Additional note, I now have 4x8TB WD Red Pro from the hardware compatiblity list on order, arriving soon.

 

Since 2014 I had been using my RN516 happily to provide network storage for my local media (photos, music, videos) and then as I added two more drives it grew into a backup of my PC drives when I scheduled a couple of robocopy jobs to backup my PC drives to shares on the RN516.

 

I started it in 2014 with one volume, multiple shares, in X-Raid on a 4TB WD Red NAS drive (3.5") and eventually expanded by added two 8TB WD Red NAS drives in 2017. Along the way, through present, I have updated the ReadyNas updated. It is currently at 6.10.6.


So setup at time of reallocated sector count notification, and before failure of volume:

Current state:
Bay #1-4 TB WD Red [Compatible]
Bay #2-8 TB WD Red [Compatible](increasing reallocated sector count)
Bay #3-8 TB WD Red [Compatible]
Bay #4-EMPTY
Bay #5-EMPTY
Bay #6-EMPTY

 

Recently in the Admin UI I noticed numerous messages indicating increasing reallocated sector count on disk 2 mentioning that this often indicates an impending failure of the disk, and be prepared to replace the disk to maintain redundancy.

 

With that in mind I went back to Western Digital and drilled down to 8TB WD Red NAS drives. I didn't know to check the compatibility chart at the time (I know now!!). I saw the WD Red Pro and Plus, with the price difference I went for the 8TB WD Red Plus NAS Drive (bummer as I would later learn.) Didn't know it wasn't compatible at the time.

 

Not knowing the intricate details of X-Raid/Raid, when the drive arrived I wasn't sure if I should hot swap it with disk #2, or if I should add it separately so it could copy data off of #2 (I now know to hot swap!!!). So unfortunately I made a big mistake and put the new 8TB into Bay #4 thinking the NAS would copy the data from failing disk #2.
It took approximately three days for the Syncing and Reshaping of the data to complete, and that was with a large number of reallocated sector count entries in the logs for disk #2.

 

Current state:
Bay #1-4 TB WD Red [Compatible]
Bay #2-8 TB WD Red [Compatible](increasing reallocated sector count)
Bay #3-8 TB WD Red [Compatible]
Bay #4-8 TB WD Red [NOT Compatible]
Bay #5-EMPTY
Bay #6-EMPTY

 

Then I hot removed #2 from the NAS:

 

Current state:

Bay #1-4 TB WD Red [Compatible]
Bay #2-EMPTY
Bay #3-8 TB WD Red [Compatible]
Bay #4-8 TB WD Red [NOT Compatible]
Bay #5-EMPTY
Bay #6-EMPTY

 

At this point I received a Volume Degraded message, which frankly freaked me out.
After doing some web searching without a ton of luck I then made the next big mistake hoping to remove the volume degraded message.
I put #2 back into the NAS and I ordered another WD Red Plus 8TB (not compatible, but still unknown to me.)

 

Current state:
Bay #1-4 TB WD Red [Compatible]
Bay #2-8 TB WD Red [Compatible](increasing reallocated sector count)
Bay #3-8 TB WD Red [Compatible]
Bay #4-8 TB WD Red [NOT Compatible]
Bay #5-EMPTY
Bay #6-EMPTY

 

This led to a crazy quick Resyncing followed by nearly two days of Restoring Data being showed on the display and in the log page in UI.
Unfortunately in the last day of Reshaping Data disk #1 failed changing the volume from Degraded to DEAD.

 

Current state:
Bay #1-4 TB WD Red [Compatible] - FAILED/DEAD
Bay #2-8 TB WD Red [Compatible](increasing reallocated sector count)
Bay #3-8 TB WD Red [Compatible]
Bay #4-8 TB WD Red [NOT Compatible]
Bay #5-EMPTY
Bay #6-EMPTY

 

Once Restoring data finished the volume was changed to DEAD.
I noticed I could see the mapped shares in Windows but there was a lot of data missing. In the Admin UI I tried to restore several of the folders by rollback but the three or four rollback attempts failed nearly immediately with an error message. I suspect this may have been a mistake to try as well because it did indicate it would remove later restores. I thought though that with disks #2,#3,#4 working (#2 with increasing warnings), that it may work. Nope.

 

This is when I think I made another major mistake. Hoping to see the full data in the shares I then rebooted the RN516. When it came back online there were no mapped shares and in the Admin UI it said there was no volume/shares and recommended I create a volume. I did not do that. I contacted support by chat.

 

Working with free chat support I uploaded my current logs and in them they determined that disk #4 was a newer, non-compatible WD Red NAS drive. :(. We tried rebooting the NAS, and also removing all drives with NAS off, and restarting. but the volume did not come back.
I ordered four 8TB WD Red Pro compatible disks which should arrive in the next few days.


Once the drives arrive their advice is to swap disk #2 when the power is off and hope that the NAS rebuilds the volume, with the data.
I'm worried as it seems to me like #2, while having an increasing reallocated sector count, may actually hold a copy, or subset, of the data on it, and disk #3 should be fine?

 

Unfortunately since taking the actions with support from the chat option, when I start up the NAS I receive the following notifications in the Admin UI:

  • Remove inactive volumes to use the disk. Disk #1,2,3,4.
  • Detected increasing reallocated sector count [####] on disk 2 messages

 

I can understand the issues with disk #1, #2, #4, but am confused as #3 should be okay. I even feel like #2 should be okay just with warnings...

Thoughts on how to proceed?

Model: RN51600|ReadyNAS 516 6-Bay Diskless
Message 1 of 13

Accepted Solutions
StephenB
Guru

Re: Hope to Get Data & Volume: Volume Degraded --> No Volume


@Augestflex wrote:

The drives have arrived, so with the configuration backed up, I've gone ahead and replaced disk #2, per chat support recommendation, with one of the new drives.

 

Upon regular bootup, the volume/shares aren't shown, and there is top message int he admin page that says:

Remove inactive volumes to use the disk. Disk #3,4.

 


I didn't think it would resync - I.suspect the support person was mostly hoping.

View solution in original post

Message 9 of 13

All Replies
StephenB
Guru

Re: Hope to Get Data & Volume: Volume Degraded --> No Volume

I wish you'd posted here before you started replacing disks.


@Augestflex wrote:

I went for the 8TB WD Red Plus NAS Drive (bummer as I would later learn.) Didn't know it wasn't compatible at the time.

 


WD Red Plus drives are compatible, even though the current model number isn't on the HCL.  Netgear is quite slow to add new models.  In general NAS-purposed drives (WD Red Plus, IronWolf), and enterprise class are all compatible.  The exception is the current WD Red line - which all use SMR technology.

 

I don't know who you talked to in support, but I believe they gave you wrong information.  For some years now, Netgear mods have described the HCL as a more of a guide than a true compatibility list.

 

In any event, there's nothing in your narrative (other than the one comment from chat) that suggests the 8 TB WD Plus drives caused any of the problems you are seeing.  You began by adding a disk instead of replacing it, then later on panicked and put a failing drive back into the NAS.  Then a second drive (the oldest disk) failed.  So it appears to me (w/o logs of course) that the problems are all caused by issues with two of the original drives.

 


@Augestflex wrote:

 

Then I hot removed #2 from the NAS:

 

Current state:

Bay #1-4 TB WD Red [Compatible]
Bay #2-EMPTY
Bay #3-8 TB WD Red [Compatible]
Bay #4-8 TB WD Red [NOT Compatible]
Bay #5-EMPTY
Bay #6-EMPTY

 

At this point I received a Volume Degraded message, which frankly freaked me out.
After doing some web searching without a ton of luck I then made the next big mistake hoping to remove the volume degraded message.

 

I put #2 back into the NAS ...

 

Current state:
Bay #1-4 TB WD Red [Compatible] - FAILED/DEAD
Bay #2-8 TB WD Red [Compatible](increasing reallocated sector count)
Bay #3-8 TB WD Red [Compatible]
Bay #4-8 TB WD Red [NOT Compatible]
Bay #5-EMPTY
Bay #6-EMPTY

 

Once Restoring data finished the volume was changed to DEAD.

Was the volume dead at this point?  Or just disk 1?

 

It looks like what happened is that disk 1 failed (perhaps stressed by by multiple resyncs).  If it failed during the resync of disk 2, then the volume would have failed.  If it failed after the resync completed, then the volume would not have failed.

 

I'm thinking disk 1 failed after the resync completed, and that the volume wasn't dead.  If it was dead, you would not have seen any data on it from Windows (and in fact you could).  The disk errors on disk 2 could have resulted in the missing data you saw.

 


@Augestflex wrote:

 

Unfortunately since taking the actions with support from the chat option, when I start up the NAS I receive the following notifications in the Admin UI:

  • Remove inactive volumes to use the disk. Disk #1,2,3,4.
  • Detected increasing reallocated sector count [####] on disk 2 messages

 

I can understand the issues with disk #1, #2, #4, but am confused as #3 should be okay. I even feel like #2 should be okay just with warnings...


Is it still showing disk 1 as dead?

 

RAID spreads the data evenly across all four disks.  One potential problem is when data isn't written for some reason, the four disks can get out of sync.  (There are counters on each disk, so this can be detected).  When this happens, the NAS will give you the inactive volume message.

 

Whether the errors on disk 2 should be understood "warnings" depends on how many there are - and also how rapidly they are increasing.  Can you let us know what the counts are?

 


@Augestflex wrote:

 

Thoughts on how to proceed?


Honestly, it depends on how much your data is worth to you.  You've ended up in a bit of a mess, and it might not be easy to untangle.  

 

One possibility here is that you will need to start over - doing a factory reset with all the new drives in place.  Then gather what you can of the old data from PCs, USB drives, or whatever other sources you have.

 

First,I wouldn't replace disk 2 at this point.

 

Second, if your data is worth the cost, then you should get professional data recovery.   Netgear offers a recovery service, you can see details here: https://kb.netgear.com/69/ReadyNAS-Data-Recovery-Diagnostics-Scope-of-Service  Note that doing more on your own might make things worse.  If professional recovery is the right path, then you should start down it right away.

 

If professional recovery is unaffordable, then I'd suggest that you should get a new set of logs, and ask one of the mods (perhaps @Marc_V ) to review both the original set you provided support, and also the current ones.  If you have case numbers, you could maybe also give them that info, as they can review the case info that support has.

 

Another thing you could try is 

  • power down
  • remove disk 1
  • reboot in READ-ONLY mode (ask if you don't know how to do that).

and see you can access any data.  If you can, then power down immediately, and post back.

 

 

 

 

 

 

 

 

Message 2 of 13
Augestflex
Aspirant

Re: Hope to Get Data & Volume: Volume Degraded --> No Volume

Hello Stephen,

 

I really appreciate your analysis and thoughtful response and am happy to hear that disk #4 may be alright. It was surprising to me that it did the sync and reshaping and showed as healthy considering chat support had indicated it is not counting as one of the disks in the RAID.

 

I very much wish I had posted before replacing disks. Esp. since many of the models on the compatibility list for the RN516 don't even appear to be sold anymore.

 

Regarding your notes about the value of the data to me, I'm less worried about the data backup of my PC as my PC is still functioning fine. I have most, but unfortunately not all, of the photos, music, videos backed up in Google Drive so I can work on a path to download that while I pursue a path to recovery if the volume cannot be rebuilt.

 

To answer your questions:

  • Was the volume dead at this point?  Or just disk 1?

It looks like the volume failed and then disk #1 failed., note I've removed snapshot messages and all the increasing reallocated sector count warnings.

Feb 15, 2022 12:15:22 PM Disk: Disk in channel 2 (Internal) changed state from RESYNC to ONLINE.
Feb 14, 2022 09:52:41 AM Disk: Disk in channel 1 (Internal) changed state from ONLINE to FAILED.
Feb 14, 2022 09:46:10 AM Volume: Volume data health changed from Degraded to Dead.
Feb 14, 2022 01:00:11 AM Volume: Volume data is Degraded.
Feb 13, 2022 03:15:09 PM Volume: Resyncing started for Volume data.
Feb 13, 2022 03:11:55 PM Disk: Disk Model:WDC WD80EFZX-68UW8N0 Serial:VJHAWEHX was added to Channel 2 of the head unit.
Feb 13, 2022 02:36:30 PM Volume: The resync operation finished on volume data. However, the volume is still degraded.
Feb 13, 2022 02:36:20 PM Disk: Disk Model:WDC WD80EFZX-68UW8N0 Serial:VJHAWEHX was removed from Channel 2 of the head unit.
Feb 13, 2022 02:36:19 PM Volume: Volume data health changed from Redundant to Degraded.
Feb 13, 2022 02:36:19 PM Volume: Resyncing started for Volume data.
Feb 12, 2022 01:42:14 PM Volume: Volume data is resynced.
Feb 09, 2022 01:51:08 PM Disk: Disk in channel 4 (Internal) changed state from RESYNC to ONLINE.
Feb 09, 2022 01:48:50 PM Volume: Resyncing started for Volume data.
Feb 09, 2022 01:48:37 PM Disk: Disk Model:WDC WD80EFBX-68AZZN0 Serial:VR0A0HPK was added to Channel 4 of the head unit.

 

  • Is it still showing disk 1 as dead?

Yes, but interestingingly I get two immediate inconsistent messages on boot. Note, I've been keeping the NAS off until the new disks arrive and we determine the best path forward.
01:27:11 PM 02/17/2022 Remove inactive volumes to use the disk. Disk #2,3,4.
01:27:11 PM 02/17/2022 Remove inactive volumes to use the disk. Disk #1,2,3,4.

 

  • Whether the errors on disk 2 should be understood "warnings" depends on how many there are - and also how rapidly they are increasing.  Can you let us know what the counts are?

They have definitely increased significantly after putting in disk #4 then removing and reinserting disk #2.

Feb 17, 2022 01:38:34 PM Disk: Detected increasing reallocated sector count: [1921] on disk 2 (Internal) [WDC WD80EFZX-68UW8N0 VJHAWEHX] 254 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
Feb 17, 2022 01:30:32 PM Disk: Detected increasing reallocated sector count: [1917] on disk 2 (Internal) [WDC WD80EFZX-68UW8N0 VJHAWEHX] 253 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
Feb 17, 2022 01:28:31 PM Disk: Detected increasing reallocated sector count: [1914] on disk 2 (Internal) [WDC WD80EFZX-68UW8N0 VJHAWEHX] 252 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.

 

I very much appreicate the tips on data recovery, and pointing me in the right direction for services as well as a note for logs. Luckily the NAS itself boots up fine and I can still download logs.

 


@StephenB wrote:

Another thing you could try is 

  • power down
  • remove disk 1
  • reboot in READ-ONLY mode (ask if you don't know how to do that).

and see you can access any data.  If you can, then power down immediately, and post back.

 


I've gone ahead and shutdown, removed disk #1 and restarted in Volume ReadOnly mode, and then shut back down.

None of the shares were readable.

 

Thank you so much for your help so far it is greatly appreciated and I continue to increase my knowledge on the RN516 and RAID.

Message 3 of 13
StephenB
Guru

Re: Hope to Get Data & Volume: Volume Degraded --> No Volume


@Augestflex wrote:

 


@StephenB wrote:

Another thing you could try is 

  • power down
  • remove disk 1
  • reboot in READ-ONLY mode (ask if you don't know how to do that).

and see you can access any data.  If you can, then power down immediately, and post back.

 


I've gone ahead and shutdown, removed disk #1 and restarted in Volume ReadOnly mode, and then shut back down.

None of the shares were readable.

 


Maybe repeat this test with drive 1 installed, but drive 2 removed.  If it works, then immediately copy the files you are missing.
 

Message 4 of 13
Augestflex
Aspirant

Re: Hope to Get Data & Volume: Volume Degraded --> No Volume

Hello Stephen,

 

I appreicate it, I went ahead and did that test removing disk #2 while powered down and then booting into Volume ReadOnly mode with no luck. The new drives should arrive today before noon so I am thinking I will try to do the replacement of disk #2 as per chat support and if it doesn't rebuild the volume I guess I will put in all of the new drives and create a new volume and shares and copy back what I can from my PC and Google Drive.

 

@StephenB  Quick question, is there a way to save the configuration I had as far as shares, permissions, etc. So I don't have to recreate all of that again?

 

Thanks!

Message 5 of 13
Augestflex
Aspirant

Re: Hope to Get Data & Volume: Volume Degraded --> No Volume

Ooh, I found Configuration Backup under System --> Settings.

 

Going to do that now though I'm not sure it will capture my actual configuration before the failure since currently when I go to the NAS it wants me to create a volume.

Message 6 of 13
Augestflex
Aspirant

Re: Hope to Get Data & Volume: Volume Degraded --> No Volume

The drives have arrived, so with the configuration backed up, I've gone ahead and replaced disk #2, per chat support recommendation, with one of the new drives.

 

Upon regular bootup, the volume/shares aren't shown, and there is top message int he admin page that says:

Remove inactive volumes to use the disk. Disk #3,4.

 

It seems the volume is marked as inactive and is still associated with disks 3 and 4, even though those disks should be okay... the only disks that had problems were #1 and #2, with questionable note that the model for #4 wasn't on the HCL even though it built just fine and was healthy before all these problems occurred.

Message 7 of 13
Augestflex
Aspirant

Re: Hope to Get Data & Volume: Volume Degraded --> No Volume

Alright, so I've tried cold swapping #1 and #2 independantly and at the same time with no luck. It is not rebuilding the volume.

 

Interestingly when I tried one last time to put the original #1 and #2 back in (also as a cold swap) on bootup it started a recover data process. However with #1 dead and #2 with very high and frequent increasing reallocated sector count warnings, I don't think it is worth going through the 2-3 days of recovery when likely #2 will fail anyways.

I still have no idea why #3 and #4 aren't of any help for the volume coming online since #3 had no errors and #4 is brand new, worked fine in the NAS after that initial three day syncing/reshaping data activity and the only outstanding bit with it is that it isn't in the HCL.

 

I'm going to toss the towel in. Put in all four new drives (replacing all the old drives) with the power off and then start up and recreate the volume and go about copying back my data from PC and Google Drive where it exists.

 

I'm hoping I'll be able to restart the the configuration backup I just took but since that was with the volume in a failed/dead state, I'm not sure I can count on that either.

 

Thank you so much for your help @StephenB . I'm bummed I did not get the volume to a working state but am okay moving on. Your time, help, and thoughtful analysis and responses have been greatly appreciated.

Message 8 of 13
StephenB
Guru

Re: Hope to Get Data & Volume: Volume Degraded --> No Volume


@Augestflex wrote:

The drives have arrived, so with the configuration backed up, I've gone ahead and replaced disk #2, per chat support recommendation, with one of the new drives.

 

Upon regular bootup, the volume/shares aren't shown, and there is top message int he admin page that says:

Remove inactive volumes to use the disk. Disk #3,4.

 


I didn't think it would resync - I.suspect the support person was mostly hoping.

Message 9 of 13
Augestflex
Aspirant

Re: Hope to Get Data & Volume: Volume Degraded --> No Volume

So I put in all four new drives and it took me to the initial page to set up everything up as it began the sync process for the new drives.

That's the good news, now the bummer.

As if to add a little salt into the fresh wound my configuration backup failed to restore, saying the volume name(s) has to be identical. I see no place to change the volume name and was never manually doing that. I think X-RAID managed that. They were data, data-0, and data-1.

 

So unfortunately resetting up all the shares, permissions, protocols, etc. while the sync job for the new drives takes place. 😞 

 

Seems a little silly as I had access set up for Windows, Linux, and other devices by permissions and I hadn't reviewed the permissions and enabled protocols for each share in some time. 😞  I did recreate the same exact share names and the configuration restore still failed. It really is tied to the volume or old array. 

 

Anyways, thanks again for the assistance. Complete restart for me which leaves a little bit of a bitter taste in my mouth after 8 years of great service and dependability from the RN516. I had choosen it after a few years of using a Buffalo Terastation and I can say the RN516 was a nice upgrade and way more pleasant to set up and use. To be fair I did make the mistake, and cause my own problems, of adding the new disk, instead of swapping with disk #2 that had warnings. I had thought that would help it get any data off of the disk with warnings, but obviously I understand now that the other drives would have been enough between them and the space reserved for data protection. Still a big bummer to not be able to recover any data despite 3 working drives (one with warnings, one with no warnings whatsoever, and one that was healthy and synced fine, worked well, but was not on HCL). I guess I'm just really bummed it couldn't even restore my share settings (permissions and protocols for each share) and cloud backup of specific paths. 

 

In either event I'll get things set back up and most of the data back in place and I'm sure time will numb the disappointment at the end.

/rant off

Message 10 of 13
StephenB
Guru

Re: Hope to Get Data & Volume: Volume Degraded --> No Volume


@Augestflex wrote:

 

As if to add a little salt into the fresh wound my configuration backup failed to restore, saying the volume name(s) has to be identical. I see no place to change the volume name and was never manually doing that. I think X-RAID managed that. They were data, data-0, and data-1.

Did you choose FlexRAID or X-RAID when you did the new install?

 

X-RAID uses data for the volume name (data-0 and data-1 are RAID group names, not quite the same thing).

Message 11 of 13
Sandshark
Sensei

Re: Hope to Get Data & Volume: Volume Degraded --> No Volume

But when the system sees drives with parts of raid groups data-0, data-1, etc that BTRFS can't put together into a usable volume, the GUI often shows them as volumes as well, though there is nothing on them you can access.

Message 12 of 13
Augestflex
Aspirant

Re: Hope to Get Data & Volume: Volume Degraded --> No Volume

Great question @StephenB , I used X-RAID. In the volumes screen I would see the existing volumes name on the left side. Additionally when I'd mouse over a drive bay on the volumes screen it would show which voume the disk was in. 

 

@Sandshark That may explain part of the problem on the configuration restore. In either event I've got my new volume with all new drives. They are synced and I am copying all of my data back to the RN516. I have only a few days of data delta  between PC and NAS. The backup to Google Drive should also include almost everything so my loss is negible, just the time, frustration, and cash outlay in new drives.

 

Thanks to both of you for your response, support. and help. Greatly appreciated.

 

Message 13 of 13
Top Contributors
Discussion stats
  • 12 replies
  • 1837 views
  • 2 kudos
  • 3 in conversation
Announcements