× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

ReadyNAS 104 Firmware 6.10.7 - All four disks inactive following hot-swap of failed vol 2

Whiskeypod
Aspirant

ReadyNAS 104 Firmware 6.10.7 - All four disks inactive following hot-swap of failed vol 2

I have four disks installed, 4 x 4TB. X-raid. One volume.
For a while the system indicated that disk 2 was degraded ("Volume: Volume data is Degraded" and "Disk: Detected increasing ATA error count: [48040] on disk 2 (Internal) [ST4000VN000-1H4168, Z302KH2D] 19342 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy".)

My Data was a fraction of the space available, and the free space was in excess of 70%. (you can see from the logs that resync took about 90 minutes.)
My NAS does not have a backup, I purchased it with four drives AS a storage and back up, as the data was meant to be safe from failures with four disks.

Yesterday, I hot-swapped drive 2 with a new one, as per the instructions.

Everything seemed to work fine. Disk 2 resynced, and then immediately the logs show that "Volume: Volume data health changed from Degraded to  Dead",  disk 2 went online, and then there is a notification that disk in channel 4 has changed from online to failed, and the volume is finally changed to the volume data health from Dead to Inactive.

The system then went on about deleting every single snapshot. (there is 18 pages of this) as now the system thinks it has run out of space.


Here is the relevant part of the logs:

Aug 06, 2022 11:26:39 PM
 
Snapshot: Snapshot 2015_04_30__01_00_54 was deleted from share or LUN Backup due to low free space on the volume.
Aug 06, 2022 11:25:49 PM
 
Volume: Volume data health changed from Dead to Inactive.
Aug 06, 2022 10:59:27 PM
 
Disk: Disk in channel 4 (Internal) changed state from ONLINE to FAILED.
Aug 06, 2022 10:59:26 PM
 
Disk: Disk in channel 2 (Internal) changed state from RESYNC to ONLINE.
Aug 06, 2022 10:59:26 PM
 
Volume: Volume data health changed from Degraded to Dead.
Aug 06, 2022 10:59:26 PM
 
Volume: Volume data is resynced.
Aug 06, 2022 08:26:02 PM
 
Volume: Resyncing started for Volume data.
Aug 06, 2022 08:24:59 PM
 
Disk: Disk Model:WDC WD40EFZX-68AWUN0 Serial:WD-WX62D123KUTC was added to Channel 2 of the head unit.
Aug 06, 2022 08:11:46 PM
 
Account: User 'Ben' was added.
Aug 06, 2022 08:11:38 PM
 
Disk: Disk Model:ST4000VN000-1H4168 Serial:Z302KH2D was removed from Channel 2 of the head unit.
Aug 06, 2022 08:11:12 PM
 
Account: User 'Ben' was deleted.
Aug 06, 2022 01:00:12 AM
 
Volume: Volume data is Degraded.

 

Now, in the readycloud app, my entire file structure is still there, but everything is showing as empty.

As I log into readycloud online I get notification: Readynas 6001010060
Then home shows No Volumes or USB Disks.
In Manage: Status Healthy. Data-0 0 free of 10.90 TB, and Data 0 free of 0. (i.e. there is another volume with no resources allocated to it.)

In the Admin page I get a Notification to remove inactive volumes to use the disk. disk #1,2,3,4.
There are TWO volumes, instead of ONE - as it used to be prior to the hot-swap.
All four disks are RED with green lights (i.e. all marked as spare).

Whiskeypod_0-1659904432846.jpeg

 

Performance tab claims all four disks are fine - all green.  

performance.jpg


And ALL disk have  a volume status of NEW.

DiskState.jpg

All my data, shares, apps, etc are gone/not accessible.

It appears to me that the ReadyNAS, in its wisdom, after resync, somehow decided to mark my original volume as DEAD, then decided that disk 4 had failed (it hasn't, as it is currently listed as healthy in the Performance tab; also there had NEVER been any notification about any degradation or issues with disk 4), and then marked my original volume from DEAD to INACTIVE. 

 

Basically, after the hot-swap, the nas went about to self-destruct as per above, and then created a new volume, so there are now two, both marked as full as all disks are marked as inactive.


BTW we did not make any mistakes in carrying out the hot-swap; we followed instructions to the letter, the system was powered-up throughout and we replaced the correct disk (as the logs clearly indicate). This  is a clear fault with the NAS's processes, which after resync decided to put a few spanners in the work, by marking the volume from 'degraded' (as disk 2 had originally failed) to 'dead', instead of 'healthy' or 'online' (as you can see in the logs). And from there things just got worse, and into a chain reaction.

How do I fix this?
How do I get my data back?

Model: RN10400|ReadyNAS 100 Series 4- Bay (Diskless)
Message 1 of 2
StephenB
Guru

Re: ReadyNAS 104 Firmware 6.10.7 - All four disks inactive following hot-swap of failed vol 2


@Whiskeypod wrote:

Performance tab claims all four disks are fine - all green.  


It appears to me that the ReadyNAS, in its wisdom, after resync, somehow decided to mark my original volume as DEAD, then decided that disk 4 had failed (it hasn't, as it is currently listed as healthy in the Performance tab; also there had NEVER been any notification about any degradation or issues with disk 4), and then marked my original volume from DEAD to INACTIVE. 

 


Regrettably, the ReadyNAS can misreport disk health - there are several cases I've seen where the NAS reports a failed disk as healthy.  Disks also can and do fail without warning, so the absence of prior notification is not definitive either.

 

I recommend powering down the NAS and testing the disk in a Windows PC with WD's dashboard utility.  You can connect the disk to the PC using either SATA or a USB adapter/dock.   Probably makes sense to test disk 2 while you are at it.

 

BTW, label the disks by slot as you remove them (and also label the disk you removed).  

 


@Whiskeypod wrote:

(you can see from the logs that resync took about 90 minutes.)


This is actually a bad sign.  RAID runs "below" the file system, it doesn't know what blocks on the disks hold data and which do not.  A resync requires reading all 12 TB on your pre-existing disks, and then writing 4 TB of data to the new one.  That can't be done in 90 minutes (despite the completion message), so something went wrong.

 


@Whiskeypod wrote:


My NAS does not have a backup, I purchased it with four drives AS a storage and back up, as the data was meant to be safe from failures with four disks.


As you have found out, devices (including disks and NAS) can fail at any time.  RAID is useful, but it is not enough to keep your data safe.  The only way to do that is to have a good back up plan in place.  Hopefully you will be able to recover your data, and take care of that.

 


@Whiskeypod wrote:

 

Basically, after the hot-swap, the nas went about to self-destruct as per above, and then created a new volume, so there are now two, both marked as full as all disks are marked as inactive.


How do I fix this?
How do I get my data back?


While it looks like it created a new volume, it didn't actually do that.  The problem is that it could not mount your existing volume due to the failures during resync.

 

Analyzing and fixing this problem is complicated (and honestly, success is not certain).  The best option is to use paid Netgear support - likely you will need a data recovery contract. 

Another option is to connect all the disks to a Windows PC (likely using an enclosure of some kind), and then using RAID recovery software that supports BTRFS.  ReclaiMe is one package that other users have used with success.  This option would also require that you have additional storage to copy off your data.

 

Also, keep the disk you removed.  It sounds like it hadn't completely failed, and you might need it as part of the recovery process. 

 

Some folks recently have had trouble getting paid support for their ReadyNAS.  The mods ( @Mark_V and @JeraldM ) should be able to help with that.  They likely will also move this discussion thread to the ReadyNAS forum area.

Message 2 of 2
Top Contributors
Discussion stats
  • 1 reply
  • 524 views
  • 0 kudos
  • 2 in conversation
Announcements