ReadyNAS 214 "Remove inactive volumes to use the disk. Disk #1,2,3,4."
Hello, I recently updated my firmware on a ReadyNAS 214. Next time I looked I got a degraded volume error on Drive bay #1.
I have 4 WD Red WD30EFRX drives but when I searched for a replacement I was directed to a newer version, which is the WD Red Plus WD30EFZX. I shutdown and replaced the drive but the new drive was not recognized. The drive was blued out in the Volume active display, but the indicator was yellow rather than green. Not sure if theproblem is incompability but I then tried different swaps and now have ended up with all the drives redded out under volume status but with a green indicator and the "Remove inactive volumes" message above. I was under the impression that I had complete data redundancy across 4 backup drives but my reading of these forums suggests that this NAS backup is not a complete safeguard. I have logs attached below and some screenshots.
Any help in getting this fixed is greatly appreciated!
Okay - understanding this is what's tripping me up.
Here's a brief (and simplified) explanation.
Sector X on disks a, b, and c are for data. The corresponding Sector X on disk d is a parity block. This is constructed using an exclusive-or (xor) of the three data blocks You can think of this as addition for our purposes here.
Every time Xa, Xb, or Xc are modified, the RAID also updates Xd.
So Xa + Xb + Xc = Xd.
If disk b is replaced, then Xb can be reconstructed by
Xb = Xd - Xa - Xc
Similarly, the contents of any of the other disks can be reconstructed from the remaining 3. That is what is happening when the RAID volume resyncs.
The reconstruction fails if
the system crashed after Xa, Xb, or Xc was modified, but before Xd was updated.
two or more disks fail (including a second disk failure during reconstruction).
a disk read gives a wrong answer (instead of failing). This is sometimes called "bit rot".
the system can't tell which disk was replaced.
The RAID system counts up the writes to each disk (maintaining an event counter for each disk). So it can detect the first failure mode (because the event counters won't match). When it sees that error, it will refuse to mount the volume. That is a fairly common cause of the inactive volume issue.
Often it is a result of a power failure, someone pulling the plug on the NAS instead of properly shutting it down, or a crash. The RAID array can usually be forcibly assembled (telling the system to ignore the event count mismatch). There can be some data loss, since there were writes that never made it to some of the disks.
Two or more disk failures sounds unlikely, but in fact it does happen. Recovery in that case is far more difficult (often impossible, or cost-prohibitive).
Figuring out what happened in your case requires analysis of the NAS logs. If you want me to take a look at them, you need to download the full log zip file from the NAS log page. Then put it into cloud storage (dropbox, icloud, etc), and send me a private message (PM) using the envelope icon in the upper right of the forum page. Put a link to the zip file in the PM (and set the permissions so anyone with the link can view/download the zip file).
Re: ReadyNAS 214 "Remove inactive volumes to use the disk. Disk #1,2,3,4."
Did I need to format the new Red Plus drive before it would sync with the 3 redundant drives? I was leery of hitting any buttons that said format because it's not clear which volume it refers to.
I then tried different swaps and now have ended up with all the drives redded out under volume status but with a green indicator and the "Remove inactive volumes" message above.
Did you try putting the original drives back in their original slots?
If that fails, remove the drive you first removed (to insert the replacement) and try booting again.
You need to be careful when experimenting with different swaps, as the RAID array can easily get out of sync (making the issue more difficult to fix).
I was under the impression that I had complete data redundancy across 4 backup drives but my reading of these forums suggests that this NAS backup is not a complete safeguard.
RAID redundancy (though helpful) is not enough to keep data safe. I'm guessing you don't have a backup of the files on the NAS - is that the case?
Re: ReadyNAS 214 "Remove inactive volumes to use the disk. Disk #1,2,3,4."
Yes, you have to format the new drive in the NAS. In older OS versions, it was automatic. But users who failed to read the users' manual were putting in drives with their collection of photos (or whatever) and losing them (they thought they'd still be there, as if a NAS is like a USB drive chassis), so Netgear made this a separate step.
Yes, you had redundancy when you had four good drives. But after you lost one, you had none, and experimentally messing with the remainder was a really bad idea. You likely caused the volume to go "out of sync". It may be recoverable, and @StephenB is better at helping you do that than I.
Re: ReadyNAS 214 "Remove inactive volumes to use the disk. Disk #1,2,3,4."
Stephen,
Thanks. Yes, I put the drives back as originally configured and I get all drives redded out with a red status indicator on the original 'degraded' drive. On my performance status I get a black ball on Disk 1 and 3 green balls on Disks 2-4.
My overview shows status as "Healthy," whereas before it was "Volume Degraded." I suspect it's not reading Disk 1, so just the 3 additional drives.
I don't have a backup since I had less than 5TB of data and 12TB (4x3TB total over 4 drives and thought I was mirroring the data on the other drives. The Volume page now shows I have 8.17 TB of data but I have no idea how it is arrayed and how to recover it. It still says to remove inactive volumes 1-4.
I kept reading that the drives were hot swappable but now I'm not so sure and so shut down each time I swap the drives. The replacement drive I bought is a WD30EFZX Red Plus drive and when I put it in and select to Format, nothing happens(?)
I have lots of photographs stored as hires original files so I hope I can recover all. Are there any repair routines to rebuild the Raid array?|
I kept reading that the drives were hot swappable but now I'm not so sure and so shut down each time I swap the drives. The replacement drive I bought is a WD30EFZX Red Plus drive and when I put it in and select to Format, nothing happens(?)
At the moment, you really don't want to be swapping drives around.
Re: ReadyNAS 214 "Remove inactive volumes to use the disk. Disk #1,2,3,4."
That you are seeing volume "data" and "data-0" means the OS can't assemble all the drives in one volume, but does recognize that there is something that needs to be assembled and has labeled them separately. That usually means they are out of sync (something changed on some, but not all, drives in the RAID). Both contain parts of your actual "data" volume, so don't delete either of them. But because file contents are spread among all drives, you can't just go into either and get data. @StephenB can help you try to manually assemble them into one via the command line. There could end up being a few corrupted files, but most will likely be recoverable.
I think you are prevented from formatting when you have an invalid volume. It's a way the OS keeps you from doing something stupid that makes recovery impossible. Just adding another drive will not fix your problem, anyway. It's too late for that. If the old drive can be read at all, cloning it to the new one may help with the recovery.
If you can assemble the volume, you are going to want to save off all the files and re-format the whole volume, so you are going to need somewhere to put them. After you've restored everything, that somewhere can become your backup, so size it appropriately.
I kept reading that the drives were hot swappable but now I'm not so sure and so shut down each time I swap the drives. The replacement drive I bought is a WD30EFZX Red Plus drive and when I put it in and select to Format, nothing happens(?)
At the moment, you really don't want to be swapping drives around.
I have only swapped the "degraded volume" for a newly purchased WD Red Plus. I'm wondering if this new drive is going to be compatible with my WD Red drives or if I should return it before the return window expires?
Re: ReadyNAS 214 "Remove inactive volumes to use the disk. Disk #1,2,3,4."
Thanks Sandshark,
You wrote:
"If you can assemble the volume, you are going to want to save off all the files and re-format the whole volume, so you are going to need somewhere to put them. After you've restored everything, that somewhere can become your backup, so size it appropriately."
Yes, I guess that is best practice. I have data spread across several USB external HDs but I never trust them bc WD often fails. So, I bot 12TB of these WD Red drives to secure my data on an NAS and now I need to duplicate it with another large harddrive? Are backups possible by just plugging in the USB port on the ReadyNas drive bay?
I'm wondering when all is said and done whether the new WD 3TB Red Plus drive I bought will be necessary to replace the degraded drive?
Okay - understanding this is what's tripping me up.
Here's a brief (and simplified) explanation.
Sector X on disks a, b, and c are for data. The corresponding Sector X on disk d is a parity block. This is constructed using an exclusive-or (xor) of the three data blocks You can think of this as addition for our purposes here.
Every time Xa, Xb, or Xc are modified, the RAID also updates Xd.
So Xa + Xb + Xc = Xd.
If disk b is replaced, then Xb can be reconstructed by
Xb = Xd - Xa - Xc
Similarly, the contents of any of the other disks can be reconstructed from the remaining 3. That is what is happening when the RAID volume resyncs.
The reconstruction fails if
the system crashed after Xa, Xb, or Xc was modified, but before Xd was updated.
two or more disks fail (including a second disk failure during reconstruction).
a disk read gives a wrong answer (instead of failing). This is sometimes called "bit rot".
the system can't tell which disk was replaced.
The RAID system counts up the writes to each disk (maintaining an event counter for each disk). So it can detect the first failure mode (because the event counters won't match). When it sees that error, it will refuse to mount the volume. That is a fairly common cause of the inactive volume issue.
Often it is a result of a power failure, someone pulling the plug on the NAS instead of properly shutting it down, or a crash. The RAID array can usually be forcibly assembled (telling the system to ignore the event count mismatch). There can be some data loss, since there were writes that never made it to some of the disks.
Two or more disk failures sounds unlikely, but in fact it does happen. Recovery in that case is far more difficult (often impossible, or cost-prohibitive).
Figuring out what happened in your case requires analysis of the NAS logs. If you want me to take a look at them, you need to download the full log zip file from the NAS log page. Then put it into cloud storage (dropbox, icloud, etc), and send me a private message (PM) using the envelope icon in the upper right of the forum page. Put a link to the zip file in the PM (and set the permissions so anyone with the link can view/download the zip file).
Re: ReadyNAS 214 "Remove inactive volumes to use the disk. Disk #1,2,3,4."
Dear StephenB,
I had exact problem with my NAS 214. It was all working fine, but suddenly all the three (it has 3 disks) disks started to show up as red. The health indicator shows green. There was a power failure in between. We have some important data. I am comfortable with command line, I have been a linux user for sometime. Could you please help?
Re: ReadyNAS 214 "Remove inactive volumes to use the disk. Disk #1,2,3,4."
Firmware 6.10.8
I suspect that StephenB has already answered this query (same, almost, as the original), but here goes:
My NAS was approaching 5% free data and I did not notice this. Then DISK 2 showed as FAILED. I remounted it and all shares and files appeared okay, so I commenced a clean up and ordered a new HDD to replace DISK 2, but 2 days later DISK 3 showed as FAILED as well (DISK 2 did not show a FAIL status again), at which point the SHARES were still accessible but the files were unable to be opened... We were to have a scheduled loss of power before I expected to receive the new HDD so I gracefully powered the NAS down and when the new HDD arrived the next day, powered the NAS up again. At this point I had no shares and no volumes. Replacing the failed disk with the new HDD didn't help.
I tried to use Wondershare's RECOVERIT but it could not connect to the NAS. The NAS is showing as healthy, but disks 1,2, 4 and red, disk 3 is grey and all show as SPARE, not RAID5.
My question is whether there is any tool or process I can use to access the data on disks 1,2,4 to re-synch disk 3 and get my shares and files again?
Re: ReadyNAS 214 "Remove inactive volumes to use the disk. Disk #1,2,3,4."
Recoverit certainly wouldn't work, since it is for Windows only.
One option is to connect the original disks to a Windows PC using USB/adapter docks and then use ReclaiMe (or some other RAID recovery software that supports both BTRFs and software RAID). You'd also need enough storage to offload your data.
It might be possible to forcibly mount the volume from tech support mode or using ssh (if it is enabled). Have you ever used the linux command line interface?
Re: ReadyNAS 214 "Remove inactive volumes to use the disk. Disk #1,2,3,4."
Hi Stephen, I have not used the Linux interface before, but I have used a number of flavors of proprietary unix before. From what I have read here, forcibly mounting the volume in tech support mode may result in loss of some of the files, but I am prepared to risk this (as Spock once said "the needs of the many outweigh the needs of the few"). Any advice is welcomed. A few queries:
(1) Should I mount the current (i.e. the "failed") disk or a fresh replacement disk in the applicable slot before trying to forcibly mount the volume/s?
(2) I (maybe stupidly) obeyed the instruction to remove the "inactive volumes to use the disk. Disk#1,2,4). Has this confused the RAID system?
(3) Once I boot the device in tech support mode, what is the command string?
(1) Should I mount the current (i.e. the "failed") disk or a fresh replacement disk in the applicable slot before trying to forcibly mount the volume/s?
(3) Once I boot the device in tech support mode, what is the command string?
Can you get a fresh log zip from the web ui?
Then put it into cloud storage, and PM me a download link. (Use the envelope icon at the top right of the forum page to send a private message).
Re: ReadyNAS 214 "Remove inactive volumes to use the disk. Disk #1,2,3,4."
To the rest of the folks in the community, an update that Stephen was able to coach and work with me to get access to the "lost" shares on my NAS, and I am in the process of unloading this data prior to replacing the degraded HDDs and re-building my RAID.
Lessons learned:
1) Watch out for warnings in the NAS logs and obey them in a timely manner
2) If the brown-stuff hits the fan, the community - and guys like Stephen in particular - is an invaluable asset to recover your data.
1) Watch out for warnings in the NAS logs and obey them in a timely manner
2) If the brown-stuff hits the fan, the community - and guys like Stephen in particular - is an invaluable asset to recover your data.
If you heed the first, you can hope you never need the second. But, stuff happens. Many simply take a NAS for granted, treating it like a simple USB storage bay instead of a complex device running an OS. In addition to email alerts, it's always good to "check in" on your NAS at least once a week via the GUI.
