Volume degraded after firmware update to 6.9.5 on RN212
NAS device was upgraded to the 6.9.5 version this evening and after reboot my drive in slot 1 is now showing as degraded. I rebooted and have power reset with no fix. Ran disk tests also with no luck. I swapped drives between slots to see if the issue follows and it does not. Slot 1 still showing degraded state. So that proves its not a drive issue but a slot issue and I am assuming related to firmware. Anyone else having this issue? Thanks
Chris
Model: RN21200|ReadyNAS 212 Series 2- Bay (Diskless)
Re: Volume degraded after firmware update to 6.9.5 on RN212
I am not seeing it in my RN202. You could of course try downgrading back to 6.9.4. Though I think it'd be better to PM @JohnCM_S and ask him if he is willing to review your logs.
If you are the original purchaser, then your RN212 should still be covered by the warranty (3 years). So another option is to request an RMA via my.netgear.com.
Re: Volume degraded after firmware update to 6.9.5 on RN212
Hi chyzm,
I have checked the logs and see communication error with the drives. Most likely it is already an issue with the bay itself but it should not be related to the firmware upgrade specifically.
It will be best to contact support so they can process an RMA for the chassis.
Re: Volume degraded after firmware update to 6.9.5 on RN212
Did you get this sorted?
I seem to have the same problem on my ReadyNAS Pro (upgraded to OS6).
I updated the firmware to 6.9.5 and later disk 1 showed as Failed. I rebooted the NAS and the disk now shows OK but the resync didn’t complete and the NAS display is flashing “Data Degraded”. Not sure what caused this.
Re: Volume degraded after firmware update to 6.9.5 on RN212
No luck. Trying to contact and get Netgear to responsed was a failure. Customer service and experience are the pits with them. After two days trying to open a case I was finally able to do so but again no response. After a week the drive came back. Mine is under warranty but again I have had no luck in getting Netgear to respond. Funny you posted this because last night I had to power reset some of my equipment and the NAS was one that got reset and same issue has come back. I give up and am looking at a Synology DS218+ to replace. Don’t want to go through the whole debacle again and will never buy a Netgear product again.
I seem to have the same problem on my ReadyNAS Pro (upgraded to OS6).
I updated the firmware to 6.9.5 and later disk 1 showed as Failed. I rebooted the NAS and the disk now shows OK but the resync didn’t complete and the NAS display is flashing “Data Degraded”. Not sure what caused this.
Re: Volume degraded after firmware update to 6.9.5 on RN212
Thanks for your reply chyzm. I've factory reset the NAS and it seems to be OK now. It's currently going through the resync process. My problem may not be related to the 6.9.5 update. I also changed the settings on the NAS to enable disk spin down after 10 minutes and it may have been this. I've changed it back to having the disks spinning all the time.
Yeah the Synology DS218+ looks nice. I've had a look at that too. But my ReadyNAS has been pretty well rock solid for more than 10 years. I've had to change out all of the 6 disks and I've also upgraded the CPU and memory on it to get a bit more performance. It's really reliable but performance is still a bit low for apps like Plex.
Yes it may not be the firmware upgrade. I did three things prior to it failing:
(i) Upgraded firmware to 6.9.5
(ii) Changed power settings to disk spin down after 10mins of inactivity
(iii) Installed Plex
It may be related to (ii). From what I've read, it's better to leave the disks spinning all of the time rather than spinning them up. I have an unproven theory that maybe the disks didn't spin up fast enough and the system mapped one as bad. But that's only speculation. This is this alos happend with disk 3 briefly but a reboot fixed it.
I've factory reset the NAS and it seems to be all fine now.
Thanks for the offer of looking at the Log files. I'll take you up on the offer. What I have noticed in mdstat is that /dev/md/data-0 all of the disks are OK but in /dev/md/data-1 it's showing 4 active devices, 5 working devices and 1 spare. UNless somehow disk 3 was also mapped as bad and that's led to a situation where the disks can't resync.
Re: Volume degraded after firmware update to 6.9.5 on RN212
I changed the disk spin up setting as you mentioned and will factory reset to see if that fixes issue. I think my issue is support. Netgear makes it so difficult to contact support and then they don’t follow up. It’s a reason to make me leave Netgear products. Thanks for the info!
From what I've read, it's better to leave the disks spinning all of the time rather than spinning them up. I
There's more than one view on that. Personally I enable spindown on all my backup NAS. On the main NAS I enable spindown on off-hours - the idea there is that I don't want to wait for spinup to complete before accessing data. That's annoying, especially with my media player.
It's hard to say if spindown extends the life of the drives (or not). But it does save power.
Re: Volume degraded after firmware update to 6.9.5 on RN212
Thanks @StephenB . I have used disk spindown before with no problems - but maybe not with these WD Red drives. What I'm wondering is whether my transient disk failure could be related to this spindown/up. I had two disks reporting briefly as failed which seem now perfectly OK following a restart. Both are showing virtually zero errors in their reported health data. What could have caused them to report as being failed? As mentioned previously, I changed three things shortly before this problem ocurred - upgraded to 6.9.5 firmware, set disk spindown after 10 min of inactivity and loaded Plex app. Sort of strange. Any ideas?
Thanks @StephenB . I have used disk spindown before with no problems - but maybe not with these WD Red drives.
I am using WD20EFRX, WD30EFRX, WD60EFRX and WD80EFZX with spindown enabled.
The firmware update process updates the OS partition on all the drives. So if you did the reboot fairly quickly after you applied the update, then the disks should still have been all spun up. Either way, the drives are all spun down as part of the shutdown process, and they all are spun up as part of the boot process. While I wouldn't rule your spindown theory out, I think it's not a likely cause.
I had two disks reporting briefly as failed which seem now perfectly OK following a restart. Both are showing virtually zero errors in their reported health data.
I’m just wondering if one of the drives was slow to spin up whether it would be flagged by the NAS as Failed?
Well, if a drive is slow to respond, that could happen. As I explained above, I don't think enabling spindown would trigger that behavior on a firmware install or reboot. I haven't seen any evidence of it happening with any of my WDC Reds (scattered across 4 different ReadyNAS).
ATA errors could explain your problem too, especially if the disk driver marks the SATA interface as down as a result. Are the ATA errors increasing?
Your disk configuration is: 3TB x 3TB x 2TB x 3TB x 3TB x 3TB.
This means that the NAS will create two raids due to the different sized disks. As follows:
md127 = 6 x 2TB (raid5) md126 = 5 x 1TB (raid5)
We can see by the raid config that this is indeed the case. However, notice how sda is a spare in the md126 raid. I suspect this happens because it cannot sync in the disk properly. More on that further down.
Your two raids are then "stuck" together by the filesystem in order to create one volume.
Label: '<masked>:data' uuid: <masked>
Total devices 2 FS bytes used 970.18GiB
devid 1 size 9.07TiB used 972.02GiB path /dev/md127
devid 2 size 3.64TiB used 0.00B path /dev/md126
=== Issue ===
Disk 1 fell out of the md126 raid and was never able to re-join the raid array. The below sequence happens every time you reboot.
[19/04/12 18:01:48] err:disk:LOGMSG_ZFS_DISK_STATUS_CHANGED Disk in channel 1 (Internal) changed state from ONLINE to FAILED.
[19/04/12 19:45:48] notice:volume:LOGMSG_RESILVERSTARTED_VOLUME Resyncing started for Volume data.
[19/04/12 19:45:52] notice:disk:LOGMSG_ZFS_DISK_STATUS_CHANGED Disk in channel 1 (Internal) changed state from ONLINE to RESYNC.
[19/04/12 23:29:08] notice:volume:LOGMSG_RESILVERCOMPLETE_DEGRADED_VOLUME The resync operation finished on volume data. However, the volume is still degraded.
[19/04/12 23:45:10] notice:system:LOGMSG_SYSTEM_REBOOT The system is rebooting.
[19/04/12 23:46:44] warning:volume:LOGMSG_HEALTH_VOLUME_WARN Volume data is Degraded.
[19/04/12 23:46:45] notice:volume:LOGMSG_RESILVERSTARTED_VOLUME Resyncing started for Volume data.
[19/04/13 00:24:08] notice:volume:LOGMSG_RESILVERCOMPLETE_DEGRADED_VOLUME The resync operation finished on volume data. However, the volume is still degraded.
A raid sync stopping like this is a safety mechanism because one or more drives are not responding properly during the sync. It is to avoid a potential double disk failure. You disks appear OK-ish with two disks having a small amount of errors.
Device: sdc
Channel: 2 <<<=== Disk 3
ATA Error Count: 9
Device: sdd
Channel: 3 <<<=== Disk 4
ATA Error Count: 1
However, the kernel logs are completely flooded with disk errors from 4 of the disks: 2, 4, 5 and 6. Below is a mere extract.
So, this is why the raid sync fails. It is likely also why disk 1 eventually is simply marked as spare in the md126 raid.
Re: Volume degraded after firmware update to 6.9.5 on RN212
Hi @Hopchen . Thanks so much for looking at my log files. Yikes!
I've backed up the NAS and done a factory reset. The NAS seems to be working fine now but looking at the latest kernel log file, there are still ata errors such as:
This seems to be pretty much happening with all of the drives. These are different errors than those reported in the log you looked at.
Would you expect the above errors normally?
DMA errors would likely be related to the Motherboard right? I did upgrade the CPU and memory on the motherboard a year or so back. But it's been fine since.
@chyzm Do you have similar errors in your kernel log file?
These are not normal no. Given your explanation of the NAS history and changing of parts, I definitely think the issue is in the chassis somewhere. I am wondering if the PSU might be going bad...
The NAS might not have proper drivers for the new CPU, but it looks more like a power issue to me. Could be wrong though.
The NAS seems to be working fine now - albeit with the SATA DMA errors (which may have been happening for ages without my knowledge). I notice that the ATA error count is creeping up a bit on all of the drives - which suggests an interface error. I pulled the unit apart and cleaned the interconnect between the motherboard and the disk backplane board just in case there was a bad connection there.
I guess I'll just keep an eye on it. Do you know if you can buy a compatible power supply these days? The unit still works fine as a NAS and is built to a high quality compared to some of the consumer NAS's.
Do you know if you can buy a compatible power supply these days? The unit still works fine as a NAS and is built to a high quality compared to some of the consumer NAS's.
Do you know if you can buy a compatible power supply these days? The unit still works fine as a NAS and is built to a high quality compared to some of the consumer NAS's.
