Readynas 3220 addling a drive into xraid volume

Hi Raslin 

 

Before we start, let me remind that a raid6 can tolerate two disk failures. Any more than that (without replacing disks) and your raid will break.

 

As per your PM, you now rebooted the NAS and afterwards getting the message: "Remove Inactive Volumes".
The reason for that is because your data raid does not start and hence your data volume cannot mount. We can see this from the raid config.

md1 : active raid6 sdi2[0] sdl2[11] sdf2[10] sdg2[9] sdd2[8] sde2[7] sdb2[6] sdh2[5] sda2[4] sdc2[3] sdm2[2] sdj2[1]
5232640 blocks super 1.2 level 6, 512k chunk, algorithm 2 [12/12] [UUUUUUUUUUUU] <<<=== Linux swap space raid

md0 : active raid1 sdh1[14](F) sdi1[0] sdl1[15] sdm1[11] sdg1[10] sdd1[9] sdf1[12] sde1[6] sdc1[5] sdb1[4] sdj1[13] sda1[1]
4190208 blocks super 1.2 [12/11] [UUUUUUUUUUU_] <<<=== OS raid

<<<=== Missing data raid (md127)

This is not good news.

 

 

You issue actually began a long time ago. In June 2018, disk 2 started to fail and had gotten worse over time. Eventually it dropped from the raid.

 

[18/06/25 20:54:03 CAT] notice:disk:LOGMSG_SMART_PENDING_SECT_30DAYS_WARN Detected increasing pending sector: count [495] on disk 2 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx] 113 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
[18/06/25 21:00:18 CAT] notice:disk:LOGMSG_SMART_REALLOC_SECT_30DAYS_WARN Detected increasing reallocated sector count: [974] on disk 2 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx] 79 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
[18/06/25 21:00:23 CAT] notice:disk:LOGMSG_SMART_PENDING_SECT_30DAYS_WARN Detected increasing pending sector: count [504] on disk 2 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx] 114 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
[18/06/25 21:03:44 CAT] warning:volume:LOGMSG_HEALTH_VOLUME Volume data health changed from Redundant to Degraded.

 

 

Your volume went to a "Degraded" state but at this point you are still good. You can tolerate two disk failures in your raid6. That disk 2 is still in the NAS by the way. It was never replaced.

 

Fast forward to August 2018. Disk 7 is now starting to show signs of problems. It continues to get worse over the next months but it hangs on.

[18/08/02 07:25:58 CAT] notice:disk:LOGMSG_SMART_UNCORR_ERR_WARN Detected high uncorrectable error count: [76] on disk 7 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx]. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
[18/08/15 12:29:19 CAT] notice:disk:LOGMSG_SMART_UNCORR_ERR_WARN Detected high uncorrectable error count: [77] on disk 7 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx]. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.

 

 

Things start to become critical in October when disk 6 exhibits problems as well.

[18/10/30 04:25:52 CAT] notice:disk:LOGMSG_SMART_UNCORR_ERR_WARN Detected high uncorrectable error count: [66] on disk 6 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx]. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
[18/11/04 20:46:23 CAT] notice:disk:LOGMSG_SMART_PENDING_SECT_30DAYS_WARN Detected increasing pending sector: count [137] on disk 6 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx] 11 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
[18/11/04 20:46:47 CAT] notice:disk:LOGMSG_SMART_UNCORR_ERR_WARN Detected high uncorrectable error count: [126] on disk 6 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx]. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
---The errors keep increasing over time---
[19/02/14 09:51:13 CAT] notice:disk:LOGMSG_SMART_REALLOC_SECT_30DAYS_WARN Detected increasing reallocated sector count: [587] on disk 6 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx] 114 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
[19/02/14 09:55:13 CAT] notice:disk:LOGMSG_SMART_PENDING_SECT_30DAYS_WARN Detected increasing pending sector: count [1163] on disk 6 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx] 55 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
[19/02/14 10:02:22 CAT] notice:disk:LOGMSG_SMART_REALLOC_SECT_30DAYS_WARN Detected increasing reallocated sector count: [611] on disk 6 (Internal) [WDC WD4000FYYZ-01UL1B1 WD-xxxxxxxxx] 115 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
[19/02/14 10:04:58 CAT] notice:disk:LOGMSG_SMART_PENDING_SECT_30DAYS_WARN Detected increasing pending sector: count [1221] on disk 6 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx] 56 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
[19/02/14 18:52:44 CAT] notice:disk:LOGMSG_SMART_REALLOC_SECT_30DAYS_WARN Detected increasing reallocated sector count: [612] on disk 6 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx] 116 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.

 

 

Eventually disk 6 gets very bad and is kicked from the raid. Your NAS is hanging by a thread at this point since two disks are gone from the raid and you are on the verge of 3rd disk failing as well (disk 7). Somehow, disk 7 seems to keep hanging on. But then more bad news when disk 3 also starts to fail in mid February 2019.

 

[19/02/16 14:38:20 CAT] notice:disk:LOGMSG_SMART_PENDING_SECT_30DAYS_WARN Detected increasing pending sector: count [104] on disk 3 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx] 16 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
[19/02/16 14:45:30 CAT] notice:disk:LOGMSG_SMART_PENDING_SECT_30DAYS_WARN Detected increasing pending sector: count [118] on disk 3 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx] 17 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
[19/02/16 14:52:39 CAT] notice:disk:LOGMSG_SMART_PENDING_SECT_30DAYS_WARN Detected increasing pending sector: count [151] on disk 3 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx] 18 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
[19/02/16 14:59:48 CAT] notice:disk:LOGMSG_SMART_PENDING_SECT_30DAYS_WARN Detected increasing pending sector: count [172] on disk 3 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx] 19 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
---Disk 3 gets rapidly worse and eventually fails---
[19/03/12 21:21:13 CAT] notice:disk:LOGMSG_SMART_PENDING_SECT_30DAYS_WARN Detected increasing pending sector: count [649] on disk 3 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx] 50 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
[19/03/12 21:28:24 CAT] notice:disk:LOGMSG_SMART_PENDING_SECT_30DAYS_WARN Detected increasing pending sector: count [661] on disk 3 (Internal) [WDC WD4000FYYZ-01UL1B1, WD-xxxxxxxxx] 51 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
[19/03/12 22:07:06 CAT] notice:disk:LOGMSG_SMART_UNCORR_ERR_WARN Detected high uncorrectable error count: [89] on disk 7 (Internal) [WDC WD4000FYYZ-01UL1B1, WWD-xxxxxxxxx]. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.

 

 

Data access is lost at this point with a triple disk failure in the raid6. A reboot of the NAS is done and you get the error of: "Remove Inactive Volumes". To be honest, recovering from this is quite tricky - if even possible. 4 bad disks... It would require cloning of disks, trying to manually assemble the raid with a mix of good original disks and new cloned ones, etc. Not an easy task.

If you have backups then this would be the time to use them. If you do not have backups it would be time, right now, to contact NETGEAR Support and enquire about data recovery. That is a separate service (from normal support) that requires a fee.

 

All the problems stem from not replacing bad disks. I wonder if you never had email alerts working? The NAS should have warned you by email about the impending failure and disk issues. I can see some troubles getting the email alerts working?

[19/03/13 18:19:23 CAT] notice:system:LOGMSG_SENT_ALERT_MESG_FAILED Alert message failed to send.
[19/03/13 18:21:26 CAT] notice:system:LOGMSG_SENT_ALERT_MESG_FAILED Alert message failed to send.
[19/03/13 18:25:38 CAT] info:system:LOGMSG_SET_ALERTCONTACT Alert contact information was saved.
[19/03/13 18:25:38 CAT] info:system:LOGMSG_SET_ALERTCONFIG Alert settings are updated.
[19/03/13 18:26:50 CAT] notice:system:LOGMSG_SENT_ALERT_MESG_FAILED Alert message failed to send.
[19/03/13 18:28:11 CAT] notice:system:LOGMSG_SENT_ALERT_MESG_FAILED Alert message failed to send.

 

 

4 disks failing is of course a lot. However, this didn't happen over night and several of these disks are nearing end-of-life expectancy with 5 years of power-on time. Disks will fail eventually and thus makes it important to replace then when they do. Anyway, I hope that sheds some light on the situation. Not a nice place to be, unfortunately. I wish I could bring you better news :(

 

Cheers