One disk gets redundant, rebulds, then another disk gets redundant. Now all dead. ReadyNAS 104
Hi folks. A few days ago my RN104 flagged my disk 1 as degraded. I did a restart on the system to see if it was only an error, and the disk started to re-build itself.
I bought a new 8TB disk, hooked that to my computer and started to transfer 3 volumes from a 4TB drive onto the new drive, with the intention of putting the new 8TB into my computer and replacing the faulty NAS disk with the older 4TB.
As soon as the faulty NAS disk 1 was finished rebuilding itself, I got an error message telling me that the volum was finished but was still flagged as degraded. Then it flagged disk 3 as degradedas well.... Like, at the same time.
Now, all four disks are marked red, and it tells me "Please remove inactive volumes in order to use the disk. Disks # 1, 2, 3, 4"
Can someone please tell me what the h*** just happened here? Did I just experience the worst luck ever, with the second drive crashing at the EXACT time the other process finishes, or is this a faulty device, or what?
I tried attaching the system log, but was not allowed to upload a zip file. You can see a short out-take on the incident below, but I have uploaded the zip file to a OneDrive folder if anyone would like to have a look at the full log. https://1drv.ms/u/s!Aq3djxq-F3Z1gcEoi_sKe9zsgtyqRQ?e=8c5UIe
Feb 27 01:00:28 NewBox readynasd[1353]: Volume 'data' is 'Degraded'. Feb 27 02:49:14 NewBox readynasd[1353]: The rebuild operation has finished on volume 'data'. However, the volume is still degraded. Feb 27 02:49:15 NewBox readynasd[1353]: Disk in channel '1' (Internal) changed state from RESYNC to ONLINE. Feb 27 02:49:15 NewBox readynasd[1353]: Disk in channel '3' (Internal) changed state from ONLINE to FAILED.
Re: One disk gets redundant, rebulds, then another disk gets redundant. Now all dead. ReadyNAS 104
Drive 1 has a lot of reallocated sectors, drive 3 has some errors which were enough to cause the volume to fail.
Feb 27 02:48:37 NewBox kernel: md/raid:md127: read error not correctable (sector 5843916768 on sdb3).
Feb 27 02:48:37 NewBox kernel: md/raid:md127: read error not correctable (sector 5843916776 on sdb3).
Feb 27 02:48:37 NewBox kernel: md/raid:md127: read error not correctable (sector 5843916784 on sdb3).
Feb 27 02:48:37 NewBox kernel: md/raid:md127: read error not correctable (sector 5843916792 on sdb3).
Feb 27 02:48:37 NewBox kernel: md/raid:md127: read error not correctable (sector 5843916800 on sdb3).
Feb 27 02:48:37 NewBox kernel: md/raid:md127: read error not correctable (sector 5843916808 on sdb3).
Feb 27 02:48:37 NewBox kernel: md/raid:md127: read error not correctable (sector 5843916816 on sdb3).
Feb 27 02:48:37 NewBox kernel: md/raid:md127: read error not correctable (sector 5843916824 on sdb3).
Feb 27 02:48:37 NewBox kernel: md/raid:md127: read error not correctable (sector 5843916832 on sdb3).
Feb 27 02:48:37 NewBox kernel: md/raid:md127: read error not correctable (sector 5843916840 on sdb3).
Drive 1 has been misbehaving since May of 2016 and should have been replaced long ago. So this wasn't really the "worst luck ever" on the timing - instead it was a ticking bomb that finally went off. Take a look in smart_history.log (at both pending and reallocated sectors).
I suggest removing the log zip from your OneDrive, as there is information in the logs that shouldn't be publicly posted.
Re: One disk gets redundant, rebulds, then another disk gets redundant. Now all dead. ReadyNAS 104
Thanks for the answer.
I have not seen this before, and I do check now and then.
I am no expert at this, but I must have been looking at the complete wrong places. I guess just checking the status in RAIDar isn't enough... Just checked my other ReadyNAS in the logfile you mentioned, and thankfully everything there is at 0.
Guess I'll just have to bite the bullet, by yet another HDD, rebuild the array and then check the remaining two disks to see how their healt is faring too..
I am no expert at this, but I must have been looking at the complete wrong places. I guess just checking the status in RAIDar isn't enough...
Unfortunately not. One issue with ReayNAS OS 6 is that it doesn't give you enough feedback on disk health. There are some alerts, but the thresholds for them are much too high.
