Received an alert last night that 1 of my disks in my 4 disk ReadyNas Ultra 4 had failed.
When I logged in to Front View, it actually shows 2 disks as dead and the system is in life support modee :
Configuration:
RAID Level X-RAID2, 4 disks
Status:
On life support - access to volume disabled. To restore volume, shutdown this device, reconnect the disk(s) if they had been inadvertently disconnected, and power the device back on. To avoid permanently rendering the disk unusable, do not reconnect the disk(s) while the power is on.
What's my play here? The disks haven't been disconnected and there is no indicaiton in the logs of the 2nd disk failure.
Thanks in advance...
Model: ReadyNASRNDU4000|ReadyNAS Ultra 4 Chassis only
Hi @TConger, I don't know how old the box is and how intensively you were using it under which environmental conditions. Depending the NAS could collect lots of dust/dirt and cleaning up carefully might fix (parts of) your problem.
Before doing this make sure the nas is no longer connected to power, neither directly nor via ups. Take out the disks keeping in mind, which disk needs to go back into which slot lateron.
Then do the cleaning, reinsert the disks, boot and cross your fingers.
there is no indicaiton in the logs of the 2nd disk failure.
Thanks in advance...
Where have you looked in the logs? You've confirmed no disk i/o failures in system.log and kernel.log?
Note that about 25% of the time disks fail with no warning (SMART stats are healthy). This is per BackBlaze data, though earlier studies (particularly one by Google) also found this to be true. Multiple disk failures in rapid succession are more likely in RAID arrays than random chance predicts - not that surprising since generally we are installing identical disks at the same time in the same chassis, with identical loads thanks to RAID striping.
I suggest powering down the NAS (actually removing the power cord for 15 minutes or so), and then power it up using the boot menu to disable the file system (volume) check. Use the procedure on page 21-22 here: http://www.downloads.netgear.com/files/GDC/RNDU2000/ReadyNAS_Ultra_UltraPlus_NVX_Pro_HW_en_8July11.p... If the volume is accessible, I suggest making a backup without delay (and don't write anything to the data volume until that is done).
Then power down again, and test the disks in a Windows PC with vendor tools - Seatools for Seagate, Lifeguard for Western Digital. Windows won't recognize the format, but the vendor tools can find and test the disks. It's ok to connect the disks via a USB adapter/dock. Label them by slot as you remove them from the NAS (so you can keep them in slot order later when you return them). I suggest looking at the SMART stat details shown by the vendor tools, and not just depending on the pass/fail results. Also, I don't recommend trying to repair the disks at this point (even if the tools suggest that).
When I said the 2nd disk failure wasn't in the logs, I had only looked in the GUI log, where 1 disk failure is noted. I then downloaded the logs and read through the system.log file where i find this: Nov 1 15:37:39 nas-AA-2D-9D kernel: end_request: I/O error, dev sdc, sector 5362360 Nov 1 15:37:39 nas-AA-2D-9D kernel: md/raid1:md0: Disk failure on sdc1, disabling device. Nov 1 15:37:39 nas-AA-2D-9D kernel: <1>md/raid1:md0: Operation continuing on 3 devices.
and 2 minutes later:
ov 1 15:39:12 nas-AA-2D-9D kernel: end_request: I/O error, dev sda, sector 7023456 Nov 1 15:39:12 nas-AA-2D-9D kernel: md/raid1:md0: Disk failure on sda1, disabling device. Nov 1 15:39:12 nas-AA-2D-9D kernel: <1>md/raid1:md0: Operation continuing on 2 devices.
Nov 1 15:44:12 nas-AA-2D-9D kernel: md/raid:md2: Disk failure on sdc5, disabling device. Nov 1 15:44:12 nas-AA-2D-9D kernel: <1>md/raid:md2: Operation continuing on 3 devices. Nov 1 15:44:12 nas-AA-2D-9D kernel: md/raid:md2: Disk failure on sda5, disabling device. Nov 1 15:44:12 nas-AA-2D-9D kernel: <1>md/raid:md2: Operation continuing on 2 devices.
So the 2 drives failed iin pretty quick succession.
The drives are 2 different drives 1 from the original confiugration 1.5 TB Seagate and the other failure is a WD 3 Tb drive taht was swapped out in Jan 2014. In 2014 I swapped out 2 of the 1.5 TB drives for 2 3 TB drives. I still have at lease one of the 1.5 Tbs in my lab, it's been sitting in my drawer since I replaced it (repalced to increase space, not because of a failure).
The system has disabled access to the volume, so I'm not able to take a backup yet.
Thanks for the link, I will try the powerdown procedure.
I'm also getting a spare worksation ready to run ddrescue incase I need to clone the disks.
The unit sits in my 23 U server rack and is pretty clean, save for a small amount of dust on the front cover that has built up since the last cleaning. It's not in heavy use, mainly sits and runs as a backup location and music storage server for my SONOS system. It's certainly not being stressed at all and is backed by a UPS so there have been no power drops in the past 5 or 6 years that I am aware of. It's probaly 8 years old, I don't remember for sure when I bought it. Initially I had 4 1.5 TB seagate drives in it but upgraded 2 of them to 3 TB in 2014.
so when I do this, i get to the boot menu, but pressing the backup button does nothing until I stop pressing the reset button, then I am able to select the 'skip vol check' option from the menu, but nothing happens. I tried leaving the skip vol check option and powering down, but when it came back up it said 'err: No disks detected!'
I powered down and took each drive out one-by-one and checked the connections and made sure they were clean and reinserted the drives and repeated the boot-menu process.
Same thing, no action from the backup button until i released the reset button. This time I selected the 'skip vol check' option and the pressed the reset button again. same results. 😞
so when I do this, i get to the boot menu, but pressing the backup button does nothing until I stop pressing the reset button, then I am able to select the 'skip vol check' option from the menu, but nothing happens. I tried leaving the skip vol check option and powering down, but when it came back up it said 'err: No disks detected!'
I powered down and took each drive out one-by-one and checked the connections and made sure they were clean and reinserted the drives and repeated the boot-menu process.
Same thing, no action from the backup button until i released the reset button. This time I selected the 'skip vol check' option and the pressed the reset button again. same results. 😞
Probably the next step is to test the disks in a PC with with lifeguard then.
Both of the Western digital disks passed the LIfegard tests.
The 1st of the Seagate 1.5 tb failed the 'Long Generic' test.
Running the 2nd one through the Long Generic test now.
If it's just the one 1.5 Tb seagate that is dead, I can replace it with one of the 1.5 Seagates I took out in 2014 - I'll think I just need to wipe it before I put it back in the NAS as it was previously a member of this raid group.
Yup, make it blank. No formatting, no partitioning.
Just deleting the partitions will work, but I suggest using the "advanced" write-zeros test in the vendor tools, as it is a good diagnostic. Note you can use the one in Lifeguard for this also, it will work with Seagate disks.
