I've been using various ReadyNAS units since 2007 and for the first time I think I may have run into a very serious problem. Our ReadyNAS 314 had 4x3TB drives in there running in X-Raid and recently it got to over 90% full. So today I bought 4x6TB drives, to replace them. Unfortunately something has gone wrong during the replacement/rebuild of the first drive.
I ejected disk 1 and replaced it with one of the new 6TB drives (hot swap). It started rebuilding. I was working away and glanced at the ReadyNAS at various points and saw that the % was gradually increasing (at one point I noticed it was at about 70%). But shortly aftewards I started to get various alerts (some almost simultaneously). My ReadyNas is set up to send me email alerts and this is what it sent me during the process: ------------------------ - 4:43pm: Disk Model:WDC WD30EFRX-68AX9N0 Serial:WD-WMC1T2693974 was removed from Channel 1 of the head unit - 4:43pm: Volume data health changed from Redundant to Degraded - 4:45pm: Disk Model:WDC WD60EFRX-68L0BN1 Serial:WD-WX21DC7780KV was added to Channel 1 of the head unit. - 4:46m: Resyncing started for Volume data. - 10:04pm: Disk in channel 1 (Internal) changed state from RESYNC to ONLINE - 10:04pm: Disk in channel 3 (Internal) changed state from ONLINE to FAILED - 10:04pm: Volume data health changed from Degraded to Dead - 10:06pm: Volume data is resynced ------------------------- I was confused and a bit worried to see that a disk had failed and the volme was "dead" and yet it also said resynced. I went into my NAS folders and I could see the shares still there. However, as I browsed the folders, it seemed that not everything was there. All of the folder structure seemed to be there, but some folders were empty or had a lot less files than they should have.
Something wasn't right. based on the above error message I could guess drive 3 had failed during the rebuild/resync and I guessed that maybe the resync of the new drive 1 might not have been fully complete. So I swithched off the Readynas and removed disk 3 (the failed one). When I turned the ReadyNas back on, my volume was no longer accesible!
I switched off the ReadyNas again and put the old HDD1 back in. I figured that the old disks 1+2+4 should still have all of the data if only 3 has failed. My thinking was that after putting old disk 1 back in, I could then replace disk 3 (the failed one) with one of the new drives and get that resynced. But when I turned the ReadyNas back on, my volume was still completely inaccesible!
I'm thinking I should never have turned the unit off - at least I had some files still accessible before I did!
It seems disk 3 had an issue during the resync of disk 1. Let's say disk 3 is bad. In theory I have old disks 1+2+4, shouldn't these have enough data to restore my volume? Can I somehow use the old disks 1+2+4 to get my volume back?
Or have I done some irreparable damage by rebooting with the old disk 1 back in?
Luckily some of the data should be backed up to the cloud (documents and photots). But a lot of the larger content on the drives like video and music is not (it was just too large to to upload).
I'm not much of an expert on disks and RAID, I'd really appreciate some feedback from anyone who knows what if anything I can do now.
My ReadyNAS details are,
Model: ReadyNAS314
Firmware: 6.9.3
Thanks so much for any help.
Mark D
ps: Incidentally I checked the old disks and the logs before replacing disk 1 to see if any drives had reported errors/issues and it all looked clean. So there was no hint that disk 3 was bad.
The SMART stats in the logs depend on the disk firmware discovering issues. Sometimes those issues don't show up until sectors are read or rewritten. FWIW, there are maitenance functions that you can run on schedule in the NAS that can help detect disk problems earlier.
If you have no backup, you will probably need data recovery, using the original disk 1, disk 2, and disk 4. (One lesson of course is that RAID alone isn't enough to keep your data safe).
Can you connect the disks to a Windows PC (using either SATA or a USB/SATA adapter/dock)? If so, try testing all the disks with Western Digital's lifeguard program.
