The next step is to test the disks in a Windows PC using vendor tools (seatools for seagate, lifeguard for western digital). They can be connected either with SATA or a USB dock/adapter.
if you have a scratch disk that you know works, then you can also try installing that disk in the NAS and try to do a factory install.
Your issue could be related to the disks themselves, the SATA bus on the NAS, or the 12 volt power from the PSU. The SATA bus wouldn't be repairable, but it is possible to replace the PSU.
What should I watch out for when testing the disks? I assume the tests are safe in such a way that the disks and the data on them can still be used in the NAS (if the disks are fine).
If the factory install works, should I just re-insert my 6 disks? Should I expect to be able to re-access my data?
What should I watch out for when testing the disks? I assume the tests are safe
Windows won't recognize the file system, so the disks won't show up in file explorer. The vendor tools will still find them. The normal tests are non-destructive, some of the advanced tests are destructive, but they will warn you of that if you try to run them.
In addition to the pass/fail reported by the test, it's useful to collect the SMART stats before and after you test each disk. The manufacturer thresholds for pass/fail are higher than the ones I use (they of course want to control warranty costs). Getting the info before and after lets you see if the disk health degrades as a result of the test (which is a bad sign). Both the tools have a way to look at the SMART stats.
If the factory install works, should I just re-insert my 6 disks? Should I expect to be able to re-access my data?
If the factory install works, then try powering down the NAS, and moving the disk to the second slot. If that boots normally, do that again (to the third slot) until you've confirmed that all the bays are working.
If that all works ok, then the SATA bus isn't the issue. But the factory install won't fix anything, so reinstalling your original disks will almost certainly give the same failure you have now.
If all this works and the disks are healthy, then the issue is probably a failing PSU (which can be replaced). If it fails, then it might still be the PSU, or it might be the system board (which can't be).
Good: the NAS seems fine. Tested all 6 bays with a factory reset disk.
Bad: 2 of the 6 disks fail. One doesn't actually get a fail from Seatools, but there clearly is something mechanically wrong with the disk (clicking), and it never even gets to the end of the short self test. The other one fails the self test fairly immediately.
I booted the pioneer with another disk, and got to see the logs. Oddly, nothing of note there. Two theories:
- The NAS was surge protected, but a really bad powercut might have affected the two disks simultaneously. Unlikely.
- Logging was broken. There was nothing logged since end of 2017, which is odd, as I have weekly scrubs scheduled, and these usually get logged.
I have a backup of the most important files, but I'd like to know my options from here. The mechanically failing disk might still be recoverable.
It's not uncommon to see multiple disk failures in RAID. There are a couple of reasons-
we tend to buy the same model drives at the same time. Then we install them all at once into the same chassis, and RAID puts them under near-identical loads. Same disks, same environment, same load - it shouldn't be surprising if they wear out about the same time.
Bad sectors are detected when we try to read or write them (not when they actually go bad). For most people, the bulk of their storage is only rarely accessed. So when a disk is replaced, it can expose pre-existing problems in other disks.
Your weekly scrubs would have uncovered any bad sectors, since every sector in the data volume is either read or written as part of the scrub. So the unexplained dropping of the scrubs is very unfortunate.
You can try RAID recovery (either software or a service). You can also try to clone the disk that isn't clicking, and then try booting up the NAS with the cloned disk in the array (use the boot menu to skip the file system check). Leave the clicking drive out. There will be file system corruption (since bad sectors can't be cloned), but if the bulk of the disk can be read you might be able to get at least some files back.
I might have a go at cloning the non-clicking disk. I can only find old posts about how to do this. What is the optimal solution for cloning XRAID disks today?
As for the clicking disk: would it be worth sending it in for mechnical repair? What are the chances the data is OK?
Once I recup one of the 2 failing disks, I assume this is the process:
- insert my 4 working disks (in their original slots)
- insert my fixed disk (in the same spot as the disk it was cloned/recovered from)
- insert a new disk with sufficient capacity in the remaining slot
- switch the ReadyNAS on and let it do its resync
I have one concern: this seems to be a one shot operation, as the rebuilding will affect the 4 good disks. Is this the optimal process?
How does XRAID deal with bad sectors there might be on the recovered disk? Can it fail the resync? Or is there just loss of some files?
As for the clicking disk: would it be worth sending it in for mechnical repair? What are the chances the data is OK?
I think this depends on the details of the drive model and what exactly failed. I found a youtube that might be helpful ( https://www.youtube.com/watch?v=PhxroolWG28 ) - this is not from a company I've ever used, so it's not a recommendation.
Whether this is worth it depends on how valuable your data is.
I might have a go at cloning the non-clicking disk. I can only find old posts about how to do this. What is the optimal solution for cloning XRAID disks today?
You want a cloning tool that does sector by sector copying. There are several out there, including Clonezilla. All these tools work identically on sections of the disk that aren't damaged - where they might vary is how they handle failed reads. It's hard to say what strategy is best - simply moving past the unreadable parts limits the possibility of causing more damage, but of course in some cases aggressive retrying might recover some additional sectors.
I might have a go at cloning the non-clicking disk. I can only find old posts about how to do this. What is the optimal solution for cloning XRAID disks today?
As for the clicking disk: would it be worth sending it in for mechnical repair? What are the chances the data is OK?
Once I recup one of the 2 failing disks, I assume this is the process:
- insert my 4 working disks (in their original slots)
- insert my fixed disk (in the same spot as the disk it was cloned/recovered from)
- insert a new disk with sufficient capacity in the remaining slot
- switch the ReadyNAS on and let it do its resync
I'd leave out the new disk initially, and restart the NAS using the boot menu option to skip the file system check. There will be no resync, or attempt to repair errors in the file system. The reason for this is that you likely won't be able to copy the entire disk without errors. So there probably will be some file system damage - and the repair attempt done by the file system check might delete files or folders. More details on that below.
If the volume mounts, then back up all the data you can before going further. When that's complete, you can either
reboot the NAS, and hot-insert the blank disk
do a factory reset with all disks in place and recover the data from the backup.
Personally I favor the factory reset, as that will give you a completely clean file system.
I have one concern: this seems to be a one shot operation, as the rebuilding will affect the 4 good disks. Is this the optimal process?
How does XRAID deal with bad sectors there might be on the recovered disk? Can it fail the resync? Or is there just loss of some files?
As far as RAID is concerned, the NAS can't detect any bad sectors on a cloned disk. All the sectors are readable, and it has no way to tell if a sector was successfully copied (or not). The RAID resync reconstructs the blank disk from data and partity blocks on the remaining disks (including the clone). It shouldn't change the contents of those disks, and if there are errors on the clone, there will be more errors on the reconstructed disk. If there is an error reading a sector on one of the non-blank disks, then the RAID sync will fail.
The file system check that follows the resync will attempt to repair any problems with the folder structure (that is metadata), and that can result in deleting folders/files where it finds issues. There could (likely will be) some errors in the files themselves on the cloned drive and the reconstructed disk(corrupted video or music for example). As explained above, RAID can't repair those errors.
These possibilities are why I suggest avoiding the resync and the file system check initially, so you can offload all the files you can. If you have known good copies of some of those files somewhere else, then you might want to compare the backup with the copies (figuring out which you want to keep).
