Dec 2 04:00:05 bucket RAIDiator: The on-line filesystem consistency check has started for Volume C. Dec 2 04:00:09 bucket RAIDiator: Volume consistency check started for Volume C. (bucket) : The on-line filesystem consistency check has started for Volume C. Dec 2 04:00:15 bucket kernel: EXT4-fs (dm-3): INFO: recovery required on readonly filesystem Dec 2 04:00:15 bucket kernel: EXT4-fs (dm-3): write access will be enabled during recovery Dec 2 04:00:19 bucket kernel: EXT4-fs (dm-3): recovery complete Dec 2 04:00:20 bucket kernel: EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: acl,user_xattr Dec 2 04:05:40 bucket RAIDiator: The on-line filesystem consistency check completed without errors for Volume C. Dec 2 04:05:44 bucket RAIDiator: Volume consistency check completed for Volume C. (bucket) : The on-line filesystem consistency check completed without errors for Volume C.
I would power down, remove each disk in turn and test it using vendor tools, use extended tests. Label the disks according to slot number. Don't power on unless you have reinserted all the disks again. What disks are they? Are you running 4.2.27?
Do you have a backup as StephenB suggested you should?
Bad disk, fine, but I'm bummed that the ReadyNAS isn't saying "hey you have a bad disk". They're Hitachi HUS724040ALE640, right off the HCL. I have zero ability to test disks using vendor tools. http://www.hgst.com/support/downloads appears to be Windows-only.
I'm running 4.2.26. I was going to upgrade to .27 but now doesn't seem to be a good time.
I have a sorta-backup from when I switched disks two months ago.
I had shut the ReadyNAS down, so to get the logs I needed to boot it. Of course, more errors...
***** File system check performed at Wed Dec 3 21:19:27 EST 2014 ***** fsck 1.42.8 (20-Jun-2013) /dev/c/c: recovering journal /dev/c/c contains a file system with errors, check forced. /dev/c/c: 561818/182755328 files (2.8% non-contiguous), 1433763287/2924085248 blocks
I don't trust the SMART stats. I replaced all my drives two months ago, and I/O errors have to be coming from somewhere. Would be nice if the ReadyNAS errored out the disk rather than sending me on a rat chase to track them down myself.
So the disk test completed overnight and I can't find any sign of what the result is. The NAS was running in the morning so I have to assume that they all passed. I'm going to kick in another test now, and try a real test tomorrow (busy today with other stuff).
To summarize - you are seeing what looks like disk i/o errors in the log - there still are no SMART reallocated or pending sectors, no ATA errors -The smart disk check appears to be completing without error. - There is no indication of which disk is failing anywhere
To summarize - you are seeing what looks like disk i/o errors in the log - there still are no SMART reallocated or pending sectors, no ATA errors -The smart disk check appears to be completing without error. - There is no indication of which disk is failing anywhere
I am seeing what are clearly disk IO errors in the logs and in interactive use ("ls: cannot access <<filenamehere>>: Input/output error")
SMART looks entirely clean to me
The boot-time disk test ends with a boot; I have to assume that means "no error"
There is no indication in the UI at all that any disk is failing
"Disk" io errors will likely show up against the disk device. dm complaining without seeing those disk device errors may just mean that the data is bad on disk. dm is a logical device that sort of lets you logically organize partitions. For example, if you suddenly lost power on a system, an extended attribute page could get "corrupted" and you'd see what you are seeing. The disk in that case could be fine. This is why the first person asked if you were having clean shutdowns. It's possible there was a real disk IO error earlier in the log or something that cause the corruption too. Which is why no one ruled out bad disk.
Can you show what the system log contained from the timeframe when you did the ls? There should have been an error logged then.
I agree that it seems possible that file system corruption could lead to io errors in the log.
If a disk read or write actually fails, then that should show up in the SMART stats (which are kept by the disk itself, not the NAS).
There are some failures where it might be possible to easily tell where the fault lies (system, or which drive). For instance, if a drive interface failure somehow hung the SATA bus controller.
First if it is working now, take a good backup. Best bet is to open a case with Netgear.
But, if you are out of warranty and still reading...
I don't see anything obvious. I'd run tests on the drives using vendor software and system memory using the nas boot tests in particular. Remember to make note which slot each drive came from. This drive test will also get the drives reseated which is good. Also this might be a good time to check and update the firmware on them.
A couple of the entries are interesting. The error on ext4_lookup that says "deleted inode referenced" means it actually walked a stale inode. That's different than garbage. It could be the code just ignored an error and processed some stale memory, but do you happen to have disk write caching enabled in the nas performance settings? If so, try turning that off for a while and see if you get more obvious in band errors. Deferred write errors are ignored by Linux (at least on the scsi stack that I'm familiar with). In theory software raid could help repair them but it's not even notified. Also almost every drive vendor has fixed firmware bugs in the area of disk write caching over the last couple of years.
Re caching: yes, I do. 😕 I'm turning off "Enable disk write cache" and "Disable full data journaling". Write speed really isn't that important, anyway.
