NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Getting "Input/output error"s! ReadyNAS doesn't care.
(Continuing from viewtopic.php?f=7&t=78769&p=442201)
A few days ago a share of mine went suddenly read-only. Pulling the logs revealed:
So I rebooted with a scan and got:
What. I did a diff with my backup and found an affected folder. I tried to put some files back and got an error. So I ssh-ed in.
What is going on here? All the SMART stuff is happy... allegedly. I don't know what to trust. What does an "I/O error" mean on a RAID volume?
A few days ago a share of mine went suddenly read-only. Pulling the logs revealed:
Dec 1 03:50:04 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4989: inode #52462369: block 4142678096: comm python: invalid block
Dec 1 03:50:04 bucket kernel: Aborting journal on device dm-0-8.
Dec 1 03:50:04 bucket kernel: EXT4-fs (dm-0): Remounting filesystem read-only
Dec 1 03:50:04 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #52462371: comm python: bad extended attribute block 10303626543104
Dec 1 03:50:05 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #52461977: comm python: deleted inode referenced: 52462373
Dec 1 03:50:05 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #52461977: comm python: deleted inode referenced: 52462379
Dec 1 03:50:21 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #52461976: comm python: deleted inode referenced: 52462372
Dec 1 03:50:21 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #52461976: comm python: deleted inode referenced: 52462374
Dec 1 03:50:21 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4989: inode #52462378: block 4142682048: comm python: invalid block
Dec 1 03:50:21 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:5021: inode #52462380: comm python: bogus i_mode (135300)
Dec 1 03:50:21 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #52462382: comm python: bad extended attribute block 9655233829812
So I rebooted with a scan and got:
***** File system check forced at Mon Dec 1 09:04:42 EST 2014 *****
fsck 1.42.8 (20-Jun-2013)
e2fsck 1.42.8 (20-Jun-2013)
/dev/c/c: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (1512517731, counted=1492196226).
Fix? yes
Free inodes count wrong (182203254, counted=182193565).
Fix? yes
/dev/c/c: ***** FILE SYSTEM WAS MODIFIED *****
/dev/c/c: 561763/182755328 files (2.8% non-contiguous), 1431889022/2924085248 blocks
What. I did a diff with my backup and found an affected folder. I tried to put some files back and got an error. So I ssh-ed in.
# cd affecteddirectory
# ls
01. xxx.mp3 04. xxx.mp3
02. xxx.mp3 05. xxx.mp3
03. xxx.mp3 06. xxx.mp3
# ls -al
ls: cannot access 03. xxx.mp3: Input/output error
ls: cannot access 01. xxx.mp3: Input/output error
ls: cannot access 02. xxx.mp3: Input/output error
ls: cannot access 04. xxx.mp3: Input/output error
total 2147484464
drwxr-sr-x 3 avi users 4096 2008-05-27 00:13 .
drwxr-sr-x 13 avi users 4096 2008-05-27 00:13 ..
-????????? ? ? ? ? ? 01. xxx.mp3
-????????? ? ? ? ? ? 02. xxx.mp3
-????????? ? ? ? ? ? 03. xxx.mp3
-????????? ? ? ? ? ? 04. xxx.mp3
#
What is going on here? All the SMART stuff is happy... allegedly. I don't know what to trust. What does an "I/O error" mean on a RAID volume?
22 Replies
Replies have been turned off for this discussion
- And it also seems to be flat-out lying to me?
Dec 2 04:00:05 bucket RAIDiator: The on-line filesystem consistency check has started for Volume C.
Dec 2 04:00:09 bucket RAIDiator: Volume consistency check started for Volume C. (bucket) : The on-line filesystem consistency check has started for Volume C.
Dec 2 04:00:15 bucket kernel: EXT4-fs (dm-3): INFO: recovery required on readonly filesystem
Dec 2 04:00:15 bucket kernel: EXT4-fs (dm-3): write access will be enabled during recovery
Dec 2 04:00:19 bucket kernel: EXT4-fs (dm-3): recovery complete
Dec 2 04:00:20 bucket kernel: EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: acl,user_xattr
Dec 2 04:05:40 bucket RAIDiator: The on-line filesystem consistency check completed without errors for Volume C.
Dec 2 04:05:44 bucket RAIDiator: Volume consistency check completed for Volume C. (bucket) : The on-line filesystem consistency check completed without errors for Volume C.
No errors. Right. - I/O errors usually indicate a bad disk.
I would power down, remove each disk in turn and test it using vendor tools, use extended tests. Label the disks according to slot number. Don't power on unless you have reinserted all the disks again.
What disks are they?
Are you running 4.2.27?
Do you have a backup as StephenB suggested you should? - Bad disk, fine, but I'm bummed that the ReadyNAS isn't saying "hey you have a bad disk". They're Hitachi HUS724040ALE640, right off the HCL. I have zero ability to test disks using vendor tools. http://www.hgst.com/support/downloads appears to be Windows-only.
I'm running 4.2.26. I was going to upgrade to .27 but now doesn't seem to be a good time.
I have a sorta-backup from when I switched disks two months ago. - Can you send me your logs (see the Sending Logs link in my sig)?
You could run the Disk Test boot menu option. - Re logs, will do. Re boot menu option, that's a great idea; I'd totally forgotten about that.
- I had shut the ReadyNAS down, so to get the logs I needed to boot it. Of course, more errors...
***** File system check performed at Wed Dec 3 21:19:27 EST 2014 *****
fsck 1.42.8 (20-Jun-2013)
/dev/c/c: recovering journal
/dev/c/c contains a file system with errors, check forced.
/dev/c/c: 561818/182755328 files (2.8% non-contiguous), 1433763287/2924085248 blocks
Logs coming your way. - The boot menu option isn't as good as using the manufacturer's tool, but it is the next best thing.
- I'll let it grind overnight with the disk test. I think I may have found someone to let me borrow their Windows box, so we'll see there.
- The disk SMART stats look O.K. but one or more of the disks could still be bad.
- I don't trust the SMART stats. I replaced all my drives two months ago, and I/O errors have to be coming from somewhere. Would be nice if the ReadyNAS errored out the disk rather than sending me on a rat chase to track them down myself.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
