NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.

Forum Discussion

Avi_Drissman's avatar
Avi_Drissman
Aspirant
Dec 03, 2014

Getting "Input/output error"s! ReadyNAS doesn't care.

(Continuing from viewtopic.php?f=7&t=78769&p=442201)

A few days ago a share of mine went suddenly read-only. Pulling the logs revealed:

Dec  1 03:50:04 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4989: inode #52462369: block 4142678096: comm python: invalid block
Dec  1 03:50:04 bucket kernel: Aborting journal on device dm-0-8.
Dec  1 03:50:04 bucket kernel: EXT4-fs (dm-0): Remounting filesystem read-only
Dec  1 03:50:04 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #52462371: comm python: bad extended attribute block 10303626543104
Dec  1 03:50:05 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #52461977: comm python: deleted inode referenced: 52462373
Dec  1 03:50:05 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #52461977: comm python: deleted inode referenced: 52462379
Dec  1 03:50:21 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #52461976: comm python: deleted inode referenced: 52462372
Dec  1 03:50:21 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #52461976: comm python: deleted inode referenced: 52462374
Dec  1 03:50:21 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4989: inode #52462378: block 4142682048: comm python: invalid block
Dec  1 03:50:21 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:5021: inode #52462380: comm python: bogus i_mode (135300)
Dec  1 03:50:21 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #52462382: comm python: bad extended attribute block 9655233829812


So I rebooted with a scan and got:

***** File system check forced at Mon Dec  1 09:04:42 EST 2014 *****
fsck 1.42.8 (20-Jun-2013)
e2fsck 1.42.8 (20-Jun-2013)
/dev/c/c: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (1512517731, counted=1492196226).
Fix? yes

Free inodes count wrong (182203254, counted=182193565).
Fix? yes


/dev/c/c: ***** FILE SYSTEM WAS MODIFIED *****
/dev/c/c: 561763/182755328 files (2.8% non-contiguous), 1431889022/2924085248 blocks


What. I did a diff with my backup and found an affected folder. I tried to put some files back and got an error. So I ssh-ed in.

# cd affecteddirectory
# ls
01. xxx.mp3  04. xxx.mp3
02. xxx.mp3  05. xxx.mp3
03. xxx.mp3  06. xxx.mp3
# ls -al
ls: cannot access 03. xxx.mp3: Input/output error
ls: cannot access 01. xxx.mp3: Input/output error
ls: cannot access 02. xxx.mp3: Input/output error
ls: cannot access 04. xxx.mp3: Input/output error
total 2147484464
drwxr-sr-x     3 avi   users       4096 2008-05-27 00:13 .
drwxr-sr-x    13 avi   users       4096 2008-05-27 00:13 ..
-?????????     ? ?     ?              ?                ? 01. xxx.mp3
-?????????     ? ?     ?              ?                ? 02. xxx.mp3
-?????????     ? ?     ?              ?                ? 03. xxx.mp3
-?????????     ? ?     ?              ?                ? 04. xxx.mp3
#


What is going on here? All the SMART stuff is happy... allegedly. I don't know what to trust. What does an "I/O error" mean on a RAID volume?
Issues with Boot|Installation|Upgrade|Expansion
Other Discussions

22 Replies

Replies have been turned off for this discussion
Show More
  • mdgm-ntgr's avatar
    mdgm-ntgr
    NETGEAR Employee Retired
    Dec 04, 2014
    If we can't detect which disk is the problem due to the nature of the problem then running extended test is necessary.

    An extended test will pickup a lot more than a short online test will.
  • Avi_Drissman's avatar
    Avi_Drissman
    Aspirant
    Dec 04, 2014
    So the disk test completed overnight and I can't find any sign of what the result is. The NAS was running in the morning so I have to assume that they all passed. I'm going to kick in another test now, and try a real test tomorrow (busy today with other stuff).
  • StephenB's avatar
    StephenB
    Guru - Experienced User
    Dec 04, 2014
    To summarize
    - you are seeing what looks like disk i/o errors in the log
    - there still are no SMART reallocated or pending sectors, no ATA errors
    -The smart disk check appears to be completing without error.
    - There is no indication of which disk is failing anywhere

    Is that correct?
  • Avi_Drissman's avatar
    Avi_Drissman
    Aspirant
    Dec 04, 2014
    StephenB wrote:
    To summarize
    - you are seeing what looks like disk i/o errors in the log
    - there still are no SMART reallocated or pending sectors, no ATA errors
    -The smart disk check appears to be completing without error.
    - There is no indication of which disk is failing anywhere


    • I am seeing what are clearly disk IO errors in the logs and in interactive use ("ls: cannot access <<filenamehere>>: Input/output error")

    • SMART looks entirely clean to me

    • The boot-time disk test ends with a boot; I have to assume that means "no error"

    • There is no indication in the UI at all that any disk is failing
  • dsm1212's avatar
    dsm1212
    Apprentice
    Dec 04, 2014
    "Disk" io errors will likely show up against the disk device. dm complaining without seeing those disk device errors may just mean that the data is bad on disk. dm is a logical device that sort of lets you logically organize partitions. For example, if you suddenly lost power on a system, an extended attribute page could get "corrupted" and you'd see what you are seeing. The disk in that case could be fine. This is why the first person asked if you were having clean shutdowns. It's possible there was a real disk IO error earlier in the log or something that cause the corruption too. Which is why no one ruled out bad disk.

    Can you show what the system log contained from the timeframe when you did the ls? There should have been an error logged then.

    steve
  • StephenB's avatar
    StephenB
    Guru - Experienced User
    Dec 04, 2014
    I agree that it seems possible that file system corruption could lead to io errors in the log.

    If a disk read or write actually fails, then that should show up in the SMART stats (which are kept by the disk itself, not the NAS).

    There are some failures where it might be possible to easily tell where the fault lies (system, or which drive). For instance, if a drive interface failure somehow hung the SATA bus controller.
  • Avi_Drissman's avatar
    Avi_Drissman
    Aspirant
    Dec 04, 2014
    It's hard to match up exactly the log entries.

    Here's something from 4am. This is afpd, so this wasn't me doing ls:

    Dec  2 04:54:07 bucket kernel: ------------[ cut here ]------------
    Dec  2 04:54:07 bucket kernel: WARNING: at fs/buffer.c:1189 __brelse+0x2b/0x30()
    Dec  2 04:54:07 bucket kernel: Hardware name: ReadyNAS
    Dec  2 04:54:07 bucket kernel: VFS: brelse: Trying to free free buffer
    Dec  2 04:54:07 bucket kernel: Modules linked in: pvgpio nv6vpd(P)
    Dec  2 04:54:07 bucket kernel: Pid: 11707, comm: afpd Tainted: P            2.6.37.6.RNx86_64.2.4 #1
    Dec  2 04:54:07 bucket kernel: Call Trace:
    Dec  2 04:54:07 bucket kernel:  [<ffffffff880d2eeb>] ? __brelse+0x2b/0x30
    Dec  2 04:54:07 bucket kernel:  [<ffffffff8803432b>] warn_slowpath_common+0x8b/0xc0
    Dec  2 04:54:07 bucket kernel:  [<ffffffff88034459>] warn_slowpath_fmt+0x69/0x70
    Dec  2 04:54:07 bucket kernel:  [<ffffffff882bca7e>] ? radix_tree_lookup_slot+0xe/0x10
    Dec  2 04:54:07 bucket kernel:  [<ffffffff880d2c1f>] ? __find_get_block_slow+0x9f/0x110
    Dec  2 04:54:07 bucket kernel:  [<ffffffff880d2eeb>] __brelse+0x2b/0x30
    Dec  2 04:54:07 bucket kernel:  [<ffffffff880d30d5>] __find_get_block+0x145/0x180
    Dec  2 04:54:07 bucket kernel:  [<ffffffff880d3131>] __getblk+0x21/0x260
    Dec  2 04:54:07 bucket kernel:  [<ffffffff88138a80>] __ext4_get_inode_loc+0x120/0x3d0
    Dec  2 04:54:07 bucket kernel:  [<ffffffff8813f3b7>] ext4_iget+0x77/0x790
    Dec  2 04:54:07 bucket kernel:  [<ffffffff881446e5>] ext4_lookup+0xd5/0x110
    Dec  2 04:54:07 bucket kernel:  [<ffffffff880b7d3b>] d_alloc_and_lookup+0x4b/0x80
    Dec  2 04:54:07 bucket kernel:  [<ffffffff880b804c>] do_lookup+0x11c/0x160
    Dec  2 04:54:07 bucket kernel:  [<ffffffff880b93c9>] do_last+0x1a9/0x650
    Dec  2 04:54:07 bucket kernel:  [<ffffffff880ba6e8>] do_filp_open+0x1f8/0x590
    Dec  2 04:54:07 bucket kernel:  [<ffffffff880bfecb>] ? dput+0x12b/0x130
    Dec  2 04:54:07 bucket kernel:  [<ffffffff880b7c0c>] ? path_put+0x2c/0x40
    Dec  2 04:54:07 bucket kernel:  [<ffffffff880b23ec>] ? vfs_fstatat+0x6c/0x70
    Dec  2 04:54:07 bucket kernel:  [<ffffffff880c5263>] ? alloc_fd+0x43/0x130
    Dec  2 04:54:07 bucket kernel:  [<ffffffff880ac9b5>] do_sys_open+0x65/0x110
    Dec  2 04:54:07 bucket kernel:  [<ffffffff880ea5d5>] compat_sys_open+0x15/0x20
    Dec  2 04:54:07 bucket kernel:  [<ffffffff88024c53>] ia32_sysret+0x0/0x5
    Dec  2 04:54:07 bucket kernel: ---[ end trace 810179052d77effb ]---


    Here's another incident:

    Dec  2 05:08:58 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4989: inode #132252657: block 4294967295: comm afpd: invalid block
    Dec  2 05:08:58 bucket kernel: Aborting journal on device dm-0-8.
    Dec  2 05:08:58 bucket kernel: EXT4-fs (dm-0): Remounting filesystem read-only
    Dec  2 05:08:58 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4989: inode #132252657: block 4294967295: comm afpd: invalid block
    Dec  2 05:08:59 bucket last message repeated 3 times
    Dec  2 05:08:59 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  2 05:08:59 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  2 05:08:59 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252665
    Dec  2 05:08:59 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #132252658: comm afpd: bad extended attribute block 281470681743360
    Dec  2 05:08:59 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252663
    Dec  2 05:08:59 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252667
    Dec  2 05:08:59 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #132252658: comm afpd: bad extended attribute block 281470681743360
    Dec  2 05:09:00 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252663
    Dec  2 05:09:00 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252667
    Dec  2 05:09:00 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252663
    Dec  2 05:09:00 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252667
    Dec  2 05:09:00 bucket last message repeated 2 times
    Dec  2 05:09:12 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252645: comm afpd: deleted inode referenced: 132252660
    Dec  2 05:09:12 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252645: comm afpd: deleted inode referenced: 132252671


    And another:

    Dec  3 00:11:33 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252645: comm afpd: deleted inode referenced: 132252660
    Dec  3 00:11:33 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252645: comm afpd: deleted inode referenced: 132252671
    Dec  3 00:11:40 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 00:11:40 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 00:11:40 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252665
    Dec  3 00:11:40 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #132252658: comm afpd: bad extended attribute block 281470681743360
    Dec  3 00:11:40 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252663
    Dec  3 00:11:40 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252667
    Dec  3 00:11:44 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 00:11:44 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 00:11:45 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252665
    Dec  3 00:11:45 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #132252658: comm afpd: bad extended attribute block 281470681743360
    Dec  3 00:11:45 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252663
    Dec  3 00:11:45 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252667
    Dec  3 00:12:01 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4989: inode #132252657: block 4294967295: comm afpd: invalid block
    Dec  3 00:12:01 bucket last message repeated 3 times


    A little one (I think this is the ls; all the others are afpd):

    Dec  3 05:12:37 bucket kernel: EXT4-fs (dm-0): error count: 38
    Dec  3 05:12:37 bucket kernel: EXT4-fs (dm-0): initial error at 1417514938: ext4_iget:4989: inode 132252657: block 4294967295
    Dec  3 05:12:37 bucket kernel: EXT4-fs (dm-0): last error at 1417583521: ext4_iget:4989: inode 132252657: block 4294967295


    And a final one:

    Dec  3 08:16:56 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:16:56 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:16:56 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252665
    Dec  3 08:16:56 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #132252658: comm afpd: bad extended attribute block 281470681743360
    Dec  3 08:16:56 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252663
    Dec  3 08:16:56 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252667
    Dec  3 08:16:57 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:16:57 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:16:57 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252665
    Dec  3 08:16:57 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #132252658: comm afpd: bad extended attribute block 281470681743360
    Dec  3 08:16:57 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252663
    Dec  3 08:16:57 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252667
    Dec  3 08:16:59 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:16:59 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:16:59 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252665
    Dec  3 08:17:00 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #132252658: comm afpd: bad extended attribute block 281470681743360
    Dec  3 08:17:00 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252663
    Dec  3 08:17:00 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252667
    Dec  3 08:17:06 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:17:06 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:17:06 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252665
    Dec  3 08:17:06 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #132252658: comm afpd: bad extended attribute block 281470681743360
    Dec  3 08:17:06 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252663
    Dec  3 08:17:06 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252667
    Dec  3 08:17:20 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #132252658: comm afpd: bad extended attribute block 281470681743360
    Dec  3 08:17:21 bucket last message repeated 4 times
    Dec  3 08:17:23 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:17:23 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:17:23 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252665
    Dec  3 08:17:23 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #132252658: comm afpd: bad extended attribute block 281470681743360
    Dec  3 08:17:23 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252663
    Dec  3 08:17:23 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252667
    Dec  3 08:17:28 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:17:28 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:17:28 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252665
    Dec  3 08:17:28 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #132252658: comm afpd: bad extended attribute block 281470681743360
    Dec  3 08:17:28 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252663
    Dec  3 08:17:28 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252667
    Dec  3 08:17:29 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:17:30 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:17:30 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252665
    Dec  3 08:17:30 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #132252658: comm afpd: bad extended attribute block 281470681743360
    Dec  3 08:17:30 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252663
    Dec  3 08:17:30 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252667
    Dec  3 08:17:31 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:17:31 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:17:31 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252665
    Dec  3 08:17:31 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #132252658: comm afpd: bad extended attribute block 281470681743360
    Dec  3 08:17:31 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252663
    Dec  3 08:17:32 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252667
    Dec  3 08:17:32 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:17:32 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252517: comm afpd: deleted inode referenced: 132252670
    Dec  3 08:17:32 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252665
    Dec  3 08:17:32 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #132252658: comm afpd: bad extended attribute block 281470681743360
    Dec  3 08:17:32 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252663
    Dec  3 08:17:32 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm afpd: deleted inode referenced: 132252667
    Dec  3 08:19:11 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm ls: deleted inode referenced: 132252665
    Dec  3 08:19:11 bucket kernel: EXT4-fs error (device dm-0): ext4_iget:4976: inode #132252658: comm ls: bad extended attribute block 281470681743360
    Dec  3 08:19:11 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm ls: deleted inode referenced: 132252663
    Dec  3 08:19:11 bucket kernel: EXT4-fs error (device dm-0): ext4_lookup:1043: inode #132252516: comm ls: deleted inode referenced: 132252667


    Meanwhile, everything is now behaving. I can go into that directory, ls, and succeed.
  • dsm1212's avatar
    dsm1212
    Apprentice
    Dec 05, 2014
    First if it is working now, take a good backup. Best bet is to open a case with Netgear.

    But, if you are out of warranty and still reading...

    I don't see anything obvious. I'd run tests on the drives using vendor software and system memory using the nas boot tests in particular. Remember to make note which slot each drive came from. This drive test will also get the drives reseated which is good. Also this might be a good time to check and update the firmware on them.

    A couple of the entries are interesting. The error on ext4_lookup that says "deleted inode referenced" means it actually walked a stale inode. That's different than garbage. It could be the code just ignored an error and processed some stale memory, but do you happen to have disk write caching enabled in the nas performance settings? If so, try turning that off for a while and see if you get more obvious in band errors. Deferred write errors are ignored by Linux (at least on the scsi stack that I'm familiar with). In theory software raid could help repair them but it's not even notified. Also almost every drive vendor has fixed firmware bugs in the area of disk write caching over the last couple of years.

    steve
  • Avi_Drissman's avatar
    Avi_Drissman
    Aspirant
    Dec 05, 2014
    I'm well out of warranty.

    Re testing the drives: working on it.

    Re caching: yes, I do. :\ I'm turning off "Enable disk write cache" and "Disable full data journaling". Write speed really isn't that important, anyway.

    Thanks for your thoughts.
  • StephenB's avatar
    StephenB
    Guru - Experienced User
    Dec 05, 2014
    Have you run the NAS memory test from the front panel?

NETGEAR Academy

Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology! 

Join Us!

ProSupport for Business

Comprehensive support plans for maximum network uptime and business peace of mind.

 

Learn More