Re: volume degraded

acng · ‎2018-03-20

Hi,

I took over our ReadyNAS environment and one ReadyNAS is in degraded state.It's configured in RAID 6.

First such error was: "Volume: volume data health changed from Redundant to Degraded"

it's followed by numerous "Volume: volume data is Degraded."

When I go to System > Volume, it shows all drives are green and when I mouse over each one of them, they all say ONLINE.

I downloaded the logs and found in mdstat.log:

/dev/md/data-0:

Version : 1.2

Creation Time : Thu Nov 19 22:47:03 2015

Raid Level : raid6

Array Size : 39021667840 (37213.96 GiB 39958.19 GB)

Used Dev Size : 3902166784 (3721.40 GiB 3995.82 GB)

Raid Devices : 12

Total Devices : 12

Persistence : Superblock is persistent

Update Time : Tue Mar 20 14:33:49 2018

State : clean, degraded

Active Devices : 11

Working Devices : 11

Failed Devices : 1

Spare Devices : 0

Layout : left-symmetric

Chunk Size : 64K

Name : 43f68b06:data-0 (local to host 43f68b06)

UUID : e92dd329:1a6ad005:49d51ef6:3035aab6

Events : 2419044

Number Major Minor RaidDevice State

0 8 3 0 active sync /dev/sda3

1 8 35 1 active sync /dev/sdc3

12 8 19 2 active sync /dev/sdb3

3 8 51 3 active sync /dev/sdd3

4 8 67 4 active sync /dev/sde3

5 8 83 5 active sync /dev/sdf3

6 8 115 6 active sync /dev/sdh3

7 8 99 7 active sync /dev/sdg3

8 8 131 8 active sync /dev/sdi3

9 8 147 9 active sync /dev/sdj3

10 8 179 10 active sync /dev/sdl3

22 0 0 22 removed

11 8 195 - faulty /dev/sdm3

My questions are:

1. What exactly is the issue here? partition sdm3 is corrupted? How to fix this kind of errors?

2. I have no idea how long this ReadyNAS has been arouond. Is it something I can buy a drive and replace it? I registered the product and it requires purchase date but I don't have that info. How do I know it's under warranty or not?

thanks,

StephenB · ‎2018-03-23

What do you see in disk_info.log and smart_history.log for that disk?

acng · ‎2018-03-27

Hi,

I don't see anything unusual for sdm in the disk_info.log:

Device:             sdm
Controller:         0
Channel:            3
Model:              WDC WD4000F9YZ-09N20L1
Serial:             WD-WCC5D0HH7DHH
Firmware:           01.01A02
Class:              SATA
RPM:                7200
Sectors:            7814037168
Pool:               data
PoolType:           RAID 6
PoolState:          3
PoolHostId:         43f68b06
Health data
ATA Error Count:                0
Reallocated Sectors:            0
Reallocation Events:            0
Spin Retry Count:               0
Current Pending Sector Count:   0
Uncorrectable Sector Count:     0
Temperature:                    41
Start/Stop Count:               7
Power-On Hours:                 16538
Power Cycle Count:              7
Load Cycle Count:               4

and no error in smart_history.log:

model                 serial                realloc_sect realloc_evnt spin_retry_cnt ioedc       cmd_timeouts pending_sect uncorrectable_err ata_errors timestamp   time
-------------------- -------------------- ------------ ------------ -------------- ---------- ------------ ------------ ----------------- ---------- ---------- -------------------

WDC WD4000F9YZ-09N20 WD-WCC5D0HH7DHH       0             0             0               -1          -1            0             0                  0           1448002169 2015-11-20 06:49:29

However, there have been bunch of errors for sdm in messages.log file.

I have a tough time attaching the messages.log file. Tried messages.log, messages.txt, and even converted to pdf. Will try attach again after this post.

Please advise. Thanks,

acng · ‎2018-03-27

Please see attached.

StephenB · ‎2018-03-27

I suggest powering down the NAS, and then testing sdm in a Windows PC with Western Digital's lifeguard program.

acng · ‎2018-03-27

Thanks but this NAS is at a remote site so it's a bit difficult to do at this moment.

I also found in kernel.log, there are alot of entries like below logged every second. Does it pretty much imply the disk is due to be replaced?

~~~~~~

Mar 20 14:34:02 ut-backup03 kernel: blk_update_request: I/O error, dev sdm, sector 9437257
Mar 20 14:34:02 ut-backup03 kernel: Buffer I/O error on dev sdm3, logical block 1, async page read
Mar 20 14:34:02 ut-backup03 kernel: ata8: EH complete
Mar 20 14:34:02 ut-backup03 kernel: ata8.00: exception Emask 0x0 SAct 0x2000040 SErr 0x0 action 0x0
Mar 20 14:34:02 ut-backup03 kernel: ata8.00: irq_stat 0x40000008
Mar 20 14:34:02 ut-backup03 kernel: ata8.00: failed command: READ FPDMA QUEUED
Mar 20 14:34:02 ut-backup03 kernel: ata8.00: cmd 60/08:30:48:00:90/00:00:00:00:00/40 tag 6 ncq 4096 in
Mar 20 14:34:02 ut-backup03 kernel: res 41/40:00:49:00:90/00:00:00:00:00/40 Emask 0x409 (media error) <F>
Mar 20 14:34:02 ut-backup03 kernel: ata8.00: status: { DRDY ERR }
Mar 20 14:34:02 ut-backup03 kernel: ata8.00: error: { UNC }
Mar 20 14:34:02 ut-backup03 kernel: ata8.00: configured for UDMA/133
Mar 20 14:34:02 ut-backup03 kernel: sd 8:0:0:0: [sdm] tag#6 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Mar 20 14:34:02 ut-backup03 kernel: sd 8:0:0:0: [sdm] tag#6 Sense Key : Medium Error [current] [descriptor]
Mar 20 14:34:02 ut-backup03 kernel: sd 8:0:0:0: [sdm] tag#6 Add. Sense: Unrecovered read error - auto reallocate failed
Mar 20 14:34:02 ut-backup03 kernel: sd 8:0:0:0: [sdm] tag#6 CDB: Read(16) 88 00 00 00 00 00 00 90 00 48 00 00 00 08 00 00
Mar 20 14:34:02 ut-backup03 kernel: blk_update_request: I/O error, dev sdm, sector 9437257
Mar 20 14:34:02 ut-backup03 kernel: Buffer I/O error on dev sdm3, logical block 1, async page read
Mar 20 14:34:02 ut-backup03 kernel: ata8: EH complete

Mar 20 14:34:02 ut-backup03 kernel: BTRFS: could not find root 8
Mar 20 14:34:02 ut-backup03 kernel: BTRFS: could not find root 8

~~~~~~

thanks.

StephenB · ‎2018-03-27

@acng wrote:

I also found in kernel.log, there are alot of entries like below logged every second. Does it pretty much imply the disk is due to be replaced?

Likely yes. Another possibility is that something is wrong with the NAS SATA connection to that bay.