WX31D55DF4Y7 appears to be showing 6 pending sectors. For the past 24hours. (Now 16:48 12-6-19 in AU).
Is the NAS telling me the degraded state is because of this drive? Id rather not replace a perfectly good drive, only to find out I didn't need to as hard drives are expensive here!
Thanks
Model: RN31400|ReadyNAS 300 Series 4- Bay (Diskless)
Is the NAS telling me the degraded state is because of this drive? Id rather not replace a perfectly good drive, only to find out I didn't need to as hard drives are expensive here!
Can you post mdstat.log? (cut and paste into a reply)
Also, look in disk_info.log for the details of the SMART stats, and look in system.log and kernel.log for disk i/o errors.
FWIW, I recently found that I had a failed WD60EFRX even though the SMART stats looked ok. It was still under warranty, so I got a recertified replacement from Western Digital.
kernel log is reporting Buffer I/O error on dev sdc3. sector 9437258 , 9437260 etc.
...
md127 : active raid5 sda3[7] sdd3[4] sdb3[6]
8776243968 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [UU_U]
/dev/md/data-0: State : clean, degraded md0 : active raid1 sda1[7] sdd1[4] sdb1[6] 4192192 blocks super 1.2 [4/3] [UU_U] unused devices: <none> /dev/md/0: State : clean, degraded
The relevant bits of mdstat are above.
Your data volume has two RAID groups - md126 and md127. That's because you vertically expanded your RAID array at some point (starting off with 3 TB drives and upgrading to 6 TB later).
sdc is part of md126 and md1 (though I didn't quote those bits). But it's not part of md0 (which is the NAS OS partition) and it's not part of md127 (half of your data volume) - and both are marked as degraded.
So sdc is the culprit - some pending sectors, some kernel errors, and it's dropped out of two of the four RAID groups in your system. It would be good to update your backup right away, since your data is at risk.
If you have ssh enabled (or are willing to do that), then you could try entering
# smartctl -x /dev/sdc
This will give you extended SMART status. The interesting part is the queue of failed commands. For instance, this snippet from one of my own WD60EFRX disks:
Error 12 [11] occurred at disk power-on lifetime: 36166 hours (1506 days + 22 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 00 0c 27 df 40 40 00 Error: UNC at LBA = 0x0c27df40 = 203939648
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 80 00 c8 00 00 0c 27 df 40 40 08 1d+08:15:46.204 READ FPDMA QUEUED
60 00 08 00 c0 00 01 06 34 36 98 40 08 1d+08:15:46.163 READ FPDMA QUEUED
60 00 80 00 b8 00 00 0c 27 e4 40 40 08 1d+08:15:46.146 READ FPDMA QUEUED
60 00 80 00 b0 00 00 0c 27 e9 40 40 08 1d+08:15:46.123 READ FPDMA QUEUED
60 00 80 00 a8 00 00 0c 27 ee 40 40 08 1d+08:15:46.094 READ FPDMA QUEUED
This particular snippet shows an unrecoverable error (UNC). I suspect you'll see an error like this around the time of the buffer error (and perhaps more). If the drive is still under warranty, you could RMA it. I've sometimes returned drives that passed Lifeguard (and Seatools), and neither vendor challenged them.
I'd suggest removing the drive and testing it in a windows PC with WD's lifeguard program. If the long generic test passes, then I suggest following up with the destructive write-zeros test. I've found that the write-zeros test will sometimes pick up issues that the non-destructive test misses (and vice-versa). If the disk passes, then check the SMART stats. If they look good, then you could try adding it to the array again (though if you do that, keep a close eye on it).
My volume is in bad shape. from /sdc/ - errors such as this
Error 6799 [6] occurred at disk power-on lifetime: 18909 hours (787 days + 21 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 00 00 90 00 49 40 00 Error: UNC at LBA = 0x00900049 = 9437257
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 07 00 20 00 00 00 90 00 49 40 08 04:42:30.908 READ FPDMA QUEUED
60 00 01 00 10 00 00 00 90 00 48 40 08 04:42:30.908 READ FPDMA QUEUED
ef 00 10 00 02 00 00 00 00 00 00 a0 08 04:42:30.907 SET FEATURES [Enable SATA feature]
27 00 00 00 00 00 00 00 00 00 00 e0 08 04:42:30.907 READ NATIVE MAX ADDRESS EXT [OBS-ACS-3]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 04:42:30.907 IDENTIFY DEVICE
I decided to check sda, sdb and sdd (which I assume are all 4 phyiscal drives). This appears for sdd.
Error 35 [10] occurred at disk power-on lifetime: 15466 hours (644 days + 10 hours)
When the command that caused the error occurred, the device was in standby mode.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 00 00 f6 f1 38 40 00 Error: UNC at LBA = 0x00f6f138 = 16183608
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 08 00 90 00 00 04 3f 0c e0 40 08 03:53:07.269 READ FPDMA QUEUED
e5 00 00 00 00 00 00 00 00 00 00 00 08 03:52:34.053 CHECK POWER MODE
e5 00 00 00 00 00 00 00 00 00 00 00 08 03:52:34.052 CHECK POWER MODE
e5 00 00 00 00 00 00 00 00 00 00 00 08 03:50:34.033 CHECK POWER MODE
e5 00 00 00 00 00 00 00 00 00 00 00 08 03:50:34.032 CHECK POWER MODE
I suspect that both drives (3&4) will need replacing? Any suggestion as to how best do this? This is a dump volume. I store everything and anything on this. Anything of critical importance is backed up to another Netgear NAS running JBOD.
Error 35 [10] occurred at disk power-on lifetime: 15466 hours (644 days + 10 hours)
When the command that caused the error occurred, the device was in standby mode.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 00 00 00 00 f6 f1 38 40 00 Error: UNC at LBA = 0x00f6f138 = 16183608
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 08 00 90 00 00 04 3f 0c e0 40 08 03:53:07.269 READ FPDMA QUEUED
e5 00 00 00 00 00 00 00 00 00 00 00 08 03:52:34.053 CHECK POWER MODE
e5 00 00 00 00 00 00 00 00 00 00 00 08 03:52:34.052 CHECK POWER MODE
e5 00 00 00 00 00 00 00 00 00 00 00 08 03:50:34.033 CHECK POWER MODE
e5 00 00 00 00 00 00 00 00 00 00 00 08 03:50:34.032 CHECK POWER MODE
I suspect that both drives (3&4) will need replacing?
I'd replace sdc, since you are seeing operational issues with it in addition to the errors.
I'm not so sure about sdd. Certainly there is evidence that it might be failing - so you should at least test it. Power down the NAS when you do that (since you don't want to risk resyncs).
FWIW, I had some file access issues a couple months ago (file explorer started timing out, also my media player was misbehaving). I had one disk (sdc) showing errors in kernel.log at the time the media player failed, and three disks showing the UNCs (including sdc). One disk was completely clean.
I tested sdc with lifeguard - it failed so I replaced it. The usual process worked - there was no volume collapse during the resync.
I then replaced a second one - though when I tested it, it passed the Lifeguard tests I mentioned above. I've set that disk aside - I'm not sure if I can trust it or not. But I didn't want two disks with those errors in the same NAS.
I've left the third disk in service for now (the error snipped I posted was on that drive). After all, it had survived two recent resyncs (and also a scrub) without causing volume loss or a degraded volume. The UNC rate is about once every 3 months. If it shows another one I'll replace it also.
As recommended, I'll replace SDC and consider replacing SDD. If SDD tests OK, at least I'll have a spare drive on-hand for any future issues. At present, I potentially have two failing drives - a precarious position to be in. So keen to get back to having 4 functional drives.
