NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
ReadyNAS Ultra Plus on OS 6.10.1 becoming unresponsive
After a few weeks having my ReadyNAS Ultra 6 Plus upgraded to OS6 I am starting to experience random situations where the NAS goes completely unresponsive. It is no longer accesible via the web inter...
Smaky wrote:
I have seen no other errors like these associated to another service or application. What does these errors mean?
Not 100% sure, but it looks like something in RAM failed to write to the RAID array.
Do you have a backup of the data?
StephenB wrote:
Do you have a backup of the data?For the most part yes, I still have the original 3TB drives that populated the array with all the data on them.
Smaky wrote:
For the most part yes, I still have the original 3TB drives that populated the array with all the data on them.
These were the OS 4.2 disks?
I suggest making a full backup (perhaps purchasing a couple large USB drives if you don't have enough storage). Ideally leave the 4.2 array instact until you've done that. Once the data is completely safe, it'll be easier to work on the problem.
Running smartctl -x on each disk might give some more information - you'd do that from ssh. I recently found a failed WD60ERFX that way in my own system - even though it had good SMART stats. Stored errors don't necessarily mean the disk is bad, but it does suggest that the disk should be tested with lifeguard. I believe Lifeguard also is the right tool for your Hitachi disks (since HGST is part of WDC)
This is the output for the two HGST drives:
smartctl -x /dev/sde smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.4.178.x86_64.1] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: HGST Deskstar NAS Device Model: HGST HDN724040ALE640 Serial Number: PK2338P4GT3WWC LU WWN Device Id: 5 000cca 249caf78b Firmware Version: MJAOA5E0 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Sun Jul 7 22:40:49 2019 CDT SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Unavailable APM feature is: Disabled Rd look-ahead is: Enabled Write cache is: Enabled DSN feature is: Unavailable ATA Security is: Disabled, NOT FROZEN [SEC1] Wt Cache Reorder: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 24) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off supp ort. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 565) minutes. SCT capabilities: (0x003d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate PO-R-- 100 100 016 - 0 2 Throughput_Performance P-S--- 137 137 054 - 77 3 Spin_Up_Time POS--- 130 130 024 - 595 (Average 595) 4 Start_Stop_Count -O--C- 098 098 000 - 11691 5 Reallocated_Sector_Ct PO--CK 100 100 005 - 0 7 Seek_Error_Rate PO-R-- 100 100 067 - 0 8 Seek_Time_Performance P-S--- 121 121 020 - 34 9 Power_On_Hours -O--C- 095 095 000 - 36843 10 Spin_Retry_Count PO--C- 100 100 060 - 0 12 Power_Cycle_Count -O--CK 100 100 000 - 203 192 Power-Off_Retract_Count -O--CK 090 090 000 - 12686 193 Load_Cycle_Count -O--C- 090 090 000 - 12686 194 Temperature_Celsius -O---- 162 162 000 - 37 (Min/Max 19/71) 196 Reallocated_Event_Count -O--CK 100 100 000 - 0 197 Current_Pending_Sector -O---K 100 100 000 - 0 198 Offline_Uncorrectable ---R-- 100 100 000 - 0 199 UDMA_CRC_Error_Count -O-R-- 200 200 000 - 103 ||||||_ K auto-keep |||||__ C event count ||||___ R error rate |||____ S speed/performance ||_____ O updated online |______ P prefailure warning General Purpose Log Directory Version 1 SMART Log Directory Version 1 [multi-sector log support] Address Access R/W Size Description 0x00 GPL,SL R/O 1 Log Directory 0x01 SL R/O 1 Summary SMART error log 0x03 GPL R/O 1 Ext. Comprehensive SMART error log 0x04 GPL R/O 7 Device Statistics log 0x06 SL R/O 1 SMART self-test log 0x07 GPL R/O 1 Extended self-test log 0x08 GPL R/O 2 Power Conditions log 0x09 SL R/W 1 Selective self-test log 0x10 GPL R/O 1 NCQ Command Error log 0x11 GPL R/O 1 SATA Phy Event Counters log 0x20 GPL R/O 1 Streaming performance log [OBS-8] 0x21 GPL R/O 1 Write stream error log 0x22 GPL R/O 1 Read stream error log 0x80-0x9f GPL,SL R/W 16 Host vendor specific log 0xe0 GPL,SL R/W 1 SCT Command/Status 0xe1 GPL,SL R/W 1 SCT Data Transfer SMART Extended Comprehensive Error Log Version: 1 (1 sectors) Device Error Count: 103 (device log contains only the most recent 4 errors) CR = Command Register FEATR = Features Register COUNT = Count (was: Sector Count) Register LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8 LH = LBA High (was: Cylinder High) Register ] LBA LM = LBA Mid (was: Cylinder Low) Register ] Register LL = LBA Low (was: Sector Number) Register ] DV = Device (was: Device/Head) Register DC = Device Control Register ER = Error register ST = Status register Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 103 [2] occurred at disk power-on lifetime: 36330 hours (1513 days + 18 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 84 -- 51 00 18 00 00 00 5a 6c 98 00 00 Error: ICRC, ABRT 24 sectors at LBA = 0x005a6c98 = 5926040 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 35 00 00 00 78 00 00 00 5a 6c 38 40 00 00:03:18.662 WRITE DMA EXT 25 00 00 00 01 00 00 00 00 08 00 40 00 00:03:18.310 READ DMA EXT 25 00 00 00 01 00 00 00 00 08 00 40 00 00:03:18.290 READ DMA EXT 25 00 00 00 01 00 00 00 00 08 00 40 00 00:03:18.284 READ DMA EXT 25 00 00 00 01 00 00 00 00 08 00 40 00 00:03:18.281 READ DMA EXT Error 102 [1] occurred at disk power-on lifetime: 36330 hours (1513 days + 18 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 84 -- 51 00 bf 00 00 00 5a 66 f9 00 00 Error: ICRC, ABRT 191 sectors at LBA = 0x005a66f9 = 5924601 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 35 00 00 00 f0 00 00 00 5a 66 c8 40 00 00:02:10.419 WRITE DMA EXT 25 00 00 00 08 00 00 00 60 1f 48 40 00 00:02:10.418 READ DMA EXT 25 00 00 00 08 00 00 00 60 1f 40 40 00 00:02:10.418 READ DMA EXT 25 00 00 00 08 00 00 00 60 1f 38 40 00 00:02:10.418 READ DMA EXT 35 00 00 00 08 00 00 00 60 1c 58 40 00 00:02:10.417 WRITE DMA EXT Error 101 [0] occurred at disk power-on lifetime: 36330 hours (1513 days + 18 hours) When the command that caused the error occurred, the device was active or idle . After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 84 -- 51 00 01 00 00 00 60 1c 5f 00 00 Error: ICRC, ABRT 1 sectors at LBA = 0x00601c5f = 6298719 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 35 00 00 00 08 00 00 00 60 1c 58 40 00 00:02:10.417 WRITE DMA EXT 25 00 00 00 08 00 00 64 bc 85 70 40 00 00:02:10.404 READ DMA EXT 25 00 00 00 08 00 00 00 60 28 18 40 00 00:02:10.403 READ DMA EXT 25 00 00 00 08 00 00 00 60 28 10 40 00 00:02:10.402 READ DMA EXT 25 00 00 00 08 00 00 00 60 27 f8 40 00 00:02:10.401 READ DMA EXT Error 100 [3] occurred at disk power-on lifetime: 36330 hours (1513 days + 18 hours) When the command that caused the error occurred, the device was active or idle . After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 84 -- 51 00 01 00 00 00 60 5e 07 00 00 Error: ICRC, ABRT 1 sectors at LBA = 0x00605e07 = 6315527 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 25 00 00 00 08 00 00 00 60 5e 00 40 00 00:02:06.250 READ DMA EXT 25 00 00 00 08 00 01 98 92 ed 38 40 00 00:02:06.237 READ DMA EXT 25 00 00 00 08 00 00 00 60 5d e0 40 00 00:02:06.237 READ DMA EXT 25 00 00 00 08 00 00 00 60 08 50 40 00 00:02:05.251 READ DMA EXT 35 00 00 00 10 00 00 00 60 0e 80 40 00 00:02:04.984 WRITE DMA EXT SMART Extended Self-test Log Version: 1 (1 sectors) Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 36560 - # 2 Short offline Completed without error 00% 12664 - # 3 Short offline Completed without error 00% 11256 - # 4 Short offline Completed without error 00% 5622 - # 5 Extended offline Completed without error 00% 5354 - # 6 Extended offline Completed without error 00% 2918 - # 7 Short offline Completed without error 00% 5 - # 8 Short offline Completed without error 00% 5 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. SCT Status Version: 3 SCT Version (vendor specific): 256 (0x0100) SCT Support Level: 1 Device State: SMART Off-line Data Collection executing in background (4) Current Temperature: 36 Celsius Power Cycle Min/Max Temperature: 30/45 Celsius Lifetime Min/Max Temperature: 19/71 Celsius Under/Over Temperature Limit Count: 0/0 SCT Temperature History Version: 2 Temperature Sampling Period: 1 minute Temperature Logging Interval: 1 minute Min/Max recommended Temperature: 0/60 Celsius Min/Max Temperature Limit: -40/70 Celsius Temperature History Size (Index): 128 (63) Index Estimated Time Temperature Celsius 64 2019-07-07 20:34 32 ************* ... ..( 29 skipped). .. ************* 94 2019-07-07 21:04 32 ************* 95 2019-07-07 21:05 33 ************** 96 2019-07-07 21:06 34 *************** 97 2019-07-07 21:07 34 *************** 98 2019-07-07 21:08 35 **************** ... ..( 3 skipped). .. **************** 102 2019-07-07 21:12 35 **************** 103 2019-07-07 21:13 34 *************** ... ..( 8 skipped). .. *************** 112 2019-07-07 21:22 34 *************** 113 2019-07-07 21:23 33 ************** ... ..( 18 skipped). .. ************** 4 2019-07-07 21:42 33 ************** 5 2019-07-07 21:43 32 ************* ... ..( 43 skipped). .. ************* 49 2019-07-07 22:27 32 ************* 50 2019-07-07 22:28 33 ************** 51 2019-07-07 22:29 33 ************** 52 2019-07-07 22:30 34 *************** 53 2019-07-07 22:31 34 *************** 54 2019-07-07 22:32 35 **************** 55 2019-07-07 22:33 36 ***************** 56 2019-07-07 22:34 36 ***************** 57 2019-07-07 22:35 37 ****************** 58 2019-07-07 22:36 37 ****************** 59 2019-07-07 22:37 38 ******************* 60 2019-07-07 22:38 38 ******************* 61 2019-07-07 22:39 37 ****************** 62 2019-07-07 22:40 37 ****************** 63 2019-07-07 22:41 37 ****************** SCT Error Recovery Control: Read: Disabled Write: Disabled Device Statistics (GP Log 0x04) Page Offset Size Value Flags Description 0x01 ===== = = === == General Statistics (rev 2) == 0x01 0x008 4 203 --- Lifetime Power-On Resets 0x01 0x018 6 92113611119 --- Logical Sectors Written 0x01 0x020 6 367341356 --- Number of Write Commands 0x01 0x028 6 221434555420 --- Logical Sectors Read 0x01 0x030 6 994363283 --- Number of Read Commands 0x03 ===== = = === == Rotating Media Statistics (rev 1) == 0x03 0x008 4 34678 --- Spindle Motor Power-on Hours 0x03 0x010 4 34677 --- Head Flying Hours 0x03 0x018 4 12686 --- Head Load Events 0x03 0x020 4 0 --- Number of Reallocated Logical Sectors 0x03 0x028 4 2 --- Read Recovery Attempts 0x03 0x030 4 0 --- Number of Mechanical Start Failures 0x04 ===== = = === == General Errors Statistics (rev 1) == 0x04 0x008 4 0 --- Number of Reported Uncorrectable Errors 0x04 0x010 4 765 --- Resets Between Cmd Acceptance and Completion 0x05 ===== = = === == Temperature Statistics (rev 1) == 0x05 0x008 1 36 --- Current Temperature 0x05 0x010 1 34 N-- Average Short Term Temperature 0x05 0x018 1 42 N-- Average Long Term Temperature 0x05 0x020 1 71 --- Highest Temperature 0x05 0x028 1 19 --- Lowest Temperature 0x05 0x030 1 62 N-- Highest Average Short Term Temperature 0x05 0x038 1 25 N-- Lowest Average Short Term Temperature 0x05 0x040 1 58 N-- Highest Average Long Term Temperature 0x05 0x048 1 25 N-- Lowest Average Long Term Temperature 0x05 0x050 4 27060 --- Time in Over-Temperature 0x05 0x058 1 60 --- Specified Maximum Operating Temperature 0x05 0x060 4 0 --- Time in Under-Temperature 0x05 0x068 1 0 --- Specified Minimum Operating Temperature 0x06 ===== = = === == Transport Statistics (rev 1) == 0x06 0x008 4 787 --- Number of Hardware Resets 0x06 0x010 4 537 --- Number of ASR Events 0x06 0x018 4 103 --- Number of Interface CRC Errors |||_ C monitored condition met ||__ D supports DSN |___ N normalized value Pending Defects log (GP Log 0x0c) not supported SATA Phy Event Counters (GP Log 0x11) ID Size Value Description 0x0001 2 0 Command failed due to ICRC error 0x0002 2 0 R_ERR response for data FIS 0x0003 2 0 R_ERR response for device-to-host data FIS 0x0004 2 0 R_ERR response for host-to-device data FIS 0x0005 2 0 R_ERR response for non-data FIS 0x0006 2 0 R_ERR response for device-to-host non-data FIS 0x0007 2 0 R_ERR response for host-to-device non-data FIS 0x0009 2 5 Transition from drive PhyRdy to drive PhyNRdy 0x000a 2 3 Device-to-host register FISes sent due to a COMRESET 0x000b 2 0 CRC errors within host-to-device FIS 0x000d 2 0 Non-CRC errors within host-to-device FISand
smartctl -x /dev/sdf smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.4.178.x86_64.1] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: HGST Deskstar NAS Device Model: HGST HDN724040ALE640 Serial Number: PK1334PCJAZHES LU WWN Device Id: 5 000cca 24ce12d34 Firmware Version: MJAOA5E0 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Sun Jul 7 22:47:55 2019 CDT SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Unavailable APM feature is: Disabled Rd look-ahead is: Enabled Write cache is: Enabled DSN feature is: Unavailable ATA Security is: Disabled, NOT FROZEN [SEC1] Wt Cache Reorder: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 24) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 560) minutes. SCT capabilities: (0x003d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate PO-R-- 100 100 016 - 0 2 Throughput_Performance P-S--- 137 137 054 - 78 3 Spin_Up_Time POS--- 150 150 024 - 600 (Average 432) 4 Start_Stop_Count -O--C- 098 098 000 - 11213 5 Reallocated_Sector_Ct PO--CK 100 100 005 - 0 7 Seek_Error_Rate PO-R-- 100 100 067 - 0 8 Seek_Time_Performance P-S--- 121 121 020 - 34 9 Power_On_Hours -O--C- 095 095 000 - 36693 10 Spin_Retry_Count PO--C- 100 100 060 - 0 12 Power_Cycle_Count -O--CK 100 100 000 - 204 192 Power-Off_Retract_Count -O--CK 090 090 000 - 12239 193 Load_Cycle_Count -O--C- 090 090 000 - 12239 194 Temperature_Celsius -O---- 176 176 000 - 34 (Min/Max 19/69) 196 Reallocated_Event_Count -O--CK 100 100 000 - 0 197 Current_Pending_Sector -O---K 100 100 000 - 0 198 Offline_Uncorrectable ---R-- 100 100 000 - 0 199 UDMA_CRC_Error_Count -O-R-- 200 200 000 - 1 ||||||_ K auto-keep |||||__ C event count ||||___ R error rate |||____ S speed/performance ||_____ O updated online |______ P prefailure warning General Purpose Log Directory Version 1 SMART Log Directory Version 1 [multi-sector log support] Address Access R/W Size Description 0x00 GPL,SL R/O 1 Log Directory 0x01 SL R/O 1 Summary SMART error log 0x03 GPL R/O 1 Ext. Comprehensive SMART error log 0x04 GPL R/O 7 Device Statistics log 0x06 SL R/O 1 SMART self-test log 0x07 GPL R/O 1 Extended self-test log 0x08 GPL R/O 2 Power Conditions log 0x09 SL R/W 1 Selective self-test log 0x10 GPL R/O 1 NCQ Command Error log 0x11 GPL R/O 1 SATA Phy Event Counters log 0x20 GPL R/O 1 Streaming performance log [OBS-8] 0x21 GPL R/O 1 Write stream error log 0x22 GPL R/O 1 Read stream error log 0x80-0x9f GPL,SL R/W 16 Host vendor specific log 0xe0 GPL,SL R/W 1 SCT Command/Status 0xe1 GPL,SL R/W 1 SCT Data Transfer SMART Extended Comprehensive Error Log Version: 1 (1 sectors) Device Error Count: 1 CR = Command Register FEATR = Features Register COUNT = Count (was: Sector Count) Register LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8 LH = LBA High (was: Cylinder High) Register ] LBA LM = LBA Mid (was: Cylinder Low) Register ] Register LL = LBA Low (was: Sector Number) Register ] DV = Device (was: Device/Head) Register DC = Device Control Register ER = Error register ST = Status register Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 1 [0] occurred at disk power-on lifetime: 36160 hours (1506 days + 16 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER -- ST COUNT LBA_48 LH LM LL DV DC -- -- -- == -- == == == -- -- -- -- -- 84 -- 51 02 9d 00 00 37 64 e6 fb 07 00 Error: ICRC, ABRT 669 sectors at LBA = 0x3764e6fb = 929359611 Commands leading to the command that caused the error were: CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name -- == -- == -- == == == -- -- -- -- -- --------------- -------------------- 35 00 00 04 00 00 00 37 64 e5 98 40 00 1d+15:08:01.838 WRITE DMA EXT 35 00 00 04 00 00 00 37 64 e1 98 40 00 1d+15:08:01.835 WRITE DMA EXT 35 00 00 04 00 00 00 37 64 dd 98 40 00 1d+15:08:01.832 WRITE DMA EXT 35 00 00 04 00 00 00 37 64 d9 98 40 00 1d+15:08:01.829 WRITE DMA EXT 35 00 00 04 00 00 00 37 64 d5 98 40 00 1d+15:08:01.821 WRITE DMA EXT SMART Extended Self-test Log Version: 1 (1 sectors) Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 36410 - # 2 Short offline Completed without error 00% 36230 - # 3 Short offline Completed without error 00% 12666 - # 4 Short offline Completed without error 00% 11258 - # 5 Short offline Completed without error 00% 5623 - # 6 Extended offline Completed without error 00% 5355 - # 7 Extended offline Completed without error 00% 2939 - # 8 Short offline Completed without error 00% 6 - # 9 Short offline Completed without error 00% 6 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. SCT Status Version: 3 SCT Version (vendor specific): 256 (0x0100) SCT Support Level: 1 Device State: SMART Off-line Data Collection executing in background (4) Current Temperature: 34 Celsius Power Cycle Min/Max Temperature: 29/43 Celsius Lifetime Min/Max Temperature: 19/69 Celsius Under/Over Temperature Limit Count: 0/0 SCT Temperature History Version: 2 Temperature Sampling Period: 1 minute Temperature Logging Interval: 1 minute Min/Max recommended Temperature: 0/60 Celsius Min/Max Temperature Limit: -40/70 Celsius Temperature History Size (Index): 128 (0) Index Estimated Time Temperature Celsius 1 2019-07-07 20:41 31 ************ ... ..(106 skipped). .. ************ 108 2019-07-07 22:28 31 ************ 109 2019-07-07 22:29 32 ************* 110 2019-07-07 22:30 32 ************* 111 2019-07-07 22:31 33 ************** 112 2019-07-07 22:32 34 *************** 113 2019-07-07 22:33 34 *************** 114 2019-07-07 22:34 35 **************** 115 2019-07-07 22:35 35 **************** 116 2019-07-07 22:36 36 ***************** ... ..( 2 skipped). .. ***************** 119 2019-07-07 22:39 36 ***************** 120 2019-07-07 22:40 35 **************** ... ..( 3 skipped). .. **************** 124 2019-07-07 22:44 35 **************** 125 2019-07-07 22:45 34 *************** ... ..( 2 skipped). .. *************** 0 2019-07-07 22:48 34 *************** SCT Error Recovery Control: Read: Disabled Write: Disabled Device Statistics (GP Log 0x04) Page Offset Size Value Flags Description 0x01 ===== = = === == General Statistics (rev 2) == 0x01 0x008 4 204 --- Lifetime Power-On Resets 0x01 0x018 6 81923585770 --- Logical Sectors Written 0x01 0x020 6 363412195 --- Number of Write Commands 0x01 0x028 6 213140746257 --- Logical Sectors Read 0x01 0x030 6 923989246 --- Number of Read Commands 0x03 ===== = = === == Rotating Media Statistics (rev 1) == 0x03 0x008 4 34537 --- Spindle Motor Power-on Hours 0x03 0x010 4 34536 --- Head Flying Hours 0x03 0x018 4 12239 --- Head Load Events 0x03 0x020 4 0 --- Number of Reallocated Logical Sectors 0x03 0x028 4 0 --- Read Recovery Attempts 0x03 0x030 4 0 --- Number of Mechanical Start Failures 0x04 ===== = = === == General Errors Statistics (rev 1) == 0x04 0x008 4 0 --- Number of Reported Uncorrectable Errors 0x04 0x010 4 522 --- Resets Between Cmd Acceptance and Completion 0x05 ===== = = === == Temperature Statistics (rev 1) == 0x05 0x008 1 34 --- Current Temperature 0x05 0x010 1 33 N-- Average Short Term Temperature 0x05 0x018 1 43 N-- Average Long Term Temperature 0x05 0x020 1 69 --- Highest Temperature 0x05 0x028 1 19 --- Lowest Temperature 0x05 0x030 1 61 N-- Highest Average Short Term Temperature 0x05 0x038 1 25 N-- Lowest Average Short Term Temperature 0x05 0x040 1 57 N-- Highest Average Long Term Temperature 0x05 0x048 1 25 N-- Lowest Average Long Term Temperature 0x05 0x050 4 17590 --- Time in Over-Temperature 0x05 0x058 1 60 --- Specified Maximum Operating Temperature 0x05 0x060 4 0 --- Time in Under-Temperature 0x05 0x068 1 0 --- Specified Minimum Operating Temperature 0x06 ===== = = === == Transport Statistics (rev 1) == 0x06 0x008 4 619 --- Number of Hardware Resets 0x06 0x010 4 396 --- Number of ASR Events 0x06 0x018 4 1 --- Number of Interface CRC Errors |||_ C monitored condition met ||__ D supports DSN |___ N normalized value Pending Defects log (GP Log 0x0c) not supported SATA Phy Event Counters (GP Log 0x11) ID Size Value Description 0x0001 2 0 Command failed due to ICRC error 0x0002 2 0 R_ERR response for data FIS 0x0003 2 0 R_ERR response for device-to-host data FIS 0x0004 2 0 R_ERR response for host-to-device data FIS 0x0005 2 0 R_ERR response for non-data FIS 0x0006 2 0 R_ERR response for device-to-host non-data FIS 0x0007 2 0 R_ERR response for host-to-device non-data FIS 0x0009 2 5 Transition from drive PhyRdy to drive PhyNRdy 0x000a 2 3 Device-to-host register FISes sent due to a COMRESET 0x000b 2 0 CRC errors within host-to-device FIS 0x000d 2 0 Non-CRC errors within host-to-device FISWhen I expanded the array a few weeks ago, I tested the drives and zeroed them out before adding them one at a time to the NAS. No errors were reported then.
Smaky wrote:
This is the output for the two HGST drives:
smartctl -x /dev/sde ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 9 Power_On_Hours -O--C- 095 095 000 - 36843 199 UDMA_CRC_Error_Count -O-R-- 200 200 000 - 103
Error 103 [2] occurred at disk power-on lifetime: 36330 hours (1513 days + 18 hours)
Error: ICRC, ABRT 24 sectors at LBA = 0x005a6c98 = 5926040
Error 102 [1] occurred at disk power-on lifetime: 36330 hours (1513 days + 18 hours)
Error: ICRC, ABRT 191 sectors at LBA = 0x005a66f9 = 5924601
Error 101 [0] occurred at disk power-on lifetime: 36330 hours (1513 days + 18 hours)
Error: ICRC, ABRT 1 sectors at LBA = 0x00601c5f = 6298719I'd definitely retest this one. Note all the ICRC error being reported reported happened 513 hours ago (about 3 weeks). But the rise in CRC count says there might be more going on here.
Both errors are linked to the SATA interface, and it is also possible that it is the drive bay that is at fault.
The single error on the other drive doesn't seem as concerning, though probably it's worth a retest also.
You could start with the disk test that's built into OS-6 - click on the settings wheel for the volume (on the volume tab). That would test all the disks in the volume and the SATA interface of the NAS. But I also suggest testing this one with lifeguard.
Did you see any errors logged for your other disks?
Ok.. I am starting the disk test now.. will come back with more feedback.
StephenB wrote:Note all the ICRC error being reported reported happened 513 hours ago (about 3 weeks). But the rise in CRC count says there might be more going on here.
Both errors are linked to the SATA interface, and it is also possible that it is the drive bay that is at fault.
There are no other errors reported on the rest of the disks in the array. I have a couple of questions:
I have a couple of questions:
a) How did you come up with the timing on the errors.. the message states 36330 hours (1513 days + 18 hours).. how is this about three weeks ago, just curious because three weeks ago is about the time I added the disk to the array.
b) How can I test the interface?
regards.
I ran the disk test on the RAID volume. No errors were reported, nor an increase on the number of SMART errors. Should I look for anything particcular in the logs? I was wondering if there is an additional log generated by the disk test itself.
Smaky wrote:
a) How did you come up with the timing on the errors.. the message states 36330 hours (1513 days + 18 hours)..
Your SMART stats included the current power-on hours - which was 36843. 36843-36330=513.
Smaky wrote:
b) How can I test the interface?
There's no easy way I know of to test just the interface. You can install a single known working disk, and move that systematically into each slot. Then you can test the disk, and see if it misbehaves. For instance, you can run a SMART test (either from the web ui or ssh). You could also try copying files from ssh.
It has been almost 10 days since the last "freeze" of the NAS occurred (and the time I created this post that same day). There have been no increase in the numnber of errors on any drive of the NAS, even after I ran a couple of full disk tests and a scrub (which each took almost a whole day to execute).
I have kept all apps disabled (Plex, Emby, uTorrent & iStat) since then.. and the NAS has been completely stable since then. I will wait for anoter 10 days or so, before reactivating Emby on the NAS (which was the only app that showed some errors on the logs from the day it became unresponsive).
Will update in a few days more or earlier if something changes.
Another disk test ran yesterday. No errors were reported and the NAS keeps being responsive. Will wait until monday when I will re-enable Emby and will see if the system remains stable.
Just for the sake of completeness I shut down my NAS yesterday and started a memory test from the boot menu. The test ran for over 12 hrs. and no memory errors (4 GB Patriot non ECC memory installed), completed over 11 passes successfully. So I think I can discard any memory issues.
I think you have ruled out hardware problems as the cause. It'll be interesting to see if re-enabling Emby causes the problem to recur.
StephenB wrote:I think you have ruled out hardware problems as the cause. It'll be interesting to see if re-enabling Emby causes the problem to recur.
It's been a few days sice I reactivated the Emby service on my NAS. Besides having the software updated as there was a new verison availbale and a couple of reboots because I got a new set of RAM to raise memory to 4 GB (vs 3 GB I previously had), there have been no events to report. NAS has been stable.
Will continue to monitor at least a couple of weeks to check whether it goes unresponsive again.
Unfortunately something happened. After enabling Emby on the NAS and letting it run for a few days. Suddenly, a few days ago at some point the array on the NAS became alerted. Upon inspection, one of the WD drives failed an became unresponsive. I noticed this only because, again, the NAS became unresposive and the only way to regain access to it was to cold boot it via pressing the power button until the NAS shut down.
Upon restart I noticed a loud sound coming out from one of the drives and once it finally boot the array state was showing as degraded with the faulty drive showing up as offline. I shut down the NAS and replaced the drive with a spare one I had, and the rebuild process started. That lasted for a bit over a day.
Before shutting down the NAS I downloaded the logs, then before shutting it down to get the failed drive replaced I disabled Emby. Once the array was rebuilt I downloaded the logs again. It has been a couple of days since then and an array scrub task ran with no signal of issues.
The drive which failed had no SATA errors reported and now it became completely inaccesible even when connected to a PC. It makes a lot of loud sounds for a couple of minutes and Windows or WD data lifeguard fails to identify it. So it really seems a total drive failure.
I cannot say the failure is related to Emby's reactivation however this was the only change applied to the NAS config since all tests and several weeks of monitoring with no issues occurred. I am wondering if there might be somehting on the logs that can point to some sort of overload caused by Emby that might have cause stress on the drive to the point of failure. Not sure what to look for or where.
Smaky wrote:
I am wondering if there might be somehting on the logs that can point to some sort of overload caused by Emby that might have cause stress on the drive to the point of failure. Not sure what to look for or where.
Very unlikely theory. RAID-5 and XRAID are designed to spread the disk load evenly across all the drives - there is no overload on just one of them.
More likely - the drive just happened to fail, and Emby was not the cause.
StephenB wrote:More likely - the drive just happened to fail, and Emby was not the cause.
Well.. that's reassuring, I was on the same suspission as one of the aspects of RAID technology is to spread I/O between the discs. I guess it is just bad luck then.
Anyways, I have also noticed that during scrubs drives go up to 47-48 °C. Normally they are around 34-38°C.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
