× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: RN316 Corrupt Files

Mook2
Aspirant

RN316 Corrupt Files

Once again I'm having issues with BTRFS errors.

 

020-04-04 14:54:44: BTRFS critical (device md126): corrupt leaf, bad key order: block=20728614748160, root=1, slot=404

 

Earlier I had this error:

 

2020-03-22 05:39:37: raidard[2956]: segfault at 7fff125bc000 ip 0000000000401d20 sp 00007fff125b8f98 error 4 in raidard[400000+6000]

 

RN316 4x6, 2x10 WD Red  FW 6.10.2

 

In January Stephen B helped me with this issue:

 

https://community.netgear.com/t5/New-ReadyNAS-Users-General/RN316-Corrupt-Files/m-p/1855098

 

I've checked all the drives with WD Lifeguard. They all passed. Ran Balance, Scrub and Defrag before these errors.

I'm losing faith in my ReadyNAS boxes. I've never had this many problems. I've used these things since Infrant was the owner. I'm spending more time checking for errors and waiting for the hammer to fall again than using these to listen to music.

 

What should I do other than what I've done.

 

Thanks,

Dave-

 

 

Model: RN31600|ReadyNAS 300 Series 6- Bay (Diskless)
Message 1 of 30
StephenB
Guru

Re: RN316 Corrupt Files

Is this the backup NAS or the main one?

 

Maybe download the log zip, and send a PM to @JohnCM_S again.  Hopefully he will analyze it.

Message 2 of 30
Mook2
Aspirant

Re: RN316 Corrupt Files

Hi Stephen, this is my backup box.

 

I neglected to mention this was in my log:

 

"readynas volume data encountered an error and was made read-only. It is recommended to backup your data."

 

I'll send the logfile and see what happens,

 

This is so disheartening,

 

Dave-

 

 

Model: RN31600|ReadyNAS 300 Series 6- Bay (Diskless)
Message 3 of 30
StephenB
Guru

Re: RN316 Corrupt Files


@Mook2 wrote:

 

This is so disheartening,

 


Yeah.  And the recurrance is a puzzle.  Hopefully a log analysis will give some clue.

Message 4 of 30
Sandshark
Sensei

Re: RN316 Corrupt Files

What drive type(s) are you using?  Some "green" desktop drives do no behave well in a RAID array.  Mixing them with others can be especially troublesome.

Message 5 of 30
Mook2
Aspirant

Re: RN316 Corrupt Files

Hi sand,

 

These are all Western Digtal Red. I tested them all before I installed them.

 

I have 5 ReadyNAS boxes. 3-316s and 2 Ultra6s running OS6,

 

4 of 5 have segfault errors. I honestly have zero faith in Netgear. This started after I updated to 6.10.2. There's an earlier issue I posted that Stephen helped me "solve" but that box now has a segfault error.

I'm not happy. If I had the money I dump these for something else.

 

Dave-

 

 

Model: RN31600|ReadyNAS 300 Series 6- Bay (Diskless)
Message 6 of 30
Mook2
Aspirant

Re: RN316 Corrupt Files

Stephen, I can only hope this can be solved.

Model: RN31600|ReadyNAS 300 Series 6- Bay (Diskless)
Message 7 of 30
StephenB
Guru

Re: RN316 Corrupt Files


@Mook2 wrote:

 

These are all Western Digtal Red. I tested them all before I installed them.

 

I have 5 ReadyNAS boxes. 3-316s and 2 Ultra6s running OS6,

 


I'm running three OS-6 ReadyNAS (RN526, RN524, RN202) with WD Reds, which are running 6.10.3 at the moment.  I've never seen this on any of them.

 

Have you tried running smartctl -x using ssh?  This will report disk errors that might have rotated out of the logs.  I discovered a couple of WD60EFRX drives were generating occasional UNCs that way in my RN526.  There was nothing in the normal smart stats.

Message 8 of 30
Sandshark
Sensei

Re: RN316 Corrupt Files

WD reds shouldn't give you any issues, but it does still sound like a drive access issue, not a firmware issue.  If it's not the drives, it may be the hardware.  The firmware puts the volume in read-only mode to prevent further degradation when there is a problem with the volume.  That's actually helping you, not hurting.  What you need to address is the source of the faults that trigger it.

 

I have only seen this type of error once, and that was on an EDA500 when the eSATA connection was loose.

Message 9 of 30
Mook2
Aspirant

Re: RN316 Corrupt Files

Stephen, running smartctl using ssh is above my pay grade!

 

 

Model: RN31600|ReadyNAS 300 Series 6- Bay (Diskless)
Message 10 of 30
Mook2
Aspirant

Re: RN316 Corrupt Files

sand, thanks.

 

Nor sure how to address the issue of the faults. When I had my first issue in Jan I did a factory reset and it cleared the problem.

 

What bothers me is that 4 of 5 boxes have the segfault message. Not sure 4 of 6 boxes can have hardware issues at the same time. The odds against that must be high but thzt'x  different issue than the one I started the thread for.

 

I did get a PM on the logfile I sent and replied. We'll see what happens.

 

Dave-

Model: RN31600|ReadyNAS 300 Series 6- Bay (Diskless)
Message 11 of 30
Sandshark
Sensei

Re: RN316 Corrupt Files

Do you have drive spin-down enabled?  If so, have you tried disabling it?  I know Netgear removed the spin-down option for rack mount systems, but I have not seen a reason why.  Maybe other models can have the same issue?

 

Are the units running RAID5 (either via XRAID or FlexRAID), or somethng else?  That may be a contributor (rack mount units with >6 drives go to RAID6).

Message 12 of 30
StephenB
Guru

Re: RN316 Corrupt Files


@Mook2 wrote:

Stephen, running smartctl using ssh is above my pay grade!


It's actually not difficult, we can walk you through it if you want to give it a try.

Message 13 of 30
Mook2
Aspirant

Re: RN316 Corrupt Files

sand, spin down not enabled.

 

X-Raid (Raid 5)

 

Dave-

Model: RN31600|ReadyNAS 300 Series 6- Bay (Diskless)
Message 14 of 30
Mook2
Aspirant

Re: RN316 Corrupt Files

Hi Stephen, i have nothing to lose. Appreciate you stepping me through that process.

 

I did ssh once a loooong time ago for another issue. I believe mdgm (spelling?) stepped me through ot.

 

Thanks,

Dave-

Model: RN31600|ReadyNAS 300 Series 6- Bay (Diskless)
Message 15 of 30
StephenB
Guru

Re: RN316 Corrupt Files

You log in using root as the username and the admin password.

Then just enter

# smartctl -x /dev/sda

 

You then scroll to the section that starts with

SMART Extended Comprehensive Error Log Version:

and see what errors are underneath that. 

 

An example of a UNC error is

Error 13 [12] occurred at disk power-on lifetime: 40103 hours (1670 days + 23 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 01 cc 96 3d c0 40 00  Error: UNC at LBA = 0x1cc963dc0 = 7727365568

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 40 00 d8 00 01 cc 96 3d c0 40 08  7d+09:05:57.296  READ FPDMA QUEUED
  60 00 40 00 d0 00 01 cd 8b f8 00 40 08  7d+09:05:57.265  READ FPDMA QUEUED
  60 00 40 00 c8 00 01 cc fa a4 40 40 08  7d+09:05:57.141  READ FPDMA QUEUED
  60 00 10 00 c0 00 00 00 5c 8f c0 40 08  7d+09:05:47.254  READ FPDMA QUEUED
  e5 00 00 00 00 00 00 00 00 00 00 00 08  7d+09:05:12.714  CHECK POWER MODE

Repeat this procedure with sdb, sdc, sdd, sde, sdf.

 

Message 16 of 30
Mook2
Aspirant

Re: RN316 Corrupt Files

Stephen, paygrade issue.

 

When I try to login using "ssh root@192.xxx.x.xxx" it takes me to the admin page. Can't login as "root" but can as "admin:. Not getting what I expected to enter "# smartctl -x \dev\sda"

 

I looked at this page to make sure I had the proper settings, etc.

 

https://kb.netgear.com/30068/ReadyNAS-OS-6-SSH-access-support-and-configuration-guides

 

 

Dave-

 

 

 

Model: RN31600|ReadyNAS 300 Series 6- Bay (Diskless)
Message 17 of 30
Mook2
Aspirant

Re: RN316 Corrupt Files

Stephen, I finally got the CLI prompt. Then I got this:

 

Please specify device type with the -d option.

Use smartctl -h to get a usage summary

 

I saw this is another thread:

 

ssh root@nas-ip-address on Windows 10 (entered on the Windows 10 search bar)

 

That obviously doesn't work. I had to use the CLI prompt.

Model: RN31600|ReadyNAS 300 Series 6- Bay (Diskless)
Message 18 of 30
StephenB
Guru

Re: RN316 Corrupt Files


@Mook2 wrote:

 

When I try to login using "ssh root@192.xxx.x.xxx" it takes me to the admin page.

 

ssh root@nas-ip-address on Windows 10 (entered on the Windows 10 search bar)

That obviously doesn't work. I had to use the CLI prompt.

 


Actually ssh root@nas-ip-address should work from the windows 10 search bar (it does on all my systems).  It even works when I enter it from the file explorer address bar.

 

Did you enter it into the browser address bar?  

 


@Mook2 wrote:

Not getting what I expected to enter "# smartctl -x \dev\sda"

 


You don't actually enter the #   The # is actually the end of the CLI prompt.

 

But you seem to have figured that out.  More importantly,  I accidentally typed the wrong slash direction: Try  "smartctl -x /dev/sda"

 

 

Message 19 of 30
Mook2
Aspirant

Re: RN316 Corrupt Files

Well, that worked! Thanks!

 

I didn't see any errors just scanning the repsonses (not that I know what I'm looking for!) but I thought these were key and I got them for each of the 6 disks:

 

sda-f

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

 

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged

 

 

Model: RN31600|ReadyNAS 300 Series 6- Bay (Diskless)
Message 20 of 30
Mook2
Aspirant

Re: RN316 Corrupt Files

While I'm at it....

 

Here's another beauty seen by clicking the Diagnostic option on my "production" box:

 

2020-03-12 17:04:11: md/raid:md127: raid level 5 active with 5 out of 6 devices, algorithm 2

Model: RN31600|ReadyNAS 300 Series 6- Bay (Diskless)
Message 21 of 30
StephenB
Guru

Re: RN316 Corrupt Files


@Mook2 wrote:

While I'm at it....

 

Here's another beauty seen by clicking the Diagnostic option on my "production" box:

 

2020-03-12 17:04:11: md/raid:md127: raid level 5 active with 5 out of 6 devices, algorithm 2


Can you post mdstat.log?

Message 22 of 30
Mook2
Aspirant

Re: RN316 Corrupt Files

Sure. 

 

Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md126 : active raid1 sde4[0] sdf4[1]
3905782144 blocks super 1.2 [2/2] [UU]

md127 : active raid5 sda3[0] sdf3[6] sde3[7] sdd3[3] sdc3[2] sdb3[1]
29278364160 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]

md1 : active raid10 sda2[0] sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1]
1566720 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU]

md0 : active raid1 sda1[0] sdf1[6] sde1[7] sdd1[3] sdc1[2] sdb1[1]
4190208 blocks super 1.2 [6/6] [UUUUUU]

unused devices: <none>
/dev/md/0:
Version : 1.2
Creation Time : Sun Feb 9 22:04:02 2020
Raid Level : raid1
Array Size : 4190208 (4.00 GiB 4.29 GB)
Used Dev Size : 4190208 (4.00 GiB 4.29 GB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent

Update Time : Sat Apr 11 21:33:02 2020
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Consistency Policy : unknown

Name : 2fe6cec4:0 (local to host 2fe6cec4)
UUID : 3321bbb5:26a8bd22:e42b8c34:ce23f3a9
Events : 584

Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1
2 8 33 2 active sync /dev/sdc1
3 8 49 3 active sync /dev/sdd1
7 8 65 4 active sync /dev/sde1
6 8 81 5 active sync /dev/sdf1
/dev/md/1:
Version : 1.2
Creation Time : Sun Mar 15 08:10:34 2020
Raid Level : raid10
Array Size : 1566720 (1530.00 MiB 1604.32 MB)
Used Dev Size : 522240 (510.00 MiB 534.77 MB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent

Update Time : Sat Apr 11 10:29:07 2020
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : near=2
Chunk Size : 512K

Consistency Policy : unknown

Name : 2fe6cec4:1 (local to host 2fe6cec4)
UUID : cb52d30e:dbbd37b6:1be9b3dd:9828e897
Events : 19

Number Major Minor RaidDevice State
0 8 2 0 active sync set-A /dev/sda2
1 8 18 1 active sync set-B /dev/sdb2
2 8 34 2 active sync set-A /dev/sdc2
3 8 50 3 active sync set-B /dev/sdd2
4 8 66 4 active sync set-A /dev/sde2
5 8 82 5 active sync set-B /dev/sdf2
/dev/md/data-0:
Version : 1.2
Creation Time : Sun Feb 9 22:04:38 2020
Raid Level : raid5
Array Size : 29278364160 (27922.02 GiB 29981.04 GB)
Used Dev Size : 5855672832 (5584.40 GiB 5996.21 GB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent

Update Time : Sat Apr 11 00:01:17 2020
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Consistency Policy : unknown

Name : 2fe6cec4:data-0 (local to host 2fe6cec4)
UUID : b3d2e1b9:bd0b0ddb:838609eb:7d617bc5
Events : 1105

Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 8 19 1 active sync /dev/sdb3
2 8 35 2 active sync /dev/sdc3
3 8 51 3 active sync /dev/sdd3
7 8 67 4 active sync /dev/sde3
6 8 83 5 active sync /dev/sdf3
/dev/md/data-1:
Version : 1.2
Creation Time : Sat Mar 14 14:55:00 2020
Raid Level : raid1
Array Size : 3905782144 (3724.84 GiB 3999.52 GB)
Used Dev Size : 3905782144 (3724.84 GiB 3999.52 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Sat Apr 11 00:01:17 2020
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Consistency Policy : unknown

Name : 2fe6cec4:data-1 (local to host 2fe6cec4)
UUID : abb14453:3769da5b:6056e7c3:109c2488
Events : 135

Number Major Minor RaidDevice State
0 8 68 0 active sync /dev/sde4
1 8 84 1 active sync /dev/sdf4

Model: RN31600|ReadyNAS 300 Series 6- Bay (Diskless)
Message 23 of 30
StephenB
Guru

Re: RN316 Corrupt Files

Thx.

 

Back on 12 March it looks like the array was resyncing (or a disk had dropped out of the array).

 

But that's not the the case now. 

Message 24 of 30
Mook2
Aspirant

Re: RN316 Corrupt Files

Thanks, Stephen, Much appreciated.

 

I have a question about support.

 

I'm not sure what the process is. These boxes are beyond free support. What are my options? Do I call Netgear open a case? What is Insight? The write-up is beyond perfect. I also saw a third-party support group for Netgear where I could pay by incident or subscribe for a period of time no matter how many issues I have but I can't seem to find that for some reason.

 

i have so many issues that I think that's the way to go but I want to choose the proper option. I don't mind paying at this point rather than drive you, others and myself crazy!

 

Thoughts?

 

Dave-

Model: RN31600|ReadyNAS 300 Series 6- Bay (Diskless)
Message 25 of 30
Top Contributors
Discussion stats
  • 29 replies
  • 3064 views
  • 0 kudos
  • 4 in conversation
Announcements