Forum Discussion

Aspirant

Nov 18, 2012

Repeating GPF

Hi

I was wondering if anyone could advise on the cause of the GPF's I'm seeing in the syslog of my ReadyNas Pro. These happen once a week at least an normally result in my having to powercycle the NAS.
Extract from syslog below

Nov 15 16:01:10 Nassy2 kernel: alloc_fd: slot 5 not NULL!
Nov 15 16:01:12 Nassy2 kernel: general protection fault: 0000 [#3] SMP
Nov 15 16:01:12 Nassy2 kernel: last sysfs file: /sys/devices/virtual/block/md3/md/sync_action
Nov 15 16:01:12 Nassy2 kernel: CPU 1
Nov 15 16:01:12 Nassy2 kernel: Modules linked in: pvgpio nv6vpd(P)
Nov 15 16:01:12 Nassy2 kernel:
Nov 15 16:01:12 Nassy2 kernel: Pid: 17596, comm: grep Tainted: P      D     2.6.37.6.RNx86_64.2.4 #1 NETGEAR ReadyNAS/
Nov 15 16:01:12 Nassy2 kernel: RIP: 0010:[<ffffffff880ac8c7>]  [<ffffffff880ac8c7>] filp_close+0x17/0x90
Nov 15 16:01:12 Nassy2 kernel: RSP: 0000:ffff88007094df28  EFLAGS: 00010286
Nov 15 16:01:12 Nassy2 kernel: RAX: fffffffffffffffd RBX: 0400c8328bf591e0 RCX: 0000000000000001
Nov 15 16:01:12 Nassy2 kernel: RDX: 0000000000000000 RSI: ffff880077c13440 RDI: 0400c8328bf591e0
Nov 15 16:01:12 Nassy2 kernel: RBP: ffff88007094df48 R08: 0000000000000001 R09: 00000000fffce810
Nov 15 16:01:12 Nassy2 kernel: R10: ffff88007094c000 R11: 0000000000000000 R12: ffff880077c134c0
Nov 15 16:01:12 Nassy2 kernel: R13: 0000000000000001 R14: 0400c8328bf591e0 R15: 0000000000000000
Nov 15 16:01:12 Nassy2 kernel: FS:  0000000000000000(0000) GS:ffff88007ee80000(0063) knlGS:00000000f764e6b0
Nov 15 16:01:12 Nassy2 kernel: CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
Nov 15 16:01:12 Nassy2 kernel: CR2: 00000000f76b55d0 CR3: 0000000070b49000 CR4: 00000000000006e0
Nov 15 16:01:12 Nassy2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 15 16:01:12 Nassy2 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 15 16:01:12 Nassy2 kernel: Process grep (pid: 17596, threadinfo ffff88007094c000, task ffff88007083db00)
Nov 15 16:01:12 Nassy2 kernel: Stack:
Nov 15 16:01:12 Nassy2 kernel:  0000000000000000 ffff880077c13440 ffff880077c134c0 0000000000000001
Nov 15 16:01:12 Nassy2 kernel:  ffff88007094df78 ffffffff880adae4 0000000000000001 0000000000000000
Nov 15 16:01:12 Nassy2 kernel:  0000000000000000 0000000000000000 00000000fffce810 ffffffff88024c53
Nov 15 16:01:12 Nassy2 kernel: Call Trace:
Nov 15 16:01:12 Nassy2 kernel:  [<ffffffff880adae4>] sys_close+0xa4/0x100
Nov 15 16:01:12 Nassy2 kernel:  [<ffffffff88024c53>] ia32_sysret+0x0/0x5
Nov 15 16:01:12 Nassy2 kernel: Code: 83 80 00 00 00 48 8b 1c 24 4c 8b 64 24 08 c9 c3 66 66 66 90 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <48> 8b 47 30 49 89 f4 48 85 c0 74 52 48 8b 47 20 48 85 c0 74 44
Nov 15 16:01:12 Nassy2 kernel: RIP  [<ffffffff880ac8c7>] filp_close+0x17/0x90
Nov 15 16:01:12 Nassy2 kernel:  RSP <ffff88007094df28>
Nov 15 16:01:12 Nassy2 kernel: ---[ end trace 2c9ff00a9fb03955 ]---
Nov 15 16:01:14 Nassy2 kernel: general protection fault: 0000 [#4] SMP
Nov 15 16:01:14 Nassy2 kernel: last sysfs file: /sys/devices/virtual/block/md3/md/sync_action
Nov 15 16:01:14 Nassy2 kernel: CPU 3
Nov 15 16:01:14 Nassy2 kernel: Modules linked in: pvgpio nv6vpd(P)
Nov 15 16:01:14 Nassy2 kernel:
Nov 15 16:01:14 Nassy2 kernel: Pid: 17596, comm: grep Tainted: P      D     2.6.37.6.RNx86_64.2.4 #1 NETGEAR ReadyNAS/
Nov 15 16:01:14 Nassy2 kernel: RIP: 0010:[<ffffffff880ac8c7>]  [<ffffffff880ac8c7>] filp_close+0x17/0x90
Nov 15 16:01:14 Nassy2 kernel: RSP: 0000:ffff88007094dcd8  EFLAGS: 00010286
Nov 15 16:01:14 Nassy2 kernel: RAX: ffff880077dff010 RBX: 01000608c5620620 RCX: 0000000000000000
Nov 15 16:01:14 Nassy2 kernel: RDX: 0000000000000000 RSI: ffff880077c13440 RDI: 01000608c5620620
Nov 15 16:01:14 Nassy2 kernel: RBP: ffff88007094dcf8 R08: ffff88007094c000 R09: 0000000000000001
Nov 15 16:01:14 Nassy2 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880077c13440
Nov 15 16:01:14 Nassy2 kernel: R13: 0000000000000000 R14: ffff88000c6d8ac0 R15: 0000000000000010
Nov 15 16:01:14 Nassy2 kernel: FS:  0000000000000000(0000) GS:ffff88007ef80000(0000) knlGS:0000000000000000
Nov 15 16:01:14 Nassy2 kernel: CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
Nov 15 16:01:14 Nassy2 kernel: CR2: 00000000f776df05 CR3: 0000000071345000 CR4: 00000000000006e0
Nov 15 16:01:14 Nassy2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 15 16:01:14 Nassy2 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 15 16:01:14 Nassy2 kernel: Process grep (pid: 17596, threadinfo ffff88007094c000, task ffff88007083db00)
Nov 15 16:01:14 Nassy2 kernel: Stack:
Nov 15 16:01:14 Nassy2 kernel:  ffff8800784a2300 000000000000007f ffff880077c13440 0000000000000000
Nov 15 16:01:14 Nassy2 kernel:  ffff88007094dd38 ffffffff88036843 0000000000000000 ffff88007083db00
Nov 15 16:01:14 Nassy2 kernel:  ffff880077c13440 ffff88007083db00 000000000000009c ffff88007e8fb840
Nov 15 16:01:14 Nassy2 kernel: Call Trace:
Nov 15 16:01:14 Nassy2 kernel:  [<ffffffff88036843>] put_files_struct+0xc3/0xd0
Nov 15 16:01:14 Nassy2 kernel:  [<ffffffff88036895>] exit_files+0x45/0x50
Nov 15 16:01:14 Nassy2 kernel:  [<ffffffff88037a20>] do_exit+0x190/0x770
Nov 15 16:01:14 Nassy2 kernel:  [<ffffffff88002bce>] ? apic_timer_interrupt+0xe/0x20
Nov 15 16:01:14 Nassy2 kernel:  [<ffffffff88035fd0>] ? kmsg_dump+0x110/0x160
Nov 15 16:01:14 Nassy2 kernel:  [<ffffffff88006506>] oops_end+0xa6/0xb0
Nov 15 16:01:14 Nassy2 kernel:  [<ffffffff88006606>] die+0x56/0x90
Nov 15 16:01:14 Nassy2 kernel:  [<ffffffff880040f2>] do_general_protection+0x152/0x160
Nov 15 16:01:14 Nassy2 kernel:  [<ffffffff885b48ef>] general_protection+0x1f/0x30
Nov 15 16:01:14 Nassy2 kernel:  [<ffffffff880ac8c7>] ? filp_close+0x17/0x90
Nov 15 16:01:14 Nassy2 kernel:  [<ffffffff880adae4>] sys_close+0xa4/0x100
Nov 15 16:01:14 Nassy2 kernel:  [<ffffffff88024c53>] ia32_sysret+0x0/0x5
Nov 15 16:01:14 Nassy2 kernel: Code: 83 80 00 00 00 48 8b 1c 24 4c 8b 64 24 08 c9 c3 66 66 66 90 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <48> 8b 47 30 49 89 f4 48 85 c0 74 52 48 8b 47 20 48 85 c0 74 44
Nov 15 16:01:14 Nassy2 kernel: RIP  [<ffffffff880ac8c7>] filp_close+0x17/0x90
Nov 15 16:01:14 Nassy2 kernel:  RSP <ffff88007094dcd8>
Nov 15 16:01:14 Nassy2 kernel: ---[ end trace 2c9ff00a9fb03956 ]---
Nov 15 16:01:14 Nassy2 kernel: Fixing recursive fault but reboot is needed!

I'm hoping it's just a memory issue which can be easily resolved. Anyone recognise the problem?

Thanks in advance :)

Hw & Hw Compatibility

Other

16 Replies

Replies have been turned off for this discussion

Andlier
Tutor
Mar 05, 2013
Hi again,

After almost 4 weeks of problem free operation after I did a full factory reset and I did not restore any previous settings, the general protection fault issue happened again. I almost believed the problem had disappeared after that last factory reset, but apparently not. I am now running 4.2.20, since that was the last firmware revision I successfully used before these problems started about half a year ago.

Any input on how to debug this issue further would be greatly appreciated!? Are there any tools/software I could run on the readynas that would detect and log if any process violates any memory boundaries for example? Any ideas on how to speed up the time between faults happen? It is tedious to have to wait 4 weeks between each time the problems occur!

Since I see the same issues with approximately the same interval on both my readynas 6 ultra and 2 pro device, with several different firmware revisions (4.2.20, 4.2.21 and 4.2.22) and factory reset did not help in neither of them I must say I suspect firmware issues, maybe in combination with my network environment/configuration... Is there any logs or procedures that could help sort out issues like that? One thing I can think of that is a bit out of the ordinary with my setup is that the ultra 6 is connected to two different LAN subnets on each of the network ports, could that be an issue? Any other similar things to look out for?
chirpa
Luminary
Mar 05, 2013
Support/Engineering needs to step up and help diagnose it.
Andlier
Tutor
Mar 05, 2013
chirpa wrote:
Support/Engineering needs to step up and help diagnose it.

I've already made a new support ticket, but that will be the third support ticket I'm submitting regarding this exact problem, the previous support sessions didn't come up with anything useful except trying factory reset... Lets hope they can find something this time.
JanneK
Aspirant
Mar 17, 2013
I have the same problem.
My Readynas Ultra 2 Plus is connected to a Netgear WNDR3700 and it happens about once per one/two week(s).
Andlier
Tutor
Apr 01, 2013
Just a quick update from my side on this, problem is not resolved yet, but there is some progress with Netgear support. After first describing my network topology to netgear and allowing netgear tech support ssh access to my ultra 6 device, they quickly decided to RMA my box. Even though the RAM test and disk test is fine and I've tried factory defaulting without success. Well, RMA took some time as usual, they still haven't picked up my old device (I opted for the advance exchange rma procedure, not free). Now the new box they sent me has been running for 3-4 weeks, and them boom, same thing happened again, well almost same thing, there is no "general protection fault" message in the log, but it did crash badly with lots of errors and call trace in the log. So I'm in contact with support again and they again want ssh access to my box. In the mean time I'm following this thread: https://www.readynas.com/forum/viewtopic.php?t=69606&p=386216 which seems to be related. I do have 100mbps switch connected to my box ( in addition to a 1gbps on the other port). But it seems a bit strange that having 100mbps switch should cause such problems... Well, still waiting a resolve for these issues, been almost 8 months since this issue first appeared for me. At least I feel Netgear is taking it seriously now. Also a bit comforting to see others on here with exactly the same symptoms, then it is even more likely that Netgear looks at this seriously.
chopsywa
Aspirant
Apr 18, 2013
We have encountered this error on a couple of client machines. It has been driving us crazy. Both machines (Pro 4 and Pro 6) have been replaced.

This is not a hardware fault as such.

In both instances the units have one Ethernet directly connected to a server on jumbo frames for VMWare. The other is plugged into the switch for management.

I believe it is something to do with a packet on the network causing the low level network drivers to crash the box.

I have found that when the problems happen I notice some other things.

1. Either just before, or soon after the GPF, there are many errors in the log as such.
Dec 29 23:28:55 PERTH-STORE01 kernel: UDP: short packet: From 192.168.60.9:17500 130/99 to 192.168.60.255:17500
The packets are always UDP to broadcast addresses. At this site there is also an NVX and there are no errors like this in the logs.
2. The management interface is non responsive, even to pings, yet the NFS interface is still alive.
3. Sometime from a couple of hours to a day or two later, it hard locks up and needs a powercyle.

We have one of these units now unplugged from the management LAN and only accessable via the connected server. So far so good. I will report back in a week or so. It usually fails within 3 or 4 days.