We just recently purchased a ReadyNAS 4200 and recently had an odd failure that we haven't been able to explain. Overnight, when noone was in the office the NAS just seemed to "stop" for lack of a better description. When I came in the next morning, all ISCSI connections were failing to connect. I couldn't ping the NAS, connect to the admin URL, nothing. When I went into the lab and looked at the NAS, The power and network lights were all green, but all the drives appeared to have no activity (no blinking green lights). I called NETGEAR support (case #16011584) and was told to follow the instructions to "reinstall the OS" on the FAQ page. I was hesitant to do that without understanding a little bit more of the nature of the failure. We finally ended up powering down the NAS and when it came back up everything was working fine.
I've seen a couple of other threads describing similar kinds of behavior from the NAS, but it doesn't appear to be exactly our situation. There was no load at all on the NAS at the time it simply stopped responding. When I looked at the logs, I didn't see anything that seemed immediately alarming. Here's the section of the kernel log right when the failure happened. Notice the complete lack of activity from 03:59:39 until we restarted it at 08:56:54.
Jun 30 03:51:12 TPANAS01 kernel: ------------------------------------------------------------------ Jun 30 03:51:12 TPANAS01 kernel: InitiatorName: iqn.1991-05.com.microsoft:tpavfarm01.ourdomain.com Jun 30 03:51:12 TPANAS01 kernel: TargetAlias: LIO Target Jun 30 03:51:12 TPANAS01 kernel: TargetPortalGroupTag: 1 Jun 30 03:51:12 TPANAS01 kernel: DefaultTime2Wait: 2 Jun 30 03:51:12 TPANAS01 kernel: DefaultTime2Retain: 20 Jun 30 03:51:12 TPANAS01 kernel: SessionType: Discovery Jun 30 03:51:12 TPANAS01 kernel: ------------------------------------------------------------------ Jun 30 03:51:12 TPANAS01 kernel: TARGET_CORE[iSCSI]: Registered fabric_sess_ptr: ffff8801f2e001c0 Jun 30 03:51:12 TPANAS01 kernel: iSCSI Login successful on CID: 1 from 10.5.94.52 to 10.5.94.51:3260,1 Jun 30 03:51:12 TPANAS01 kernel: Incremented iSCSI Connection count to 1 from node: iqn.1991-05.com.microsoft:tpavfarm01.ourdomain.com Jun 30 03:51:12 TPANAS01 kernel: Established iSCSI session from node: iqn.1991-05.com.microsoft:tpavfarm01.ourdomain.com Jun 30 03:51:12 TPANAS01 kernel: Incremented number of active iSCSI sessions to 1 on iSCSI Target Portal Group: 1 Jun 30 03:51:12 TPANAS01 kernel: Decremented iSCSI connection count to 0 from node: iqn.1991-05.com.microsoft:tpavfarm01.ourdomain.com Jun 30 03:51:12 TPANAS01 kernel: TARGET_CORE[iSCSI]: Deregistered fabric_sess Jun 30 03:51:12 TPANAS01 kernel: Released iSCSI session from node: iqn.1991-05.com.microsoft:tpavfarm01.ourdomain.com Jun 30 03:51:12 TPANAS01 kernel: Decremented number of active iSCSI Sessions on iSCSI TPG: 1 to 0 Jun 30 03:59:36 TPANAS01 kernel: Decremented iSCSI connection count to 0 from node: iqn.1998-01.com.vmware:QaESXiLab-3db4d83d Jun 30 03:59:36 TPANAS01 kernel: TARGET_CORE[iSCSI]: Deregistered fabric_sess Jun 30 03:59:36 TPANAS01 kernel: Released iSCSI session from node: iqn.1998-01.com.vmware:QaESXiLab-3db4d83d Jun 30 03:59:36 TPANAS01 kernel: Decremented number of active iSCSI Sessions on iSCSI TPG: 1 to 0 Jun 30 03:59:38 TPANAS01 kernel: Decremented iSCSI connection count to 0 from node: iqn.1991-05.com.microsoft:tpavfarm01.ourdomain.com Jun 30 03:59:38 TPANAS01 kernel: TARGET_CORE[iSCSI]: Deregistered fabric_sess Jun 30 03:59:38 TPANAS01 kernel: Released iSCSI session from node: iqn.1991-05.com.microsoft:tpavfarm01.ourdomain.com Jun 30 03:59:38 TPANAS01 kernel: Decremented number of active iSCSI Sessions on iSCSI TPG: 1 to 1 Jun 30 03:59:38 TPANAS01 kernel: Stopped 1 thread set(s) (2 total threads). Jun 30 03:59:39 TPANAS01 kernel: Decremented iSCSI connection count to 0 from node: iqn.1991-05.com.microsoft:tpavfarm02.ourdomain.com Jun 30 03:59:39 TPANAS01 kernel: TARGET_CORE[iSCSI]: Deregistered fabric_sess Jun 30 03:59:39 TPANAS01 kernel: Released iSCSI session from node: iqn.1991-05.com.microsoft:tpavfarm02.ourdomain.com Jun 30 03:59:39 TPANAS01 kernel: Decremented number of active iSCSI Sessions on iSCSI TPG: 1 to 0 Jun 30 03:59:39 TPANAS01 kernel: Stopped 1 thread set(s) (2 total threads). Jun 30 08:56:54 TPANAS01 kernel: klogd 1.4.1#18, log source = /proc/kmsg started.
Just wondering if anyone has encountered this issue and if there are any additional troubleshooting steps we can take to try to definitively determine the root cause and a fix.
I am seeing this exact issue on one of my three 4200s, and the same level of logging (i.e. not helpful). Since the power light is still on but graceful shutdown doesn't work and none of the nics are responding (all static) I am assuming it has frozen during the middle of the night.
All three 4200 units were recently installed and upgraded with the 10GbE SFP+ option, however the one with the issue had the 10Gb card installed after firmware updates to 4.2.17 Long story short, I suspect the Broadcom 10Gb card on the other side or the SFP+ cable (direct connect) may be causing the problem due to repeated disconnects and reconnects in the ESXi logs during the night. Still, no excuse for a flatline on the ReadyNAS side.
So ... I have disconnected the 10Gb cable from the ReadyNAS and have remapped iSCSI and NAS shares over 1Gb and we'll see how it fairs tonight. I may use a ping monitor to pinpoint exactly when it's going down and compare with other logs. Also curious to see if one connects a monitor and usb keyboard if there will be some sort of panic message on the screen. Following that other steps I might try would be disabling iSCSI entirely, removing the 10Gb card, OS Reinstall, and Factory Reset. Anyone else have insight into this issue? Anyone else observing this behavior?
I had a similar problem and resolved it with help from Netgear by setting ACLs on the iSCSI volumes. Also check that the 10GBe cables are "supported", I would suggest to contact Netgear direct for this, for me I got it working with Cisco SFP+ cables although Netgear was unsure if they would work.
Once this was done, it's been running stable since 21st March 2011 😄
