ReadyNAS Pro unresponsive on 4.2.21 - first time in years.

The OS partition is 4 GB on the X86 platforms, and 2 GB on the Sparc. DF is fine. On the x86, it should be /dev/md0 (at least that the case for the Pro). Not sure when it becomes dangerously full. Though if you haven't installed any addons, the simple way to increase space is to delete the logs via Frontview. Since you have the zip already, you won't lose any information.

Try both "df -h" and "df -i". The first can be used to see if the space on the OS partition (/dev/md0) is filled up. The latter can be used to see if the inodes are used up (i.e. lots of small files). Too full would be something approaching 100% full (say over about 80% perhaps). Normally the OS partition would be well under half full.

The output of "df -h" is shown in disk_usage.log in the logs zip file but using SSH would give you up-to-date output.

I presume this is on your beta unit?

ProServer:~# df -h /dev/md0
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              4.0G  699M  3.2G  19% /

ProServer:~# df -i /dev/md0
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/md0              131072   20327  110745   16% /

Why would you presume it's a beta unit? It's my only unit, and it's production.

The thing that's especially troubling is that I'm hosting a bunch of VMs on this ReadyNas Pro via iSCSI. So, not only did I lose the file server, but I also lost all my VMs when the ReadyNAS stopped responding. The ESXi server is connected to the ReadyNAS NIC2 with a direct Cat6 cable, so when I lost the switch, the ESXi-to-ReadyNAS link was not affected.

Any other ideas? Thanks!

By beta I meant a unit received as part of the beta test back in 2008 not how it's currently used. If it this unit then perhaps PM a Jedi to draw their attention to the thread. One of them should be able to review your logs.

If it's a unit you purchased I'd suggest opening a tech support (http://www.ReadyNAS.com/support) case. Edit the thread title (i.e. subject of first post in the thread) to include your case number.

Space used on OS partition looks fine.

OK. Now we are on the same page. This unit was in fact a beta-tester’s unit that I received back in 2008 from Netgear. Since a few of the Jedi have left the company, I’m not sure to whom I should PM. Any suggestions?

yoh-dah and chirpa are the Jedi Council Alumni (former members of the council)

Try contacting a Jedi Council member (this will show in their rank) listed when you follow "The Team" link on this page. Choose one with a recent date for last visiting the forum.

This is to update the status: A member of the Jedi Council got in touch with me via PM. I have emailed the logs and waiting on the ruling.

Further update: I had to open a case with Netgear support. Eventually, the shipped me a replacement ReadyNAS Pro Business. About 17 days after I installed the replacement ReadyNAS Pro 6 and moved my disks from the “defective” unit to the “replacement” unit, the same problem recurred. Last Saturday, I noticed that my Squeezebox Boom and Squeezebox Radio lost connection to the Squeezebox server (or whatever it’s now called since it has been renamed so many times). I tried to ping my ReadyNAS and got nothing in response. When I pressed the front-panel button on the ReadyNAS Pro, the LED screen responded to the button press. So, I figured I would leave it alone for one more day to see if the problem goes away or gets worse. The following day (Sunday), the ReadyNAS was not responding to pings, either, but it also stopped responding to the front-panel button presses.

So, I logged into the ESXi server (which is using the ReadNAS as its iSCSI targets) and it still still saw the iSCSI volumes located on the ReadyNAS. The ESXi server is connected to the ReadyNAS via a direct CAT6 cable (iSCSI NIC2 <—> ReadyNAS NIC2). Moreover, I was still able to ping VMs running on the ISCSI server, which are physically located on LUNs provisioned on the ReadyNAS. I have one Windows 7 VM, which was responding to pings, but I could not RDP into it. There are several Cisco Unified Communications (RedHat based VMs) that are LUNs on the ReadyNAS, which were also responding to pings, but I could not SSH into them. Finally, I had to power down the ESXi server and then I pushed and held the front-panel button on the ReadyNAS until it turned off.

When I turned the ReadyNAS back on, it started a syncing process, which lasted several hours. When I powered up the ESXi server, it took it a long time to boot up. It had happened before (after I upgraded to RAIDiator 4.2.21). The VM whose LUN was provisioned last on the ReadyNAS causes the ESXi server to hang for a long time (close to 10 minutes) before it boots all the way up and is accessible via the vSphere Client. Basically, the ESXi server cannot establish a connection to this last LUN, which happens to be my Windows 7 VM. Connections to other LUNs on the ReadyNAS are fine. In fact, after I received the replacement ReadyNAS, I deleted the Windows 7 LUN, and then created a new LUN and installed Windows 7 anew. Still, this is the VM running off the LUN that was provisioned last on the ReadyNAS, and it manifests the same behavior as it did before I recreated this LUN and before I replaced the ReadyNAS. The only way to get the ESXi server to connect to this LUN is to go to the ReadyNAS FrontView, disable this LUN, then re-enable this LUN, and then to go to the vSphere Client, and rescan the LUNs. Then, the ESXi server finds the datastore on this LUN and the Windows 7 VM shows up in the list of VMs. At this point, I can manually start this VM and all is good until the next reboot of the ESXi server, which occurs when my ReadyNAS Pro freezes - generally within 14-20 days after another reboot.

So, my strong suspicion is that my ReadyNAS freezes are caused by the iSCSI protocol. One of the symptoms of a problem is that my ESXi server cannot automatically start the VM that uses the LUN created last on the ReadyNAS because it cannot connect to the datastore on this LUN upon initial boot. This seems to point to some sort of incompatibility between ESXi 5 and RAIDiator 4.2.21. Since the hardware has now been replaced, everything points to the software issue. Moreover, the communication between the ESXi server and the ReadyNAS occurs only via a direct CAT6 cable connecting the two servers’ NIC2s. I bound iSCSI to NIC2 on the ESXi server, and therefore, the source address with which ESXi communicates with the ReadyNAS is its NIC2’s IP address. Therefore, the ReadyNAS only knows of the ESXi server’s IP on its NIC2 and because of its internal routing table, it only sends packets to the ESXi server out of its own NIC2. When the ReadyNAS becomes unresponsive, not only do I lose communication to the ReadyNAS NIC1 - the IP that the rest of my network sees (which is accessible via an ethernet switch), but also the communication based on the ReadyNAS NIC2 (direct cable connection to the iSCSI server) is also affected. This rules out any issues on my LAN.

I’m not sure how many users of the ReadyNAS Pro Business use it as an iSCSI target. I’m pretty sure that I’m running the ReadyNAS in a pretty complex environment - more complex than most users. I believe there’s actually a bug in the RAIDiator software - probably a memory leak - related to iSCSI, which makes the ReadyNAS completely unresponsive within 12-16 days.

So, almost 6 months later, I'm still experiencing the same issue. I have five VMs that are hosted on the ReadyNAS via iSCSI, and this morning, the ReadyNAS Pro 6 became unresponsive again. ReadyNAS' interface plugged into my LAN switch does not respond to pings. The other ReadyNAS interface plugged directly into my ESXi host is still working because the virtual machines are alive even though they are actually on the ReadyNAS datastore.

I have a Windows VM also hosted on the ReadyNAS datastore, which has a virtual NIC on the e-switch that is on the ESXi physical interface connected directly to the ReadyNAS. I tried to HTTP to FrontView from that VM, and I can see FrontView's frame but no content inside of the frame - so the normal FrontView menu items are not there. I also tried to SSH to the ReadyNAS via that Windows VM"s interface bridged to the ESXi host's interface plugged directly into the ReadyNAS, and I can get the "login as" and the "password" prompts but I never get to the ReadyNAS' shell. Strangely enough, the ReadyNAS is still operating as the ESXi datastore. To me it sounds as some sort of memory leak. Besides the ReadyNAS being used as a file server and an ESXi datastore, it is also running the Logitech Media Server v 7.7.2. One of these services is what causes the ReadyNAS to become unresponsive. My suspicion has been from the get go that this is an issue with iSCSI. In the past, the behavior was very similar, but I was able to HTTP to the ReadyNAS via the direct link between the ESXi host and the ReadyNAS secondary interface (from the Windows VM) and then toggle the ReadyNAS primary interface (by changing the duplex setting from auto to 1000/full and then back to auto). This would fix the unresponsiveness and the ReadyNAS would continue working fine. This time around, I can't get to the ReadyNAS in this manner either via HTTP or SSH, so I will have to power down all of my VMs and then physically reset the ReadyNAS.

The way I usually notice that the ReadyNAS becomes unresponsive is the fact that my Squeezebox devices stop streaming music. This is exactly what happened today.

I had two tickets open with Netgear support on this issue starting back in June of 2012. The first ticket was closed without resolution. The second ticket was opened for about 3.5 months. Netgear asked me to open SSH access for them to troubleshoot the problem, and then their engineer disappeared. I left the SSH access open thinking that they are working on this issue behind the scenes. As a result of SSH access to the ReadyNAS having been open from the Internet for several months, my ReadyNAS was hacked into and infected with a Trojan, which started attacking other servers on the Internet. It became a soldier in the army of bots attacking a company in Europe, and after the company complained to their ISP, and that ISP forwarded the complaint to my ISP, Time Warner shut off my Internet access. This happened twice within a month, and after the second time, I started investigating in earnest which host was infected on my network, and was very surprised to find out that it was the ReadyNAS. I was able to unmount all of my personal data and then wiped out all other files, and then did the factory reset with software reinstall. I was able to restore the ReadyNAS back to the normal functionality back at the end of 2012, and I was hoping the saga with my ReadyNAS was over. However, as I learned today, the saga is still ongoing.

So, I decided that I have had enough. I am abandoning the ReadyNAS ship (the sinking ship), and I'm switching to a competitor. This is no longer a viable platform for my computing needs. After the Jedi left, this line of products is heading for a crash of galactic proportions.

sirozha wrote:
OK. Now we are on the same page. This unit was in fact a beta-tester’s unit that I received back in 2008 from Netgear. Since a few of the Jedi have left the company, I’m not sure to whom I should PM. Any suggestions?

Hi. Was using the upgraded version resolved this for you? I just want an assurance first.