I have been running a ReadyNAS Ultra 6 Plus (6x2 TByte in X-Raid 2 mode) for some years with few problems and much success. The device was always upgraded to the latest RAIDiator and has been running 4.2.25 since whenever that was released. A few non-standard services have been running for at least a few months with problems: SSH, Plex Media Server, and the CrashPlan backup program. There have been no configuration changes for at least a few months.
Today, for the first time, I have encountered a serious problem. First, when logging myself into ReadyNAS FrontView, I got a notice of new log items, but when I went to the logs, the page indicated that "No logs exist." According to the advice on this forum, I attempted to reboot the server from the admin page, but that did not have any effect. So I tried to remember where the server was physically located, found it, and did a reboot by pressing the power button twice (i.e., a soft reboot).
When the server was rebooted things were worse, rather than better. Now I could no longer open any of the SMB shares from any of my home workstations. Logging myself into FrontView showed still no logs, but neither was there any error. All the shares were still listed under Shares/Share Listing. Status/Health showed all green lights. The only item which was even close to being out of the safe range was Temp Sys at 57 degrees Celsius, but that might always have been around that level.
When I ssh-ed into the server, things appeared fine. The underlying data files in the filesystem appear still to be there. I did a 'ps -edalf' (I'll be damned if I can remember what all those switches mean, but they just came from muscle memory from my days as UN*X sysadmin). Nothing suspicious there at first glance, but I've appended the results below.
Does anybody have any help or advice? I'd hate to nuke the settings, reconfigure everything, and reinstall and reconfigure my addons (if I even remember how exactly to do so).
Since the last report, my ReadyNAS has continued to deteriorate without my intervention.
Now, on the health status page, there is listed as malfunctioning:
Fan CAS 0 RPM Out of Spec Temp AUX -17.5 C / 0 F [Normal 0-0 C / 32-32 F] Out of Spec
I do not recall seeing either of those items listed at all previously.
Moreover, Frontview is failing left and right.
For example, my timezone has changed to being listed as "GMT -11:00 Samoa" and the NTP synchronization has been turned off. Correcting these setting and hitting apply just results in them being reset to the incorrect values.
Now, under Services/Standard File Protocols all the protocols are suddenly listed as disabled. Click the checkbox to enable followed by apply, just resets the buttons to off.
Finally, under Volumes/Volume Settings/RAID Settings, the listing has changed to:
Configuration: RAID Level X-RAID2, 6 disks Status: Dead. This volume is no longer available.
Update on resolution: After some trials and tribulations, tech support were able to log themselves in remotely and fix the problem.
Tech support did not say what the problem was, but said they moved a bunch of /tmp files to a separate partition. Reviewing these tmp files turns up a huge number of jna[0-9]*.tmp files inside tmp. Each was 136 kBytes, but they totaled 2.2 gBytes. My guess is that some process went rogues and just spammed these tmp files until the system partition was filled up and everything started going to hell. I could have fixed that myself had I known that this was the issue before things go so bad that I could no longer ssh myself into the ReadyNAS.
Yeah, I was wondering if jna*.tmp files were Java related. If they are, it is quite possibly CrashPlan.
Still, I wish the ReadyNAS would not allow itself to become essentially bricked by something as simple as a overfull tmp directory. A simple solution would be to at least purge the tmp directory on every boot. That would reduce the problem to one solvable by reboot and seems the right thing to do in any case.
Afraid not. That is the first thing that tech support suggested. I tried it (and it took, resetting e.g. the admin password), but the problem was not fixed until they logged themselves in remotely and removed the tmp junk.
Do we know the exact path name(s) to tmp files on our RN devices?
Mine runs and runs and while I do occasionally download all logs and then clear them (the Backup logs do get really big if left alone) there is no other regular maintenance that I do software-wise.
One more bit of information: The problem was tracked down to the java app, CrashPlan. Reviewing the logs it turns out that, about a month ago, it started crashing every couple of minutes. Each of those crashes left a /tmp/jna*.tmp file behind. The reason for the crashes in turn was that CrashPlan kept running out of memory. Altering the JVM's memory ceiling from 512 MBytes to 2 GBytes, https://support.code42.com/CrashPlan/Latest/Troubleshooting/CrashPlan_Runs_Out_Of_Memory_And_Crashes... seems to have fixed the problem for now. However, given that the ReadyNAS only has one GByte of physical RAM, I hope CrashPlan doesn't mind virtual.
I reset mine to 1 GB some time ago (I also ran into the crashing problem). Then I upgraded the physical memory to 2 GB, which seemed to improve Crashplan's performance.
Interesting. I recall upgrading the memory on my ReadyNAS to 1 gByte, but that may have been my old NV. How much memory can you put into a ultra 6+ and what types does it take?
Ok, I have gone over the logs now and reconstruct a timeline of what happened.
Oct. 1, 2013: I installed CrashPlan 3.5.3. For the next few months it worked beautifully.
~ Mid-December 2013: CrashPlan performance starts rapidly deteriorating as the amount of backed-up files (~ 3 TBytes in mostly large files) starts hitting the java memory limit of half a gigabyte (of 1 gigabyte installed memory).
Dec. 23, 2013: CrashPlan hits the memory limit hard, crashing, leaving a 136 kByte jna*.tmp file in /tmp, and restarting. Backups basically stop but because the CrashPlan client manages to connect to the mothership every time before it crashes, I don't any warnings from them that the client is no longer running.
Jan. 18, 2014: The tmp files have used up almost all the space on the system partition and other unrelated services start malfunctioning. I post here and a few hours later the partition is completely full and the ReadyNAS is basically bricked.
Jan. 30, 2014: ReadyNAS is able to log themselves in remotely and fix the problem by moving the contents of /tmp to another partition. I change the Java memory limit to 2 GByte, which ends the crashes, but slows down the CrashPlan startup so much that it is basically useless.
Feb. 5, 2014: I get the GSkill 2x4Gbyte memory modules recommended in the above-linked thread, install and test them. CrashPlan works at full capacity once again.
