I had a hard time coming up with a good subject for this problem, since I don't really know what's wrong and how to describe it. Here's the story:
Suddenly I noticed that CIFS was not behaving normally. I could browse shares, but not add any files.
This can of course be because of a million things, so I figured I try disabling and enabling that service in RAIDiator (4.2.27). So I disabled it and tried enabling it again, but selecting that service and clicking apply does absolutely nothing. The checkbox disappears and nothing more happens. No error message. Weird...? FTP still worked though, at least for browsing.
So I restarted the NAS. Then the CIFS service was still off and can't be started and the shares (naturally) disappeared from the network. Uh?! Not only that, but FTP had disabled itself. And, yeah, it won't start either. SSH had stopped working. Addons like Transmission as well won't start. I have Crashplan running on it, which I can't access either, so I don't really know if that starts or not.
Browsing via HTTPS still works, so the volume and the shares are still there. Everything looks green. Disks are ok etc. Status display on the front of the NAS seems normal.
So I turn to the logs. "No logs exist." Ehh? I try do download all logs anyway and it gives me some XML error message, which I guess is normal when no logs exist. I sure haven't deleted them.
So I try the memory test on the boot menu. No problems there. I also try the disk test on the boot menu. Finished fine and the NAS booted up again.
So I figure that the OS Reinstall on the boot menu should be able to solve this. Something is very corrupt/weird with how the NAS behaves! I think I managed to download a copy of the settings before I proceeded, since I wasn't sure what would actually be reset, and then I did the OS Reinstall.
The only thing that has changed is that the admin password is reset to standard - and I can't even change that. I fill in the fields and try to apply it. And, yes, nothing happens.
Then it struck me, maybe it's the browser (Chrome) or something?? So I tried IE as well, but no luck there. Also tried Safari in my phone.
I then tried getting SSH to work again by trying to upload that addon file, but I just got "Update file is not valid for this architecture" as error message.
So now I'm pretty much stuck here. I'm out of ideas! The NAS says everything is A-OK, but refuses to run anything I want to run on it and it doesn't tell me why. I can't even start rsync, which does not bode well either...
It's possible that these are symptoms of a full OS partition.
Do you have a backup?
If you had ssh enabled you could look at how full the OS partition is with some simple commands, and also fix it. Sometimes deleting some logs helps to free up some space and get things working again, and in your case may allow you to install the ssh add on. Try deleting all the backup job logs. I wouldn't delete the system logs just yet, and definitely don't download any logs, that will make things worse, and may have already.
The other methods for fixing a full OS partition are remotely (Netgear support), or a factory default.
I think you are going to need help with one of these. There may be someone here who can help you with this as a courtesy.
Disk problems manifest in a wide range of symptoms so it's worth checking all your disks with vendor tools too. You'll need to power down, remove and label the disks and hook them up to a PC to do this.
and definitely don't download any logs, that will make things worse, and may have already.
Ouch! I did not know that. 😞
I'm guessing that the OS partition is not the same thing and the volume C then, since that one is at about 85%.
Yeah, since I can't access it via SSH I guess I'm out of luck regarding those simple commands, right? 😞
I would of course appreciate the help with the "Tech support" boot menu option if someone at Netgear would be willing to help me out. I do not have a local full backup of the volume and Crashplan was nowhere near done with this either.
Could I trouble someone with this? I suspect logs related to Crashplan could maybe be the problem. 😕 Although I've been running Crashplan for a couple of years already.
/usr/local/crashplan/cache had hundreds of MBs in it as well.
I changed the crashplan configuration to store the cache and the tmp files on the data volume.
Hopefully this helps.
Your services configuration was corrupt (probably due to trying to start/stop services when the OS partition was full), so I replace it with the default one. So you may need to re-enable some services and install some apps over the top of themselves.
Thanks mdgm so much for your help! Everything is fine again. I had to reenable FTP and redo the share settings for that. And also reupload the SSH plugin. But that was pretty much it!
(Btw, good call vandermerwe. :wink:)
It seems your new setting for the Crashplan cache folder didn't stick. I found this in the config file, which seems to be the default location:
<cachePath>/usr/local/crashplan/cache</cachePath>
But if that was [only] a couple of hundred megs, then I guess the real problem was whatever was in tmp, right?
I have definitely learned a valuable lesson here. So I'm keen in making sure I don't run into this problem again in the future. Therefore, I have a couple of questions that I hope you can help me with:
1. The 2.5GB in tmp, what did it consist of mainly? What was the main culprit? Something I've installed? Or can this happen with just standard everything?
2. Can I delete anything in tmp at anytime? I mean if it allows itself to be deleted that is and is not locked.
3. Is it possible to schedule some form of routine clean-up of tmp? Or should I manually set a reminder for myself to go in and do some house cleaning now and then?
4. One final question, which I guess I already know the answer to, but I might as well ask. Is it possible to expand the OS partition or does that require some custom firmware or something? THAT I'm not getting into...!
The cache was several hundred MBs. I removed it. Not sure why the change didn't stick.
1. There were a whole of files starting with jna and ending in .tmp in the /tmp directory. Did the change of /tmp directory for Crashplan stick?
2. You can delete files from there, though some of the files in there may be in use by an application at the time e.g. if a Frontview backup job is in progress you shouldn't delete the temporary backup job log for that backup.
3. You should periodically check the fullness of the OS partition and take steps to find out what's going on if the usage is unexpectedly high.
Thanks for your replies! If you're interested, I did a little more digging and thinking. 🙂
I found out that every time Crashplan starts it puts one of those jna1234567890123456789.tmp files of exactly 136883 bytes in tmp. They're definitely related to Java. From my googling it seems it might have to do with how Java is installed and that there might be a way to stop this from happening by unpacking some jar file to somewhere and pointing some config towards that instead. Well... not for me... 🙂
I'm not sure how such files could constitute 2.5GB though. That would be 18.000+ of them. Perhaps the files grow over time? Perhaps they are created at other times as well? Were there that many or of larger sizes in there? In any case, if deleted, Crashplan still works.
I did not find any specific temp path in the Crashplan config. Only for cache. The only references to "tmp" are for files/folders that should be hidden in the GUI I think so you don't select them for backup.
Right now the total Crashplan cache size is small, but I'll keep an eye on it in parallel with the tmp folder.
Would it be possible to force the creation of these jna*.tmp files on the data volume via some symbolic link or something? I found this, but I'm not sure I'm understanding it. Running what program? http://stackoverflow.com/questions/1010 ... -io-tmpdir
I found the underlying problem for these jna files being created and eventually filling the partition. After just a few days of use now there were a large amount of them again in tmp. After some more searching I found about 20,000 log files of CrashPlan restarting (hence the creation of the jna files). Not much info in those log files, but I figured out it's because CrashPlan is running out of RAM due to my backup size.
One symptom of this happening could be that the GUI disconnects, which it has been doing actually, but not for very long, so the partition must have filled pretty rapidly over a series of days, maybe weeks, but probably not months. It's not like I'm in the CrashPlan GUI that often to react to the disconnections.
I found these recommendations on memory allocation. I had no idea about this either. 🙂
I have about 100,000 files in 5.1 TB selected atm. According to these recommendations I should allocate 5 GB of memory. I have 4 GB + the 2 GB swap. I tried settings of both 5120 and 4096, but CrashPlan wouldn't start for some reason. I then tried a more moderate setting of 2048 and that has been running stable now for a while. No crashes. For now...
I noticed someone running 2 x 4 GB in the Pro 6, but I don't think that's an option really for me. DDR2 is getting ridiculously expensive. 🙂 This also pushes me towards the RN516 when it's time to upgrade. Now I have another reason to upgrade besides running out of space on C again. 2 x 8 GB on the RN516 vs. 1 x 4 on the RN316.
I've just run into this also (seeing a bunch of restart.2015*.log files but am not seeing jna files)- and opened a ticket with crashplan support.
Even with 8 GB of ram you will be limited to ~3700M heap size. The Crashplan JVM is 32 bit, so the address space is 4 GB. Some of the address space is reserved. You can probe it with java -XmxAAAAm -showversion. If AAAA is small enough, the command will display some java info. If it gets much over 3715 on my pro6 it will crash.
FWIW my volume size is ~8.5TB and ~550K files. The size (and crashplan) have been stable for quite a while. Crashplan began to fail a couple of weeks ago. There was about 100 GB of churn (17K files) in one folder that may have triggered it.
Hmm, well that sucks. How can they advertise unlimited cloud storage and then their software won't handle more than whatever fits in 32 bits of memory space? Is it how CrashPlan is implemented to use Java that's the problem? I guess Java can run in 64 bit mode as well? Even on our NAS:es?
It's good to have your numbers as a reference point at least!
Hmm, well that sucks. How can they advertise unlimited cloud storage and then their software won't handle more than whatever fits in 32 bits of memory space? Is it how CrashPlan is implemented to use Java that's the problem? I guess Java can run in 64 bit mode as well? Even on our NAS:es?
They are actively working the case, I'll post an update when I know the resolution.
One of my questions is whether crashplan can run with a 64bit mode jvm on linux.
I've just run into this also (seeing a bunch of restart.2015*.log files but am not seeing jna files)- and opened a ticket with crashplan support.
Even with 8 GB of ram you will be limited to ~3700M heap size. The Crashplan JVM is 32 bit, so the address space is 4 GB. Some of the address space is reserved. You can probe it with java -XmxAAAAm -showversion. If AAAA is small enough, the command will display some java info. If it gets much over 3715 on my pro6 it will crash.
FWIW my volume size is ~8.5TB and ~550K files. The size (and crashplan) have been stable for quite a while. Crashplan began to fail a couple of weeks ago. There was about 100 GB of churn (17K files) in one folder that may have triggered it.
Just to follow up on this...
I was able to get backups going again with help from Crashplan support. The fix required adjusting the retention for deleted files from the default "never" down to 6 months, and then compacting the archive. That reduced the size of the server archive from ~19 TB down to ~10 TB. (De-duplication is against the server achive, not the local storage - so reducing that size helps). Then I rebuilt the cache and rebooted the NAS.
I am getting more memory for the Pro, since this will likely happen again at some point in the future.
One small mystery - my backups are working, with much less memory than Crashplan says is needed. One reason might be because compression is off and de-duplication is set to "minimal".
StephenB wrote:
One of my questions is whether crashplan can run with a 64bit mode jvm on linux.
I've reduced my selected files for backup to 4.6TB and I also changed the retention for deleted files to 6 months like you have now. I tried compacting, but it still shows as 3.9TB server side. Maybe that has to do with that the backup is still trying to complete initially for a large portion of the selected files. I still only have 2.7TB actually backed up yet. Yeah, speed towards the US datacenter is not very good for me. 🙂 But a lot of what it is backing up now is very static, so once done it should be better.
One thing I don't understand though is if it is the amount of data on the server or the selected amount on the client that causes the out of memory issues. Or a combination maybe? For you the solution was to reduce the server archive size and then both de-duplication and compression would be beneficial. However, de-duplication and compression might be detrimental to memory consumption on the client side you theorize. So do you think there might be a trade-off one has to make here to strike the right balance between the two? My guess would be that these are features you want to use. I have both on automatic.
About 64-bit, that doesn't help us on our systems since Embedded Java only exists in 32-bit, right? 😕 So until that changes there's no point in being able to allocate more than the ~3700MBs right? Still, if you run with ~3700MB then there might be a good idea to have 8GB for the system as a whole.
I reset the daily resync to 14 days while I was compacting. That eliminated the endless restarts, but it still took quite a while. I also needed to do it twice, not sure why. After the "deep compacting" step it should also do a version pruning step. I started with Crashplan in 2012, and have been backing up weekly image backups for all our PCs on the Pro. The churn from those created a lot of space on the server archive (even with de-duplication).
My backup speed is usually in the 30-50 mb/s range (not including de-duplication savings). Though I'm in the US.
There's not a lot of information on the memory usage, but my impression is that the answer is a combination. There's a hash value stored for each block on the cloud server, and every new block in the local NAS needs to be hashed (and that hash looked up in the cloud server table). It isn't very clear though, esp. since there is also an on-disk cache (without any info on what is stored in that). My cache is about 1.5 GB right now (and it is set up to be on the data volume, not the OS partition).
On 64 bit Java - I thought it was available for the Pro, but I must admit I haven't checked. I did order an 8 GB memory upgrade, under the theory that even with ~3700 MB dedicated to crashplan, I'd want at least 6 GB for the system. Plus the DDR2 memory is getting expensive, and I only wanted to buy it once.
Yeah, the compact button does not seem to do anything at the moment. Did you get some sort of progress bar when you did this? I think the backup needs to be stopped for something to happen. Or maybe the compact command is queued now after backup completes. Just 4.7 months left! 😄 My speeds are 1-2 Mbit/s... If I ever get to complete this backup I'll take another look at compacting. For now it seems to be under control memory wise though and I should be able to increase the allocation from 2048 towards ~3700 if things get hairy. Just need to check /tmp once in a while for accumulating Java files.
I used to have my PC backups on the NAS selected for backup as well, but deselected those since there are too many changes and it takes too long to upload to the server. 5.1TB -> 4.6TB. I feel it's enough with the live version on the computers + the one backup on the NAS. The data unique to the NAS is what is being backup up now. And being without all those versions related to the PC backups might be what I need to fly below the radar here in terms of memory consumption.
It's a bit weird, my cache now is only 41MB in /usr/local/crashplan/cache so it doesn't seem to be filling up my OS partition very much. If things change I might take another stab at moving the cache to C instead though, like you have.
It seems to me as well that CrashPlan can handle a lot more than the memory requirements state. Perhaps they include really large margins or error. Good news for us if that's the case. 🙂
Yeah, the compact button does not seem to do anything at the moment. Did you get some sort of progress bar when you did this?
I believe it started with "synchronizing", then "compacting (deep)" and "pruning versions (deep)". There was a progress bar, with % completion. In my case it took a very long time (~10 days). I think it is a background task, and when it found stuff to backup it would switch over to the normal backup progress.
BTW, if you look in /usr/local/crashplan/bin, are you seeing a lot of restart*.log files? One of my main symptoms was seeing a new restart log file appear whenever crashplan tried to back up a file to the server. You'll also see these resets in the history log (either tail /usr/local/crashplan/log/history.log.0 or using the client's history page).
Also the files appeared to be backing up (very slowly) from the gui, but every time it restarted it tried to back them up again. And looking at backup_files.log.0 I could see a reported transfer rate of 0 bps.
"tail /usr/local/crashplan/log/backup_files.log.0 -n 250" gives you a pretty good picture of how things are going.
Yeah, I tried compacting from one of the client computers so now I now what it should look like when it's doing the pruning and compacting. It seems like that doesn't start at the moment since it's busy backing up.
Yes, I did see those files when I had the problems earlier, but they were pretty small. The Java related files in /tmp were much larger. Since I increased the memory allocation I haven't had any more restarts.
The restart logs are small - its not that they fill up the OS partition, just that they are a sign that the application decided it was very sick and restarted itself. I didn't see them on compacting, only backup attempts. It's good that you aren't seeing them.
