This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
I have upgraded my RN516 to 6.5.0 and some weird problem started. Logitechmediaserver didn't work anymore. After logging in I discovered the / filesystem was not writable because there seemed to be 'No space left on device'. There was stil 20% free, but obviously 'df' doesn't mean that much with btfs.
The problem seemed to be over 300.000 directories in /var/tmp all named systemd-private-XXXXX (some random string). All of them empty. After removing them, the filesystem was writable again.
Here I made a mistake; I thought it would be a good thing to do some balancing of the filesystem, but what I also wanted to do was give it a reboot; /var/tmp was filling up with systemd-private subdirs quickly, probably because of a (set of?) failing startup services. I started the balancing, forgot about it and tried to reboot the system a few hours later.
Now, almost 30 hours later; the system is pingable, but I can't do anything with it anymore. Disks are spinning, so I think it is still busy balancing.
I read that I can safely power off and on the NAS, but I wouldn't want to loose all my data.
What is the wise thing to do? I can wait a little while, but I really need the NAS back again.
Re: Safe to hard reboot while balancing is running?
It's broken due to general packaging violations in the lms-1.08 package. Please purge it (`dpkg -P lms`) before doing anything else. For LMS functionality, you'll want to switch to the much nicer and cleaner "Logitech Media Server NT" package from our Apps library.
Re: Safe to hard reboot while balancing is running?
The NAS is still busy and pingable, but it hasn't rebooted yet. I am about to power off and on the NAS, but fear loosing my data. I'm pretty sure the only thing it's doing is balancing and all other processes have died. I think reboot is just waiting for balancing to finish.
I hope that when I power cycle the NAS, services will become available again and balancing will continue.
Re: Safe to hard reboot while balancing is running?
The NAS is back online. Volume seems to be okay, but all my shares are missing.
Logging in to the device with ssh shows me everything is still there.
Probably I just have to recreate the shares and set all the permissions again.
Hope that's it.
I still think it would be better to buy myself an EXT4 based system like Synology to migrate all the stuff and get rid of the ReadyNAS.
Nothing personal, but support here has been so-so and the choice for btrfs has prove to be just plain wrong. I want to be able to completely rely on my storage. This 516 has been giving me nothing but trouble.
Re: Safe to hard reboot while balancing is running?
JasperA,
I just saw your post now after the weekend.
The problem you ran into is clearly not a problem with BTRFS. The problem you encountered could just as easily be run into when using EXT4. A 4GB EXT4 partition has a limited number of inodes and systemd is available for use with EXT4 as well as BTRFS. If you have a huge number of directories on a 4GB EXT4 partition you will use up all the inodes.
Did you happen to make a configuration backup before running into this problem? If so, that would be useful for restoring your configuration.
If you fill up the root volume (again this is the case with both EXT4 and BTRFS) then some config may get corrupted when the system attempts to write to it. If there's no space to write to it then an attempt to overwrite the config may result in an empty file.
BTRFS stores both data and metadata (data about data). As it needs to it preallocates space. If the space on the volume is fully allocated but the allocation for data or metadata is fully used then you will get the errors about no space being left on the device like you would if you were to fill all the inodes on an EXT4 volume.
You can check the usage of the root volume by using commands such as
# btrfs fi show
and
# btrfs fi df /
For the data volume it's best to keep volume usage under 80% and schedule a regular balance. For the root volume you shouldn't need to run a balance, but if you do choose to do so as the volume is quite small you should get the root volume usage well under 50% (probably under about 30%) before doing so. My root volume usage is under 15%. You can use the -dusage filter when running a balance to start by only balancing chunks that are nearly empty.
When you are running a balance and shutdown the system without cancelling it, it will automatically start again on the next boot. There is a command you can run to cancel a balance.
BTRFS is a great filesystem and it's inevitable that more and more NAS vendors will use it as time progresses.
Which apps did you install on your system? Have you installed any packages via SSH? If so, which ones? Did you check to see if updates were available for the apps you had installed before updating the firmware on your system?
You may wish to uninstall the LogitechMediaServer app you have installed and use this one instead.
From how I see it you have installed some apps on your system and you have done some things using SSH (the use of which is unsupported) which may have made the problem worse.
We'd certainly like to get some more information to see if we can reproduce what happened on your system. Some logs would be very helpful (see the Sending Logs link in my sig).
I just had a look at my RN516 and I can see no systemd-private directories under /var/tmp
Re: Safe to hard reboot while balancing is running?
Thank you for the information. At this moment /var/tmp/ keeps filling up with systemd-private-xxxxxx subdirectories.
I think I'd best do a clean firmware install. There must be a way to preserve the /data volume. At this moment I'm unable to create shares with the old names. I don't have a backup of the configuration.
I did install and de-install several apps. (through the web-interface as well as with apt-get install) I think it was a nameserver most recently. Removed that one.
When I click 'Apps' in the admin interface, the page remains empty. Installed apps -> empty, Available apps -> empty.
Re: Safe to hard reboot while balancing is running?
The 'Download logs' button does nothing. I can collect them via SSH if you like. Please tell me which files you need. (Running ReadyNAS OS 6 ) by the way.
Re: Safe to hard reboot while balancing is running?
Apart from those two failed services, I remember what I previously installed; PowerDNS:
root@truck ~# dpkg -l | grep pdns rc pdns-backend-mysql 3.1-4.1 amd64 generic MySQL backend for PowerDNS rc pdns-server 3.1-4.1 amd64 extremely powerful and versatile nameserver root@truck ~# apt-get remove pdns-backend-mysql pdns-server Reading package lists... Done Building dependency tree Reading state information... Done Package 'pdns-backend-mysql' is not installed, so not removed Package 'pdns-server' is not installed, so not removed You might want to run 'apt-get -f install' to correct these: The following packages have unmet dependencies: libc6-i386 : Depends: libc6 (= 2.19-13.netgear1) E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
Not sure if it's wise to do an 'apt-get -f install' at this point...
Re: Safe to hard reboot while balancing is running?
I sent you my logs, which I was able to download after I removed the service file for plexmediaserver. At this moment /var/tmp is filling up at the speed of about 500 new directories per minute. Which takes about 10 hours before it reaches 300.000 directories in /var/tmp. /tmp is filling up with these dirs too by the way; so it will be about 600.000 directories in the / filesystem in about 10 hours time. I believe that was the point where / was running out of space a few days ago.
So, while shares are back, the system is not usable still, because it will crash in about 10 hours. Ofcourse I can run a cron-job to remove these directories, but I want to solve the root cause.
Re: Safe to hard reboot while balancing is running?
The logs you sent are flooded with messages like this:
May 24 00:11:15 truck systemd[1]: systemd-pdns.service holdoff time over, scheduling restart.
May 24 00:11:15 truck systemd[1]: Cannot add dependency job for unit network-online.target, ignoring: Unit network-online.target failed to load: No such file or directory. See system logs and 'systemctl status network-online.target' for details.
May 24 00:11:15 truck systemd[29309]: Failed at step EXEC spawning /usr/sbin/pdns_server: No such file or directory
May 24 00:11:15 truck systemd[1]: systemd-pdns.service: control process exited, code=exited status=203
May 24 00:11:15 truck systemd[1]: Unit systemd-pdns.service entered failed state.
So perhaps you should start by removing that systemd-pdns.service file.
Re: Safe to hard reboot while balancing is running?
root@truck log# apt-get -f install Reading package lists... Done Building dependency tree Reading state information... Done Correcting dependencies... Done The following extra packages will be installed: libc6-i386 The following packages will be upgraded: libc6-i386 1 upgraded, 0 newly installed, 0 to remove and 77 not upgraded. 5 not fully installed or removed. Need to get 0 B/2386 kB of archives. After this operation, 10.2 kB of additional disk space will be used. Do you want to continue? [Y/n] y perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LC_CTYPE = "UTF-8", LANG = "en_US.UTF-8" are supported and installed on your system. perl: warning: Falling back to the standard locale ("C"). locale: Cannot set LC_CTYPE to default locale: No such file or directory locale: Cannot set LC_ALL to default locale: No such file or directory (Reading database ... 36783 files and directories currently installed.) Preparing to replace libc6-i386 2.19-13.netgear1 (using .../libc6-i386_2.19-18+deb8u3.netgear1_amd64.deb) ... Unpacking replacement libc6-i386 ... dpkg: error processing /var/cache/apt/archives/libc6-i386_2.19-18+deb8u3.netgear1_amd64.deb (--unpack): trying to overwrite '/lib/ld-linux.so.2', which is also in package lms 1.0.8 Errors were encountered while processing: /var/cache/apt/archives/libc6-i386_2.19-18+deb8u3.netgear1_amd64.deb E: Sub-process /usr/bin/dpkg returned an error code (1) root@truck log# ll /lib/ld-linux.so.2 lrwxrwxrwx 1 root root 20 Nov 8 2015 /lib/ld-linux.so.2 -> /lib32/ld-linux.so.2* root@truck log# ll /lib32/ld-linux.so.2 lrwxrwxrwx 1 root root 10 Feb 6 2015 /lib32/ld-linux.so.2 -> ld-2.19.so* root@truck log# ll /lib32/ld-2.19.so -rwxr-xr-x 1 root root 134308 Feb 6 2015 /lib32/ld-2.19.so*
Re: Safe to hard reboot while balancing is running?
It's broken due to general packaging violations in the lms-1.08 package. Please purge it (`dpkg -P lms`) before doing anything else. For LMS functionality, you'll want to switch to the much nicer and cleaner "Logitech Media Server NT" package from our Apps library.
Re: Safe to hard reboot while balancing is running?
I had pretty much the exact same problem a year ago: I initiated a restart while a balance was running, which caused the system to hang. I eventually had to force reboot my system (after waiting two weeks!) and the end result was a damaged filesystem. I had to do a full reformat. I managed to salvage most but not all of the data, but it cost a lot of time.
I was told the problem with balance preventing reboots would be fixed in a future update. But I guess that hasn't been done. (This was on OS 6.2.2)
Re: Safe to hard reboot while balancing is running?
Balance doesn't prevent reboots. That had nothing to do with the issue here. This issue was the result of bad interaction between improperly-packaged apps and manually installed extra packages.
Re: Safe to hard reboot while balancing is running?
Indeed, removing the lms package did the trick. That and removing remains of pdns and syslog-ng which I installed previously. /var/tmp isn't filling up with subdirs anymore and the NAS seems happy.
