NAS 104 locking up

Could you send in your system logs?

I've seen the same on OS6, though on the #notsupported method of running it on an Ultra2. In my case, btrfs/kernel memory is doing something weird. Via SSH, the system says 'out of disk space', even though clearly has space, likely some memory leak. The GUI (readynasd) is unusable because it cannot write to its database (disk full), so it won't even load up. Only a reboot resolves it for ~12 hours.

If you have SSH, are you able to even try 'touch ~/test', or does it report disk full in that state?

$ ssh root@10.1.2.1
root@10.1.2.1's password: 
Welcome to ReadyNASOS 6.0.5
# touch test
touch: cannot touch `test': No space left on device
# free
             total       used       free     shared    buffers     cached
Mem:       1009736     765592     244144          0       1272     563856
-/+ buffers/cache:     200464     809272
Swap:       523964          0     523964
# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          4.0G  2.2G  1.4G  62% /
tmpfs            10M  4.0K   10M   1% /dev
/dev/md0        4.0G  2.2G  1.4G  62% /
tmpfs           494M  4.0K  494M   1% /dev/shm
tmpfs           494M  7.1M  486M   2% /run
tmpfs           494M     0  494M   0% /sys/fs/cgroup
tmpfs           494M     0  494M   0% /media
/dev/md127      927G  215G  675G  25% /data
/dev/md127      927G  215G  675G  25% /home
/dev/md127      927G  215G  675G  25% /apps

Out of curiosity, I went poking around, and found that if I disable the space_cache option on this mount point, the 'out of disk space' issue goes away, not requiring a reboot to get back to sane function.

# mount
tmpfs on /dev type tmpfs (rw,relatime,size=10240k,mode=755)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
/dev/md0 on / type btrfs (rw,relatime,space_cache)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=24,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /media type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=755)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/md127 on /data type btrfs (rw,relatime,space_cache)
/dev/md127 on /home type btrfs (rw,relatime,space_cache)
/dev/md127 on /apps type btrfs (rw,relatime,space_cache)
# mount -o remount,nospace_cache /
# mount
tmpfs on /dev type tmpfs (rw,relatime,size=10240k,mode=755)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
/dev/md0 on / type btrfs (rw,relatime,nospace_cache)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=24,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /media type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=755)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/md127 on /data type btrfs (rw,relatime,space_cache)
/dev/md127 on /home type btrfs (rw,relatime,space_cache)
/dev/md127 on /apps type btrfs (rw,relatime,space_cache)
# touch test
# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          4.0G  2.1G  1.4G  62% /
tmpfs            10M  4.0K   10M   1% /dev
/dev/md0        4.0G  2.1G  1.4G  62% /
tmpfs           494M  4.0K  494M   1% /dev/shm
tmpfs           494M  7.1M  486M   2% /run
tmpfs           494M     0  494M   0% /sys/fs/cgroup
tmpfs           494M     0  494M   0% /media
/dev/md127      927G  215G  675G  25% /data
/dev/md127      927G  215G  675G  25% /home
/dev/md127      927G  215G  675G  25% /apps

RN104 uses ext4 for the root filesystem, so this is likely something different.

chirpa, can you try running:

mount -o remount,clear_cache /

I've also seen apps that would create large files on root, then unlink them while they're still in use -- thereby not actually freeing the space. So that would be another possibility.

I'll give that a try. Why does mtab list root as btrfs and not ext4? Backwards compatible, so no real hardm? Or is this a side affect of OS6 on old x86 models?

I recall BackupExec RALUS had this issue once. Do you remember what proc tools were used to narrow down which process was 'eating' memory?

Anyways, don't want to side-track AndrewT's thread if his issue turns out to be not related to this disk space eating issue.

All x86 boxes do btrfs on root; ARM does ext4. So your mtab should be accurate. You can look at the entries in /proc/*/fd/ to see open file descriptors.

Jedi Knight wrote:
Could you send in your system logs?

Which system logs ? The logs via Web interface do not show any errors. The Readynas locked up again last night ;(

See the Sending System Logs link in his sig.

If i try to download the logs i see the following message...

same problem on a mac and pc...

Gateway Time-out

The gateway did not receive a timely response from the upstream server or application.
Apache/2.2.22 (Debian) Server at 192.168.1.5 Port 443

Installed software version 6.0.6 , I can now download the logs files. unfortunately they do not go back to the last crash .. Playing the waiting game..

Any release notes of 6.0.6 (no expecting any still waiting for release notes for 6.0.5)

Andrew~

104 Lockup last night thought 6.0.6 might have fixed the problem... log file on there way to Netgear..