RN314 FW 6.7.1 root partition full, how to fix ?

[jukkaforss]

Hi,
 
I rebooted my readynas and after boot GUI didn't start, cli is still working fine.
I logged in and it seems that root partition is full, but not full. It says 100% full, but there is only 1,4GB data in it.

oot@readynas:~# df -k
Filesystem      1K-blocks       Used  Available Use% Mounted on
udev                10240          4      10236   1% /dev
/dev/md0          4190208    1484212          0 100% /
tmpfs             1015564          0    1015564   0% /dev/shm
tmpfs             1015564       8752    1006812   1% /run
tmpfs              507784         12     507772   1% /run/lock
tmpfs             1015564          0    1015564   0% /sys/fs/cgroup
/dev/md127     8776244352 6082575816 2690603384  70% /data
/dev/md127     8776244352 6082575816 2690603384  70% /home
/dev/md127     8776244352 6082575816 2690603384  70% /apps
tmpfs               51200          0      51200   0% /var/replicate/shm
root@readynas:~#

Also btrfs fi says same thing.

oot@readynas:~# btrfs fi df /
Data, single: total=3.65GiB, used=900.52MiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=142.06MiB, used=18.44MiB
GlobalReserve, single: total=512.00MiB, used=388.38MiB
root@readynas:~# btrfs fi usage /
Overall:
    Device size:		   4.00GiB
    Device allocated:		   4.00GiB
    Device unallocated:		   1.00MiB
    Device missing:		     0.00B
    Used:			 937.43MiB
    Free (estimated):		   2.78GiB	(min: 2.78GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 388.38MiB)

Data,single: Size:3.65GiB, Used:900.52MiB
   /dev/md0	   3.65GiB

Metadata,DUP: Size:142.06MiB, Used:18.44MiB
   /dev/md0	 284.12MiB

System,DUP: Size:32.00MiB, Used:16.00KiB
   /dev/md0	  64.00MiB

Unallocated:
   /dev/md0	   1.00MiB
root@readynas:~# 

There should be 2.7GB free in it.
 
There is one big core file in root file system, but I can't delete it, it says no space left on device.

root@readynas:/#  find / -not -path "*/data/*" -not -path "*/apps/*" -size +100000k -print0 |xargs -0 ls -sd |sort -nr
find: `/proc/4946/task/4946/fd/5': No such file or directory
find: `/proc/4946/task/4946/fdinfo/5': No such file or directory
find: `/proc/4946/fd/5': No such file or directory
find: `/proc/4946/fdinfo/5': No such file or directory
49032 /var/cores/core-readynasd
root@readynas:/#  ls -al /var/cores/core-readynasd
-rw------- 1 root root 254840832 May 11 16:29 /var/cores/core-readynasd
root@readynas:/# rm /var/cores/core-readynasd
rm: cannot remove '/var/cores/core-readynasd': No space left on device

I even tried install FW again using Recovery USB boot, but no help.
 
How I can fix this problem, without wiping whole box ?

[jukkaforss · Accepted solution]

Skywalker fixed root fs somehow, and not it is working fine.

[jak0lantash]

To be able to look "under" mountpoints, mount-bind the root volume to /mnt:

mount --bind / /mnt

Then look inside it:

cd /mnt
du -d1 -h

It will tell you which is the biggest folder, then cd into it and re-run:

cd <big_folder>
du -d1 -h

Repeat the operation until there is no more big folder, only the total is big, which means it's big file(s) in the current directory, then:

ls -lhS

When you're finished cleaning up the big files, unmount the temporary bind:

cd
umount /mnt

[jak0lantash]

If deleting data doesn't work, you may also need to balance the empty chunks:

btrfs balance start -musage=0 -dusage=0 /

[jukkaforss]

---

@jak0lantash wrote:

If deleting data doesn't work, you may also need to balance the empty chunks:

btrfs balance start -musage=0 -dusage=0 /

---

root@readynas:/etc# btrfs balance start -musage=0 -dusage=0 /
Done, had to relocate 0 out of 13 chunks
root@readynas:/etc# 

It didn't help, root is still full.

[jukkaforss]

---

@jak0lantash wrote:

To be able to look "under" mountpoints, mount-bind the root volume to /mnt:

mount --bind / /mnt

Then look inside it:

cd /mnt
du -d1 -h

It will tell you which is the biggest folder, then cd into it and re-run:

cd <big_folder>
du -d1 -h

Repeat the operation until there is no more big folder, only the total is big, which means it's big file(s) in the current directory, then:

ls -lhS

When you're finished cleaning up the big files, unmount the temporary bind:

cd
umount /mnt

---

It is not possible to remove files, rm command always gives error.

root@readynas:/mnt/var/cache/apt/archives# rm libgsm1_1.0.13-4_amd64.deb
rm: cannot remove 'libgsm1_1.0.13-4_amd64.deb': No space left on device
root@readynas:/mnt/var/cache/apt/archives# 

[jukkaforss]

Is there a way to to factory reset with out disks, so that data volume is saved ?

I need to get this system up and running as soon as possible !!

[jukkaforss · Accepted solution]

Skywalker fixed root fs somehow, and not it is working fine.