NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Solved
RN314 don't boot anymore after trying to fix "no space left on device" error.
Hi, Here's the full story that led me to a ReadyNAS RN314 that no longer starts with a "flush_space+5a" message on the LCD. Actually, after seeing that I lost (I mean visible on the network) th...
Unfortunalty not yet:
# mount /dev/md0 /sysroot
Killed
I noticed that each time I get a Killed error message the md devices dispear and I need to reboot and call then btrfs asseble --scan command to restore them.
I called btrfschk comand to check/repair the file /dev/md0.
I ran
btrfs check --progress /dev/md0
btrfs check --init-csum-tree --progress /dev/md0
btrfs check --init-extent-tree --progress /dev/md0
After I retried the mount and it worked.
# mount
rootfs on / type rootfs (rw,size=2011292k,nr_inodes=502823)
proc on /proc type proc (rw,noatime,nodiratime)
sysfs on /sys type sysfs (rw,noatime,nodiratime)
udev on /dev type devtmpfs (rw,noatime,nodiratime,size=10240k,nr_inodes=502828,mode=755)
devpts on /dev/pts type devpts (rw,noatime,nodiratime,mode=600,ptmxmode=000)
/dev/md0 on /sysroot type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=5,subvol=/)
# btrfs fi usage /sysroot/
Overall:
Device size: 4.00GiB
Device allocated: 4.00GiB
Device unallocated: 0.00B
Device missing: 0.00B
Used: 2.29GiB
Free (estimated): 1.53GiB (min: 1.53GiB)
Data ratio: 1.00
Metadata ratio: 1.96
Global reserve: 64.00MiB (used: 0.00B)
Data,single: Size:3.57GiB, Used:2.04GiB
/dev/md0 3.57GiB
Metadata,single: Size:8.00MiB, Used:0.00B
/dev/md0 8.00MiB
Metadata,DUP: Size:204.56MiB, Used:125.88MiB
/dev/md0 409.13MiB
System,single: Size:4.00MiB, Used:0.00B
/dev/md0 4.00MiB
System,DUP: Size:8.00MiB, Used:16.00KiB
/dev/md0 16.00MiB
Unallocated:
/dev/md0 0.00B
#
curso wrote:
/dev/md0 on /sysroot type btrfsSo it is mounted now.
Are you seeing anything odd when you do a ls -als /sysroot ? Normally you'd see something like this:
16 drwxr-xr-x 1 root root 260 Dec 24 00:05 . 16 drwxr-xr-x 1 root root 260 Dec 24 00:05 .. 0 drwxrwxr-x 1 admin admin 226 Jun 13 2024 apps 0 drwxr-xr-x 1 root root 1338 May 22 2023 bin 0 drwxr-xr-x 1 root root 0 May 21 2012 boot 32 drwxrwxr-x+ 1 root root 570 Jun 3 2024 data 0 drwxr-xr-x 16 root root 3440 Jun 13 2024 dev 0 drwxr-xr-x 1 root root 2800 Nov 17 15:05 etc 0 drwxr-xr-x 1 root root 54 May 22 2023 frontview 0 -rw-r--r-- 1 root root 0 Nov 9 2016 ftp_ban.tbl 0 drwxr-xr-x 1 admin admin 10 Mar 12 2023 home 4 lrwxrwxrwx 1 root root 5 Nov 22 2016 homes -> /home 0 drwxr-xr-x 1 root root 462 Feb 20 2024 lib 0 drwxr-xr-x 1 root root 40 May 22 2023 lib64 0 drwxr-xr-x 1 root root 32 Jun 4 2012 media 0 drwxr-xr-x 1 root root 0 Jan 25 2016 mnt 0 drwxr-xr-x 1 root root 98 Oct 11 2022 opt 0 dr-xr-xr-x 258 root root 0 Jun 13 2024 proc 4 -rw------- 1 root root 1024 Nov 23 2016 .rnd 0 drwx------ 1 root root 92 Nov 3 2023 root 0 drwxrwxr-x 25 root admin 840 Dec 24 05:00 run 0 drwxr-xr-x 1 root root 2590 May 22 2023 sbin 0 drwxr-xr-x 1 root root 0 Jun 4 2012 srv 0 dr-xr-xr-x 12 root root 0 Aug 22 10:29 sys 0 drwxrwxrwt 1 root root 134 Dec 24 09:17 tmp 4 -rw-r--r-- 1 root root 25 Apr 30 2019 .update_fail 0 -rw-r--r-- 1 root root 0 May 22 2023 .update_success 0 drwxr-xr-x 1 root root 70 Oct 11 2022 usr 0 drwxr-xr-x 1 root root 146 May 22 2023 varThis is a problem:
Device allocated: 4.00GiB Device unallocated: 0.00BTry running btrfs balance /sysroot/ and see if you get unallocated space back.
FWIW, my own system shows this:
Device size: 4.00GiB Device allocated: 1.53GiB Device unallocated: 2.46GiB Device missing: 0.00B Used: 1021.90MiB Free (estimated): 2.71GiB (min: 1.48GiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 16.00MiB (used: 0.00B)I rebooted and miracle it seems that everything is back.
Time will tell if all back to normal but anyway thank already for all you ahelp and support.
I have to leave for Christmas Eve and will be for two days.
Thanks again, I will keep you posted.
As mentionned it seems back. I rebooted in normal mode I have control back.
But When I will come back I will reboot in 'Tech support' mode to do a balance because I guess the FS must be unmounted.
I just ran it and I got an errror
root@readynas:~# btrfs balance --full-balance // ERROR: error during balancing '//': Input/output error There may be more info in syslog - try dmesg | tail root@readynas:~# root@readynas:~# dmesg | tail [ 2098.551351] BTRFS info (device md0): relocating block group 2818244608 flags data [ 2103.283051] BTRFS info (device md0): relocating block group 2389180416 flags data [ 2108.078305] BTRFS info (device md0): relocating block group 1960116224 flags data [ 2108.388648] BTRFS info (device md0): relocating block group 1531052032 flags data [ 2112.394487] BTRFS info (device md0): relocating block group 1101987840 flags data [ 2112.700304] BTRFS info (device md0): relocating block group 672923648 flags data [ 2118.422916] BTRFS info (device md0): relocating block group 243859456 flags data [ 2124.663008] BTRFS warning (device md0): csum failed ino 278 off 243941376 csum 2628970915 expected csum 3116378598 [ 2124.678379] BTRFS warning (device md0): csum failed ino 278 off 243941376 csum 2628970915 expected csum 3116378598 [ 2126.027199] BTRFS info (device md0): 8 enospc errors during balance root@readynas:~#I guess I have still some inconstistance on the filesystem.
How I fix them to make the balancing working?
curso wrote:
How I fix them to make the balancing working?
Check the usage again, looking specifically at unallocated space.. Despite the checksum, the balance did appear to work.
What does this return?: btrfs inspect-internal inode-resolve 278 //
Normally it will return the file name of the corrupted file.
I ran it here is the result
root@readynas:/tmp# btrfs inspect-internal inode-resolve 278 // ERROR: ino paths ioctl: No such file or directory root@readynas:/tmp# ls -il / total 48 256 drwxrwxr-x 1 root root 674 Dec 24 16:27 apps 257 drwxr-xr-x 1 root root 1738 Mar 16 2024 bin 374 drwxr-xr-x 1 root root 0 May 21 2012 boot 12249745 -rwxr-xr-x 1 root root 4160 Jun 4 2013 cp_check_md5sums.sh 256 drwxr-xr-x 1 root root 586 Dec 24 16:55 data 1025 drwxr-xr-x 16 root root 3380 Dec 24 16:01 dev 422 drwxr-xr-x 1 root root 3574 Nov 23 12:22 etc 194065248 drwxr-xr-x 1 root root 54 Mar 16 2024 frontview 89278 -rw-r--r-- 1 root root 0 Jun 18 2014 ftp_ban.tbl 256 drwxr-xr-x 1 admin admin 28 Mar 17 2018 home 9699 lrwxrwxrwx 1 root root 5 Jun 18 2014 homes -> /home 1320 drwxr-xr-x 1 root root 822 Mar 16 2024 lib 12093 drwxr-xr-x 1 root root 1304 Aug 15 2017 lib32 2197 drwxr-xr-x 1 root root 40 Mar 16 2024 lib64 2199 drwxr-xr-x 1 root root 0 Mar 23 2024 media 2200 drwxr-xr-x 1 root root 26 Jan 25 2016 mnt 2201 drwxr-xr-x 1 root root 98 Oct 11 2022 opt 193170984 drwxr-xr-x 1 root root 4 Aug 26 2023 path 1 dr-xr-xr-x 328 root root 0 Dec 24 16:00 proc 2351 drwx------ 1 root root 620 Dec 24 16:56 root 1235 drwxrwxr-x 28 root admin 1080 Dec 24 16:02 run 2355 drwxr-xr-x 1 root root 2944 Mar 16 2024 sbin 2510 drwxr-xr-x 1 root root 0 Feb 28 2012 selinux 2511 drwxr-xr-x 1 root root 0 Jun 4 2012 srv 1 dr-xr-xr-x 12 root root 0 Dec 24 16:40 sys 188474452 -rw------- 1 root root 18 Feb 21 2024 syslog 2513 drwxrwxrwt 1 root root 178 Dec 25 20:07 tmp 2514 drwxr-xr-x 1 root root 112 Oct 11 2022 usr 8070 drwxr-xr-x 1 root root 188 Mar 16 2024 var 181144103 drwxr-xr-x 1 root root 42 May 7 2022 yacht root@readynas:/tmp#Then I do not no which file(s) is(are) corrupted.
But by curiosity I ran again btrfs balance // and this time it stopped with a "no space left on device" error message. I ran several times with the same error message, but when I run "", the allocated space was no more 4.0GB 8-), there is progress.
root@readynas:/tmp# btrfs fi df -H // Data, single: total=2.59GB, used=2.23GB System, DUP: total=33.55MB, used=16.38kB Metadata, DUP: total=134.22MB, used=42.73MB GlobalReserve, single: total=33.55MB, used=0.00B root@readynas:/tmp# btrfs fi usage / Overall: Device size: 4.00GiB Device allocated: 2.72GiB Device unallocated: 1.27GiB Device missing: 0.00B Used: 2.16GiB Free (estimated): 1.61GiB (min: 993.96MiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 32.00MiB (used: 0.00B) Data,single: Size:2.41GiB, Used:2.08GiB /dev/md0 2.41GiB Metadata,DUP: Size:128.00MiB, Used:40.75MiB /dev/md0 256.00MiB System,DUP: Size:32.00MiB, Used:16.00KiB /dev/md0 64.00MiB Unallocated: /dev/md0 1.27GiB root@readynas:/tmp# ls -al /var/cores/core-readynasd -rw------- 1 root root 281935872 Aug 16 2018 /var/cores/core-readynasd root@readynas:/tmp# ls -alh /var/cores/core-readynasd -rw------- 1 root root 269M Aug 16 2018 /var/cores/core-readynasd root@readynas:/tmp#You now have some unallocated space so no need to run balance again.
curso wrote:
I ran it
No, you didn't. The command is
btrfs inspect-internal inode-resolve 278 //The two dashes (inspect-internal and inode-resolve) matter.
curso wrote:
-rw------- 1 root root 269M Aug 16 2018 /var/cores/core-readynasd
You can delete this file.
OK
Thanks for your support and help.
I whish you a nice and happy end of year 2024.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
