NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Solved
RN314 don't boot anymore after trying to fix "no space left on device" error.
Hi, Here's the full story that led me to a ReadyNAS RN314 that no longer starts with a "flush_space+5a" message on the LCD. Actually, after seeing that I lost (I mean visible on the network) th...
# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
unused devices: <none>
#
StephenB wrote:You too!
Thanks
I ran mdadm --assemble --scan and it created the /dev/md * devices then I ran the the btrff check ...
Here are the results:
# mdadm --assemble --scan mdadm: /dev/md/data-0 has been started with 4 drives. mdadm: /dev/md/1 has been started with 4 drives. mdadm: /dev/md/0 has been started with 4 drives. # ls /dev autofs input mcelog network_throughput sda2 sdd2 tty1 tty20 tty31 tty42 tty53 tty7 vpd bsg kmsg md null sda3 sdd3 tty10 tty21 tty32 tty43 tty54 tty8 zero btrfs-control loop-control md0 port sdb sde tty11 tty22 tty33 tty44 tty55 tty9 bus loop0 md1 ptmx sdb1 sde1 tty12 tty23 tty34 tty45 tty56 ttyS0 console loop1 md127 ptp0 sdb2 sg0 tty13 tty24 tty35 tty46 tty57 ttyS1 cpu loop2 mem ptp1 sdb3 sg1 tty14 tty25 tty36 tty47 tty58 ttyS2 cpu_dma_latency loop3 memory_bandwidth pts sdc sg2 tty15 tty26 tty37 tty48 tty59 ttyS3 disk loop4 mpt2ctl random sdc1 sg3 tty16 tty27 tty38 tty49 tty6 urandom full loop5 mpt3ctl rfkill sdc2 sg4 tty17 tty28 tty39 tty5 tty60 vcs fuse loop6 mptctl rtc0 sdc3 snd tty18 tty29 tty4 tty50 tty61 vcs1 hwrng loop7 net sda sdd tty tty19 tty3 tty40 tty51 tty62 vcsa i2c-0 mapper network_latency sda1 sdd1 tty0 tty2 tty30 tty41 tty52 tty63 vcsa1 # btrfs check /dev/md0 Checking filesystem on /dev/md0 UUID: 1892115e-e870-45e4-87e6-4d9b5a202889 checking extents checking free space cache checking fs roots checking csums checking root refs found 2540609536 bytes used err is 0 total csum bytes: 2292960 total tree bytes: 134774784 total fs tree bytes: 126566400 total extent tree bytes: 4456448 btree space waste bytes: 21996338 file data blocks allocated: 3943616512 referenced 2357968896 #I ran again the cat /dev/mdstat:
# cat /proc/mdstat Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md0 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1] 4190208 blocks super 1.2 [4/4] [UUUU] md1 : active raid6 sda2[0] sdd2[3] sdc2[2] sdb2[1] 1047424 blocks super 1.2 level 6, 64k chunk, algorithm 2 [4/4] [UUUU] md127 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1] 8776244352 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU] bitmap: 0/22 pages [0KB], 65536KB chunk unused devices: <none> #Unfortunalty not yet:
# mount /dev/md0 /sysroot KilledI noticed that each time I get a Killed error message the md devices dispear and I need to reboot and call then btrfs asseble --scan command to restore them.
I called btrfschk comand to check/repair the file /dev/md0.
I ran
btrfs check --progress /dev/md0 btrfs check --init-csum-tree --progress /dev/md0 btrfs check --init-extent-tree --progress /dev/md0After I retried the mount and it worked.
# mount rootfs on / type rootfs (rw,size=2011292k,nr_inodes=502823) proc on /proc type proc (rw,noatime,nodiratime) sysfs on /sys type sysfs (rw,noatime,nodiratime) udev on /dev type devtmpfs (rw,noatime,nodiratime,size=10240k,nr_inodes=502828,mode=755) devpts on /dev/pts type devpts (rw,noatime,nodiratime,mode=600,ptmxmode=000) /dev/md0 on /sysroot type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=5,subvol=/) # btrfs fi usage /sysroot/ Overall: Device size: 4.00GiB Device allocated: 4.00GiB Device unallocated: 0.00B Device missing: 0.00B Used: 2.29GiB Free (estimated): 1.53GiB (min: 1.53GiB) Data ratio: 1.00 Metadata ratio: 1.96 Global reserve: 64.00MiB (used: 0.00B) Data,single: Size:3.57GiB, Used:2.04GiB /dev/md0 3.57GiB Metadata,single: Size:8.00MiB, Used:0.00B /dev/md0 8.00MiB Metadata,DUP: Size:204.56MiB, Used:125.88MiB /dev/md0 409.13MiB System,single: Size:4.00MiB, Used:0.00B /dev/md0 4.00MiB System,DUP: Size:8.00MiB, Used:16.00KiB /dev/md0 16.00MiB Unallocated: /dev/md0 0.00B #
curso wrote:
/dev/md0 on /sysroot type btrfsSo it is mounted now.
Are you seeing anything odd when you do a ls -als /sysroot ? Normally you'd see something like this:
16 drwxr-xr-x 1 root root 260 Dec 24 00:05 . 16 drwxr-xr-x 1 root root 260 Dec 24 00:05 .. 0 drwxrwxr-x 1 admin admin 226 Jun 13 2024 apps 0 drwxr-xr-x 1 root root 1338 May 22 2023 bin 0 drwxr-xr-x 1 root root 0 May 21 2012 boot 32 drwxrwxr-x+ 1 root root 570 Jun 3 2024 data 0 drwxr-xr-x 16 root root 3440 Jun 13 2024 dev 0 drwxr-xr-x 1 root root 2800 Nov 17 15:05 etc 0 drwxr-xr-x 1 root root 54 May 22 2023 frontview 0 -rw-r--r-- 1 root root 0 Nov 9 2016 ftp_ban.tbl 0 drwxr-xr-x 1 admin admin 10 Mar 12 2023 home 4 lrwxrwxrwx 1 root root 5 Nov 22 2016 homes -> /home 0 drwxr-xr-x 1 root root 462 Feb 20 2024 lib 0 drwxr-xr-x 1 root root 40 May 22 2023 lib64 0 drwxr-xr-x 1 root root 32 Jun 4 2012 media 0 drwxr-xr-x 1 root root 0 Jan 25 2016 mnt 0 drwxr-xr-x 1 root root 98 Oct 11 2022 opt 0 dr-xr-xr-x 258 root root 0 Jun 13 2024 proc 4 -rw------- 1 root root 1024 Nov 23 2016 .rnd 0 drwx------ 1 root root 92 Nov 3 2023 root 0 drwxrwxr-x 25 root admin 840 Dec 24 05:00 run 0 drwxr-xr-x 1 root root 2590 May 22 2023 sbin 0 drwxr-xr-x 1 root root 0 Jun 4 2012 srv 0 dr-xr-xr-x 12 root root 0 Aug 22 10:29 sys 0 drwxrwxrwt 1 root root 134 Dec 24 09:17 tmp 4 -rw-r--r-- 1 root root 25 Apr 30 2019 .update_fail 0 -rw-r--r-- 1 root root 0 May 22 2023 .update_success 0 drwxr-xr-x 1 root root 70 Oct 11 2022 usr 0 drwxr-xr-x 1 root root 146 May 22 2023 varThis is a problem:
Device allocated: 4.00GiB Device unallocated: 0.00BTry running btrfs balance /sysroot/ and see if you get unallocated space back.
FWIW, my own system shows this:
Device size: 4.00GiB Device allocated: 1.53GiB Device unallocated: 2.46GiB Device missing: 0.00B Used: 1021.90MiB Free (estimated): 2.71GiB (min: 1.48GiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 16.00MiB (used: 0.00B)I rebooted and miracle it seems that everything is back.
Time will tell if all back to normal but anyway thank already for all you ahelp and support.
I have to leave for Christmas Eve and will be for two days.
Thanks again, I will keep you posted.
As mentionned it seems back. I rebooted in normal mode I have control back.
But When I will come back I will reboot in 'Tech support' mode to do a balance because I guess the FS must be unmounted.
I just ran it and I got an errror
root@readynas:~# btrfs balance --full-balance // ERROR: error during balancing '//': Input/output error There may be more info in syslog - try dmesg | tail root@readynas:~# root@readynas:~# dmesg | tail [ 2098.551351] BTRFS info (device md0): relocating block group 2818244608 flags data [ 2103.283051] BTRFS info (device md0): relocating block group 2389180416 flags data [ 2108.078305] BTRFS info (device md0): relocating block group 1960116224 flags data [ 2108.388648] BTRFS info (device md0): relocating block group 1531052032 flags data [ 2112.394487] BTRFS info (device md0): relocating block group 1101987840 flags data [ 2112.700304] BTRFS info (device md0): relocating block group 672923648 flags data [ 2118.422916] BTRFS info (device md0): relocating block group 243859456 flags data [ 2124.663008] BTRFS warning (device md0): csum failed ino 278 off 243941376 csum 2628970915 expected csum 3116378598 [ 2124.678379] BTRFS warning (device md0): csum failed ino 278 off 243941376 csum 2628970915 expected csum 3116378598 [ 2126.027199] BTRFS info (device md0): 8 enospc errors during balance root@readynas:~#I guess I have still some inconstistance on the filesystem.
How I fix them to make the balancing working?
curso wrote:
How I fix them to make the balancing working?
Check the usage again, looking specifically at unallocated space.. Despite the checksum, the balance did appear to work.
What does this return?: btrfs inspect-internal inode-resolve 278 //
Normally it will return the file name of the corrupted file.
I ran it here is the result
root@readynas:/tmp# btrfs inspect-internal inode-resolve 278 // ERROR: ino paths ioctl: No such file or directory root@readynas:/tmp# ls -il / total 48 256 drwxrwxr-x 1 root root 674 Dec 24 16:27 apps 257 drwxr-xr-x 1 root root 1738 Mar 16 2024 bin 374 drwxr-xr-x 1 root root 0 May 21 2012 boot 12249745 -rwxr-xr-x 1 root root 4160 Jun 4 2013 cp_check_md5sums.sh 256 drwxr-xr-x 1 root root 586 Dec 24 16:55 data 1025 drwxr-xr-x 16 root root 3380 Dec 24 16:01 dev 422 drwxr-xr-x 1 root root 3574 Nov 23 12:22 etc 194065248 drwxr-xr-x 1 root root 54 Mar 16 2024 frontview 89278 -rw-r--r-- 1 root root 0 Jun 18 2014 ftp_ban.tbl 256 drwxr-xr-x 1 admin admin 28 Mar 17 2018 home 9699 lrwxrwxrwx 1 root root 5 Jun 18 2014 homes -> /home 1320 drwxr-xr-x 1 root root 822 Mar 16 2024 lib 12093 drwxr-xr-x 1 root root 1304 Aug 15 2017 lib32 2197 drwxr-xr-x 1 root root 40 Mar 16 2024 lib64 2199 drwxr-xr-x 1 root root 0 Mar 23 2024 media 2200 drwxr-xr-x 1 root root 26 Jan 25 2016 mnt 2201 drwxr-xr-x 1 root root 98 Oct 11 2022 opt 193170984 drwxr-xr-x 1 root root 4 Aug 26 2023 path 1 dr-xr-xr-x 328 root root 0 Dec 24 16:00 proc 2351 drwx------ 1 root root 620 Dec 24 16:56 root 1235 drwxrwxr-x 28 root admin 1080 Dec 24 16:02 run 2355 drwxr-xr-x 1 root root 2944 Mar 16 2024 sbin 2510 drwxr-xr-x 1 root root 0 Feb 28 2012 selinux 2511 drwxr-xr-x 1 root root 0 Jun 4 2012 srv 1 dr-xr-xr-x 12 root root 0 Dec 24 16:40 sys 188474452 -rw------- 1 root root 18 Feb 21 2024 syslog 2513 drwxrwxrwt 1 root root 178 Dec 25 20:07 tmp 2514 drwxr-xr-x 1 root root 112 Oct 11 2022 usr 8070 drwxr-xr-x 1 root root 188 Mar 16 2024 var 181144103 drwxr-xr-x 1 root root 42 May 7 2022 yacht root@readynas:/tmp#Then I do not no which file(s) is(are) corrupted.
But by curiosity I ran again btrfs balance // and this time it stopped with a "no space left on device" error message. I ran several times with the same error message, but when I run "", the allocated space was no more 4.0GB 8-), there is progress.
root@readynas:/tmp# btrfs fi df -H // Data, single: total=2.59GB, used=2.23GB System, DUP: total=33.55MB, used=16.38kB Metadata, DUP: total=134.22MB, used=42.73MB GlobalReserve, single: total=33.55MB, used=0.00B root@readynas:/tmp# btrfs fi usage / Overall: Device size: 4.00GiB Device allocated: 2.72GiB Device unallocated: 1.27GiB Device missing: 0.00B Used: 2.16GiB Free (estimated): 1.61GiB (min: 993.96MiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 32.00MiB (used: 0.00B) Data,single: Size:2.41GiB, Used:2.08GiB /dev/md0 2.41GiB Metadata,DUP: Size:128.00MiB, Used:40.75MiB /dev/md0 256.00MiB System,DUP: Size:32.00MiB, Used:16.00KiB /dev/md0 64.00MiB Unallocated: /dev/md0 1.27GiB root@readynas:/tmp# ls -al /var/cores/core-readynasd -rw------- 1 root root 281935872 Aug 16 2018 /var/cores/core-readynasd root@readynas:/tmp# ls -alh /var/cores/core-readynasd -rw------- 1 root root 269M Aug 16 2018 /var/cores/core-readynasd root@readynas:/tmp#You now have some unallocated space so no need to run balance again.
curso wrote:
I ran it
No, you didn't. The command is
btrfs inspect-internal inode-resolve 278 //The two dashes (inspect-internal and inode-resolve) matter.
curso wrote:
-rw------- 1 root root 269M Aug 16 2018 /var/cores/core-readynasd
You can delete this file.
OK
Thanks for your support and help.
I whish you a nice and happy end of year 2024.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
