RN314 don't boot anymore after trying to fix "no space left on device" error.

---

curso wrote:

Weird, I don't see any /dev/md* devices. But rnutil get_disk_info sees and list well my  my 4 x 3TB disks.

 

---

It sounds like md0 hasn't been assembled.  What you see in cat /proc/mdstat ?

---

curso wrote:

By the way merry Christmas.

 

---

You too!

# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
unused devices: <none>
#

---

StephenB  wrote:

You too!

Thanks

---

I ran  mdadm --assemble --scan and it created the /dev/md * devices then I ran the the btrff check ...

Here are the results:

# mdadm --assemble --scan
mdadm: /dev/md/data-0 has been started with 4 drives.
mdadm: /dev/md/1 has been started with 4 drives.
mdadm: /dev/md/0 has been started with 4 drives.
# ls /dev
autofs              input               mcelog              network_throughput  sda2                sdd2                tty1                tty20               tty31               tty42               tty53               tty7                vpd
bsg                 kmsg                md                  null                sda3                sdd3                tty10               tty21               tty32               tty43               tty54               tty8                zero
btrfs-control       loop-control        md0                 port                sdb                 sde                 tty11               tty22               tty33               tty44               tty55               tty9
bus                 loop0               md1                 ptmx                sdb1                sde1                tty12               tty23               tty34               tty45               tty56               ttyS0
console             loop1               md127               ptp0                sdb2                sg0                 tty13               tty24               tty35               tty46               tty57               ttyS1
cpu                 loop2               mem                 ptp1                sdb3                sg1                 tty14               tty25               tty36               tty47               tty58               ttyS2
cpu_dma_latency     loop3               memory_bandwidth    pts                 sdc                 sg2                 tty15               tty26               tty37               tty48               tty59               ttyS3
disk                loop4               mpt2ctl             random              sdc1                sg3                 tty16               tty27               tty38               tty49               tty6                urandom
full                loop5               mpt3ctl             rfkill              sdc2                sg4                 tty17               tty28               tty39               tty5                tty60               vcs
fuse                loop6               mptctl              rtc0                sdc3                snd                 tty18               tty29               tty4                tty50               tty61               vcs1
hwrng               loop7               net                 sda                 sdd                 tty                 tty19               tty3                tty40               tty51               tty62               vcsa
i2c-0               mapper              network_latency     sda1                sdd1                tty0                tty2                tty30               tty41               tty52               tty63               vcsa1
# btrfs check /dev/md0
Checking filesystem on /dev/md0
UUID: 1892115e-e870-45e4-87e6-4d9b5a202889
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 2540609536 bytes used err is 0
total csum bytes: 2292960
total tree bytes: 134774784
total fs tree bytes: 126566400
total extent tree bytes: 4456448
btree space waste bytes: 21996338
file data blocks allocated: 3943616512
 referenced 2357968896
# 

I ran again the cat /dev/mdstat:

# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md0 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
      4190208 blocks super 1.2 [4/4] [UUUU]

md1 : active raid6 sda2[0] sdd2[3] sdc2[2] sdb2[1]
      1047424 blocks super 1.2 level 6, 64k chunk, algorithm 2 [4/4] [UUUU]

md127 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1]
      8776244352 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/22 pages [0KB], 65536KB chunk

unused devices: <none>
#

---

curso wrote:

I ran  mdadm --assemble --scan and it created the /dev/md * devices then I ran the the btrfs check ...

 

---

So no errors.

If you try a mount /dev/md0 /sysroot now, does it succeed?

Unfortunalty not yet:

# mount /dev/md0 /sysroot
Killed

I noticed that each time I get a Killed error message the md devices dispear and I need to reboot and call then btrfs asseble --scan command to restore them.

I called btrfschk comand to check/repair the file /dev/md0.

I ran 

btrfs check --progress /dev/md0
btrfs check --init-csum-tree --progress /dev/md0
btrfs check --init-extent-tree --progress /dev/md0

After I retried the mount and it worked.

# mount
rootfs on / type rootfs (rw,size=2011292k,nr_inodes=502823)
proc on /proc type proc (rw,noatime,nodiratime)
sysfs on /sys type sysfs (rw,noatime,nodiratime)
udev on /dev type devtmpfs (rw,noatime,nodiratime,size=10240k,nr_inodes=502828,mode=755)
devpts on /dev/pts type devpts (rw,noatime,nodiratime,mode=600,ptmxmode=000)
/dev/md0 on /sysroot type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=5,subvol=/)
# btrfs fi usage /sysroot/
Overall:
    Device size:                   4.00GiB
    Device allocated:              4.00GiB
    Device unallocated:              0.00B
    Device missing:                  0.00B
    Used:                          2.29GiB
    Free (estimated):              1.53GiB      (min: 1.53GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.96
    Global reserve:               64.00MiB      (used: 0.00B)

Data,single: Size:3.57GiB, Used:2.04GiB
   /dev/md0        3.57GiB

Metadata,single: Size:8.00MiB, Used:0.00B
   /dev/md0        8.00MiB

Metadata,DUP: Size:204.56MiB, Used:125.88MiB
   /dev/md0      409.13MiB

System,single: Size:4.00MiB, Used:0.00B
   /dev/md0        4.00MiB

System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/md0       16.00MiB

Unallocated:
   /dev/md0          0.00B
#

---

curso wrote:
/dev/md0 on /sysroot type btrfs

So it is mounted now.

Are you seeing anything odd when you do a ls -als /sysroot ?  Normally you'd see something like this:

16 drwxr-xr-x    1 root  root   260 Dec 24 00:05 .
16 drwxr-xr-x    1 root  root   260 Dec 24 00:05 ..
 0 drwxrwxr-x    1 admin admin  226 Jun 13  2024 apps
 0 drwxr-xr-x    1 root  root  1338 May 22  2023 bin
 0 drwxr-xr-x    1 root  root     0 May 21  2012 boot
32 drwxrwxr-x+   1 root  root   570 Jun  3  2024 data
 0 drwxr-xr-x   16 root  root  3440 Jun 13  2024 dev
 0 drwxr-xr-x    1 root  root  2800 Nov 17 15:05 etc
 0 drwxr-xr-x    1 root  root    54 May 22  2023 frontview
 0 -rw-r--r--    1 root  root     0 Nov  9  2016 ftp_ban.tbl
 0 drwxr-xr-x    1 admin admin   10 Mar 12  2023 home
 4 lrwxrwxrwx    1 root  root     5 Nov 22  2016 homes -> /home
 0 drwxr-xr-x    1 root  root   462 Feb 20  2024 lib
 0 drwxr-xr-x    1 root  root    40 May 22  2023 lib64
 0 drwxr-xr-x    1 root  root    32 Jun  4  2012 media
 0 drwxr-xr-x    1 root  root     0 Jan 25  2016 mnt
 0 drwxr-xr-x    1 root  root    98 Oct 11  2022 opt
 0 dr-xr-xr-x  258 root  root     0 Jun 13  2024 proc
 4 -rw-------    1 root  root  1024 Nov 23  2016 .rnd
 0 drwx------    1 root  root    92 Nov  3  2023 root
 0 drwxrwxr-x   25 root  admin  840 Dec 24 05:00 run
 0 drwxr-xr-x    1 root  root  2590 May 22  2023 sbin
 0 drwxr-xr-x    1 root  root     0 Jun  4  2012 srv
 0 dr-xr-xr-x   12 root  root     0 Aug 22 10:29 sys
 0 drwxrwxrwt    1 root  root   134 Dec 24 09:17 tmp
 4 -rw-r--r--    1 root  root    25 Apr 30  2019 .update_fail
 0 -rw-r--r--    1 root  root     0 May 22  2023 .update_success
 0 drwxr-xr-x    1 root  root    70 Oct 11  2022 usr
 0 drwxr-xr-x    1 root  root   146 May 22  2023 var

This is a problem:

    Device allocated:              4.00GiB
    Device unallocated:              0.00B

 Try running btrfs balance /sysroot/ and see if you get unallocated space back.

FWIW, my own system shows this:

    Device size:                   4.00GiB
    Device allocated:              1.53GiB
    Device unallocated:            2.46GiB
    Device missing:                  0.00B
    Used:                       1021.90MiB
    Free (estimated):              2.71GiB      (min: 1.48GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               16.00MiB      (used: 0.00B)

I rebooted and miracle it seems that everything is back.

Time will tell if all back to normal but anyway thank already for all you ahelp and support.

I have to leave for Christmas Eve and will be for two days.

Thanks again, I will keep you posted.

As mentionned it seems back. I rebooted in normal mode I have control back.

But When I will come back I will reboot in 'Tech support' mode to do a balance because I guess the FS must be unmounted.

---

curso wrote:

When I will come back I will reboot in 'Tech support' mode to do a balance because I guess th FS muste be unmounted.

---

Balances are done on mounted file systems.

btrfs balance // will work when you are booted normally.

I just ran it  and I got an errror

root@readynas:~# btrfs balance --full-balance //
ERROR: error during balancing '//': Input/output error
There may be more info in syslog - try dmesg | tail
root@readynas:~#
root@readynas:~# dmesg | tail
[ 2098.551351] BTRFS info (device md0): relocating block group 2818244608 flags data
[ 2103.283051] BTRFS info (device md0): relocating block group 2389180416 flags data
[ 2108.078305] BTRFS info (device md0): relocating block group 1960116224 flags data
[ 2108.388648] BTRFS info (device md0): relocating block group 1531052032 flags data
[ 2112.394487] BTRFS info (device md0): relocating block group 1101987840 flags data
[ 2112.700304] BTRFS info (device md0): relocating block group 672923648 flags data
[ 2118.422916] BTRFS info (device md0): relocating block group 243859456 flags data
[ 2124.663008] BTRFS warning (device md0): csum failed ino 278 off 243941376 csum 2628970915 expected csum 3116378598
[ 2124.678379] BTRFS warning (device md0): csum failed ino 278 off 243941376 csum 2628970915 expected csum 3116378598
[ 2126.027199] BTRFS info (device md0): 8 enospc errors during balance
root@readynas:~#

 I guess I have still some inconstistance on the filesystem.

How I fix them to make the balancing working?

---

curso wrote:

 

How I fix them to make the balancing working?

 

---

Check the usage again, looking specifically at unallocated space..  Despite the checksum, the balance did appear to work.

What does this return?:  btrfs inspect-internal inode-resolve 278 //

Normally it will return the file name of the corrupted file.

I ran it here is the result

root@readynas:/tmp#  btrfs inspect-internal inode-resolve 278 //
ERROR: ino paths ioctl: No such file or directory
root@readynas:/tmp# ls -il /
total 48
      256 drwxrwxr-x   1 root  root   674 Dec 24 16:27 apps
      257 drwxr-xr-x   1 root  root  1738 Mar 16  2024 bin
      374 drwxr-xr-x   1 root  root     0 May 21  2012 boot
 12249745 -rwxr-xr-x   1 root  root  4160 Jun  4  2013 cp_check_md5sums.sh
      256 drwxr-xr-x   1 root  root   586 Dec 24 16:55 data
     1025 drwxr-xr-x  16 root  root  3380 Dec 24 16:01 dev
      422 drwxr-xr-x   1 root  root  3574 Nov 23 12:22 etc
194065248 drwxr-xr-x   1 root  root    54 Mar 16  2024 frontview
    89278 -rw-r--r--   1 root  root     0 Jun 18  2014 ftp_ban.tbl
      256 drwxr-xr-x   1 admin admin   28 Mar 17  2018 home
     9699 lrwxrwxrwx   1 root  root     5 Jun 18  2014 homes -> /home
     1320 drwxr-xr-x   1 root  root   822 Mar 16  2024 lib
    12093 drwxr-xr-x   1 root  root  1304 Aug 15  2017 lib32
     2197 drwxr-xr-x   1 root  root    40 Mar 16  2024 lib64
     2199 drwxr-xr-x   1 root  root     0 Mar 23  2024 media
     2200 drwxr-xr-x   1 root  root    26 Jan 25  2016 mnt
     2201 drwxr-xr-x   1 root  root    98 Oct 11  2022 opt
193170984 drwxr-xr-x   1 root  root     4 Aug 26  2023 path
        1 dr-xr-xr-x 328 root  root     0 Dec 24 16:00 proc
     2351 drwx------   1 root  root   620 Dec 24 16:56 root
     1235 drwxrwxr-x  28 root  admin 1080 Dec 24 16:02 run
     2355 drwxr-xr-x   1 root  root  2944 Mar 16  2024 sbin
     2510 drwxr-xr-x   1 root  root     0 Feb 28  2012 selinux
     2511 drwxr-xr-x   1 root  root     0 Jun  4  2012 srv
        1 dr-xr-xr-x  12 root  root     0 Dec 24 16:40 sys
188474452 -rw-------   1 root  root    18 Feb 21  2024 syslog
     2513 drwxrwxrwt   1 root  root   178 Dec 25 20:07 tmp
     2514 drwxr-xr-x   1 root  root   112 Oct 11  2022 usr
     8070 drwxr-xr-x   1 root  root   188 Mar 16  2024 var
181144103 drwxr-xr-x   1 root  root    42 May  7  2022 yacht
root@readynas:/tmp#

Then I do not no which file(s) is(are) corrupted.

But by curiosity I ran again btrfs balance // and this time it stopped with a "no space left on device" error message. I ran several times with the same error message, but when I run "", the allocated space was no more 4.0GB 8-), there is progress.

root@readynas:/tmp# btrfs fi df -H //
Data, single: total=2.59GB, used=2.23GB
System, DUP: total=33.55MB, used=16.38kB
Metadata, DUP: total=134.22MB, used=42.73MB
GlobalReserve, single: total=33.55MB, used=0.00B
root@readynas:/tmp# btrfs fi usage /
Overall:
    Device size:                   4.00GiB
    Device allocated:              2.72GiB
    Device unallocated:            1.27GiB
    Device missing:                  0.00B
    Used:                          2.16GiB
    Free (estimated):              1.61GiB      (min: 993.96MiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               32.00MiB      (used: 0.00B)

Data,single: Size:2.41GiB, Used:2.08GiB
   /dev/md0        2.41GiB

Metadata,DUP: Size:128.00MiB, Used:40.75MiB
   /dev/md0      256.00MiB

System,DUP: Size:32.00MiB, Used:16.00KiB
   /dev/md0       64.00MiB

Unallocated:
   /dev/md0        1.27GiB
root@readynas:/tmp#  ls -al /var/cores/core-readynasd
-rw------- 1 root root 281935872 Aug 16  2018 /var/cores/core-readynasd
root@readynas:/tmp#  ls -alh /var/cores/core-readynasd
-rw------- 1 root root 269M Aug 16  2018 /var/cores/core-readynasd
root@readynas:/tmp#

You now have some unallocated space so no need to run balance again.

---

curso wrote:

I ran it 

 

---

No, you didn't.  The command is 

btrfs inspect-internal inode-resolve 278 //

The two dashes (inspect-internal and inode-resolve) matter.

---

curso wrote:

-rw------- 1 root root 269M Aug 16 2018 /var/cores/core-readynasd

---

You can delete this file.

---

StephenB  a écrit :

You now have some unallocated space so no need to run balance again.

---

Sorry it was my cut and paste that I messed up choosing a wrong format 8-(.

Here is a snapshot.

OK 

Thanks for your support and help.

I whish you a nice and happy end of year 2024.