× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

OS6 stuck at booting / mounting btrfs

MaxxMark
Luminary

OS6 stuck at booting / mounting btrfs

I rebooted my NAS today after it became a bit sluggish and prior to wanting to update to 6.9.1 (i'm at the latest 6.8.x version right now).

 

After rebooting the NAS didnt come up, and the LCD displays "Booting #########>  89%".

 

I can log in to the nas via SSH and I noticed that [btrfs-transaction] is consuming 99% CPU.

 

Some further digging shows me that it is seems to be hanging on "/bin/mount /dev/disk/by-label/33ea2307:data /data -t btrfs"

 

The last bit of dmesg is attached (whichttp://www.maxxmark.com/dropbox/nas.dmesg.out.txt h describes it indeed hanging).

 

Is there someway to see what (and if) it is still doing something? I was reluctant to just try and reobot it again. Any suggestions?

 

 

Edit: after a while it stopped from showing [btrfs-transaction] consuming 100% and went to the /bin/mount as described above to consume 100% cpu.

And the LCD display now showed:

Fail to startup

#################> 94%

 

Edit 2:

I just noticed that my nas around the time of rebooting mailed "Balance complete for volume data". So my guess it was balancing (which surely would explain the sluggishness of the nas). Is it possible it tried to (or did) cancel the balancing, and restarted the balancing on reboot, and is now continueing the balance? 

Model: RNDP600E |ReadyNAS Pro Pioneer Edition|EOL
Message 1 of 5

Accepted Solutions
MaxxMark
Luminary

Re: OS6 stuck at booting / mounting btrfs

I'm one small step closer.

 

I based it off of this post: https://askubuntu.com/questions/157917/how-do-i-recover-a-btrfs-partition-that-will-not-mount

 

I have done a btrfsck on the system which seems to state that there ís something wrong with my volume:

 

 

root@NAS:/var/log/frontview# btrfsck -p /dev/md127 
Checking filesystem on /dev/md127
UUID: d3d6c597-3eb2-4923-bb50-ee21e11dcfec
checking extents [o]
checking free space cache [o]
checking fs roots [O]
checking csums
checking root refs
checking quota groups
Counts for qgroup id: 0/262 are different
our: referenced 12925101330432 referenced compressed 12925101330432
disk: referenced 12849220059136 referenced compressed 12849220059136
diff: referenced 75881271296 referenced compressed 75881271296
our: exclusive 12925101330432 exclusive compressed 12925101330432
disk: exclusive 12849220059136 exclusive compressed 12849220059136
diff: exclusive 75881271296 exclusive compressed 75881271296
Counts for qgroup id: 0/270 are different
our: referenced 1132319191040 referenced compressed 1132319191040
disk: referenced 1131488161792 referenced compressed 1131488161792
diff: referenced 831029248 referenced compressed 831029248
our: exclusive 1132319191040 exclusive compressed 1132319191040
disk: exclusive 1131488161792 exclusive compressed 1131488161792
diff: exclusive 831029248 exclusive compressed 831029248
Counts for qgroup id: 0/277 are different
our: referenced 20862586880 referenced compressed 20862586880
disk: referenced 20729131008 referenced compressed 20729131008
diff: referenced 133455872 referenced compressed 133455872
our: exclusive 20862586880 exclusive compressed 20862586880
disk: exclusive 20729131008 exclusive compressed 20729131008
diff: exclusive 133455872 exclusive compressed 133455872
Counts for qgroup id: 0/278 are different
our: referenced 83479613440 referenced compressed 83479613440
disk: referenced 83248349184 referenced compressed 83248349184
diff: referenced 231264256 referenced compressed 231264256
our: exclusive 83479613440 exclusive compressed 83479613440
disk: exclusive 83248349184 exclusive compressed 83248349184
diff: exclusive 231264256 exclusive compressed 231264256
Counts for qgroup id: 0/279 are different
our: referenced 41265213440 referenced compressed 41265213440
disk: referenced 33458343936 referenced compressed 33458343936
diff: referenced 7806869504 referenced compressed 7806869504
our: exclusive 41265213440 exclusive compressed 41265213440
disk: exclusive 33458343936 exclusive compressed 33458343936
diff: exclusive 7806869504 exclusive compressed 7806869504
found 14220525928448 bytes used, no error found
total csum bytes: 141941444
total tree bytes: 1367834624
total fs tree bytes: 827392000
total extent tree bytes: 347897856
btree space waste bytes: 251305664
file data blocks allocated: 21737524551680
 referenced 14189602930688
 

Then I followed up with mounting the partition:

 

root@NAS:/mnt# mount -t btrfs -o recovery,nospace_cache,nospace_cache /dev/md127 /mnt/datadisk/

root@NAS:/mnt# cd datadisk/

root@NAS:/mnt/datadisk# ls -l total 136 drwxr-xr-x 1 root root 230 Nov 23 18:09 . drwxr-xr-x 1 root root 16 Dec 20 18:32 .. drwxrwxrwx+ 1 admin admin 1352 Nov 17 00:56 Backup drwxrwxrwx+ 1 guest guest 20 Aug 20 15:11 Documents drwxr-xr-x 1 admin admin 94 Aug 20 15:07 home drwxrwxrwx+ 1 guest guest 20 Aug 20 15:11 Music drwxrwxrwx+ 1 guest guest 38 Sep 15 17:41 Pictures drwxrwxrwx+ 1 502 dialout 170 Dec 18 18:56 Videos root@NAS:/mnt/datadisk# du -sh 13T .
root@NAS:/mnt/datadisk# df -h Filesystem Size Used Avail Use% Mounted on udev 10M 4.0K 10M 1% /dev /dev/md0 4.0G 555M 3.1G 15% / tmpfs 998M 0 998M 0% /dev/shm tmpfs 998M 3.7M 994M 1% /run tmpfs 499M 1.1M 498M 1% /run/lock tmpfs 998M 0 998M 0% /sys/fs/cgroup /dev/md127 14T 13T 701G 95% /mnt/datadisk

 

Edit:

 

I've unmounted the partition and tried to mount it normally again and that worked like a charm:

 

 

root@NAS:/mnt# mount -t btrfs /dev/md127 /mnt/datadisk/
root@NAS:/mnt# cd datadisk/
root@NAS:/mnt/datadisk# ls -l
total 136
drwxr-xr-x  1 root  root       230 Nov 23 18:09 .
drwxr-xr-x  1 root  root        16 Dec 20 18:32 ..
drwxrwxrwx+ 1 admin admin     1352 Nov 17 00:56 Backup
drwxrwxrwx+ 1 guest guest       20 Aug 20 15:11 Documents
drwxr-xr-x  1 admin admin       94 Aug 20 15:07 home
drwxrwxrwx+ 1 guest guest       20 Aug 20 15:11 Music
drwxrwxrwx+ 1 guest guest       38 Sep 15 17:41 Pictures
drwxrwxrwx+ 1   502 dialout    170 Dec 18 18:56 Videos

root@NAS:/mnt/datadisk# df -h Filesystem Size Used Avail Use% Mounted on udev 10M 4.0K 10M 1% /dev /dev/md0 4.0G 555M 3.1G 15% / tmpfs 998M 0 998M 0% /dev/shm tmpfs 998M 3.1M 995M 1% /run tmpfs 499M 1.1M 498M 1% /run/lock tmpfs 998M 0 998M 0% /sys/fs/cgroup /dev/md127 14T 13

 

 

 

Now i'm contemplating on what to do next... I might just do a reboot and hope for the best (as it SHOULD come up again, because there is now nothing that should prevent the mount action from failing). But then again... I have this strange feeling that it might not come up correct again 🙂

 

View solution in original post

Message 2 of 5

All Replies
MaxxMark
Luminary

Re: OS6 stuck at booting / mounting btrfs

I'm one small step closer.

 

I based it off of this post: https://askubuntu.com/questions/157917/how-do-i-recover-a-btrfs-partition-that-will-not-mount

 

I have done a btrfsck on the system which seems to state that there ís something wrong with my volume:

 

 

root@NAS:/var/log/frontview# btrfsck -p /dev/md127 
Checking filesystem on /dev/md127
UUID: d3d6c597-3eb2-4923-bb50-ee21e11dcfec
checking extents [o]
checking free space cache [o]
checking fs roots [O]
checking csums
checking root refs
checking quota groups
Counts for qgroup id: 0/262 are different
our: referenced 12925101330432 referenced compressed 12925101330432
disk: referenced 12849220059136 referenced compressed 12849220059136
diff: referenced 75881271296 referenced compressed 75881271296
our: exclusive 12925101330432 exclusive compressed 12925101330432
disk: exclusive 12849220059136 exclusive compressed 12849220059136
diff: exclusive 75881271296 exclusive compressed 75881271296
Counts for qgroup id: 0/270 are different
our: referenced 1132319191040 referenced compressed 1132319191040
disk: referenced 1131488161792 referenced compressed 1131488161792
diff: referenced 831029248 referenced compressed 831029248
our: exclusive 1132319191040 exclusive compressed 1132319191040
disk: exclusive 1131488161792 exclusive compressed 1131488161792
diff: exclusive 831029248 exclusive compressed 831029248
Counts for qgroup id: 0/277 are different
our: referenced 20862586880 referenced compressed 20862586880
disk: referenced 20729131008 referenced compressed 20729131008
diff: referenced 133455872 referenced compressed 133455872
our: exclusive 20862586880 exclusive compressed 20862586880
disk: exclusive 20729131008 exclusive compressed 20729131008
diff: exclusive 133455872 exclusive compressed 133455872
Counts for qgroup id: 0/278 are different
our: referenced 83479613440 referenced compressed 83479613440
disk: referenced 83248349184 referenced compressed 83248349184
diff: referenced 231264256 referenced compressed 231264256
our: exclusive 83479613440 exclusive compressed 83479613440
disk: exclusive 83248349184 exclusive compressed 83248349184
diff: exclusive 231264256 exclusive compressed 231264256
Counts for qgroup id: 0/279 are different
our: referenced 41265213440 referenced compressed 41265213440
disk: referenced 33458343936 referenced compressed 33458343936
diff: referenced 7806869504 referenced compressed 7806869504
our: exclusive 41265213440 exclusive compressed 41265213440
disk: exclusive 33458343936 exclusive compressed 33458343936
diff: exclusive 7806869504 exclusive compressed 7806869504
found 14220525928448 bytes used, no error found
total csum bytes: 141941444
total tree bytes: 1367834624
total fs tree bytes: 827392000
total extent tree bytes: 347897856
btree space waste bytes: 251305664
file data blocks allocated: 21737524551680
 referenced 14189602930688
 

Then I followed up with mounting the partition:

 

root@NAS:/mnt# mount -t btrfs -o recovery,nospace_cache,nospace_cache /dev/md127 /mnt/datadisk/

root@NAS:/mnt# cd datadisk/

root@NAS:/mnt/datadisk# ls -l total 136 drwxr-xr-x 1 root root 230 Nov 23 18:09 . drwxr-xr-x 1 root root 16 Dec 20 18:32 .. drwxrwxrwx+ 1 admin admin 1352 Nov 17 00:56 Backup drwxrwxrwx+ 1 guest guest 20 Aug 20 15:11 Documents drwxr-xr-x 1 admin admin 94 Aug 20 15:07 home drwxrwxrwx+ 1 guest guest 20 Aug 20 15:11 Music drwxrwxrwx+ 1 guest guest 38 Sep 15 17:41 Pictures drwxrwxrwx+ 1 502 dialout 170 Dec 18 18:56 Videos root@NAS:/mnt/datadisk# du -sh 13T .
root@NAS:/mnt/datadisk# df -h Filesystem Size Used Avail Use% Mounted on udev 10M 4.0K 10M 1% /dev /dev/md0 4.0G 555M 3.1G 15% / tmpfs 998M 0 998M 0% /dev/shm tmpfs 998M 3.7M 994M 1% /run tmpfs 499M 1.1M 498M 1% /run/lock tmpfs 998M 0 998M 0% /sys/fs/cgroup /dev/md127 14T 13T 701G 95% /mnt/datadisk

 

Edit:

 

I've unmounted the partition and tried to mount it normally again and that worked like a charm:

 

 

root@NAS:/mnt# mount -t btrfs /dev/md127 /mnt/datadisk/
root@NAS:/mnt# cd datadisk/
root@NAS:/mnt/datadisk# ls -l
total 136
drwxr-xr-x  1 root  root       230 Nov 23 18:09 .
drwxr-xr-x  1 root  root        16 Dec 20 18:32 ..
drwxrwxrwx+ 1 admin admin     1352 Nov 17 00:56 Backup
drwxrwxrwx+ 1 guest guest       20 Aug 20 15:11 Documents
drwxr-xr-x  1 admin admin       94 Aug 20 15:07 home
drwxrwxrwx+ 1 guest guest       20 Aug 20 15:11 Music
drwxrwxrwx+ 1 guest guest       38 Sep 15 17:41 Pictures
drwxrwxrwx+ 1   502 dialout    170 Dec 18 18:56 Videos

root@NAS:/mnt/datadisk# df -h Filesystem Size Used Avail Use% Mounted on udev 10M 4.0K 10M 1% /dev /dev/md0 4.0G 555M 3.1G 15% / tmpfs 998M 0 998M 0% /dev/shm tmpfs 998M 3.1M 995M 1% /run tmpfs 499M 1.1M 498M 1% /run/lock tmpfs 998M 0 998M 0% /sys/fs/cgroup /dev/md127 14T 13

 

 

 

Now i'm contemplating on what to do next... I might just do a reboot and hope for the best (as it SHOULD come up again, because there is now nothing that should prevent the mount action from failing). But then again... I have this strange feeling that it might not come up correct again 🙂

 

Message 2 of 5
MaxxMark
Luminary

Re: OS6 stuck at booting / mounting btrfs

Another interesting thing is that I noticed the system started mailing me multiple mails that the data partition went from "inactive to redundant"  and from "redundant to inactive" a few times after eachother. This started right after I mounted the partition manually (screenshot attached).

 

 

Message 3 of 5
MaxxMark
Luminary

Re: OS6 stuck at booting / mounting btrfs

Decided to give it a reboot. I couldn't think of anything that I could do (next to doing a full backup from the recoverymount).

 

Choose shutdown from the webinterface, it rebooted, and came up again as it should!

 

So for future reference, you could do the same as I described above. Don't just assume all was lost, but check if it is actually gone. 

 

As a warning I should say that the btrfsck output of mine didn't state that there were any problems (except for quota issues). There seemed to be no integrity errors or data errors. If there were, I would not have been comfortable to reboot the unit.

Message 4 of 5
StephenB
Guru

Re: OS6 stuck at booting / mounting btrfs

Thanks for sharing this.

Message 5 of 5
Top Contributors
Discussion stats
  • 4 replies
  • 1506 views
  • 2 kudos
  • 2 in conversation
Announcements