Orbi WiFi 7 RBE973
Reply

Readynas 104 not booting up properly

zz9pa
Aspirant

Readynas 104 not booting up properly

Hi,

got an issue with my readynas shortly after the update to 6.2.5.

It doesnt seem to want to boot up properly. It seems to hang during "booting...."  with the hard drive going (i left it over night in case it was running some fsck) and the power light flashing slowly.

 

I can ssh in - which is comforting.

I reboot it  (shutdown -r ) - and it will start up and allow web logins - but I end up getting a "readonly filesystem" and cant save anything. Rebooting again - and its back to locking up with the flashing power light

 

 

Any ideas ? 

 

 

root@ReadyNAS:/# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 08:53 ? 00:00:03 /sbin/init
root 2 0 0 08:53 ? 00:00:00 [kthreadd]
root 3 2 0 08:53 ? 00:00:00 [ksoftirqd/0]
root 4 2 0 08:53 ? 00:00:00 [kworker/0:0]
root 6 2 0 08:53 ? 00:00:03 [rcu_kthread]
root 7 2 0 08:53 ? 00:00:00 [watchdog/0]
root 8 2 0 08:53 ? 00:00:00 [khelper]
root 9 2 0 08:53 ? 00:00:00 [kworker/u:1]
root 182 2 0 08:53 ? 00:00:00 [sync_supers]
root 184 2 0 08:53 ? 00:00:00 [bdi-default]
root 186 2 0 08:53 ? 00:00:00 [kblockd]
root 192 2 0 08:53 ? 00:00:00 [ata_sff]
root 199 2 0 08:53 ? 00:00:00 [khubd]
root 204 2 0 08:53 ? 00:00:00 [md]
root 222 2 0 08:53 ? 00:00:00 [rpciod]
root 235 2 0 08:53 ? 00:00:02 [kswapd0]
root 236 2 0 08:53 ? 00:00:00 [fsnotify_mark]
root 237 2 0 08:53 ? 00:00:00 [nfsiod]
root 240 2 0 08:53 ? 00:00:00 [xfsalloc]
root 241 2 0 08:53 ? 00:00:00 [xfs_mru_cache]
root 242 2 0 08:53 ? 00:00:00 [xfslogd]
root 244 2 0 08:53 ? 00:00:00 [mdcsrepair]
root 245 2 0 08:53 ? 00:00:00 [crypto]
root 261 2 0 08:53 ? 00:00:00 [crypto]
root 262 2 0 08:53 ? 00:00:00 [crypto_ret]
root 340 2 0 08:53 ? 00:00:00 [scsi_eh_0]
root 343 2 0 08:53 ? 00:00:00 [scsi_eh_1]
root 346 2 0 08:53 ? 00:00:00 [scsi_eh_2]
root 349 2 0 08:53 ? 00:00:00 [scsi_eh_3]
root 355 2 0 08:53 ? 00:00:00 [kworker/u:5]
root 366 2 0 08:53 ? 00:00:00 [scsi_eh_4]
root 369 2 0 08:53 ? 00:00:00 [scsi_eh_5]
root 375 2 0 08:53 ? 00:00:00 [target_completi]
root 376 2 0 08:53 ? 00:00:00 [tmr]
root 377 2 0 08:53 ? 00:00:00 [iscsi_ttx]
root 378 2 0 08:53 ? 00:00:00 [iscsi_trx]
root 379 2 0 08:53 ? 00:00:00 [iscsi_ttx]
root 380 2 0 08:53 ? 00:00:00 [iscsi_trx]
root 381 2 0 08:53 ? 00:00:00 [iscsi_ttx]
root 382 2 0 08:53 ? 00:00:00 [iscsi_trx]
root 383 2 0 08:53 ? 00:00:00 [iscsi_ttx]
root 384 2 0 08:53 ? 00:00:00 [iscsi_trx]
root 392 2 0 08:53 ? 00:00:00 [mtdblock0]
root 397 2 0 08:53 ? 00:00:00 [mtdblock1]
root 402 2 0 08:53 ? 00:00:00 [mtdblock2]
root 407 2 0 08:53 ? 00:00:00 [mtdblock3]
root 412 2 0 08:53 ? 00:00:00 [mtdblock4]
root 468 2 0 08:53 ? 00:00:00 [mv_crypto]
root 478 2 0 08:53 ? 00:00:00 [kpca9554d]
root 604 2 0 08:53 ? 00:00:00 [ubi_bgt0d]
root 636 2 0 08:53 ? 00:00:00 [md0_raid1]
root 645 2 0 08:53 ? 00:00:00 [md1_raid6]
root 667 2 0 08:53 ? 00:00:00 [jbd2/md0-8]
root 668 2 0 08:53 ? 00:00:00 [ext4-dio-unwrit]
root 701 1 0 08:53 ? 00:00:04 /lib/systemd/systemd-journald
root 703 1 0 08:53 ? 00:00:00 /sbin/udevd
root 728 2 1 08:53 ? 00:00:14 [md127_raid5]
root 749 2 0 08:53 ? 00:00:00 [md126_raid5]
root 766 2 0 08:53 ? 00:00:00 [md125_raid1]
root 798 2 0 08:53 ? 00:00:00 [btrfs-genwork-1]
root 800 2 0 08:53 ? 00:00:02 [btrfs-submit-1]
root 802 2 0 08:53 ? 00:00:00 [btrfs-delalloc-]
root 804 2 0 08:53 ? 00:00:00 [btrfs-fixup-1]
root 806 2 0 08:53 ? 00:00:00 [btrfs-endio-1]
root 810 2 0 08:53 ? 00:00:00 [btrfs-endio-met]
root 812 2 0 08:53 ? 00:00:00 [btrfs-endio-wri]
root 815 2 0 08:53 ? 00:00:00 [btrfs-freespace]
root 817 2 0 08:53 ? 00:00:00 [btrfs-delayed-m]
root 819 2 0 08:53 ? 00:00:00 [btrfs-cache-1]
root 821 2 0 08:53 ? 00:00:00 [btrfs-readahead]
root 823 2 0 08:53 ? 00:00:00 [btrfs-flush_del]
root 825 2 0 08:53 ? 00:00:00 [btrfs-qgroup-re]
root 1094 703 0 08:53 ? 00:00:00 /sbin/udevd
root 1114 703 0 08:53 ? 00:00:00 /sbin/udevd
root 1204 2 0 08:53 ? 00:00:00 [flush-9:0]
root 1222 2 0 08:54 ? 00:00:02 [btrfs-cleaner]
root 1223 2 2 08:54 ? 00:00:29 [btrfs-transacti]
root 1224 2 0 08:54 ? 00:00:00 [flush-btrfs-1]
root 1226 2 0 08:54 ? 00:00:02 [btrfs-worker-3]
root 1234 1 0 08:54 ? 00:00:00 /usr/sbin/mdcsrepaird -v
root 1235 1 0 08:54 ? 00:00:00 /sbin/mdadm -F -s -y -p /frontview/bin/md_event
avahi 1239 1 0 08:54 ? 00:00:00 avahi-daemon: running [ReadyNAS.local]
root 1242 1 0 08:54 ? 00:00:00 /usr/sbin/sshd -D
root 1244 1 0 08:54 ? 00:00:00 /usr/sbin/cron -f
root 1247 1 0 08:54 ? 00:00:00 /usr/sbin/connmand -n
avahi 1248 1239 0 08:54 ? 00:00:00 avahi-daemon: chroot helper
root 1249 1 0 08:54 ? 00:00:00 /usr/sbin/raidard -S
root 1251 1 0 08:54 ? 00:00:00 /usr/sbin/minidlnad -S
root 1257 1 0 08:54 ? 00:00:00 /lib/systemd/systemd-logind
83 1260 1 0 08:54 ? 00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root 1264 1 0 08:54 ttyS0 00:00:00 /sbin/agetty -s ttyS0 115200 38400 9600
root 1281 1 0 08:54 ? 00:00:00 /usr/sbin/netatalk
root 1296 1281 1 08:54 ? 00:00:15 /usr/sbin/afpd -d -F /etc/netatalk/afp.conf
root 1297 1281 0 08:54 ? 00:00:00 /usr/sbin/cnid_metad -d -F /etc/netatalk/afp.conf
root 1321 1 0 08:54 ? 00:00:00 /usr/sbin/nmbd
root 1440 1 0 08:54 ? 00:00:00 /usr/sbin/smbd
root 1758 1 0 08:54 ? 00:00:00 /usr/sbin/apache2 -k start
root 1760 1758 0 08:54 ? 00:00:00 apache_log
root 1790 1758 13 08:54 ? 00:02:35 /usr/sbin/apache2 -k start
admin 1793 1758 0 08:54 ? 00:00:01 /usr/sbin/apache2 -k start
admin 1794 1758 0 08:54 ? 00:00:01 /usr/sbin/apache2 -k start
admin 1797 1758 0 08:54 ? 00:00:01 /usr/sbin/apache2 -k start
admin 1798 1758 0 08:54 ? 00:00:01 /usr/sbin/apache2 -k start
admin 1799 1758 0 08:54 ? 00:00:01 /usr/sbin/apache2 -k start
admin 1838 1758 0 08:54 ? 00:00:01 /usr/sbin/apache2 -k start
admin 1862 1758 0 08:54 ? 00:00:01 /usr/sbin/apache2 -k start
admin 1863 1758 0 08:54 ? 00:00:01 /usr/sbin/apache2 -k start
root 1933 1 0 08:54 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe
mysql 2287 1933 0 08:54 ? 00:00:02 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --pid-file=/var/run/mysqld/mysqld
root 2288 1933 0 08:54 ? 00:00:00 logger -t mysqld -p daemon.error
root 2348 2 1 08:54 ? 00:00:20 [btrfs-endio-met]
root 2542 1242 0 08:55 ? 00:00:00 sshd: root@pts/0
root 2552 2542 0 08:55 pts/0 00:00:00 -bash
root 2872 2 0 08:59 ? 00:00:00 [kworker/0:1]
root 2966 2552 0 09:01 pts/0 00:00:00 rm m1
admin 3038 1758 0 09:02 ? 00:00:00 /usr/sbin/apache2 -k start
root 3223 1 0 09:05 ? 00:00:00 /frontview/bin/fvbackup-q
root 3224 1 0 09:05 ? 00:00:00 /usr/sbin/readynasd -v 3 -t
root 3225 1 0 09:05 ? 00:00:00 /usr/sbin/minissdpd
daemon 3226 1 0 09:05 ? 00:00:00 /usr/sbin/upnphttpd
root 3232 3224 0 09:05 ? 00:00:00 /bin/sh -e /run/proc_readynasd.uPSh5E
root 3324 1242 0 09:06 ? 00:00:00 sshd: root@pts/1
root 3337 3324 0 09:07 pts/1 00:00:00 -bash
root 3448 2 0 09:08 ? 00:00:00 [kworker/0:2]
root 3743 3337 0 09:13 pts/1 00:00:00 ps -ef

 

 

 

root@ReadyNAS:/var/log# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
rootfs 4173504 842984 3121012 22% /
tmpfs 10240 4 10236 1% /dev
/dev/md0 4173504 842984 3121012 22% /
tmpfs 253640 0 253640 0% /dev/shm
tmpfs 253640 536 253104 1% /run
tmpfs 253640 0 253640 0% /sys/fs/cgroup
tmpfs 253640 0 253640 0% /media
/dev/md127 6822351104 5609475816 1211342040 83% /data
/dev/md127 6822351104 5609475816 1211342040 83% /home
/dev/md127 6822351104 5609475816 1211342040 83% /apps

 

 

If that helps

 

 

Message 1 of 16
mdgm-ntgr
NETGEAR Employee Retired

Re: Readynas 104 not booting up properly

Can you send me your logs (see the Sending Logs link in my sig)?

Message 2 of 16
zz9pa
Aspirant

Re: Readynas 104 not booting up properly

Sent...

The logs include an attached USB_HDD_1 - where I'm trying to copy off some data to - "just in case".

The problem existed before I plugged that in.

Message 3 of 16
mdgm-ntgr
NETGEAR Employee Retired

Re: Readynas 104 not booting up properly

Backing up your data is a good idea.

 

The space on the volume is fully allocated and the metadata is overfull.

Message 4 of 16
zz9pa
Aspirant

Re: Readynas 104 not booting up properly

df says its only 83% full ? 

 

So - what do I need to do to fix it ? 

Message 5 of 16
StephenB
Guru

Re: Readynas 104 not booting up properly

df s not reliable with btrfs.  What does "btrfs fi df" say?

 

https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_bu...

Note you don't need sudo on the NAS.

Message 6 of 16
zz9pa
Aspirant

Re: Readynas 104 not booting up properly

# btrfs fi df /data
Data, single: total=6.33TiB, used=5.20TiB
System, RAID1: total=32.00MiB, used=768.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, RAID1: total=9.00GiB, used=8.69GiB
Metadata, DUP: total=3.00GiB, used=2.61GiB

 

 

I cant delete anything anyway - if I try to do anything to create/modify, or delete files - i just get :

 

# touch somefile
touch: cannot touch `somefile': Read-only file system

 

 

etc

Message 7 of 16
StephenB
Guru

Re: Readynas 104 not booting up properly


@zz9pa wrote:

 

# touch somefile
touch: cannot touch `somefile': Read-only file system

 


 

Do the Web UI (or the logs) say why the file system was mounted read-only?
Message 8 of 16
zz9pa
Aspirant

Re: Readynas 104 not booting up properly

I cant get into the WebUI. I cant see anything in the logs about "readonly" 

 

root@ReadyNAS:/var/log# find . -type f -exec grep -y read {} /dev/null \; | grep -y only

./dbbroker.log: Pipe::readOnly()

 

(including the /dev/null is a kludge to get it to print the filename and the line)

 

Is there anywhere in particular I should look ? 

Message 9 of 16
StephenB
Guru

Re: Readynas 104 not booting up properly

I'd download the logs, and look in both system.log and kernel.log

Message 10 of 16
zz9pa
Aspirant

Re: Readynas 104 not booting up properly

Ok - so in the kernel.log :

Aug 25 09:05:37 ReadyNAS kernel: BTRFS error (device md125) in __btrfs_free_extent:5648: errno=-28 No space left
Aug 25 09:05:37 ReadyNAS kernel: BTRFS info (device md125): forced readonly
Aug 25 09:05:37 ReadyNAS kernel: BTRFS debug (device md125): run_one_delayed_ref returned -28
Aug 25 09:05:37 ReadyNAS kernel: BTRFS error (device md125) in btrfs_run_delayed_refs:2688: errno=-28 No space left
Aug 25 09:05:37 ReadyNAS kernel: BTRFS error (device md125) in __btrfs_free_extent:5648: errno=-28 No space left
Aug 25 09:05:37 ReadyNAS kernel: BTRFS debug (device md125): run_one_delayed_ref returned -28
Aug 25 09:05:37 ReadyNAS kernel: BTRFS error (device md125) in btrfs_run_delayed_refs:2688: errno=-28 No space left
Aug 25 09:21:55 ReadyNAS kernel: usb 1-1: new high-speed USB device number 2 using ehci_marvell

 

 

even though there is space.

Also - mounting read-only does kind of limit the options to deleting stuff ? !

 

 

How do I fix this?

(And how the hell does it happen in the first place?)

Message 11 of 16
StephenB
Guru

Re: Readynas 104 not booting up properly

The usual cause is inappropriate snapshot settings.  If you enable them on folders that have a lot of churn they tie up a lot of space quickly.

 

At this point I suggest copying of the data and doing a factory reset.  Then restoring data from backup.

Message 12 of 16
zz9pa
Aspirant

Re: Readynas 104 not booting up properly

Seriously - this *is* by backup of the data!

The whole point of a unit like this is to backup stuff !

 

I've tried to delete stuff - but then it goes readonly again - and when I reboot - the files I deleted are back.

(I tried a "sync" - didnt seem to make any difference)

 

There must be something better than wipe and restart ? 

And - if I do wipe and restart - whats to stop it just doing it again in a few months???

 

Message 13 of 16
StephenB
Guru

Re: Readynas 104 not booting up properly


@zz9pa wrote:

Seriously - this *is* by backup of the data!

The whole point of a unit like this is to backup stuff !


 

Well, if it is your backup, then you can do the factory reset and rebuild it w/o data loss.

If you can't do that w/o data loss, then as far as I'm concerned its not your backup.


@zz9pa wrote:

I've tried to delete stuff - but then it goes readonly again - and when I reboot - the files I deleted are back.

(I tried a "sync" - didnt seem to make any difference)

 

There must be something better than wipe and restart ? 

And - if I do wipe and restart - whats to stop it just doing it again in a few months???

 

Something is clearly messed up with the file system.  It's possible that Netgear can go in and make some adjustments to fix it.  But in my opinion doing the factory default on the current firmware is a more certain way to ensure its fully taken care of.

 

As far as preventing it in the future - that depends on exactly what happened.  I haven't seen any issues with BTRFS for the 2 1/2 years I've been running it.  Though my volumes don't get dangerously full, and I keep an eye on snapshot usage.  

 

Do you have snapshots turned on?  If so, are you running torrents or other applications that create a lot of churn?

 

Message 14 of 16
mdgm-ntgr
NETGEAR Employee Retired

Re: Readynas 104 not booting up properly

I mentioned the problem in an earlier post but did not elaborate as it was getting quite late in the evening. The metadata is overfull.

The space on the volume is fully allocated so no further space can be allocated to metadata.


Doing a balance from the UI can help but as the metadata is overfull I don't think running a balance would work.  If you want to try that backup your data first.

Your data volume has three layers and some of your disks are not on the compatibility list.

 

You could contact support and see if you can purchase assistance to attempt to fix this without a factory default but considering the complexity of this situation and your use of disks outside the compatibility list they may advise you to backup and factory reset the unit anyway.

You can schedule running a balance every few weeks or so (remember to schedule it when you are not actively using the NAS) in addition to filling your volume no more than 80-85% and that should help avoid running into a problem like this.


A balance moves data and metadata around so that chunks are emptied and then returned to unallocated space. If a balance finds a chunk that is already empty it will also return it to unallocated space.

 

metadata is data about data. BTRFS allocates space for system (not much is needed for this), data and metadata.

Sometimes if you say add a lot of large files and the volume becomes near fully allocated and then delete them and then add lots of small files too much has been allocated to data and there isn't enough free space to allocate more to metadata or vice versa.

We have changes coming which should alleviate this problem somewhat, however scheduling a regular balance is still recommended.

Message 15 of 16
zz9pa
Aspirant

Re: Readynas 104 not booting up properly

Ok - so what I've learned is : 

 

1) Its ok to run without any backup for a day or two - while you backup the backup, then reformat and restore the backup to the backup - nothing bad can happen during this time.

2) Its ok for the unit to lock up if you fill it more than 80-85% full, and the only way to fix it (without doing a 2nd backup) - is to pay for support to dial in...

 

 

Is that about right ?

Message 16 of 16
Top Contributors
Discussion stats
  • 15 replies
  • 3748 views
  • 0 kudos
  • 3 in conversation
Announcements