ReadyNAS Pro 6 won't boot after latest firmware upgrade
Hello loyal ReadyNAS community!
I need your help with getting my NAS back to a responsive state.
Yesterday, I upgraded one of my ReadyNAS Pro 6 to the latest firmware via the /admin UI (OS 6.10.2). Upon a successful download via the admin UI, it attempted to install and reboot. However, it refuses to complete a reboot.
As of this moment, dispite how many times I attempt to force restart the machine, It always hangs at 95%, 97%, or 99%. If I press the power button on the chassis once, it changes the LED display to complain about "nmb.service".
Unfortunately, dispite being a decade+ long user of many ReadyNASes, I should have known better to foolishly upgraded without documenting my current version! My bad. We can assume that the previous OS was ~18 months old.
What I've attempted:
1/ Several attempts at force shutdown and power on via chassis power button. This always results in the hang at 95 -> 99% during "Booting..." 2/ SSH into the machine (SSH root@__ip__) which successfully connects. (This gives me hope of a solution!)
@mdgm you've been so kind to help me in the past with previous generations of hardware. Do you have some recommended next steps?
I'm very thankful for any attention you, or other knowledgeable users can offer. I didn't know if I must travel the USB recovery route since it is responsive over SSH.
Re: ReadyNAS Pro 6 won't boot after latest firmware upgrade
Since my initial post, I've looked to see if the 4GB Root volume is full. Here is the results of that query. To ensure others can follow along, here are the steps I performed:
1/ Login via SSH as Root 2/ CD to /etc/default 3/ I ran df -h Here are the results:
Finally, I took at look at what apps I've installed via this command ls -la /apps
total 12
drwxrwxr-x 1 root root 310 Dec 7 22:18 .
drwxr-xr-x 1 root root 278 Dec 7 22:18 ..
drwxr-xr-x 1 root root 52 Nov 22 2013 APPGENIE
drwxr-xr-x 1 root root 0 Jun 20 2017 .backup
drwxr-xr-x 1 admin admin 50 May 8 2017 couchpotato
dr-xr-xr-x 1 root root 0 Nov 20 2013 DO_NOT_DELETE
-rw-r--r-- 1 www-data www-data 29 Aug 27 15:00 .eolapps
-rw-r--r-- 1 www-data www-data 55 Aug 27 15:00 .featuredapps
drwxr-xr-x 1 media root 0 May 29 2016 .forked-daapd
drwxr-xr-x 1 admin admin 1276 Dec 7 22:18 .freeapps
-rw-r--r-- 1 root root 0 Jan 12 2015 .localapp
drwxr-xr-x 1 admin admin 324 Sep 5 2018 plexmediaserver
drwxr-xr-x 1 admin admin 100 Nov 23 2013 pythonr6
drwxr-xr-t 1 root root 34 Nov 24 2013 .readydlna
drwxr-xr-x 1 admin admin 348 May 30 2016 sabnzbd
drwxr-xr-x 1 admin admin 172 Jul 19 2014 shellinabox
drwxr-xr-x 1 admin root 492 Mar 1 2015 sickbeard
drwxr-xr-x 1 root root 12 Jun 8 2017 .xdg
Please let me know if you have any advice on where I can prune some data. Although it's possible that I've filled the system drive in a way I don't understand, it appears as though I'm only 52% utilized.
Re: ReadyNAS Pro 6 won't boot after latest firmware upgrade
Space is fully allocated, but not necessarily all used.
Try entering
# btrfs balance start -dlimit=3 / # btrfs fi show /
and see if that frees up some of the allocated space. If it does, then try again w/o the -dlimit=3. You will get a warning that it will take a long time - but the file system is small, and it really shouldn't.
# btrfs balance start /
# btrfs fi show /
Note that /apps isn't actually in the root partition - it is just a mount point for /data/.apps. There are quite a few other mount points in / that can make looking for space confusing. If you want to see just the root partition, you can remount it
# mount --bind / /mnt
and then search /mnt.
When done you can undo the mount with
# cd / # umount /mnt
The cd matters, as the umount will fail if you are in a folder of /mnt.
Re: ReadyNAS Pro 6 won't boot after latest firmware upgrade
One step forward... one step back.
I performed all that you recommended above. It concluded with a rebalancing. Upon the conclusion, I rebooted the NAS expecting some progress, however. None was made, in fact now I'm at 80% booting.
I tried to re-run the btrfs balance start -dlimit=3 / to see what remained but now I get this:
ERROR: error during balancing '/': No space left on device
There may be more info in syslog - try dmesg | tail
I tried dmesg | tail and received:
[11370.177179] e1000e: eth0 NIC Link is Down
[11375.055247] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[11382.113552] e1000e: eth0 NIC Link is Down
[11385.052561] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[11389.847788] e1000e: eth0 NIC Link is Down
[11392.729800] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[25036.270432] BTRFS info (device md0): relocating block group 5564071936 flags data
[25038.825514] BTRFS info (device md0): found 1123 extents
[25040.382697] BTRFS info (device md0): found 1123 extents
[25040.540836] BTRFS info (device md0): 2 enospc errors during balance
I ran btfs fi show / again, and this time it reports:
Label: '<redacted>:root' uuid: <redacted>
Total devices 1 FS bytes used 1.70GiB
devid 1 size 4.00GiB used 2.34GiB path /dev/md0
Label: '<redacted>:root' uuid: <redacted>
Total devices 1 FS bytes used 1.70GiB
devid 1 size 4.00GiB used 2.34GiB path /dev/md0
Advice?
Well, the first balance did do its job, as the allocated space dropped from 100% to 58%.
But it looks like there was some collateral damage while the file system was completely allocated.
I think the best approach is do a factory default, reconfigure the NAS, and restore the files from the backup - painful, but certain to fully fix it. Do you have a backup of the files?
Re: ReadyNAS Pro 6 won't boot after latest firmware upgrade
Can you help clarify what you mean by "backup of files?". My understanding of the NAS file system is as follows:
A/ System drive: Used for the ReadyNAS / Linux to operate. I do not have a backup of this.
B/ File drives: 6 bays filled with business files stored in xraid format. Each drive is 3 terabytes and given the sheer size of a fully populated NAS I also do not have a backup of these.
Knowing this, what is the solution? I feel like I'm hearing: 1/ Pull all 6 drives
2/ Insert USB drive with new firmware "return to factory defaults"
Re: ReadyNAS Pro 6 won't boot after latest firmware upgrade
You cannot factory default without the drives, the factory default occurs on the drives, and wipes out all data.
By backup, he means just that, make another copy somewhere; probably on a USB drive. Big USB drives are cheap these days, way cheaper than data recovery, should your NAS fail and your files are damaged in the process. if they are business files, as you say, then you should aleady be making one periodically. RAID can help keep you from losing files, but it's main purpose is to allow continued operation through a drive failure. It is not backup.
The process is:
Backup on another device (including the NAS configuration if it's complex to re-create)..
Factory Default.
Put everything back on the NAS
Retain and periodically update the backup for the next time (which you hope will not happen, but you should prepare for).
Old IT saying: If you only have one copy of it, you must not think it's important.
Re: ReadyNAS Pro 6 won't boot after latest firmware upgrade
One thing to remember about btrfs and balancing, especially if you're very near the edge of "full". If you try to balance blocks bigger than btrfs can find space to shuffle them then the balance will fail with "no space." BUT, if you take the approach of starting with 0% as the block usage limit and slowly ramp it up, you may well be able to accumulate enough space to allow for more complete balancing.
MAX=70
i=0
while [ $i -le $MAX ]; do
btrfs balance start -dusage=$i -musage=$i /
i=$(( i + 5 ))
done
I've not yet seen a situation where this approach doesn't eventually work, as long as the filesystem isn't actually consuming all the space. And I've seen it happen so many times that fragged allocations cause package update problems (esp where apt-get update happens), that I take the step fo doing a balance EVERY time before a package install/update and absolutely before applying new OS code.
