- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
ReadyNAS Pro 6 won't boot after latest firmware upgrade
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
ReadyNAS Pro 6 won't boot after latest firmware upgrade
Hello loyal ReadyNAS community!
I need your help with getting my NAS back to a responsive state.
Yesterday, I upgraded one of my ReadyNAS Pro 6 to the latest firmware via the /admin UI (OS 6.10.2). Upon a successful download via the admin UI, it attempted to install and reboot. However, it refuses to complete a reboot.
As of this moment, dispite how many times I attempt to force restart the machine, It always hangs at 95%, 97%, or 99%. If I press the power button on the chassis once, it changes the LED display to complain about "nmb.service".
Unfortunately, dispite being a decade+ long user of many ReadyNASes, I should have known better to foolishly upgraded without documenting my current version! My bad. We can assume that the previous OS was ~18 months old.
What I've attempted:
1/ Several attempts at force shutdown and power on via chassis power button. This always results in the hang at 95 -> 99% during "Booting..."
2/ SSH into the machine (SSH root@__ip__) which successfully connects. (This gives me hope of a solution!)
@mdgm you've been so kind to help me in the past with previous generations of hardware. Do you have some recommended next steps?
I'm very thankful for any attention you, or other knowledgeable users can offer. I didn't know if I must travel the USB recovery route since it is responsive over SSH.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: ReadyNAS Pro 6 won't boot after latest firmware upgrade
Since my initial post, I've looked to see if the 4GB Root volume is full. Here is the results of that query.
To ensure others can follow along, here are the steps I performed:
1/ Login via SSH as Root
2/ CD to /etc/default
3/ I ran df -h
Here are the results:
Filesystem Size Used Avail Use% Mounted on udev 10M 4.0K 10M 1% /dev /dev/md0 4.0G 1.8G 1.7G 52% / tmpfs 1.9G 8.0K 1.9G 1% /dev/shm tmpfs 1.9G 600K 1.9G 1% /run tmpfs 960M 16K 960M 1% /run/lock tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/md127 14T 12T 1.4T 90% /data /dev/md127 14T 12T 1.4T 90% /apps /dev/md127 14T 12T 1.4T 90% /home
I then ran:
lsof | grep deleted it ran for a bit, but didn't report anything to the console. This seemed strange to me.
Then I ran:
btrf fi show
which returned:
Label: '<redacted>:root' uuid: <redacted> Total devices 1 FS bytes used 1.69GiB devid 1 size 4.00GiB used 4.00GiB path /dev/md0 Label: '<redacted>:data' uuid: <redacted> Total devices 1 FS bytes used 11.86TiB devid 1 size 13.62TiB used 13.38TiB path /dev/md127
Then I ran:
btrfs fi df /
which returned:
Data, single: total=3.32GiB, used=1.67GiB System, DUP: total=8.00MiB, used=4.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=332.69MiB, used=22.46MiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=16.00MiB, used=0.00B
I then ran:
cat /var/log/frontview/initrd.log
Which returned:
[2013/11/20 21:14:32] Factory default initiated by button! [2013/11/20 21:14:47] Defaulting to X-RAID2 mode, RAID level 5 [2013/11/20 21:14:52] Factory default initiated on ReadyNASOS 6.0.4. [2013/11/22 21:20:27] Updated from ReadyNASOS 6.0.4 to 6.1.4. [2014/07/13 19:54:55] Updated from ReadyNASOS 6.1.4 (1382402794) to 6.1.8 (1398980083). [2015/01/12 00:58:31] Updated from ReadyNASOS 6.1.8 () to 6.2.2 (ReadyNASOS). [2015/07/09 22:30:43] Updated from ReadyNASOS 6.2.2 (ReadyNASOS) to 6.2.4 (ReadyNASOS). [2016/05/30 00:10:34] Updated from ReadyNASOS 6.2.4 (ReadyNASOS) to 6.5.0 (ReadyNASOS). [2017/01/16 12:20:30] Updated from ReadyNASOS 6.5.0 (ReadyNASOS) to 6.6.1 (ReadyNASOS). [2017/06/08 06:32:29 UTC] Updated from ReadyNASOS 6.6.1 (ReadyNASOS) to 6.7.4 (ReadyNASOS). [2017/10/13 16:14:23 UTC] Updated from ReadyNASOS 6.7.4 (ReadyNASOS) to 6.8.1 (ReadyNASOS). [2018/08/29 01:50:56 UTC] Updated from ReadyNASOS 6.8.1 (ReadyNASOS) to 6.9.3 (ReadyNASOS). [2019/01/07 17:01:59 UTC] Updated from ReadyNASOS 6.9.3 (ReadyNASOS) to 6.9.4 (ReadyNASOS). [2019/01/29 15:44:06 UTC] Updated from ReadyNASOS 6.9.4 (ReadyNASOS) to 6.9.5 (ReadyNASOS). [2019/12/08 05:13:50 UTC] Updated from ReadyNASOS 6.9.5 (ReadyNASOS) to 6.10.2 (ReadyNASOS).
(so apparently I was on 6.9.5 last time :))
When I ran du -csh /var/* it returned
0 /var/agentx 104M /var/backups 65M /var/cache 3.0M /var/cores 0 /var/ftp 514M /var/lib 0 /var/local 4.0K /var/lock 65M /var/log 0 /var/mail 8.0K /var/netatalk 0 /var/opt 8.0K /var/readydrop 18M /var/readynasd 4.0K /var/run 8.0K /var/spool 0 /var/tmp 4.0K /var/www 767M total
I looked deeper via du -csh /var/log/* and it returned
12K /var/log/alternatives.log 340K /var/log/apache2 296K /var/log/apt 0 /var/log/audit 12K /var/log/btmp 0 /var/log/clamav 4.0K /var/log/dbbroker.log 64K /var/log/dmesg 56K /var/log/dmesg.0 16K /var/log/dmesg.1.gz 16K /var/log/dmesg.2.gz 16K /var/log/dmesg.3.gz 16K /var/log/dmesg.4.gz 680K /var/log/dpkg.log 0 /var/log/faillog 12M /var/log/frontview 0 /var/log/fsck 50M /var/log/journal 12K /var/log/lastlog 4.0K /var/log/LeafP2P.log 56K /var/log/netatalk.log 0 /var/log/news 4.0K /var/log/proftpd.log 4.0K /var/log/readydropd.log 928K /var/log/readynasd 796K /var/log/samba 220K /var/log/wtmp 65M total
I then looked deeper into frontview via du -csh /var/log/frontview/* which returned
148K /var/log/frontview/backup 100K /var/log/frontview/cgi.log 108K /var/log/frontview/clamscan.log 600K /var/log/frontview/error.log 0 /var/log/frontview/http-access.log 804K /var/log/frontview/http-access.log.old 388K /var/log/frontview/http-error.log 4.0K /var/log/frontview/initrd.log 400K /var/log/frontview/msmtp.log 3.3M /var/log/frontview/msmtp.log.old 568K /var/log/frontview/msmtp.queue.log 976K /var/log/frontview/msmtp.queue.log.old 4.0K /var/log/frontview/password_recovery.log 0 /var/log/frontview/ssl_access.log 2.1M /var/log/frontview/ssl_access.log.old 568K /var/log/frontview/status.log 1.5M /var/log/frontview/status.log.old 12M total
Finally, I took at look at what apps I've installed via this command ls -la /apps
total 12 drwxrwxr-x 1 root root 310 Dec 7 22:18 . drwxr-xr-x 1 root root 278 Dec 7 22:18 .. drwxr-xr-x 1 root root 52 Nov 22 2013 APPGENIE drwxr-xr-x 1 root root 0 Jun 20 2017 .backup drwxr-xr-x 1 admin admin 50 May 8 2017 couchpotato dr-xr-xr-x 1 root root 0 Nov 20 2013 DO_NOT_DELETE -rw-r--r-- 1 www-data www-data 29 Aug 27 15:00 .eolapps -rw-r--r-- 1 www-data www-data 55 Aug 27 15:00 .featuredapps drwxr-xr-x 1 media root 0 May 29 2016 .forked-daapd drwxr-xr-x 1 admin admin 1276 Dec 7 22:18 .freeapps -rw-r--r-- 1 root root 0 Jan 12 2015 .localapp drwxr-xr-x 1 admin admin 324 Sep 5 2018 plexmediaserver drwxr-xr-x 1 admin admin 100 Nov 23 2013 pythonr6 drwxr-xr-t 1 root root 34 Nov 24 2013 .readydlna drwxr-xr-x 1 admin admin 348 May 30 2016 sabnzbd drwxr-xr-x 1 admin admin 172 Jul 19 2014 shellinabox drwxr-xr-x 1 admin root 492 Mar 1 2015 sickbeard drwxr-xr-x 1 root root 12 Jun 8 2017 .xdg
Please let me know if you have any advice on where I can prune some data. Although it's possible that I've filled the system drive in a way I don't understand, it appears as though I'm only 52% utilized.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: ReadyNAS Pro 6 won't boot after latest firmware upgrade
Space is fully allocated, but not necessarily all used.
Try entering
# btrfs balance start -dlimit=3 /
# btrfs fi show /
and see if that frees up some of the allocated space. If it does, then try again w/o the -dlimit=3. You will get a warning that it will take a long time - but the file system is small, and it really shouldn't.
# btrfs balance start / # btrfs fi show /
Note that /apps isn't actually in the root partition - it is just a mount point for /data/.apps. There are quite a few other mount points in / that can make looking for space confusing. If you want to see just the root partition, you can remount it
# mount --bind / /mnt
and then search /mnt.
When done you can undo the mount with
# cd /
# umount /mnt
The cd matters, as the umount will fail if you are in a folder of /mnt.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: ReadyNAS Pro 6 won't boot after latest firmware upgrade
One step forward... one step back.
I performed all that you recommended above. It concluded with a rebalancing. Upon the conclusion, I rebooted the NAS expecting some progress, however. None was made, in fact now I'm at 80% booting.
I tried to re-run the btrfs balance start -dlimit=3 / to see what remained but now I get this:
ERROR: error during balancing '/': No space left on device There may be more info in syslog - try dmesg | tail
I tried dmesg | tail and received:
[11370.177179] e1000e: eth0 NIC Link is Down [11375.055247] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None [11382.113552] e1000e: eth0 NIC Link is Down [11385.052561] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None [11389.847788] e1000e: eth0 NIC Link is Down [11392.729800] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None [25036.270432] BTRFS info (device md0): relocating block group 5564071936 flags data [25038.825514] BTRFS info (device md0): found 1123 extents [25040.382697] BTRFS info (device md0): found 1123 extents [25040.540836] BTRFS info (device md0): 2 enospc errors during balance
I ran btfs fi show / again, and this time it reports:
Label: '<redacted>:root' uuid: <redacted> Total devices 1 FS bytes used 1.70GiB devid 1 size 4.00GiB used 2.34GiB path /dev/md0
Advice?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: ReadyNAS Pro 6 won't boot after latest firmware upgrade
@AtariPezman wrote:
Label: '<redacted>:root' uuid: <redacted> Total devices 1 FS bytes used 1.70GiB devid 1 size 4.00GiB used 2.34GiB path /dev/md0Advice?
Well, the first balance did do its job, as the allocated space dropped from 100% to 58%.
But it looks like there was some collateral damage while the file system was completely allocated.
I think the best approach is do a factory default, reconfigure the NAS, and restore the files from the backup - painful, but certain to fully fix it. Do you have a backup of the files?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: ReadyNAS Pro 6 won't boot after latest firmware upgrade
Can you help clarify what you mean by "backup of files?". My understanding of the NAS file system is as follows:
A/ System drive: Used for the ReadyNAS / Linux to operate. I do not have a backup of this.
B/ File drives: 6 bays filled with business files stored in xraid format. Each drive is 3 terabytes and given the sheer size of a fully populated NAS I also do not have a backup of these.
Knowing this, what is the solution? I feel like I'm hearing:
1/ Pull all 6 drives
2/ Insert USB drive with new firmware "return to factory defaults"
3/ Flash the device back to fresh install
4/ Reintroduce the 6 drives
Is this correct?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: ReadyNAS Pro 6 won't boot after latest firmware upgrade
You cannot factory default without the drives, the factory default occurs on the drives, and wipes out all data.
By backup, he means just that, make another copy somewhere; probably on a USB drive. Big USB drives are cheap these days, way cheaper than data recovery, should your NAS fail and your files are damaged in the process. if they are business files, as you say, then you should aleady be making one periodically. RAID can help keep you from losing files, but it's main purpose is to allow continued operation through a drive failure. It is not backup.
The process is:
- Backup on another device (including the NAS configuration if it's complex to re-create)..
- Factory Default.
- Put everything back on the NAS
- Retain and periodically update the backup for the next time (which you hope will not happen, but you should prepare for).
Old IT saying: If you only have one copy of it, you must not think it's important.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: ReadyNAS Pro 6 won't boot after latest firmware upgrade
One thing to remember about btrfs and balancing, especially if you're very near the edge of "full". If you try to balance blocks bigger than btrfs can find space to shuffle them then the balance will fail with "no space." BUT, if you take the approach of starting with 0% as the block usage limit and slowly ramp it up, you may well be able to accumulate enough space to allow for more complete balancing.
MAX=70 i=0 while [ $i -le $MAX ]; do btrfs balance start -dusage=$i -musage=$i / i=$(( i + 5 )) done
I've not yet seen a situation where this approach doesn't eventually work, as long as the filesystem isn't actually consuming all the space. And I've seen it happen so many times that fragged allocations cause package update problems (esp where apt-get update happens), that I take the step fo doing a balance EVERY time before a package install/update and absolutely before applying new OS code.