× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: Ready NAS Ultra 6 Plus - OS 6.9.5 >> 6.10 Upgrade Fail - Safe Mode Options

UltraGar
Aspirant

Ready NAS Ultra 6 Plus - OS 6.9.5 >> 6.10 Upgrade Fail - Safe Mode Options

Hello,

I realize this is a legacy device, and that OS 6 is not officially supported, but I hope someone here can help.

 

I previously updated my legacy Ready NAS Ultra 6 Plus (6x 3TB drives) from OS 4.2.XX to OS 6.9.5 without any problems.  Tonight I was prompted to upgrade the OS to 6.10, without performing a backup (my bad), and after reboot, the NAS:

 

1. Passes the Root FS check;

2. Boots in increments going to 99% and stops;

3. Enters into 6.9.5 safe mode.

 

RAIDar 6.5.0 detects the NAS and shows:

1. Name: readnasos

2. Firmware: 6.9.5

3. Status: Volume data RAID Level 5, Inactive, 100% of the volume used

4. All Drives show a green checkmark.

5. Offline Setup (LAN) is the only option.  No option to Download Logs or perform Diagnostics.

 

Logging into the Admin interface, the message is "ReadyNAS failed to boot successfully" and provides several options include formatting the Volumes, OS Reinstall, and Boot into Read-Only mode.

 

I want to avoid formatting the Volume for obvious reasons.

 

I've performed OS Reinstall several times with no success.  I moved all 6 drives to another Ultra 6 Plus unit (running 6.9.5) and the same problem occurred.  Tried to use NAS #2 to OS Reinstall with no success.  Moved the drives back to NAS #1, where the problem persists.

 

I want to boot into Read Only mode and access the data, but the Boot Menu on this Legacy NAS does not provide the option, at least as a top-level option.

 

I also attempted a USB Recovery using the 6.9.5 x86 firmware, using 3 different USB sticks, but could not get the NAS to enter USB Recovery mode.

 

I hope someone else has run into the same problem as I have and can offer some suggestions.

 

Thanks!

Model: RNDP600U|ReadyNAS Ultra 6 Plus Chassis only
Message 1 of 9
StephenB
Guru

Re: Ready NAS Ultra 6 Plus - OS 6.9.5 >> 6.10 Upgrade Fail - Safe Mode Options

Do you have experience with linux commands?  There are some things you could try in tech support mode.

 

ReclaiMe RAID recovery software is another option, though it is expensive (and requires a way to connect all your disks to a PC).

Message 2 of 9
UltraGar
Aspirant

Re: Ready NAS Ultra 6 Plus - OS 6.9.5 >> 6.10 Upgrade Fail - Safe Mode Options

I don't have a lot of experience with Linux, but am willing to try. Would I be doing steps similar to this to try mounting the volume in Read Only mode?

 

https://community.netgear.com/t5/Using-your-ReadyNAS-in-Business/Disk-volume-no-longer-active-force-...

 

 

Message 3 of 9
StephenB
Guru

Re: Ready NAS Ultra 6 Plus - OS 6.9.5 >> 6.10 Upgrade Fail - Safe Mode Options

 


@UltraGar wrote:

Would I be doing steps similar to this to try mounting the volume in Read Only mode?

 

https://community.netgear.com/t5/Using-your-ReadyNAS-in-Business/Disk-volume-no-longer-active-force-...

 


You could try that, and if those commands work do a backup of the NAS.

 

You'd boot in tech-support mode, and telnet into the NAS.  User/pass is root/infr8ntdebug

Then enter

# rnutil chroot

Which will start RAID and chroot.

 

It might be worth trying to re-apply the firmware update next.  You'd do this by

# touch /.os_update
# reboot

If that fails, you'd have to reboot in tech support mode, and go in again.

 

 

Message 4 of 9
UltraGar
Aspirant

Re: Ready NAS Ultra 6 Plus - OS 6.9.5 >> 6.10 Upgrade Fail - Safe Mode Options


@StephenB wrote:

 


@UltraGar wrote:

Would I be doing steps similar to this to try mounting the volume in Read Only mode?

 

https://community.netgear.com/t5/Using-your-ReadyNAS-in-Business/Disk-volume-no-longer-active-force-...

 


You could try that, and if those commands work do a backup of the NAS.

 

You'd boot in tech-support mode, and telnet into the NAS.  User/pass is root/infr8ntdebug

Then enter

# rnutil chroot

Which will start RAID and chroot.

 

It might be worth trying to re-apply the firmware update next.  You'd do this by

# touch /.os_update
# reboot

If that fails, you'd have to reboot in tech support mode, and go in again.

 

 


Thanks for the valuable information @StephenB!  I was able to enter Tech Support mode and use the login information to perform some tasks.

 

The rnutil chroot command did not work because there's no bash available, however, using the information from the other thread, I found mdadm and btrfs already installed and was able to mount the RAID volume in read-only and it appears intact.  It's reassuring to at least see the files and folders there.

 

It's a little slow going in part to getting acquainted with the Busybox Linux, getting SSH running through Dropbear, and not quite succeeding with WinSCP (which likes bash or ksh) or SFTP yet.  Will also try to transfer files via USB once I figure out how to find and mount it.

 

I took a peek at the Boot volume and noticed that I'm only using about 25% of the avalable space (2.7G free).  Thought maybe it was full and caused a problem on the upgrade.  Once I get the files off, I'll try re-running the upgrade.  At USB 2.0 transfer speeds, this could be a while.

 

As a related question, would I be able to take these OS6-formatted drives and transfer them over to a newer unit, like the RN426 (6 drives in the legacy unit, 6 slots in the new unit), use the boot menu to get into Read-Only mode to access the data that way (I don't have Read-Only option on the Ultra 6 Plus)?  If so, and assuming no corruption or failed disks, would I be able to simply transfer the files via SMB instead?

 

Anyway, thanks again for the help @StephenB!  I'll check in again soon! 

 

Message 5 of 9
StephenB
Guru

Re: Ready NAS Ultra 6 Plus - OS 6.9.5 >> 6.10 Upgrade Fail - Safe Mode Options


@UltraGar wrote:

The rnutil chroot command did not work 

 


It should have - perhaps that's a side effect of the failed OS update?  It is possible to manually mount the OS partition and then do a chroot.

 


@UltraGar wrote:

I took a peek at the Boot volume and noticed that I'm only using about 25% of the avalable space (2.7G free).  Thought maybe it was full and caused a problem on the upgrade. 

 


That is a common issue, so it's good that you ruled it out.

 


@UltraGar wrote:

As a related question, would I be able to take these OS6-formatted drives and transfer them over to a newer unit, like the RN426 (6 drives in the legacy unit, 6 slots in the new unit), use the boot menu to get into Read-Only mode to access the data that way (I don't have Read-Only option on the Ultra 6 Plus)?  If so, and assuming no corruption or failed disks, would I be able to simply transfer the files via SMB instead?

 

 


You should be able to do that (though of course the failed OS update could get in the way there too).  You could then re-purpose the Ultra Plus as a backup NAS.

 

Another thing you could try is doing a fresh factory install to a single (spare) disk.  If that comes up as 6.9.5, you could try updating it to 6.10.  Then power down and reinsert your original disks.

 

 

Message 6 of 9
UltraGar
Aspirant

Re: Ready NAS Ultra 6 Plus - OS 6.9.5 >> 6.10 Upgrade Fail - Safe Mode Options


@StephenB wrote:

@UltraGar wrote:

The rnutil chroot command did not work 

 



It should have - perhaps that's a side effect of the failed OS update?  It is possible to manually mount the OS partition and then do a chroot.

As I Telnet or SSH in, I get into Busybox, which is very limiting.  I input:

 

# rnutil chroot
chroot: can't execute 'bash': No such file or directory

 

I can mount and see the OS partition, but I don't seem to be able to use any of the commands in the OS partition's /bin or /sbin directories.  Busybox doesn't really give me the means to compile or use other programs that are not already included.

 

Another thing you could try is doing a fresh factory install to a single (spare) disk.  If that comes up as 6.9.5, you could try updating it to 6.10.  Then power down and reinsert your original disks.

After recovering my files, I managed to get a functioning NAS with the spare disk, with a successful boot first into 6.9.5, then 6.10.1 after an upgrade.  Operation seems normal enough - can change settings, create shares, upload and access files.  The GUI tells me without a volume the settings will be lost, but they seem to stick.

 

So I reinserted the original disks and performed an OS Re-Install.  Rebooted again and ended right back into Safe Mode in OS 6.10.1. Smiley Frustrated

 

Is there a way I can copy the boot partition off the spare disk onto the 6-disk boot partition and force OS 6.10.1?  I tried accessing the spare partitions with Busybox and in Windows 10 with a WinBtrfs driver and failed.  The NAS and Windows see the physical drive, but I can't mount them; and without mounting, I can't transfer the files anyway.

 

Looking like I might have to bite the bullet and factory reset the 6 disks.  Any other ideas before I hit the big red button?

 

Thanks!

Message 7 of 9
mdgm
Virtuoso

Re: Ready NAS Ultra 6 Plus - OS 6.9.5 >> 6.10 Upgrade Fail - Safe Mode Options

The NAS uses mdadm raid with BTRFS on top.

The tools provided are sufficient to diagnose it and if possible to give access back to data. You just need to know what to use and why/when/how to use it.

A factory reset is a drastic step and if just the root volume is messed up it would be a shame to wipe all your data if your backup is not up to date, when the problem *might* be straightforward to fix.
Message 8 of 9
UltraGar
Aspirant

Re: Ready NAS Ultra 6 Plus - OS 6.9.5 >> 6.10 Upgrade Fail - Safe Mode Options

Noticed these entries in dmesg when I mounted the md0 partition:

[ 3540.419911] BTRFS info (device md0): has skinny extents
[ 3571.847396] BTRFS critical (device md0): corrupt leaf, slot offset bad: block=52969472, root=1, slot=160
[ 3571.881779] BTRFS critical (device md0): corrupt leaf, slot offset bad: block=52969472, root=1, slot=160
[ 3571.881810] BTRFS error (device md0): Error removing orphan entry, stopping orphan cleanup
[ 3571.881814] BTRFS error (device md0): could not do orphan cleanup -22

and this when I mounted md127:

[ 4113.834071] BTRFS info (device md127): has skinny extents
[ 4125.078527] BTRFS error (device md127): qgroup generation mismatch, marked as inconsistent

Performed a btrfs scrub:

scrub status for 
        scrub started at Wed Jul 24 04:24:29 2019 and was aborted after 00:00:00
        total bytes scrubbed: 0.00B with 0 errors

Next, with the boot partition unmounted, performed a read-only btrfs check, receiving hundreds of entries:

root 5 inode 33098 errors 2001, no inode item, link count wrong
        unresolved ref dir 9076 index 1653 namelen 22 name libenca0:amd64.symbols filetype 1 errors 4, no inode ref
root 5 inode 33099 errors 2001, no inode item, link count wrong
        unresolved ref dir 9076 index 1654 namelen 21 name libenca0:amd64.shlibs filetype 1 errors 4, no inode ref
root 5 inode 33100 errors 2001, no inode item, link count wrong
        unresolved ref dir 9076 index 1655 namelen 21 name libenca0:amd64.postrm filetype 1 errors 4, no inode ref
root 5 inode 33101 errors 2001, no inode item, link count wrong
        unresolved ref dir 9076 index 1656 namelen 19 name libenca0:amd64.list filetype 1 errors 4, no inode ref
root 5 inode 33102 errors 2001, no inode item, link count wrong
        unresolved ref dir 9076 index 1657 namelen 23 name libexempi3:amd64.postrm filetype 1 errors 4, no inode ref
root 5 inode 33103 errors 2001, no inode item, link count wrong
        unresolved ref dir 9076 index 1658 namelen 23 name libexempi3:amd64.shlibs filetype 1 errors 4, no inode ref
root 5 inode 33104 errors 2001, no inode item, link count wrong
        unresolved ref dir 9076 index 1659 namelen 21 name libexempi3:amd64.list filetype 1 errors 4, no inode ref
root 5 inode 33105 errors 2001, no inode item, link count wrong
        unresolved ref dir 9076 index 1660 namelen 25 name libstemmer0d:amd64.shlibs filetype 1 errors 4, no inode ref
root 5 inode 33106 errors 2001, no inode item, link count wrong
        unresolved ref dir 9076 index 1661 namelen 25 name libstemmer0d:amd64.postrm filetype 1 errors 4, no inode ref
root 5 inode 33107 errors 2001, no inode item, link count wrong
        unresolved ref dir 9076 index 1662 namelen 26 name libstemmer0d:amd64.symbols filetype 1 errors 4, no inode ref
root 5 inode 33108 errors 2001, no inode item, link count wrong
        unresolved ref dir 9076 index 1663 namelen 23 name libstemmer0d:amd64.list filetype 1 errors 4, no inode ref
root 5 inode 33109 errors 2001, no inode item, link count wrong
        unresolved ref dir 9076 index 1664 namelen 18 name dconf-service.list filetype 1 errors 4, no inode ref
root 5 inode 33110 errors 2001, no inode item, link count wrong
        unresolved ref dir 9076 index 1665 namelen 34 name dconf-gsettings-backend:amd64.list filetype 1 errors 4, no inode ref

...

root 5 inode 68897 errors 2001, no inode item, link count wrong unresolved ref dir 9038 index 19 namelen 39 name freeapp-collection_1559671174_amd64.deb filetype 1 errors 4, no inode ref root 5 inode 68964 errors 2001, no inode item, link count wrong unresolved ref dir 9015 index 89 namelen 16 name dpkg.status.2.gz filetype 1 errors 4, no inode ref root 5 inode 70254 errors 2001, no inode item, link count wrong unresolved ref dir 10223 index 247 namelen 58 name apt.readynas.com_packages_readynasos_dists_6.9.5_InRelease filetype 1 errors 4, no inode ref root 5 inode 70279 errors 2001, no inode item, link count wrong unresolved ref dir 10223 index 248 namelen 78 name apt.readynas.com_packages_readynasos_dists_6.9.5_updates_binary-amd64_Packages filetype 1 errors 4, no inode ref root 5 inode 70287 errors 2001, no inode item, link count wrong unresolved ref dir 9038 index 20 namelen 39 name freeapp-collection_1560294402_amd64.deb filetype 1 errors 4, no inode ref root 5 inode 70288 errors 2001, no inode item, link count wrong unresolved ref dir 10219 index 20 namelen 15 name extended_states filetype 1 errors 4, no inode ref root 5 inode 70292 errors 2001, no inode item, link count wrong unresolved ref dir 9076 index 2376 namelen 27 name freeapp-collection.postinst filetype 1 errors 4, no inode ref root 5 inode 70293 errors 2001, no inode item, link count wrong unresolved ref dir 9076 index 2375 namelen 26 name freeapp-collection.preinst filetype 1 errors 4, no inode ref root 5 inode 70294 errors 2001, no inode item, link count wrong unresolved ref dir 9076 index 2374 namelen 24 name freeapp-collection.prerm filetype 1 errors 4, no inode ref root 5 inode 70299 errors 2001, no inode item, link count wrong unresolved ref dir 9076 index 2373 namelen 23 name freeapp-collection.list filetype 1 errors 4, no inode ref root 5 inode 70300 errors 2001, no inode item, link count wrong unresolved ref dir 9076 index 2378 namelen 26 name freeapp-collection.md5sums filetype 1 errors 4, no inode ref root 5 inode 70309 errors 2001, no inode item, link count wrong unresolved ref dir 9049 index 146 namelen 10 name status-old filetype 1 errors 4, no inode ref root 5 inode 70311 errors 2001, no inode item, link count wrong unresolved ref dir 9049 index 147 namelen 6 name status filetype 1 errors 4, no inode ref root 5 inode 70352 errors 2001, no inode item, link count wrong unresolved ref dir 9015 index 92 namelen 13 name dpkg.status.0 filetype 1 errors 4, no inode ref root 5 inode 70353 errors 2001, no inode item, link count wrong unresolved ref dir 9015 index 91 namelen 16 name dpkg.status.1.gz filetype 1 errors 4, no inode ref root 5 inode 73409 errors 2001, no inode item, link count wrong unresolved ref dir 11299 index 20 namelen 14 name system.journal filetype 1 errors 4, no inode ref root 5 inode 74235 errors 2001, no inode item, link count wrong unresolved ref dir 10223 index 287 namelen 46 name mirrors.kernel.org_debian_dists_jessie_Release filetype 1 errors 4, no inode ref root 5 inode 74236 errors 2001, no inode item, link count wrong unresolved ref dir 10223 index 288 namelen 50 name mirrors.kernel.org_debian_dists_jessie_Release.gpg filetype 1 errors 4, no inode ref root 5 inode 75323 errors 2001, no inode item, link count wrong unresolved ref dir 10223 index 301 namelen 67 name security.debian.org_dists_jessie_updates_main_binary-amd64_Packages filetype 1 errors 4, no inode ref root 5 inode 75375 errors 2001, no inode item, link count wrong unresolved ref dir 2716 index 6689 namelen 15 name remote-fw-index filetype 1 errors 4, no inode ref root 5 inode 75458 errors 2001, no inode item, link count wrong unresolved ref dir 10223 index 302 namelen 50 name security.debian.org_dists_jessie_updates_InRelease filetype 1 errors 4, no inode ref root 5 inode 75482 errors 2001, no inode item, link count wrong unresolved ref dir 9037 index 690 namelen 15 name srcpkgcache.bin filetype 1 errors 4, no inode ref root 5 inode 75483 errors 2001, no inode item, link count wrong unresolved ref dir 9037 index 692 namelen 12 name pkgcache.bin filetype 1 errors 4, no inode ref root 5 inode 75609 errors 2001, no inode item, link count wrong unresolved ref dir 405 index 541 namelen 13 name .flash_update filetype 1 errors 4, no inode ref root 5 inode 75613 errors 2001, no inode item, link count wrong unresolved ref dir 1460 index 152 namelen 24 name .oauth2-vault-email.json filetype 1 errors 4, no inode ref found 933789696 bytes used err is 1
total csum bytes: 0
total tree bytes: 7962624
total fs tree bytes: 7323648
total extent tree bytes: 589824
btree space waste bytes: 1334127
file data blocks allocated: 504639488
referenced 247123968

Checked with options -s 0 and -s 1 and the same result.

 

And finally attempting btrfs check --repair

enabling repair mode
Checking filesystem on /dev/md0
UUID: 
checking extents
incorrect offsets 7803 2164316543
items overlap, can't fix
btrfs: cmds-check.c: 4185: fix_item_offset: Assertion `!(ret)' failed.
Aborted

At least I got all the data off the NAS.  Seems like the next step is a Factory Reset, unless I've overlooked something...

Message 9 of 9
Top Contributors
Discussion stats
  • 8 replies
  • 2486 views
  • 1 kudo
  • 3 in conversation
Announcements