× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

ReadyNAS 4200 USB Recovery

lawrencealan83
Aspirant

ReadyNAS 4200 USB Recovery

I came in to a restarted server this morning, and lost shares. I figured I would attempt to upgrade to the latest firmware, if I was already getting my hands dirty.

 

After the process ran, I couldn't reach the system. I did an OS reinstall, and after about 30m the machine came online at the default IP on all interfaces.

I was able to SSH in at that point, but when I tried to do anything – like launch apache – I was getting missing library "errors lib___.so is missing"

 

I ran an apt-get update, and at this point the system is only reachable through tech support mode. I was able to go in and verify my data is there (we have a backup, but I'd rather not have to use it)

 

I need to do a reinstall off of a USB drive, but none of the instructions I've found seem to work for a 4220.

 

Model: ReadyNAS-4200|ReadyNAS 4200
Message 1 of 13
jak0lantash
Mentor

Re: ReadyNAS 4200 USB Recovery

The system and settings are stored on the HDDs.

The Flash card contains an archive of the system and default settings.

 

When performing a Factory Default, the system and settings are extracted from the Flash to the HDDs.

When booting a NAS, the system and settings are read from the HDDs.

When performing an OS Reinstall, some system files and some settings are extracted from the Flash and overwrite the ones located on the HDDs.

When performing a USB Boot Recovery, the content of the USB stick overwrites the Flash.

 

In your current situation, I believe you don't need to perform a USB Boot Recovery.

If your system volume is corrupted/broken, that's what you need to fix, and that's stored on your HDDs.

 

At this stage, your probably better off trying to copy the data from the volume out and resetting the whole thing.

If you're confident with working via SSH, something like that might be a good start: https://thorsten-wagener.de/readynas-pro4-stuck-booting/

Though I believe some steps are missing, you could be able to put the pieces back together.

Message 2 of 13
lawrencealan83
Aspirant

Re: ReadyNAS 4200 USB Recovery

I believe there are important files missing that would be reinstalled. I want to attempt a USB recovery so that I may do a new OS reinstall.

 

The problem is that it's a substantial amount of data and would take quite a long time, I'd much rather repair it in place.

Message 3 of 13
jak0lantash
Mentor

Re: ReadyNAS 4200 USB Recovery

If you believe that the system volume on the HDDs AND the system archive on the Flash is corrupt, then USB Boot Recovery is the right step. Though it's very unlickely, I guess it's possible.

These are the instructions: https://kb.netgear.com/30267/RAIDiator-4-2-USB-Recovery-Tool

 

FWIW, an OS Reinstall does NOT overwrite the whole system and settings. So even if the Flash content is OK, it doesn't mean that OS Reinstall will fix your issue. I still believe that the issue is not the Flash.

Another approach could be to install a fresh system from the Flash on the OS volume located on the HDDs (without touching the data volume). But I don't think there are instructions online for it.

 

If time is a constraint, NETGEAR can assist you with doing this. But that would require purchasing a Data Recovery contract. You can also contact other Data Recovery specialists.

Message 4 of 13
lawrencealan83
Aspirant

Re: ReadyNAS 4200 USB Recovery

I connected via the RS232 port, and I can log in to the console as if I were SSH'd in. So the system is at least intact enough for it to boot.

 

There are, as expected, a lot of failed services.

 

After connecting via console I can reconfigure the network interfaces manually, so then I am able to reach the server via SSH

 

As soon as I try to start up smbd.service, I get .so errors, the console closes and I have to kill screen and login again.

 

I believe this is a repairable problem.

 

Message 5 of 13
lawrencealan83
Aspirant

Re: ReadyNAS 4200 USB Recovery

root@SERVER:~# systemctl smb.service start
systemctl: error while loading shared libraries: liblz4.so.1: cannot open shared object file: No such file or directory

Message 6 of 13
lawrencealan83
Aspirant

Re: ReadyNAS 4200 USB Recovery

Making progress, but things are still missing.

 

Anyone want to export an apt package list for the latest OS?

https://askubuntu.com/questions/9135/how-to-backup-settings-and-list-of-installed-packages

 

 

 


Welcome to Debian GNU/Linux 8 (jessie)!

[ SKIP ] Ordering cycle found, skipping /apps/pydio/web/data/files
[ OK ] Listening on /dev/initctl Compatibility Named Pipe.
[ OK ] Created slice User and Session Slice.
[ OK ] Listening on Journal Socket (/dev/log).
[ OK ] Listening on udev Control Socket.
[ OK ] Set up automount Arbitrary Executab...ats File System Automount Point.
[ OK ] Started Dispatch Password Requests to Console Directory Watch.
[ OK ] Listening on Journal Socket.
[ OK ] Listening on udev Kernel Socket.
[ OK ] Started Forward Password Requests to Wall Directory Watch.
[ OK ] Reached target Paths.
[ OK ] Reached target Encrypted Volumes.
[ OK ] Created slice System Slice.
Starting Create list of required st... nodes for the current kernel...
Mounting POSIX Message Queue File System...
[ OK ] Created slice system-serial\x2dgetty.slice.
Mounting Debug File System...
Starting MD arrays...
Starting Remount Root and Kernel File Systems...
[ OK ] Started ReadyNAS LCD splasher.
[ OK ] Reached target Slices.
Mounting RPC Pipe File System...
[ OK ] Created slice system-getty.slice.
Starting Journal Service...
Mounting RPC Pipe File System...
Starting Load Kernel Modules...
Starting ReadyNASOS system prep...
[ OK ] Mounted RPC Pipe File System.
[ OK ] Mounted RPC Pipe File System.
[ OK ] Mounted Debug File System.
[ OK ] Mounted POSIX Message Queue File System.
[ OK ] Started Create list of required sta...ce nodes for the current kernel.
[FAILED] Failed to start Remount Root and Kernel File Systems.
See 'systemctl status systemd-remount-fs.service' for details.
[ OK ] Started Load Kernel Modules.
Starting Apply Kernel Variables...
Mounting FUSE Control File System...
Mounting Configuration File System...
Starting Load/Save Random Seed...
Starting Rebuild Hardware Database...
Starting Create Static Device Nodes in /dev...
[ OK ] Mounted Configuration File System.
[ OK ] Mounted FUSE Control File System.
[ OK ] Started Apply Kernel Variables.
[ OK ] Started Load/Save Random Seed.
[ OK ] Started ReadyNASOS system prep.
[ OK ] Started Create Static Device Nodes in /dev.
Starting udev Kernel Device Manager...
[ OK ] Started Journal Service.
Starting Flush Journal to Persistent Storage...
[ OK ] Started Flush Journal to Persistent Storage.
Starting Create Volatile Files and Directories...
[ OK ] Started Create Volatile Files and Directories.
Starting Update UTMP about System Boot/Shutdown...
[ OK ] Started Update UTMP about System Boot/Shutdown.
[ OK ] Started Rebuild Hardware Database.
[ OK ] Started udev Kernel Device Manager.
[ OK ] Found device /dev/md1.
Activating swap md1...
[ OK ] Activated swap md1.
[ OK ] Found device /dev/disk/by-label/7c6f00e4:data.
[ OK ] Started MD arrays.
[ OK ] Reached target Local File Systems (Pre).
Mounting /data...
[ OK ] Reached target Swap.
Starting udev Coldplug all Devices...
[ OK ] Started udev Coldplug all Devices.
Starting udev Wait for Complete Device Initialization...
[ OK ] Reached target Sound Card.
[ OK ] Created slice system-systemd\x2dbacklight.slice.
Starting Load/Save Screen Backlight...ness of backlight:acpi_video0...
[ OK ] Started Load/Save Screen Backlight ...htness of backlight:acpi_video0.
[ OK ] Found device /dev/ttyS0.
[ OK ] Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
[ OK ] Started udev Wait for Complete Device Initialization.
[ OK ] Reached target System Initialization.
[ OK ] Listening on D-Bus System Message Bus Socket.
[ OK ] Started Timer for Debian Package List Updater.
[ OK ] Listening on RPCbind Server Activation Socket.
[ OK ] Listening on Avahi mDNS/DNS-SD Stack Activation Socket.
[ OK ] Reached target Sockets.
[ OK ] Started Daily Cleanup of Temporary Directories.
[ OK ] Started Log Truncate Timer.
[ OK ] Started Timer for ReadyNAS Update Service.
[ OK ] Reached target Timers.
[ OK ] Mounted /data.
Mounting /tmp...
Mounting /usr...
Mounting Home Directory...
Mounting /data/Guest...
Mounting Apps Directory...
Mounting /exports...
[ OK ] Mounted Home Directory.
[ OK ] Mounted Apps Directory.
[ OK ] Mounted /exports.
[ OK ] Mounted /data/Guest.
[ OK ] Mounted /usr.
[ OK ] Mounted /tmp.
[ OK ] Reached target Basic System.
Starting Login Service...
[ OK ] Started SNMP Daemon.
[ OK ] Started MD repair service.
[ OK ] Started fast remote file copy program daemon.
[ OK ] Started D-Bus System Message Bus.
Starting Connection service...
[ OK ] Started MD monitoring service.
Starting LSB: site24x7 Agent...
Starting Avahi mDNS/DNS-SD Stack...
[ OK ] Started RAIDar discovery service.
[ OK ] Started Regular background program processing daemon.
Mounting /apps/pydio/web/data/personal...
[ OK ] Mounted /apps/pydio/web/data/personal.
[ OK ] Started Login Service.
[FAILED] Failed to start Connection service.
See 'systemctl status connman.service' for details.
[DEPEND] Dependency failed for Wait for network to be configured by ConnMan.
Starting Wait until network is up...
[ OK ] Reached target Network.
Starting Samba NMB Daemon...
[ OK ] Started SSH Server.
Starting RPC bind service...
Starting NFS Server...
[ OK ] Reached target Network is Online.
[ OK ] Started Avahi mDNS/DNS-SD Stack.
[ OK ] Stopped Connection service.
Starting Connection service...
[FAILED] Failed to start Connection service.
See 'systemctl status connman.service' for details.
[ OK ] Started RPC bind service.
[ OK ] Started NFS Server.
[ OK ] Stopped Connection service.
Starting Connection service...
Starting NFS Mount Daemon...
Starting NFSv4 ID-name mapping daemon...
Starting NFS file locking service....
[ OK ] Reached target RPC Port Mapper.
[FAILED] Failed to start Connection service.
See 'systemctl status connman.service' for details.
[ OK ] Started NFS Mount Daemon.
[ OK ] Started NFS file locking service..
[ OK ] Reached target Remote File Systems (Pre).
[ OK ] Reached target Remote File Systems.
Starting Permit User Sessions...
Starting LSB: Monitor for system resources and process activity...
Starting LSB: Start/stop fail2ban...
Starting LSB: Start and stop the mysql database server daemon...
Starting LSB: Start or stop the inetd daemon....
Starting LSB: CUPS Printing spooler and server...
Starting ProFTPD FTP Server...
Starting LSB: OpenLDAP standalone s...ht Directory Access Protocol)...
[ OK ] Started NFSv4 ID-name mapping daemon.
[ OK ] Started Permit User Sessions.
[ OK ] Stopped Connection service.
Starting Connection service...
[ OK ] Started Getty on tty1.
[ OK ] Started Serial Getty on ttyS0.
[ OK ] Reached target Login Prompts.
[FAILED] Failed to start Connection service.
See 'systemctl status connman.service' for details.
[ OK ] Started LSB: Start or stop the inetd daemon..
[FAILED] Failed to start LSB: OpenLDAP stand...ight Directory Access Protocol).
See 'systemctl status slapd.service' for details.
[ OK ] Started LSB: Monitor for system resources and process activity.
[ OK ] Started LSB: CUPS Printing spooler and server.
[ OK ] Stopped Connection service.
Starting Connection service...
[FAILED] Failed to start Connection service.
See 'systemctl status connman.service' for details.
[ OK ] Stopped Connection service.
[FAILED] Failed to start Connection service.
See 'systemctl status connman.service' for details.
[ OK ] Started ProFTPD FTP Server.
[FAILED] Failed to start Samba NMB Daemon.
See 'systemctl status nmb.service' for details.
[ OK ] Started LSB: Start/stop fail2ban.

ReadyNASOS 6.7.2 SERVER ttyS0

SERVER login:

Message 7 of 13
mdgm-ntgr
NETGEAR Employee Retired

Re: ReadyNAS 4200 USB Recovery

So is this a 4200 (shipped with RAIDiator 4.2.x) or a 4220 (shipped with OS6)?

 

Have you checked how full the 4GB root volume is?

 

# btrfs fi show
# btrfs fi df /

 

Message 8 of 13
lawrencealan83
Aspirant

Re: ReadyNAS 42XX/4220 USB Recovery

Sorry, 4220 with ReadyNAS OS 6 my mistake.

 

The root drive is about 90% full.

 

I've got almost everything working at this point, but I would like to force a reinstall of ALL the dpkg/apt packages, as there are some packages definitely not correctly installed. (which is why I need a package list, and I don't want to use what my system returns)

 

root@SERVER:~# btrfs fi show
Label: '7c6f00e4:root'  uuid: 1a2ac9e9-9d77-4dfa-b12a-eaf1400ae9f1
Total devices 1 FS bytes used 3.23GiB
devid    1 size 4.00GiB used 4.00GiB path /dev/md0

Label: '7c6f00e4:data'  uuid: 865f4430-04dc-4e38-8b58-4328dcd64c0b
Total devices 1 FS bytes used 17.06TiB
devid    1 size 21.80TiB used 17.52TiB path /dev/md127

root@SERVER:~# btrfs fi df /
Data, single: total=3.57GiB, used=3.18GiB
System, DUP: total=8.00MiB, used=4.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=204.56MiB, used=50.80MiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=20.00MiB, used=0.00B
root@SERVER:~# df -H
Filesystem      Size  Used Avail Use% Mounted on
udev             11M  4.1k   11M   1% /dev
/dev/md0        4.3G  3.6G  418M  90% /
tmpfs           4.2G     0  4.2G   0% /dev/shm
tmpfs           4.2G  8.8M  4.2G   1% /run
tmpfs           2.1G  1.5M  2.1G   1% /run/lock
tmpfs           4.2G     0  4.2G   0% /sys/fs/cgroup
/dev/md127       24T   19T  5.2T  79% /data
/dev/md127       24T   19T  5.2T  79% /apps
/dev/md127       24T   19T  5.2T  79% /home
/dev/md127       24T   19T  5.2T  79% /data/Guest
root@SERVER:~#

 

What I had done in the past is at add a bind in fstab for the /usr folder to a folder in my RAID exports, as my 4GB root volume was getting pretty full.

This worked, even through previous updates, until it didn't, which is what I suspect the issue with the upgrade is/was. I've commented the `/usr` bind out for the time being.

 

 

/data/usr /usr              none          bind                                                  0       0
/data/tmp /tmp              none          bind                                                  0       0

 

Pydio uses /tmp to compress folders, which ends up killing the SERVER if someone attempts to compress a folder over 2GB (raw photos, videos)

 

 

 

 

Message 9 of 13
lawrencealan83
Aspirant

Re: ReadyNAS 42XX/4220 USB Recovery

Things I am still experiencing, for instance – PHP libs w/ missing dependencies from a custom PHP build:

 

Failed loading /usr/lib/php5/20100525/xcache.so:  /usr/lib/php5/20100525/xcache.so: cannot open shared object file: No such file or directory
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php5/20131226/gd.so' - libX11.so.6: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php5/20131226/imap.so' - libc-client.so.2007e: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php5/20131226/mcrypt.so' - libmcrypt.so.4: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php5/20131226/mysql.so' - libmysqlclient.so.18: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php5/20131226/mysqli.so' - libmysqlclient.so.18: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php5/20131226/pdo_mysql.so' - libmysqlclient.so.18: cannot open shared object file: No such file or directory in Unknown on line 0
Message 10 of 13
mdgm-ntgr
NETGEAR Employee Retired

Re: ReadyNAS 42XX/4220 USB Recovery

Your 4GB root volume is way too full. No wonder you are having issues.

Message 11 of 13
lawrencealan83
Aspirant

Re: ReadyNAS 42XX/4220 USB Recovery

Safe to delete files in `/var/cores/`? 

 

root@SERVER:/var/cores# ls -rtaflh
total 1.8G
drwxr-xr-x 1 root root  184 May 22 14:16 .
drwxr-xr-x 1 root root  180 May 22 18:01 ..
-rw------- 1 root root  31M Jan 29  2015 core-leafp2p
-rw------- 1 root root    0 Jul 12  2016 core-systemd-journal
-rw------- 1 root root 396K Aug  5  2016 core-killall
-rw------- 1 root root 9.1M Nov 23 15:51 core-john
-rw------- 1 root root 3.2M Apr 26 21:39 core-smbd
-rw------- 1 root root 1.8G May  2 15:39 core-mount.exfat
-rw------- 1 root root 165M May 22 14:16 core-readynasd
root@SERVER:/var/cores#

 

Message 12 of 13
mdgm-ntgr
NETGEAR Employee Retired

Re: ReadyNAS 42XX/4220 USB Recovery

core files are useful for debugging service crashes. You could either move them across to somewhere on the data volume or delete them if you don't need them.

Clearly you've been having some issues with an exfat mount.

Message 13 of 13
Top Contributors
Discussion stats
  • 12 replies
  • 8475 views
  • 0 kudos
  • 3 in conversation
Announcements