I have since a long time my ReadyNAS ultra 4 running on 4x3TB HDD (lost my account with the death of readynas.com).
My NAS, during the first years was a little bit tweaked (outside add-ons installation) :
-2GB RAM
-midnight commander installed to manage external file transfer, even when logged off (Ajaxplorer also did that, but there wasn't any feedback, and remote SFTP was somewhat buggy).
-chron job to manage several foscam IP cams, with continuous recording and then deleting all the videos without any movement
It worked like a charm until now, and of course, i forgot to heed the best warning out there : "when it's working, don't touch anything".
I didn't really look at updates since ages, and obvioulsy, many things had to be updated :
- raidiator 4.2.31 to install :
- lastest version of some add-ons to install
- and, browsing the forums, i discovered my bios was 1.09 and the lastest was 1.10.
In order :
- 2 weeks ago, i installed raidiator 4.2.31 : worked perfectly
- 1 week ago, as i couldn't make ajaxplorer 5.0 working with the pydio app, i decided to try ajaxplorer 5.2 : the app is now bricked, reinstalling the 5.0 version doesn't work.
- the day after, i tried installing the 1.10 bios, without success (felt like the installation didn't begin, and the logs showed that i was still with the 1.09 bios)
And the real problem comes here :
- i installed the latest php add-on : worked
- i installed forkeddaapd add-on : installation worked
- i tried to reinstall the latest bios (1.10) : the installation launched and finished, without any prompt
- i thought it worked.
- rebooted the nas, and it started without a hitch, except that my admin password didn't work anymore, and neither the defaut netgear one. All my add-ons were working, as i could connect to their webpage, and they were running
- i made the big mistake : i tried rebooting the nas, to see if this password thing was just a bug.
Now the nas is stuck to the booting process (displayed in the front screen, i waited an entire night without any change). I tried many things :
- it doens't appear in raidar
- it doesn't get any ip
- it is unreachable
- os reinstall in boot menu launch the reinstallation, but then the booting message stays on
- i can't make usb recovery to work, it doesn't recognise any of the usb stick i tried.
- i don't want to try factory reset
- i tried to remove disk 1, exactly the same behavior
- i removed all disks : reboots to the classical error : no disk
- i haven't yet tested with a blank disk (still need to find one)
Then, i tried to probe with the telnet mode :
- as i first thought it is a bios problem (silly me, if it was, i don't think it would boot at all), i reapplied the bios add-on, located in opt/biosflash : didn't change anything
- looking at forums, it is possible my firmare got corrupted, and even the one in the flash memory : i would try to reapply with telnet the firmware, but i don't know where to start with it : i don't know where i can find the firmware in the nas (or can i mount the usbcard and find it ?), and don't know how to reapply it.
- i just realised i can try to manually delete the lastest addons, to see if it changes anything (maybe one addon got incompatible with my firmware), but would deleting the folder be enough ?
Thanks for the reply. this shell command didn't work but i tried all the commnands and just one worked. it showed me that the sorting is logical : -sda through sdd for the 4 hdd -sde for the flash drive inside -sdf for the usb stick with the recovery firmware
I mounted the last 2 to check. The files had the same size, the flash files are 3 weeks old, when the firmware update worked perfectly. I wasn't confident enough to try and replace the files.
I tried the memory check and disk check, no error. i rebooted the nas to see where it is, it's still stuck, but it appears on my router's device list, and the frontview webpage loads, but is empty, just the exterior frame appears. Nothing else work...
I'd try the install on a blank disk next - I think it's important to sort out if the problem is really with the flash or if it's something borked on the OS partition before you go further.
Still haven't found my spare HDD to test it, but i've looked at the logs (i'm a bit stuck, many things i try in telnet are blocked, because my working directory isn't the nas OS, but the telnet busybox, and the chroot command fails (segmentation fault).
As i looked at logs, i opened the kernel log, which stops at the 7th of january (when everything went wrong), and it lists many services which failed to start because a segfault due to an error in libc-2.13, also in dpkg.
i feel like i'm well trapped. In order to have better commands, for example to fix my chroot issue, i need to chroot in the os partition...
It's hard to say where in the process above that your volume got so borked. I've never tried to recover one in that condition. Only had it happen once (may years ago) and I had a good backup and just did a factory default and restored.
I did want to comment on your "when it's working, don't touch anything" manttra, though. That's actually no way to treat a NAS. "Install and forget" can get you into a very deep mess before you realize it, and you may realize it only after there is any chance of recovery. Had you done the steps you mentioned as the updates became available/necessary, you may have avoided this.
I "check in" on my primary NAS at least once a week and boot my backup NAS for a quick peek at least monthly, or when somehting on the primary (file issue, available update, whatever) makes me think I should do it now.
I managed to debrick the nas : - i first reimported the lib and lib64 folder, didn't work, i had others libs which gave problems - i then deleted the folders and then imported : managed to liberate chroot, but not much more - as it was still stuck, after much unfruitful tries, i went for a fresh os install, by backuping most of the partition (if i can later reimport some config files) and deleting almost all folders - it still didn't work... But when trying and failing to mount the main raid array, i was curious as to why mdadm told me that the md0 and md2 array are from 3 disks, instead of 4 - long story short, my disk 1 was corrupted (don't know why disktest didn't say anything) , and corrupted my os partition, which is why first removing the disk didn't help. - i removed it, it booted but i had to reinstall the os, again, to get this fresh start. - it's now resyncing the disk 1, but i got back frontview and access to the nas 😊
But as good things don't last, 10 minutes later, as i started parametring frontview, my admin password magically changed, and neither the default password nor mine work. I'm waiting for the resyncing to finish before looking at this problem.
But as good things don't last, 10 minutes later, as i started parametring frontview, my admin password magically changed, and neither the default password nor mine work.
In case you've forgotten: the default admin password for your NAS is netgear1.
If that fails, then try doing another OS reinstall after the resync finishes.
I haven't forgotten, i used it for the fresh reinstall. As you point, i'm waiting for the resync to finish, but i'm afraid my disk is most likely corrupted, the password recovery frontview page fail to load the secret question, thus fail to provide the recovery.
@tiranor2 wrote: I haven't forgotten, i used it for the fresh reinstall. As you point, i'm waiting for the resync to finish, but i'm afraid my disk is most likely corrupted, the password recovery frontview page fail to load the secret question, thus fail to provide the recovery.
Have you tried testing the drives in a windows PC? (connecting either with a usb adapter or SATA, and testing with vendor tools - lifeguard or seatools)?
If not, have you checked the SMART stats using smartctl?
- disconnected again the first disk (as it seemed it was the one corrupting everything)
- in telnet, cleaning again the os partition.
- when i checked the flash memory (because yesterday morning i had an extract error when trying for an os reinstall), the md5checksum showed me that kernel.up was missing.
- i verified in my raidiator 4.2.31 folder that the md5 checksum was correct, and then copied the kernel.up in the flash memory.
- i then retried an OS reinstall, which worked, and managed to boot my fresh OS, on 3 disks only.
- as i didn't want to repeat the previous day errors, i went back to telnet mode, to fully check my disk 1 (the faulty one), while offline and unmounted. To be sure i wouldn't accidently do something irreversible to the other disks, in removed them.
- i then used smartctl (it didn't work the last time i tried, but i guess i made a typo at the moment)
- it showed me the disk reported an error about 7 days ago, and Raw_Read_Error_Rate and Seek_Error_Rate were through the roof (something like hundreds of millions and hundreds of billions)
- i then did a complete dd rewrite of the disk (the weird thing is that in my case, with a raid array resync, the resync would have done exactly the same thing, i don't know why it failed the previous day),
- then checked with smartctl which reported that the disk is good to go
- put in back this morning, it's resyncing without a hitch (50% yet to go), but contrary to yesterday, i still have my frontview login working.
Small bug occuring, i got an error when trying to change the nas hostname, i followed a previous post by mdgm which worked (the etc/hostname file was missing).
So far so good, but this adventure made me realize that i should consider changing/upgrading my nas, and use my ultra4 for backuping the main nas. My nas for the moment, is used to backup my computers main files, but i didn't have the need to backup the nas main files, as i thought i could recreate them, but the years are passing, and i guess i don't have as much time now than i did before.
I'm now hesitating about different solutions, all with used nas (a lot cheaper, all between 100-200€) :
- Pro6, which i would upgrade to OS 6, and probably upgrade RAM and CPU
- RN316 (the least powerful, but the easiest)
- proliant microserver gen7 (the cheapest), i guess with something like OMV
- proliant microserver gen8 (the most powerful and most flexible), with XPenology or OMV
Raw_Read_Error_Rate and Seek_Error_Rate were through the roof (something like hundreds of millions and hundreds of billions)
What's in these fields can be vendor-specific, so the values aren't useful unless you know what's encoded in them (and the format). From what I've read, in the case of Raw_Read_Error_Rate, Seagate disks have an error count encoded in the top 16 bits, and and sectors read count in the bottom 32 bits.
I'm now hesitating about different solutions, all with used nas (a lot cheaper, all between 100-200€) :
- Pro6, which i would upgrade to OS 6, and probably upgrade RAM and CPU
- RN316 (the least powerful, but the easiest)
The RN316 would also be newer than the Pro6 - since the Pro/Ultra lines were discontinued when it was introduced. It'd be similar in performance to the Ultra.
According to cpubenchmark, the RN316 CPU is 3 times more powerful than the Atom in the Ultra4 (the ultra2 and ultra4 unfortunatly have the Atom D410, which is single core).
The Pro 6 cpu is about twice as powerful as the RN316 dual core Atom, and is upgradable if necessary, but as you said, it has old hardware.
I could aim for a RN516, but it's not available in used market here.
The Pro 6 cpu is about twice as powerful as the RN316 dual core Atom, and is upgradable if necessary, but as you said, it has old hardware.
Both are limited by network for large file transfers (running ~100 mb/sec). The CPU obviously would make a difference for something like Plex (particularly doing real-time transcoding). But both could handle a single 1080p stream and neither could handle 4K (fwiw, neither can the RN516).
