RN104 post latest firmware boot crashes to error submit_one_bio+28 error

Hi Paul,

when it boots to safe mode its like a temporary setup for analysing or recovery so it was just a way of checking everthing on the machine functions and for me it did, you could at this point slide the disks back in and re-form the array and get your data off ( my next plan ) but after everything had booted ok like that i wondered if i put everything back to normal would it then boot... and did for a time, then its like having its throat cut and a crash takes it down, nothing in the logs at all but will try this morning to get logs etc seems odd that 6.9.4 comes along and does this after being stable for a long time.. questioning the hardware as it does just stop dead in its tracks...

Lee

---

Leedanskin wrote:

questioning the hardware as it does just stop dead in its tracks...

 

---

It could be hardware, but I am thinking it could also be a btrfs file system issue. 

ok from a hardware standpoint i now believe all to be perfectly fine in safe mode with the disks in no errors all of the disk have no issues reported and it has funtioned perfectly for  an hour while i try to get this thing back online, there seems no solution to activating the volume via the 6.9.4 gui it locates all the drives says they are fine but the volume is not activated, raidar say's all the same things sees all 4 drives system is happy with an inactive volume, ssh'ing into the box allows me to check the file system but in safe mode appears to have no mdadm available to make the volume active which would allow me to get my data off.

I have the option off downgrading the firmware via the safe mode, so my guess is the internal firmware that is installed that is not the safemode version has a dodgy block, or file system error or that there is something just nasty in the 6.9.4 update.. that is causing some instablity but the disks are fine and the main system is fine.

Does safe mode have the option of just making the volume 0 active ? via ssh or via the Gui?

Lee

Just to add now on reboot comes back with the error submit_one_bio+ac thinking the internal firmware is fubar'd

is there a 6.9.3 ?

Lee

---

Leedanskin wrote:

 

is there a 6.9.3 ?

 

 

---

Yes,  here: https://kb.netgear.com/000055199/ReadyNAS-OS-6-Software-Version-6-9-3

Though perhaps try over-installing 6.9.4 first. https://kb.netgear.com/000060110/ReadyNAS-OS-6-Software-Version-6-9-4

I don't think it's the firmware though.  If you have a scratch disk, you can do a factory default with just that disk in place, and see if you get the same symptoms.  

I have a spare 2TB HDD so I'm booting into Factory Default mode now...

Booted ok 

RAIDar shows Status: Healthy and Firmware: 6.9.4

I'm just going to ignore it for a few minutes and see if it stays stable or hangs again

OK, so following StephenB's suggestion, I'll try over-installing 6.9.4 first. 

I downloaded 6.9.4 on my Mac; went to the admin page and did Manual Install Firmware; got the OK message confirming 6.9.4 had been installed; now it's rebooting...

Interesting! Now, when I go to the Admin site, it shows the installed apps which didn't appear after the first attempt at updating to 6.9.4 a week or so back!

I'm going to switch the blank 2TB for the 4 * 4TB HDD and see if it's fixed...  Nope, too hopeful. Still get the "submit_one_bio+28" error :( 

It took a couple of attempts but I've downgraded to 6.9.3 and now it's busy synchronising (presumably because I've been through so many combinations of which disks are inserted/removed).

Fingers crossed...

Incidentally, I've also removed the UPS data cable and one of the LAN cables in the hope of a simpler config might help it succeed.

Recover Data has made it past 1% which is a major milestone - it's not been able to get past about 0.75% without hanging, so I think (hope!) I've got rid of the "submit_one_bio+28" error for good this time.

Some or all of these steps got me there - hopefully they'll be of some help to others who encounter this problem:

• downgrade Firmware to 6.9.3 
  • download it from https://kb.netgear.com/000055199/ReadyNAS-OS-6-Software-Version-6-9-3
  • unzip that file - you only need the IMG file
  • despite saying that FW version 6.9.3 was now installed, rebooting said I was back on 6.9.4 a couple of times
  • when it succeeded, I had 3 of my 4 drives installed (locations 1, 3 & 4, I think it was) 
• I also removed one of the two LAN cables and the UPS data cable
• you don't have long from it booting to hanging, so be prepared - I kept the IMG file on my Mac's desktop so I could find it quickly.

Good luck Leedanskin - I hope this helps you get your NAS working again too.

Thanks StephenB and JohnCM_S for trying to help.

paul_henman

Paul,

Followed your guide to downgrading to 6.9.3 and seems to have worked a treat. Wanted to say thanks, greatly appreciated.

kind regards

Brad

paul_henman and Bradek3788,

We can't reproduce the issue in local, and want to fix it asap,

you have downloaded 6.9.3, could do you please download system logs and send to us?

How do I send all logs to ReadyNAS Community moderators?
https://kb.netgear.com/21543/How-do-I-send-all-logs-to-ReadyNAS-Community-moderators

I'm glad it worked for you; I hope it's still stable and running fine.

Unfortunately I came home from work (after it had been running on 6.9.3 for a couple of days) to find it had rebooted into safe mode and was on 6.9.4! 

I tried restoring 6.9.3 a couple of times but it just wouldn't stick, and then the RAID-5 volume disappeared - I hate to think that this means all my date is gone.

I've left it turned off since then because I have no patience left for dealing with it at present.

evan2 paul_henman

Thanks i have copied the logs t a word doc as couldn't get the download option to work and will upload.

Cheers

Bradek3788

I received your logs, it is only status logs, it is not complete.

Could you please enable secure diagnostic mode and PM me 5 digit number?

https://kb.netgear.com/000053266/ReadyNAS-OS-6-Enabling-Secure-Diagnostics-Mode

then we can remote access to check your NAS.

evan2

Confirm sdm sent

Bradek3788

Get your debug code, we will access your NAS to check.

Thanks for your support.

just got back from a work trip first chance I have had to get this working, ok so took all system disks out and added a new system disk, which allowed me to downgrade through manual firmware to 6.9.3, make sure that you generate a volume and generate a new system that boots with checking root FS each time it boots till you have a fully functioning nas at 6.9.3.

Then replace disks when booting make sure that you boot to boot mode and re-install OS to disks as the option or it just upgrades back to 6.9.4 and then causes you to start all over again!

this then gave me a system that boots to 6.9.3, i had to restart after this to get a stable system but all is now fine and no data lost so very happy, backing up like crazy!

Ta @paul_henman got there in the end!

Lee

Glad it worked for you Leedanskin

paul_henman Leedanskin

I'm testing a new version of update and will post if all seems ok after couple of days.

fingers crossed.

Great let us know, did anyone comeback with a conclusion to it?

Leedanskin paul_henman evan2 StephenB JohnCM_S

Hi all,

just to confirm several days, i think 5 days and the new modified firmware update has worked without issue.

kind regards

Brad

Where's the new version of the firmware update?

The newest I can find is 6.9.4 and the unzipped IMG file is dated September 2nd, so this is the same one that caused the problem, isn't it?

---

paul_henman wrote:

Where's the new version of the firmware update? 

---

I suspect they gave Bradek3788 something to test, which is not publicly available, and probably shouldn't be shared.

Likely this would be released as a hot-fix (which is downloaded automatically if your NAS can reach the internet).

paul_henman StephenB Leedanskin evan2

Hi Guys,

Correct they logged in to test and suplied a patched version, just wanted to let everyone who has the issue that the patch worked for me so a light at the end of the tunnel.