Re: RN104 post latest firmware boot crashes to error submit_one_bio+28 error

Leedanskin · ‎2018-09-19

After latest firmware update rn104 boots starts network appears on network looks happy then crashes to error submit_one_bio+28 error and all stops is this a root volume error or something more sinister.. any pointers gratefully recieved!

JohnCM_S · ‎2018-09-19

Hi Leedanskin,

Can you download the system logs and provide it to us? You can upload it to a file sharing site and PM me the download link.

Regards,

Leedanskin · ‎2018-09-19

Hi john

would love to do that how do i get the system logs if the system crashes back to this and is no longer available? all appears normal till it then crashes

Lee

JohnCM_S · ‎2018-09-20

Hi Leedanskin,

How long does it take before the system crashes? Can you download the logs before it happens?

Regards,

paul_henman · ‎2018-09-20

I've had exactly the same problem since I upgraded to 6.9.4 last week.

I got a tweet from NetgearHelps yesterday saying to boot into Tech Support mode, which I did and it's showing "Debug code: XXXXX" but I can't find any further info about that message.

RAIDar detects it, which it didn't after a normal boot (NAS was hung).

I can't connect (via browser) to the IP address shown by RAIDar, so I guess my next step is to ssh into the box to see what I can see.

StephenB · ‎2018-09-20

@paul_henman wrote:

boot into Tech Support mode, which I did and it's showing "Debug code: XXXXX" but I can't find any further info about that message.

That is NOT an error code. It's a PIN that Netgear can use to remotely access your system. Somehow you need to get the code to the person who asked you to go into tech support mode.

@paul_henman wrote:

I guess my next step is to ssh into the box to see what I can see.

You can't use ssh in tech support mode, but you could use telnet. You'd need to start RAID, mount the OS partition and data volume manually.

If Netgear wants to take a look, then the right next step is to let them.

paul_henman · ‎2018-09-20

Whoops! Thanks for removing that info.

Their tweet said "contact support" but my warranty's expired, so they expect me to pay US$80+ in order to have them look at a problem caused by their OS update?

StephenB · ‎2018-09-20

@paul_henman wrote:

Whoops! Thanks for removing that info.

That was a just-in-case precaution - the access should be secured and limited to Netgear support.

Perhaps PM some of the netgear mods here, and see if someone is wanting to take a courtesy look. The error message is unusual.

Have you tried testing the disks with vendor diags in a Windows PC (Lifeguard for WDC, Seatools for Seagate)?

JohnCM_S · ‎2018-09-20

@Leedanskin @paul_henman

Can any of you hook up a console cable to the NAS and reproduce the issue again so we can get the full error?

Regards,

paul_henman · ‎2018-09-20

How do we do that? Which port / cable type?

Leedanskin · ‎2018-09-21

Would love to get the console up, as Paul mentioned what cable and how do I boot into service mode?

Lee

Leedanskin · ‎2018-09-22

Ok after booting to safe mode with the disks out and then restarting the machine I can start the machine and all is well after running for a period of time the machine then errors with the same error submit_one_bio+28.

I will leave for a while and download the logs, I might rollback to 6.9.2 but want to check first if anyone else other than Paul and I are seeing this issue.

Lee

paul_henman · ‎2018-09-22

I booted without any disks too, but the logs were empty - presumably they're stored on the drives.

I think I have it configured as RAID 5 (it was a long time ago so I'm not certain) so I'll try various combinations of 3 drives to see if it'll boot without hanging.

Leedanskin · ‎2018-09-23

Hi Paul,

when it boots to safe mode its like a temporary setup for analysing or recovery so it was just a way of checking everthing on the machine functions and for me it did, you could at this point slide the disks back in and re-form the array and get your data off ( my next plan ) but after everything had booted ok like that i wondered if i put everything back to normal would it then boot... and did for a time, then its like having its throat cut and a crash takes it down, nothing in the logs at all but will try this morning to get logs etc seems odd that 6.9.4 comes along and does this after being stable for a long time.. questioning the hardware as it does just stop dead in its tracks...

Lee

StephenB · ‎2018-09-23

@Leedanskin wrote:

questioning the hardware as it does just stop dead in its tracks...

It could be hardware, but I am thinking it could also be a btrfs file system issue.

Leedanskin · ‎2018-09-23

ok from a hardware standpoint i now believe all to be perfectly fine in safe mode with the disks in no errors all of the disk have no issues reported and it has funtioned perfectly for an hour while i try to get this thing back online, there seems no solution to activating the volume via the 6.9.4 gui it locates all the drives says they are fine but the volume is not activated, raidar say's all the same things sees all 4 drives system is happy with an inactive volume, ssh'ing into the box allows me to check the file system but in safe mode appears to have no mdadm available to make the volume active which would allow me to get my data off.

I have the option off downgrading the firmware via the safe mode, so my guess is the internal firmware that is installed that is not the safemode version has a dodgy block, or file system error or that there is something just nasty in the 6.9.4 update.. that is causing some instablity but the disks are fine and the main system is fine.

Does safe mode have the option of just making the volume 0 active ? via ssh or via the Gui?

Lee

Leedanskin · ‎2018-09-23

Just to add now on reboot comes back with the error submit_one_bio+ac thinking the internal firmware is fubar'd

is there a 6.9.3 ?

Lee

StephenB · ‎2018-09-23

@Leedanskin wrote:

is there a 6.9.3 ?

Yes, here: https://kb.netgear.com/000055199/ReadyNAS-OS-6-Software-Version-6-9-3

Though perhaps try over-installing 6.9.4 first. https://kb.netgear.com/000060110/ReadyNAS-OS-6-Software-Version-6-9-4

I don't think it's the firmware though. If you have a scratch disk, you can do a factory default with just that disk in place, and see if you get the same symptoms.

paul_henman · ‎2018-09-23

I have a spare 2TB HDD so I'm booting into Factory Default mode now...

Booted ok

RAIDar shows Status: Healthy and Firmware: 6.9.4

I'm just going to ignore it for a few minutes and see if it stays stable or hangs again

OK, so following @StephenB's suggestion, I'll try over-installing 6.9.4 first.

I downloaded 6.9.4 on my Mac; went to the admin page and did Manual Install Firmware; got the OK message confirming 6.9.4 had been installed; now it's rebooting...

paul_henman · ‎2018-09-23

Interesting! Now, when I go to the Admin site, it shows the installed apps which didn't appear after the first attempt at updating to 6.9.4 a week or so back!

I'm going to switch the blank 2TB for the 4 * 4TB HDD and see if it's fixed... Nope, too hopeful. Still get the "submit_one_bio+28" error 😞

paul_henman · ‎2018-09-23

It took a couple of attempts but I've downgraded to 6.9.3 and now it's busy synchronising (presumably because I've been through so many combinations of which disks are inserted/removed).

Fingers crossed...

Incidentally, I've also removed the UPS data cable and one of the LAN cables in the hope of a simpler config might help it succeed.

paul_henman · ‎2018-09-23

Recover Data has made it past 1% which is a major milestone - it's not been able to get past about 0.75% without hanging, so I think (hope!) I've got rid of the "submit_one_bio+28" error for good this time.

Some or all of these steps got me there - hopefully they'll be of some help to others who encounter this problem:

downgrade Firmware to 6.9.3
- download it from https://kb.netgear.com/000055199/ReadyNAS-OS-6-Software-Version-6-9-3
- unzip that file - you only need the IMG file
- despite saying that FW version 6.9.3 was now installed, rebooting said I was back on 6.9.4 a couple of times
- when it succeeded, I had 3 of my 4 drives installed (locations 1, 3 & 4, I think it was)
I also removed one of the two LAN cables and the UPS data cable
you don't have long from it booting to hanging, so be prepared - I kept the IMG file on my Mac's desktop so I could find it quickly.

Good luck @Leedanskin - I hope this helps you get your NAS working again too.

Thanks @StephenB and @JohnCM_S for trying to help.

Bradek3788 · ‎2018-09-24

I have exactly the same error message since upgrading to latest firmware last week. No idea how to do a roll back or what to attempt first. any help appreciated.

evan2 · ‎2018-09-24

@paul_henman

Could do you please boot into tech mode again and PM me debug code?

Let us remote access to check the problem.

paul_henman · ‎2018-09-24

I successfully went back to 6.9.3 so I no longer have the problem. However @Leedanskin and @Bradek3788 would probably be grateful for some help in getting their NAS running again.