Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

[corinbishop]

installed 6.4.1 from 6.2.5 yesterday. All went ok. Did all the 'calculating'. worked fine. 

failed overnight with out_of_memory error 354. now won't boot. 

tried reinstallOS. No luck. 

totally stuck and need it for business!

[LifeVibes · Accepted solution]

You could try RO mode, see https://community.netgear.com/t5/Using-your-ReadyNAS/Readynas-104-won-t-boot-Error-354-out-of-memory...

[corinbishop]

did a memory test from boot menu. fails at 00:00:01 
I'm guessing that's not good?

[StephenB]

---

@corinbishop wrote:

did a memory test from boot menu. fails at 00:00:01 
I'm guessing that's not good?

---

Not good.

I think you should contact support (likely an RMA is in your future).

[corinbishop]

Slight progress.. will boot if I disconnect the network cable. I can reconnect the network cable, get an IP but then shortly after I get the error. Sounds to me like a process problem leaking memory. 

[StephenB]

Of course if the memory is ok, it could be that.  But If boot memory diag detects bad memory, then it probably isn't a memory leak.

Anyway, if it boots in tech support mode (with the cable attached), then Netgear could take a look.  So maybe try that.

If you use support, they will charge - though its conceivable someone here will kindly offer to take a look.

[corinbishop]

Thanks. I suspect software related as it happened after the upgrade (jeez, why did i do it??!!)
The unit will run ok until its get and ip address (and therefore starts the network services).

Ive logged a support call and they are giving me one freebie support case.

I'll let you know what happens
Cheers

[mdgm-ntgr]

Do you recall which services and apps you are running?

[corinbishop]

It was smb, ntfs, rsync, UpnP, http, https, ftp, ssh

Anti-virus off. No apps

[thenorthface]

Has anybody solved this yet ? It's happened to mine last night after the new firmware update keeps saying out of memory 354 ? Is there a way to go back to the old firmware

[mdgm-ntgr]

Hi thenorthface,

I have sent you a PM.

The two examples of this problem in this thread are the first I've seen of this.

Generally OOM would suggest either running too much on the NAS or a memory leak.

[corinbishop]

Hiya, 

I don't think downgrade is possible. 

I've opened a support case with netgear so I'll let you know what happens. 

So far they've asked a load of questions about ping/admin panel/raidar etc. They also asked me to try support mode:

My response to them this morning was:

_________________________________________________________________
Hi xxxxxx and thanks for taking on this problem.

I'll try to give as much detail as possible and hopefully it will help:

So:

As explained before, I did the upgrade from 625 to 641. All proceeded as planned, I was able to use the device after restart and it took a couple of hours to clear the 'calculating' status on the volume/share screen. The device continued to work until later that evening when I checked on it and found the device unresponsive. Error message was as described. All lights were on (steady) including the ACT.

I tried to reboot and got the same message on startup. I was unable to ping the machine or connect.

I tried the reinstallOS from the boot menu. It did the reinstall but it froze at the same point in boot up.

The next thing I tried was to boot it up with no network connection (i.e. unplugged everything except power).

This actually worked and the device sat there quite happily saying no-ip with all lights flashing.

I reconnected the network cable, it got an IP address but it froze with the error message 354 in under a minute. I was able to ping the machine before it froze.

I tried this a couple of times with varying levels of time 'on' before freezing again.

At no point until this morning was I able to access the admin panel.

+++++++++++++

After your email:

This morning I booted again (without the network cable) and continually pinged the device. Roughly 1 min after boot (and the no-ip message) I connected the cable, it got an IP and I was able to ping the device.

I quickly tried the admin page and was able to log in. However all I got was the main menu, it wouldn't show logs, status, performance or anything.

Within 1 minute the device then froze again.

I've tried this once more but with no success.

I also tried leaving the device with no network cable connected for longer, however eventually it froze again (just too longer to freeze than when the cable was connected)

++++++++++++++ Support mode:

I tried support mode and it shows the following with no network cable connected:

DebugMode[57550]
0.0.0.0

and with network cable connected

DebugMode[57550]
192.168.0.106

all lights (including ACT) are steady, not blinking.

I am able to ping the device ok.

I've not been able to access any system logs I'm afraid.
____________________________________________________________________

[corinbishop]

Hi mgdm, 

I'm running very little on the NAS and just using basic SMB and NTFS file sharing. I'm at 70% capacity and everything is pretty standard. 
I'd say it's a software problem with one of the services. It would be great if they had a 'boot with basic' option where it could disable all but the basic admin panel and perhaps ssh services etc. 

I'm running on my backup server for the moment and luckily it had mirrored before this problem. 

Fingers crossed it will get fixed soon. I noticed my usual IT supplier selling the 10400 in a end of line sale cheap (€190). So I bought it just in case  (I don't know if I could just move the disks over of if that would transfer the problem too?). Or could I use 6.2.5 and then move the disks? I'd hate to have to restore from backup as it would take so long but needs must.

[corinbishop]

Latest update:

Netgear support were able to access the techsupport mode no problem. 

They did another install of 6.4.1 which rebooted but again failed after a short time (this time out_of_memory 390 not 354). 

The course of action was to delete my snapshots. I had quite a lot going back quite some time and with large amounts of data (1TB!). 

They thought this could be a problem for the new software. My question, if this is the cause, will be whether 6.4.1 can't handle large number of snapshots or just can't handle upgrading a large number of snapshots created under 6.2.5. 

The deletion of snapshots is going to take some time so it might be a while before I know if it worked. 

[StephenB]

Older firmware also had problems if the volume got too full of snapshots.  I don't recall the out-of-memory specifically.  But it has always been important to keep an eye on snapshot use.  That's one reason why 6.4.x is giving more information about them.

[mdgm-ntgr]

Support is still diagnosing the problem. It's unusual for a system to crash in tech support mode with just the data volume mounted.

If the problem is with the chassis, then moving the disks across to another 104 would help, but if the problem is on the disks then it won't.

The next step for support probably is to try a read-only filesystem check.

[corinbishop]

Hi. 

Yes, it is unusual. It's the fact that it only crashes after a period of time/activity which would suggest it's not a hardward fault and is a combination of something wrong with the filesystem and this particular version of the OS. 

I hope I don't have to swap to a new machine and/or fresh installation. I do have a copy of the data but it's going to be a looooong restoration process. Having said that, at this stage, even if the problem were fixed my confidence level wouldn't be high.

I've been asked to reboot techsupport mode with a USB. Hopefully they can get some useful info written to the USB. 

[StephenB]

---

@corinbishop wrote:

Hi. 

Yes, it is unusual. It's the fact that it only crashes after a period of time/activity which would suggest it's not a hardward fault and is a combination of something wrong with the filesystem and this particular version of the OS. 

---

You might well be right that it is software, but I don't think this symptom rules out hardware.  For instance, If there's memory corruption due to faulty block of memory, it could well take a while to cause an outright crash.

---

@corinbishop wrote:

I hope I don't have to swap to a new machine and/or fresh installation. 

---

Actually migrating to a replacement RN104 would be simpler (and much faster than the fresh installation).

I hope it all gets sorted quickly.

[kohdee]

Could you possibly send me a picture of the LCD with that error message? I'm compiling a list of LCD error messages since 6.4.1 and putting them on the KB.

Out of memory on a RN104 is really easy to do, being that RN104 has 512M RAM and 256M of it is used by the system to boot, but if you're failing instantly in the boot, it does sound like bad RAM, which is only replacable by RMA of chassis 
If you have a spare disk to test on and it works out fine, then it might be something software related. 

[corinbishop]

---

@kohdee wrote:

Could you possibly send me a picture of the LCD with that error message? I'm compiling a list of LCD error messages since 6.4.1 and putting them on the KB.

Out of memory on a RN104 is really easy to do, being that RN104 has 512M RAM and 256M of it is used by the system to boot, but if you're failing instantly in the boot, it does sound like bad RAM, which is only replacable by RMA of chassis 
If you have a spare disk to test on and it works out fine, then it might be something software related. 

---

sure. pic attached. 

(note that it's currently error 390, previously it was 354 but I've not seen 354 come up for quite a while)

I'm thinking it's possible a RAM problem. Just seems strange it only appeared after upgrade to 641.  Depends on the scope of the upgrade perhaps. It's non-downgradable so perhaps there's a change to the lower level of the hardware drivers/software/firmware etc. 

I have done the mem test again and it does fail at 00:00:01 !!!

Nothing back from netgear in the last few days after I sent them a crash dump they gathered via tech support mode. 

I think at this stage an RMA would be the best option to rule out hardware. I do have spare disks but I'm not convinced that a single disk with a small amount of data is really going to be conclusive as it could still be a problem with the larger file structure. The only one unusual thing I have is I had upgraded a failed disk recently from a 3TB to 4TB with a view to changing all the disks to eventually increasing the volume size XRAID styleee...

So it's gone from 3TB x4 to, 4TB+3TBx3.

It shouldn't really make any difference with xraid but who knows, it might have been a situation that was not looked at in the new firmware.

[thenorthface]

my rn104 is the same as yours but i still have the out of memory 354 error message after updating thefirmware, tech support have taken my dump file but that was a few days ago and still no answer ! 

[corinbishop]

---

@thenorthface wrote:

my rn104 is the same as yours but i still have the out of memory 354 error message after updating thefirmware, tech support have taken my dump file but that was a few days ago and still no answer ! 

---

Hi thenorthface, 

Well, it does sound like there is a major issue with this firmware and certain models of NAS. Did you try a mem test too?  Might be interesting to know. I still suspect firmware/hardware compatibility issuse. 

Like you, I'm still waiting for support and it's gone quiet. Id really like to know if it there is any way to migrate back to 6.2.5 which was very stable. Even if it meant them sending me a new machine and I wipe/restore. I really need to get back up and running soon. I'm running on my backup and that's not a good situation if there is another failure. 

[StephenB]

6.4.2-RC1 might be worth a try, since there are fixes for the RN104.  Though if you are engaged with support, it is best to let them do the troubleshooting.

BTW it is possible to downgrade to 6.2.5 with a factory reset.  There is a thread here on how to do that.  However, it'd be best to find a way forward, not back.

[corinbishop]

I'll see if they want to go for 642rc. However i think at this stage it would be best to let support decide. Id agree forward is best but then whatever is going to get me running is an issue too!

[corinbishop]

Hi folks. 

Support have decided to disable quotas and see how I go. So far, so good. System is stable and I'm giving the unit a good thrashing - starting with syncing all the data changes Id made on the backup unit since the failure. The comparison alone will take a while.

If anyone else can suggest any other tests I could give it then let me know.

cheers

[corinbishop]

sorry. forgot to mention that I'm also on 6.4.2-T72 which I'm guessing is a specific OS versions pre RC