Reply

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

StephenB
Guru

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@corinbishop wrote:

sorry. forgot to mention that I'm also on 6.4.2-T72 which I'm guessing is a specific OS versions pre RC


Yes, it is also known as beta-2.

 

It'd be good if turning off quotas avoids the problem, but it'd be far better if the problem was actually fixed.  Did they tell you where the memory was going?  What apps are you running?

 

As far as other tests go, you could try running some of the maintenance tests (scrub, balance, defrag).  

 

Also you could try doing a couple things at once (a backup job to a USB drive, combined with over-the-network file transfers, and maybe have an app do something else at the same time).

 

 

Message 26 of 107
LifeVibes
Tutor

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

We have the same problem with one of our NAS here at work, so it’s not the only one.
So good to hear there might be workarounds and that it doesn’t seem to be a HW (memory) problem. Found it already a bit too coincidental that the mem failure would’ve just happened after fw upgrade. So prolly will save us RMA hassle.
Luckily the NAS was only rarely used until now for some less important backups. But it will normally get also another purpose in the very near future so would like to get it fully working again soon.

Symptoms are (almost exactly) the same as described above:
After fw upgrade from 6.2.5 to 6.4.1 system became unusable, normal boot never finishes and results in an out of memory error 354 after half an hour or so. Only hard power off (unplug) possible.
RAIDar sees the NAS booting "System starting up...", and then after a while "Management service is offline", and then after about 20 min it disappears and then shortly after also the OOM 354 error is displayed on the NAS.
During the period where it shows "Management service is offline" it shows the option buttons: Admin Page, Browse, Locate, Download Logs, Diagnostics
All of the buttons seem to work ok for a while, except for Admin Page:
Not able to access admin page: After supplying user/pw "Connecting to ReadyNAS Admin Page..." and then after a while "ReadyNAS Admin Page is offline."
Can perform locate (LEDs on), shares seem to be accessible for a short while and I can download logs a few times and Diagnostics are ok.
 
Tried several things:
Mem test displays 00:00:01 immediately after launching it and staying there forever. Hard reboot needed (power unplug). Don’t know if this display is pattern #, test time or error #.
Tried USB OS (re)install and OS reinstall from Boot Menu, didn’t help.

Tried the boot without network cable, as above, but has the same OOM result. So this is slightly different from above.
What does kinda work is booting in "Volume read only" mode from boot menu.
In this mode the NAS seems to boot and work normal, except for the fact you can’t write to it or delete anything of course and that the calculation (of used/free space) doesn’t seem to end.
Was able to do full backup (1.5T) and later a full CRC check of the backed up data on NAS. NAS was on for about a week like this and no problems.
Also scrub, balance and defrag don’t give errors, altho wonder how this can really work on a RO volume.

 

Yesterday tried to flash 6.4.2 RC1 but same problem still exists. Except that the error # now also is 390 instead of 354.
Volume RO is still the only mode where we can still do something.
There are no quota on the shares, so cant disable that. 😞

Config:
RN10400
2 x 4TB (X-raid Raid-1) 60% Free
very basic config:
Services: SMB, HTTP, HTTPS (AV disabled)
no apps
no quota
besides admin only one extra user.
3 shares: one with about 1.5T data with snapshots and bit rot enabled. Only 36 snapshots, so not really a large number. Permissions: admin + user.
2 test ones, currently empty, (snapshots disabled, bit rot enabled). Permissions: anonymous.

Message 27 of 107
corinbishop
Guide

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@LifeVibes wrote:

We have the same problem with one of our NAS here at work, so it’s not the only one.
So good to hear there might be workarounds and that it doesn’t seem to be a HW (memory) problem. Found it already a bit too coincidental that the mem failure would’ve just happened after fw upgrade. So prolly will save us RMA hassle.
Luckily the NAS was only rarely used until now for some less important backups. But it will normally get also another purpose in the very near future so would like to get it fully working again soon.

Symptoms are (almost exactly) the same as described above:
After fw upgrade from 6.2.5 to 6.4.1 system became unusable, normal boot never finishes and results in an out of memory error 354 after half an hour or so. Only hard power off (unplug) possible.
RAIDar sees the NAS booting "System starting up...", and then after a while "Management service is offline", and then after about 20 min it disappears and then shortly after also the OOM 354 error is displayed on the NAS.
During the period where it shows "Management service is offline" it shows the option buttons: Admin Page, Browse, Locate, Download Logs, Diagnostics
All of the buttons seem to work ok for a while, except for Admin Page:
Not able to access admin page: After supplying user/pw "Connecting to ReadyNAS Admin Page..." and then after a while "ReadyNAS Admin Page is offline."
Can perform locate (LEDs on), shares seem to be accessible for a short while and I can download logs a few times and Diagnostics are ok.
 
Tried several things:
Mem test displays 00:00:01 immediately after launching it and staying there forever. Hard reboot needed (power unplug). Don’t know if this display is pattern #, test time or error #.
Tried USB OS (re)install and OS reinstall from Boot Menu, didn’t help.

Tried the boot without network cable, as above, but has the same OOM result. So this is slightly different from above.
What does kinda work is booting in "Volume read only" mode from boot menu.
In this mode the NAS seems to boot and work normal, except for the fact you can’t write to it or delete anything of course and that the calculation (of used/free space) doesn’t seem to end.
Was able to do full backup (1.5T) and later a full CRC check of the backed up data on NAS. NAS was on for about a week like this and no problems.
Also scrub, balance and defrag don’t give errors, altho wonder how this can really work on a RO volume.

 

Yesterday tried to flash 6.4.2 RC1 but same problem still exists. Except that the error # now also is 390 instead of 354.
Volume RO is still the only mode where we can still do something.
There are no quota on the shares, so cant disable that. 😞

Config:
RN10400
2 x 4TB (X-raid Raid-1) 60% Free
very basic config:
Services: SMB, HTTP, HTTPS (AV disabled)
no apps
no quota
besides admin only one extra user.
3 shares: one with about 1.5T data with snapshots and bit rot enabled. Only 36 snapshots, so not really a large number. Permissions: admin + user.
2 test ones, currently empty, (snapshots disabled, bit rot enabled). Permissions: anonymous.


Your situation is identical to mine so I suspect something else has been done that's not been detailed. Unless it's specificly this special version of 642+quotas off?

Message 28 of 107
mdgm-ntgr
NETGEAR Employee Retired

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

This problem appears to happen on the 104 in some cases when the filesystem is quite full after updating from 6.2.x.

 

Even if you don't have quotas enabled for any shares we do use quotas still to calculate how much space is used by snapshots for instance.

 

There is likely a combination of things using up memory on affected systems.


Having run into this problem if you boot into read-only mode and backup your data, do a factory default (wipes all data, settings, everything) and restore your data from backup the system should be running fine with quotas enabled.

If you run into this if you prefer we can disable quotas from the backend so that the NAS will boot in normal mode.

Message 29 of 107
corinbishop
Guide

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

Thanks for the excellent explaination. One that makes sense to me.

One last question

Would it be better for me, in the long run, to wipe and start aknew or continue as i am? I have two full backups.

Thanks again
Message 30 of 107
mdgm-ntgr
NETGEAR Employee Retired

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

Can you send me your logs (see the Sending Logs link in my sig)? There will be some information in that, that will help with answering that question.

Message 31 of 107
corinbishop
Guide

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

Thanks. I've sent those logs to email in the link/instructions. 

 

c


@mdgm wrote:

Can you send me your logs (see the Sending Logs link in my sig)? There will be some information in that, that will help with answering that question.


 

Message 32 of 107
LifeVibes
Tutor

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


This problem appears to happen on the 104 in some cases when the filesystem is quite full after updating from 6.2.x.

I guess quite full is quite relative, as ours still has 60% free...

Even if you don't have quotas enabled for any shares we do use quotas still to calculate how much space is used by snapshots for instance.

yeah, I guessed there probably wouldve also been some internal quotas.

There is likely a combination of things using up memory on affected systems.

Now the tricky part is to find out the combination. But seeing that it seems mostly quite basic configs (min services, no apps, no AV etc), and the fact that it boots ok in RO mode and that in normal mode there seems a small window where it seems quite ok, except for the Admin Page which might be another clue, before it breaks down, I guess reduces the # combos.

If there is something we can do, let us know. Already have a looot of logs we could send Smiley LOL

Having run into this problem if you boot into read-only mode and backup your data, do a factory default (wipes all data, settings, everything) and restore your data from backup the system should be running fine with quotas enabled.

That wouldve been my ultimate last resort option and was already preparing for it by making a new backup. On ext4 target this time as previous was on NTFS to be able to read under Windows. Just hope the backup is a bit quicker this time, as it took about 2.5 days for backing up 1.5TB Smiley Frustrated

Is it also ok to backup/restore the configuration, or better to redo the config from scratch after the factory default?

Will the restore also restore the snapshots etc?

And then hope that by restoring the backup the memory combo isnt triggered again. 🙂


If you run into this if you prefer we can disable quotas from the backend so that the NAS will boot in normal mode.

Think we prefer to start from scratch then, cause who knows what the quota disable might cause in the future 😛 😉

Message 33 of 107
corinbishop
Guide

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

Update:

My system is stable. Support recommended a fresh install would be good. I have a stand-by NAS which is a mirror of the main machine (which I used during the outage) and once that's sync'd (and backed up to external drives too!) I'll do a wipe/restore. 

The one and only thing that's not good is read and write speed. I've gone from a comfortable 30-40MBytes/s read/write to 8.5 write, 27 Read. Which is actually worse than my old NV+ which I use as a backup. 

I'm going to do a restart and see if that improves but I doubt it will. 

 

 

Message 34 of 107
qborosfinest
Aspirant

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

Count me in, I just ran into this as well - Reported in #26383504 . 

 

The scenario is nearly identical to the author and other users who have reported. 

Although my original ticket was regarding an NFS mount issue, I ran into this as a result of upgrading from 6.2.2 -> 6.4.1 per troubleshooting steps. There was a version I had to step through, I can't recall right now which one exactly and I cannot access any logs at the moment. So far I'm hesistant to try more reboots or boot into other modes until instructed by support.

 

Upon first boot after the upgrade to 6.4.1, immediately noticed "calculating" on the volume was taking quite some time and never-ending.

The unit ran for approximately 12 hours (give or take) seemingly okay in terms of r/w operations..I may have even rebooted once more in those 12 hours..in any case then the issue began at some point.

When I hard reboot (unplug) after encountering the error/crash, UI doesn't ever seem to fully load but I am able to get an active terminal. free disk was above 60%, and maybe unrelated but saw a spike in kswapd cpu usage just as it crashed. This probably happens all within a minute or two at most before hitting err 354 again. I'm not running anything out of the ordinary (ssh, https, http, rsync, nfs, smb, and No AV or other apps).

 

I have an abundant number of snapshots that I don't care for. Is my understanding correct that support can delete the snapshots + disable quotas to alleviate this issue? For disabling quotas is that simply a config change or something further..? 

 

Thanks all! 

Message 35 of 107
Wytse
Aspirant

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

Also the Readynas 104, having trouble with the 354 out of memory message. Thought it was because i was racking up the date on my NAS but it states there is still like 4 tb of space.

I do have 4x4 tb WD Red installed. 

Anyone have any news if there is a fix yet.

 

Kind regards.

Message 36 of 107
qborosfinest
Aspirant

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

and occuring for you immediatley after upgrading to 6.4.1?

 

I'm waiting for an update from Netgear, they should be accessing the box soon as I'm waiting in Tech Support Mode right now.

Message 37 of 107
nilsel
Initiate

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

Same thing happening here. Upgraded to latest version after I had to do a os-reinstall (forgot admin-password, but it worked fine for 4-5 months or more).

A day or two after OS updates, it displayed "__out_of_memory+354". Unplug cord was only way to shut it down.

Memory-test from boot-menu also halts at 00:00:01.

Booted the NAS in read-only mode, and started the "scrubbing" process. Maybe not so wise (it takes 2+ days). Back from work today it displayed the same out of memory message. Don't know if it finished scrubbing.

Unplugged the unit for now until I have access to a 12TB backup option, or a miracle of a software update is released :).

 

specs: 4x3TB Seagate Barracuda disks. A bit over 70% disk usage. Quite a lot of old snapshots.

No 3rd party apps, NFS, SMB, AFP, HTTP(s) on.

 

Message 38 of 107
qborosfinest
Aspirant

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

According to a level 3 engineer, they tried to mount data volume while in tech support mode and terminal died. The LCD on NAS showed the same 354 Out of Memory. I have rebooted back into tech support mode and waiting..

 

Starting to lose faith here...is anyone else from Netgear watching this? Case # 26383504

 

Message 39 of 107
corinbishop
Guide

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

I think I'm one of the lucky ones who's managed to get back up and running (see previous posts for their solution). 

However I'm left with no confidence in the machine's stability and I've got a much lower write speed (read seems ok) as a result. 

 

I am going to schedule a complete wipe/restore soon. I would recommend that you do the same (assuming if you've got your data backed up. 

 

I think if anyone else is thinking of upgrading from 6.2.5 then 

 

A) backup x2

B) go to 6.4.0 first before 6.4.1 this does seem to be the major factor in problems

Message 40 of 107
qborosfinest
Aspirant

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@corinbishop wrote:

I think I'm one of the lucky ones who's managed to get back up and running (see previous posts for their solution). 

However I'm left with no confidence in the machine's stability and I've got a much lower write speed (read seems ok) as a result. 

 

I am going to schedule a complete wipe/restore soon. I would recommend that you do the same (assuming if you've got your data backed up. 

 

I think if anyone else is thinking of upgrading from 6.2.5 then 

 

A) backup x2

B) go to 6.4.0 first before 6.4.1 this does seem to be the major factor in problems


 

Hi Corinbishop,

Thanks for the info shared thus far. Is my understanding correct that support can delete the snapshots + disable quotas to alleviate this issue? For disabling quotas is that simply a config change or something further..? Can we do this ourselves and if so , can you share the steps?

Message 41 of 107
corinbishop
Guide

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

I didnt do the steps myself. It was support. Look back thru here.
My understanding was

1) delete (not disable) snapshots
2) install 6.4.2 rc
3) disable quotas
Message 42 of 107
LifeVibes
Tutor

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

Message 43 of 107
qborosfinest
Aspirant

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

Thanks corinbishop. Next step for me is to figure out how to offload all my data to an external USB drive. Waiting for Netgear support to provide some instructions on this step.

 

For what it's worth, I don't see anything related to this issue in the rel notes for 6.4.2.

Message 44 of 107
LifeVibes
Tutor

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


Next step for me is to figure out how to offload all my data to an external USB drive. Waiting for Netgear support to provide some instructions on this step.

If youre not in tech support mode yet, you could try to boot in Volume read only mode from the Boot Menu. That worked for me. It booted ok and was able to do a full backup + CRC comp afterwards without problems.

I used this FAQ as a guideline How do I back up data from my ReadyNAS OS 6 system to a USB disk?

And I would advise EXT4 target (eg external USB). Seems much faster than NTFS backup.

Prolly also preserves permissions better if that would be important. Only reason to choose NTFS would be if you also need to be able to read it under windows (eg on another pc).

For what it's worth, I don't see anything related to this issue in the rel notes for 6.4.2.

Guess they dont have a real clue about the issue yet.

Message 45 of 107
StephenB
Guru

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

I'd add

C) disable the AV service before the upgrade.


@corinbishop wrote:

 

 A) backup x2

 

This is always a good idea.


@corinbishop wrote:

 

B) go to 6.4.0 first before 6.4.1 this does seem to be the major factor in problems


I don't think so.  I'm seeing some users who are finding that only 6.4.0 works for them.  But some other users have found the opposite.  6.4.2 has some specific fixes for the RAID offloading hardware in the RN104, and I think some users need those fixes.

 

I'm not seeing enough of a pattern to make a recommendation either way.

 

 

 

 

Message 46 of 107
LifeVibes
Tutor

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

Although wasnt expected seeing the results with beta and RC versions, but just for completeness, 6.4.2 final release doesnt fix the problem.
Message 47 of 107
Wytse
Aspirant

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

I'm just a noob with these things, but if i exit 3 drives and only have one drive my NAS starts.

If i put all the 4 drives back in, and restart, i get the message 354 again.

But it does ask me to set the volumes anew. Does this mean i lost all my data on those drives? 

I hope not, cause it was like 10 tb of data.

 

 

Message 48 of 107
LifeVibes
Tutor

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


I'm just a noob with these things, but if i exit 3 drives and only have one drive my NAS starts.

I guess starting is relative, cause with only one disk out of 4 this isnt config which RAID can recover from anymore AFAIK. I would imagine there would be another error message which would happen earlier in the boot process than the OOM.

 

I did forget to put it in my original post of things I tried. I also tried all combos of removing disks in my 2 disk config. No disk gives the expected no disk warning. Either one disk gives the same problem in my case, as its RAID1 so both disks should be the same (copy of eachother).


But it does ask me to set the volumes anew. Does this mean i lost all my data on those drives? 

I hope not, cause it was like 10 tb of data.


I guess not, unless you reset the volumes or so. But seeing that you get the same error again when putting them all back should be ok.

Did you try booting in Volume Read only mode already to see if it boots and chekc your data and do a backup?

Message 49 of 107
StephenB
Guru

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@Wytse wrote:

I'm just a noob with these things, but if i exit 3 drives and only have one drive my NAS starts.

 

  


I think the out_of_memory is happening after the RAID array is mounted.  with only 1 drive in the NAS, that can't happen.

 

You can easily destroy any chance of data recovery if you are playing around with disks.  I suggest contacting support.

Message 50 of 107
Top Contributors
Discussion stats
  • 106 replies
  • 4199 views
  • 7 kudos
  • 11 in conversation
Announcements