NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Solved
Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
installed 6.4.1 from 6.2.5 yesterday. All went ok. Did all the 'calculating'. worked fine.
failed overnight with out_of_memory error 354. now won't boot.
tried reinstallOS. No luck.
totally stuck and need it for business!
106 Replies
Replies have been turned off for this discussion
Same thing happening here. Upgraded to latest version after I had to do a os-reinstall (forgot admin-password, but it worked fine for 4-5 months or more).
A day or two after OS updates, it displayed "__out_of_memory+354". Unplug cord was only way to shut it down.
Memory-test from boot-menu also halts at 00:00:01.
Booted the NAS in read-only mode, and started the "scrubbing" process. Maybe not so wise (it takes 2+ days). Back from work today it displayed the same out of memory message. Don't know if it finished scrubbing.
Unplugged the unit for now until I have access to a 12TB backup option, or a miracle of a software update is released :).
specs: 4x3TB Seagate Barracuda disks. A bit over 70% disk usage. Quite a lot of old snapshots.
No 3rd party apps, NFS, SMB, AFP, HTTP(s) on.
According to a level 3 engineer, they tried to mount data volume while in tech support mode and terminal died. The LCD on NAS showed the same 354 Out of Memory. I have rebooted back into tech support mode and waiting..
Starting to lose faith here...is anyone else from Netgear watching this? Case # 26383504
I think I'm one of the lucky ones who's managed to get back up and running (see previous posts for their solution).
However I'm left with no confidence in the machine's stability and I've got a much lower write speed (read seems ok) as a result.
I am going to schedule a complete wipe/restore soon. I would recommend that you do the same (assuming if you've got your data backed up.
I think if anyone else is thinking of upgrading from 6.2.5 then
A) backup x2
B) go to 6.4.0 first before 6.4.1 this does seem to be the major factor in problems
corinbishop wrote:I think I'm one of the lucky ones who's managed to get back up and running (see previous posts for their solution).
However I'm left with no confidence in the machine's stability and I've got a much lower write speed (read seems ok) as a result.
I am going to schedule a complete wipe/restore soon. I would recommend that you do the same (assuming if you've got your data backed up.
I think if anyone else is thinking of upgrading from 6.2.5 then
A) backup x2
B) go to 6.4.0 first before 6.4.1 this does seem to be the major factor in problems
Hi Corinbishop,
Thanks for the info shared thus far. Is my understanding correct that support can delete the snapshots + disable quotas to alleviate this issue? For disabling quotas is that simply a config change or something further..? Can we do this ourselves and if so , can you share the steps?
- I didnt do the steps myself. It was support. Look back thru here.
My understanding was
1) delete (not disable) snapshots
2) install 6.4.2 rc
3) disable quotas Thanks corinbishop. Next step for me is to figure out how to offload all my data to an external USB drive. Waiting for Netgear support to provide some instructions on this step.
For what it's worth, I don't see anything related to this issue in the rel notes for 6.4.2.
Next step for me is to figure out how to offload all my data to an external USB drive. Waiting for Netgear support to provide some instructions on this step.If youre not in tech support mode yet, you could try to boot in Volume read only mode from the Boot Menu. That worked for me. It booted ok and was able to do a full backup + CRC comp afterwards without problems.
I used this FAQ as a guideline How do I back up data from my ReadyNAS OS 6 system to a USB disk?
And I would advise EXT4 target (eg external USB). Seems much faster than NTFS backup.
Prolly also preserves permissions better if that would be important. Only reason to choose NTFS would be if you also need to be able to read it under windows (eg on another pc).
For what it's worth, I don't see anything related to this issue in the rel notes for 6.4.2.
Guess they dont have a real clue about the issue yet.
After successful factory reset, I am back online and working again. So far stable but definitely degradation in write speed from where I was prior to 6.4.1.
Here some points below:
- Noticed about 20-25 MB/s write speed with rsync via NFS mount. Previously I would see burst of 80-100 MB/s and settle in at 30-40 MB/s. NFS mount is using async and the share has bitrot protect (copy-on-write enabled) just like pre 6.4.1.
- Oddly enough, seeing VERY slow write speed 5-15 MB/s when rsync between USB3 drive mounted to NAS to RAID-5 volume (executed all within NAS). I didnt test this pre 6.4.1.
- Only running basic (smb,afp,nfs,ssh,http). No AV or other apps.
Other info:
- RAID-5, Volume resyncing in progress, looks like it's going to take days/week to complete.Maybe longer since I already started data restoration.
- Snapshots and quotas disabled (from UI). Snapshots were enabled in the previous config before this mess began.
- Majority of data is large video files.
Let me know if I can share anything else to help or run any tests..
qborosfinest wrote:After successful factory reset, I am back online and working again. So far stable but definitely degradation in write speed from where I was prior to 6.4.1.
Here some points below:
- Noticed about 20-25 MB/s write speed with rsync via NFS mount. Previously I would see burst of 80-100 MB/s and settle in at 30-40 MB/s. NFS mount is using async and the share has bitrot protect (copy-on-write enabled) just like pre 6.4.1.
- Oddly enough, seeing VERY slow write speed 5-15 MB/s when rsync between USB3 drive mounted to NAS to RAID-5 volume (executed all within NAS). I didnt test this pre 6.4.1.
- Only running basic (smb,afp,nfs,ssh,http). No AV or other apps.
Other info:
- RAID-5, Volume resyncing in progress, looks like it's going to take days/week to complete.Maybe longer since I already started data restoration.
- Snapshots and quotas disabled (from UI). Snapshots were enabled in the previous config before this mess began.
- Majority of data is large video files.
Let me know if I can share anything else to help or run any tests..
It's interesting you're seeing the same speed degredation as me. Sounds like we have a similar setup. Was it a full reset/wipe/new etc? I was going to do that in the hope the speed issue would be fixed but I won't both if there's going to be no change.
If the resync is still in process, then the performance of course suffers, so I think you probably need to wait until the resync is done and then post back.
I did a factory reset on my RN102 a couple days ago (no apps installed at this point). It's jbod, not xraid - so there is no resync. I've been restoring the data with rsync backup from my Pro-6, and am seeing restore speeds of about 1 TB/day (which amounts to 12 MB/s). I don't recall what the rsync speed was before, so I don't know if that's a drop-off or not (rsync is computationally intensive).
It should be finished by Monday, then I'll run a CIFS benchmark.
- Yes. The we should have waited a bit longer and checked the issues others were having. It has been frustrating. I agree with all you've said.
I've one more option. I have a spare, unused, unopened 104. If i move the disks over what will happen?
corinbishop wrote:
I've one more option. I have a spare, unused, unopened 104. If i move the disks over what will happen?I thinks its clear that the performance dropoff is a software issue, so I'd be very surprised if migrating the disks help.
If migrating the disks would help with this OOM error on boot issue, it would be from moving to a model such as the RN214, not to another RN104. I haven't seen any reports of a user trying this so don't know whether it would help. However even if this did resolve it, it wouldn't deal with the underlying cause that the system configuration and/or usage was not done in a way that is recommended.
I haven't seen any reports of this problem on models other than the 100 series.
Updating the firmware is a common requirement for escalating cases. If an agent escalated a case with a minor issue on old firmware then the case would likely be de-escalated again with the agent told to get you to update the firmware so that we can rule out encountering an issue that we have already fixed.
We have general advice on updating the firmware in ReadyNAS OS 6: Firmware Upgrade Guide and Tips which includes a suggestion to update your backup (which you should be doing regularly regardless of whether you update the firmware).
The Guide also mentions some common reasons why problems might be encountered:- Systems that are completely full.
- Systems that have high filesystem fragmentation.
- Systems that have large quantities of hourly, daily, monthly snapshots.
The first and last of these should be easy for you to verify before you update the firmware. The middle one may usually (but not always) be somewhat related to the other two, but advanced users could get a good indication by looking at the metadata usage in btrfs.log. If the metadata usage is huge then this would suggest that the way the system was configured and/or used was far from ideal.
It appears that all systems encountering this problem are affected by one or more of the issues described in these bullet points.Some suggestions going forward would be to keep volume usage under 80%, run regular scheduled volume maintenance (defrag & balance) and to only use bit-rot protection and snapshots on shares which are suited to using those, not on every share.
Performance during a resync will be reduced. So if your volume is currently syncing you should test performance again once the sync completes.
It is important to note that the initial sync when creating a volume, is a special case when it comes to redundancy. During this initial sync if you have multiple disks and have chosen a redundant RAID option (e.g. the default X-RAID) then the volume is redundant throughout the initial sync.
- Balance done. Took 25mins ish.
Kicked off a scub last night. This morning it's done 1.35%! Slooooowwwscrub started 8pm Friday.... at 19% now... is it normal to be that slow?
How big is the data volume?
It does sound slow, but if it is making progress I'd let it continue.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
