× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973

Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

Wytse
Aspirant

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

Thank you for your answers.

I tried booting in the read-only mode, and the data is all still there, very happy and now backing this up as we speak (write).

As for contacting support, i'm following this thread and hoping when there is a sollution that i can take part in it.

I don't think contacting support would help speed things up as multiple people experience this problem.

Thank you also for caring.

Message 51 of 107
mdgm-ntgr
NETGEAR Employee Retired

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

If you've run into this and haven't sent me your logs yet can you please do so: http://kb.netgear.com/app/answers/detail/a_id/25625/~/how-do-i-send-readynas-system-logs-to-netgear-...

We can try disabling quotas from the backend if you like, but I think the best solution would be to backup your data, update to 6.4.2 and do a factory default (wipes all data, settings, everything) on that and restore your data from backup.

Message 52 of 107
corinbishop
Guide

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

Hi mgdm, 

 

I'm going to do the wipe etc. Can I just confirm the procedure to follow, bearing in mind that I appear to be on a special version of OS 6.4.2-T72.

 

1. Backup (in my case it'll be 2 backups, one to an old NV+ and one to external HD's). 

2. Settings>Configuration Backup>Everything

3. Perform the Factory Default as in http://kb.netgear.com/app/answers/detail/a_id/20898/session/L2F2LzEvdGltZS8xNDU0NTg1MTU4L3NpZC9saUJp...

4. After the reset has been performed, restore the configuration settings taken in step 2 (restart at all?).

5. Restore data from back drives.

6. Have a beer (or 2).

 

Are there any issues regarding my asymetrical drives? I have 4TB, 3TB, 3TB, 3TB. 

Message 53 of 107
StephenB
Guru

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@corinbishop wrote:

Hi mgdm, 

 

I'm going to do the wipe etc. Can I just confirm the procedure to follow, bearing in mind that I appear to be on a special version of OS 6.4.2-T72.

 

1. Backup (in my case it'll be 2 backups, one to an old NV+ and one to external HD's). 

2. Settings>Configuration Backup>Everything

3. Perform the Factory Default as in http://kb.netgear.com/app/answers/detail/a_id/20898/session/L2F2LzEvdGltZS8xNDU0NTg1MTU4L3NpZC9saUJp...

4. After the reset has been performed, restore the configuration settings taken in step 2 (restart at all?).

5. Restore data from back drives.

6. Have a beer (or 2).

 

Are there any issues regarding my asymetrical drives? I have 4TB, 3TB, 3TB, 3TB. 


Step 1.5 - uninstall all apps

Step 3.5 - reinstall all apps.

 

The most convenient way to do the default is from the web ui (systems->settings)

 

If you are using flexraid, you will need to destroy the xraid volume created by the default, and recreate your old volumes.  You don't need to wait for the xraid volume to sync. 

 

Your drive configuration won't be a problem, though if you are using the normal raid5/xraid you are losing 1 TB of space.

Message 54 of 107
corinbishop
Guide

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@StephenB wrote:

@corinbishop wrote:

Hi mgdm, 

 

I'm going to do the wipe etc. Can I just confirm the procedure to follow, bearing in mind that I appear to be on a special version of OS 6.4.2-T72.

 

1. Backup (in my case it'll be 2 backups, one to an old NV+ and one to external HD's). 

2. Settings>Configuration Backup>Everything

3. Perform the Factory Default as in http://kb.netgear.com/app/answers/detail/a_id/20898/session/L2F2LzEvdGltZS8xNDU0NTg1MTU4L3NpZC9saUJp...

4. After the reset has been performed, restore the configuration settings taken in step 2 (restart at all?).

5. Restore data from back drives.

6. Have a beer (or 2).

 

Are there any issues regarding my asymetrical drives? I have 4TB, 3TB, 3TB, 3TB. 


Step 1.5 - uninstall all apps

Step 3.5 - reinstall all apps.

 

The most convenient way to do the default is from the web ui (systems->settings)

 

If you are using flexraid, you will need to destroy the xraid volume created by the default, and recreate your old volumes.  You don't need to wait for the xraid volume to sync. 

 

Your drive configuration won't be a problem, though if you are using the normal raid5/xraid you are losing 1 TB of space.


 

 

Thanks a mill. I'm using Xraid and planning to eventually upgrade all the drives to 4TB ones. A 3TB failed recently (one of the infamous Seagate 3TB **bleep**e drives!) and I replaced it with 4TB. 

 

Just to be sure: a screenshot of the volumes page is here: https://www.dropbox.com/s/uve7gitjn56sxmw/Screenshot%202016-02-04%2012.00.20.png?dl=0

 

 

Message 55 of 107
StephenB
Guru

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@corinbishop wrote:

 

Just to be sure: a screenshot of the volumes page is here: https://www.dropbox.com/s/uve7gitjn56sxmw/Screenshot%202016-02-04%2012.00.20.png?dl=0

 

 


You are running normal xraid.  Post back on how it goes (and whether the reset resolves your problem).

Message 56 of 107
corinbishop
Guide

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

Thanks. 

 

I'm currently running a stable system. Support got me working ok. I'm doing the reset to provide a fresh, new starting point. The only problem I've been left with is poor write speed. The other recent posts are from someone who has the same problem but has not managed to get back online yet. So he's doing the reset to get him running, I'm doing it for long term certainty!

Message 57 of 107
qborosfinest
Aspirant

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

I'm working on 2nd backup right now, then going with factory reset per Support recommendation - they have not provided any alternative workarounds. I will post my findings after. In some regard, glad we're not alone and it's impacting many. Let's hold hands..

Message 58 of 107
LifeVibes
Tutor

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

We are certainly not the only ones, just do a search on "104 boot" and youll find a lot of similar reports of 104 (and other RN) not booting after 6.4 updates, prolly all more or less related.

Only "real solution" seems to be factory reset.

Message 59 of 107
StephenB
Guru

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@LifeVibes wrote:
just do a search on "104 boot" and youll find a lot of similar reports of 104 (and other RN) not booting after 6.4 updates, prolly all more or less related.

 

You are certainly not alone, but there are multiple (and unrelated) issues.  One is that the boot loader is updated when you first move to 6.4.x, and that update in particular sometimes fails.

 

Message 60 of 107
mdgm-ntgr
NETGEAR Employee Retired

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@LifeVibes wrote:

This problem appears to happen on the 104 in some cases when the filesystem is quite full after updating from 6.2.x.

I guess quite full is quite relative, as ours still has 60% free...


Looking at your logs whilst your volume is nowhere near full you do have a huge amount of metadata

 

btrfs-progs v4.1.2
=== filesystem /data ===
Data, single: total=1.34TiB, used=1.34TiB
System, DUP: total=8.00MiB, used=172.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=74.50GiB, used=42.39GiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=512.00MiB, used=0.00B

Here's what I get on my 516 for a comparison (though I have no snapshots):

# btrfs fi df /data
Data, single: total=12.42TiB, used=12.36TiB
System, DUP: total=32.00MiB, used=1.34MiB
Metadata, DUP: total=15.50GiB, used=14.07GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


 I can see you had a number of your snapshots that were not upgraded when you updated to 6.2.4.

One would have to suspect that at least some of your data is not suited to snapshots.

Message 61 of 107
LifeVibes
Tutor

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@mdgm wrote:

Looking at your logs whilst your volume is nowhere near full you do have a huge amount of metadata
 I can see you had a number of your snapshots that were not upgraded when you updated to 6.2.4.

One would have to suspect that at least some of your data is not suited to snapshots.


Ah so it seems it comes down to snapshots after all.

But a bit strange that data could not be suited for snapshots. Would expect snapshots to be able to handle any data.

AFAIK we dont have any special data on this NAS, mosltly media files and

Is there some more info/investigations I need/can to do (to help finding the root)?

 

Otherwise was planning to finally do the factory reset today, to make use of the weekend to restore the backup.

Message 62 of 107
StephenB
Guru

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@LifeVibes wrote:

But a bit strange that data could not be suited for snapshots. Would expect snapshots to be able to handle any data.

 


You might want to read my post on how btrfs snapshots work here: https://community.netgear.com/t5/ReadyNAS-in-Business/ReadyNAS-312-Need-Help-Understanding-Snapshots...

 

Snapshots can "handle" any data, but if you are rewriting a lot of files in the share then they can use a lot of space and generate a lot of fragmentation.  One example of a folder where snapshots aren't very efficient is a torrent folder.  You are better off keeping snapshots disabled on shares where there is a lot of "churn". 

Message 63 of 107
LifeVibes
Tutor

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@StephenB wrote:
Snapshots can "handle" any data, but if you are rewriting a lot of files in the share then they can use a lot of space and generate a lot of fragmentation.  One example of a folder where snapshots aren't very efficient is a torrent folder.  You are better off keeping snapshots disabled on shares where there is a lot of "churn".

Actually this NAS was used as safegueard storage/backup for media files etc.

Almost all changes are just additions, hardly any deletions/replacements, but changes could be big. And also very low freq, only every x months a change/addition or so. So the "churn", as I see it, should be very low.

But so dont immediatly see any unsuitable data use for snapshots here.

Message 64 of 107
StephenB
Guru

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@LifeVibes wrote:

Almost all changes are just additions, hardly any deletions/replacements, but changes could be big. And also very low freq, only every x months a change/addition or so. So the "churn", as I see it, should be very low.

 

But so dont immediatly see any unsuitable data use for snapshots here.


With deletions/replacements, the original data stays in the snapshot, additions have no effect.  No fragmentation occurs.  Tagging a media file (after the next snapshot is taken) will result in fragmentation.

 

I agree that your usage sounds good for snapshots, though I do suggest going in periodically and deleting the oldest snapshot in every share.  I keep mine for 3 months, and go in about one a month.

 

I'd love to see the firmware manage max retention time automatically, but it doesn't.

Message 65 of 107
kohdee
NETGEAR Expert

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

There are some enhancements to snapshots in the pipeline for our next major release.

Message 66 of 107
StephenB
Guru

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@kohdee wrote:

There are some enhancements to snapshots in the pipeline for our next major release.


Great news!  

Message 67 of 107
qborosfinest
Aspirant

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

After successful factory reset, I am back online and working again. So far stable but definitely degradation in write speed from where I was prior to 6.4.1.

Here some points below:

- Noticed about 20-25 MB/s write speed with rsync via NFS mount. Previously I would see burst of 80-100 MB/s and settle in at 30-40 MB/s. NFS mount is using async and the share has bitrot protect (copy-on-write enabled) just like pre 6.4.1.

- Oddly enough, seeing VERY slow write speed 5-15 MB/s when rsync between USB3 drive mounted to NAS to RAID-5 volume (executed all within NAS). I didnt test this pre 6.4.1.

- Only running basic (smb,afp,nfs,ssh,http). No AV or other apps.

 

 Other info:

- RAID-5, Volume resyncing in progress, looks like it's going to take days/week to complete.Maybe longer since I already started data restoration.

- Snapshots and quotas disabled (from UI).  Snapshots were enabled in the previous config before this mess began.

- Majority of data is large video files.

 

Let me know if I can share anything else to help or run any tests..

Message 68 of 107
corinbishop
Guide

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@qborosfinest wrote:

After successful factory reset, I am back online and working again. So far stable but definitely degradation in write speed from where I was prior to 6.4.1.

Here some points below:

- Noticed about 20-25 MB/s write speed with rsync via NFS mount. Previously I would see burst of 80-100 MB/s and settle in at 30-40 MB/s. NFS mount is using async and the share has bitrot protect (copy-on-write enabled) just like pre 6.4.1.

- Oddly enough, seeing VERY slow write speed 5-15 MB/s when rsync between USB3 drive mounted to NAS to RAID-5 volume (executed all within NAS). I didnt test this pre 6.4.1.

- Only running basic (smb,afp,nfs,ssh,http). No AV or other apps.

 

 Other info:

- RAID-5, Volume resyncing in progress, looks like it's going to take days/week to complete.Maybe longer since I already started data restoration.

- Snapshots and quotas disabled (from UI).  Snapshots were enabled in the previous config before this mess began.

- Majority of data is large video files.

 

Let me know if I can share anything else to help or run any tests..


It's interesting you're seeing the same speed degredation as me. Sounds like we have a similar setup. Was it a full reset/wipe/new etc? I was going to do that in the hope the speed issue would be fixed but I won't both if there's going to be no change. 

Message 69 of 107
StephenB
Guru

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

If the resync is still in process, then the performance of course suffers, so I think you probably need to wait until the resync is done and then post back.

 

I did a factory reset on my RN102 a couple days ago (no apps installed at this point).  It's jbod, not xraid - so there is no resync.  I've been restoring the data with rsync backup from my Pro-6, and am seeing restore speeds of about 1 TB/day (which amounts to 12 MB/s).  I don't recall what the rsync speed was before, so I don't know if that's a drop-off or not (rsync is computationally intensive).

 

It should be finished by Monday, then I'll run a CIFS benchmark.

Message 70 of 107
corinbishop
Guide

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@StephenB wrote:

If the resync is still in process, then the performance of course suffers, so I think you probably need to wait until the resync is done and then post back.

 

I did a factory reset on my RN102 a couple days ago (no apps installed at this point).  It's jbod, not xraid - so there is no resync.  I've been restoring the data with rsync backup from my Pro-6, and am seeing restore speeds of about 1 TB/day (which amounts to 12 MB/s).  I don't recall what the rsync speed was before, so I don't know if that's a drop-off or not (rsync is computationally intensive).

 

It should be finished by Monday, then I'll run a CIFS benchmark.




I've not transfers or backups going on at all and seeing poor performance. 

 

Services on are: SMB, AFP, FTP, Rsync, HTTP, HTTPs, SSH.

No antivirus. No apps.

best write performance from a windows device has been just under 9Mb/s. Best write performance has been 28.

previously at least 30 write, 40 read.

 

Message 71 of 107
qborosfinest
Aspirant

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@corinbishop wrote:


It's interesting you're seeing the same speed degredation as me. Sounds like we have a similar setup. Was it a full reset/wipe/new etc? I was going to do that in the hope the speed issue would be fixed but I won't both if there's going to be no change. 


 

Correct, full reset/wipe was done. Will see if performance is better after resync finishes although I can see resync process is intelligent enough to pause to allow NFS activity so I doubt it. Speeds through NFS are also dropped down to 10-15 MB/s now..Are you considering downgrading perhaps?

 

Perhaps not the forum to vent but this was a bit of frustrating experience, and again a lesson why you shouldn't upgrade fw blindly, neither should support force it down your throat to support you if your product is mostly working.

I started with a NFS performance/stability issue and upgraded to 6.4.1 because my issue would not be escalated otherwise. This of course turned into 354 out of memory and all the headaches of a factory reset followed. Now I'm left with less than half the read/write speeds I had when I started and I'll spend the next couple of days restoring.

In hindsight, that initial issue was relatively minor and not worth all this.

But in the end, I do understand this is a consumer product, and I should not expect the same level of support or diligence as I'm used to in the enterprise world at work.

Message 72 of 107
corinbishop
Guide

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

Yes. The we should have waited a bit longer and checked the issues others were having. It has been frustrating. I agree with all you've said.

I've one more option. I have a spare, unused, unopened 104. If i move the disks over what will happen?
Message 73 of 107
StephenB
Guru

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717


@corinbishop wrote:



I've one more option. I have a spare, unused, unopened 104. If i move the disks over what will happen?

I thinks its clear that the performance dropoff is a software issue, so I'd be very surprised if migrating the disks help.

Message 74 of 107
mdgm-ntgr
NETGEAR Employee Retired

Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

If migrating the disks would help with this OOM error on boot issue, it would be from moving to a model such as the RN214, not to another RN104. I haven't seen any reports of a user trying this so don't know whether it would help. However even if this did resolve it, it wouldn't deal with the underlying cause that the system configuration and/or usage was not done in a way that is recommended.

I haven't seen any reports of this problem on models other than the 100 series.


Updating the firmware is a common requirement for escalating cases. If an agent escalated a case with a minor issue on old firmware then the case would likely be de-escalated again with the agent told to get you to update the firmware so that we can rule out encountering an issue that we have already fixed.

We have general advice on updating the firmware in ReadyNAS OS 6: Firmware Upgrade Guide and Tips which includes a suggestion to update your backup (which you should be doing regularly regardless of whether you update the firmware).

The Guide also mentions some common reasons why problems might be encountered:

 

  • Systems that are completely full.
  • Systems that have high filesystem fragmentation.
  • Systems that have large quantities of hourly, daily, monthly snapshots.

The first and last of these should be easy for you to verify before you update the firmware. The middle one may usually (but not always) be somewhat related to the other two, but advanced users could get a good indication by looking at the metadata usage in btrfs.log. If the metadata usage is huge then this would suggest that the way the system was configured and/or used was far from ideal.

It appears that all systems encountering this problem are affected by one or more of the issues described in these bullet points. 

 

Some suggestions going forward would be to keep volume usage under 80%, run regular scheduled volume maintenance (defrag & balance) and to only use bit-rot protection and snapshots on shares which are suited to using those, not on every share.

Performance during a resync will be reduced. So if your volume is currently syncing you should test performance again once the sync completes.

It is important to note that the initial sync when creating a volume, is a special case when it comes to redundancy. During this initial sync if you have multiple disks and have chosen a redundant RAID option (e.g. the default X-RAID) then the volume is redundant throughout the initial sync.

Message 75 of 107
Top Contributors
Discussion stats
  • 106 replies
  • 5439 views
  • 7 kudos
  • 11 in conversation
Announcements