Forum Discussion

Guide

Jan 14, 2016

Solved

Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

installed 6.4.1 from 6.2.5 yesterday. All went ok. Did all the 'calculating'. worked fine. failed overnight with out_of_memory error 354. now won't boot. tried reinstallOS. No luck. tota...

Booting

Installation & Upgrade

LifeVibes
Feb 15, 2016
You could try RO mode, see https://community.netgear.com/t5/Using-your-ReadyNAS/Readynas-104-won-t-boot-Error-354-out-of-memory-After-upgrade-to/m-p/1044151#M103244

LifeVibes

Tutor

Jan 28, 2016

This problem appears to happen on the 104 in some cases when the filesystem is quite full after updating from 6.2.x.

I guess quite full is quite relative, as ours still has 60% free...

Even if you don't have quotas enabled for any shares we do use quotas still to calculate how much space is used by snapshots for instance.

yeah, I guessed there probably wouldve also been some internal quotas.

There is likely a combination of things using up memory on affected systems.

Now the tricky part is to find out the combination. But seeing that it seems mostly quite basic configs (min services, no apps, no AV etc), and the fact that it boots ok in RO mode and that in normal mode there seems a small window where it seems quite ok, except for the Admin Page which might be another clue, before it breaks down, I guess reduces the # combos.

If there is something we can do, let us know. Already have a looot of logs we could send :smileylol:

Having run into this problem if you boot into read-only mode and backup your data, do a factory default (wipes all data, settings, everything) and restore your data from backup the system should be running fine with quotas enabled.

That wouldve been my ultimate last resort option and was already preparing for it by making a new backup. On ext4 target this time as previous was on NTFS to be able to read under Windows. Just hope the backup is a bit quicker this time, as it took about 2.5 days for backing up 1.5TB :smileyfrustrated:

Is it also ok to backup/restore the configuration, or better to redo the config from scratch after the factory default?

Will the restore also restore the snapshots etc?

And then hope that by restoring the backup the memory combo isnt triggered again. :)

If you run into this if you prefer we can disable quotas from the backend so that the NAS will boot in normal mode.

Think we prefer to start from scratch then, cause who knows what the quota disable might cause in the future :P ;)

mdgm-ntgr

NETGEAR Employee Retired

Feb 04, 2016

LifeVibes wrote:

This problem appears to happen on the 104 in some cases when the filesystem is quite full after updating from 6.2.x.

I guess quite full is quite relative, as ours still has 60% free...

Looking at your logs whilst your volume is nowhere near full you do have a huge amount of metadata

btrfs-progs v4.1.2
=== filesystem /data ===
Data, single: total=1.34TiB, used=1.34TiB
System, DUP: total=8.00MiB, used=172.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=74.50GiB, used=42.39GiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=512.00MiB, used=0.00B

Here's what I get on my 516 for a comparison (though I have no snapshots):

# btrfs fi df /data
Data, single: total=12.42TiB, used=12.36TiB
System, DUP: total=32.00MiB, used=1.34MiB
Metadata, DUP: total=15.50GiB, used=14.07GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

I can see you had a number of your snapshots that were not upgraded when you updated to 6.2.4.

One would have to suspect that at least some of your data is not suited to snapshots.

LifeVibes
Tutor
Feb 05, 2016
mdgm wrote:
Looking at your logs whilst your volume is nowhere near full you do have a huge amount of metadata
I can see you had a number of your snapshots that were not upgraded when you updated to 6.2.4.
One would have to suspect that at least some of your data is not suited to snapshots.
Ah so it seems it comes down to snapshots after all.
But a bit strange that data could not be suited for snapshots. Would expect snapshots to be able to handle any data.
AFAIK we dont have any special data on this NAS, mosltly media files and
Is there some more info/investigations I need/can to do (to help finding the root)?

Otherwise was planning to finally do the factory reset today, to make use of the weekend to restore the backup.
- StephenB
  Guru - Experienced User
  Feb 05, 2016
  LifeVibes wrote:
  
  But a bit strange that data could not be suited for snapshots. Would expect snapshots to be able to handle any data.
  
  You might want to read my post on how btrfs snapshots work here: https://community.netgear.com/t5/ReadyNAS-in-Business/ReadyNAS-312-Need-Help-Understanding-Snapshots/td-p/936581
  
  Snapshots can "handle" any data, but if you are rewriting a lot of files in the share then they can use a lot of space and generate a lot of fragmentation. One example of a folder where snapshots aren't very efficient is a torrent folder. You are better off keeping snapshots disabled on shares where there is a lot of "churn".
  - LifeVibes
    Tutor
    Feb 05, 2016
    StephenB wrote:
    Snapshots can "handle" any data, but if you are rewriting a lot of files in the share then they can use a lot of space and generate a lot of fragmentation. One example of a folder where snapshots aren't very efficient is a torrent folder. You are better off keeping snapshots disabled on shares where there is a lot of "churn".
    Actually this NAS was used as safegueard storage/backup for media files etc.
    Almost all changes are just additions, hardly any deletions/replacements, but changes could be big. And also very low freq, only every x months a change/addition or so. So the "churn", as I see it, should be very low.
    But so dont immediatly see any unsuitable data use for snapshots here.