Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717

[corinbishop]

Thanks. 

I was just thinking I could get my system back to 6.2.5 by moving to an older system. 

I think at this stage I'm going to review all my snapshots and scheduled maintenence bits. I was relying on snapshots for more long-term retrieval of photographs (in the case of clients wanting deleted images) but in realty I don't need them - I just need to protect against accidental short term deletion. So I'll shorten things. 

I'm hoping that the write speed, which does appear to be a software issue, will be be addressed in future releases. I was really happy with the speed until this update and I just hope it can be fixed. The write speed is slower than my old NV+ which is shocking!

c

[StephenB]

---

@StephenB wrote:
 It should be finished by Monday, then I'll run a CIFS benchmark.

---

Just to follow up - post factory reset, my RN102 is back up to 70-75 MB/s for both read and write.  The write speed is actually faster than what I had with 6.2.x.

(I'm not suggesting that you do another reset; this just confirms that my performance drop-off had a different cause from yours).

[corinbishop]

Thanks Stephen, 

That's interesting. I've not done the full wipe/reset/restore yet. Still planning it. I'm going to go through some disk maintenence first (defreag, scrub, balance etc.) just to make sure that everything is as good as it can be. If speed doesn't improve then I'll do the full wipe/reset/restore.

[LifeVibes]

---

@StephenB wrote:

---

---

---

With deletions/replacements, the original data stays in the snapshot, additions have no effect.  No fragmentation occurs.  Tagging a media file (after the next snapshot is taken) will result in fragmentation.

I agree that your usage sounds good for snapshots, though I do suggest going in periodically and deleting the oldest snapshot in every share.  I keep mine for 3 months, and go in about one a month.

---

Thats als how I understood it. We normally also dont tag media files (certainly not after backing it up to the NAS).

@StephenB wrote:

I'd love to see the firmware manage max retention time automatically, but it doesn't.

Isnt it supposed to already have some automatic (retention) management, see SM section "Smart Snapshot Management" or do you mean something different?

I guess its kinda working. Cause we have daily snapshot enabled since long, and only 36 snapshots (5 months protection) exist.

But maybe its not working good enough, which caused the big meta data mystery on my system.

[LifeVibes]

---

@qborosfinest wrote:

---

Perhaps not the forum to vent but this was a bit of frustrating experience, and again a lesson why you shouldn't upgrade fw blindly, neither should support force it down your throat to support you if your product is mostly working.

---

What else is a forum for?

But I guess its also a bit of a frustration for the Netgear support people ;).

Speaking of venting frustrations, the Rich Text editor on this board never seems to do what I want, esp related to quoted text

[LifeVibes]

---

@mdgm wrote:

We have general advice on updating the firmware in ReadyNAS OS 6: Firmware Upgrade Guide and Tips which includes a suggestion to update your backup (which you should be doing regularly regardless of whether you update the firmware).

---

Maybe a good idea to implement some of these guidelines in a precheck in the check fw upgrade process. Should be perfectly feasible, eg to first to scrub, defrag, size check or whatever seems necessary.

• Systems that are completely full.
• Systems that have high filesystem fragmentation.
• Systems that have large quantities of hourly, daily, monthly snapshots.

The first and last of these should be easy for you to verify before you update the firmware. The middle one may usually (but not always) be somewhat related to the other two, but advanced users could get a good indication by looking at the metadata usage in btrfs.log. If the metadata usage is huge then this would suggest that the way the system was configured and/or used was far from ideal.

It appears that all systems encountering this problem are affected by one or more of the issues described in these bullet points. 

Some suggestions going forward would be to keep volume usage under 80%, run regular scheduled volume maintenance (defrag & balance) and to only use bit-rot protection and snapshots on shares which are suited to using those, not on every share.

Still a bit puzzled how ours ended up in the big meta data case, as the use let alone config doesnt necessarily explain it AFAIK, see above.

The only thing which wasnt done was regular defrag and balance. But as said there were rarely any deletions/mods, so should not cause too much fragmentation.

Unless the cause is the smart snapshot management, which would delete snapshots and its linked (meta)data periodically, causing fragmentation?

[AlexPe]

The snapshots being deleted doesn't effect the fragmentation of the file. If anything, it likely helps with files that have lots of extents(Fragments). If you have a file or files that have lots of edits/writes to the same file or files, this will cause fragmentation to occur on the filesystem overtime. 

The Firmware guide is recommended to support the health of the filesystem while keeping performance optimal. It doesn't guarantee that the volume will be error free, but it does keep it maintained. As does smart snapshot management.

[StephenB]

---

@LifeVibes wrote:

---

@mdgm wrote:

but advanced users could get a good indication by looking at the metadata usage in btrfs.log. If the metadata usage is huge then this would suggest that the way the system was configured and/or used was far from ideal.

Still a bit puzzled how ours ended up in the big meta data case, as the use let alone config doesnt necessarily explain it AFAIK, see above.

The only thing which wasnt done was regular defrag and balance. But as said there were rarely any deletions/mods, so should not cause too much fragmentation.

---

I agree it is puzzling.  Do you know for certain that you were in the "big metadata case".  I wasn't, and my snapshot's weren't using much space either.  I'd cleared them all in November, and balance/defrag/scrub are all run.  Disk space didn't drop noticeably post-reset.  The only enabled app was SMB-plus (plex was installed, but was disabled).

But I still saw a marked increase in performance after the reset - nearly 2x over what I had before the reset.

In my case, the btrfs volumes were quite old (back to 6.0.0 timeframe).  Netgear did make some adjustments later to the btrfs configuration, but they required a reset to get take advantage of them, and I didn't do that.  So that could be part of the explanation, but it doesn't seem very convincing.

However, I am ok with a little mystery in my life, and I am happy to get to ~70 MB/s performance again on the RN102.

[LifeVibes]

---

@AlexPe wrote:

The snapshots being deleted doesn't effect the fragmentation of the file. If anything, it likely helps with files that have lots of extents(Fragments). If you have a file or files that have lots of edits/writes to the same file or files, this will cause fragmentation to occur on the filesystem overtime. 

As said, indeed no changes to files so files itself cant become more fragmented. But I guess it could cause more disk fragmentation, also depending on how btrfs is mounted (COW & autodefrag).

But so if this could not cause it, still puzzled.

@StephenB wrote:

I agree it is puzzling.  Do you know for certain that you were in the "big metadata case".

 mdgm saw in the logs, see https://community.netgear.com/t5/Using-your-ReadyNAS/Readynas-104-won-t-boot-Error-354-out-of-memory...

@StephenB wrote:
However, I am ok with a little mystery in my life, and I am happy to get to ~70 MB/s performance again on the RN102.

Dont mind some mysteries, but these Id like to dig into. Also to try and minimize the risk in the future.

Anyway in the meantime I want for the factory default.

Some things I noticed:

-after restoring the saved config, the default dirs (My Pictures etc) still there which was a bit annoying as I had to remove them again.

-bitrot was not enabled anymore on the share, so seems this was not stored in or restored correctly from config

-Searched for a way to keep snapshots, but that doesnt seem to be possible? Although maybe couldve recreated it by getting the diffs/copies from all snapshots and backup them separately. And then later put them back snapshot by snapshot. But that wouldve been a hell of a job.

[StephenB]

---

@LifeVibes wrote:

---

-bitrot was not enabled anymore on the share, so seems this was not stored in or restored correctly from config

---

That setting was preserved on my shares.

---

@LifeVibes wrote:

---

-Searched for a way to keep snapshots, but that doesnt seem to be possible?

---

I don't think that is easy.  I think you maybe could use btfrs send and btrfs receive to copy the full subvolume to an external drive, and then do it in the opposite direction after the factory reset.  But those aren't commands I've played with (and they might not preserve snapshots).

[corinbishop]

my update:

I'm still running on tech support versions of 6.4.2. I was up and running again after snapshots were deleted and quota's disabled. 

I'm ultimately planning a wipe/reset/restore but in the mean time I have to balance this with continuing to work!

I'm seeing very poor write/read performance - approv 6Mbs write, 27Mbs read. Should be nearer 40 both ways. 

I've done a defrag. Completed ok. No change in performance. 

Now going to try scrub and balance. Which one first? or does it matter?

[StephenB]

---

@corinbishop wrote:
 Now going to try scrub and balance. Which one first? or does it matter?

---

It shouldn't matter, though I'd be inclined to balance first.

[corinbishop]

Thanks

[corinbishop]

Balance done. Took 25mins ish.
Kicked off a scub last night. This morning it's done 1.35%! Slooooowww

[corinbishop]

scrub started 8pm Friday.... at 19% now... is it normal to be that slow?

[StephenB]

How big is the data volume? 

It does sound slow, but if it is making progress I'd let it continue.

[corinbishop]

8tb. 5tb used. 

four disk x-raid

4tb (recently used to replace broken 3tb drive - will eventually upgrade all drives to 4tb)

3tb

3tb

3tb

[StephenB]

I have a 6 TB jbod volume (4 TB full) with a scrub in progress on an RN202.  

It's about 10% complete (approximately 1% per hour so far).  Volume fullness does matter with a scrub (unlike a resync).

I'd expect faster performance on the RN104 than you are seeing.  If you have disk spindown enabled, perhaps turn that off for now.

[ctw1969]

Following the firmware update I too am now experiencing this problem, with out of memory error 390 on Firmware 6.4.2.

I have 4 x 2TB drives in (i think) raid 5, but may be one of the standard drive array formats that come with this unit. Anyway there is around 6TB of volume space available, of which I am using just over 4TB.

The unit has been sitting under my study desk comfortable for over a year without issue, and on a move around of funiture necessitated me to reboot, and get prompted for a firmware update. Now wishing I had not!

I did notice that there were around 50 snapshot backups on various folders, but other than this there was no other backup in place, so I need to sort out access to offload just over 4TB of data so that any other work on rebuilding the drives can be done if recovery cannot be done in situe. I do not use the management console of this unit that often i.e. once when setting up, and then when doing the firmware upgrade, and the nas just has a shared folder use for document and media storage.

Any help and assistance would be greatly appriciated, as not sure where to start on my recovery journey!

[LifeVibes · Accepted solution]

You could try RO mode, see https://community.netgear.com/t5/Using-your-ReadyNAS/Readynas-104-won-t-boot-Error-354-out-of-memory...

[ctw1969]

Thank you for this. In addition, I went here for instructions on how to get to the RO (Read Only) option: http://kb.netgear.com/app/answers/detail/a_id/22891 

Just adding to assist anyone going through this issue and come across this board

[nilsel]

Hi,

This is the route I went too, support kindly disabled quotas and snapshots for me in support-mode (had 55 snapshots on large media shares). The unit worked fine until I started fiddling with the filesystem during backup, and it hung with out of memory 390.

I rebooted into read-only mode, and used ssh to back up my remaining files to external usb drives (it seems you cannot configure backup jobs with the web-interface in read-only mode).

After a factory reset and running 6.4.2, it has been running for a week without crashing, so all seems good. Not finished the data-recovery yet but I'm optimistic.

If in doubt, use your NAS in read-only mode, and don't try to defrag or scrub, it will probably crash until it completes the process if you have the same issues (as it seems). Tech support can help you fix the snapshots so you can boot normally, but after that, turn off all services and backup asap.

[mdgm-ntgr]

If you'd like to disable quotas send me a PM and I can let you know what I need to try to do that.

[LifeVibes]

---

@nilsel wrote:

I rebooted into read-only mode, and used ssh to back up my remaining files to external usb drives (it seems you cannot configure backup jobs with the web-interface in read-only mode).

I was able to configure the backup job via web interface in RO mode (didnt have ssh enabled).

[corinbishop]

ok. update:

defrag, balance and scrub all done. unit restarted. 

still seeing awful write speeds < 9MB/s, read at 25-27 

I don't know what else to do. This is unusable as for as I'm concerned and it's slower than my old NV+

If someone can tell me that there is a good chance it will be better if I wipe/reset/restore then I'll do it. Otherwise if not then I'll be buying a new machine (not netgear). 

Regretting upgrading the software. Life was perfect on 6.2.5. Write speeds 40MBs, no stability issues. 

I've spent so much time on this device since the upgrade that I can no longer justify the time. 

However, like I said above, if something really thinks it will be better with a complete wipe/reset/restore then it's the one last thing I'll do.