NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Solved
Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
installed 6.4.1 from 6.2.5 yesterday. All went ok. Did all the 'calculating'. worked fine.
failed overnight with out_of_memory error 354. now won't boot.
tried reinstallOS. No luck.
tota...
corinbishop wrote:
I've one more option. I have a spare, unused, unopened 104. If i move the disks over what will happen?
I thinks its clear that the performance dropoff is a software issue, so I'd be very surprised if migrating the disks help.
If migrating the disks would help with this OOM error on boot issue, it would be from moving to a model such as the RN214, not to another RN104. I haven't seen any reports of a user trying this so don't know whether it would help. However even if this did resolve it, it wouldn't deal with the underlying cause that the system configuration and/or usage was not done in a way that is recommended.
I haven't seen any reports of this problem on models other than the 100 series.
Updating the firmware is a common requirement for escalating cases. If an agent escalated a case with a minor issue on old firmware then the case would likely be de-escalated again with the agent told to get you to update the firmware so that we can rule out encountering an issue that we have already fixed.
We have general advice on updating the firmware in ReadyNAS OS 6: Firmware Upgrade Guide and Tips which includes a suggestion to update your backup (which you should be doing regularly regardless of whether you update the firmware).
The Guide also mentions some common reasons why problems might be encountered:
- Systems that are completely full.
- Systems that have high filesystem fragmentation.
- Systems that have large quantities of hourly, daily, monthly snapshots.
The first and last of these should be easy for you to verify before you update the firmware. The middle one may usually (but not always) be somewhat related to the other two, but advanced users could get a good indication by looking at the metadata usage in btrfs.log. If the metadata usage is huge then this would suggest that the way the system was configured and/or used was far from ideal.
It appears that all systems encountering this problem are affected by one or more of the issues described in these bullet points.
Some suggestions going forward would be to keep volume usage under 80%, run regular scheduled volume maintenance (defrag & balance) and to only use bit-rot protection and snapshots on shares which are suited to using those, not on every share.
Performance during a resync will be reduced. So if your volume is currently syncing you should test performance again once the sync completes.
It is important to note that the initial sync when creating a volume, is a special case when it comes to redundancy. During this initial sync if you have multiple disks and have chosen a redundant RAID option (e.g. the default X-RAID) then the volume is redundant throughout the initial sync.
Thanks.
I was just thinking I could get my system back to 6.2.5 by moving to an older system.
I think at this stage I'm going to review all my snapshots and scheduled maintenence bits. I was relying on snapshots for more long-term retrieval of photographs (in the case of clients wanting deleted images) but in realty I don't need them - I just need to protect against accidental short term deletion. So I'll shorten things.
I'm hoping that the write speed, which does appear to be a software issue, will be be addressed in future releases. I was really happy with the speed until this update and I just hope it can be fixed. The write speed is slower than my old NV+ which is shocking!
c
mdgm wrote:We have general advice on updating the firmware in ReadyNAS OS 6: Firmware Upgrade Guide and Tips which includes a suggestion to update your backup (which you should be doing regularly regardless of whether you update the firmware).
Maybe a good idea to implement some of these guidelines in a precheck in the check fw upgrade process. Should be perfectly feasible, eg to first to scrub, defrag, size check or whatever seems necessary.
- Systems that are completely full.
- Systems that have high filesystem fragmentation.
- Systems that have large quantities of hourly, daily, monthly snapshots.
The first and last of these should be easy for you to verify before you update the firmware. The middle one may usually (but not always) be somewhat related to the other two, but advanced users could get a good indication by looking at the metadata usage in btrfs.log. If the metadata usage is huge then this would suggest that the way the system was configured and/or used was far from ideal.
It appears that all systems encountering this problem are affected by one or more of the issues described in these bullet points.Some suggestions going forward would be to keep volume usage under 80%, run regular scheduled volume maintenance (defrag & balance) and to only use bit-rot protection and snapshots on shares which are suited to using those, not on every share.
Still a bit puzzled how ours ended up in the big meta data case, as the use let alone config doesnt necessarily explain it AFAIK, see above.
The only thing which wasnt done was regular defrag and balance. But as said there were rarely any deletions/mods, so should not cause too much fragmentation.
Unless the cause is the smart snapshot management, which would delete snapshots and its linked (meta)data periodically, causing fragmentation?
The snapshots being deleted doesn't effect the fragmentation of the file. If anything, it likely helps with files that have lots of extents(Fragments). If you have a file or files that have lots of edits/writes to the same file or files, this will cause fragmentation to occur on the filesystem overtime.
The Firmware guide is recommended to support the health of the filesystem while keeping performance optimal. It doesn't guarantee that the volume will be error free, but it does keep it maintained. As does smart snapshot management.
AlexPe wrote:The snapshots being deleted doesn't effect the fragmentation of the file. If anything, it likely helps with files that have lots of extents(Fragments). If you have a file or files that have lots of edits/writes to the same file or files, this will cause fragmentation to occur on the filesystem overtime.
As said, indeed no changes to files so files itself cant become more fragmented. But I guess it could cause more disk fragmentation, also depending on how btrfs is mounted (COW & autodefrag).
But so if this could not cause it, still puzzled.
StephenB wrote:
I agree it is puzzling. Do you know for certain that you were in the "big metadata case".
mdgm saw in the logs, see https://community.netgear.com/t5/Using-your-ReadyNAS/Readynas-104-won-t-boot-Error-354-out-of-memory-After-upgrade-to/m-p/1045637#M103452
StephenB wrote:
However, I am ok with a little mystery in my life, and I am happy to get to ~70 MB/s performance again on the RN102.Dont mind some mysteries, but these Id like to dig into. Also to try and minimize the risk in the future.
Anyway in the meantime I want for the factory default.
Some things I noticed:
-after restoring the saved config, the default dirs (My Pictures etc) still there which was a bit annoying as I had to remove them again.
-bitrot was not enabled anymore on the share, so seems this was not stored in or restored correctly from config
-Searched for a way to keep snapshots, but that doesnt seem to be possible? Although maybe couldve recreated it by getting the diffs/copies from all snapshots and backup them separately. And then later put them back snapshot by snapshot. But that wouldve been a hell of a job.
LifeVibes wrote:
mdgm wrote:
but advanced users could get a good indication by looking at the metadata usage in btrfs.log. If the metadata usage is huge then this would suggest that the way the system was configured and/or used was far from ideal.
Still a bit puzzled how ours ended up in the big meta data case, as the use let alone config doesnt necessarily explain it AFAIK, see above.
The only thing which wasnt done was regular defrag and balance. But as said there were rarely any deletions/mods, so should not cause too much fragmentation.
I agree it is puzzling. Do you know for certain that you were in the "big metadata case". I wasn't, and my snapshot's weren't using much space either. I'd cleared them all in November, and balance/defrag/scrub are all run. Disk space didn't drop noticeably post-reset. The only enabled app was SMB-plus (plex was installed, but was disabled).
But I still saw a marked increase in performance after the reset - nearly 2x over what I had before the reset.
In my case, the btrfs volumes were quite old (back to 6.0.0 timeframe). Netgear did make some adjustments later to the btrfs configuration, but they required a reset to get take advantage of them, and I didn't do that. So that could be part of the explanation, but it doesn't seem very convincing.
However, I am ok with a little mystery in my life, and I am happy to get to ~70 MB/s performance again on the RN102.
Following the firmware update I too am now experiencing this problem, with out of memory error 390 on Firmware 6.4.2.
I have 4 x 2TB drives in (i think) raid 5, but may be one of the standard drive array formats that come with this unit. Anyway there is around 6TB of volume space available, of which I am using just over 4TB.
The unit has been sitting under my study desk comfortable for over a year without issue, and on a move around of funiture necessitated me to reboot, and get prompted for a firmware update. Now wishing I had not!
I did notice that there were around 50 snapshot backups on various folders, but other than this there was no other backup in place, so I need to sort out access to offload just over 4TB of data so that any other work on rebuilding the drives can be done if recovery cannot be done in situe. I do not use the management console of this unit that often i.e. once when setting up, and then when doing the firmware upgrade, and the nas just has a shared folder use for document and media storage.
Any help and assistance would be greatly appriciated, as not sure where to start on my recovery journey!
Thank you for this. In addition, I went here for instructions on how to get to the RO (Read Only) option: http://kb.netgear.com/app/answers/detail/a_id/22891
Just adding to assist anyone going through this issue and come across this board
Hi,
This is the route I went too, support kindly disabled quotas and snapshots for me in support-mode (had 55 snapshots on large media shares). The unit worked fine until I started fiddling with the filesystem during backup, and it hung with out of memory 390.
I rebooted into read-only mode, and used ssh to back up my remaining files to external usb drives (it seems you cannot configure backup jobs with the web-interface in read-only mode).
After a factory reset and running 6.4.2, it has been running for a week without crashing, so all seems good. Not finished the data-recovery yet but I'm optimistic.
If in doubt, use your NAS in read-only mode, and don't try to defrag or scrub, it will probably crash until it completes the process if you have the same issues (as it seems). Tech support can help you fix the snapshots so you can boot normally, but after that, turn off all services and backup asap.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
