- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
Thank you for your answers.
I tried booting in the read-only mode, and the data is all still there, very happy and now backing this up as we speak (write).
As for contacting support, i'm following this thread and hoping when there is a sollution that i can take part in it.
I don't think contacting support would help speed things up as multiple people experience this problem.
Thank you also for caring.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
If you've run into this and haven't sent me your logs yet can you please do so: http://kb.netgear.com/app/answers/detail/a_id/25625/~/how-do-i-send-readynas-system-logs-to-netgear-...
We can try disabling quotas from the backend if you like, but I think the best solution would be to backup your data, update to 6.4.2 and do a factory default (wipes all data, settings, everything) on that and restore your data from backup.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
Hi mgdm,
I'm going to do the wipe etc. Can I just confirm the procedure to follow, bearing in mind that I appear to be on a special version of OS 6.4.2-T72.
1. Backup (in my case it'll be 2 backups, one to an old NV+ and one to external HD's).
2. Settings>Configuration Backup>Everything
3. Perform the Factory Default as in http://kb.netgear.com/app/answers/detail/a_id/20898/session/L2F2LzEvdGltZS8xNDU0NTg1MTU4L3NpZC9saUJp...
4. After the reset has been performed, restore the configuration settings taken in step 2 (restart at all?).
5. Restore data from back drives.
6. Have a beer (or 2).
Are there any issues regarding my asymetrical drives? I have 4TB, 3TB, 3TB, 3TB.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
@corinbishop wrote:
Hi mgdm,
I'm going to do the wipe etc. Can I just confirm the procedure to follow, bearing in mind that I appear to be on a special version of OS 6.4.2-T72.
1. Backup (in my case it'll be 2 backups, one to an old NV+ and one to external HD's).
2. Settings>Configuration Backup>Everything
3. Perform the Factory Default as in http://kb.netgear.com/app/answers/detail/a_id/20898/session/L2F2LzEvdGltZS8xNDU0NTg1MTU4L3NpZC9saUJp...
4. After the reset has been performed, restore the configuration settings taken in step 2 (restart at all?).
5. Restore data from back drives.
6. Have a beer (or 2).
Are there any issues regarding my asymetrical drives? I have 4TB, 3TB, 3TB, 3TB.
Step 1.5 - uninstall all apps
Step 3.5 - reinstall all apps.
The most convenient way to do the default is from the web ui (systems->settings)
If you are using flexraid, you will need to destroy the xraid volume created by the default, and recreate your old volumes. You don't need to wait for the xraid volume to sync.
Your drive configuration won't be a problem, though if you are using the normal raid5/xraid you are losing 1 TB of space.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
@StephenB wrote:
@corinbishop wrote:Hi mgdm,
I'm going to do the wipe etc. Can I just confirm the procedure to follow, bearing in mind that I appear to be on a special version of OS 6.4.2-T72.
1. Backup (in my case it'll be 2 backups, one to an old NV+ and one to external HD's).
2. Settings>Configuration Backup>Everything
3. Perform the Factory Default as in http://kb.netgear.com/app/answers/detail/a_id/20898/session/L2F2LzEvdGltZS8xNDU0NTg1MTU4L3NpZC9saUJp...
4. After the reset has been performed, restore the configuration settings taken in step 2 (restart at all?).
5. Restore data from back drives.
6. Have a beer (or 2).
Are there any issues regarding my asymetrical drives? I have 4TB, 3TB, 3TB, 3TB.
Step 1.5 - uninstall all apps
Step 3.5 - reinstall all apps.
The most convenient way to do the default is from the web ui (systems->settings)
If you are using flexraid, you will need to destroy the xraid volume created by the default, and recreate your old volumes. You don't need to wait for the xraid volume to sync.
Your drive configuration won't be a problem, though if you are using the normal raid5/xraid you are losing 1 TB of space.
Thanks a mill. I'm using Xraid and planning to eventually upgrade all the drives to 4TB ones. A 3TB failed recently (one of the infamous Seagate 3TB **bleep**e drives!) and I replaced it with 4TB.
Just to be sure: a screenshot of the volumes page is here: https://www.dropbox.com/s/uve7gitjn56sxmw/Screenshot%202016-02-04%2012.00.20.png?dl=0
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
@corinbishop wrote:
Just to be sure: a screenshot of the volumes page is here: https://www.dropbox.com/s/uve7gitjn56sxmw/Screenshot%202016-02-04%2012.00.20.png?dl=0
You are running normal xraid. Post back on how it goes (and whether the reset resolves your problem).
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
Thanks.
I'm currently running a stable system. Support got me working ok. I'm doing the reset to provide a fresh, new starting point. The only problem I've been left with is poor write speed. The other recent posts are from someone who has the same problem but has not managed to get back online yet. So he's doing the reset to get him running, I'm doing it for long term certainty!
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
I'm working on 2nd backup right now, then going with factory reset per Support recommendation - they have not provided any alternative workarounds. I will post my findings after. In some regard, glad we're not alone and it's impacting many. Let's hold hands..
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
We are certainly not the only ones, just do a search on "104 boot" and youll find a lot of similar reports of 104 (and other RN) not booting after 6.4 updates, prolly all more or less related.
Only "real solution" seems to be factory reset.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
@LifeVibes wrote:
just do a search on "104 boot" and youll find a lot of similar reports of 104 (and other RN) not booting after 6.4 updates, prolly all more or less related.
You are certainly not alone, but there are multiple (and unrelated) issues. One is that the boot loader is updated when you first move to 6.4.x, and that update in particular sometimes fails.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
@LifeVibes wrote:
This problem appears to happen on the 104 in some cases when the filesystem is quite full after updating from 6.2.x.I guess quite full is quite relative, as ours still has 60% free...
Looking at your logs whilst your volume is nowhere near full you do have a huge amount of metadata
btrfs-progs v4.1.2 === filesystem /data === Data, single: total=1.34TiB, used=1.34TiB System, DUP: total=8.00MiB, used=172.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=74.50GiB, used=42.39GiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=512.00MiB, used=0.00B
Here's what I get on my 516 for a comparison (though I have no snapshots):
# btrfs fi df /data Data, single: total=12.42TiB, used=12.36TiB System, DUP: total=32.00MiB, used=1.34MiB Metadata, DUP: total=15.50GiB, used=14.07GiB GlobalReserve, single: total=512.00MiB, used=0.00B
I can see you had a number of your snapshots that were not upgraded when you updated to 6.2.4.
One would have to suspect that at least some of your data is not suited to snapshots.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
@mdgm wrote:Looking at your logs whilst your volume is nowhere near full you do have a huge amount of metadata
I can see you had a number of your snapshots that were not upgraded when you updated to 6.2.4.One would have to suspect that at least some of your data is not suited to snapshots.
Ah so it seems it comes down to snapshots after all.
But a bit strange that data could not be suited for snapshots. Would expect snapshots to be able to handle any data.
AFAIK we dont have any special data on this NAS, mosltly media files and
Is there some more info/investigations I need/can to do (to help finding the root)?
Otherwise was planning to finally do the factory reset today, to make use of the weekend to restore the backup.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
@LifeVibes wrote:
But a bit strange that data could not be suited for snapshots. Would expect snapshots to be able to handle any data.
You might want to read my post on how btrfs snapshots work here: https://community.netgear.com/t5/ReadyNAS-in-Business/ReadyNAS-312-Need-Help-Understanding-Snapshots...
Snapshots can "handle" any data, but if you are rewriting a lot of files in the share then they can use a lot of space and generate a lot of fragmentation. One example of a folder where snapshots aren't very efficient is a torrent folder. You are better off keeping snapshots disabled on shares where there is a lot of "churn".
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
@StephenB wrote:
Snapshots can "handle" any data, but if you are rewriting a lot of files in the share then they can use a lot of space and generate a lot of fragmentation. One example of a folder where snapshots aren't very efficient is a torrent folder. You are better off keeping snapshots disabled on shares where there is a lot of "churn".
Actually this NAS was used as safegueard storage/backup for media files etc.
Almost all changes are just additions, hardly any deletions/replacements, but changes could be big. And also very low freq, only every x months a change/addition or so. So the "churn", as I see it, should be very low.
But so dont immediatly see any unsuitable data use for snapshots here.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
@LifeVibes wrote:
Almost all changes are just additions, hardly any deletions/replacements, but changes could be big. And also very low freq, only every x months a change/addition or so. So the "churn", as I see it, should be very low.
But so dont immediatly see any unsuitable data use for snapshots here.
With deletions/replacements, the original data stays in the snapshot, additions have no effect. No fragmentation occurs. Tagging a media file (after the next snapshot is taken) will result in fragmentation.
I agree that your usage sounds good for snapshots, though I do suggest going in periodically and deleting the oldest snapshot in every share. I keep mine for 3 months, and go in about one a month.
I'd love to see the firmware manage max retention time automatically, but it doesn't.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
There are some enhancements to snapshots in the pipeline for our next major release.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
After successful factory reset, I am back online and working again. So far stable but definitely degradation in write speed from where I was prior to 6.4.1.
Here some points below:
- Noticed about 20-25 MB/s write speed with rsync via NFS mount. Previously I would see burst of 80-100 MB/s and settle in at 30-40 MB/s. NFS mount is using async and the share has bitrot protect (copy-on-write enabled) just like pre 6.4.1.
- Oddly enough, seeing VERY slow write speed 5-15 MB/s when rsync between USB3 drive mounted to NAS to RAID-5 volume (executed all within NAS). I didnt test this pre 6.4.1.
- Only running basic (smb,afp,nfs,ssh,http). No AV or other apps.
Other info:
- RAID-5, Volume resyncing in progress, looks like it's going to take days/week to complete.Maybe longer since I already started data restoration.
- Snapshots and quotas disabled (from UI). Snapshots were enabled in the previous config before this mess began.
- Majority of data is large video files.
Let me know if I can share anything else to help or run any tests..
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
@qborosfinest wrote:After successful factory reset, I am back online and working again. So far stable but definitely degradation in write speed from where I was prior to 6.4.1.
Here some points below:
- Noticed about 20-25 MB/s write speed with rsync via NFS mount. Previously I would see burst of 80-100 MB/s and settle in at 30-40 MB/s. NFS mount is using async and the share has bitrot protect (copy-on-write enabled) just like pre 6.4.1.
- Oddly enough, seeing VERY slow write speed 5-15 MB/s when rsync between USB3 drive mounted to NAS to RAID-5 volume (executed all within NAS). I didnt test this pre 6.4.1.
- Only running basic (smb,afp,nfs,ssh,http). No AV or other apps.
Other info:
- RAID-5, Volume resyncing in progress, looks like it's going to take days/week to complete.Maybe longer since I already started data restoration.
- Snapshots and quotas disabled (from UI). Snapshots were enabled in the previous config before this mess began.
- Majority of data is large video files.
Let me know if I can share anything else to help or run any tests..
It's interesting you're seeing the same speed degredation as me. Sounds like we have a similar setup. Was it a full reset/wipe/new etc? I was going to do that in the hope the speed issue would be fixed but I won't both if there's going to be no change.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
If the resync is still in process, then the performance of course suffers, so I think you probably need to wait until the resync is done and then post back.
I did a factory reset on my RN102 a couple days ago (no apps installed at this point). It's jbod, not xraid - so there is no resync. I've been restoring the data with rsync backup from my Pro-6, and am seeing restore speeds of about 1 TB/day (which amounts to 12 MB/s). I don't recall what the rsync speed was before, so I don't know if that's a drop-off or not (rsync is computationally intensive).
It should be finished by Monday, then I'll run a CIFS benchmark.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
@StephenB wrote:If the resync is still in process, then the performance of course suffers, so I think you probably need to wait until the resync is done and then post back.
I did a factory reset on my RN102 a couple days ago (no apps installed at this point). It's jbod, not xraid - so there is no resync. I've been restoring the data with rsync backup from my Pro-6, and am seeing restore speeds of about 1 TB/day (which amounts to 12 MB/s). I don't recall what the rsync speed was before, so I don't know if that's a drop-off or not (rsync is computationally intensive).
It should be finished by Monday, then I'll run a CIFS benchmark.
I've not transfers or backups going on at all and seeing poor performance.
Services on are: SMB, AFP, FTP, Rsync, HTTP, HTTPs, SSH.
No antivirus. No apps.
best write performance from a windows device has been just under 9Mb/s. Best write performance has been 28.
previously at least 30 write, 40 read.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
@corinbishop wrote:It's interesting you're seeing the same speed degredation as me. Sounds like we have a similar setup. Was it a full reset/wipe/new etc? I was going to do that in the hope the speed issue would be fixed but I won't both if there's going to be no change.
Correct, full reset/wipe was done. Will see if performance is better after resync finishes although I can see resync process is intelligent enough to pause to allow NFS activity so I doubt it. Speeds through NFS are also dropped down to 10-15 MB/s now..Are you considering downgrading perhaps?
Perhaps not the forum to vent but this was a bit of frustrating experience, and again a lesson why you shouldn't upgrade fw blindly, neither should support force it down your throat to support you if your product is mostly working.
I started with a NFS performance/stability issue and upgraded to 6.4.1 because my issue would not be escalated otherwise. This of course turned into 354 out of memory and all the headaches of a factory reset followed. Now I'm left with less than half the read/write speeds I had when I started and I'll spend the next couple of days restoring.
In hindsight, that initial issue was relatively minor and not worth all this.
But in the end, I do understand this is a consumer product, and I should not expect the same level of support or diligence as I'm used to in the enterprise world at work.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
I've one more option. I have a spare, unused, unopened 104. If i move the disks over what will happen?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
@corinbishop wrote:
I've one more option. I have a spare, unused, unopened 104. If i move the disks over what will happen?
I thinks its clear that the performance dropoff is a software issue, so I'd be very surprised if migrating the disks help.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Readynas 104 won't boot. Error 354 out_of_memory. After upgrade to 6.4.1 #26323717
If migrating the disks would help with this OOM error on boot issue, it would be from moving to a model such as the RN214, not to another RN104. I haven't seen any reports of a user trying this so don't know whether it would help. However even if this did resolve it, it wouldn't deal with the underlying cause that the system configuration and/or usage was not done in a way that is recommended.
I haven't seen any reports of this problem on models other than the 100 series.
Updating the firmware is a common requirement for escalating cases. If an agent escalated a case with a minor issue on old firmware then the case would likely be de-escalated again with the agent told to get you to update the firmware so that we can rule out encountering an issue that we have already fixed.
We have general advice on updating the firmware in ReadyNAS OS 6: Firmware Upgrade Guide and Tips which includes a suggestion to update your backup (which you should be doing regularly regardless of whether you update the firmware).
The Guide also mentions some common reasons why problems might be encountered:
- Systems that are completely full.
- Systems that have high filesystem fragmentation.
- Systems that have large quantities of hourly, daily, monthly snapshots.
The first and last of these should be easy for you to verify before you update the firmware. The middle one may usually (but not always) be somewhat related to the other two, but advanced users could get a good indication by looking at the metadata usage in btrfs.log. If the metadata usage is huge then this would suggest that the way the system was configured and/or used was far from ideal.
It appears that all systems encountering this problem are affected by one or more of the issues described in these bullet points.
Some suggestions going forward would be to keep volume usage under 80%, run regular scheduled volume maintenance (defrag & balance) and to only use bit-rot protection and snapshots on shares which are suited to using those, not on every share.
Performance during a resync will be reduced. So if your volume is currently syncing you should test performance again once the sync completes.
It is important to note that the initial sync when creating a volume, is a special case when it comes to redundancy. During this initial sync if you have multiple disks and have chosen a redundant RAID option (e.g. the default X-RAID) then the volume is redundant throughout the initial sync.