RN316 - OS 6.7.0 Beta hang - Out of Memory+2f9 on LCD

[Michael_Oz]

---

@Skywalker wrote:

It's all looking healthy

---

@Skywalker  Not any more, another hang with "out of memory+2f9' on the display.

Hard power reset.

Rebooted.

Downloaded logs.

Admin page inaccessable after a few minutes.

SSH unresponsive.

RAIDar has intermittent contact, says "management services are offline" or such.

Eventually out of memory msg again.

Rebooting, will try turning off quotas.

Logs emailed.

[Michael_Oz]

I managed to unclick the quota setting and click OK. The spinner was still spinning when the Admin page failed.

I managed to get on SSH for longer than the Admin page works.

Got a ps list, again an awful lot of Apache processes.

Then hang, "bfq_lookup_next_entity+eb" on display.

This is the same pattern as before. (except the buttons work)

Hard power reset, boot.

Lost Admin page again before I could check the quota checkbox.

Again SSH lasted longer, consecutive ps shows extra groups of Apache over time.

Using the three press power button instigates a power down, but s l o o o w l y, I imagine it is swapping.

NO, spoke too soon, it got to "shutting down goodbye", for a looong time, then out of memory again.

Hard power reset, boot....

[Michael_Oz]

I managed to get to the Volume settings, the quota checkbox was still checked.

I unchecked it, clicked OK, got the spinner, again the Admin page failed while the spinner hadn't finished...

Perhaps someone can PM me the commands to turn it off?

[mdgm-ntgr]

That's simple. if the volume is e.g. "data" (no quotes) you'd do:

# btrfs quota disable /data

The hard part is that if disabling quotas is necessary it has to be done ASAP after booting and in some cases a low-level diagnostic mode may be better for doing this than normal mode.

[Michael_Oz]

@mdgm-ntgr  Thanks. I submitted it shortly after booting, it is taking some time to return, but putty has noty timed out yet.

Disc light are still flashing and touch panel is responsive, so far...smb is dead as is html admin

If it hangs again I'll boot to Tech Support mode and let you know....

[Michael_Oz]

BTW Prior to Beta4 T206 u/g I did a defrag and balance and disc test.

[Michael_Oz]

btrfs is still running, disc lights active & display/touch responsive.

Now 01:07, retiring, will report back in the morning.

[Michael_Oz]

@mdgm-ntgr  Why does it always make a liar of you a few minutes after you post...

Hung, putty timeout, disc lights steady, touch/display unresponsive...

reboot into tech support mode.

I agree to the terms...

01:25 still retiring, will check back in the morn...

[Michael_Oz]

@mdgm-ntgr  I'm back on deck.

There was no disc activity, so I rebooted.

I entered

btrfs quota disable /N316AR6

It came back instantly, so I'm assuming the earlier command finished.

Admin page confirmed quota setting disabled.

Touch-wood it looks stable.

Do you want logs or can I do any diagnostics?

[jak0lantash]

---

@mdgm wrote:

# btrfs quota disable /data

---

I don't know about 6.7.0, but on 6.6.1, if you disable the quotas this way, and create/edit/delete a share, it automatically reactivate the quotas on the volume.

[Michael_Oz]

Thanks, I won't do that then.

----------------------

@mdgm-ntgr@Skywalker

Post Mortem

Mar 14 08:13:18 Logs Downloaded
Mar 16 20:44:58 Reboot, u/g to 6.7.0-T206
Mar 17 10:23:20 Logs Downloaded
Mar 17 10:23 Enabled quotas 'Calculating'...
Mar 17 11:19:57 Logs Downloaded
Mar 17~11:23 'Calculating' finished
Mar 17 11:23:38 Logs Downloaded (logs sent)
Mar 17 19:45:22 Logs Downloaded
Mar 18 09:09:24 Logs Downloaded

Mar 19 00:05:07 Start Backup to USB HDD ~200GB
Mar 19 00:45:53 End Backup - Success
Mar 19 00:57:53/54 2 Snaspshots pruned
Mar 19~01:04 lots of oom-killer invocations
                       (apache2 ? do these get restarted?)
                       2GB swap free?
                       readynasd 499843 total_vm

Mar 19 01:07:57 Last log before Hang
Mar 19 ?? 'out of memory+2f9' (noticed ~15:00)
Mar 19 15:14 Rebooted
Mar 19 15:15:34 Logs Downloaded (logs sent)
[numerious hangs/reboots]
Mar 19 18:07:40 Logs Downloaded
Mar 19 18:10:07 Logs Downloaded
Mar 19 23:50 Rebooted
Mar 19~23:50 SSH btrfs quota disable /N316AR6 (did not return to prompt)
Mar 20~01:11 lots of oom-killer (eating its children...)
                       1.9G swap free
                        readynasd 516271 total_vm
Mar 20 01:12:26 Last log before Hang
Mar 20~01:20 Putty timeout, touch/display unresponsive, disc light steady (hang)
Mar 20~01:20+ Boot into Tech Support Mode - left overnight [apparently nothing done?]
Mar 20 08:46:41 Reboot
Mar 20 08:48:45 ME-NAS-316A kernel: BTRFS info (device dm-0): quota is disabled
Mar 20 08:50:45 Logs Downloaded
Mar 20 10:52:11 Logs Downloaded

Summary hang ~ 40hrs after quota enabled, during which there was ~200GB backup PC to NAS, ~200 backup NAS to USB, couple of snapshots taken and pruned, little other activity.

[Michael_Oz]

p.s. If you want any of the logs shown above, give me the date/time you want.

[FramerV]

Hi Michael_Oz,

You could send it using the steps on the link:

How do I send all logs to ReadyNAS Community moderators?

Please update the thread once sent. Thanks.

Regards,

[Michael_Oz]

Latest logs sent.

Subject line:

Re: RN316 - OS 6.7.0 Beta4 T206 hang - Out of Memory+2f9 on LCD - Attn: mdgm - Attn:Skywalker - Attn: FramerV

Meanwhile I've been running a memory test, 1 pass so far, no errors.

[mdgm-ntgr]

OOM (Out of Memory) errors do not indicate a problem with the memory (RAM) at all. If you had a problem with the memory you'd get a different error.

OOM errors indicate that the system has encountered a situation where it has insufficient memory to run everything that it is being asked to so it is forced to start force shutting down services to try and free up some memory. If it has to keep on shutting down services the system will crash.

This is typically caused by trying to do too much on a system or a memory leak of some kind. Either way it's software related not a problem with the hardware.

[Michael_Oz]

Thanks I know that. Studied Computer Science & 30+ years in the industry. Just haven't bothered with Linux in depth...

I saw some errors including a stack dump mentioning btrfs_create_repair_bio, during oom; just wanted to make sure non-ecc memory was OK before other tests & maint. (7 passes 0 errors BTW)

[Michael_Oz]

When I booted into the boot menu I noticed the buttons were acting differently.

The up button was instantly responsive, the down button was not and took frequent presses to get action.

I then tried after a normal boot, it is variably responsive, up tends to be ok, down very laggy, but some time very responsive.

You may want to check those recent changes.

(Defrag was running, maybe related to the 'normal' situation, see my other post, perhaps nice levels of whatever procees handled it need tuning?)

[Michael_Oz]

Had another hang.

Display shows NAS name/ip, buttons unresponsive.

SMB down.

Web Admin down.

SSH down.

No ping response.

RAIDar doesn't see it.

Scrub was running for a few days.

No response to power button press.

Hard reset.

No mention of Scrub completing in log.

No Scrub process running.

 Logs downloaded, nothing stands out AFAIK.

Do you want logs emailed?

 edit/ BTW, do you prefer adding further problems with the same system, like this post, added to existing thread about that system, or new thread raised for subsequent problems? /edit

[Michael_Oz]

@mdgm-ntgr  does the 6.7.1.Beta2 have quota enabled? Should I be on it?

[mdgm-ntgr]

The production ReadyNAS OS 6.7.1 release is now available! There are some fixes that may help, yes. If you disable quota on 6.7.1 the change should survive share settings changes.

[jak0lantash]

---

@mdgm wrote:

The production ReadyNAS OS 6.7.1 release is now available! There are some fixes that may help, yes. If you disable quota on 6.7.1 the change should survive share settings changes.

---

It would have been great if that could have made it to the Release Notes... Same for the quota issue that resulted in 6.7.0 being pulled...