Re: WAX610Y no longer responsive on network - syslog errors repeating -> NBUF alloc failed 1210

SquareJ · ‎2022-04-11

My WAX610y stops responding occasionally so I configured a syslog server and am now seeing these after rebooting about 24 hours ago. It is running the latest firmware: 9.6.2.6. I also did a factory reset yesterday and uploaded the last saved backup. Please help.

I randomly found this which may provide some insight

Apr 11 09:17:37 10.100.10.5 kernel: [82784.399506] Normal free:6104kB min:2488kB low:3108kB high:3732kB active_anon:13312kB inactive_anon:5692kB active_file:1552kB inac
tive_file:1336kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:402432kB managed:387956kB mlocked:0kB dirty:0kB writeback:0kB mapped:4084kB shmem:6008kB
slab_reclaimable:4044kB slab_unreclaimable:152332kB kernel_stack:1488kB pagetables:660kB unstable:0kB bounce:0kB free_pcp:2564kB local_pcp:676kB free_cma:0kB writeback_
tmp:0kB pages_scanned:356 all_unreclaimable? no
Apr 11 09:17:37 10.100.10.5 kernel: [82784.399514] lowmem_reserve[]: 0 0 0
Apr 11 09:17:37 10.100.10.5 kernel: [82784.399524] Normal: 380*4kB (UME) 473*8kB (UME) 45*16kB (UME) 3*32kB (ME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096
kB = 6120kB
Apr 11 09:17:37 10.100.10.5 kernel: [82784.399562] 2238 total pagecache pages
Apr 11 09:17:37 10.100.10.5 kernel: [82784.399567] 100608 pages RAM
Apr 11 09:17:37 10.100.10.5 kernel: [82784.399571] 0 pages HighMem/MovableOnly
Apr 11 09:17:37 10.100.10.5 kernel: [82784.399576] 3619 pages reserved
Apr 11 09:17:37 10.100.10.5 kernel: [82784.399581] 0 pages cma reserved

And then this

Apr 11 09:17:37 10.100.10.5 kernel: [82784.815598] Exception stack(0x80b01f60 to 0x80b01fa8)
Apr 11 09:17:37 10.100.10.5 kernel: [82784.815610] 1f60: 00000001 00000000 00000000 80103360 00000000 80b00000 00000000 80a3530c
Apr 11 09:17:37 10.100.10.5 kernel: [82784.815622] 1f80: 80b01fb8 80a2ba30 80b61010 00000000 00000000 80b01fb0 80110d5c 80110d60
Apr 11 09:17:37 10.100.10.5 kernel: [82784.815629] 1fa0: 60000013 ffffffff

Finally, nothing but lines like these from here on
Apr 11 10:01:43 10.100.10.5 kernel: [85429.889202] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3782
Apr 11 10:01:43 10.100.10.5 kernel: [85430.401189] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3782
Apr 11 10:01:44 10.100.10.5 kernel: [85430.913209] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3782
Apr 11 10:01:44 10.100.10.5 kernel: [85431.425188] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3782
Apr 11 10:01:44 10.100.10.5 kernel: [85431.937206] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3782
Apr 11 10:01:45 10.100.10.5 kernel: [85432.449179] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3782
Apr 11 10:01:45 10.100.10.5 kernel: [85432.961182] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3782
Apr 11 10:01:47 10.100.10.5 kernel: [85433.473149] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3782
Apr 11 10:01:48 10.100.10.5 kernel: [85433.985197] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3782
Apr 11 10:01:48 10.100.10.5 kernel: [85434.497197] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3782
Apr 11 10:01:48 10.100.10.5 kernel: [85435.009196] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3782
Apr 11 10:01:48 10.100.10.5 kernel: [85435.521174] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3782
Apr 11 10:01:49 10.100.10.5 kernel: [85436.033179] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3782

RaghuHR · ‎2022-04-11

Hi @SquareJ

Please download the detailed logs from the monitoring page and send me link to download via PM.

Also share your network topology.

Thanks,

Raghu

SquareJ · ‎2022-04-18

Just noticed this in my latest syslog, page allocation failure. Not sure if relevent.

Apr 17 20:11:41 10.100.10.5 kernel: [82409.972239] swapper/0: page allocation failure: order:6, mode:0x208c020
Apr 17 20:11:41 10.100.10.5 kernel: [82409.972259] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P 4.4.60 #5
Apr 17 20:11:41 10.100.10.5 kernel: [82409.972265] Hardware name: Generic DT based system
Apr 17 20:11:41 10.100.10.5 kernel: [82409.972295] [<80116950>] (unwind_backtrace) from [<801132a4>] (show_stack+0x10/0x14)
Apr 17 20:11:41 10.100.10.5 kernel: [82409.972313] [<801132a4>] (show_stack) from [<803080bc>] (dump_stack+0x7c/0x9c)
Apr 17 20:11:41 10.100.10.5 kernel: [82409.972331] [<803080bc>] (dump_stack) from [<8019c960>] (warn_alloc_failed+0xe4/0x110)
Apr 17 20:11:41 10.100.10.5 kernel: [82409.972347] [<8019c960>] (warn_alloc_failed) from [<8019f718>] (__alloc_pages_nodemask+0x6e8/0x768)
Apr 17 20:11:41 10.100.10.5 kernel: [82409.972362] [<8019f718>] (__alloc_pages_nodemask) from [<8019f964>] (alloc_kmem_pages+0x18/0xac)
Apr 17 20:11:41 10.100.10.5 kernel: [82409.972376] [<8019f964>] (alloc_kmem_pages) from [<801b252c>] (kmalloc_order+0x10/0x20)
Apr 17 20:11:41 10.100.10.5 kernel: [82409.972421] [<801b252c>] (kmalloc_order) from [<7fd4e63c>] (__qdf_mem_malloc+0x28/0x90 [qdf])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.972649] [<7fd4e63c>] (__qdf_mem_malloc [qdf]) from [<a81a6524>] (dp_peer_create_wifi3+0x25c/0x74c [wifi_3_0])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.972976] [<a81a6524>] (dp_peer_create_wifi3 [wifi_3_0]) from [<a88008b8>] (ol_ath_node_alloc+0x1e8/0x830 [qca_ol])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.973567] [<a88008b8>] (ol_ath_node_alloc [qca_ol]) from [<a5b65880>] (ieee80211_alloc_node+0x98/0x460 [umac])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.974180] [<a5b65880>] (ieee80211_alloc_node [umac]) from [<a5cacad8>] (wlan_mlme_peer_obj_create_handler+0x74/0x16c [umac])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.974720] [<a5cacad8>] (wlan_mlme_peer_obj_create_handler [umac]) from [<a5ca67c8>] (wlan_objmgr_peer_obj_create+0x138/0x3d0 [umac])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.975252] [<a5ca67c8>] (wlan_objmgr_peer_obj_create [umac]) from [<a5cacc18>] (sm_dbg_fs_open+0x48/0x13c [umac])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.975748] [<a5cacc18>] (sm_dbg_fs_open [umac]) from [<a5b67968>] (ieee80211_dup_bss+0xc/0xb0 [umac])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.976224] [<a5b67968>] (ieee80211_dup_bss [umac]) from [<a5bb2a54>] (mlme_recv_auth_ap_defer+0xb8/0xc40 [umac])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.976704] [<a5bb2a54>] (mlme_recv_auth_ap_defer [umac]) from [<a5bb43e0>] (mlme_recv_auth_ap+0x730/0x9d8 [umac])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.977178] [<a5bb43e0>] (mlme_recv_auth_ap [umac]) from [<a5bae004>] (ieee80211_mlme_recv_auth+0x94/0xbc [umac])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.977657] [<a5bae004>] (ieee80211_mlme_recv_auth [umac]) from [<a5b951d4>] (ieee80211_send_auth+0x888/0xd90 [umac])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.978109] [<a5b951d4>] (ieee80211_send_auth [umac]) from [<a5b99b18>] (ieee80211_recv_mgmt+0xc1c/0x372c [umac])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.978570] [<a5b99b18>] (ieee80211_recv_mgmt [umac]) from [<a5c27a38>] (ieee80211_input+0x9f8/0xea8 [umac])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.979053] [<a5c27a38>] (ieee80211_input [umac]) from [<a5c2808c>] (ieee80211_input_all+0x1a4/0x4e8 [umac])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.979533] [<a5c2808c>] (ieee80211_input_all [umac]) from [<a5c28528>] (ieee80211_mgmt_input+0x158/0x174 [umac])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.980048] [<a5c28528>] (ieee80211_mgmt_input [umac]) from [<a5ca8cd8>] (tgt_mgmt_txrx_rx_frame_handler+0x5f4/0xc98 [umac])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.980527] [<a5ca8cd8>] (tgt_mgmt_txrx_rx_frame_handler [umac]) from [<a8818810>] (ol_ath_mgmt_handler+0x4e4/0x704 [qca_ol])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.980792] [<a8818810>] (ol_ath_mgmt_handler [qca_ol]) from [<a8818b94>] (ol_ath_mgmt_rx_event_handler+0x164/0x2d4 [qca_ol])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.981045] [<a8818b94>] (ol_ath_mgmt_rx_event_handler [qca_ol]) from [<a8857af8>] (__wmi_control_rx+0x134/0x354 [qca_ol])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.981309] [<a8857af8>] (__wmi_control_rx [qca_ol]) from [<a8858204>] (wmi_control_rx+0x4e8/0x658 [qca_ol])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.981577] [<a8858204>] (wmi_control_rx [qca_ol]) from [<a885f93c>] (htc_rx_completion_handler+0x3b4/0x68c [qca_ol])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.981847] [<a885f93c>] (htc_rx_completion_handler [qca_ol]) from [<a8864ab4>] (hif_pci_ce_recv_data+0x9c/0x1cc [qca_ol])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.982110] [<a8864ab4>] (hif_pci_ce_recv_data [qca_ol]) from [<a8862200>] (ce_engine_service_reg+0xe0/0x51c [qca_ol])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.982369] [<a8862200>] (ce_engine_service_reg [qca_ol]) from [<a88613a0>] (ce_per_engine_service+0x8c/0x24c [qca_ol])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.982607] [<a88613a0>] (ce_per_engine_service [qca_ol]) from [<a8866520>] (ce_tasklet+0x68/0x5a0 [qca_ol])
Apr 17 20:11:41 10.100.10.5 kernel: [82409.982739] [<a8866520>] (ce_tasklet [qca_ol]) from [<801245d0>] (tasklet_action+0x8c/0xe8)
Apr 17 20:11:41 10.100.10.5 kernel: [82409.982757] [<801245d0>] (tasklet_action) from [<80123c80>] (__do_softirq+0xe4/0x228)
Apr 17 20:11:41 10.100.10.5 kernel: [82409.982772] [<80123c80>] (__do_softirq) from [<80124060>] (irq_exit+0x98/0x108)
Apr 17 20:11:41 10.100.10.5 kernel: [82409.982787] [<80124060>] (irq_exit) from [<8015b9d4>] (__handle_domain_irq+0x90/0xb4)
Apr 17 20:11:41 10.100.10.5 kernel: [82409.982802] [<8015b9d4>] (__handle_domain_irq) from [<8010137c>] (gic_handle_irq+0x50/0x94)
Apr 17 20:11:41 10.100.10.5 kernel: [82409.982815] [<8010137c>] (gic_handle_irq) from [<801024c0>] (__irq_svc+0x40/0x74)
Apr 17 20:11:41 10.100.10.5 kernel: [82409.982822] Exception stack(0x80b01f60 to 0x80b01fa8)
Apr 17 20:11:41 10.100.10.5 kernel: [82409.982834] 1f60: 00000001 00000000 00000000 80103360 00000000 80b00000 00000000 80a3530c
Apr 17 20:11:41 10.100.10.5 kernel: [82409.982846] 1f80: 80b01fb8 80a2ba30 80b61010 00000000 00000000 80b01fb0 80110d5c 80110d60
Apr 17 20:11:41 10.100.10.5 kernel: [82409.982853] 1fa0: 60000013 ffffffff
Apr 17 20:11:41 10.100.10.5 kernel: [82409.982865] [<801024c0>] (__irq_svc) from [<80110d60>] (arch_cpu_idle+0x2c/0x50)
Apr 17 20:11:41 10.100.10.5 kernel: [82409.982879] [<80110d60>] (arch_cpu_idle) from [<80153528>] (cpu_startup_entry+0x134/0x214)
Apr 17 20:11:41 10.100.10.5 kernel: [82409.982896] [<80153528>] (cpu_startup_entry) from [<80a00c90>] (start_kernel+0x3b0/0x438)
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983081] Mem-Info:
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983102] active_anon:3418 inactive_anon:1451 isolated_anon:0
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983102] active_file:2563 inactive_file:699 isolated_file:0
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983102] unevictable:0 dirty:0 writeback:0 unstable:0
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983102] slab_reclaimable:980 slab_unreclaimable:38105
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983102] mapped:3377 shmem:1533 pagetables:167 bounce:0
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983102] free:1848 free_pcp:575 free_cma:0
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983141] Normal free:7392kB min:2488kB low:3108kB high:3732kB active_anon:13672kB inactive_anon:5804kB active_file:10252kB inactive_file:2796kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:402432kB managed:387956kB mlocked:0kB dirty:0kB writeback:0kB mapped:13508kB shmem:6132kB slab_reclaimable:3920kB slab_unreclaimable:152420kB kernel_stack:1480kB pagetables:668kB unstable:0kB
bounce:0kB free_pcp:2300kB local_pcp:624kB free_cma:0kB writeback_tmp:0kB pages_scanned:6452 all_unreclaimable? no
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983148] lowmem_reserve[]: 0 0 0
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983159] Normal: 172*4kB (UME) 298*8kB (UME) 95*16kB (UME) 54*32kB (ME) 16*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 7344kB
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983198] 4789 total pagecache pages
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983203] 100608 pages RAM
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983208] 0 pages HighMem/MovableOnly
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983213] 3619 pages reserved
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983218] 0 pages cma reserved
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983249] wlan: [0:E:ANY] ol_ath_node_alloc: 764: Unable to attach txrx peer for mac b8:f0:09:95:ad:6c, vdev_id 2
Apr 17 20:11:41 10.100.10.5 kernel: [82409.983249]