I'm trying to troubleshoot intermittent WAN outages that happen once or twice a day. Yesterday I saw a BUG message in the router logfile right before the last outage occurred:
Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] [<ffffffff8114aa74>] local_bh_enable+0xc4/0x118 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] [<ffffffff811207a0>] dump_stack+0x8/0x48 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] Call Trace: Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] BUG: at kernel/softirq.c:138 local_bh_enable() Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] Adding packet sink. Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] Adding IPCOMP6 protocol processing.
Has anyone else seen this or have any advice regarding this message? A larger chunk of the log is attached below.
Thanks,
Wed Jan 11 13:30:12 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_get(tp a80000000400b980,handle 0x00000000) Wed Jan 11 13:30:12 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_init(tp a80000000400b980) Wed Jan 11 13:30:12 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_delete(tp a80000000400b980,arg 0xa80000000400b688),p a80000000343a780,f 0000000000000000 Wed Jan 11 13:30:12 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_walk(tp a80000000400b980,walker a8000000030ab900),p a80000000343a780 Wed Jan 11 13:30:12 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_destroy(tp a80000000400b980),p a80000000343a780 Wed Jan 11 13:29:47 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] HTB: quantum of class 11024 is big. Consider r2q change. Wed Jan 11 13:29:47 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] HTB: quantum of class 10001 is big. Consider r2q change. Wed Jan 11 13:29:47 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] p->perfect 0000000000000000 p->h a800000004fac400 Wed Jan 11 13:29:47 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_dump(tp a80000000343a500,fh 0x0,skb a800000006d9e080,t a8000000034d6010),p a80000000343a480,r 0000000000000000,b a8000000034d6038 Wed Jan 11 13:29:47 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_change(tp a80000000343a500,handle 0x00000000,tca a800000007c05a80,arg a800000004f8b950),opt a800000004fac030,p a80000000343a480,r 0000000000000000,*arg 0x0 Wed Jan 11 13:29:47 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_get(tp a80000000343a500,handle 0x00000000) Wed Jan 11 13:29:47 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_init(tp a80000000343a500) Wed Jan 11 13:29:47 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_delete(tp a80000000343a500,arg 0xa80000000343a408),p a80000000343a480,f 0000000000000000 Wed Jan 11 13:29:47 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_walk(tp a80000000343a500,walker a800000004b1f900),p a80000000343a480 Wed Jan 11 13:29:47 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_destroy(tp a80000000343a500),p a80000000343a480 Wed Jan 11 13:28:28 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] HTB: quantum of class 11024 is big. Consider r2q change. Wed Jan 11 13:28:28 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] HTB: quantum of class 10001 is big. Consider r2q change. Wed Jan 11 13:28:28 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] p->perfect 0000000000000000 p->h a800000004face00 Wed Jan 11 13:28:28 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_dump(tp a80000000343a500,fh 0x0,skb a80000000540cb00,t a800000003be3010),p a80000000343a480,r 0000000000000000,b a800000003be3038 Wed Jan 11 13:28:28 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_change(tp a80000000343a500,handle 0x00000000,tca a800000007c05a80,arg a800000004397950),opt a800000004d6c630,p a80000000343a480,r 0000000000000000,*arg 0x0 Wed Jan 11 13:28:28 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_get(tp a80000000343a500,handle 0x00000000) Wed Jan 11 13:28:28 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_init(tp a80000000343a500) Wed Jan 11 13:28:28 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_delete(tp a80000000343a500,arg 0xa80000000343a408),p a80000000343a480,f 0000000000000000 Wed Jan 11 13:28:28 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_walk(tp a80000000343a500,walker a800000004f8b900),p a80000000343a480 Wed Jan 11 13:28:28 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_destroy(tp a80000000343a500),p a80000000343a480 Wed Jan 11 13:28:23 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] HTB: quantum of class 11024 is big. Consider r2q change. Wed Jan 11 13:28:23 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] HTB: quantum of class 10001 is big. Consider r2q change. Wed Jan 11 13:28:23 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] p->perfect 0000000000000000 p->h a800000004d6c800 Wed Jan 11 13:28:23 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_dump(tp a80000000400b980,fh 0x0,skb a800000006d93080,t a800000006d5b010),p a80000000343a780,r 0000000000000000,b a800000006d5b038 Wed Jan 11 13:28:23 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_change(tp a80000000400b980,handle 0x00000000,tca a800000007c05a80,arg a8000000030ab950),opt a800000004d6c630,p a80000000343a780,r 0000000000000000,*arg 0x0 Wed Jan 11 13:28:23 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_get(tp a80000000400b980,handle 0x00000000) Wed Jan 11 13:28:23 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_init(tp a80000000400b980) Wed Jan 11 13:28:23 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_delete(tp a80000000400b980,arg 0xa80000000400b688),p a80000000343a780,f 0000000000000000 Wed Jan 11 13:28:23 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_walk(tp a80000000400b980,walker a800000004393900),p a80000000343a780 Wed Jan 11 13:28:23 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_destroy(tp a80000000400b980),p a80000000343a780 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] HTB: quantum of class 11024 is big. Consider r2q change. Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] HTB: quantum of class 10001 is big. Consider r2q change. Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] p->perfect 0000000000000000 p->h a800000004fac600 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_dump(tp a80000000343a500,fh 0x0,skb a800000006d71500,t a8000000037a6010),p a80000000343a480,r 0000000000000000,b a8000000037a6038 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_change(tp a80000000343a500,handle 0x00000000,tca a800000007c05a80,arg a800000004397950),opt a8000000071eee30,p a80000000343a480,r 0000000000000000,*arg 0x0 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_get(tp a80000000343a500,handle 0x00000000) Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_init(tp a80000000343a500) Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] HTB: quantum of class 11024 is big. Consider r2q change. Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] HTB: quantum of class 10001 is big. Consider r2q change. Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] p->perfect 0000000000000000 p->h a800000004d6c600 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_dump(tp a80000000400b980,fh 0x0,skb a800000006d87200,t a800000006d99010),p a80000000343a780,r 0000000000000000,b a800000006d99038 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_change(tp a80000000400b980,handle 0x00000000,tca a800000007c05a80,arg a80000000486b950),opt a800000004fac630,p a80000000343a780,r 0000000000000000,*arg 0x0 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_get(tp a80000000400b980,handle 0x00000000) Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] tcindex_init(tp a80000000400b980) Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth1: no IPv6 routers present Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth0.4093: no IPv6 routers present Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] IPsec: device down: eth1 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth0.4093: add 33:33:ff:0d:04:62 mcast address to master interface Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth0.4093: add 33:33:00:00:00:01 mcast address to master interface Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth0.4093: add 01:00:5e:00:00:01 mcast address to master interface Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth0.4093: Setting MAC address to 10 da 43 0d 04 62. Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] IPsec: device down: eth0.4093 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth0.4093: del 33:33:00:00:00:01 mcast address from master interface Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth0.4093: del 33:33:00:00:00:01 mcast address from vlan interface Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth0.4093: del 01:00:5e:00:00:01 mcast address from master interface Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth0.4093: del 01:00:5e:00:00:01 mcast address from vlan interface Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth0.4093: del 33:33:ff:10:75:00 mcast address from master interface Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth0.4093: del 33:33:ff:10:75:00 mcast address from vlan interface Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth0.1: no IPv6 routers present Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth0.1: add 33:33:ff:0d:04:53 mcast address to master interface Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth0.1: add 01:00:5e:00:00:01 mcast address to master interface Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth0.1: add 33:33:00:00:00:01 mcast address to master interface Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] VLAN (eth0.1): Setting underlying device (eth0) to promiscious mode. Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] device eth0 entered promiscuous mode Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] eth0.1: Setting MAC address to 10 da 43 0d 04 53. Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] Loading ifDev module Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] using bcm switch bcm53115 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] Loading /root/Arun/renego/fvs336gv3/fvs336gv3/fvs336gv3_common/comps/switchdriver/src/switchdriver/linux/kernel/mii/octeon/mii.c Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] Now do mknod /dev/led_dev c 242 0 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] registering of the char device is sucess Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] xlr8Nat module loaded successfully Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] initializing xlr8Nat proc entries Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] initializing xlr8Nat engine queue Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] initializing flow cache database Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] registering backend engine with xlr8Nat framework Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] N_HDLC line discipline registered. Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] HDLC line discipline: version $Revision: 4.8 $, maxframe=4096 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] [<ffffffff8112a734>] handle_sysn32+0x54/0x98 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] [<ffffffff813042d8>] sys_sendto+0xd8/0x130 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] [<ffffffff81303e98>] sock_sendmsg+0xb0/0x100 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] [<c0000000002fbf24>] key_output+0x36c/0x4d8 [vipsec] Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] [<c0000000002b60a4>] key_parse+0xc4c/0xf08 [vipsec] Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] [<c0000000002fb4c0>] key_sendup0+0x1d0/0x338 [vipsec] Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] [<c0000000002f4e54>] copy_mbuf_to_skbuff+0xf4/0x2b0 [vipsec] Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] [<ffffffff81308de4>] sock_queue_rcv_skb+0x9c/0x248 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] [<ffffffff8114aa74>] local_bh_enable+0xc4/0x118 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] [<ffffffff811207a0>] dump_stack+0x8/0x48 Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] Call Trace: Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] BUG: at kernel/softirq.c:138 local_bh_enable() Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] Adding packet sink. Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] Adding IPCOMP6 protocol processing. Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] Adding ESP6 protocol processing. Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] Adding AH6 protocol processing. Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] Adding IPCOMP4 protocol processing. Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] Adding ESP4 protocol processing. Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] Adding AH4 protocol processing. Wed Jan 11 13:27:18 2017((GMT-0600)) [FVS336GV3][Kernel][KERNEL] Software Backend Registered with XLR8 IPSec Framework
Model: FVS336Gv3|ProSafe dual WAN gigabit firewall with SSL and IPSec VPN
I have inquired these messages with our engineering team and it is best to open a support ticket for them to have it interpreted. We have some additional questions that may help in isolating your issue. Please see below:
1. When the WAN drops out, do both WAN ports drop together?
2. Does customer use single WAN or Load Balancing across both WAN ports?
3. Does the modem connecting into the FVS336Gv3 also lose the internet connection?
4. Does customer take action to bring the WAN online again, or it resolves itself?
5. Can customer change modem in front of the WAN port seeing the issue, to see if it resolve the issue?
6. If customer uses only single WAN port, can they use the second WAN port to see if issue happens there also?
Thanks for your reply. Per your suggestion I have now opened a case on this issue. I'll post more here when we resolve the issue. For now here are the answers to the questions you posted:
1. The light on only one port goes out but users report the internet down even though the light is still on. 2. We are configured for load balancing. 3. We don't have a modem. we are connected to two ports provided to our research lab by the University. 4. The issue usually resolves itself if you wait long enough. If you can't wait then disconnecting and reconnecting the port that has the light out will reestablish the connection. 5. We do not have access to anything upstream of the router. 6. Both ports are used in load balancing mode.
Regards,
Model: FVS336Gv3|ProSafe dual WAN gigabit firewall with SSL and IPSec VPN
