This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
M5300 unaccessible after reload and/or uplink reboot
Hi,
Starting with the beta firmware 11.0.0.30 there is a strange behaviour to be seen on a M5300 based stack. Here are the facts.
The stack:
1 M5300-28G3
1 M5300-28GF3
Role: OSPF Area Boarder Router
VLAN:
20 - OSPF Backbone + Switch Management VLAN
48 - Clients Network
XYZ - Multiple others
Uplink:
The stack is connected to the backbone via one 10Gb line. I know, should be more but by now it is just like this.
The Problem:
I see the stack not beeing accessible by any of its IP addresses anymore from VLAN 48 after:
1. The uplink switch in VLAN 20 (backbone) gets rebooted
2. The stack itself is being rebooted (reloaded)
All the other network traffic is still running fine over it but its own IP adresses do not even respond to pings anymore.
Connecting to the stack via console and reloading it from there does not solve the problem. I have to disconnect the whole stack from the power and then reconnect it. That's the only thing I can do in order to get the stack back into a status where it is accessible through its IP addresses.
-------------
So, this is strange and as I mentioned I started seeing this after I installed the firmware version 11.0.0.30.
Could someone from Netgear take a look to this, please?
Unfortunately I have to inform you that I'm about to give up. I'm not able to reproduce the issue. Once I did a config change I thought might be the problem I was not able to get it to run into the problem again. Here's a short description.
The switch in question was originally connected to the backbone switch with one 10GB line. That line was configured to carry a couple of VLANs where VLAN 48 was one of them. This was - let's say - a leftover from when the network topology was changed/a new area introduced.
Then the setup was changed so the uplink was not just one line but a LAG with two 10GB links. This LAG was configured to also carry VLAN 48 beside a couple of others. I did miss to take away the VLAN tagging/membership settings from the first single line when the LAG was fully set up. Once the new area was introduced, I forgott to remove VLAN 48 from the LAG, as well.
So, I did remove any other VLAN setting from both, the original single line as well as the LAG thus only VLAN20 (the backbone VLAN) was active on the LAG connecting to the backbone switch. After I did that change, the problem seemed to be resolved and I was not able to force it again.
I even applied the "old" startup-config from which I know that it was running when the problem occured. No chance, I do not get the switch to run into the issue again.
So, therefore I have to give up now which - on the one hand - is something realy good... 😉 On the other hand I feel sorry not being able to show you how to force this issue.
Re: M5300 unaccessible after reload and/or uplink reboot
Hi Carl,
Thank you very much for your answer. I will create the files needed and send them according to your instructions. If you don't mind I would prefere to send the information you asked for with the email I'm about to send to the community moderators.
Re: M5300 unaccessible after reload and/or uplink reboot
Hi Carl,
Thanks for your answer. I was unable to come back to this one any earlier, sorry for that.
If this happens, working on a PC in VLAN 48, I cannot access any of this IPs on the switch. Neither VLAN20 nor VLAN48. Super interessting: I do still have access to all the other networks - including internet from VLAN48. This network traffic is being routed through the switch in question here.So it looks like network traffic is flowing through the switch without issues. But accessing - Mgmt UI, telnet, SSH - the switch directly is not possible. Even pings do timeout sent to one of the switch's IPs.
And yes, the routing settings on the workstation is set correctly.
Maybe the following information helps finding out the root cause. If you configure an VLAN specific IP on the switch and while the switch is running, there is no interface active on the switch participating with that VLAN, the switch considers that VLAN specific IP do be down/disabled. Example: The backbone uplink runs on VLAN 20. On the switch there is only one interface being configured to participate in that VLAN. If this interface is down, e.g. because the other side switch is being rebooted or down or unplugged, the switch reports the whole VLAN IP to be down/disabled and it is impossible to even ping the IP assigned to this VLAN on the switch from any other source through the network. This, by its own is somehow strange, as of my point of view and should/could be changed. For this case here anyways this might give an idea where the problem comes from? I don't know but I have a feeling... 😉
Re: M5300 unaccessible after reload and/or uplink reboot
Hi Carl,
Any news on this one? Have you been able to find anything?
I just upgraded all the M5300 switches to the latest public firmware -> 11.0.0.31
The problem is still the same. As soon as I restart the backbone uplink switch, this stack (ABR) connected to it is not accessible through its management IP anymore.
What firmware version are you using before upgrading to v11.0.0.30? Kindly try to revert back the firmware (to when it was working fine before) prior to using firmware version 11.0.0.30 and confirm if you will not experience the problem.
Thanks for your answer. Right now I'm running firmware version 11.0.0.31 on all the switches which is the latest, official one. Do you want me to go through this procedure anyways?
Yes, kindly proceed with it for as long as the switches are not being used for production at the moment so that we could isolate the problem and verify if the problem is only present when using firmwares v11.0.0.30 & v11.0.0.31. Be reminded to perform a factory reset on the switches after reverting back to the previous firmware (to when it was working fine before) then reconfigure it from scratch in order to start clean using the firmware uploaded.
OK. Would it be sufficient to load a configuration backup once the switch was factory reset? It's so much to configure and error-prone doing it all manually. The switch is productive and I'm facing the risk to bring the whole area down if this does not work.
This is to let you know the latest results. For a test: I was NOT changing the firmware but did reset the switch and loaded the configuration which I saved before back into the switch. I then did the test of restarting the uplink backbone switch and the result was, that the switch was becoming accesible again after the uplink switch was up again. So this looked like the problem was solved by resetting and reloading the switch's configuration.
Unfortunatelly this is not true. This morning - after the switch was running for about two days - I performed the same test again and now the problem reappeared. By this I can confirm that resetting the switch does not solve the problem.
I will now change the the firmware back to the former one -> 11.0.0.28. Once that was done I will wait two days and test it again. After that I will let you know the results.
I think I can confirm that this is an issue which was not introduced with the latest firmware thus must have exists longer in the firmware. I just went through the following procedure:
1. Uploaded the former firmware (11.0.0.28) to the stack
2. Set this older firmware the active one and restarted the stack
3. Rest the stack to factory default
4. After a restart I loaded back the configuration saved before and restarted the stack
5. Restarted the uplink backbone switch
Result: Again the management IP address/the stack is not accassible anymore. And again network traffic is flowing fine through the stack, e.g. routing over the stack and through the backbone area (to which the stack mgmt IP belongs) works, but the mgmt IP itself is not accassible.
It becomes accasible again by either disrubting the power supply or reloading/restarting the stack through a USB console session.
I guess it's your turn now. Let me know how I can help you but please check this out. Right now it is not working reliably.
I inquired your concern to the higher tier of NETGEAR Support and they recommend that you open an online case with NETGEAR Support in order for your concern to be forwarded to the engineering team for further investigation. Be reminded to upload the configuration file of the switches to the online case.
Thank you very much for taking care about this one!
Unfortunatelly I'm not entitled to open a support case there. As a none-original purchaser of these switches they will not accept any request from my side. Any chance you or they can open a ticket. I will support them as much as I can but someone else has to open a ticket, sorry... ;(
As far as I know, opening a chat / online case should be done by the original owner of the NETGEAR device. I'm wondering what you meant by how the last time was possible.
Someone over there at Netgear was able to open a ticket regarding that case.
Beside that I'm wondering because isn't it true that Netgear claims the community portal to also be the place where people get help which are not the original owner of Netgear devices? Furthermore is Netgear not interessted in obvious bugs in the firmware?
I ask because I invest time to go through procedures in order to give you the answers you asked for and now we are having a discussion about who should open a ticket with Netgear support or even who is able to do so.
Common guys, I showed you a bug in the firmware and I'm willing to support you guys in figuring out the cause for it but Netgear doesn't help me. This makes me sad, honestly.
So, a very simple question: Is Netgear interessted in users finding and reporting bugs and help Netgear to find the cause or not?
I found out that you have a Lifetime Chat Support with NETGEAR. Feel free to open a chat case anytime and use this forum thread as reference in order for your concern to be forwarded to the engineering team.
I have revieve and download your configuration file to my M5300. And connection is fine.
But in your TechSupport file , there have many logs:
“<12> May 26 22:05:14 ABR-0003-1 ARP[ipMapForwarding]: ipmap_arp_api.c(1149) 3026 %% Received ARP Request on interface vlan 48 with bad target IP address 255.255.255.255. Sender IP is 0.0.0.0, sender MAC is d0:66:7b:e1:7c:1c.”
And cpu usage is 58.02%.
So the ping package may be affect by this.
In order to prove this , I simulation the illegal ARP package as in your technical support file. And send the the illegal arp package to M5300 at 100Mbps.
After reset the uplink device my pc also ping fail to M5300. But ping success once stop ARP packets.
So suggest to check in you network ,why the device(it mac is d0:66:7b:e1:7c:1c) always send illegal arp package to your M5300.
Stop the illegal arp ,then try to access to M5300.
Thank you so much for jumping into this one, highly appreciated!
I did an inspection regarding the information you've provided and found out the following: The source of those packages is a Samsung TV. That TV is creating two packages like the one you mentioned during its startup. I just tested it by doing "normal" TV usage... 😉
It looks like the scenario as you tested it - creating this kind of packages with 100Mbps - is not what happens in real live. So far, after doing a couple of restarts and therefore having a couple of these packages sent through the network, the stack IP is still accassible through VLAN 48 once the uplink switch has been restarted. It looks like there is something else causing the inaccessibility after a while of continued operation of the stack.
I will further test it by:
1. restart the uplink switch after 1 hour of stack operation
2. restart the uplink switch tomorrow
3. restart the uplink switch after 24 hours of stack operation
3. continue to do so
Once the inaccessability reoccurs I'll create a TechSupport file immediately and provide the information to you.
Hth!
Greez,
dialsc
PS: For what it's worth I've just seen the following behaviour. I was - due to tests asked by your colleague - still running firmware 11.0.0.28. I change the config to make the version 11.0.0.31 the active one. Then I triggered a restart through the web UI. After the stack has rebooted, the same problem occurs. The mgmt IP is not accessible from my workstation -> VLAN48. After triggering a second reload through the command interface the mgmt IP has become available.
I will report the result once done.
