Firmware 7.3.1.7

The console was completely unresponsive too. :rolleyes:

Man, I really nailed it. Hard.

At any rate, I'm going to try other common script kiddie methods on the switch now that I've power cycled it.

I have console logging turned on and am capturing all of the console output.

I'll catalog each of these easy denial of service methods and package them up for Netgear support. Hopefully they agree with me that an enterprise L3 switch should not be so easily DOS'ed and they get these issues fixed.

Have you tried emailing level2supportnetgear.com? They might be more willing to take onboard your findings as opposed to the frontline helpdesk guy :)

stevenb wrote:
Have you tried emailing level2supportnetgear.com? They might be more willing to take onboard your findings as opposed to the frontline helpdesk guy :)

No, I haven't tried that. They won't sacrifice my firstborn for bypassing the gatekeepers? ;)

Seriously, though, I don't want to cause ill will because I'm using a support resource that hasn't been offered to me.

We haven't purchased a support contract (was like $5K annually for our switches), so I just figured I was stuck battling consumer oriented support drones to pound through firmware bug reports. Is that not correct?

Don't know about the support but I am seriously interested in the outcome as I am about to purchase 2 GSM7328s's for my Network Core with about 8 stacks of 5 GS748TS's at the access layer. I may hold off on this purchase for the time being. Do you have any other comments of wisdom on these switch's?♦

Markwirez wrote:
Don't know about the support but I am seriously interested in the outcome as I am about to purchase 2 GSM7328s's for my Network Core with about 8 stacks of 5 GS748TS's at the access layer. I may hold off on this purchase for the time being. Do you have any other comments of wisdom on these switch's?♦

Well, we did have an experience where one of our more powerful servers flooded the switch with 400 million PPS of ARPs during an incident involving misconfigured LACP on the server side (~190 million PPS is the maximum that the switch can handle, IIRC) . The switches didn't like that (~90% packet loss, no management access). ;)

As soon as the problem machine was disconnected, though, everything returned to normal. The switches didn't even reload.

The moral of the story is that the switch hardware seems up to the task even if the GUI/CLI seems a bit fragile. If Netgear's firmware guys can harden things a bit, these switches will be everything we wanted.


On a completely different topic, this hard crash is 100% reproducible. It leaves no console log messages even at the debug level. It just instantly locks hard before any messages make it out to the console log.

Seems we may be running into the same issues you have for this particular firmware version. Our logs show a bunch of sshd login attempts before it crashes (no reboot) and one can assume after reading your findings that, that is what is going on here too. Have you had any further results with this? Any further issues? Have you had a chance to take a look at the new firmware and see if netgear actually fixed this problem?

jturner wrote:

Seems we may be running into the same issues you have for this particular firmware version. Our logs show a bunch of sshd login attempts before it crashes (no reboot) and one can assume after reading your findings that, that is what is going on here too. Have you had any further results with this? Any further issues? Have you had a chance to take a look at the new firmware and see if netgear actually fixed this problem?

The only thing I can add is that the 6.2.0.14 firmware is also vulnerable to the same problem, but it isn't quite as sensitive. It takes a little more abuse before it takes a nosedive. Our only solution thus far is the block TCP and UDP (allow ICMP for troubleshooting purposes) to every IP address active on the L3 switch. Of course, allow TCP access to your management station(s).

Another approach that might work for some is to turn off SSH access for the switch, but you're hosed again the moment you turn it back on, so it is far from ideal. We had a situation where we'd tried this approach and the switch always crashed again before we had a chance to finish up in SSH and turn it off again. If you only use the GUI, though, this might work well enough.

I have not had time to install and test the new firmware on our non-production switch and there's no way I'm going to put it on our production switches until I've tested it thoroughly. Netgear tech support indicated that the fix likely wouldn't be in that version because it was too new of an issue to get integrated, but sometimes the firmware guys and the tech support guys don't communicate about every little firmware change. Maybe it is fixed in the 8.x.x.x beta, but I didn't see it specifically in the "changelog".

8.0.0.25 beta is barely functional. I tried an upgrade and the switch got stuck in an endless reboot. After clearing the config at the boot menu, it was finally able to boot properly.

The big issue is that there are so many syntax changes in the commands that converting the config is a tedious and time consuming process. It could probably be done if you had the patience, but expect several hours (at least) of pulling your hair out trying to get the same config you had in 7.3.1.7.

I had the same issue only on my GSM7328S after which I did a recovery and it said it was successful with the firmware upgrade but unsuccessful when checking the bootcode. So I pulled it from the network let sit over weekend and boot it back up and its running fine now without reboot weird.

Firmware v8.0.1.2 is immune to the "long username hang" problem. They seem to have instituted a 99 character limit in some places, a 90 character limit in other places, and a 32 character limit elsewhere. Regardless, it seems to prevent the buffer overflow from taking down the switch.

It will require further testing to see if it can handle being exposed to the Internet without reloading on a regular basis. We have our test switch with v8.0.1.2 exposed to the Internet, but who knows how long it will be before it gets attacked. Even then, how long do you go without a reload before you call it "stable"? :confused: