Firmware 7.3.1.7

We did go ahead and split our stack and did a test install of 7.3.1.7. All seems ok at first glance except for one thing.

The config directive 'ip http secure-server' seems to be no longer implemented. All config directives beginning with 'ip http' seem to be missing. The manual for this version of firmware indicates I should be able to use that config directive.

That section of the manual states that the web server is on by default. Well, the insecure port 80 server is indeed running and works, but the port 443 (SSL) server is disabled upon upgrade.

When I find a way to re-enable the SSL web GUI, I'll report back.

Well, I don't know why SSL was disabled on upgrade, but I did figure out how to get it enabled again.

Apparently 'ip http secure-server' is a global mode command that isn't entered in config mode. In other words, you don't have to type 'conf t' before entering it. Go figure.

I'm hoping to get some time Sunday to put this through its paces.

Well, I didn't have a chance to put it through its paces yet, but I thought I'd report my experiences so far.

The switch is linked back to the rest of our network with a single 1Gbps link. It is otherwise disconnected from the stack. It has no hosts or other ports in use for anything. It has its own IP address, different than the stack it was separated from. All IP addresses on the switch were removed or changed after disconnecting from the stack and before reconnecting it to the rest of the network.

It is running MSTP, a routed VLAN, and several unused layer 2 VLANs.

It reloaded yesterday without any apparent reason. Nothing was logged. Not a good sign. That's one of the problems we'd had with previous 7.x.x.x firmwares.

I'm going to try increasing the logging level to see if I can catch why it reloads if it reloads again.

It reloaded again a couple of days ago.

I suspect that someone is trying various ssh vulnerability attacks against the switch and one of those causes a reload.

I'm going to investigate the possibility of using an ACL to block all access to the switch from all locations accept our management workstations. Of course, I'll be watching for the absence of future reloads. It's hard to definitively determine the absence of something.

With prior 7.x.x.x versions, I couldn't get ACLs to do that, but maybe this version will or I'll discover what I missed before ;) .

I got ACLs working to restrict access to the switch IP. I applied:

conf t
access-list 110 permit ip  0.0.0.0  0.0.0.0
access-list 110 permit ip  0.0.0.0  0.0.0.0
access-list 110 deny ip any  0.0.0.0
access-list 110 permit every
access-list 110 deny every
ip access-group 110 in
exit
save

We'll see if the reloads go away. If so, something at the IP layer on the network was tickling the switch into reloading. If not, then it might be something at layer 2 or a cause internal to the switch.

So far, 12 days without a random reload by the switch after implementing the ACL for the management IP. I did have to modify the ACL to allow the switch to communicate with the NTP server, but, other than that, the ACL I posted above seems to have done the trick so far. Without NTP, the switch was losing minutes (system clock running slower than real time) at the rate of 1 or 2 minutes per day.

I'm going to let it sit this way for another couple weeks before adding some real traffic to the switch. If it can handle some real traffic for a month without trouble, I'll add in layer 3 VLANs and dynamic routing. If it can handle that for a month without trouble, it gets my stamp of approval and I'll roll it out to our other switches.

It has now been a full month since I implemented the ACLs for the switch management and there still are no reloads.

That would point to one of the following as the source of the reloads:

I can test item #1 above easily enough. That's the one I'm concerned about because the logs will definitely fill up over time. If the switch reloads when the logs get full, then the reloads will be unavoidable.

All the others would be successfully mitigated permanently by the ACLs.

It would be nice if I could nail down a reproducible problem so I can file a bug report and get a fixed firmware version.

I've been trying to get Netgear support to forward my firmware fix request to the firmware guys at Netgear, but their response so far has been "reflash, reset, check your config." Useless. I've already reflashed. The switch reloads itself, so how is it going to help if it tell it to do so again? I've checked my config a bunch of times in isolating the issue to the point that I reported it and pinpointed for them the source of the problem.

I'm pressing them on this and won't back down until they fix it. I've even offered them direct access to my test switch so their firmware engineers can debug the problem directly in the environment in which it happens.

That said, if any of you are having problems with your GSM73xx series switches randomly reloading when your VLAN IP addresses on the switch are exposed to the Internet, please open a trouble ticket with them on the issue.

It's a really backwards thing, but because they have fewer buyers on their most expensive switches, they receive the least QA and slowest fixes to firmware problems. Grrr.

I'm hoping that if we can get enough people to gang up on them that they'll finally give this issue some attention and get it fixed.

[QUOTE=advantagecom;187408]It has now been a full month since I implemented the ACLs for the switch management and there still are no reloads.

That would point to one of the following as the source of the reloads:

It's now more than 3 months without a reload. Just for kicks, I setup the ACL so it exposed *only* SSH and the switch has been running that way for a little over a week. The hope was that I could get a reload with just one port exposed to vastly narrow down the root cause. Of course, the script kiddies haven't obliged and the logs have been quiet the entire week.

I fired up a dictionary attack against SSH on the switch and there are now tens of thousands of log messages with no reload caused, so it definitely isn't the logs filling up that causes the reload.

It also isn't just normal failed logins causing the reload. I've generated around 10,000 failed logins on the switch so far without it causing a reload.

One common vector of attack is a buffer overflow. The next thing to try is pasting in a huge text file for the username and doing the same for the password. Maybe it will finally keel over. ;)

Well, sending a huge username knocked the switch offline, but it didn't reload. The entire tcp/ip stack on the switch died, so now it is completely inaccessible over the network. :D

The ethernet port active on that switch still negotiates a link, but I don't know if it is passing traffic.

I'll try the serial console when I get to the office tomorrow to see if that access method still works.