Reply
Topic Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
Re: Firmware 7.3.1.7
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-03-25
08:18 PM
2009-03-25
08:18 PM
Firmware 7.3.1.7
Has anyone running a GSM or FSM series L3 managed switch tried firmware 7.3.1.7?
I'm really curious about stability of this new release. 7.1.1.7 and 7.2.1.6 were not very stable (IP routing, VLAN, spanning tree). We had first hand experience with 7.1.1.7 and others reported bad experiences with 7.2.1.6.
We're currently running 6.2.0.14 for stability reasons, but I wish we could make use of some of the new features found in the 7.x.x.x releases such as MAC based VLANs and LAG hash algorithm selection.
None of the release notes mention fixes related to IP routing, VLAN, or spanning tree stability problems, though, so I'm hesitant to actually use that version in production.
If I absolutely have to, I can break our stack apart to get a switch for testing because one of our GSM7352S doesn't have many ports in use and they can be moved to a different switch temporarily. It's a bit of a hassle after that, though, because you then have a switch that's not the same as the others and various split stack configuration issues.
I'm really curious about stability of this new release. 7.1.1.7 and 7.2.1.6 were not very stable (IP routing, VLAN, spanning tree). We had first hand experience with 7.1.1.7 and others reported bad experiences with 7.2.1.6.
We're currently running 6.2.0.14 for stability reasons, but I wish we could make use of some of the new features found in the 7.x.x.x releases such as MAC based VLANs and LAG hash algorithm selection.
None of the release notes mention fixes related to IP routing, VLAN, or spanning tree stability problems, though, so I'm hesitant to actually use that version in production.
If I absolutely have to, I can break our stack apart to get a switch for testing because one of our GSM7352S doesn't have many ports in use and they can be moved to a different switch temporarily. It's a bit of a hassle after that, though, because you then have a switch that's not the same as the others and various split stack configuration issues.
Message 1 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-03-27
08:14 PM
2009-03-27
08:14 PM
Re: Firmware 7.3.1.7
We did go ahead and split our stack and did a test install of 7.3.1.7. All seems ok at first glance except for one thing.
The config directive 'ip http secure-server' seems to be no longer implemented. All config directives beginning with 'ip http' seem to be missing. The manual for this version of firmware indicates I should be able to use that config directive.
That section of the manual states that the web server is on by default. Well, the insecure port 80 server is indeed running and works, but the port 443 (SSL) server is disabled upon upgrade.
When I find a way to re-enable the SSL web GUI, I'll report back.
The config directive 'ip http secure-server' seems to be no longer implemented. All config directives beginning with 'ip http' seem to be missing. The manual for this version of firmware indicates I should be able to use that config directive.
That section of the manual states that the web server is on by default. Well, the insecure port 80 server is indeed running and works, but the port 443 (SSL) server is disabled upon upgrade.
When I find a way to re-enable the SSL web GUI, I'll report back.
Message 2 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-03-27
10:55 PM
2009-03-27
10:55 PM
Re: Firmware 7.3.1.7
Well, I don't know why SSL was disabled on upgrade, but I did figure out how to get it enabled again.
Apparently 'ip http secure-server' is a global mode command that isn't entered in config mode. In other words, you don't have to type 'conf t' before entering it. Go figure.
I'm hoping to get some time Sunday to put this through its paces.
Apparently 'ip http secure-server' is a global mode command that isn't entered in config mode. In other words, you don't have to type 'conf t' before entering it. Go figure.
I'm hoping to get some time Sunday to put this through its paces.
Message 3 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-04-18
03:22 AM
2009-04-18
03:22 AM
Re: Firmware 7.3.1.7
Well, I didn't have a chance to put it through its paces yet, but I thought I'd report my experiences so far.
The switch is linked back to the rest of our network with a single 1Gbps link. It is otherwise disconnected from the stack. It has no hosts or other ports in use for anything. It has its own IP address, different than the stack it was separated from. All IP addresses on the switch were removed or changed after disconnecting from the stack and before reconnecting it to the rest of the network.
It is running MSTP, a routed VLAN, and several unused layer 2 VLANs.
It reloaded yesterday without any apparent reason. Nothing was logged. Not a good sign. That's one of the problems we'd had with previous 7.x.x.x firmwares.
I'm going to try increasing the logging level to see if I can catch why it reloads if it reloads again.
The switch is linked back to the rest of our network with a single 1Gbps link. It is otherwise disconnected from the stack. It has no hosts or other ports in use for anything. It has its own IP address, different than the stack it was separated from. All IP addresses on the switch were removed or changed after disconnecting from the stack and before reconnecting it to the rest of the network.
It is running MSTP, a routed VLAN, and several unused layer 2 VLANs.
It reloaded yesterday without any apparent reason. Nothing was logged. Not a good sign. That's one of the problems we'd had with previous 7.x.x.x firmwares.
I'm going to try increasing the logging level to see if I can catch why it reloads if it reloads again.
Message 4 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-04-28
01:53 AM
2009-04-28
01:53 AM
Re: Firmware 7.3.1.7
It reloaded again a couple of days ago.
I suspect that someone is trying various ssh vulnerability attacks against the switch and one of those causes a reload.
I'm going to investigate the possibility of using an ACL to block all access to the switch from all locations accept our management workstations. Of course, I'll be watching for the absence of future reloads. It's hard to definitively determine the absence of something.
With prior 7.x.x.x versions, I couldn't get ACLs to do that, but maybe this version will or I'll discover what I missed before 😉 .
I suspect that someone is trying various ssh vulnerability attacks against the switch and one of those causes a reload.
I'm going to investigate the possibility of using an ACL to block all access to the switch from all locations accept our management workstations. Of course, I'll be watching for the absence of future reloads. It's hard to definitively determine the absence of something.
With prior 7.x.x.x versions, I couldn't get ACLs to do that, but maybe this version will or I'll discover what I missed before 😉 .
Message 5 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-04-28
02:53 AM
2009-04-28
02:53 AM
Re: Firmware 7.3.1.7
I got ACLs working to restrict access to the switch IP. I applied:
We'll see if the reloads go away. If so, something at the IP layer on the network was tickling the switch into reloading. If not, then it might be something at layer 2 or a cause internal to the switch.
conf t
access-list 110 permit ip0.0.0.0 0.0.0.0
access-list 110 permit ip0.0.0.0 0.0.0.0
access-list 110 deny ip any0.0.0.0
access-list 110 permit every
access-list 110 deny every
ip access-group 110 in
exit
save
We'll see if the reloads go away. If so, something at the IP layer on the network was tickling the switch into reloading. If not, then it might be something at layer 2 or a cause internal to the switch.
Message 6 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-05-06
12:49 PM
2009-05-06
12:49 PM
Re: Firmware 7.3.1.7
So far, 12 days without a random reload by the switch after implementing the ACL for the management IP. I did have to modify the ACL to allow the switch to communicate with the NTP server, but, other than that, the ACL I posted above seems to have done the trick so far. Without NTP, the switch was losing minutes (system clock running slower than real time) at the rate of 1 or 2 minutes per day.
I'm going to let it sit this way for another couple weeks before adding some real traffic to the switch. If it can handle some real traffic for a month without trouble, I'll add in layer 3 VLANs and dynamic routing. If it can handle that for a month without trouble, it gets my stamp of approval and I'll roll it out to our other switches.
I'm going to let it sit this way for another couple weeks before adding some real traffic to the switch. If it can handle some real traffic for a month without trouble, I'll add in layer 3 VLANs and dynamic routing. If it can handle that for a month without trouble, it gets my stamp of approval and I'll roll it out to our other switches.
Message 7 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-05-25
12:42 PM
2009-05-25
12:42 PM
Re: Firmware 7.3.1.7
It has now been a full month since I implemented the ACLs for the switch management and there still are no reloads.
That would point to one of the following as the source of the reloads:
That would point to one of the following as the source of the reloads:
I can test item #1 above easily enough. That's the one I'm concerned about because the logs will definitely fill up over time. If the switch reloads when the logs get full, then the reloads will be unavoidable.
All the others would be successfully mitigated permanently by the ACLs.
It would be nice if I could nail down a reproducible problem so I can file a bug report and get a fixed firmware version.
Message 8 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-06-19
01:35 AM
2009-06-19
01:35 AM
Re: Firmware 7.3.1.7
I've been trying to get Netgear support to forward my firmware fix request to the firmware guys at Netgear, but their response so far has been "reflash, reset, check your config." Useless. I've already reflashed. The switch reloads itself, so how is it going to help if it tell it to do so again? I've checked my config a bunch of times in isolating the issue to the point that I reported it and pinpointed for them the source of the problem.
I'm pressing them on this and won't back down until they fix it. I've even offered them direct access to my test switch so their firmware engineers can debug the problem directly in the environment in which it happens.
That said, if any of you are having problems with your GSM73xx series switches randomly reloading when your VLAN IP addresses on the switch are exposed to the Internet, please open a trouble ticket with them on the issue.
It's a really backwards thing, but because they have fewer buyers on their most expensive switches, they receive the least QA and slowest fixes to firmware problems. Grrr.
I'm hoping that if we can get enough people to gang up on them that they'll finally give this issue some attention and get it fixed.
I'm pressing them on this and won't back down until they fix it. I've even offered them direct access to my test switch so their firmware engineers can debug the problem directly in the environment in which it happens.
That said, if any of you are having problems with your GSM73xx series switches randomly reloading when your VLAN IP addresses on the switch are exposed to the Internet, please open a trouble ticket with them on the issue.
It's a really backwards thing, but because they have fewer buyers on their most expensive switches, they receive the least QA and slowest fixes to firmware problems. Grrr.
I'm hoping that if we can get enough people to gang up on them that they'll finally give this issue some attention and get it fixed.
Message 9 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-08-05
12:57 AM
2009-08-05
12:57 AM
Re: Firmware 7.3.1.7
[QUOTE=advantagecom;187408]It has now been a full month since I implemented the ACLs for the switch management and there still are no reloads.
That would point to one of the following as the source of the reloads:
That would point to one of the following as the source of the reloads:
It's now more than 3 months without a reload. Just for kicks, I setup the ACL so it exposed *only* SSH and the switch has been running that way for a little over a week. The hope was that I could get a reload with just one port exposed to vastly narrow down the root cause. Of course, the script kiddies haven't obliged and the logs have been quiet the entire week.
I fired up a dictionary attack against SSH on the switch and there are now tens of thousands of log messages with no reload caused, so it definitely isn't the logs filling up that causes the reload.
It also isn't just normal failed logins causing the reload. I've generated around 10,000 failed logins on the switch so far without it causing a reload.
One common vector of attack is a buffer overflow. The next thing to try is pasting in a huge text file for the username and doing the same for the password. Maybe it will finally keel over. 😉
Message 10 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-08-05
01:26 AM
2009-08-05
01:26 AM
Re: Firmware 7.3.1.7
Well, sending a huge username knocked the switch offline, but it didn't reload. The entire tcp/ip stack on the switch died, so now it is completely inaccessible over the network. 😄
The ethernet port active on that switch still negotiates a link, but I don't know if it is passing traffic.
I'll try the serial console when I get to the office tomorrow to see if that access method still works.
The ethernet port active on that switch still negotiates a link, but I don't know if it is passing traffic.
I'll try the serial console when I get to the office tomorrow to see if that access method still works.
Message 11 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-08-05
11:45 AM
2009-08-05
11:45 AM
Re: Firmware 7.3.1.7
The console was completely unresponsive too. :rolleyes:
Man, I really nailed it. Hard.
At any rate, I'm going to try other common script kiddie methods on the switch now that I've power cycled it.
I have console logging turned on and am capturing all of the console output.
I'll catalog each of these easy denial of service methods and package them up for Netgear support. Hopefully they agree with me that an enterprise L3 switch should not be so easily DOS'ed and they get these issues fixed.
Man, I really nailed it. Hard.
At any rate, I'm going to try other common script kiddie methods on the switch now that I've power cycled it.
I have console logging turned on and am capturing all of the console output.
I'll catalog each of these easy denial of service methods and package them up for Netgear support. Hopefully they agree with me that an enterprise L3 switch should not be so easily DOS'ed and they get these issues fixed.
Message 12 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-08-05
12:40 PM
2009-08-05
12:40 PM
Re: Firmware 7.3.1.7
Have you tried emailing level2supportnetgear.com? They might be more willing to take onboard your findings as opposed to the frontline helpdesk guy 🙂
Message 13 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-08-05
01:25 PM
2009-08-05
01:25 PM
Re: Firmware 7.3.1.7
stevenb wrote: Have you tried emailing level2support netgear.com? They might be more willing to take onboard your findings as opposed to the frontline helpdesk guy 🙂
No, I haven't tried that. They won't sacrifice my firstborn for bypassing the gatekeepers? 😉
Seriously, though, I don't want to cause ill will because I'm using a support resource that hasn't been offered to me.
We haven't purchased a support contract (was like $5K annually for our switches), so I just figured I was stuck battling consumer oriented support drones to pound through firmware bug reports. Is that not correct?
Message 14 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-08-05
07:50 PM
2009-08-05
07:50 PM
Re: Firmware 7.3.1.7
Don't know about the support but I am seriously interested in the outcome as I am about to purchase 2 GSM7328s's for my Network Core with about 8 stacks of 5 GS748TS's at the access layer. I may hold off on this purchase for the time being. Do you have any other comments of wisdom on these switch's?♦
Message 15 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-08-05
09:24 PM
2009-08-05
09:24 PM
Re: Firmware 7.3.1.7
Markwirez wrote: Don't know about the support but I am seriously interested in the outcome as I am about to purchase 2 GSM7328s's for my Network Core with about 8 stacks of 5 GS748TS's at the access layer. I may hold off on this purchase for the time being. Do you have any other comments of wisdom on these switch's?♦
Well, we did have an experience where one of our more powerful servers flooded the switch with 400 million PPS of ARPs during an incident involving misconfigured LACP on the server side (~190 million PPS is the maximum that the switch can handle, IIRC) . The switches didn't like that (~90% packet loss, no management access). 😉
As soon as the problem machine was disconnected, though, everything returned to normal. The switches didn't even reload.
The moral of the story is that the switch hardware seems up to the task even if the GUI/CLI seems a bit fragile. If Netgear's firmware guys can harden things a bit, these switches will be everything we wanted.
On a completely different topic, this hard crash is 100% reproducible. It leaves no console log messages even at the debug level. It just instantly locks hard before any messages make it out to the console log.
Message 16 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-11-05
12:08 PM
2009-11-05
12:08 PM
Re: Firmware 7.3.1.7
Seems we may be running into the same issues you have for this particular firmware version. Our logs show a bunch of sshd login attempts before it crashes (no reboot) and one can assume after reading your findings that, that is what is going on here too. Have you had any further results with this? Any further issues? Have you had a chance to take a look at the new firmware and see if netgear actually fixed this problem?
Message 17 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-11-06
03:04 PM
2009-11-06
03:04 PM
Re: Firmware 7.3.1.7
jturner wrote: Seems we may be running into the same issues you have for this particular firmware version. Our logs show a bunch of sshd login attempts before it crashes (no reboot) and one can assume after reading your findings that, that is what is going on here too. Have you had any further results with this? Any further issues? Have you had a chance to take a look at the new firmware and see if netgear actually fixed this problem?
The only thing I can add is that the 6.2.0.14 firmware is also vulnerable to the same problem, but it isn't quite as sensitive. It takes a little more abuse before it takes a nosedive. Our only solution thus far is the block TCP and UDP (allow ICMP for troubleshooting purposes) to every IP address active on the L3 switch. Of course, allow TCP access to your management station(s).
Another approach that might work for some is to turn off SSH access for the switch, but you're hosed again the moment you turn it back on, so it is far from ideal. We had a situation where we'd tried this approach and the switch always crashed again before we had a chance to finish up in SSH and turn it off again. If you only use the GUI, though, this might work well enough.
I have not had time to install and test the new firmware on our non-production switch and there's no way I'm going to put it on our production switches until I've tested it thoroughly. Netgear tech support indicated that the fix likely wouldn't be in that version because it was too new of an issue to get integrated, but sometimes the firmware guys and the tech support guys don't communicate about every little firmware change. Maybe it is fixed in the 8.x.x.x beta, but I didn't see it specifically in the "changelog".
Message 18 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-11-15
07:21 PM
2009-11-15
07:21 PM
Re: Firmware 7.3.1.7
8.0.0.25 beta is barely functional. I tried an upgrade and the switch got stuck in an endless reboot. After clearing the config at the boot menu, it was finally able to boot properly.
The big issue is that there are so many syntax changes in the commands that converting the config is a tedious and time consuming process. It could probably be done if you had the patience, but expect several hours (at least) of pulling your hair out trying to get the same config you had in 7.3.1.7.
The big issue is that there are so many syntax changes in the commands that converting the config is a tedious and time consuming process. It could probably be done if you had the patience, but expect several hours (at least) of pulling your hair out trying to get the same config you had in 7.3.1.7.
Message 19 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2009-11-16
09:01 AM
2009-11-16
09:01 AM
Re: Firmware 7.3.1.7
I had the same issue only on my GSM7328S after which I did a recovery and it said it was successful with the firmware upgrade but unsuccessful when checking the bootcode. So I pulled it from the network let sit over weekend and boot it back up and its running fine now without reboot weird.
Message 20 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2010-03-26
12:50 AM
2010-03-26
12:50 AM
Re: Firmware 7.3.1.7
Firmware v8.0.1.2 is immune to the "long username hang" problem. They seem to have instituted a 99 character limit in some places, a 90 character limit in other places, and a 32 character limit elsewhere. Regardless, it seems to prevent the buffer overflow from taking down the switch.
It will require further testing to see if it can handle being exposed to the Internet without reloading on a regular basis. We have our test switch with v8.0.1.2 exposed to the Internet, but who knows how long it will be before it gets attacked. Even then, how long do you go without a reload before you call it "stable"? :confused:
It will require further testing to see if it can handle being exposed to the Internet without reloading on a regular basis. We have our test switch with v8.0.1.2 exposed to the Internet, but who knows how long it will be before it gets attacked. Even then, how long do you go without a reload before you call it "stable"? :confused:
Message 21 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2010-07-13
07:28 PM
2010-07-13
07:28 PM
Re: Firmware 7.3.1.7
We left our test switch with v8.0.1.2 exposed to the Internet for about 45 days. We never experienced any reloads or hangs. Given that we have about 6000 *new* malicious IP addresses attack or probe our network every day and there are thousands of attacks across our network every hour, I'd say that's a good result.
Our test switch is no longer connected to the Internet because we're using it in some internal server testing, but we're happy enough with the results that we will likely push this firmware out to production in the near future.
Our test switch is no longer connected to the Internet because we're using it in some internal server testing, but we're happy enough with the results that we will likely push this firmware out to production in the near future.
Message 22 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2010-07-14
04:48 AM
2010-07-14
04:48 AM
Re: Firmware 7.3.1.7
advantagecom wrote: We left our test switch with v8.0.1.2 exposed to the Internet for about 45 days. We never experienced any reloads or hangs. Given that we have about 6000 *new* malicious IP addresses attack or probe our network every day and there are thousands of attacks across our network every hour, I'd say that's a good result.
Our test switch is no longer connected to the Internet because we're using it in some internal server testing, but we're happy enough with the results that we will likely push this firmware out to production in the near future.
Hi,
Is it worth upgrading to 8.* version from 7.1.1.7 ? Is it much more stable than 7.1 ?
Thanks,
Message 23 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2010-07-14
11:46 AM
2010-07-14
11:46 AM
Re: Firmware 7.3.1.7
alex74 wrote: Hi,
Is it worth upgrading to 8.* version from 7.1.1.7 ? Is it much more stable than 7.1 ?
Thanks,
In our experience, the entire 7.x.x.x line of firmwares is buggy and unstable. 7.1.1.7 is even worse than 7.3.1.7. We had massive problems with STP and L3 routing with the 7.x.x.x firmwares. Furthermore, the 7.x.x.x firmwares are vulnerable to multiple remote DOS attacks that leave you with a hard-locked switch that requires a power cycle to get it going again.
Yes, it is definitely worth the upgrade. Just make sure you follow the instructions Netgear supplies.
Message 24 of 26
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2010-07-15
04:22 AM
2010-07-15
04:22 AM
Re: Firmware 7.3.1.7
advantagecom wrote: In our experience, the entire 7.x.x.x line of firmwares is buggy and unstable. 7.1.1.7 is even worse than 7.3.1.7. We had massive problems with STP and L3 routing with the 7.x.x.x firmwares. Furthermore, the 7.x.x.x firmwares are vulnerable to multiple remote DOS attacks that leave you with a hard-locked switch that requires a power cycle to get it going again.
Yes, it is definitely worth the upgrade. Just make sure you follow the instructions Netgear supplies.
Upgraded to 8.*, had a big issue with switches as they didn't take older config and decided to go into infinite rebooting loop. I manually removed config - that worked so I just copy-pasted everything back in and that was it... There was no problem upgrading switch from 7.1->7.2->7.3 though.
Message 25 of 26