FVS336Gv3 Weird Problem - VPN stuck and possibly blocking DMZ + DHCP on Port 4 ?

[Marksmt]

Hi

I wonder if anyone has come across this. Just replaced a UTM-25 with a FVS336G v3 with firmware version: 4.3.4-2 which has been perfectly stable for about a month or so. Came to work this morning and due to a LAN crash staff had power cycled the router and apart from mopping up some support issues everything seemed stable until the Wireless LAN was reported as not connecting to the internet later in the day

We run a separate WLAN on DMZ port 4 with pass-thru DHCP. After elminating our wireless equipment I traced the problem back to the router which seemed very sluggish and was taking about 2 to 3 minutes to issue an IP via DHCP (direct patch to test laptop)

I remoted in to the web console from the internal LAN (port 1) and after poking around and finding not much unusual with the DMZ I noticed a problem with our (one and only) overseas hard VPN link. I couldn't ping it. The web console showed the connection UP but the disconnect button did nothing at all. The VPN logs showed some problems connecting despite the link status showing UP

Tue Feb 28 12:32:02 2017 (GMT +0000): [FVS336GV3] [IKE] ERROR: Peer is requesting for phase-2 establishment, could not start negotiation as phase-1 is not established
Tue Feb 28 12:32:02 2017 (GMT +0000): [FVS336GV3] [IKE] INFO: Sending Informational Exchange: notify payload[INVALID-COOKIE]
(repeated many times)

Also the main log showed a "clock skew detected" error (?) although the date and time appeared to be synced correctly

Also, the VPN connection up time in seconds showed some huge number (maybe around 5000000 or so seconds). I calculated it to be 6000 days or roughly 16 years - which would be problematic as it was rebooted this AM and has only been installed a month or so.

Unable to disconnect/reconnect (or annoy users with a 2nd FWR reboot in one day) I disabled and re-enabled the VPN policy and the VPN suddenly connected OK and I could ping our remote VPN site etc. 

Also, (and weirdly) the router then became responsive and I could obtain an IP via the DMZ WLAN test laptop and browse sites, do DNS lookups etc. All good. No further changes made or needed.

It looks like the VPN component was stuck in some kind of loop perhaps and dragging down the routers CPU (?)

As I don't have the option of a newer firmware has anyone experienced this? Is this likely to be an ongoing issue?

Regards

[Marksmt · Accepted solution]

Hi

Thanks for the suggestion. Unfortunately, I went down this route with the UTM-25 and don't really have the luxury of continued disruption in the office doing the same with this new FWR so I've managed to get an RMA on the unit and have reverted back to the UTM-25 for the time being.

I've seen posts elsewhere on the net suggesting the clock skew issue was related to faulty PSU bricks. I guess only the tech guys who look at the returned unit will know. Hopefully there will be a new firmware release soon. It seems odd that it was perfectly stable for several weeks then seems to have thrown a complete "wobbler" in recent days. It resides on a surge protected UPS and the other equipment in the rack is fine

If the new unit fails the same way I'll have to look at an alternative product

Regards

[Marksmt]

More information whicy may help others who find themselves in the same situation

We had a a serious problem in our domain which I now know is caused by the Netgear FWR deciding to act as an internal DHCP server. This has been a nightmare. Random round the site users losing authentication via a domain controller whilst working and, on investigation suddenly aren't connected to our Active Directory DC DHCP but have been assigned a high range IP of say .251 or .253 by the FWR even though it has not been set up to do so. The ipconfig also showed the network as company.com instead of company.local (the DC)

Rebooting the FVS unit after yesterday's issue with VPN blocking VLAN4 DHCP and then rebooting the workstation resolved the issue. Users can  now connect to the A/D domain and resources if they reboot and log in again. 

I think I will have to raise an RMA on this unit. It has worked fine for a month or two and I thought I had finally resolved the endless crash/reboot cycles with the older UTM-25 unit we had, only to find the FWR goes AWOL after a few weeks of use. Not a happy bunny 😞

[Marksmt]

Ran out of time to edit 2nd post as I was interrupted to do some work ....

*Edit*
A few minutes after the last reboot the VPN link uptime shows 541686032.0 seconds with 3296 packets exchangedand the VPN dead. Again, no control over the VPN connection and I must have got lucky yesterday because disabling and re-enabling the current VPN policy has not revived the VPN link. After grabbing screendumps and because I don't have time to spend all day wiping and reconfiguring I'm going to fail the unit out. Also, DHCP working on DMZ/port4 but no ip based traffic passing through although DNS seems to work (ping 8.8.8.8 fails) from our DMZ/WLAN

[JohnRo]

Hello Marksmt, 

I suggest saving the configuration file of the firewall and performing a factory reset. Once done, reconfigure the VPN connectivity (only the VPN - no other settings). See if it will show the same issue. I have checked our logged cases but I did not find anything that is similar to yours. 

Thanks,

[Marksmt · Accepted solution]

Hi

Thanks for the suggestion. Unfortunately, I went down this route with the UTM-25 and don't really have the luxury of continued disruption in the office doing the same with this new FWR so I've managed to get an RMA on the unit and have reverted back to the UTM-25 for the time being.

I've seen posts elsewhere on the net suggesting the clock skew issue was related to faulty PSU bricks. I guess only the tech guys who look at the returned unit will know. Hopefully there will be a new firmware release soon. It seems odd that it was perfectly stable for several weeks then seems to have thrown a complete "wobbler" in recent days. It resides on a surge protected UPS and the other equipment in the rack is fine

If the new unit fails the same way I'll have to look at an alternative product

Regards

[JohnRo]

Hi Marksmt, 

Hopefully the replacement unit will be stable so that you won't experience any disruption. For now, I suggest marking your post above as a (temporary) solution. You can always open a new post if have more questions. 

Thanks,

[SamirD]

Did you perform any type of firmware update or any other change prior to this behavior?  It's odd for these units to go that far off the rails like that.

[Marksmt]

No, no updates or changes after testing and final commissioning.

if this information helps anyone else...

It was upgraded to latest firmware as a precautionary measure and tested. There was an initial hiccup on first install, whereby it wouldn't resolve DNS and first rollout was abandoned + reverted, (fixed by changing LAN DNS from our ISP's DNS to Google 8.8.8.8). As a DNS issue was acknowledged by our ISP I assumed this was down to them, but I now believe this was coincidental and that for some reason the FVS unit wouldn't resolve our original DNS. I noticed that Wireless LAN going via the DMZ would resolve ok at the time, but workstations on the LAN would not. This was because I had copied over the original config correctly and whoever set the UTM up had put the WLAN on 8.8.8.8 and the LAN used our ISP's DNS). The UTM config was input to the FVS by printing out every config page, taking the unit home and spending a Sunday afternoon laboriously typing in the config and checking for errors.

Once DNS issues were resolved,  VPN was set up, configuration input checked, made live one evening, and was working perfectly for many weeks - in contrast to the UTM-25 which I had to revert to, which becomes unstable and requires a regular reboot every 3 or 4 weeks. I don't have the luxury of being able to tinker around with with a FWR once live as it is in 24/7 international use.

The fault, as highlighted above initially manifested as a failure of the Wireless LAN (DMZ port 4) but investigations showed the fault to be at the router. Checking the VPN link showed that also to be down. Then there were failures due to DHCP becoming enabled on the LAN. This was easily resolved by disabling LAN DHCP again, but this didn't resolve the VPN or WLAN issue satisfactorily. I also documented above that a kludge could be used to bring the VPN up but this always failed again soon after.

I did find post elsewhere suggesting that the clock skew VPN issue was due to a faulty power adapter but I can't put my hands on the post at the moment. I have just received an RMA replacement and this will have to wait until I can find time configure and test.

Hope this helps