GC752XP Switch Firmware 1.0.5.38 Issues high CPU Rebooting

[RonBay]

I started having issues recently on firmware 1.0.5.38 with high CPU and random reboots after using more ports than normal.  I have IGMP snooping disabled on VLAN's and have two Insight switches. 

I noticed I had a POE camera that was flapping a port due to it's own issues, so the port and POE were turned off.  However, the switch issues continued.

I was able to downgrade to 1.0.5.36 and unplugged everything from the switch except LAN and Internet access. Booted and connected to Insight fine.  Thought maybe corrupt firmware update through Insight, I manually downloaded from website and installed back 1.0.5.38 through local console.  Shortly after booting it went back to 90+% utilization and after 6 minutes rebooted. Further reboots all yielded same issue.

I've been on FW 1.0.5.36 continuously for over a day and not having any issues with CPU utilization around 20%. So I'm eliminating any hardware issues with connected devices or the switch itself as it's been stable on this version.

[sbs101]

We have the same problem with one of our GC752XP-switches after updating the firmware from 1.0.5.36 to 1.0.5.38.

After the update of the firmware via Insight we experienced frequent reboots every 6 or 7 minutes. After some time the reboots stopped and currently the switch keeps running. In the monitoring we however notice strange things:

In the memory log we see this line:

<9>1 2023-07-26T14:29:37.297+2:00Z S24-1 OpEN tRpcsrv.01000 - :openapi_logging.c(1294) 522 %% CloudAgent: check_auto_reclaim:Reset file does not exist.

and the application information claims that it is not connected to the cloud:

App Name	App Status	Version
CloudAgent	Not connected to cloud	6.2.0.60200019
ConfigAgent	Operational	6.2.0.60200019
DiscoveryAgent	Operational	6.2.0.60200019

[BrendanM]

Hi @RonBay Hi @sbs101 

I'm sorry to hear that you have had this problem. If either of you still have a unit running 1.0.5.38 and which is showing the problem, can you please export a tech support file from the switch and send it to us?

To get the tech support file, on the local admin page, go to Maintenance - Export - HTTP File Export. From the menu, choose Tech Support and click Apply.

Please then email this to switch_support (a) netgear.com referencing this thread.

Thank you,

Brendan M

[RonBay]

Hi @BrendanM 

The switch is currently in use.  However, maybe later this week I can replace the firmware, collect the logs and roll back. 

[Gedruspax]

I have 9 of these switches in my corporate production environment but I have not updated to 1.0.5.38 after seeing this post.  They are all still on 1.0.5.36.  If there is indeed an issue with 1.0.5.38 then I certainly don't want to update them so lets keep this thread updated on what's happening with this.

[sbs101]

I have provided the requested log file to the switch support. I hope it helps to solve the issue.

BTW: I have to admit that I made a mistake in my above reply. Our switches are not GC752XP but GC728XP, but it looks like the issue is about the same.

Some more details on our setup which might help debugging: We use the switches mostly to power PoE Access Points, which means that there is a huge amount of traffic coming from the APs after the PoE power is switched on after the reboot of the switch and all APs boot simultaniously. Besides that we have maintained many PoE schedules via Insight.

[BrendanM]

Hi @sbs101 

I didn't see the email with the tech support file. I will PM you.

Thanks,

BrendanM

[Gedruspax]

Do either of you have any further updates on this?

[Gedruspax]

I noticed this weekend that this firmware has been pulled from insight and is no longer available for download.

[BrendanM]

Hi @Gedruspax 

While this issue described in this thread was not widely seen, we have removed the firmware out of an abundance of caution.

Hi @RonBay @sbs101 

We have identified a potential fix for this issue and one of my colleagues will be sending you a PM to see if you will be in a position to try the fix.