WAC720 rebooting for no apparent reason

[PC-David]

I have 10x WAC720 AP's in an ensemble, all on the latest firmware (3.9.1.13), and they seem to randomly reboot causing all connected clients to drop or switch to another access point. They don't all do it at once, though sometimes a few will go off at the same time or in succession. It seems to happen more often when there's a lot of people around with devices, but it has also happened when nobody is around and there's only ~3 idle devices connected. The AP's are powered from a mix of PoE switches (HP and Dell) and injectors around the building, all on battery backups.

I enabled syslog and checked it around the time of a reboot but nothing seems out of the ordinary. CPU utilization is low and memory is under 75%. They're all in very heavily air conditioned spaces so it's unlikely to be thermal either unless there's an internal problem. It's driving us mad and causing disruption to some of our operations, and I'd really hate to waste $1,000 +labor replacing all of these... but I'm running out of options.

[plemans]

have you been able to isolate it to a few devices or is it everyone of them? 

Reason I ask is if you can isolate it to a few, maybe its the POE connection causing the issue. 

with multiple devices having the same issue, I wouldn't be looking at a hardware issue with the router. 

I'd either be looking at the firmware or whats powering them. 

[PC-David]

It is unfortunately all of them. Some do it more than others, but they all do it.

[plemans]

Did it just start recently or how long has it been going on? 

I'd try downgrading the firmware and getting in touch with a mod from the business side. 

If its a firmware issue, it'd be good for them to know. 

Those devices do have a lifetime warranty but its doubtful that all 10 would have hardware failure at once. 

@BretD @DavidGo 

[PC-David]

It's been happening for at least the last year. I can try downgrading the firmware if I have time this weekend.

[DaneA]

@PC-David,

Kindly answer the questions below: 

1. From your initial post, you mentioned “The AP's are powered from a mix of PoE switches (HP and Dell) and injectors around the building, all on battery backups.”  What is the model of the switches where the APs are connected and how many APs are connected to each switch?

2. Are there any other PoE devices being powered by the switches aside from the WAC720 APs?

3. Are there any PoE related logs being seen on the switches?

Kindly provide the syslogs when reboot is seen. Also, please get the WAC720 AP logs from the web-GUI.  Kindly upload the syslogs and AP logs via Google Drive then send the Google Drive link to me via private message so that I can download it. 

Regards,

DaneA
NETGEAR Community Team

[PC-David]

• Switch #1 is an HP ProCurve 3500yl-48G (J8693A). It has 3 access points, 5 phones, and 5 IP cameras.
• Switch #2 is the same model with 5 access points, 3 phones, and 8 IP cameras.
• Switch #3 is a Dell PowerConnect 5524P with 1 access point and no other PoE devices.
• Access point #10 uses a PoE injector... I am not sure of the model I would have to look tomorrow.

With PoE budgets of nearly 400W the 2 HP switches have a huge amount of headroom compared to their average demand which is under 100W. None of the switches have any PoE related log entries or errors.

I will send you the logs when I get into the office tomorrow.