Anyone else wake up to a dozen failed firmware update emails/alerts for the WAX620s?

[jasondunn]

I have three WAX620 APs deployed in my home. V10.8.5.7 firmware on all of them. I do not pay the fee to get the app-based service, I manually update the firmware on all of them. I woke up this morning to 12 emails and 12 app alerts, three per AP, all saying the same vague thing. Why exactly would devices that I control and need do manual firmware updates on be pushed a firmware update that failed? This is somewhat concerning to me as I didn't see any communications from Netgear about an emergency firmware they were pushing out to all APs or the like.

Failure Reason: An error occurred while updating the firmware.

Failure Reason: The device configuration has failed

[BrendanM · Accepted solution]

Here is some additional context on this topic. On 30th March, NETGEAR released firmware updates (v10.8.5.7/v9.9.4.7) for our Insight-manageable wireless access points. Following long-term stress testing of this firmware, it was found that access points running this firmware could, under certain conditions, encounter a memory leak. This was dependent on configuration and load on the access point. As result of this memory leak, some affected APs may have unexpectedly rebooted. The firmwares were taken down on 13th April. On the same day, we performed some mitigation activities on the Insight platform to stabilize the affected APs. As a result of this mitigation activity, some Insight accounts, to which the affected APs were registered, may have received some false positive Insight notifications between 07:00 to 08:45 PDT on 13th April. These notifications will have been similar to those received by @jasondunn and can be safely ignored. A new firmware is being released today to address this issue. It is recommended to upgrade to this new version as soon as possible.

@jasondunn  - are you certain that your APs are not managed by Insight? You can check the management mode by logging on to the admin page and going to Management - Configuration - System - Basic - Management Mode. Please send me a PM if you are unsure.

@RipTWD  - since your APs are running v10.8.5.7, then the unexplained reboot may have been caused by the memory leak described above. When the new firmware releases today, please upgrade as soon as possible. We should still try to understand how your APs got to v10.8.5.7. Did you manually update the APs after March 30th?

[RipTWD]

I saw that too.  My 4 WAX-615's all self-rebooted at around 10:44 ET this morning.  

[RennaD]

Hello @jasondunn @RipTWD,

If you received the notifications last April 13th around 7:15AM PST to 8:50AM PST then this is expected. On that day, engineering have applied changes through Insight which would help stabilize APs running on latest firmware. As a result you may have received the notification "The device configuration has failed."  and in some cases, "Failure Reason: An error occurred while updating the firmware." Some APs, which has ran into low memory, would have rebooted.

Have a lovely day,
RennaD
NETGEAR Team

[jasondunn]

Renna,

I thought I owned these APs and I could decide when and how firmware updates were applied? What is the circumstance when Netgear decides a firmware update is forced upon an AP?

[RennaD]

Hello @jasondunn,

What was applied was not a firmware update, but rather an enhancement to APs that are on latest firmware which your APs are on (V10.8.5.7). Are your WAX620 set to Insight Management mode? Can you check if Schedule Firmware Update is enabled on the Insight Portal?

Have a lovely day,
RennaD
NETGEAR Team

[RipTWD]

Mine were not set to automatically update

[BrendanM · Accepted solution]

Here is some additional context on this topic. On 30th March, NETGEAR released firmware updates (v10.8.5.7/v9.9.4.7) for our Insight-manageable wireless access points. Following long-term stress testing of this firmware, it was found that access points running this firmware could, under certain conditions, encounter a memory leak. This was dependent on configuration and load on the access point. As result of this memory leak, some affected APs may have unexpectedly rebooted. The firmwares were taken down on 13th April. On the same day, we performed some mitigation activities on the Insight platform to stabilize the affected APs. As a result of this mitigation activity, some Insight accounts, to which the affected APs were registered, may have received some false positive Insight notifications between 07:00 to 08:45 PDT on 13th April. These notifications will have been similar to those received by @jasondunn and can be safely ignored. A new firmware is being released today to address this issue. It is recommended to upgrade to this new version as soon as possible.

@jasondunn  - are you certain that your APs are not managed by Insight? You can check the management mode by logging on to the admin page and going to Management - Configuration - System - Basic - Management Mode. Please send me a PM if you are unsure.

@RipTWD  - since your APs are running v10.8.5.7, then the unexplained reboot may have been caused by the memory leak described above. When the new firmware releases today, please upgrade as soon as possible. We should still try to understand how your APs got to v10.8.5.7. Did you manually update the APs after March 30th?

[schumaku]

---

@BrendanM wrote:

....
@RipTWD  - since your APs are running v10.8.5.7, then the unexplained reboot may have been caused by the memory leak described above. When the new firmware releases today, please upgrade as soon as possible. We should still try to understand how your APs got to v10.8.5.7. Did you manually update the APs after March 30th?

---

v10.8.5.7 was certainly made available by Insight on 30 March 2024, we took that plunge, too.

Same for all 615, 620, 625, 630E, ... in our inventory. 

Greetings to Ireland!

[jasondunn]

Ok, this is clear now. Yes I updated all three APs myself with the 10.8.5.7 firmware a couple of weeks ago. What’s not clear is why I received an email this morning telling me that 10.8.6.3 is available but when I log into my AP running 10.8.5.7 and force a firmware check it says there’s no new firmware available. 🫤

Do the emails go out before the firmware is actually available?

[BrendanM]

Hi @jasondunn - as of writing this, the firmware has been posted to the NETGEAR support site and has also been made available for installation through Insight (I suspect you received the Insight notification). It will become available via the 'Check for upgrade' button in the local admin page of the AP very soon.

[RipTWD]

>> Following long-term stress testing of this firmware, it was found that access points running this firmware could, under certain conditions, encounter a memory leak. This was dependent on configuration and load on the access point. As result of this memory leak, some affected APs may have unexpectedly rebooted. 

All four, at the same time?  I applaud myself on how perfectly I load balanced my access points 🙂

I'll update this weekend & keep an eye on things.

Thanks

[jasondunn]

I'm getting emails urging me to update the firmware, yet just a few minutes ago I checked again and my AP will not see a newer firmware. I have one of my three APs still under the trial of Insight cloud management, so I tried from the app, and that failed with a very "There was an error" message. Why would Netgear send communications to customers urging them to update but not have the update available? This is quite frustrating.

[RennaD]

Hello @jasondunn,

Checking the attached screenshot, the AP's firmware is already the latest which is version 10.8.6.3. As for the AP that is on Insight, can you try to Update again. If same issue exist I suggest to do a local update via the UI. Please see steps below:

1. Go to Management - Maintenance - Upgrade - Firmware Upgrade
2. Click Upgrade Now if the Latest available version is 10.8.6.3
3. If it's not, kindly download it using the link below then unzip the file:
https://www.downloads.netgear.com/files/GDC/WAX620/WAX620_firmware_V10.8.6.3.zip
4. Select Local for Upgrade Options
5. click Browse File, locate the firmware file then click Upgrade

Have a lovely day,
RennaD
NETGEAR Team

[jasondunn]

I took a screenshot of the wrong AP, that one was updated, but the other two are stuck on V10.8.5.7 and no amount of checking for updates will get the AP to update, so I'll pursue the manual update route. But why doesn't the manually-initiated process work?

[aabecst]

Are the AP offline from Insight ? 

Try rebooting the AP using the POE port,  once the AP come up-online again,  Try upgrade using the "firmware upgrade" buttons on insight.

[jasondunn]

I don't use Inisght to manage them (paying a fee isn't worth it for me), so I usually upgrade via IP connection. That wasn't working (no new updates were detected) so I had to download the firmware and apply it manually. But all are updated now to the latest.

[schumaku]

Plenty of other reasons to use full Insight, several more than just updating the APs. Seamless roaming of WPA3-SAE clients for example... which can't work without assistance (different from the comparable simple WPA2-PSK).