Re: RBR850 Frequent Disconnects Starting March 9th AM EST

MeGrimlock · ‎2020-03-19

Similar to others this morning, so far so good on the updated FW. Network seems stable and no drops from any devices so far. Of course I'll be testing the entire day, so hopefully I can report the same later tonight.

@Blanca_O - Can we learn what the root cause was and what the fix was, so that we can be aware of what to look for if possible, triage and report back if anything goes sideways during the day?

Chivvalry · ‎2020-03-19

New firmware downloaded and installed and daughter reports that WoW is working. Also, my video conferencing is working as well! Seems like a successful fix.

tigerisak · ‎2020-03-19

Im on V3.2.10.13_1.2.13T1. But are still having dropouts for Zoom, Aircall, Teams and IPTV. Any suggestions?

d1mmy · ‎2020-03-19

I'm not even getting an updated firmware pushed to me and my devices are still offline. My devices are still on the original, bad, 3.2.10.11_1.2.12. I assume xxx.13T1 may mean it's a test version?

@tigerisak wrote:
Im on V3.2.10.13_1.2.13T1. But are still having dropouts for Zoom, Aircall, Teams and IPTV. Any suggestions?

tigerisak · ‎2020-03-19

Yes its a BETA that can be downloaded through NG if you contact support and sign a NDA. But it doesent help me.

bluegoose987 · ‎2020-03-19

That is not at all what is being indicated by NG or other users... NG has been in saying they are pushing it not to contact support and sign an NDA.

Ah sorry, that was for T1... not 10.11 Apologies

FURRYe38 · ‎2020-03-19

Downgrade to v3.2.9.2 then let the system get pushed to v10.11...

@tigerisak wrote:
Im on V3.2.10.13_1.2.13T1. But are still having dropouts for Zoom, Aircall, Teams and IPTV. Any suggestions?

FURRYe38 · ‎2020-03-19

Downgrade to v3.2.9.2 then let the system get pushed to v10.11...factory reset then setup from scratch. v10.13T1 is not recommended.

@d1mmy wrote:
I'm not even getting an updated firmware pushed to me and my devices are still offline. My devices are still on the original, bad, 3.2.10.11_1.2.12. I assume xxx.13T1 may mean it's a test version?

@tigerisak wrote:
Im on V3.2.10.13_1.2.13T1. But are still having dropouts for Zoom, Aircall, Teams and IPTV. Any suggestions?

adr1974 · ‎2020-03-19

I'm confused...I thought 3.2.10.11_1.2.12 was the update which caused all the issue.

xnolightx · ‎2020-03-19

I'm not having stellar results today. I pulled the latest firmware (V3.2.10.11_1.2.12) hours ago and at first things seemed to be stable. In the last half hour I've had 3 disconnects from VPN and about 10 disconnects from Remote Desktop. I did not perform a hard reset before upgrading, so that might be the next avenue.

I don't want to be pessimistic, but if that doesn't work I'm not going to spend any more time troubleshooting this. I'm just too busy at the moment to have an unstable internet connection.

computim · ‎2020-03-19

@adr1974 yes, it is but NG obviously noticed something in their code and resolved it. I can confirm after reviewing my logs that all my systems appear stable from the upgrade this morning.

adr1974 · ‎2020-03-19

i hear you. i updated on mar 9 or so, and system has been pretty stable (with the exception of my voip phone...but i upgraded the phone's firmware yesterday and it's been working fine since). wondering what changed for people if this is the same version which was released ten days ago. wouldn't they change the release number if they changed the code?

ScottD67 · ‎2020-03-19

Still good after 4-5 hours of heavy use this morning. Lots of streaming, video conferences, wifi calls and no drops to report on the IOT devices where it was happening earlier. FWIW I was stable after downgrading to v3.1.5.23_1.0.46 on both the RBR and RBS. I removed the site block which let both the RBR and RBS upgrade to 3.2.9.2 overnight last night and then I forced the RBR and RBS to pull 3.2.10.11 when I saw it available this morning. I never did a master reset in that process.

FURRYe38 · ‎2020-03-19

I would factory reset and setup up from scratch when you get a chance too to see if this helps any.

@xnolightx wrote:
I'm not having stellar results today. I pulled the latest firmware (V3.2.10.11_1.2.12) hours ago and at first things seemed to be stable. In the last half hour I've had 3 disconnects from VPN and about 10 disconnects from Remote Desktop. I did not perform a hard reset before upgrading, so that might be the next avenue.

I don't want to be pessimistic, but if that doesn't work I'm not going to spend any more time troubleshooting this. I'm just too busy at the moment to have an unstable internet connection.

FURRYe38 · ‎2020-03-19

Thanks for letting us know. Hope it keeps working for you. Be sure to do a backup configuration to file for safe keeping. Saves time when or if a reset is needed.

Enjoy.

@ScottD67 wrote:
Still good after 4-5 hours of heavy use this morning. Lots of streaming, video conferences, wifi calls and no drops to report on the IOT devices where it was happening earlier. FWIW I was stable after downgrading to v3.1.5.23_1.0.46 on both the RBR and RBS. I removed the site block which let both the RBR and RBS upgrade to 3.2.9.2 overnight last night and then I forced the RBR and RBS to pull 3.2.10.11 when I saw it available this morning. I never did a master reset in that process.

xnolightx · ‎2020-03-19

This may have been load on our corporate VPN/WAN as thousands of users are home. I'm going to be keeping an eye on it today. Hopefully it was just the load on the VPN/WAN and not the router.

FURRYe38 · ‎2020-03-19

Fw wasn't changed. It was on there cloud system side.

@adr1974 wrote:
i hear you. i updated on mar 9 or so, and system has been pretty stable (with the exception of my voip phone...but i upgraded the phone's firmware yesterday and it's been working fine since). wondering what changed for people if this is the same version which was released ten days ago. wouldn't they change the release number if they changed the code?

FURRYe38 · ‎2020-03-19

Hope so too.

@xnolightx wrote:
This may have been load on our corporate VPN/WAN as thousands of users are home. I'm going to be keeping an eye on it today. Hopefully it was just the load on the VPN/WAN and not the router.

adr1974 · ‎2020-03-19

gotcha -- thanks!

ErnestTheGreat · ‎2020-03-19

The FW you have is test FW, there may be other issues in that FW. My suggestion would be to install 3.2.9.2 FW and have it updated to 3.2.10.11 and see if that helps. The 3.2.10.11 version has the update now with the fix. If the issue continues, I would suggest to PM one of the admins with your info for further assistance..

warpdag · ‎2020-03-19

Here is some food for thoughts for y'all:

"Original" broken firmware from last week, still available here: http://www.downloads.netgear.com/files/GDC/RBK852/RBR850-V3.2.10.11_1.2.12.zip

File in the zip dated 3/12, so that's the original one

=> MD5: 0787706e2d5ee94805725604de6048e3

"New" firmware on the Orbi update server, available here: https://http.fw.updates1.netgear.com/rbr850/default/ww/RBR850-V3.2.10.11_1.2.12.chk

Manifest of the file available here (that's what Orbi checks for its updates): https://http.fw.updates1.netgear.com/rbr850/default/ww/fileinfo.txt

=> MD5: 0787706e2d5ee94805725604de6048e3

Firmware revisions matched when the "update" was rolled out, that was a first clue.

Both MD5 hashes match, so exact same firmware. Interesting.

The same goes for the satellite firmware.

Placebo effect? I'll let you jump to your own conclusions.

cruisencode · ‎2020-03-19

@warpdag wrote:
Here is some food for thoughts for y'all:

"Original" broken firmware from last week, still available here: http://www.downloads.netgear.com/files/GDC/RBK852/RBR850-V3.2.10.11_1.2.12.zip
File in the zip dated 3/12, so that's the original one
=> MD5: 0787706e2d5ee94805725604de6048e3

"New" firmware on the Orbi update server, available here: https://http.fw.updates1.netgear.com/rbr850/default/ww/RBR850-V3.2.10.11_1.2.12.chk
Manifest of the file available here (that's what Orbi checks for its updates): https://http.fw.updates1.netgear.com/rbr850/default/ww/fileinfo.txt
=> MD5: 0787706e2d5ee94805725604de6048e3

Firmware revisions matched when the "update" was rolled out, that was a first clue.

Both MD5 hashes match, so exact same firmware. Interesting.

The same goes for the satellite firmware.

Placebo effect? I'll let you jump to your own conclusions.

I believe the "original" version was deployed on 3/9. So, if the file is dated 3/12 then this would not be the same file.

warpdag · ‎2020-03-19

The "fix" was "pushed" last night

Chivvalry · ‎2020-03-19

@warpdag wrote:
The "fix" was "pushed" last night

Most likely scenario is that the fix was in place in the new 3/12 file and they spent the rest of the time testing it to make sure. Two screw ups in a row would have been catastrophic.

cruisencode · ‎2020-03-19

@warpdag wrote:
The "fix" was "pushed" last night

I was commenting on the fact that the FW must be "new" if the file in the zip is dated 3/12, since the original (broken) version was pushed on 3/9.