BR500 Connectivity Issue - Subscribe Here for Updates

[BretD]

Dear NETGEAR BR500 customers,

Over the past two weeks we have experienced extreme instability with our VPN cloud ecosystem. We believe it is due to the scale of customer base combined with our design tradeoffs on passing cloud status. We are working around the clock to identify a fix, test it and roll it out to production with as little ongoing disruption as possible. Unfortunately the current situation requires us to regularly reboot our cloud servers which is causing the disruption you are experiencing.

We are committed to having a robust and reliable VPN service and realize that we have not achieved this. For this we are truly sorry and we will continue to work on this until we have achieved this objective. We will be posting regular (several a day) updates on our corrective action status at this location in our community until we have completely resolved this issue. If you would prefer to get these status updates sent to you directly, login to the NETGEAR Community and subscribe to this thread. You can respond via the BR500 update thread. (Subscribe instructions can be found below).

Sincerely,

NETGEAR BR500 team

Insight Instant VPN Router

Subscribe Instructions:

This thread is locked except for Administrators. As updates on this the BR500 connectivity issues are avaliable this thread will be updated. For each reply thread subscribers will receive a community email update. (Depending on your community settings). Thank you for your understanding.

[BretD]

Update 10/30/2019 10:00 PM PT: Insight VPN Services are operational. 

[BretD]

Update 10/31/2019 2:00 AM PT: Insight VPN Services are operational.

• There was a temporary disruption around 11:30pm~00:30am PDT time on one server in East Coast, about 50 routers might have experienced an intermittent VPN disconnection/reconnection.

[BretD]

Update 10/31/2019 6:00 AM PT: Insight VPN Services are operational. 

[BretD]

Update 10/31/2019 10:00 AM PT: Insight VPN Services are operational.

• 3 additional servers are scheduled to go online today.

[BretD]

Update 10/31/2019 5PM PT: Current status - Since this issue surfaced last Thursday, ours VPN servers would start to drop connections when the number of connections got to a critical level.

To mitigate this issue, we added a new server in our West Coast datacenter on Oct 28th, and subsequently added 3 more servers at 12pm on Oct 31st, one on the West Coast, one on the East Coast of the US and one in Ireland. We now have 8 servers serving different regions.

We haven’t had any VPN disconnections since these 3 new servers were added. The system has been stable since our last post.

This is the temporary solution to keep the service stable. On the development side, the team continues to work on fixing the issue related to the unnecessary traffic flooding causing the servers to disconnect VPN requests from the BR500s in the field. We are targeting to release a test build to QA test and validation by early next week.

We apologize for the ongoing uncertainty.

[BretD]

Update: 10/31/2019 6PM PST Update. All servers are working. All servers have been stable since last update 4 hours ago.

Total connection is 508. Detail distribution is as below:

1: US-W               72

2: US-E                 93

3: EU                    93

4: US-E                 77

5: US-W               89

6: US-W               32

7: EU                    36

8: US-E                 16

[BretD]

Update: 10/31/2019 PST 10PM All servers heathy and responsive. Connect counts below:

Region	Current	4Hrs Ago
1: US-W	75	72
2: US-E	91	93
3: EU	87	93
4: US-E	77	77
5: US-W	85	89
6: US-W	35	32
7: EU	37	36
8: US-E	19	16
Total	506	508

[BretD]

2:00 am , 11/01/2019 PT Update.

All servers healthy and responsive. Connect counts below:

Region	Current	4Hrs Ago
1: US-W	74	75
2: US-E	92	91
3: EU	82	87
4: US-E	82	77
5: US-W	85	85
6: US-W	39	35
7: EU	44	37
8: US-E	21	19
Total	519	508

[BretD]

6:00 am , 11/01/2019 PT Update.

All servers healthy and responsive. Connect counts below:

Region	Current	4Hrs Ago
1: US-W	75	74
2: US-E	94	92
3: EU	74	82
4: US-E	77	82
5: US-W	84	85
6: US-W	43	39
7: EU	48	44
8: US-E	23	21
Total	518	519

[BretD]

10:00 am , 11/01/2019 PDT Update.

All servers healthy and responsive. Connect counts below:

Region	Current	4Hrs Ago
1: US-W	75	75
2: US-E	92	94
3: EU	71	74
4: US-E	76	77
5: US-W	86	84
6: US-W	51	43
7: EU	50	48
8: US-E	23	23
Total	524	518

[BretD]

2PM , 11/01/2019 PDT Update.

All servers healthy and responsive. Connect counts below:

Region	Current	4hrs Ago
1: US-W	79	75
2: US-E	93	92
3: EU	65	71
4: US-E	74	76
5: US-W	80	86
6: US-W	51	51
7: EU	49	50
8: US-E	26	23
Total	517	524

[BretD]

6PM , 11/01/2019 PDT Update.

All servers healthy and responsive. Connect counts below:

Region	Current	4hrs ago
1: US-W	82	79
2: US-E	89	93
3: EU	59	65
4: US-E	76	74
5: US-W	75	80
6: US-W	53	51
7: EU	49	49
8: US-E	24	26
Total	507	517

[BretD]

10PM , 11/01/2019 PDT Update.

All servers healthy and responsive. Connect counts below:

Region	Current	4H Ago
1: US-W	81	82
2: US-E	89	89
3: EU	57	59
4: US-E	75	76
5: US-W	74	75
6: US-W	55	53
7: EU	50	49
8: US-E	26	24
Total	507	507

[BretD]

2:17 am , 11/02/2019 PDT Update.

All servers healthy and responsive. Connect counts below:

Region	Current	4H Ago
1: US-W	81	81
2: US-E	91	89
3: EU	57	57
4: US-E	77	75
5: US-W	71	74
6: US-W	55	55
7: EU	51	50
8: US-E	27	26
Total	510	507

Note :

• Team is working on validation of v.1.0.6.1 conductor and proxy package.

[BretD]

6:17 am , 11/02/2019 PDT Update.

All servers healthy and responsive. Connect counts below:

Region	Current	4H Ago
1: US-W	80	81
2: US-E	91	91
3: EU	51	57
4: US-E	78	77
5: US-W	70	71
6: US-W	55	55
7: EU	56	51
8: US-E	29	27
Total	510	510

[BretD]

10 am , 11/02/2019 PDT Update.

All servers healthy and responsive. Connect counts below (Since total is below 500, we will pay attention to it)

Region	Current	4H Ago
1: US-W	78	80
2: US-E	88	91
3: EU	49	51
4: US-E	77	78
5: US-W	65	70
6: US-W	60	55
7: EU	49	56
8: US-E	27	29
Total	493	510

[BretD]

2PM, 11/02/2019 PDT.

 All servers healthy and responsive.

Region	Current	4H Ago	8H Ago
1: US-W	78	78	80
2: US-E	88	88	91
3: EU	46	49	51
4: US-E	79	77	78
5: US-W	65	65	70
6: US-W	63	60	55
7: EU	54	49	56
8: US-E	32	27	29
Total	505	493	510

[BretD]

6PM, 11/02/2019 PDT.

All servers healthy and responsive.

Region	Current	-4H	-8H	-12H
1: US-W	79	78	78	80
2: US-E	88	88	88	91
3: EU	43	46	49	51
4: US-E	80	79	77	78
5: US-W	64	65	65	70
6: US-W	64	63	60	55
7: EU	56	54	49	56
8: US-E	31	32	27	29
Total	505	505	493	510

[BretD]

10PM, 11/02/2019 PDT.

All servers healthy and responsive.

Region	Current	-4H	-8H	-12H	-16H
1: US-W	76	79	78	78	80
2: US-E	87	88	88	88	91
3: EU	42	43	46	49	51
4: US-E	80	80	79	77	78
5: US-W	64	64	65	65	70
6: US-W	63	64	63	60	55
7: EU	55	56	54	49	56
8: US-E	31	31	32	27	29
Total	498	505	505	493	510

[BretD]

2:00 am , 11/03/2019 PDT.

 

All servers healthy and responsive.

Region	current	-4H	-8H	-12H	-16H	-20H
1: US-W	77	76	79	78	78	80
2: US-E	90	87	88	88	88	91
3: EU	57	42	43	46	49	51
4: US-E	78	80	80	79	77	78
5: US-W	61	64	64	65	65	70
6: US-W	65	63	64	63	60	55
7: EU	38	55	56	54	49	56
8: US-E	31	31	31	32	27	29
Total	497	498	505	505	493	510

[BretD]

6:05 am , 11/03/2019 PDT.

All servers healthy and responsive.

Region	current	-4H	-8H	-12H	-16H	-20H	-24H
1: US-W	76	77	76	79	78	78	80
2: US-E	88	90	87	88	88	88	91
3: EU	56	57	42	43	46	49	51
4: US-E	79	78	80	80	79	77	78
5: US-W	61	61	64	64	65	65	70
6: US-W	64	65	63	64	63	60	55
7: EU	38	38	55	56	54	49	56
8: US-E	33	31	31	31	32	27	29
Total	495	497	498	505	505	493	510

[BretD]

10 am , 11/03/2019 PST. (New half day update)

No server rebooted and no VPN connection disrupted since last update.

Region	Current	-12H	-24H
1: US-W	78	76	78
2: US-E	87	87	88
3: EU	39	42	49
4: US-E	80	80	77
5: US-W	62	64	65
6: US-W	65	63	60
7: EU	57	55	49
8: US-E	34	31	27
Total	502	498	493

[BretD]

10 pm , 11/03/2019 PST.

No server rebooted and no customer VPN connection disrupted since last update.

Region	Current	-12H	-1D	-1.5D
1: US-W	79	78	76	78
2: US-E	84	87	87	88
3: EU	31	39	42	49
4: US-E	83	80	80	77
5: US-W	60	62	64	65
6: US-W	69	65	63	60
7: EU	62	57	55	49
8: US-E	33	34	31	27
Total	501	502	498	493

[BretD]

 10AM , 11/04/2019 PST.

No server rebooted and no customer VPN connection disrupted since last update.

Region	Current	-0.5Day	-1D	-1.5D	-2D
1: US-W	80	79	78	76	78
2: US-E	86	84	87	87	88
3: EU	28	31	39	42	49
4: US-E	85	83	80	80	77
5: US-W	62	60	62	64	65
6: US-W	72	69	65	63	60
7: EU	70	62	57	55	49
8: US-E	35	33	34	31	27
Total	518	501	502	498	493