NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
BR500 Connectivity Issue - Subscribe Here for Updates
Dear NETGEAR BR500 customers,
Over the past two weeks we have experienced extreme instability with our VPN cloud ecosystem. We believe it is due to the scale of customer base combined with our desi...
10AM , 11/04/2019 PST.
No server rebooted and no customer VPN connection disrupted since last update.
|
Region |
Current |
-0.5Day |
-1D |
-1.5D |
-2D |
|
1: US-W |
80 |
79 |
78 |
76 |
78 |
|
2: US-E |
86 |
84 |
87 |
87 |
88 |
|
3: EU |
28 |
31 |
39 |
42 |
49 |
|
4: US-E |
85 |
83 |
80 |
80 |
77 |
|
5: US-W |
62 |
60 |
62 |
64 |
65 |
|
6: US-W |
72 |
69 |
65 |
63 |
60 |
|
7: EU |
70 |
62 |
57 |
55 |
49 |
|
8: US-E |
35 |
33 |
34 |
31 |
27 |
|
Total |
518 |
501 |
502 |
498 |
493 |
10PM , 11/04/2019 PST.
No server rebooted.
New connections are OK.
There were two cases reported in CESMB-1124 that routers were removed from VPN group, result in group mismatch between VPN server and Insight server, investigating as P1 issue.
|
Region |
Current |
-0.5D |
-1Day |
-1.5D |
-2D |
|
1: US-W |
79 |
80 |
79 |
78 |
76 |
|
2: US-E |
87 |
86 |
84 |
87 |
87 |
|
3: EU |
22 |
28 |
31 |
39 |
42 |
|
4: US-E |
79 |
85 |
83 |
80 |
80 |
|
5: US-W |
63 |
62 |
60 |
62 |
64 |
|
6: US-W |
73 |
72 |
69 |
65 |
63 |
|
7: EU |
65 |
70 |
62 |
57 |
55 |
|
8: US-E |
38 |
35 |
33 |
34 |
31 |
|
Total |
506 |
518 |
501 |
502 |
498 |
10AM , 11/05/2019 PST.
All servers operational. No server rebooted. No customer issue reported.
Region
Current
-0.5D
-1D
-1.5Day
-2D
1: US-W
76
79
80
79
78
2: US-E
87
87
86
84
87
3: EU
25
22
28
31
39
4: US-E
82
79
85
83
80
5: US-W
63
63
62
60
62
6: US-W
76
73
72
69
65
7: EU
67
65
70
62
57
8: US-E
39
38
35
33
34
Total
515
506
518
501
502
10PM , 11/05/2019 PST.
All servers operational.
As reported at 4PM, VPN server 4 in Virginia had memory leak issue between 1:30PM and 3PM.
User may experience intermittent VPN connection drop if they were using VPN during that period, it was fully operational after VPN service was restarted(Actual restart takes a few seconds to complete)
No customer issue reported.
Region
Current
-6H
-0.5D
-1D
-1.5D
-2D
1: US-W
78
77
76
79
80
79
2: US-E
83
85
87
87
86
84
3: EU
20
23
25
22
28
31
4: US-E
79
82
82
79
85
83
5: US-W
58
57
63
63
62
60
6: US-W
76
74
76
73
72
69
7: EU
64
62
67
65
70
62
8: US-E
36
38
39
38
35
33
Total
494
498
515
506
518
501
Update on new server software:
New VPN conductor server and proxy server software have been deployed on QA platform, It has been operational since 11/04. QA is performing functional and load/stress tests.
10AM , 11/06/2019 PST.
All servers operational. No customer issue reported.
Region
Current
-0.5Day
-2D
-1.5D
-2D
1: US-W
81
78
76
79
80
2: US-E
86
83
87
87
86
3: EU
24
20
25
22
28
4: US-E
81
79
82
79
85
5: US-W
58
58
63
63
62
6: US-W
73
76
76
73
72
7: EU
67
64
67
65
70
8: US-E
37
36
39
38
35
Total
507
494
515
506
518
10PM , 11/06/2019 PST.
All servers operational. No customer issue reported.
Region
Current
-0.5Day
-1D
-1,5DD
-2D
1: US-W
80
81
78
76
79
2: US-E
86
86
83
87
87
3: EU
33
24
20
25
22
4: US-E
80
81
79
82
79
5: US-W
55
58
58
63
63
6: US-W
71
73
76
76
73
7: EU
69
67
64
67
65
8: US-E
39
37
36
39
38
Total
513
507
494
515
506
Current Status:
We are testing candidate builds for a permanent fix and hope to finish QA by early next week. In the interim we continue to closely monitor server loading to prevent outages.
10AM , 11/07/2019 PST.
All servers operational. No customer issue reported.
Region
Current
-0.5Day
-1D
-1.5D
-2D
1: US-W
82
80
81
78
76
2: US-E
90
86
86
83
87
3: EU
36
33
24
20
25
4: US-E
81
80
81
79
82
5: US-W
54
55
58
58
63
6: US-W
78
71
73
76
76
7: EU
66
69
67
64
67
8: US-E
40
39
37
36
39
Total
527
513
507
494
515
Current Status:
We are testing candidate builds for a permanent fix and hope to finish QA by early next week. In the interim we continue to closely monitor server loading to prevent outages.
10PM , 11/07/2019 PST.
All servers operational. No customer issue reported.
Region
Current
-0.5Day
-1Day
-1.5D
-2D
-2.5D
1: US-W
80
82
80
81
78
76
2: US-E
87
90
86
86
83
87
3: EU
35
36
33
24
20
25
4: US-E
82
81
80
81
79
82
5: US-W
51
54
55
58
58
63
6: US-W
80
78
71
73
76
76
7: EU
65
66
69
67
64
67
8: US-E
40
40
39
37
36
39
Total
520
527
513
507
494
515
Dev Update: A check is being performed on the integrity of the VPN database. Actions upon analysis it needed.
10AM , 11/08/2019 PST.
All servers operational. No customer issue reported.
Region
Current
-0.5Day
-1Day
-1.5Day
-2D
-2.5D
-3D
1: US-W
82
80
82
80
81
78
76
2: US-E
88
87
90
86
86
83
87
3: EU
36
35
36
33
24
20
25
4: US-E
70
82
81
80
81
79
82
5: US-W
63
51
54
55
58
58
63
6: US-W
78
80
78
71
73
76
76
7: EU
67
65
66
69
67
64
67
8: US-E
46
40
40
39
37
36
39
Total
530
520
527
513
507
494
515
On server status monitor:
Requested CloudOPS to lower memory usage alarm level from 60% to 40%, in effort to reduce expected down time to be less than 30 minutes for any server might running to issue.
On server update:
We received new server code with fixes to bugs exposed in high availability tests, high availability and stress tests will resume on Monday.
On VPN support cases:
Bug reported that specific user can’t delete VPN group, or router does not belong to any group.
This is related to 10/24 deployment. We are trying to identify the root cause.
10PM , 11/08/2019 PST.
All servers operational. No customer issue reported.
Region
Current
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
1: US-W
82
82
80
82
80
81
78
2: US-E
86
88
87
90
86
86
83
3: EU
35
36
35
36
33
24
20
4: US-E
59
70
82
81
80
81
79
5: US-W
65
63
51
54
55
58
58
6: US-W
79
78
80
78
71
73
76
7: EU
63
67
65
66
69
67
64
8: US-E
48
46
40
40
39
37
36
Total
517
530
520
527
513
507
494
10AM , 11/09/2019 PST.
All servers operational. No customer issue reported.
Region
Current
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
1: US-W
83
82
82
80
82
80
81
2: US-E
85
86
88
87
90
86
86
3: EU
34
35
36
35
36
33
24
4: US-E
56
59
70
82
81
80
81
5: US-W
67
65
63
51
54
55
58
6: US-W
80
79
78
80
78
71
73
7: EU
64
63
67
65
66
69
67
8: US-E
54
48
46
40
40
39
37
Total
523
517
530
520
527
513
507
Note:
Thanks CloudOPS team! Alarm will be triggered if any VPN proxy server memory usage level reaches 40%.
Response team will then monitor server health closely to determine if any action should be taken.
This should minimize user VPN downtime to less than 30 minutes even if any server runs into problem.
10PM , 11/09/2019 PST.
All servers operational. No alarm received. No customer issue reported.
Region
Current
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
1: US-W
79
83
82
82
80
82
80
2: US-E
84
85
86
88
87
90
86
3: EU
35
34
35
36
35
36
33
4: US-E
55
56
59
70
82
81
80
5: US-W
66
67
65
63
51
54
55
6: US-W
80
80
79
78
80
78
71
7: EU
63
64
63
67
65
66
69
8: US-E
50
54
48
46
40
40
39
Total
512
523
517
530
520
527
513
10AM , 11/10/2019 PST.
All servers operational. No alarm received. No customer issue reported.
Region
Current
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
1: US-W
83
79
83
82
82
80
82
2: US-E
85
84
85
86
88
87
90
3: EU
35
35
34
35
36
35
36
4: US-E
51
55
56
59
70
82
81
5: US-W
64
66
67
65
63
51
54
6: US-W
81
80
80
79
78
80
78
7: EU
65
63
64
63
67
65
66
8: US-E
54
50
54
48
46
40
40
Total
518
512
523
517
530
520
527
10PM , 11/10/2019 PST.
All servers operational. No alarm received. No customer issue reported.
Region
Current
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
1: US-W
87
83
79
83
82
82
80
2: US-E
84
85
84
85
86
88
87
3: EU
35
35
35
34
35
36
35
4: US-E
50
51
55
56
59
70
82
5: US-W
67
64
66
67
65
63
51
6: US-W
80
81
80
80
79
78
80
7: EU
65
65
63
64
63
67
65
8: US-E
55
54
50
54
48
46
40
Total
523
518
512
523
517
530
520
10AM , 11/11/2019 PST.
All servers operational. No alarm received. No customer issue reported.
Region
Current
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
1: US-W
87
87
83
79
83
82
82
2: US-E
86
84
85
84
85
86
88
3: EU
35
35
35
35
34
35
36
4: US-E
47
50
51
55
56
59
70
5: US-W
68
67
64
66
67
65
63
6: US-W
81
80
81
80
80
79
78
7: EU
65
65
65
63
64
63
67
8: US-E
57
55
54
50
54
48
46
Total
526
523
518
512
523
517
530
Notes:
We will continue to focus on new VPN server software validation this week
10PM , 11/11/2019 PST.
All servers operational. No alarm received. No customer issue reported.
Region
Current
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
1: US-W
87
87
87
83
79
83
82
2: US-E
86
86
84
85
84
85
86
3: EU
35
35
35
35
35
34
35
4: US-E
45
47
50
51
55
56
59
5: US-W
69
68
67
64
66
67
65
6: US-W
83
81
80
81
80
80
79
7: EU
62
65
65
65
63
64
63
8: US-E
58
57
55
54
50
54
48
Total
525
526
523
518
512
523
517
10AM , 11/12/2019 PST.
All servers operational. No alarm received.
Region
Current
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
1: US-W
87
87
87
87
83
79
83
2: US-E
86
86
86
84
85
84
85
3: EU
35
35
35
35
35
35
34
4: US-E
39
45
47
50
51
55
56
5: US-W
75
69
68
67
64
66
67
6: US-W
81
83
81
80
81
80
80
7: EU
67
62
65
65
65
63
64
8: US-E
63
58
57
55
54
50
54
Total
533
525
526
523
518
512
523
Notes:
- VPN group info mismatch issue: We have validated automated procedure to sync Insight database with VPN server database. The sync is scheduled on Indian 11/13. Actual time depends on final approvals.
- On Server Fix: Validation is still ongoing with QA servers, there were some abnormities that needed clarifications from development.
- One customer reported long site to site VPN traffic latency. We are investigating.
10PM , 11/12/2019 PST.
All servers operational. No alarm received. No new support case reported.
Region
Current
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
1: US-W
86
87
87
87
87
83
79
2: US-E
85
86
86
86
84
85
84
3: EU
36
35
35
35
35
35
35
4: US-E
37
39
45
47
50
51
55
5: US-W
75
75
69
68
67
64
66
6: US-W
81
81
83
81
80
81
80
7: EU
65
67
62
65
65
65
63
8: US-E
63
63
58
57
55
54
50
Total
528
533
525
526
523
518
512
3.30 AM , 11/13/2019 PST.
All servers operational.
Region
current
-5hours
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
1: US-W
95
86
87
87
87
87
83
79
2: US-E
84
85
86
86
86
84
85
84
3: EU
65
36
35
35
35
35
35
35
4: US-E
61
37
39
45
47
50
51
55
5: US-W
59
75
75
69
68
67
64
66
6: US-W
97
81
81
83
81
80
81
80
7: EU
65
65
67
62
65
65
65
63
8: US-E
63
63
63
58
57
55
54
50
Total
589
528
533
525
526
523
518
512
Note:
Service on one of the server in US-W (44.224.142.224) went down. The same was detected and made operational with in 10 mins.
11AM , 11/13/2019 PST.
All servers operational. VPN service was restarted on proxy server 5 (44.224.142.224). Service disruptions was reported.
Region
Current
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
1: US-W
94
86
87
87
87
87
83
2: US-E
83
85
86
86
86
84
85
3: EU
66
36
35
35
35
35
35
4: US-E
58
37
39
45
47
50
51
5: US-W
63
75
75
69
68
67
64
6: US-W
98
81
81
83
81
80
81
7: EU
63
65
67
62
65
65
65
8: US-E
66
63
63
58
57
55
54
Total
591
528
533
525
526
523
518
Notes:
- Proxy server 5 had memory leak issue stating 2:43AM. Up to 75 connections for users connecting to Oregon server were affected between 2:43AM and 3:07AM PST, 11/13/2019. Service was back to normal after 3:07AM. Total duration was 24 minutes. CloudOPS, should this type of event happen again:
- Send alert to this email thread as soon as service disruption is detected, with potential hit connection number and region.
- Send update as soon as service is restored
- Dev team finished database sync up to unblock all users hit by CESMB-1124 ticket. Continue to verify if there is any remain issue. Due to this fix, system brought back about 60 active connections. CloudOPS, for planned ECR like this fix, let’s also send out notice before ECR is performed, with ECR purpose and potential risk, if any.
10PM , 11/13/2019 PST.
All servers operational. No service disruption, no new issue reported.
Region
Current
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
1: US-W
90
94
86
87
87
87
87
2: US-E
82
83
85
86
86
86
84
3: EU
68
66
36
35
35
35
35
4: US-E
60
58
37
39
45
47
50
5: US-W
67
63
75
75
69
68
67
6: US-W
95
98
81
81
83
81
80
7: EU
62
63
65
67
62
65
65
8: US-E
62
66
63
63
58
57
55
Total
586
591
528
533
525
526
523
11AM , 11/14/2019 PST.
All servers operational. No service disruption, no new issue reported.
Region
11AM
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
1: US-W
94
90
94
86
87
87
87
2: US-E
80
82
83
85
86
86
86
3: EU
70
68
66
36
35
35
35
4: US-E
62
60
58
37
39
45
47
5: US-W
66
67
63
75
75
69
68
6: US-W
97
95
98
81
81
83
81
7: EU
60
62
63
65
67
62
65
8: US-E
63
62
66
63
63
58
57
Total
592
586
591
528
533
525
526
10PM , 11/14/2019 PST.
All servers operational. No service disruption, no new issue reported.
Region
10PM
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
1: US-W
92
94
90
94
86
87
87
2: US-E
76
80
82
83
85
86
86
3: EU
68
70
68
66
36
35
35
4: US-E
63
62
60
58
37
39
45
5: US-W
64
66
67
63
75
75
69
6: US-W
97
97
95
98
81
81
83
7: EU
64
60
62
63
65
67
62
8: US-E
60
63
62
66
63
63
58
Total
584
592
586
591
528
533
525
11AM , 11/15/2019 PST.
All servers operational. No service disruption, no new issue reported.
Region
11AM
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
-3.5D
1: US-W
88
92
94
90
94
86
87
87
2: US-E
81
76
80
82
83
85
86
86
3: EU
71
68
70
68
66
36
35
35
4: US-E
63
63
62
60
58
37
39
45
5: US-W
71
64
66
67
63
75
75
69
6: US-W
95
97
97
95
98
81
81
83
7: EU
61
64
60
62
63
65
67
62
8: US-E
64
60
63
62
66
63
63
58
Total
594
584
592
586
591
528
533
525
11PM , 11/15/2019 PST.
All servers operational. No service disruption, no new issue reported.
Region
11PM
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
1: US-W
87
88
92
94
90
94
86
2: US-E
77
81
76
80
82
83
85
3: EU
69
71
68
70
68
66
36
4: US-E
62
63
63
62
60
58
37
5: US-W
65
71
64
66
67
63
75
6: US-W
96
95
97
97
95
98
81
7: EU
61
61
64
60
62
63
65
8: US-E
59
64
60
63
62
66
63
Total
576
594
584
592
586
591
528
10AM , 11/16/2019 PST.
All servers operational. No service disruption, no new issue reported.
Region
10AM
-0.5Day
-1D
-1.5D
-2D
-2.5D
-3D
1: US-W
87
87
88
92
94
90
94
2: US-E
77
77
81
76
80
82
83
3: EU
69
69
71
68
70
68
66
4: US-E
62
62
63
63
62
60
58
5: US-W
67
65
71
64
66
67
63
6: US-W
97
96
95
97
97
95
98
7: EU
60
61
61
64
60
62
63
8: US-E
61
59
64
60
63
62
66
Total
580
576
594
584
592
586
591
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
