Orbi WiFi 7 RBE973

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

BretD
Admin

BR500 Connectivity Issue - Subscribe Here for Updates

Dear NETGEAR BR500 customers,

Over the past two weeks we have experienced extreme instability with our VPN cloud ecosystem. We believe it is due to the scale of customer base combined with our design tradeoffs on passing cloud status. We are working around the clock to identify a fix, test it and roll it out to production with as little ongoing disruption as possible. Unfortunately the current situation requires us to regularly reboot our cloud servers which is causing the disruption you are experiencing.

We are committed to having a robust and reliable VPN service and realize that we have not achieved this. For this we are truly sorry and we will continue to work on this until we have achieved this objective. We will be posting regular (several a day) updates on our corrective action status at this location in our community until we have completely resolved this issue. If you would prefer to get these status updates sent to you directly, login to the NETGEAR Community and subscribe to this thread. You can respond via the BR500 update thread. (Subscribe instructions can be found below).

Sincerely,

NETGEAR BR500 team

Insight Instant VPN RouterInsight Instant VPN Router

Subscribe Instructions:

This thread is locked except for Administrators. As updates on this the BR500 connectivity issues are avaliable this thread will be updated. For each reply thread subscribers will receive a community email update. (Depending on your community settings). Thank you for your understanding.How-To-Subscribe-Community.png

 

Message 1 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

Update 10/30/2019 10:00 PM PT: Insight VPN Services are operational. 

Message 2 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

Update 10/31/2019 2:00 AM PT: Insight VPN Services are operational.

  • There was a temporary disruption around 11:30pm~00:30am PDT time on one server in East Coast, about 50 routers might have experienced an intermittent VPN disconnection/reconnection.
Message 3 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

Update 10/31/2019 6:00 AM PT: Insight VPN Services are operational. 

Message 4 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

Update 10/31/2019 10:00 AM PT: Insight VPN Services are operational.

  • 3 additional servers are scheduled to go online today.
Message 5 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

Update 10/31/2019 5PM PT: Current status - Since this issue surfaced last Thursday, ours VPN servers would start to drop connections when the number of connections got to a critical level.


To mitigate this issue, we added a new server in our West Coast datacenter on Oct 28th, and subsequently added 3 more servers at 12pm on Oct 31st, one on the West Coast, one on the East Coast of the US and one in Ireland. We now have 8 servers serving different regions.

 

We haven’t had any VPN disconnections since these 3 new servers were added. The system has been stable since our last post.

This is the temporary solution to keep the service stable. On the development side, the team continues to work on fixing the issue related to the unnecessary traffic flooding causing the servers to disconnect VPN requests from the BR500s in the field. We are targeting to release a test build to QA test and validation by early next week.

We apologize for the ongoing uncertainty.

Message 6 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

Update: 10/31/2019 6PM PST Update. All servers are working. All servers have been stable since last update 4 hours ago.

Total connection is 508. Detail distribution is as below:

1: US-W               72

2: US-E                 93

3: EU                    93

4: US-E                 77

5: US-W               89

6: US-W               32

7: EU                    36

8: US-E                 16

Message 7 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

Update: 10/31/2019 PST 10PM All servers heathy and responsive. Connect counts below:

Region

Current

4Hrs Ago

1: US-W

75

72

2: US-E

91

93

3: EU

87

93

4: US-E

77

77

5: US-W

85

89

6: US-W

35

32

7: EU

37

36

8: US-E

19

16

Total

506

508

Message 8 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

2:00 am , 11/01/2019 PT Update.

 

All servers healthy and responsive. Connect counts below:

 

Region

Current

4Hrs Ago

1: US-W

74

75

2: US-E

92

91

3: EU

82

87

4: US-E

82

77

5: US-W

85

85

6: US-W

39

35

7: EU

44

37

8: US-E

21

19

Total

519

508

Message 9 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

6:00 am , 11/01/2019 PT Update.

 

All servers healthy and responsive. Connect counts below:

 

Region

Current

4Hrs Ago

1: US-W

75

74

2: US-E

94

92

3: EU

74

82

4: US-E

77

82

5: US-W

84

85

6: US-W

43

39

7: EU

48

44

8: US-E

23

21

Total

518

519

Message 10 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

10:00 am , 11/01/2019 PDT Update.

 

All servers healthy and responsive. Connect counts below:

 

 

Region

Current

4Hrs Ago

1: US-W

75

75

2: US-E

92

94

3: EU

71

74

4: US-E

76

77

5: US-W

86

84

6: US-W

51

43

7: EU

50

48

8: US-E

23

23

Total

524

518

Message 11 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

2PM , 11/01/2019 PDT Update.

All servers healthy and responsive. Connect counts below:

 

Region

Current

4hrs Ago

1: US-W

79

75

2: US-E

93

92

3: EU

65

71

4: US-E

74

76

5: US-W

80

86

6: US-W

51

51

7: EU

49

50

8: US-E

26

23

Total

517

524

Message 12 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

6PM , 11/01/2019 PDT Update.

 

All servers healthy and responsive. Connect counts below:

 

 

Region

Current

4hrs ago

1: US-W

82

79

2: US-E

89

93

3: EU

59

65

4: US-E

76

74

5: US-W

75

80

6: US-W

53

51

7: EU

49

49

8: US-E

24

26

Total

507

517

Message 13 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

10PM , 11/01/2019 PDT Update.

 

All servers healthy and responsive. Connect counts below:

 

 

Region

Current

4H Ago

1: US-W

81

82

2: US-E

89

89

3: EU

57

59

4: US-E

75

76

5: US-W

74

75

6: US-W

55

53

7: EU

50

49

8: US-E

26

24

Total

507

507

Message 14 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

2:17 am , 11/02/2019 PDT Update.

 

All servers healthy and responsive. Connect counts below:

 

 

Region

Current

4H Ago

1: US-W

81

81

2: US-E

91

89

3: EU

57

57

4: US-E

77

75

5: US-W

71

74

6: US-W

55

55

7: EU

51

50

8: US-E

27

26

Total

510

507

 

Note :

  • Team is working on validation of v.1.0.6.1 conductor and proxy package.
Message 15 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

6:17 am , 11/02/2019 PDT Update.

 

All servers healthy and responsive. Connect counts below:

 

 

Region

Current

4H Ago

1: US-W

80

81

2: US-E

91

91

3: EU

51

57

4: US-E

78

77

5: US-W

70

71

6: US-W

55

55

7: EU

56

51

8: US-E

29

27

Total

510

510

Message 16 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

10 am , 11/02/2019 PDT Update.

 

All servers healthy and responsive. Connect counts below (Since total is below 500, we will pay attention to it)

 

Region

Current

4H Ago

1: US-W

78

80

2: US-E

88

91

3: EU

49

51

4: US-E

77

78

5: US-W

65

70

6: US-W

60

55

7: EU

49

56

8: US-E

27

29

Total

493

510

Message 17 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

2PM, 11/02/2019 PDT.

 All servers healthy and responsive.

 

Region

Current

4H Ago

8H Ago

1: US-W

78

78

80

2: US-E

88

88

91

3: EU

46

49

51

4: US-E

79

77

78

5: US-W

65

65

70

6: US-W

63

60

55

7: EU

54

49

56

8: US-E

32

27

29

Total

505

493

510

Message 18 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

6PM, 11/02/2019 PDT.

All servers healthy and responsive.

 

Region

Current

-4H

-8H

-12H

1: US-W

79

78

78

80

2: US-E

88

88

88

91

3: EU

43

46

49

51

4: US-E

80

79

77

78

5: US-W

64

65

65

70

6: US-W

64

63

60

55

7: EU

56

54

49

56

8: US-E

31

32

27

29

Total

505

505

493

510

Message 19 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 10/30

10PM, 11/02/2019 PDT.

 

All servers healthy and responsive.

 

Region

Current

-4H

-8H

-12H

-16H

1: US-W

76

79

78

78

80

2: US-E

87

88

88

88

91

3: EU

42

43

46

49

51

4: US-E

80

80

79

77

78

5: US-W

64

64

65

65

70

6: US-W

63

64

63

60

55

7: EU

55

56

54

49

56

8: US-E

31

31

32

27

29

Total

498

505

505

493

510

Message 20 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

Message 21 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

Message 22 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

10 am , 11/03/2019 PST. (New half day update)

 

No server rebooted and no VPN connection disrupted since last update.

 

Region

Current

-12H

-24H

1: US-W

78

76

78

2: US-E

87

87

88

3: EU

39

42

49

4: US-E

80

80

77

5: US-W

62

64

65

6: US-W

65

63

60

7: EU

57

55

49

8: US-E

34

31

27

Total

502

498

493

 

Message 23 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

10 pm , 11/03/2019 PST.

 

No server rebooted and no customer VPN connection disrupted since last update.

 

Region

Current

-12H

-1D

-1.5D

1: US-W

79

78

76

78

2: US-E

84

87

87

88

3: EU

31

39

42

49

4: US-E

83

80

80

77

5: US-W

60

62

64

65

6: US-W

69

65

63

60

7: EU

62

57

55

49

8: US-E

33

34

31

27

Total

501

502

498

493

Message 24 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

 10AM , 11/04/2019 PST.

 

No server rebooted and no customer VPN connection disrupted since last update.

 

Region

Current

-0.5Day

-1D

-1.5D

-2D

1: US-W

80

79

78

76

78

2: US-E

86

84

87

87

88

3: EU

28

31

39

42

49

4: US-E

85

83

80

80

77

5: US-W

62

60

62

64

65

6: US-W

72

69

65

63

60

7: EU

70

62

57

55

49

8: US-E

35

33

34

31

27

Total

518

501

502

498

493

Message 25 of 295
Discussion stats
  • 294 replies
  • 21511 views
  • 7 kudos
  • 1 in conversation
Announcements

Need More Help?