Orbi WiFi 7 RBE973

Re: BR500 Connectivity Issue - Subscribe Here for Updates

BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

10PM , 11/04/2019 PST.

 

No server rebooted.

New connections are OK.

There were two cases reported in CESMB-1124 that routers were removed from VPN group, result in group mismatch between VPN server and Insight server, investigating as P1 issue.

 

Region

Current

-0.5D

-1Day

-1.5D

-2D

1: US-W

79

80

79

78

76

2: US-E

87

86

84

87

87

3: EU

22

28

31

39

42

4: US-E

79

85

83

80

80

5: US-W

63

62

60

62

64

6: US-W

73

72

69

65

63

7: EU

65

70

62

57

55

8: US-E

38

35

33

34

31

Total

506

518

501

502

498

Message 26 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

10AM , 11/05/2019 PST.

 

All servers operational. No server rebooted. No customer issue reported.

 

Region

Current

-0.5D

-1D

-1.5Day

-2D

1: US-W

76

79

80

79

78

2: US-E

87

87

86

84

87

3: EU

25

22

28

31

39

4: US-E

82

79

85

83

80

5: US-W

63

63

62

60

62

6: US-W

76

73

72

69

65

7: EU

67

65

70

62

57

8: US-E

39

38

35

33

34

Total

515

506

518

501

502

Message 27 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

10PM , 11/05/2019 PST.

All servers operational.

As reported at 4PM, VPN server 4 in Virginia had memory leak issue between 1:30PM and 3PM.

User may experience intermittent VPN connection drop if they were using VPN during that period, it was fully operational after VPN service was restarted(Actual restart takes a few seconds to complete)

No customer issue reported.

 

Region

Current

-6H

-0.5D

-1D

-1.5D

-2D

1: US-W

78

77

76

79

80

79

2: US-E

83

85

87

87

86

84

3: EU

20

23

25

22

28

31

4: US-E

79

82

82

79

85

83

5: US-W

58

57

63

63

62

60

6: US-W

76

74

76

73

72

69

7: EU

64

62

67

65

70

62

8: US-E

36

38

39

38

35

33

Total

494

498

515

506

518

501

 

Update on new server software:

New VPN conductor server and proxy server software have been deployed on QA platform, It has been operational since 11/04. QA is performing functional and load/stress tests.

Message 28 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

10AM , 11/06/2019 PST.

All servers operational. No customer issue reported.

 

Region

Current

-0.5Day

-2D

-1.5D

-2D

1: US-W

81

78

76

79

80

2: US-E

86

83

87

87

86

3: EU

24

20

25

22

28

4: US-E

81

79

82

79

85

5: US-W

58

58

63

63

62

6: US-W

73

76

76

73

72

7: EU

67

64

67

65

70

8: US-E

37

36

39

38

35

Total

507

494

515

506

518

Message 29 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

10PM , 11/06/2019 PST.

All servers operational. No customer issue reported.

 

Region

Current

-0.5Day

-1D

-1,5DD

-2D

1: US-W

80

81

78

76

79

2: US-E

86

86

83

87

87

3: EU

33

24

20

25

22

4: US-E

80

81

79

82

79

5: US-W

55

58

58

63

63

6: US-W

71

73

76

76

73

7: EU

69

67

64

67

65

8: US-E

39

37

36

39

38

Total

513

507

494

515

506

 

Current Status:

We are testing candidate builds for a permanent fix and hope to finish QA by early next week.   In the interim we continue to closely monitor server loading to prevent outages.

Message 30 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

10AM , 11/07/2019 PST.

All servers operational. No customer issue reported.

 

Region

Current

-0.5Day

-1D

-1.5D

-2D

1: US-W

82

80

81

78

76

2: US-E

90

86

86

83

87

3: EU

36

33

24

20

25

4: US-E

81

80

81

79

82

5: US-W

54

55

58

58

63

6: US-W

78

71

73

76

76

7: EU

66

69

67

64

67

8: US-E

40

39

37

36

39

Total

527

513

507

494

515

 

Message 31 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

Current Status:

We are testing candidate builds for a permanent fix and hope to finish QA by early next week.   In the interim we continue to closely monitor server loading to prevent outages.

Message 32 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

10PM , 11/07/2019 PST.

All servers operational. No customer issue reported.

 

Region

Current

-0.5Day

-1Day

-1.5D

-2D

-2.5D

1: US-W

80

82

80

81

78

76

2: US-E

87

90

86

86

83

87

3: EU

35

36

33

24

20

25

4: US-E

82

81

80

81

79

82

5: US-W

51

54

55

58

58

63

6: US-W

80

78

71

73

76

76

7: EU

65

66

69

67

64

67

8: US-E

40

40

39

37

36

39

Total

520

527

513

507

494

515

 

Dev Update: A check is being performed on the integrity of the VPN database. Actions upon analysis it needed.

Message 33 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

10AM , 11/08/2019 PST.

All servers operational. No customer issue reported.

 

Region

Current

-0.5Day

-1Day

-1.5Day

-2D

-2.5D

-3D

1: US-W

82

80

82

80

81

78

76

2: US-E

88

87

90

86

86

83

87

3: EU

36

35

36

33

24

20

25

4: US-E

70

82

81

80

81

79

82

5: US-W

63

51

54

55

58

58

63

6: US-W

78

80

78

71

73

76

76

7: EU

67

65

66

69

67

64

67

8: US-E

46

40

40

39

37

36

39

Total

530

520

527

513

507

494

515

 

On server status monitor:

Requested CloudOPS to lower memory usage alarm level from 60% to 40%, in effort to reduce expected down time to be less than 30 minutes for any server might running to issue.

 

On server update:

We received new server code with fixes to bugs exposed in high availability tests, high availability and stress tests will resume on Monday.

 

On VPN support cases:

Bug reported that specific user can’t delete VPN group, or router does not belong to any group.

This is related to 10/24 deployment. We are trying to identify the root cause.

Message 34 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Message 35 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Message 36 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Message 37 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Message 38 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Message 39 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

10AM , 11/11/2019 PST.

All servers operational. No alarm received. No customer issue reported.

 

Region

Current

-0.5Day

-1D

-1.5D

-2D

-2.5D

-3D

1: US-W

87

87

83

79

83

82

82

2: US-E

86

84

85

84

85

86

88

3: EU

35

35

35

35

34

35

36

4: US-E

47

50

51

55

56

59

70

5: US-W

68

67

64

66

67

65

63

6: US-W

81

80

81

80

80

79

78

7: EU

65

65

65

63

64

63

67

8: US-E

57

55

54

50

54

48

46

Total

526

523

518

512

523

517

530

 

Notes:

We will continue to focus on new VPN server software validation this week

Message 40 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Message 41 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

10AM , 11/12/2019 PST.

All servers operational. No alarm received.

 

Region

Current

-0.5Day

-1D

-1.5D

-2D

-2.5D

-3D

1: US-W

87

87

87

87

83

79

83

2: US-E

86

86

86

84

85

84

85

3: EU

35

35

35

35

35

35

34

4: US-E

39

45

47

50

51

55

56

5: US-W

75

69

68

67

64

66

67

6: US-W

81

83

81

80

81

80

80

7: EU

67

62

65

65

65

63

64

8: US-E

63

58

57

55

54

50

54

Total

533

525

526

523

518

512

523

 

Notes:

  1. VPN group info mismatch issue: We have validated automated procedure to sync Insight database with VPN server database. The sync is scheduled on Indian 11/13. Actual time depends on final approvals.
  2. On Server Fix: Validation is still ongoing with QA servers, there were some abnormities that needed clarifications from development.
  3. One customer reported long site to site VPN traffic latency. We are investigating.
Message 42 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Message 43 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

3.30 AM , 11/13/2019 PST.

All servers operational.

 

Region

current

-5hours

-0.5Day

-1D

-1.5D

-2D

-2.5D

-3D

1: US-W

95

86

87

87

87

87

83

79

2: US-E

84

85

86

86

86

84

85

84

3: EU

65

36

35

35

35

35

35

35

4: US-E

61

37

39

45

47

50

51

55

5: US-W

59

75

75

69

68

67

64

66

6: US-W

97

81

81

83

81

80

81

80

7: EU

65

65

67

62

65

65

65

63

8: US-E

63

63

63

58

57

55

54

50

Total

589

528

533

525

526

523

518

512

 

Note: 

Service on one of the server in US-W (44.224.142.224) went down. The same was detected and made operational with in 10 mins.

Message 44 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

11AM , 11/13/2019 PST.

All servers operational. VPN service was restarted on proxy server 5 (44.224.142.224). Service disruptions was reported.

 

Region

Current

-0.5Day

-1D

-1.5D

-2D

-2.5D

-3D

1: US-W

94

86

87

87

87

87

83

2: US-E

83

85

86

86

86

84

85

3: EU

66

36

35

35

35

35

35

4: US-E

58

37

39

45

47

50

51

5: US-W

63

75

75

69

68

67

64

6: US-W

98

81

81

83

81

80

81

7: EU

63

65

67

62

65

65

65

8: US-E

66

63

63

58

57

55

54

Total

591

528

533

525

526

523

518

 

Notes:

  1. Proxy server 5 had memory leak issue stating 2:43AM. Up to 75 connections for users connecting to Oregon server were affected between 2:43AM and 3:07AM PST, 11/13/2019. Service was back to normal after 3:07AM. Total duration was 24 minutes. CloudOPS, should this type of event happen again:
    1. Send alert to this email thread as soon as service disruption is detected, with potential hit connection number and region.
    2. Send update as soon as service is restored
  2.  Dev team finished database sync up to unblock all users hit by CESMB-1124 ticket. Continue to verify if there is any remain issue. Due to this fix, system brought back about 60 active connections. CloudOPS, for planned ECR like this fix, let’s also send out notice before ECR is performed, with ECR purpose and potential risk, if any.
Message 45 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Message 46 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

11AM , 11/14/2019 PST.

All servers operational. No service disruption, no new issue reported.

 

Region

11AM

-0.5Day

-1D

-1.5D

-2D

-2.5D

-3D

1: US-W

94

90

94

86

87

87

87

2: US-E

80

82

83

85

86

86

86

3: EU

70

68

66

36

35

35

35

4: US-E

62

60

58

37

39

45

47

5: US-W

66

67

63

75

75

69

68

6: US-W

97

95

98

81

81

83

81

7: EU

60

62

63

65

67

62

65

8: US-E

63

62

66

63

63

58

57

Total

592

586

591

528

533

525

526

Message 47 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Message 48 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Message 49 of 295
BretD
Admin

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Message 50 of 295
Discussion stats
  • 294 replies
  • 21511 views
  • 7 kudos
  • 1 in conversation
Announcements

Need More Help?