//
Latest VPN Business Community News:
Reply
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

10PM , 11/04/2019 PST.

 

No server rebooted.

New connections are OK.

There were two cases reported in CESMB-1124 that routers were removed from VPN group, result in group mismatch between VPN server and Insight server, investigating as P1 issue.

 

Region

Current

-0.5D

-1Day

-1.5D

-2D

1: US-W

79

80

79

78

76

2: US-E

87

86

84

87

87

3: EU

22

28

31

39

42

4: US-E

79

85

83

80

80

5: US-W

63

62

60

62

64

6: US-W

73

72

69

65

63

7: EU

65

70

62

57

55

8: US-E

38

35

33

34

31

Total

506

518

501

502

498

Please click KUDOS or REPLY If you found this helpful.
Message 26 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

10AM , 11/05/2019 PST.

 

All servers operational. No server rebooted. No customer issue reported.

 

Region

Current

-0.5D

-1D

-1.5Day

-2D

1: US-W

76

79

80

79

78

2: US-E

87

87

86

84

87

3: EU

25

22

28

31

39

4: US-E

82

79

85

83

80

5: US-W

63

63

62

60

62

6: US-W

76

73

72

69

65

7: EU

67

65

70

62

57

8: US-E

39

38

35

33

34

Total

515

506

518

501

502

Please click KUDOS or REPLY If you found this helpful.
Message 27 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

10PM , 11/05/2019 PST.

All servers operational.

As reported at 4PM, VPN server 4 in Virginia had memory leak issue between 1:30PM and 3PM.

User may experience intermittent VPN connection drop if they were using VPN during that period, it was fully operational after VPN service was restarted(Actual restart takes a few seconds to complete)

No customer issue reported.

 

Region

Current

-6H

-0.5D

-1D

-1.5D

-2D

1: US-W

78

77

76

79

80

79

2: US-E

83

85

87

87

86

84

3: EU

20

23

25

22

28

31

4: US-E

79

82

82

79

85

83

5: US-W

58

57

63

63

62

60

6: US-W

76

74

76

73

72

69

7: EU

64

62

67

65

70

62

8: US-E

36

38

39

38

35

33

Total

494

498

515

506

518

501

 

Update on new server software:

New VPN conductor server and proxy server software have been deployed on QA platform, It has been operational since 11/04. QA is performing functional and load/stress tests.

Please click KUDOS or REPLY If you found this helpful.
Message 28 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

10AM , 11/06/2019 PST.

All servers operational. No customer issue reported.

 

Region

Current

-0.5Day

-2D

-1.5D

-2D

1: US-W

81

78

76

79

80

2: US-E

86

83

87

87

86

3: EU

24

20

25

22

28

4: US-E

81

79

82

79

85

5: US-W

58

58

63

63

62

6: US-W

73

76

76

73

72

7: EU

67

64

67

65

70

8: US-E

37

36

39

38

35

Total

507

494

515

506

518

Please click KUDOS or REPLY If you found this helpful.
Message 29 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

10PM , 11/06/2019 PST.

All servers operational. No customer issue reported.

 

Region

Current

-0.5Day

-1D

-1,5DD

-2D

1: US-W

80

81

78

76

79

2: US-E

86

86

83

87

87

3: EU

33

24

20

25

22

4: US-E

80

81

79

82

79

5: US-W

55

58

58

63

63

6: US-W

71

73

76

76

73

7: EU

69

67

64

67

65

8: US-E

39

37

36

39

38

Total

513

507

494

515

506

 

Current Status:

We are testing candidate builds for a permanent fix and hope to finish QA by early next week.   In the interim we continue to closely monitor server loading to prevent outages.

Please click KUDOS or REPLY If you found this helpful.
Message 30 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

10AM , 11/07/2019 PST.

All servers operational. No customer issue reported.

 

Region

Current

-0.5Day

-1D

-1.5D

-2D

1: US-W

82

80

81

78

76

2: US-E

90

86

86

83

87

3: EU

36

33

24

20

25

4: US-E

81

80

81

79

82

5: US-W

54

55

58

58

63

6: US-W

78

71

73

76

76

7: EU

66

69

67

64

67

8: US-E

40

39

37

36

39

Total

527

513

507

494

515

 

Please click KUDOS or REPLY If you found this helpful.
Message 31 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

Current Status:

We are testing candidate builds for a permanent fix and hope to finish QA by early next week.   In the interim we continue to closely monitor server loading to prevent outages.

Please click KUDOS or REPLY If you found this helpful.
Message 32 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates 11/3

10PM , 11/07/2019 PST.

All servers operational. No customer issue reported.

 

Region

Current

-0.5Day

-1Day

-1.5D

-2D

-2.5D

1: US-W

80

82

80

81

78

76

2: US-E

87

90

86

86

83

87

3: EU

35

36

33

24

20

25

4: US-E

82

81

80

81

79

82

5: US-W

51

54

55

58

58

63

6: US-W

80

78

71

73

76

76

7: EU

65

66

69

67

64

67

8: US-E

40

40

39

37

36

39

Total

520

527

513

507

494

515

 

Dev Update: A check is being performed on the integrity of the VPN database. Actions upon analysis it needed.

Please click KUDOS or REPLY If you found this helpful.
Message 33 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

10AM , 11/08/2019 PST.

All servers operational. No customer issue reported.

 

Region

Current

-0.5Day

-1Day

-1.5Day

-2D

-2.5D

-3D

1: US-W

82

80

82

80

81

78

76

2: US-E

88

87

90

86

86

83

87

3: EU

36

35

36

33

24

20

25

4: US-E

70

82

81

80

81

79

82

5: US-W

63

51

54

55

58

58

63

6: US-W

78

80

78

71

73

76

76

7: EU

67

65

66

69

67

64

67

8: US-E

46

40

40

39

37

36

39

Total

530

520

527

513

507

494

515

 

On server status monitor:

Requested CloudOPS to lower memory usage alarm level from 60% to 40%, in effort to reduce expected down time to be less than 30 minutes for any server might running to issue.

 

On server update:

We received new server code with fixes to bugs exposed in high availability tests, high availability and stress tests will resume on Monday.

 

On VPN support cases:

Bug reported that specific user can’t delete VPN group, or router does not belong to any group.

This is related to 10/24 deployment. We are trying to identify the root cause.

Please click KUDOS or REPLY If you found this helpful.
Message 34 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Please click KUDOS or REPLY If you found this helpful.
Message 35 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Please click KUDOS or REPLY If you found this helpful.
Message 36 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Please click KUDOS or REPLY If you found this helpful.
Message 37 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Please click KUDOS or REPLY If you found this helpful.
Message 38 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Please click KUDOS or REPLY If you found this helpful.
Message 39 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

10AM , 11/11/2019 PST.

All servers operational. No alarm received. No customer issue reported.

 

Region

Current

-0.5Day

-1D

-1.5D

-2D

-2.5D

-3D

1: US-W

87

87

83

79

83

82

82

2: US-E

86

84

85

84

85

86

88

3: EU

35

35

35

35

34

35

36

4: US-E

47

50

51

55

56

59

70

5: US-W

68

67

64

66

67

65

63

6: US-W

81

80

81

80

80

79

78

7: EU

65

65

65

63

64

63

67

8: US-E

57

55

54

50

54

48

46

Total

526

523

518

512

523

517

530

 

Notes:

We will continue to focus on new VPN server software validation this week

Please click KUDOS or REPLY If you found this helpful.
Message 40 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Please click KUDOS or REPLY If you found this helpful.
Message 41 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

10AM , 11/12/2019 PST.

All servers operational. No alarm received.

 

Region

Current

-0.5Day

-1D

-1.5D

-2D

-2.5D

-3D

1: US-W

87

87

87

87

83

79

83

2: US-E

86

86

86

84

85

84

85

3: EU

35

35

35

35

35

35

34

4: US-E

39

45

47

50

51

55

56

5: US-W

75

69

68

67

64

66

67

6: US-W

81

83

81

80

81

80

80

7: EU

67

62

65

65

65

63

64

8: US-E

63

58

57

55

54

50

54

Total

533

525

526

523

518

512

523

 

Notes:

  1. VPN group info mismatch issue: We have validated automated procedure to sync Insight database with VPN server database. The sync is scheduled on Indian 11/13. Actual time depends on final approvals.
  2. On Server Fix: Validation is still ongoing with QA servers, there were some abnormities that needed clarifications from development.
  3. One customer reported long site to site VPN traffic latency. We are investigating.
Please click KUDOS or REPLY If you found this helpful.
Message 42 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Please click KUDOS or REPLY If you found this helpful.
Message 43 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

3.30 AM , 11/13/2019 PST.

All servers operational.

 

Region

current

-5hours

-0.5Day

-1D

-1.5D

-2D

-2.5D

-3D

1: US-W

95

86

87

87

87

87

83

79

2: US-E

84

85

86

86

86

84

85

84

3: EU

65

36

35

35

35

35

35

35

4: US-E

61

37

39

45

47

50

51

55

5: US-W

59

75

75

69

68

67

64

66

6: US-W

97

81

81

83

81

80

81

80

7: EU

65

65

67

62

65

65

65

63

8: US-E

63

63

63

58

57

55

54

50

Total

589

528

533

525

526

523

518

512

 

Note: 

Service on one of the server in US-W (44.224.142.224) went down. The same was detected and made operational with in 10 mins.

Please click KUDOS or REPLY If you found this helpful.
Message 44 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

11AM , 11/13/2019 PST.

All servers operational. VPN service was restarted on proxy server 5 (44.224.142.224). Service disruptions was reported.

 

Region

Current

-0.5Day

-1D

-1.5D

-2D

-2.5D

-3D

1: US-W

94

86

87

87

87

87

83

2: US-E

83

85

86

86

86

84

85

3: EU

66

36

35

35

35

35

35

4: US-E

58

37

39

45

47

50

51

5: US-W

63

75

75

69

68

67

64

6: US-W

98

81

81

83

81

80

81

7: EU

63

65

67

62

65

65

65

8: US-E

66

63

63

58

57

55

54

Total

591

528

533

525

526

523

518

 

Notes:

  1. Proxy server 5 had memory leak issue stating 2:43AM. Up to 75 connections for users connecting to Oregon server were affected between 2:43AM and 3:07AM PST, 11/13/2019. Service was back to normal after 3:07AM. Total duration was 24 minutes. CloudOPS, should this type of event happen again:
    1. Send alert to this email thread as soon as service disruption is detected, with potential hit connection number and region.
    2. Send update as soon as service is restored
  2.  Dev team finished database sync up to unblock all users hit by CESMB-1124 ticket. Continue to verify if there is any remain issue. Due to this fix, system brought back about 60 active connections. CloudOPS, for planned ECR like this fix, let’s also send out notice before ECR is performed, with ECR purpose and potential risk, if any.
Please click KUDOS or REPLY If you found this helpful.
Message 45 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Please click KUDOS or REPLY If you found this helpful.
Message 46 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

11AM , 11/14/2019 PST.

All servers operational. No service disruption, no new issue reported.

 

Region

11AM

-0.5Day

-1D

-1.5D

-2D

-2.5D

-3D

1: US-W

94

90

94

86

87

87

87

2: US-E

80

82

83

85

86

86

86

3: EU

70

68

66

36

35

35

35

4: US-E

62

60

58

37

39

45

47

5: US-W

66

67

63

75

75

69

68

6: US-W

97

95

98

81

81

83

81

7: EU

60

62

63

65

67

62

65

8: US-E

63

62

66

63

63

58

57

Total

592

586

591

528

533

525

526

Please click KUDOS or REPLY If you found this helpful.
Message 47 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Please click KUDOS or REPLY If you found this helpful.
Message 48 of 91
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Please click KUDOS or REPLY If you found this helpful.
Message 49 of 91
Highlighted
Community Manager

Re: BR500 Connectivity Issue - Subscribe Here for Updates

Please click KUDOS or REPLY If you found this helpful.
Message 50 of 91
Need More Help?