Orbi WiFi 7 RBE973
Reply

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

tuna_ertemalp
Luminary

RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, etc.

I was doing this on my RBR50 for the last 2.5 years: Every 3am, email me the log. That also resulted in the log getting reset, so every morning I would have a copy of the previous day's log in my email. And I saved them in case I had to go back & look at something. And these logs showed an incredible number of entries of attacks coming in and were being deflected by the router.

 

The same option exists on RBRE960 (using the latest FW V6.0.3.85_3.1.15), so I have the same settings for Administration/Logs (including the all the checkboxes at the bottom of that page) and Security/E-Mail. Yet, what I get at 3am is just a few lines of the log, like for 3am-6am yesterday morning, not 3am-3am from yesterday to today. Even when I go into the Administration/Logs in the middle of the day right now and hit SEND LOG, I receive these 9 lines:

 

[Internet connected] IP address: <redacted>, Friday, Mar 25,2022 05:18:48
[Time synchronized with NTP server] Friday, Mar 25,2022 04:48:48
[Internet connected] IP address: <redacted>, Friday, Mar 25,2022 04:48:48
[Time synchronized with NTP server] Friday, Mar 25,2022 03:48:49
[Internet connected] IP address: <redacted>, Friday, Mar 25,2022 03:48:48
[Time synchronized with NTP server] Friday, Mar 25,2022 03:18:49
[Internet connected] IP address: <redacted>, Friday, Mar 25,2022 03:18:49
[email sent to: <redacted>] Friday, Mar 25,2022 03:00:07
[DHCP IP: (<redacted>)] to MAC address <redacted>, Friday, Mar 25,2022 07:05:47

 

For starters, notice how the line I marked in blue is listed way out of time order.

 

Plus, on my screen, I am literally staring at dozens and dozens of log lines since yesterday 3am; here they are, redacted & trimmed:

 

[email sent to: <redacted>] Friday, Mar 25,2022 11:12:38
[DHCP IP: (<redacted>)] to MAC address <redacted>, Friday, Mar 25,2022 10:57:14
[DHCP IP: (<redacted>)] to MAC address <redacted>, Friday, Mar 25,2022 10:56:46
[DHCP IP: (<redacted>)] to MAC address <redacted>, Friday, Mar 25,2022 10:55:28
[Time synchronized with NTP server] Friday, Mar 25,2022 10:48:48
[Internet connected] IP address: <redacted>, Friday, Mar 25,2022 10:48:48
. . . . .
[DHCP IP: (<redacted>)] to MAC address <redacted>, Friday, Mar 25,2022 07:05:54
[DHCP IP: (<redacted>)] to MAC address <redacted>, Friday, Mar 25,2022 07:05:47
[DHCP IP: (<redacted>)] to MAC address <redacted>, Friday, Mar 25,2022 07:05:46
. . . . .

[DHCP IP: (<redacted>)] to MAC address <redacted>, Friday, Mar 25,2022 06:08:14
[Time synchronized with NTP server] Friday, Mar 25,2022 05:18:48
[Internet connected] IP address: <redacted>, Friday, Mar 25,2022 05:18:48
[Time synchronized with NTP server] Friday, Mar 25,2022 04:48:48
[Internet connected] IP address: <redacted>, Friday, Mar 25,2022 04:48:48
[Time synchronized with NTP server] Friday, Mar 25,2022 03:48:49
[Internet connected] IP address: <redacted>, Friday, Mar 25,2022 03:48:48
[Time synchronized with NTP server] Friday, Mar 25,2022 03:18:49
[Internet connected] IP address: <redacted>, Friday, Mar 25,2022 03:18:49
[email sent to: <redacted>] Friday, Mar 25,2022 03:00:07

 

I highlighted in red/blue the 9 lines that SEND LOG decided to send me. Notice how few they are AND how the one blue random line from the middle of the actual log is added to the end of the emailed log, completely out of time order.

 

This definitely looks like a bad bug with handling string buffers during log emailing. I hope nobody can use it to attack the router.

 

Additionally, as I mentioned, RBR50 logs had a huge number of attacks listed in the log. I am not seeing any such log entry on RBRE960 logs despite having checked the box to include "Known DoS attacks and Port Scans" along with all the other checkboxes. I wonder if that (or all) checkbox(es) isn't (aren't) respected properly. Or, if my 30-day free trial of ORBI Armor is filtering them out without letting them get into the logs... Either of these is not good. Router logs should have all things that happened that the user wants/needs to see.

 

In general, logging on a router should work, especially at this price point. This is something that needs to be fixed in a future FW drop. Who needs to hear this directly?

 

Thanks

Tuna

 

Message 1 of 23
tuna_ertemalp
Luminary

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

When I edit this post, everything looks beautiful in their colors & fonts, but not when posted, at least not for me. I am sorry. Don't know how to fix that. But the content stands, even if it is hard to read...

 

Tuna

 

Message 2 of 23
CrimpOn
Guru

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

I, also, have two old RBR50's sending me logs.  They send "when the log is full", rather than at a specific time.  While I agree that the product should do what it says it will do, I wonder as a diagnostic effort what would happen if your 960 is set to send logs when full. (I have also found it somewhat humorous that email settings fall under Security on the older Orbi, while logs fall under Administration.  Same on the 960?)

 

Also, while there is a log setting for "Known DoS attacks and port scans" in the log settings, there is also a check box in the WAN Setup page to "Disable Port Scan and DoS Protection".  Is there a similar setting on the 960?  (Ah, yes. On page 62 of the user manual.)

 

 I could not help but notice your 960 appears to be connecting to the Internet a lot. All those "internet Conected" followed by "Time Synchronized" entries.  Since they all appear in the log, it is clear that the 960 did not reboot (which would clear the log).

 

Message 3 of 23
TC_in_Montana
Virtuoso

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

@tuna_ertemalp 

 

Scheduled and on-demand E-Mailing of router logs has been an issue on Netgear products since some of the first AX capable devices.

 

I have the same issue on my 960 and have had the same issue in regards to this since day 1.   Some days I get 1 line, some days I get 20 lines, and they are always from the earliest entry forward, except that the latest entry it decides to actual include in the mail on that run is listed last.

 

Some days the router logs clear after mailing, some days they do not.  It's all a crapshoot.

 

I hate to say this, but if nothing else, I am brutally honest.   Do not expect consistent and complete router logs through the automated mailing process - at least for now.   Hopefully it is something being worked on, and will be corrected in a future firmware update.

 

 

Message 4 of 23
tuna_ertemalp
Luminary

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

@CrimpOn 

 


@CrimpOn wrote:

I, also, have two old RBR50's sending me logs.  They send "when the log is full", rather than at a specific time.  While I agree that the product should do what it says it will do, I wonder as a diagnostic effort what would happen if your 960 is set to send logs when full. (I have also found it somewhat humorous that email settings fall under Security on the older Orbi, while logs fall under Administration.  Same on the 960?)

"When Full" is of limited use for me. I liked waking up and looking at the logs to see how I was being attacked... LOL

 


Also, while there is a log setting for "Known DoS attacks and port scans" in the log settings, there is also a check box in the WAN Setup page to "Disable Port Scan and DoS Protection".  Is there a similar setting on the 960?  (Ah, yes. On page 62 of the user manual.)

I checked. The log setting to report is enabled, and the WAN setting to disable is disabled. So, it should work.

 


@CrimpOn wrote:

I, also, have two old RBR50's sending me logs.  They send "when the log is full", rather than at a specific time.  While I agree that the product should do what it says it will do, I wonder as a diagnostic effort what would happen if your 960 is set to send logs when full. (I have also found it somewhat humorous that email settings fall under Security on the older Orbi, while logs fall under Administration.  Same on the 960?)

 

Also, while there is a log setting for "Known DoS attacks and port scans" in the log settings, there is also a check box in the WAN Setup page to "Disable Port Scan and DoS Protection".  Is there a similar setting on the 960?  (Ah, yes. On page 62 of the user manual.)

 

 I could not help but notice your 960 appears to be connecting to the Internet a lot. All those "internet Conected" followed by "Time Synchronized" entries.  Since they all appear in the log, it is clear that the 960 did not reboot (which would clear the log).

 


Yes, I noticed that, too. RBR50 used to sync time with the NTP server once a day or once per reboot or something like that, and that didn't trigger an "Internet Connected" entry in the log. It seems RBRE960 feels the need to sync the time wayyyyyyy more frequently and a Internet Connected line is written into the log just before that happens. They certainly are not reboots.

 

While there, let me say that I hate that the log clears at reboot. Yikes! The log leading up to a crash resulting in a reboot is valuable! Like, that is a no brainer. The fact that there isn't the slightest amount of non-volatile memory in this expensive hardware to store the log in a way that is persisted across crashes & reboots, and reported properly is insane!

 

Tuna

 

 

 

Message 5 of 23
CrimpOn
Guru

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

I view "Internet Connected" as the key log entry.  There will always be a Time Sync immediately after the internet connection.  "Hey, I'm on the internet.  Wonder what time it is?"  I'd put money on NTP not having anything to do with the Connection happening.  There is some other cause.  Since I keep all these logs, I just searched.  My Orbi put "Internet connected" into the log file on Monday, Dec 13.

 

My Orbi has been 'up' for 119 days (since Nov 25, 2021) and during that time it has 'connected' to the internet 3-4 times. The last time being Dec 13, 2021.  In every case after Nov 25, there was a 'disconnected' message immediately before the 'connected'.

 

A word about "Full" vs. at a certain time.  It is pretty clear that there is a maximum log file size. (Hence the concept "full".)  If a log is send once per day, it will be either (a) not completely full yet, or (b) have gone past full and wrapped around, and thus an unknown number of log entries have been written over.  Most days, it takes more than 24 hours to fill my log files, so once per day would be convenient. I find several emails, however, that came in less than 24 hours.  Since the number of DHCP lease renewals is pretty much constant, the major difference is the number of DoS entries.  When some A**H*** out there decides to go fishing, the logs can fill really quickly.

 

Anyway, the question is more about diagnostics rather than the end goal.  If 'when full' actually works, that is a ton better than an email with 9 lines of drivel.

Message 6 of 23
tuna_ertemalp
Luminary

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

@CrimpOn 

 

Good argument about Full vs Daily. I'll switch to Full and see what happens.

Message 7 of 23
Mikey94025
Hero

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e


@tuna_ertemalp wrote:

In general, logging on a router should work, especially at this price point. This is something that needs to be fixed in a future FW drop. Who needs to hear this directly?

 


Consider filing a Netgear Support request, not posting the community forum, if you feel that Netgear engineering needs to know about or fix something for you.  If you purchased your AXE router within 90 days ago then you have technical support available at https://my.netgear.com/home.aspx

 

Message 8 of 23
tuna_ertemalp
Luminary

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

@CrimpOn 

 

No dice. Since I switched to "email when full" few days back on Friday, just now, this Wednesday 5am, I received my first log email, only containing a whopping 11 lines, from Sunday morning... 😞

 

[DHCP IP: (192.168.1.7)] to MAC address <redacted>, Sunday, Mar 27,2022 06:34:23
[Time synchronized with NTP server] Sunday, Mar 27,2022 06:18:51
[Internet connected] IP address: <redacted>, Sunday, Mar 27,2022 06:18:51
[DHCP IP: (192.168.1.73)] to MAC address <redacted>C, Sunday, Mar 27,2022 06:17:58
[Time synchronized with NTP server] Sunday, Mar 27,2022 05:18:50
[Internet connected] IP address: <redacted>, Sunday, Mar 27,2022 05:18:50
[Time synchronized with NTP server] Sunday, Mar 27,2022 04:48:49
[Internet connected] IP address: <redacted>, Sunday, Mar 27,2022 04:48:49
[DHCP IP: (192.168.1.64)] to MAC address <redacted>, Sunday, Mar 27,2022 04:46:58
[Time synchronized with NTP server] Sunday, Mar 27,2022 03:48:49
[Internet connected] IP address: <redacted>, Sunday, Mar 27,2022 03:48:48

 

The current live log on RBRE960 ranges between:

 

[Admin login] from source 192.168.1.59, Wednesday, Mar 30,2022 06:50:38

. . . .

[DHCP IP: (192.168.1.12)] to MAC address <redacted, Monday, Mar 28,2022 07:06:17

 

So, it feels like the router tried to email only entries from Friday through Monday morning, even though it did that on Wednesday morning, but failed to send all of it, yet seemingly still truncated the Fri-to-Mon morning entries from the log, instead of emailing everything properly and resetting the log to empty.

 

Tuna

 

Message 9 of 23
TC_in_Montana
Virtuoso

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

Orbi 900 series has not properly E-Mailed router logs since day 1.

Engineers are aware of the issue.   Hopefully they are working on it.

 

 

Message 10 of 23
CrimpOn
Guru

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

I agree with @Mikey94025  If you have any time left on the "complimentary" 90 days of support, open a ticket with Netgear support.  Would be fun if you could pry some sort of acknowledgement out of them that the problem is actually being worked on. Maybe even a case number or something.

Message 11 of 23
tuna_ertemalp
Luminary

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

@Mikey94025 @CrimpOn 

 

I have 78 days left. I will. It is on my to-do list. 

 

Tuna

 

Message 12 of 23
tuna_ertemalp
Luminary

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

Here is the response. The customer facing support person was really quick in turning this around. One could wish that the engineering were this quick. 😁

"Hi Tuna,

I have looked into your case and was able to speak to our Level 2 support about it.

According to our Level 2 support, there are currently no updates regarding this issue. And that our engineering team is working on it, hopefully, they would be able to address this issue ASAP.

I was advised that if you really want to monitor the progress of this case, we advise that you call for help regarding it and open a case regarding your concern.

Again, thank you for choosing NETGEAR.

Respectfully,

Ross
NETGEAR Support Expert"
Message 13 of 23
TC_in_Montana
Virtuoso

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

@tuna_ertemalp 

 

At least you know that they know!!   

 

My guess is that their support and engineers have been working on issues on the 700/800 series that are more pressing.   While this is an annoyance, it doesn't cause crashes or disconnects.

 

I'm anxiously waiting on a fix for this as well.

Message 14 of 23
CrimpOn
Guru

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e


@tuna_ertemalp wrote:
I have looked into your case and was able to speak to our Level 2 support about it.

we advise that you call for help regarding it and open a case regarding your concern.

So, you have a "your case", but need to open a case.  Was the problem perhaps that you did not call, and thus whatever communication you made does not count? (chat? email?)

 

very confusing.

Message 15 of 23
tuna_ertemalp
Luminary

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

Exactly my feelings... LOL

 

My response:

 

"Thank you for looking into this and talking to L2. However, I am confused as to what I am supposed to do. You said, "we advise that you call for help regarding it and open a case regarding your concern." I have already opened this case 45883819 that we are communicating over currently. Are you asking me actually to use a physical phone to place an actual phone call to open a new duplicate case? What is the difference between this case and a phone-initiated case? Besides, the support person answering the phone won't have the entire background we have here, so I will need to convince him/her to open a new case for a case that already exists. So, I am confused as to what you need me to do... Why can't we continue to use this case# or you go ahead and open a more proper case based on this one and let me know?

Thanks
Tuna"

Message 16 of 23
CrimpOn
Guru

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

Well done.
Message 17 of 23
tuna_ertemalp
Luminary

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

Continuing...

 

"Hi Tuna,


Thank you for taking the time to contact us at NETGEAR Email support.

On your initial email, you were asking for an acknowledgment if our engineering team is actually working on the issue of incomplete logs being generated by the router. I asked our Level 2 support regarding your concern and was advised that our engineering team is actually aware of this issue and are currently working on it.

He also informed me that if you wish to monitor the progress of this case, then we recommend that we troubleshoot your concern regarding the incomplete logs that you are receiving from your router. And then escalate your case so that they can provide you with real-time updates instead of going through the forum.

If you wish to monitor the status of this case, you can either call us back and ask for troubleshooting steps on how to resolve the incomplete logs that you are receiving from the router, or I can change the status of this case from an inquiry to a technical support case and provide you with some troubleshooting steps like resetting the router to factory settings and reconfiguring it. Just some basic troubleshooting steps.

By the way, your case number is 45883819.

Looking forward to your reply.

Again, thank you for choosing NETGEAR.

Respectfully,

Ross
NETGEAR Support Expert"

 

 

My response:

 

"Hello Ross,

Then please change the status of this case from an inquiry to a technical support case. However, there are no value in troubleshooting steps. The issue is not with my particular router or my particular settings. This has been happening to every owner of every RBRE960 since the product has been available, across all firmware versions that were released for it, regardless the number of customized settings, including fresh out of the box using the latest firmware with no user modified settings (that is actually the state I had discovered this issue in). Besides, it seems the engineering team already knows about it, so trying to troubleshoot this would not create any additional useful information. I, and all Netgear 960 users interested in seeing proper logs, are simply interested in knowing about the progress of engineering towards the release of a firmware that fixes it. How do we achieve that without taking unnecessary steps?

Thanks
Tuna"

Message 18 of 23
CrimpOn
Guru

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

I have this image of Don Quixote and the windmill.

Message 19 of 23
tuna_ertemalp
Luminary

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

Is that you, Sancho?

Message 20 of 23
tuna_ertemalp
Luminary

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

Here is a positive update: After exactly 1 month, 4 techs, many msgs, and one phone call, I was able to talk to a Level 2 tech (Eduard), we collected a bunch of data to reflect the experience while on the phone, the zip'd file has been given to him, and the issue is now getting (re?)escalated to engineering as I type this. My final point was "The logging issue I reported was already reported by many customers, either as cases or community posts, it happens on EVERY RBRE960 shipped, with EVERY f/w version, under ANY network setup, for ALL customers of this router. This is not specific to me. Logging is simply very badly broken for RBRE960 using any f/w." He reaffirmed that this is something that simply has to work for their flagship router. Eduard will keep me updated when he gets updates/requests from engineering.

 

Tuna

 

Message 21 of 23
tuna_ertemalp
Luminary

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

And, after opening a case on April 11, 2022, and having gotten it escalated to ENG, and then gotten a partial fix in a private F/W that I tested and reported back on May 18 (which also made my router to not recognize my paid Armor subscription anymore), it just got closed on Aug 2, without a publicly available F/W to include that fix I tested and preferably adding other fixes to further problems I reported on that fix. Sad. Here are the last few exchanges... I'm pretty sure I won't buy anything Netgear anymore...

 

2022-07-25 19:46:53

And I think everyone is wondering: For this expensive flagship product, there were monthly firmware releases in Nov, Dec and Jan. Then, since Jan, for the last 6 months, there have not been any public updates to the firmware. Clearly there are bugs to be fixed that are being worked on, with some already fixed. What is the reason that this advanced product isn't getting regular updates to keep customers happy, without pesky bugs?

 

2022-08-02 13:32:53

Hi Tuna,
We just got an update from our ENG team. Here's their final statement regarding this concern:
"We are constantly monitoring this unit. We fix the issues being raised and add enhancements. The customer can always contact us if he sees any problem."

 

2022-08-02 15:10:13

Not sure what value that statement adds to the issue at hand.
All the issues I reported for the last few months, now partly fixed, partly still broken, could have been fully fixed and released, and they still are not. I have already done my "customer can always contact us" part, repeatedly. It is the "We fix the issues being raised" part that didn't happen... 😞
I'm saddened by this pre-canned standard amateurishly dismissive response.

 

2022-08-02 21:59:16

Hi Tuna,

We were informed by our ENG team that they have noted your last feedback. They also recommended to close this case now. I will be closing this case.

 

Your case was closed on 2022-08-02

 
😔
Message 22 of 23
FURRYe38
Guru

Re: RBRE960 emailing logs has string buffer bug, doesn't send the whole log, doesn't list attacks, e

We've found that there is a log corruption problem that NG needs to fix across all Orbi lines. 

 

Message 23 of 23
Top Contributors
Discussion stats
  • 22 replies
  • 2913 views
  • 4 kudos
  • 5 in conversation
Announcements

Orbi 770 Series