NVRAM bug - FOUND IT!!

[SLK-Purdue]

I have posted for some time about various issues with corrupt config parameters (NVRAM).  If nothing else, I am a really stubborn geek with 50+ years of experience and I don't like unexplained failures.  Computing should be a predictable thing!!  I wasn't going to let this bug beat me.

Long story short - the script below shows a 100% failure that corrupts NVRAM.  The issue is a software error (maybe hardware assisted) if the size of NVRAM goes over 0xFFFF.   Not clear if this is the max physical size of NVRAM or a software bug with 16-bit ariithmatic but, either way, it is a serious software bug.  You might say "Why do I care?  My config can't ever be that big!"   You would be surprised.  In addition to normal information, NVRAM also contains information about every device your router sees.  There is no delete/garbage collection.  NVRAM grows to that size and then you have a problem but don't even know it.   Eventually, the corruption will cause a noticeable problem but that may be a long time after the actual failure event or when you try to reload your config after a disaster.

NG has open source for some of the Orbi software but this part is proprietary so I can't work on a fix.  (would be happy to do that for FREE if NG would provide the source).

100% failure scrupt follows.  Exist in every version of firmware I have tried.

Scott

--------

= PuTTY log 2021.10.22 11:20:37 =~=~=~=~=~=~=~=~=~=~=~=

< load my production NVRAM>

root@RBR50:/# nvram show | grep size:

size: 65364 bytes (168108 left)

root@RBR50:/# nvram set junk1=1234567890

root@RBR50:/# nvram show | grep size:

size: 65381 bytes (168091 left)

root@RBR50:/# nvram set junk2=1234567890

root@RBR50:/# nvram set junk3=1234567890

root@RBR50:/# nvram set junk4=1234567890

root@RBR50:/# nvram set junk5=1234567890

root@RBR50:/# nvram show | grep size:

size: 65449 bytes (168023 left)

root@RBR50:/# nvram set junk6=1234567890

root@RBR50:/# nvram show | grep size:

size: 65466 bytes (168006 left)

root@RBR50:/# nvram set junk7=1234567890

root@RBR50:/# nvram show | grep size:

size: 65483 bytes (167989 left)

root@RBR50:/# nvram set junk8=1234567890

root@RBR50:/# nvram show | grep size:

size: 65500 bytes (167972 left)

root@RBR50:/# nvram set junk9=12345

root@RBR50:/# nvram show | grep size:

size: 65513 bytes (167959 left)   <size = 0xFFE9, OK>

root@RBR50:/# nvram set junk10=12345

root@RBR50:/# nvram show | grep size:

size: 12 bytes (233460 left)  <NVRAM toasted>

[FURRYe38]

I would contact a forum moderator and get this information to them:

@Blanca_O @DarrenM @ChristineT 

[CrimpOn]

Congratulations.  My Orbi matches your exactly used plus free = 233,472.  Looks like a classic case of using an 'int' where it should be a 'long int'.

Is this a V1 or V2 Orbi?

How many devices are represented in the Orbi tables (LAN reservation, previous but not connected, etc.?)

[SLK-Purdue]

Tested on V1. 

54 active connected devices and 56 items in the previous but not connected per WebUI..  Looks like there are 198 items in the NVRAM "access_control" table.

Scott

---

@CrimpOn wrote:

Congratulations.  My Orbi matches your exactly used plus free = 233,472.  Looks like a classic case of using an 'int' where it should be a 'long int'.

Is this a V1 or V2 Orbi?

How many devices are represented in the Orbi tables (LAN reservation, previous but not connected, etc.?)

---

[SLK-Purdue]

---

@FURRYe38 wrote:

I would contact a forum moderator and get this information to them:

@Blanca_O @DarrenM @ChristineT 

---

Done.

Scott

[CrimpOn]

---

@SLK-Purdue wrote:

54 active connected devices and 56 items in the previous but not connected per WebUI..  Looks like there are 198 items in the NVRAM "access_control" table.

---

My access_control list has 77 items in it. Looking at the Access Control web page

37 Active devices

36 Allowed but not connected

  1 Blocked but not connected

----

74 Total.  I think the missing 3 are three entries for ETH0.  74+3 = 77.

I would expect you to have 54+56+3 = 113, not 198.

I do not have Access Control enabled, so there is no particular difficulty in deleting "allowed but not connected' devices because they will simply authenticate the next time they show up.  But, where did those extra 85 entries in your system come from?

[SLK-Purdue]

My bad -  access_control table has 105 entries.   Counting error.  I think that matches current totals as you did.

Thanks,

Scott

---

@CrimpOn wrote:

---

@SLK-Purdue wrote:

54 active connected devices and 56 items in the previous but not connected per WebUI..  Looks like there are 198 items in the NVRAM "access_control" table.

---

My access_control list has 77 items in it. Looking at the Access Control web page

37 Active devices

36 Allowed but not connected

  1 Blocked but not connected

----

74 Total.  I think the missing 3 are three entries for ETH0.  74+3 = 77.

I would expect you to have 54+56+3 = 113, not 198.

I do not have Access Control enabled, so there is no particular difficulty in deleting "allowed but not connected' devices because they will simply authenticate the next time they show up.  But, where did those extra 85 entries in your system come from?

---

[SLK-Purdue]

My ARM assembly language skills aren't as good as some other hardware architectures but, based on a disassembly of NVRAM, it sure looks like that code has a hard coded size constant of 0x10000.  That would explain the bug.  No clue on the actually allocated size of NVRAM in the EEPROM but I am guessing the buffer size is the real issue.   Would likely be an easy fix if NVRAM source code were part of the NG "open source" but, sadly, it isn't.

NG - why keep this a secret?  There is nothing in using EEPROM for NVRAM that is your secret intellectual property.   Everybody does that.  I understand not wanting to release the secret sauce behind for mesh algorithm but that not in reading and writing NVRAM.

AGAIN - this is a VERY serious problem for all users.  Send me the source code and I will fix this FOR FREE and send you back code and give you all rights.  You're getting a consultant with 50+ years of systems experience that bills at $500/hr when I work for money.

Scott

[FURRYe38]

Wondering if this issue is recent or how far it goes back in FW versions...

Wondering if also this is only or Orbi AC or does this effect more products globally. 

[SLK-Purdue]

---

@FURRYe38 wrote:

Wondering if this issue is recent or how far it goes back in FW versions...

Wondering if also this is only or Orbi AC or does this effect more products globally. 

---

If this were part of the open source code, we would know.  Sadly, it isn't 

Since this is not something that would likely be introduced in an update, I would guess bases on experience that this is a day 1 bug. 

Scott 

[FURRYe38]

Well maybe however the NVRAM and configuration section maybe something that is used across multiple product lines. Similar code base can be used across different products. Like the UI, basically the same format and feature listings, though some areas change per product lines. Hopefully this is only effected Orbi AC then. 

Hopefully NG will take a look and get it corrected. Thanks for your deep dive man. Wish NG had you on as a dev. 

[SLK-Purdue]

---

@FURRYe38 wrote:

Well maybe however the NVRAM and configuration section maybe something that is used across multiple product lines. Similar code base can be used across different products. Like the UI, basically the same format and feature listings, though some areas change per product lines. Hopefully this is only effected Orbi AC then. 

Hopefully NG will take a look and get it corrected. Thanks for your deep dive man. Wish NG had you on as a dev. 

---

Been pretty clear that I am available and the price is right. 

[FURRYe38]