Always practice your disaster protection plan!

[SLK-Purdue]

After a 50+ year career in IT, I religiously believe in backups.  More importantly, I believe in practicing reloading your backups as part of your disaster protection plan.   I back up EVERYTHING and I practice reloading periodically.  BEWARE - Think through your reload plan carefully because if your reload fails (and it will at least one) you can be left with not choice but to restart at square 1.

I backup my Orbi config periodically via the WebUI.   A few months ago, I decided to practice the Orbi reload with a spare RBR50.   It failed!  OK, that was disturbing but I tried a fresh backup and an immediate reload.  That failed.  The message indicated that the NVRAM was corrupt.   I factory reset the spare RBR50, backed it up, and tried the reload again.   That worked.   Hmmm ....

As part of the forced firmware upgrade by NG (another unhappy story), I did a full factory reset to my production config.  (1 router = 3 satellites)   I backup that up and it did successfully reload to the spare router.   I had a known good process.   Since then, I have periodically backed up my production RBR50 and regularly practiced reloading the spare.   After about two month, the reload that had regularly worked failed again!

Long story but this sent me down the black hole of understanding Orbi backups and learning things like ORBICFG to decode the backup file.

Short form - there is a bug in Orbi firmware that corrupts NVRAM.  You have no way of knowing this without checking a backup.  I have seen the corruption happen in a little as 2 days of operation.  This problem can explain a lot of strange behavior since once NVRAM is corrupt, router operation is unpredictably.  I have some debugging skills and I have tried to understand this but I don't have enough information from NG.  (hint, hint - I would like to help but I need more technical iinformation about Orbi software and NVRAM operation.)

Bottom line - it is possible that a reload of an Orbi config backup in a disaster won't work!  There is a bug in Orbi firmware that corrupts NVRAM (your config) during normal operations.  You can look at your backup using ORBICFG (https://github.com/Fysac/orbicfg ).  If you see non-ASCII junk in the output, you have a corrupt NVRAM.

AGAIN BEWARE - don't try a reload on a production system without a full plan.   A reload failure will possibly require a factory reset and leave you in a bad situation at square 1.

[CrimpOn]

Thanks for the effort on this.  Time to install a C compiler. The gist of this appears to be:

Plan A.

• Make a configuration backup every time there has been a change to the Orbi configuration
  (add anything to or edit a device table, change advanced setup parameters, allow/block devices, modify DNS, etc. etc.)
• Keep a log of every change made to the router.
• Keep the previous (good) version of the cfg file until the new cfg file has been validated.
• Test each new cfg file for integrity - how?
  * Does the orbicfg tool indicate corruption in an obvious, quick way? (have not installed it yet)
  * Loading every cfg file on the spare Orbi would verify integrity, but many users have no spare router.
• When the latest cfg file is corrupt, perform a Factory Reset, load the previous (good) cfg file, use that log to make the last changes, and resume until the next time a cfg file is corrupt.

Plan B

• If the 'good' cfg file will not load:
• Factory Reset, and
• Configure the router manually.

[SLK-Purdue]

Yes, running ORBICFG on your backup will tell you if it is good.   Normal output will be ASCII commands.  When NVRAM get corrut, you get very obvious non-ASCII crud in the output.

Scott

---

@CrimpOn wrote:

Thanks for the effort on this.  Time to install a C compiler. The gist of this appears to be:

Plan A.

• Make a configuration backup every time there has been a change to the Orbi configuration
  (add anything to or edit a device table, change advanced setup parameters, allow/block devices, modify DNS, etc. etc.)
• Keep a log of every change made to the router.
• Keep the previous (good) version of the cfg file until the new cfg file has been validated.
• Test each new cfg file for integrity - how?
  * Does the orbicfg tool indicate corruption in an obvious, quick way? (have not installed it yet)
  * Loading every cfg file on the spare Orbi would verify integrity, but many users have no spare router.
• When the latest cfg file is corrupt, perform a Factory Reset, load the previous (good) cfg file, use that log to make the last changes, and resume until the next time a cfg file is corrupt.

Plan B

• If the 'good' cfg file will not load:
• Factory Reset, and
• Configure the router manually.

---

[CrimpOn]

---

@SLK-Purdue wrote:

Yes, running ORBICFG on your backup will tell you if it is good.   Normal output will be ASCII commands.  When NVRAM get corrupted, you get very obvious non-ASCII crud in the output.

---

This is SO FUN.  I have not found non-ASCII characters in the output (yet), but discovered that line 1 of my output reads: "=666951560373216".  Notice there is no parameter name.  I guess that it might load (??), but what would the Orbi do with that first line?

My "go to" tool for comparing files is WinMerge (https://winmerge.org/ ) I redirect standard output to a text file which can be sorted or fed into WinMerge.  Every difference gets highlighted.  I plan to make a backup every day for a while and "see what changes" from one day to the next. (My previous backup was last August on V2.7.2.106, which is a completely different firmware release.)

For those not comfortable with running C programs on Windows, I complied orbicfg on Windows 10 with the "Tiny C Compiler" (tcc). The exe file is here: https://www.dropbox.com/s/3w1pss8hqk9c2q3/orbicfg.exe?dl=0 

I am wondering if in addition to validating the checksum (which might be what causes the Orbi upload configuration operation to fail), what code could be used to report non-ASCII characters to standard error? Something in the while loop at line 66?

[SLK-Purdue]

That first line looks wrong. Haven't seen that before. 

You are going to be surprised about how much changes in "normal" operation. 

Scott

---

@CrimpOn wrote:

---

@SLK-Purdue wrote:

Yes, running ORBICFG on your backup will tell you if it is good.   Normal output will be ASCII commands.  When NVRAM get corrupted, you get very obvious non-ASCII crud in the output.

---

This is SO FUN.  I have not found non-ASCII characters in the output (yet), but discovered that line 1 of my output reads: "=666951560373216".  Notice there is no parameter name.  I guess that it might load (??), but what would the Orbi do with that first line?

My "go to" tool for comparing files is WinMerge (https://winmerge.org/ ) I redirect standard output to a text file which can be sorted or fed into WinMerge.  Every difference gets highlighted.  I plan to make a backup every day for a while and "see what changes" from one day to the next. (My previous backup was last August on V2.7.2.106, which is a completely different firmware release.)

For those not comfortable with running C programs on Windows, I complied orbicfg on Windows 10 with the "Tiny C Compiler" (tcc). The exe file is here: https://www.dropbox.com/s/3w1pss8hqk9c2q3/orbicfg.exe?dl=0 

I am wondering if in addition to validating the checksum (which might be what causes the Orbi upload configuration operation to fail), what code could be used to report non-ASCII characters to standard error? Something in the while loop at line 66?

---