Re: Nighthawk X6 R8000 Upgrading to latest firmware may have bricked my router

daveb78 · ‎2018-12-07

This morning, in the Nighthawk iOS app, I noticed that a firmware update was available for my Nighthawk X6 R8000 AC3200 router. The app indicated it would take approximately 3 minutes for the update, and I started it. I've found the Nighthawk app to be a very convenient way to update firmwar, since it is completely automatic after the upgrade is initiated through the app.

About 20 minutes later, I noticed that my WiFi signal still had not come back. I went to physically check thr router, and after observing it for several minutes, it appeared to be in an endless boot-loop. With no end in sight, I was forced to dicsonnect it, and I'm now on my ISP-provided router.

My nighthawk X6 seems to be in a bad state. Powering it on, the boot sequence ends with a single solid red LED illuminated. I've attempted a factory reset (long-press the back panel reset button with a straightened paperclip) to no avail.

I'm afraid that this firmware upgrade bricked my router. I've never had an experience where a manufacturuer supplied firmware has resulted in a non-operable device.

Can anyone help me bring it back to life? Does the Nighthawk X6 have any mechanism to revert the device to the preiously installed firmware forlloing a failed upgrade?

Thanks in advance!

daveb78 · ‎2018-12-16

Thanks Bill,

Unfortuntaely, I have a visual impairment that firmly puts that solution beyond my capabilities.

So, the only solutions available to me would appear to be:

Paying someone else to do it...That is, if it is even possible to locate a Third Party repair shop (or individual for that matter) that would do such a thing.
Outright replacement.

My thanks to the community, but it appears that I'm hosed.

View solution in original post

michaelkenward · ‎2018-12-08

@daveb78 wrote:

I've never had an experience where a manufacturuer supplied firmware has resulted in a non-operable device.

Sadly, I have seen similar reports from users of stuff made by most manufacturers.

It is more likely when people allow something to update itself over wifi, where the risk of interruption is much higher.

One way out is TFTP.

How to upload firmware to a NETGEAR router using Windows TFTP | Answer | NETGEAR Support

myersw · ‎2018-12-08

Why so they have an app that does not worn about doing over wireless?

Crap firmware and crap app. Yes this is your troll friend.

daveb78 · ‎2018-12-08

@michaelkenward thanks for pointing me at the tftp solution. Unfortunately, it didn't work-out.

When I boot-up the router, while holding-down the reset button, the router goes through a repeating sequence where the power LED is either off, illuinated white, then illuimnated orange/amber. Some or all of the other LEDs are illuminated white, except when the power LED is solid organge/amber. The power LED is never in a state that I would consider to be "flashing". This is not quite what I would have expected from the description in step 10 "Watch the Power LED. It starts with an orange color, and then start flashing". I allowed the repeating sequence to occur at least 10 times, then release the reset button, and hit ENTER on the tftp command. The response I received was "Connect request failed" - so I'm not certain if this is because I've done something incorrect (but don't think so) or that my router is beyond recovery.

Also, after you pointed me at the Windows TFTP solution, I also happened upon Netgear's similar KB article using a TFTP Client. What I found intriguing was that those instructions, while similar, don't mention either waiting for at least "10 flashes" or releaseing the reset button. I find this disconcerting, as I would have expected the router-side of the procedure to be identical, regardless of the TFTP client.

@michaelkenward wrote:

It is more likely when people allow something to update itself over wifi, where the risk of interruption is much higher.

I totally hear you on that. I had assumed (perhaps incorrectly) the because Netgear specifically warns against updating firmware over WiFi, and that they also provide an app feature to upgrade firmware, that the app itself plays no active role in the firmware upgrade process. I.e.the app merely "tells" the router to upgrade, and all the work is offloaded to the router itself with no WiFi involvement in the upgrade process. Again thought, that's merely an assumption on my part, mostly because that's how I would've done it to avoid the ptifalls

IrvSp · ‎2018-12-08

@daveb78 wrote:

I totally hear you on that. I had assumed (perhaps incorrectly) the because Netgear specifically warns against updating firmware over WiFi, and that they also provide an app feature to upgrade firmware, that the app itself plays no active role in the firmware upgrade process. I.e.the app merely "tells" the router to upgrade, and all the work is offloaded to the router itself with no WiFi involvement in the upgrade process. Again thought, that's merely an assumption on my part, mostly because that's how I would've done it to avoid the ptifalls

See, this is part of the problem. People read or in some cases do not. At one time the README file associated with the Firmware (you can go back maybe a years worth and see this) SPECIFICALLY said you should use a WIRED DEVICE to update the f/w. Now even when you buy a new router there is a sticker on it (at least on my R8000 I bought 8 months ago) TELLING you to use the NIGHTHAWK app (my R8000 said NETGAR UP which was replaced with it) to set up the router. You were to use the attached USERID's and PASSWORD's to hook up the device. The first thing it does IS check for a f/w update and if you allow it, it will update itself.

'You' are not doing it wirelessly. It is all done internally between the router and NG F/W server. Yes, something could happen and corrupt the d/l, depending on a whole bunch of factors. Now where if gets 'messy' is people see 'wireless'. So they think they can D/L the f/w to the Laptop and then UNZIP the file, open a browser, go to Genie and do a Router Update pointing back to the flash on the WIRELESS device... an invitation to trouble. That is why WIRED was at one time strongly suggested.

Since NG has added the AUTO UPDATE feature to firmware, we've hit a lot of posts with a 'bricked' or non-responsive router due to that. Should not matter, but some f/w is just not working right. On top of that, NG at one time stated you should RESET the router and manually reenter all settings after a f/w update. I don't know if they DID something that this wasn't required anymore (but it is the first thing to try if you have an update problem) and have REALLY REALLY tested this or just that it seemed to work so they removed that request?

michaelkenward · ‎2018-12-09

@IrvSp wrote:

On top of that, NG at one time stated you should RESET the router and manually reenter all settings after a f/w update. I don't know if they DID something that this wasn't required anymore (but it is the first thing to try if you have an update problem) and have REALLY REALLY tested this or just that it seemed to work so they removed that request?

Did they really suggest a reset after every update?

My line has always been that you do not reset unless you have post-flash problems.

@IrvSp wrote:

Since NG has added the AUTO UPDATE feature to firmware, we've hit a lot of posts with a 'bricked' or non-responsive router due to that.

Indeed. But is this cause and effect? Or just a coincidence?

Over in the Orbi zone you will find that people complain constantly about the inability to disable autoupdates. Too many complaints of systems failing with new firmware.

Netgear seems to be stuck between trying to make it easier to update firmware because people have a hard time and end up with very old firmware, and forcing updates on users who want to manage their own devices.

IrvSp · ‎2018-12-09

@michaelkenward wrote:

@IrvSp wrote:

On top of that, NG at one time stated you should RESET the router and manually reenter all settings after a f/w update. I don't know if they DID something that this wasn't required anymore (but it is the first thing to try if you have an update problem) and have REALLY REALLY tested this or just that it seemed to work so they removed that request?

Did they really suggest a reset after every update?

My line has always been that you do not reset unless you have post-flash problems.

No, actually it wasn't a suggestion... I looked at R7000-Firmware-Version-1-0-3-24 flash, and from it:

=============

Note: Please remember to do factory default after firmware upgrade. Enter "Backup Settings" in the web GUI and click "Erase" button of "Revert to factory default settings".

=============

First line at the top...

Other things to do:

=========

Note: To avoid wireless disconnect issue during the firmware download process, NETGEAR recommends that firmware upgrade be performed on a computer with wired connection.

Write down all the settings which you changed from the default values, since you may need to re-enter them manually.

=========

Didn't bother to look at every one to determine when it stopped.

@michaelkenward wrote:

@IrvSp wrote:

Since NG has added the AUTO UPDATE feature to firmware, we've hit a lot of posts with a 'bricked' or non-responsive router due to that.

Indeed. But is this cause and effect? Or just a coincidence?

.
.

.

Netgear seems to be stuck between trying to make it easier to update firmware because people have a hard time and end up with very old firmware, and forcing updates on users who want to manage their own devices.

Not everyone has a perfect line, and some have longer runs from modem to router, power fluctuations, etc. Even AP's that get updated.

The 'best' way, D/L the flash and unzip it to a wired PC... Once down if you can UNZIP it, it is probably OK. Then sending the flash to the router via wired would probably never have a line drop. However in today's day and age I suspect not every home has a wired connection? Almost every desktop PC comes with both a wired and wireless NIC. Wireless capability means you can put the router anywhere and not have to have a PC near it. So you are basically doing the flash direct from the Internet when you kick it off within the router, either using wireless and the router update button or Auto Update. All that needs to happen is a glitch in power or the internet and you might toast the router. Wireless method of feeding a download flash would probably be the most problematic. TCP/IP packets get dropped all the time. Even ISP feeds... look at your modem... Mine shows this data

=============

Downstream

	DCID	Freq	Power	SNR	Modulation	Octets	Correcteds	Uncorrectables
Downstream 1	23	717.00 MHz	-3.20 dBmV	38.61 dB	256QAM	3373068172	2893	18872
Downstream 2	16	675.00 MHz	-4.40 dBmV	37.64 dB	256QAM	3082618035	118	2420
Downstream 3	17	681.00 MHz	-4.20 dBmV	38.98 dB	256QAM	3218541165	2523	20697
Downstream 4	18	687.00 MHz	-3.90 dBmV	38.61 dB	256QAM	3163539557	117	2304
Downstream 5	19	693.00 MHz	-4.00 dBmV	37.64 dB	256QAM	3149494398	133	2229
Downstream 6	20	699.00 MHz	-3.90 dBmV	38.98 dB	256QAM	3261358637	126	1268
Downstream 7	21	705.00 MHz	-3.60 dBmV	38.98 dB	256QAM	3265960605	4748	4849
Downstream 8	22	711.00 MHz	-3.50 dBmV	38.61 dB	256QAM	3220795892	5170	24964
Downstream 9	24	723.00 MHz	-2.90 dBmV	38.61 dB	256QAM	2891225299	157	1983
Downstream 10	25	729.00 MHz	-3.10 dBmV	38.98 dB	256QAM	3978673882	127	715
Downstream 11	26	735.00 MHz	-3.50 dBmV	38.98 dB	256QAM	4019583536	115	1860
Downstream 12	27	741.00 MHz	-3.90 dBmV	38.61 dB	256QAM	4033157086	1784	12869
Downstream 13	28	747.00 MHz	-4.20 dBmV	38.98 dB	256QAM	4041107260	145	732
Downstream 14	29	753.00 MHz	-4.10 dBmV	38.61 dB	256QAM	4023810168	1283	14767
Downstream 15	30	759.00 MHz	-3.30 dBmV	38.61 dB	256QAM	4038039606	2453	22494
Downstream 16	31	765.00 MHz	-3.10 dBmV	38.61 dB	256QAM	3157177367	2918	22325

===============================

What is worse for me, that is only 8+ days from the last boot of it. I didn't even know it happened, and I've got a UPS on it and the router. A lot of uncorrectables, but it is miniscule compared to the OCTET count. All you need is one of those be sent through either as it was missed by the modem or corrected improperly and sent through while you are d/l'ing to the router the flash. I don't know if NG's code checks the flash once it is completely down or just reads it, writes it, reads the next one? Looking at my DEBUG.HTM does show plenty of storage in the R8000 and NVRAM available though.

daveb78 · ‎2018-12-09

First let me say thanks to everyone whos made a contribution to my issue so far. I've done some more research, and additional tinkering,, and would like to bring this discussion thread back on-topic. Which is recovering my Nighthawk X6 R800.

Here are the vital details, about the current state of my troubleshooting efforts:

Per the KB article that @michaelkenward pointed me to, I attempted a Windows TFTP file transfer of firmware. This consistently resulted int "Connection Request Failed".
I noticed that when following the KB article instructions, thr router appeared not to enter the state described "10. Watch the Power LED. It starts with an orange color, and then start flashing." At no point does the Power LED start flashing. Rather, my router appears to be in a continuous boot-loop.
My router state is not impacted by holding-down the reset button while poering it on. It enters the apparent boot-loop regardless of the reset button.
Further research in the Netgear community forums led to several posts that claim while in a boot loop. Demonstrated by continuous pinging of the router IP, where most ping requests time-out, but for a very brief period in the boot loop. Several post authors claim that during this brief period, it is possible to perfrom the TFTP transfer of new firmware if it is initiated at just the right moment.
Continuous pinging of my router does show that in a boot loop cycle lasting about 100 seconds, there is a very small window where the router responds to the ping (only about 4 sequential requests).

So my conclusion thus far, is that my router is indeed stuck in a continuous boot loop. I've attempted to manually initiate the TFTP upload inside the brief window, of oppourtunity, as others have described, but the results remain the same. So, rather than trying to manually hit an impossibly small window, I authored a PowerShell script that:

Continuously pings my router.
Upon a successful response, that others indicated should include "TTL=100" (though I'm personally not certain that's important - I've observed that pings that don't time out always have TTL=100), the TFTP transfer is nearly instantaneously attempted.
If the upload fails, pinging resumes for another attempt at the next window.

Unfortunately, this also resulted in failure, after running this for a few hours. Were it actually possible to sneak in the TFTP upload in this impossibly smal window, I'd expect that a programatic approach (as I've outlined above) would have met with success relatively quickly. Instead, having made (literally) hundreds up upload attempts, none succeeded.

I've attached the PowerShell script (unfortuately as a PDF due to attachment filetype restrictions), in case anyone wants to inspect it - if I've made a mistake that anyone spots, I'd certainly like to know about it.

So, does anyone have any suggestion about what I might try next to get my router out this continuous boot loop?

daveb78 · ‎2018-12-14

Is there anyone out there in the community who can help me get my router out of this apparent boot loop?

Netgear, are you there?

antinode · ‎2018-12-14

> Is there anyone out there in the community who can help me get my
> router out of this apparent boot loop?

   I wouldn't bet on it (there being anyone, that is), but I seem to
recall some reports of clever/lucky timing of the TFTP "put" command
being helpful. There apparently is also a USB-serial_port (low voltage)
technique, which works at an even lower level than TFTP (and requires
opening the box). I'd expect a Web search for keywords like, say:
      Netgear serial port
to find things.

> I've attached the PowerShell script [...]

   I've never dealt with such a problem, and I do little with
PowerShell, so I know nothing, but you don't seem to have gotten any
useful advice yet, so...

   If timing matters, then I wouldn't do this:

      $pingTimeout = 2000 # ping timeout in ms.

As I understand it, "ping" sends a query, and then waits for a response.
If you don't get an immediate response, then what's the value in waiting
up to two seconds before retrying? If there is a susceptible time
window, and it's open for only, say, one second, then you could easily
miss it that way. I'd use the shortest "ping" timeout possible -- just
large enough to discriminate between success and failure.

   Such a tight polling loop wastes CPU, but, for a non-general-use case
like this, who cares?

> # if the ping result contains a TTL=100, [...]

   Why not simply check the exit status of the "ping" program?

> # 2. That it be executed from the same directory as the firmware file
> to be uploaded (defined by $firmwareFile)

   Does the current directory really matter if you use an appropriate
path for "$firmwareFile"?

daveb78 · ‎2018-12-14

Thanks for taking a look at my script @antinode!

@antinode wrote:

> Is there anyone out there in the community who can help me get my
> router out of this apparent boot loop?

   I wouldn't bet on it (there being anyone, that is), but I seem to
recall some reports of clever/lucky timing of the TFTP "put" command
being helpful. There apparently is also a USB-serial_port (low voltage)
technique, which works at an even lower level than TFTP (and requires
opening the box). I'd expect a Web search for keywords like, say:
      Netgear serial port
to find things.

>> Yes, your correct about the clever/lucky timing reports...it was comming accross those that formed the basis for my approach.

> I've attached the PowerShell script [...]

   I've never dealt with such a problem, and I do little with
PowerShell, so I know nothing, but you don't seem to have gotten any
useful advice yet, so...

   If timing matters, then I wouldn't do this:

      $pingTimeout = 2000 # ping timeout in ms.

As I understand it, "ping" sends a query, and then waits for a response.
If you don't get an immediate response, then what's the value in waiting
up to two seconds before retrying? If there is a susceptible time
window, and it's open for only, say, one second, then you could easily
miss it that way. I'd use the shortest "ping" timeout possible -- just
large enough to discriminate between success and failure.

   Such a tight polling loop wastes CPU, but, for a non-general-use case
like this, who cares?

>> Quite correct. I'd actually tested with a timeout of 3ms, but published to the forum with a longer value by accident. I did note, by observation that when a series of ping requests succeed, the first ping response time was nearly 2 seconds, while the subsequent pings in the success series were consistently 2ms - making the smallest possible timeout value 3ms. The problem is that since the first ping response takes nearly 2 seconds to respond, it likely means that the first interpretation of success is after the second consecutive successful ping response.

> # if the ping result contains a TTL=100, [...]

   Why not simply check the exit status of the "ping" program?
>> That was my first instinct as well. However, it seems that the usual ways of determining success don't apply to ping, an so I had to go the route that I did. The probably unnecessary piece is specifically checking for a value of "100" - but did so to align with what others were reporting for a manual approach.

"A successful PING does NOT always return an %errorlevel% of 0
Therefore to reliably detect a successful ping - pipe the output into FIND and look for the text "TTL""

https://ss64.com/nt/ping.html

> # 2. That it be executed from the same directory as the firmware file
> to be uploaded (defined by $firmwareFile)

   Does the current directory really matter if you use an appropriate
path for "$firmwareFile"?

>> You're right, it doesn't. That instruction is to simplify the process for anyone else out there who wants to try an use my script.

Now that the work week is over, I can disconnect my PC from my LAN, connect it up to my router, and try again overnight, as see if I'm able to sneak the firmware in via TFTP over a longer period.

I'll report back results tomorrow.

daveb78 · ‎2018-12-15

So, I ran continuous ping and TFTP attempts for about 14 straight hours - using a ping timeout of 3ms. After a successful ping response, a TFTP transfer was immediately attempted. My 14-hour run resulted in:

52008 ping requests.
546 successful ping responses.
546 failed TFTP transfer attempts.
No successfull TFTP transfer attempts.

It would appear that I'm completely hosed.

Before this issue, I was extremely satisfied with my Nighthawk X6 R8000, and Netgear in general - this being my first experience with a Netgear router.

Now my previously positive experience has been completely soured. If using Netgear tools (properly), to upgrade Netgear firmware, in a Netgear router, results in a non-functional device - with no meaningful support from Netgear (the company itself - I'm not referring to those in the community who tried to help me) then I have to say that Netgear isn't standing behind their product.

When others ask me about my experience with Netger (and they do), where before I would have said "Im having a great experience with my router! It performs very well, and the Nighthawk app is quite convenient.", I'll now have to say "You can get a Netgear router if you like, but be warned that Netgear could break it at any time with a bad firmware flash, and Netgear won't help you fix it.".

Since Netgear hasn't stepped-up to help me, I'll have to consider a replacement from another vendor - I have no intention of using my ISP-provided gateway as my router indefinitely.

All suggestions from the community on a replacement for my now useless Nighthawk X6 R8000 would be most welcome!

myersw · ‎2018-12-15

Other option is break it open and use a serial connection which means getting a serial to USB adapter in most cases. Have recovered r8000 that way. At that point you have nothing to lose unless still under warrenty as you will break the seal to open it up.

daveb78 · ‎2018-12-16

Thanks Bill,

Unfortuntaely, I have a visual impairment that firmly puts that solution beyond my capabilities.

So, the only solutions available to me would appear to be:

Paying someone else to do it...That is, if it is even possible to locate a Third Party repair shop (or individual for that matter) that would do such a thing.
Outright replacement.

My thanks to the community, but it appears that I'm hosed.