Keep getting NETWORK CHANGED message since upgrading to 6.9.0

[amrob2]

Yesterday I upgraded the firmware on my RN 316 to the latest firmware 6.9.0. Since upgrading I have noticed that I keep getting the words NETWORK CHANGED on the NAS display which will then change to NO IP ADDRESS followed by 192.168.0.22.

This happens when I copy large quantities of data to the NAS. If I am not copying anything, the NAS display stays blank, but the second a large file transfer starts, the display will do as explained above.  I also use the NAS as a DLNA server connected to my Samsung tv and whenever the NETWORK CHANGED message appears, the tv loses access to the DLNA server, so the video I am watching on the tv will error and I have to wait for the tv to reconnect to the NAS again.

This is what I have done to ascertain where the problem lies:

The NAS has a static IP address of 192.168.0.22 allocated on ethernet 0, ethernet 1 is set to DHCP
The network router is used as a DHCP server, so it will automatically allocate IP address 192.168.0.22 via the MAC address of ethernet 0 on the NAS (even though the ethernet 0 is set to static with the same IP)
The network router does not allocate an IP address via the MAC address of ethernet 1, so if that port is used then the first available free IP will be allocated.

When the problem first occurred, I disconnected the ethernet cable from the NAS and plugged it back into the same port - just in case the ethernet cable had become loose - this had no effect.

I then disconnected the ethernet cable from ethernet 0 and plugged a different ethernet cable into ethernet 1 which was also connected to a different port on the router. This had no effect and the NETWORK CHANGED message continued to appear on the NAS whenever a large file transfer was initiated.

Doing this proved that it wasn't a problem with Static v DHCP on the network or a faulty ethernet cable / router port as different cables and ports had been used by carrying out this second test and the second test showed that the problem continued when I went from Static to a DHCP connection.

As the problem only started yesterday after installing OS 6.9.0 I have to assume that it is a bug in that OS update that is causing the netowrk connection to play up as I have never had any problems prior to this update.

Is anyone else having similar issues?  Is this a known problem?

[Skywalker · Accepted solution]

Thanks for the detailed information.  We are now able to duplicate the issue using a Nighthawk S8000 switch.  After updating the firmware to the version I mentioned earlier, the problem went away.  You can now download 6.9.1-T119 here, and it should resolve the issue.  Please confirm if it works for you.

[Sandshark]

Setting your NAS to a static IP in the same range that your router is giving out adresses will cause that.  The router does not know that the NAS has that address, and is also handing it out to your TV. 

Using a static IP on the NAS is not really the best way to accomplish having a constant IP.  It is better to reserve the address on the router side and let the NAS use DHCP.  Doing it that way prevents issues when you upgrade the router if it uses a different address range.

If you cannot reserve an address in your router, then put the static IP well up out from where your router will give out addresses, like 192.168.0.200.

Since you have a NAS with two ethernet ports, you should be able to go in through the one set for DHCP and set the other one up the same.  Then go to your router and reserve the IP it already has or some new one.  If it's some new one, you'll need to reboot the NAS for it to claim it.

[amrob2]

I may not have made it very clear in the opening message, but the tv does NOT have the same IP as the NAS. I allocate static IP's to some of my devices and let some be DHCP.  Static IP's are on things like servers, bluray players and tv's and printers. This is because I use certain ranges of IP's for specific device types.

No IP is duplicated anywhere on the network.

This problem has only started since upgrading the firmware. No network infrastructure changes have taken place to cause this, nor has there been any configuration changes on the network. The only change has been the firmware on the server - and that is why I assume that there is an issue with OS 6.9.0.

Per the opening post, I ascertained whether there were any problems with the static IP v DHCP IP allocation and the problem occured regardless of how the IP was allocated.

[Tinyhorns]

I also have serious network issues with 6.9.0.

My bonding keeps going up and down, rendering the NAS useless.

Worked like a charm for the last 2 years with different firmwares, but now, not working at all.

I have tried 2 different switches, same result with both.

Switch 1, Cisco3750, LACP configured.

Switch 2, HP 1920-16G LACP configured.

NAS has LACP configured as well.

Something is seriously broken with network implementation in 6.9.0.

See logs below:

[17/10/27 22:55:28 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth1 offline.
[17/10/27 22:55:50 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth1 offline.
[17/10/27 22:57:57 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth0 offline.
[17/10/27 22:58:02 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth1 offline.
[17/10/27 22:59:03 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth0 offline.
[17/10/27 23:00:57 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth1 offline.
[17/10/27 23:01:18 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth0 offline.
[17/10/27 23:02:24 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth1 offline.
[17/10/27 23:05:05 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth1 offline.
[17/10/27 23:05:30 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth0 offline.
[17/10/27 23:05:30 CEST] warning:system:LOGMSG_SENT_ALERT_MESG_FAILED Alert message failed to send.
[17/10/27 23:25:16 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth1 offline.
[17/10/27 23:29:17 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth0 offline.
[17/10/27 23:30:43 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth1 offline.
[17/10/27 23:32:24 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth0 offline.
[17/10/27 23:33:54 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth0 offline.
[17/10/27 23:33:54 CEST] warning:system:LOGMSG_SENT_ALERT_MESG_FAILED Alert message failed to send.
[17/10/27 23:34:48 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth0 offline.
[17/10/27 23:36:31 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth1 offline.
[17/10/27 23:39:17 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth0 offline.
[17/10/27 23:40:03 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth1 offline.
[17/10/27 23:54:30 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth1 offline.
[17/10/27 23:54:56 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth0 offline.
[17/10/27 23:57:22 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth0 offline.
[17/10/27 23:57:27 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth1 offline.
[17/10/27 23:59:07 CEST] warning:system:LOGMSG_BOND_NETWORK_SLAVE_NIC_DOWN Bond interface bond0 has slave interface eth0 offline.

[amrob2]

Further to @Sandshark reply, I changed the NAS to DHCP this morning just to see if that would make any difference.

2 minutes into a large data transfer and again "NETWORK CHANGED" "No IP Address" "192.168.0.22"

There is a DEFINITE problem with OS 6.9.0. It is not my configuration causing this as I have changed my configuration to what you suggested.

Can someone PLEASE flag this up to the support people to investigate. I have a second RN 316 server that is awaiting a firmware upgrade and that is not going to happen until this is resolved.

I should know not to upgrade to .0 firmware updates. I was bitten on the bum recently with iOS upgrades on my watch, iPads and iPhones where none of them work correctly even after three software patches.

[Tinyhorns]

do you have shell-access to the box?

I did see link up/down alot in my dmesg.

If you can login to it, can you type 'dmesg -T' in it and see if the timestamps does tell you anything fun?

I have not started troubleshoot it yet, but, it might be a bug in the linux-kernel that is in use, but im not sure yet.

but from what I can see in my own logs, it seem like the OS thinks that the network interface is switching between up and down, that might explain your problems as well as my own.

// T

[amrob2]

@Tinyhorns I don't have shell access, but from what I have seen of your problem above, I'd say it is a similar one to mine. It is like there is something in this firmware release that is making network connectivity reset momentarily and then restart whenever disk access is initiated - hence the "No IP Address" message I get on the NAS display, which then changes to the MAC assigned IP of "192.168.0.22".  It is annoying that nothing appears in the normal NAS logs which are viewable via the normal control panel.

We cannot be the only two people having this sort of problem and it is pretty annoying that there doesn't seem to be anyway of speaking to Netgear support people without having a paid support package.

[Tinyhorns]

@amrob2 It seems like I have the option to use "chat-support" during office-hours on my product still.
I will try and get their attention on monday if they do not spot this in here first.

Imho, this is clearly related to the 6.9.0 upgrade, and hopefully it this will get to their attention, they can start looking at it.

 edit: I would not upgrade your 2nd ReadyNAS just yet mate.

// T

[amrob2]

Dont't worry @Tinyhorns my second NAS will most certainly not be receiving firmware update 6.9.0

I must make a point of never installing .0 firmware updates in future as they are the ones that usually are buggy. I will always wait for the .x releases which will usually contain the fixes.

Still, it coule be worse. This could be apple and we could be discussing ios 11 which has had 3 software patches so far which have all failed to addres some of the serious bugs in it

[Tinyhorns]

@amrob2 could be alot worse, as long as there is no dataloss, I can survive without the nas for a week or 2.

// T

[amrob2]

@Tinyhorns that is what concerns me. I have terrabytes of data on my NAS and I regularly dump 40+gb to it at a time. I don't have any precise way of knowing, short of examing every file if the "writes" are being processed accurately at the moment.

[Tinyhorns]

After some more testing, it seems like putting heavy load on the NAS amplifies the disconnects.

I started a copy over the network at 18:40ish.

Clearly the "resets" of the interfaces happens more often then.

The switch keeps reporting the same.

[Sat Oct 28 18:10:51 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:10:58 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:24:03 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:24:53 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:25:19 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:25:28 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:25:29 2017] bond0: link status definitely down for interface eth1, disabling it
[Sat Oct 28 18:26:35 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:26:38 2017] bond0: link status definitely down for interface eth1, disabling it
[Sat Oct 28 18:35:54 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:36:44 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:39:16 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:39:40 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:39:41 2017] bond0: link status definitely down for interface eth1, disabling it
[Sat Oct 28 18:40:34 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:40:36 2017] bond0: link status definitely down for interface eth1, disabling it
[Sat Oct 28 18:41:19 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:41:20 2017] bond0: link status definitely down for interface eth1, disabling it
[Sat Oct 28 18:42:13 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:42:14 2017] bond0: link status definitely down for interface eth1, disabling it
[Sat Oct 28 18:43:11 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:43:14 2017] bond0: link status definitely down for interface eth1, disabling it
[Sat Oct 28 18:43:16 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:44:12 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:44:16 2017] bond0: link status definitely down for interface eth1, disabling it
[Sat Oct 28 18:44:17 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:45:09 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:45:10 2017] bond0: link status definitely down for interface eth1, disabling it
[Sat Oct 28 18:46:13 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:46:14 2017] bond0: link status definitely down for interface eth1, disabling it

[Sat Oct 28 18:43:19 2017] bond0: link status definitely up for interface eth0, 1000 Mbps full duplex
[Sat Oct 28 18:44:08 2017] e1000e: eth0 NIC Link is Down
[Sat Oct 28 18:44:08 2017] e1000e 0000:01:00.0 eth0: speed changed to 0 for port eth0
[Sat Oct 28 18:44:08 2017] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Sat Oct 28 18:44:12 2017] e1000e: eth0 NIC Link is Down
[Sat Oct 28 18:44:12 2017] e1000e 0000:01:00.0 eth0: speed changed to 0 for port eth0
[Sat Oct 28 18:44:12 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:44:14 2017] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Sat Oct 28 18:44:14 2017] bond0: link status definitely up for interface eth0, 1000 Mbps full duplex
[Sat Oct 28 18:44:16 2017] e1000e: eth1 NIC Link is Down
[Sat Oct 28 18:44:16 2017] e1000e 0000:04:00.0 eth1: speed changed to 0 for port eth1
[Sat Oct 28 18:44:16 2017] bond0: link status definitely down for interface eth1, disabling it
[Sat Oct 28 18:44:16 2017] bond0: first active interface up!
[Sat Oct 28 18:44:17 2017] e1000e: eth0 NIC Link is Down
[Sat Oct 28 18:44:17 2017] e1000e 0000:01:00.0 eth0: speed changed to 0 for port eth0
[Sat Oct 28 18:44:17 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:44:17 2017] bond0: first active interface up!
[Sat Oct 28 18:44:19 2017] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Sat Oct 28 18:44:19 2017] bond0: link status definitely up for interface eth1, 1000 Mbps full duplex
[Sat Oct 28 18:44:19 2017] bond0: first active interface up!
[Sat Oct 28 18:44:20 2017] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Sat Oct 28 18:44:20 2017] bond0: link status definitely up for interface eth0, 1000 Mbps full duplex
[Sat Oct 28 18:45:09 2017] e1000e: eth0 NIC Link is Down
[Sat Oct 28 18:45:09 2017] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Sat Oct 28 18:45:09 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:45:10 2017] e1000e: eth1 NIC Link is Down
[Sat Oct 28 18:45:10 2017] bond0: link status definitely down for interface eth1, disabling it
[Sat Oct 28 18:45:10 2017] bond0: first active interface up!
[Sat Oct 28 18:45:12 2017] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Sat Oct 28 18:45:12 2017] bond0: link status definitely up for interface eth0, 1000 Mbps full duplex
[Sat Oct 28 18:45:12 2017] bond0: first active interface up!
[Sat Oct 28 18:45:13 2017] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Sat Oct 28 18:45:13 2017] bond0: link status definitely up for interface eth1, 1000 Mbps full duplex
[Sat Oct 28 18:46:13 2017] e1000e: eth0 NIC Link is Down
[Sat Oct 28 18:46:13 2017] e1000e 0000:01:00.0 eth0: speed changed to 0 for port eth0
[Sat Oct 28 18:46:13 2017] bond0: link status definitely down for interface eth0, disabling it
[Sat Oct 28 18:46:13 2017] bond0: first active interface up!
[Sat Oct 28 18:46:14 2017] e1000e: eth1 NIC Link is Down
[Sat Oct 28 18:46:14 2017] e1000e 0000:04:00.0 eth1: speed changed to 0 for port eth1
[Sat Oct 28 18:46:14 2017] bond0: link status definitely down for interface eth1, disabling it
[Sat Oct 28 18:46:14 2017] bond0: first active interface up!
[Sat Oct 28 18:46:16 2017] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Sat Oct 28 18:46:16 2017] bond0: link status definitely up for interface eth0, 1000 Mbps full duplex
[Sat Oct 28 18:46:16 2017] bond0: first active interface up!
[Sat Oct 28 18:46:17 2017] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Sat Oct 28 18:46:17 2017] bond0: link status definitely up for interface eth1, 1000 Mbps full duplex

[amrob2]

Yes it is the same with me @Tinyhorns small file copies of 100mb or so are fine, but when you enter into the gb and up file size, that is when the disconnects start and the display starts doing the "No IP Address" followed by the allocated IP address.

It is likely something really small and silly that has been changed and very quick to fix, but as it seems Netgear themselves don't monitor this forum, it could likely take several days for a fix to come through.

[Tinyhorns]

Im sure they do not check it on weekends, but, hopefully, maybe during next week 🙂

[Tinyhorns]

@mdgm-ntgr Any thoughts on this post, are you guys aware of the network issues with 6.9.0 as well?

// T

[Tinyhorns]

@amrob2 Good news, I have manage to open a ticket with the support, and I have sent my logs to them so they can start investigate.
Hopefully they will have a fix soon for our issues.

// T

[amrob2]

@Tinyhorns let's hope so! I cannot keep going with my NAS like this. I have seen on other threads where people are having problems with backups that they have reverted back to 6.8.1 against Netgear's advice and as I have around 10TB of data on my NAS I cannot risk bricking the device and potentially losing all my data.

I can't even run a backup from this NAS to my second NAS (which is used to backup my primary NAS) owing to the network dropping errors.

When you contacted Netgear and they opened the ticket, did they give any indications as to whether they agree with you and I that there is a bug in 6.9.0?

[mdgm-ntgr]

We are monitoring this community (that's what I spend most of my time doing) and we have seen a higher than normal level of reports of networking issues for users running 6.9.0. We are investigating.

It is early Monday morning in the U.S. and I have relayed my concerns to the engineering team.

Feel free to send your logs in (see the Sending Logs link in my sig).

[Tinyhorns]

@mdgm-ntgr have a ticket open with attached logs.

Im guessing the engineers will request more info if needed in my case.

// T

[bugsplat]

---

@amrob2 wrote:

...

We cannot be the only two people having this sort of problem and it is pretty annoying that there doesn't seem to be anyway of speaking to Netgear support people without having a paid support package.

---

Had the same problem after I updated two RN314s.  One didn't develop the problem (so far), while one did.  I tried the usual --- changed cable, switched ports, switched switches using different ports ... nothing worked.  Originally using DHCP, but using static IP didn't solve the problem either.

I reinstalled the 6.9.0 firmware and that seemed to have fixed the problem ... for now.  A backup (to the other RN314) is scheduled later today, so I'll see how that goes.

[Tinyhorns]

I tried to reapply the 6.9.0-image again.

But, after putting som load on the box, same error as before.

So, it did not help for me.

 Edit: MD5 (ReadyNASOS-6.9.0-x86_64.img) = ef5899fc20c48270ac149bdf8d498536 <-- Image currently in use.

// T

[Mon Oct 30 17:24:47 2017] bond0: link status definitely up for interface eth0, 1000 Mbps full duplex
[Mon Oct 30 17:24:47 2017] bond0: now running without any active interface!
[Mon Oct 30 17:24:47 2017] bond0: link status definitely up for interface eth1, 1000 Mbps full duplex
[Mon Oct 30 17:24:47 2017] IPv6: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
[Mon Oct 30 17:25:18 2017] Adjusting tsc more than 11% (8932621 vs 8751289)
[Mon Oct 30 17:25:23 2017] nfsd: last server has exited, flushing export cache
[Mon Oct 30 17:25:23 2017] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[Mon Oct 30 17:25:23 2017] NFSD: starting 90-second grace period (net ffffffff88f13980)
[Mon Oct 30 17:34:03 2017] e1000e: eth0 NIC Link is Down
[Mon Oct 30 17:34:03 2017] e1000e 0000:01:00.0 eth0: speed changed to 0 for port eth0
[Mon Oct 30 17:34:03 2017] bond0: link status definitely down for interface eth0, disabling it
[Mon Oct 30 17:34:03 2017] bond0: first active interface up!

[Tinyhorns]

Good new, the engineers at Netgear has confirmed the bug, and working on a fix that will be available in 6.9.1

Lets just hope that the patch will be out soon.

// T

[bugsplat]

Well, my backup failed because of this bug ... so for now, I've shut down the RN314  

[amrob2]

Oh that is good news @Tinyhorns. The sooner a fix is out the better as I have not been able to even attempt a backup of my server since upgrading to 6.9.0 and I would be devastated if something happened to a hard drive on it between now and the fix being released.

It does make me wonder how much testing was carried out prior to releasing 6.9.0 though. Considering the network connection resets/drops (whatever!) as soon as you put a read/write load of as little as 1GB on it, this should have easily been picked up in the development stage and most definitely in the UAT stage.

[StephenB]

---

@amrob2 wrote:

Considering the network connection resets/drops (whatever!) as soon as you put a read/write load of as little as 1GB on it, this should have easily been picked up in the development stage and most definitely in the UAT stage.

---

FWIW, this is not showing up on my two RN52x systems.  So it might not be that easy to detect (even though it is clearly impacting some users).