Keep getting NETWORK CHANGED message since upgrading to 6.9.0

Thanks mdgm-ntgr the packages log shows the following:

ii  readynasos                     6.9.0+2                    amd64        ReadyNASOS base system

Does that mean that the hot fix has been applied or not? If it does, then it has not cured the problem that I am experiencing, which is the one in the opening post on this thread where the display changes to "NETWORK CHANGED", "No IP Address" then "192.168.0.22"

I have noticed that unlike before that I can do copying (reading) large chunks of data (300gb at a time) from the NAS to a different computer on my network without the problem kicking in, but trying to write files of more than 2gb at a time still results in the error.

 6.9.0+4 is where you should be at, just try another reboot or two and check back after 15-20 minutes 

If one wants, its possible to use 'apt-get' to get the firmware in place.

but that requires ssh-access.

# apt-get update && apt-get upgrade readynasos
......
# dpkg -l |grep readynasos
ii  readynasos                     6.9.0+4                    amd64        ReadyNASOS base system

That's not the way we'd recommend doing it.

mdgm-ntgr

So, I did upgrade, and after putting some load on the box.

My issue is still the same.

[Wed Nov  1 08:11:47 2017] e1000e: eth1 NIC Link is Down
[Wed Nov  1 08:11:47 2017] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:11:47 2017] e1000e: eth1 NIC Link is Down
[Wed Nov  1 08:11:47 2017] bond0: link status definitely down for interface eth1, disabling it
[Wed Nov  1 08:11:47 2017] bond0: first active interface up!
[Wed Nov  1 08:11:49 2017] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:11:49 2017] bond0: link status definitely up for interface eth0, 1000 Mbps full duplex
[Wed Nov  1 08:11:49 2017] bond0: first active interface up!
[Wed Nov  1 08:11:50 2017] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:11:51 2017] bond0: link status definitely up for interface eth1, 1000 Mbps full duplex
[Wed Nov  1 08:12:39 2017] e1000e: eth0 NIC Link is Down
[Wed Nov  1 08:12:39 2017] e1000e 0000:01:00.0 eth0: speed changed to 0 for port eth0
[Wed Nov  1 08:12:39 2017] bond0: link status definitely down for interface eth0, disabling it
[Wed Nov  1 08:12:39 2017] bond0: first active interface up!
[Wed Nov  1 08:12:41 2017] e1000e: eth1 NIC Link is Down
[Wed Nov  1 08:12:41 2017] e1000e 0000:04:00.0 eth1: speed changed to 0 for port eth1
[Wed Nov  1 08:12:41 2017] bond0: link status definitely down for interface eth1, disabling it
[Wed Nov  1 08:12:41 2017] bond0: first active interface up!
[Wed Nov  1 08:12:43 2017] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:12:43 2017] bond0: link status definitely up for interface eth0, 1000 Mbps full duplex
[Wed Nov  1 08:12:43 2017] bond0: first active interface up!
[Wed Nov  1 08:12:44 2017] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:12:44 2017] bond0: link status definitely up for interface eth1, 1000 Mbps full duplex
[Wed Nov  1 08:13:35 2017] e1000e: eth0 NIC Link is Down
[Wed Nov  1 08:13:35 2017] e1000e 0000:01:00.0 eth0: speed changed to 0 for port eth0
[Wed Nov  1 08:13:35 2017] bond0: link status definitely down for interface eth0, disabling it
[Wed Nov  1 08:13:35 2017] bond0: first active interface up!
[Wed Nov  1 08:13:38 2017] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:13:38 2017] bond0: link status definitely up for interface eth0, 1000 Mbps full duplex
[Wed Nov  1 08:13:40 2017] e1000e: eth1 NIC Link is Down
[Wed Nov  1 08:13:40 2017] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:13:40 2017] e1000e: eth1 NIC Link is Down
[Wed Nov  1 08:13:40 2017] bond0: link status definitely down for interface eth1, disabling it
[Wed Nov  1 08:13:40 2017] bond0: first active interface up!
[Wed Nov  1 08:13:40 2017] e1000e: eth0 NIC Link is Down
[Wed Nov  1 08:13:40 2017] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:13:40 2017] e1000e: eth0 NIC Link is Down
[Wed Nov  1 08:13:40 2017] bond0: link status definitely down for interface eth0, disabling it
[Wed Nov  1 08:13:40 2017] bond0: first active interface up!
[Wed Nov  1 08:13:43 2017] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:13:43 2017] bond0: link status definitely up for interface eth1, 1000 Mbps full duplex
[Wed Nov  1 08:13:43 2017] bond0: first active interface up!
[Wed Nov  1 08:13:43 2017] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:13:43 2017] bond0: link status definitely up for interface eth0, 1000 Mbps full duplex

Thanks bedlam1 I have shutdown and rebooted three times now (with gaps of several hours between each cycle) but still am only on 6.9.0+2 - although it appears from Tinyhorns post that the hot fix he received hasn't cured the problem anyway :smileyfrustrated:

That is most frustrating Tinyhorns I still have not received the hot fix despite powering down and rebooting three times in the last 24 hours since we were told a hot fix is available, so I cannot even confirm if the fix would resume normal functionality on my 316 :smileyfrustrated:

Rebooted both my RN314s last night, and I immediately check the ReadyNAS os to see where they were at ... lo and behold, they updated to the +4 hot-fixed immediately after reboot.

I'll do some heavy-duty file transfers when I get home tonight to see If I still have the problem (subject of this thread).

Well that is strange where everyone else is picking up the hot fix, but my 316 has not.  It has full network connectivity, it can see the internet (it happily sends out email alerts when I reboot/shut it down). So I guess I am going to have to wait until next week until everyone receives it (per mdgm-ntgr post on the subject yesterday) :smileysad:

Perhaps PM mdgm, and ask if he can check the logs.  There might be something there that explains why the hot-fix isn't being installed.

This is affecting me too - only since upgrading to OS 6.9.0

NAS314 with 3 x 4TB drives

Static IP address on NAS and reserved in router

The patch appears to have uploaded - package.log shows:

ii readynasos 6.9.0+4 amd64 ReadyNASOS base system

I don't know which log has the bond details ??

When transferring large file to the NAS or in a backup using Remote Rsync Server to an an attached USB3 drive (4TB):

'Network changed' is displayed periodically followed after abut 2 second the IP address - which is the correct IP address.

That is the exact same problem as what I am experiencing, but my box has not taken on the hot fix for some reason.

But it would appear that 6.9.0+4 has not resolved the problem from what you have said anyway :smileysad:

I am loathe to trust transfers and backups with this problem.

Checking the integrity of the transfers or backups (which are a transfer anyway) would be a mammoth task.

Or am I ignorant of a setting which can be used to md5sum(slow!!) ,or whatever, the original and the transferred files?

Is this relevant - there is nothing wrong with the cable connection at either end nor the usb3 cable:

Nov 02 09:19:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=configuration
Nov 02 09:19:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=disconnect
Nov 02 09:19:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=idle
Nov 02 09:19:23 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable removed
Nov 02 09:19:26 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable changed: State=ready
Nov 02 09:20:01 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=configuration
Nov 02 09:20:01 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=disconnect
Nov 02 09:20:01 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=idle
Nov 02 09:20:01 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable removed
Nov 02 09:20:03 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable changed: State=ready
Nov 02 09:20:08 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=configuration
Nov 02 09:20:08 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=disconnect
Nov 02 09:20:08 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=idle
Nov 02 09:20:09 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable removed
Nov 02 09:20:14 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable changed: State=ready
Nov 02 09:20:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=configuration
Nov 02 09:20:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=disconnect
Nov 02 09:20:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=idle
Nov 02 09:20:23 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable removed
Nov 02 09:20:29 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable changed: State=ready

---

amrob2 wrote:

That is the exact same problem as what I am experiencing, but my box has not taken on the hot fix for some reason.

 

But it would appear that 6.9.0+4 has not resolved the problem from what you have said anyway :smileysad:

---

Yup, still hosed ... tried copying over a 10GB file tonight --- total disaster with 6.9.0+4  :smileyfrustrated:

davidr1 using ssh, you can see what the kernel thinks about bonding.

That requires enabled ssh tho, and Im not sure thats the recomended way.

But, my flapping bond shows up in the default log-view in the webgui.

if you have ssh enabled do the following.

# dmesg -T

dmesg will show the message buffer of the kernel, -T will show you timestamps instead of epoch-time.

// T

I too have this identical issue (Bond interface bond0 has slave interface eth0 offline) since updating to 6.9.0 and as others have said it seems to occur when moving multi Gb files.  I also have the hotifx installed and am at ReadyNAS OS 6.9.0+4. So the hotfix did not fix the issue

I reproduced the issue on my RN314, don't see the issue on RN628X.

but file transfer is still successfully when system log show interface offline,

the issue should be exist on older FW,

6.9.0 added fail log if there is enterface offline in bonding, so 6.9.0 can see the offline issue,

I filed a bug to track the issue.

evan2 Hi,

I agree that the filetransfer do work, but, it will take longer time, and there is a lot of tcp retransmiting going on and dropped packets.

RX packets:87859001 errors:0 dropped:30499 overruns:0 frame:0

IO-wait on the connected clients are sky-high as well, as expected.

Thats how I did notice it in the first place.

But, I do disagree that it this bug exist on older firmware, I say this bug was introduced in 6.9.0.

I do continuous moninotring of the connected clients, and it was a clear spike in IO-wait/Load on the box.

Please see the graph attached.

Guess when I did the FW-upgrade to 6.9.0?

// T

I agree with Tinyhorns that this bug was introduced in 6.9.0 evan2 and I can prove it quite easily.

My RN316 server is used as a DLNA server as well as a file server. The DLNA server is streaming to a Samsung smart tv for around 8 hours a day and never had any problems prior to 6.9.0

Now, as soon as a file transfer of a GB or more is initiated, the network connection on the NAS disconnects per the details explained numerous times in this thread already and the DLNA connection to the smart tv is lost. This is annoying and unacceptable when you are trying to stream video files to a tv for 8 hours or more and the stream is constantly being interrupted whenever a file transfer takes place.

Perhaps the bug was there in previous versions of OS 6, but 6.9.0 has intensified the problem - although I can say that I have never before had any issues using DLNA on this server, so personally I do not believe that the bug ever existed before.

If you have access to a NAS with 6.9.0+2 or +4 on it, set it up as a DLNA server and stream a video/audio file to a tv or other device that can take advantage of DLNA. Then try and copy a file of more than 1GB - you'll see DLNA will go down as soon as the network problems start.

---

amrob2 wrote:

 

...as soon as a file transfer of a GB or more is initiated, the network connection on the NAS disconnects per the details explained numerous times in this thread already

---

I do understand your frustration, and I agree that this is a problem that is affecting other users that needs to be fixed.

But I think it's important that everyone understand that this is not affecting all users and platforms. 

I just transferred a 125 GiB file to/from my RN526 with no problems.  The Windows 10 PC I was using was quite busy doing other things (loaded over 70%) - some of which were also accessing the NAS. But the file transfer speed still ran between 100-300 MB/s in both directions, with no dropped connections or other misbehavior. 

Note the PC and the NAS are equipped with 10 gigabit ethernet.  I don't know if this matters, but ReadyCloud is turned off and I'm not using NIC bonding (no point to it with 10gbase-T)

---

StephenB wrote:

I do understand your frustration, and I agree that this is a problem that is affecting other users that needs to be fixed.

 

But I think it's important that everyone understand that this is not affecting all users and platforms. 

---

I know it isn't affecting everyone, but while there are only a few of us complaining on here, there will be many more people with the same NAS box models that are affected who are not complaining - probably because they don't know that these forums exist. Not everyone will be running DLNA or have the NAS located in line of sight where the display is showing error messages, so they may not even be aware that there is a problem.

The users that are experiencing problems but don't know it will likely be thinking "My network is a bit slow" not realising that it is the NAS that is causing the slow transfers where the network connection fails and restarts.

You are correct in saying that it is not affecting everyone, but regardless of whether it is or not, I have given good money to Netgear for my NAS, I entrust Netgear to look after my extremely valuable data and I expect Netgear to only put firmware onto my box that works and has been thoroughly tested. I also expect Netgear to fix any firmware faults within a reasonable time frame - it is now 7 days since the firmware update and the fault has not been corrected, the presumed fix (6.9.0+4) which I have not received but others have, has not resolved the issue. I therefore think that I am not being unreasonable to post on here about the problem - and each time I do post on here, I am speaking for myself and other users on this thread who have the exact same issue. I do not claim to speak for every ReadyNAS owner, nor would I ever try and speak for every ReadyNAS owner.

StephenB Hello Sir,

Maybe not all platform are affected, and maybe everyone doesnt see the performance penalties due to lack of monitoring and stuff like that.
Like I said, I did only notice it in xymon, one of the boxes started to send out alerts regarding very high (load 20+) cpu-load, in my case, related to very high io-wait and my switch started to send snmp-traps telling me that I have flapping LACP.

Pretty sure most of the private persons that uses the ReadyNAS does not have that in place, so, there for I'm almost sure that this can go unnoticed for a long time in most usecases, since as long as you are not logged in to the gui, or have some other notification-system, it will go unnoticed.

If you do a backup of your PC/Mac to the RN, if it takes 20 min or 24 min, doesnt really matter, and is nothing you reflect on.

Most protocolls have very good fault tolerance, so the file operation will not be aborted, just retried over and over again until finished.

In this case, it will go unnoticed as well.

So, stating that most people doesnt have this issue is in my view, a bit strong.

Do you not agree?

amrob2

In my case, the display on the physical RN does not show this error at all.

I do not login to the gui that often, but, the gui does not say that much, just a small notice that eth1 or eth0 is down for a while.

// T

---

amrob2 wrote:

---

StephenB wrote:

I do understand your frustration, and I agree that this is a problem that is affecting other users that needs to be fixed.

 

But I think it's important that everyone understand that this is not affecting all users and platforms. 

---

I know it isn't affecting everyone, but while there are only a few of us complaining on here, there will be many more people with the same NAS box models that are affected who are not complaining - probably because they don't know that these forums exist. Not everyone will be running DLNA or have the NAS located in line of sight where the display is showing error messages, so they may not even be aware that there is a problem.

I don't disagree - not everyone posts, and depending on their usage they might not realize that they have the problem.

But I think (given the wording in several of the posts) that people generally reading or searching the forum could think that the problem affects all 6.9.0 systems. I want to make sure they don't jump to that conclusion. 

Certainly the networking problems are on Netgear's short list of issues, and I hope they resolve it soon with a new hot-fix.

6.9.0 incorporated a network driver update from Intel to fix a transmit hang under heavy load.  This update is the most likely cause for the issues people are seeing.  That being said, we have transferred more than 1,000TB of data on RN31x series units trying to reproduce this issue, and have not seen anything like what has been described here.  So there is clearly a combination of factors involved here.

We plan to post a test build with that driver update backed out in a few hours, once it passes sanity checks.

I suspect that the main things affecting the issue are:

- bonded/teaming nics (all 4 of mine use lacp layer 3+4 mode, 1gbit)

- dhcp aquired ips. Mine are all static assigned.

I am not sure that if this issue is the same or related to the other thread regarding the resetting of lacp layer mode, but seems both are network/driver related.