Keep getting NETWORK CHANGED message since upgrading to 6.9.0

That is the exact same problem as what I am experiencing, but my box has not taken on the hot fix for some reason.

But it would appear that 6.9.0+4 has not resolved the problem from what you have said anyway :smileysad:

I am loathe to trust transfers and backups with this problem.

Checking the integrity of the transfers or backups (which are a transfer anyway) would be a mammoth task.

Or am I ignorant of a setting which can be used to md5sum(slow!!) ,or whatever, the original and the transferred files?

Is this relevant - there is nothing wrong with the cable connection at either end nor the usb3 cable:

Nov 02 09:19:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=configuration
Nov 02 09:19:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=disconnect
Nov 02 09:19:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=idle
Nov 02 09:19:23 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable removed
Nov 02 09:19:26 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable changed: State=ready
Nov 02 09:20:01 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=configuration
Nov 02 09:20:01 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=disconnect
Nov 02 09:20:01 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=idle
Nov 02 09:20:01 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable removed
Nov 02 09:20:03 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable changed: State=ready
Nov 02 09:20:08 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=configuration
Nov 02 09:20:08 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=disconnect
Nov 02 09:20:08 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=idle
Nov 02 09:20:09 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable removed
Nov 02 09:20:14 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable changed: State=ready
Nov 02 09:20:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=configuration
Nov 02 09:20:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=disconnect
Nov 02 09:20:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=idle
Nov 02 09:20:23 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable removed
Nov 02 09:20:29 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable changed: State=ready

---

amrob2 wrote:

That is the exact same problem as what I am experiencing, but my box has not taken on the hot fix for some reason.

 

But it would appear that 6.9.0+4 has not resolved the problem from what you have said anyway :smileysad:

---

Yup, still hosed ... tried copying over a 10GB file tonight --- total disaster with 6.9.0+4  :smileyfrustrated:

davidr1 using ssh, you can see what the kernel thinks about bonding.

That requires enabled ssh tho, and Im not sure thats the recomended way.

But, my flapping bond shows up in the default log-view in the webgui.

if you have ssh enabled do the following.

# dmesg -T

dmesg will show the message buffer of the kernel, -T will show you timestamps instead of epoch-time.

// T

I too have this identical issue (Bond interface bond0 has slave interface eth0 offline) since updating to 6.9.0 and as others have said it seems to occur when moving multi Gb files.  I also have the hotifx installed and am at ReadyNAS OS 6.9.0+4. So the hotfix did not fix the issue

I reproduced the issue on my RN314, don't see the issue on RN628X.

but file transfer is still successfully when system log show interface offline,

the issue should be exist on older FW,

6.9.0 added fail log if there is enterface offline in bonding, so 6.9.0 can see the offline issue,

I filed a bug to track the issue.

evan2 Hi,

I agree that the filetransfer do work, but, it will take longer time, and there is a lot of tcp retransmiting going on and dropped packets.

RX packets:87859001 errors:0 dropped:30499 overruns:0 frame:0

IO-wait on the connected clients are sky-high as well, as expected.

Thats how I did notice it in the first place.

But, I do disagree that it this bug exist on older firmware, I say this bug was introduced in 6.9.0.

I do continuous moninotring of the connected clients, and it was a clear spike in IO-wait/Load on the box.

Please see the graph attached.

Guess when I did the FW-upgrade to 6.9.0?

// T

I agree with Tinyhorns that this bug was introduced in 6.9.0 evan2 and I can prove it quite easily.

My RN316 server is used as a DLNA server as well as a file server. The DLNA server is streaming to a Samsung smart tv for around 8 hours a day and never had any problems prior to 6.9.0

Now, as soon as a file transfer of a GB or more is initiated, the network connection on the NAS disconnects per the details explained numerous times in this thread already and the DLNA connection to the smart tv is lost. This is annoying and unacceptable when you are trying to stream video files to a tv for 8 hours or more and the stream is constantly being interrupted whenever a file transfer takes place.

Perhaps the bug was there in previous versions of OS 6, but 6.9.0 has intensified the problem - although I can say that I have never before had any issues using DLNA on this server, so personally I do not believe that the bug ever existed before.

If you have access to a NAS with 6.9.0+2 or +4 on it, set it up as a DLNA server and stream a video/audio file to a tv or other device that can take advantage of DLNA. Then try and copy a file of more than 1GB - you'll see DLNA will go down as soon as the network problems start.

---

amrob2 wrote:

 

...as soon as a file transfer of a GB or more is initiated, the network connection on the NAS disconnects per the details explained numerous times in this thread already

---

I do understand your frustration, and I agree that this is a problem that is affecting other users that needs to be fixed.

But I think it's important that everyone understand that this is not affecting all users and platforms. 

I just transferred a 125 GiB file to/from my RN526 with no problems.  The Windows 10 PC I was using was quite busy doing other things (loaded over 70%) - some of which were also accessing the NAS. But the file transfer speed still ran between 100-300 MB/s in both directions, with no dropped connections or other misbehavior. 

Note the PC and the NAS are equipped with 10 gigabit ethernet.  I don't know if this matters, but ReadyCloud is turned off and I'm not using NIC bonding (no point to it with 10gbase-T)

---

StephenB wrote:

I do understand your frustration, and I agree that this is a problem that is affecting other users that needs to be fixed.

 

But I think it's important that everyone understand that this is not affecting all users and platforms. 

---

I know it isn't affecting everyone, but while there are only a few of us complaining on here, there will be many more people with the same NAS box models that are affected who are not complaining - probably because they don't know that these forums exist. Not everyone will be running DLNA or have the NAS located in line of sight where the display is showing error messages, so they may not even be aware that there is a problem.

The users that are experiencing problems but don't know it will likely be thinking "My network is a bit slow" not realising that it is the NAS that is causing the slow transfers where the network connection fails and restarts.

You are correct in saying that it is not affecting everyone, but regardless of whether it is or not, I have given good money to Netgear for my NAS, I entrust Netgear to look after my extremely valuable data and I expect Netgear to only put firmware onto my box that works and has been thoroughly tested. I also expect Netgear to fix any firmware faults within a reasonable time frame - it is now 7 days since the firmware update and the fault has not been corrected, the presumed fix (6.9.0+4) which I have not received but others have, has not resolved the issue. I therefore think that I am not being unreasonable to post on here about the problem - and each time I do post on here, I am speaking for myself and other users on this thread who have the exact same issue. I do not claim to speak for every ReadyNAS owner, nor would I ever try and speak for every ReadyNAS owner.

StephenB Hello Sir,

Maybe not all platform are affected, and maybe everyone doesnt see the performance penalties due to lack of monitoring and stuff like that.
Like I said, I did only notice it in xymon, one of the boxes started to send out alerts regarding very high (load 20+) cpu-load, in my case, related to very high io-wait and my switch started to send snmp-traps telling me that I have flapping LACP.

Pretty sure most of the private persons that uses the ReadyNAS does not have that in place, so, there for I'm almost sure that this can go unnoticed for a long time in most usecases, since as long as you are not logged in to the gui, or have some other notification-system, it will go unnoticed.

If you do a backup of your PC/Mac to the RN, if it takes 20 min or 24 min, doesnt really matter, and is nothing you reflect on.

Most protocolls have very good fault tolerance, so the file operation will not be aborted, just retried over and over again until finished.

In this case, it will go unnoticed as well.

So, stating that most people doesnt have this issue is in my view, a bit strong.

Do you not agree?

amrob2

In my case, the display on the physical RN does not show this error at all.

I do not login to the gui that often, but, the gui does not say that much, just a small notice that eth1 or eth0 is down for a while.

// T

---

amrob2 wrote:

---

StephenB wrote:

I do understand your frustration, and I agree that this is a problem that is affecting other users that needs to be fixed.

 

But I think it's important that everyone understand that this is not affecting all users and platforms. 

---

I know it isn't affecting everyone, but while there are only a few of us complaining on here, there will be many more people with the same NAS box models that are affected who are not complaining - probably because they don't know that these forums exist. Not everyone will be running DLNA or have the NAS located in line of sight where the display is showing error messages, so they may not even be aware that there is a problem.

I don't disagree - not everyone posts, and depending on their usage they might not realize that they have the problem.

But I think (given the wording in several of the posts) that people generally reading or searching the forum could think that the problem affects all 6.9.0 systems. I want to make sure they don't jump to that conclusion. 

Certainly the networking problems are on Netgear's short list of issues, and I hope they resolve it soon with a new hot-fix.

6.9.0 incorporated a network driver update from Intel to fix a transmit hang under heavy load.  This update is the most likely cause for the issues people are seeing.  That being said, we have transferred more than 1,000TB of data on RN31x series units trying to reproduce this issue, and have not seen anything like what has been described here.  So there is clearly a combination of factors involved here.

We plan to post a test build with that driver update backed out in a few hours, once it passes sanity checks.

I suspect that the main things affecting the issue are:

- bonded/teaming nics (all 4 of mine use lacp layer 3+4 mode, 1gbit)

- dhcp aquired ips. Mine are all static assigned.

I am not sure that if this issue is the same or related to the other thread regarding the resetting of lacp layer mode, but seems both are network/driver related.

Skywalker Can we help in some way?

Do you need additional information?

Debug-mode?

The current image seems to load alot of different drivers.

[    2.940935] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[    2.941027] e1000e: Intel(R) PRO/1000 Network Driver - 3.3.6-NAPI
[    3.148764] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.3.0-k
[    3.148903] Intel(R) 10GbE PCI Express Linux Network Driver - version 5.2.1

// T

TeknoJnky Im using static ip, same issue.

And I have done it both static and DHCP all with the same results.

Also experiencing the same problem here with an RN314 after the 6.9.0 update.  When a client starts its backups, I am getting hit with similar warnings:

Thu Nov 2 2017 4:19:45    System: Bond interface bond0 has slave interface eth1 offline.
Thu Nov 2 2017 4:19:28    System: Bond interface bond0 has slave interface eth0 offline.
Thu Nov 2 2017 4:17:36    System: Bond interface bond0 has slave interface eth1 offline.
Thu Nov 2 2017 4:14:11    System: Bond interface bond0 has slave interface eth1 offline.
Thu Nov 2 2017 4:12:24    System: Bond interface bond0 has slave interface eth1 offline.
Thu Nov 2 2017 4:11:32    System: Bond interface bond0 has slave interface eth1 offline.

The unit is bonded to a Netgear GS716T switch.

Looking forward to a timely fix along with everyone else!

---

Tinyhorns wrote:

davidr1 using ssh, you can see what the kernel thinks about bonding.

That requires enabled ssh tho, and Im not sure thats the recomended way.

 

But, my flapping bond shows up in the default log-view in the webgui.

 

if you have ssh enabled do the following.

# dmesg -T

dmesg will show the message buffer of the kernel, -T will show you timestamps instead of epoch-time.

 

// T

 

---

File attached. Note that the NAS has just been turned on.

David

---

TeknoJnky wrote:

I suspect that the main things affecting the issue are:

 

- bonded/teaming nics (all 4 of mine use lacp layer 3+4 mode, 1gbit)

- dhcp aquired ips. Mine are all static assigned.

 

 

---

In my case I have a STATIC IP set so the second point is not correct.

As for the frustration angle - I too am feeling the pain, but we need to cut the Netgear developers some slack too. Given this is not impacting all users it will be hardder to replicate. While we'd all like this fixed urgently, tracking down such issues takes time - especially determining the specific factors to trigger it. I'd prefer time be taken to get it right rather than rush an untested fix that could cause other issues.

Here is a summary of my setup:
- RN 314 + EDA 500

- bonded ethernet

- teaming mode IEEE 802.3ad LACP

- layer 2+3

- static IPv4 address

- dynamic DHCP IPv6 address

- connected to Netgear Nighthawk S8000 switch (ports 6 and 7)

This may be unrelated since I updated the switch software at the same time :-( so have been holding off chasing the issue below:

• I am NOT using link aggregation on the 2 matching switch ports because the moment I set aggregation I lose access to the NAS shares and cannot ping the NAS
• This means that the ports are bonded but not connected to an aggregated link!

---

Skywalker wrote:

Thanks for the detailed information.  We are now able to duplicate the issue using a Nighthawk S8000 switch.  After updating the firmware to the version I mentioned earlier, the problem went away.  You can now download 6.9.1-T119 here, and it should resolve the issue.  Please confirm if it works for you.

---

Hi Skywalker

Is that the same image for the RN316 machines? I notice the URL mentions X86, but isn't the 316 processor an AMD chip?

I just want to check before I install and potentially brick my box.

---

Skywalker wrote:

Thanks for the detailed information.  We are now able to duplicate the issue using a Nighthawk S8000 switch.  After updating the firmware to the version I mentioned earlier, the problem went away.  You can now download 6.9.1-T119 here, and it should resolve the issue.  Please confirm if it works for you.

---

I've installed the image on my RN314. I also re-enabled link aggregation on ports 6 & 7 of my S8000. While I still can't ping my NAS, I CAN access the shares and all is working fine otherwise. I have copied a 42Gb test file from the NAS to a local drive and have a peak speed of 108 Mb/sec and average speed of 95.7 (my usual speed pre 6.9.0). Copying the same file back is also as pre 6.9.0 (peaks at 75.9, ave 66.7)

Logs are reporting no bond issues. I'd say this looks fixed with the update. :smileyhappy:

Skywalker ignore the question about whether the firmware update is for RN 316's, I saw that someone else had loaded it onto their 314 and I know that the 316 has the same processor as them.

I have loaded the update onto my 316 and put through quite intensive data writes with two laptops copying via a GB network 100GB of data from each laptop to the server while at the same time the server being connected via DLNA on my tv with video streaming.

There were no problems at all, the copy processes completed with no network disconnects, the display on the NAS stayed completely blank and the DLNA streaming continued uninterrupted - so I would say that you have managed to resolve the issue (at least from my perspective).

Thanks to everyone who helped get this fixed, including end users who provided logs to Netgear for investigating :smileyhappy:

I have just completed transferring 260GB of files each ranging from 6.6GB to 19.3GB without a problem.

At the same time DLNA was uninterrupted.

It appears that the problem has been solved. Would it be correct that it is 314x specific and if so does that mean that future updates might again be a problem for the 314 design? (It is not the Nighthawk switch as I don't have one - I have the FVS318G router).

Thank you sincerely to everybody at Netgear and in the community who have obviously worked very hard and with the customers to solve this problem.

David