Re: Keep getting NETWORK CHANGED message since upgrading to 6.9.0

amrob2 · ‎2017-11-02

I agree with @Tinyhorns that this bug was introduced in 6.9.0 @evan2 and I can prove it quite easily.

My RN316 server is used as a DLNA server as well as a file server. The DLNA server is streaming to a Samsung smart tv for around 8 hours a day and never had any problems prior to 6.9.0

Now, as soon as a file transfer of a GB or more is initiated, the network connection on the NAS disconnects per the details explained numerous times in this thread already and the DLNA connection to the smart tv is lost. This is annoying and unacceptable when you are trying to stream video files to a tv for 8 hours or more and the stream is constantly being interrupted whenever a file transfer takes place.

Perhaps the bug was there in previous versions of OS 6, but 6.9.0 has intensified the problem - although I can say that I have never before had any issues using DLNA on this server, so personally I do not believe that the bug ever existed before.

If you have access to a NAS with 6.9.0+2 or +4 on it, set it up as a DLNA server and stream a video/audio file to a tv or other device that can take advantage of DLNA. Then try and copy a file of more than 1GB - you'll see DLNA will go down as soon as the network problems start.

StephenB · ‎2017-11-02

@amrob2 wrote:

...as soon as a file transfer of a GB or more is initiated, the network connection on the NAS disconnects per the details explained numerous times in this thread already

I do understand your frustration, and I agree that this is a problem that is affecting other users that needs to be fixed.

But I think it's important that everyone understand that this is not affecting all users and platforms.

I just transferred a 125 GiB file to/from my RN526 with no problems. The Windows 10 PC I was using was quite busy doing other things (loaded over 70%) - some of which were also accessing the NAS. But the file transfer speed still ran between 100-300 MB/s in both directions, with no dropped connections or other misbehavior.

Note the PC and the NAS are equipped with 10 gigabit ethernet. I don't know if this matters, but ReadyCloud is turned off and I'm not using NIC bonding (no point to it with 10gbase-T)

amrob2 · ‎2017-11-02

@StephenB wrote:
I do understand your frustration, and I agree that this is a problem that is affecting other users that needs to be fixed.

But I think it's important that everyone understand that this is not affecting all users and platforms.

I know it isn't affecting everyone, but while there are only a few of us complaining on here, there will be many more people with the same NAS box models that are affected who are not complaining - probably because they don't know that these forums exist. Not everyone will be running DLNA or have the NAS located in line of sight where the display is showing error messages, so they may not even be aware that there is a problem.

The users that are experiencing problems but don't know it will likely be thinking "My network is a bit slow" not realising that it is the NAS that is causing the slow transfers where the network connection fails and restarts.

You are correct in saying that it is not affecting everyone, but regardless of whether it is or not, I have given good money to Netgear for my NAS, I entrust Netgear to look after my extremely valuable data and I expect Netgear to only put firmware onto my box that works and has been thoroughly tested. I also expect Netgear to fix any firmware faults within a reasonable time frame - it is now 7 days since the firmware update and the fault has not been corrected, the presumed fix (6.9.0+4) which I have not received but others have, has not resolved the issue. I therefore think that I am not being unreasonable to post on here about the problem - and each time I do post on here, I am speaking for myself and other users on this thread who have the exact same issue. I do not claim to speak for every ReadyNAS owner, nor would I ever try and speak for every ReadyNAS owner.

Tinyhorns · ‎2017-11-02

@StephenB Hello Sir,

Maybe not all platform are affected, and maybe everyone doesnt see the performance penalties due to lack of monitoring and stuff like that.
Like I said, I did only notice it in xymon, one of the boxes started to send out alerts regarding very high (load 20+) cpu-load, in my case, related to very high io-wait and my switch started to send snmp-traps telling me that I have flapping LACP.

Pretty sure most of the private persons that uses the ReadyNAS does not have that in place, so, there for I'm almost sure that this can go unnoticed for a long time in most usecases, since as long as you are not logged in to the gui, or have some other notification-system, it will go unnoticed.

If you do a backup of your PC/Mac to the RN, if it takes 20 min or 24 min, doesnt really matter, and is nothing you reflect on.

Most protocolls have very good fault tolerance, so the file operation will not be aborted, just retried over and over again until finished.

In this case, it will go unnoticed as well.

So, stating that most people doesnt have this issue is in my view, a bit strong.

Do you not agree?

@amrob2

In my case, the display on the physical RN does not show this error at all.

I do not login to the gui that often, but, the gui does not say that much, just a small notice that eth1 or eth0 is down for a while.

// T

StephenB · ‎2017-11-02

@amrob2 wrote:

@StephenB wrote:

I do understand your frustration, and I agree that this is a problem that is affecting other users that needs to be fixed.

But I think it's important that everyone understand that this is not affecting all users and platforms.

I know it isn't affecting everyone, but while there are only a few of us complaining on here, there will be many more people with the same NAS box models that are affected who are not complaining - probably because they don't know that these forums exist. Not everyone will be running DLNA or have the NAS located in line of sight where the display is showing error messages, so they may not even be aware that there is a problem.

I don't disagree - not everyone posts, and depending on their usage they might not realize that they have the problem.

But I think (given the wording in several of the posts) that people generally reading or searching the forum could think that the problem affects all 6.9.0 systems. I want to make sure they don't jump to that conclusion.

Certainly the networking problems are on Netgear's short list of issues, and I hope they resolve it soon with a new hot-fix.

Skywalker · ‎2017-11-02

6.9.0 incorporated a network driver update from Intel to fix a transmit hang under heavy load. This update is the most likely cause for the issues people are seeing. That being said, we have transferred more than 1,000TB of data on RN31x series units trying to reproduce this issue, and have not seen anything like what has been described here. So there is clearly a combination of factors involved here.

We plan to post a test build with that driver update backed out in a few hours, once it passes sanity checks.

TeknoJnky · ‎2017-11-02

I suspect that the main things affecting the issue are:

- bonded/teaming nics (all 4 of mine use lacp layer 3+4 mode, 1gbit)

- dhcp aquired ips. Mine are all static assigned.

I am not sure that if this issue is the same or related to the other thread regarding the resetting of lacp layer mode, but seems both are network/driver related.

Tinyhorns · ‎2017-11-02

@Skywalker Can we help in some way?

Do you need additional information?

Debug-mode?

The current image seems to load alot of different drivers.

[    2.940935] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[    2.941027] e1000e: Intel(R) PRO/1000 Network Driver - 3.3.6-NAPI
[    3.148764] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.3.0-k
[    3.148903] Intel(R) 10GbE PCI Express Linux Network Driver - version 5.2.1

// T

Tinyhorns · ‎2017-11-02

@TeknoJnky Im using static ip, same issue.

amrob2 · ‎2017-11-02

And I have done it both static and DHCP all with the same results.

xbone · ‎2017-11-02

Also experiencing the same problem here with an RN314 after the 6.9.0 update. When a client starts its backups, I am getting hit with similar warnings:

Thu Nov 2 2017 4:19:45   System: Bond interface bond0 has slave interface eth1 offline.
Thu Nov 2 2017 4:19:28   System: Bond interface bond0 has slave interface eth0 offline.
Thu Nov 2 2017 4:17:36   System: Bond interface bond0 has slave interface eth1 offline.
Thu Nov 2 2017 4:14:11   System: Bond interface bond0 has slave interface eth1 offline.
Thu Nov 2 2017 4:12:24   System: Bond interface bond0 has slave interface eth1 offline.
Thu Nov 2 2017 4:11:32   System: Bond interface bond0 has slave interface eth1 offline.

The unit is bonded to a Netgear GS716T switch.

Looking forward to a timely fix along with everyone else!

davidr1 · ‎2017-11-02

@Tinyhorns wrote:
@davidr1 using ssh, you can see what the kernel thinks about bonding.
That requires enabled ssh tho, and Im not sure thats the recomended way.

But, my flapping bond shows up in the default log-view in the webgui.

if you have ssh enabled do the following.
# dmesg -T
dmesg will show the message buffer of the kernel, -T will show you timestamps instead of epoch-time.

// T

File attached. Note that the NAS has just been turned on.

David

stubbsy1 · ‎2017-11-02

@TeknoJnky wrote:
I suspect that the main things affecting the issue are:

- bonded/teaming nics (all 4 of mine use lacp layer 3+4 mode, 1gbit)
- dhcp aquired ips. Mine are all static assigned.

In my case I have a STATIC IP set so the second point is not correct.

As for the frustration angle - I too am feeling the pain, but we need to cut the Netgear developers some slack too. Given this is not impacting all users it will be hardder to replicate. While we'd all like this fixed urgently, tracking down such issues takes time - especially determining the specific factors to trigger it. I'd prefer time be taken to get it right rather than rush an untested fix that could cause other issues.

Here is a summary of my setup:
- RN 314 + EDA 500

- bonded ethernet

- teaming mode IEEE 802.3ad LACP

- layer 2+3

- static IPv4 address

- dynamic DHCP IPv6 address

- connected to Netgear Nighthawk S8000 switch (ports 6 and 7)

This may be unrelated since I updated the switch software at the same time 😞 so have been holding off chasing the issue below:

I am NOT using link aggregation on the 2 matching switch ports because the moment I set aggregation I lose access to the NAS shares and cannot ping the NAS
This means that the ports are bonded but not connected to an aggregated link!

Skywalker · ‎2017-11-02

Thanks for the detailed information. We are now able to duplicate the issue using a Nighthawk S8000 switch. After updating the firmware to the version I mentioned earlier, the problem went away. You can now download 6.9.1-T119 here, and it should resolve the issue. Please confirm if it works for you.

amrob2 · ‎2017-11-02

@Skywalker wrote:
Thanks for the detailed information. We are now able to duplicate the issue using a Nighthawk S8000 switch. After updating the firmware to the version I mentioned earlier, the problem went away. You can now download 6.9.1-T119 here, and it should resolve the issue. Please confirm if it works for you.

Hi @Skywalker

Is that the same image for the RN316 machines? I notice the URL mentions X86, but isn't the 316 processor an AMD chip?

I just want to check before I install and potentially brick my box.

stubbsy1 · ‎2017-11-02

@Skywalker wrote:
Thanks for the detailed information. We are now able to duplicate the issue using a Nighthawk S8000 switch. After updating the firmware to the version I mentioned earlier, the problem went away. You can now download 6.9.1-T119 here, and it should resolve the issue. Please confirm if it works for you.

I've installed the image on my RN314. I also re-enabled link aggregation on ports 6 & 7 of my S8000. While I still can't ping my NAS, I CAN access the shares and all is working fine otherwise. I have copied a 42Gb test file from the NAS to a local drive and have a peak speed of 108 Mb/sec and average speed of 95.7 (my usual speed pre 6.9.0). Copying the same file back is also as pre 6.9.0 (peaks at 75.9, ave 66.7)

Logs are reporting no bond issues. I'd say this looks fixed with the update.

amrob2 · ‎2017-11-02

@Skywalker ignore the question about whether the firmware update is for RN 316's, I saw that someone else had loaded it onto their 314 and I know that the 316 has the same processor as them.

I have loaded the update onto my 316 and put through quite intensive data writes with two laptops copying via a GB network 100GB of data from each laptop to the server while at the same time the server being connected via DLNA on my tv with video streaming.

There were no problems at all, the copy processes completed with no network disconnects, the display on the NAS stayed completely blank and the DLNA streaming continued uninterrupted - so I would say that you have managed to resolve the issue (at least from my perspective).

Thanks to everyone who helped get this fixed, including end users who provided logs to Netgear for investigating

davidr1 · ‎2017-11-02

I have just completed transferring 260GB of files each ranging from 6.6GB to 19.3GB without a problem.

At the same time DLNA was uninterrupted.

It appears that the problem has been solved. Would it be correct that it is 314x specific and if so does that mean that future updates might again be a problem for the 314 design? (It is not the Nighthawk switch as I don't have one - I have the FVS318G router).

Thank you sincerely to everybody at Netgear and in the community who have obviously worked very hard and with the customers to solve this problem.

David

Tinyhorns · ‎2017-11-03

@Skywalker Good morning Sir.

After applying the supplied image, and battering the RN with heavy R/W, I can not reproduce the same error I have seen before.
4 clients doing Timemachine backups, 2 nfs-clients doing md5sum-checks on various files and one iscsi-vmware-initiator doing its thing.

High load expected on the RN, but no signs everywhere of flapping.

I must say I'm very impressed on the speed that you have solved this.
5-6 days (Weekend included) after people starting to notice the issue, you have pinpointed the bug, eliminated it and deployed a fix to the "public".

Good job guys, keep it up.

// T

vikingfmo · ‎2017-11-03

Hi,

Please don't forget the RN214 (ARM processor). Since I upgraded to 6.9.0, I have the message:

Fri Nov 3 2017 13:35:37	
System: Bond interface bond0 has slave interface eth0 offline

I tried to use Static IP or DHCP, same issue.

Logs for eth0:

root@viking-nas:~# dmesg -T | grep "eth0"
[Fri Nov  3 13:34:44 2017] al_eth 0000:00:01.0 eth0: AnnapurnaLabs unified 1Gbe/10Gbe found at mem fe000000, mac addr a0:63:91:9c:1b:fa
[Fri Nov  3 13:34:47 2017] al_eth 0000:00:01.0 eth0: using MSI-X per Queue interrupt mode
[Fri Nov  3 13:34:47 2017] al_eth 0000:00:01.0 eth0: phy[4]: device 8:04, driver Atheros 8035 ethernet
[Fri Nov  3 13:34:47 2017] al_eth 0000:00:01.0 eth0: phy[4]:supported 2ef adv 2ef
[Fri Nov  3 13:34:47 2017] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[Fri Nov  3 13:34:51 2017] al_eth 0000:00:01.0 eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[Fri Nov  3 13:34:51 2017] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[Fri Nov  3 13:35:05 2017] al_eth 0000:00:01.0 eth0: al_eth_down
[Fri Nov  3 13:35:05 2017] bond0: Adding slave eth0
[Fri Nov  3 13:35:05 2017] al_eth 0000:00:01.0 eth0: using MSI-X per Queue interrupt mode
[Fri Nov  3 13:35:05 2017] al_eth 0000:00:01.0 eth0: phy[4]: device 8:04, driver Atheros 8035 ethernet
[Fri Nov  3 13:35:05 2017] al_eth 0000:00:01.0 eth0: phy[4]:supported 2ef adv 2ef
[Fri Nov  3 13:35:05 2017] bond0: Enslaving eth0 as a backup interface with a down link
[Fri Nov  3 13:35:09 2017] al_eth 0000:00:01.0 eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[Fri Nov  3 13:35:09 2017] bond0: link status definitely up for interface eth0, 1000 Mbps full duplex
[Fri Nov  3 13:35:19 2017] NETDEV WATCHDOG: eth0 (al_eth): transmit queue 3 timed out
[Fri Nov  3 13:35:19 2017] al_eth 0000:00:01.0 eth0: al_eth_reset_task restarting interface
[Fri Nov  3 13:35:19 2017] al_eth 0000:00:01.0 eth0: al_eth_down
[Fri Nov  3 13:35:19 2017] al_eth 0000:00:01.0 eth0: free uncompleted tx skb qid 3 idx 0x0
[Fri Nov  3 13:35:19 2017] al_eth 0000:00:01.0 eth0: free uncompleted tx skb qid 3 idx 0x1
[Fri Nov  3 13:35:19 2017] al_eth 0000:00:01.0 eth0: free uncompleted tx skb qid 3 idx 0x2
[Fri Nov  3 13:35:19 2017] al_eth 0000:00:01.0 eth0: free uncompleted tx skb qid 3 idx 0x3
[Fri Nov  3 13:35:19 2017] al_eth 0000:00:01.0 eth0: free uncompleted tx skb qid 3 idx 0x4
[Fri Nov  3 13:35:19 2017] al_eth 0000:00:01.0 eth0: free uncompleted tx skb qid 3 idx 0x5
[Fri Nov  3 13:35:19 2017] al_eth 0000:00:01.0 eth0: free uncompleted tx skb qid 3 idx 0x6
[Fri Nov  3 13:35:19 2017] al_eth 0000:00:01.0 eth0: free uncompleted tx skb qid 3 idx 0x7
[Fri Nov  3 13:35:19 2017] al_eth 0000:00:01.0 eth0: free uncompleted tx skb qid 3 idx 0x8
[Fri Nov  3 13:35:19 2017] al_eth 0000:00:01.0 eth0: free uncompleted tx skb qid 3 idx 0x9
[Fri Nov  3 13:35:19 2017] al_eth 0000:00:01.0 eth0: free uncompleted tx skb qid 3 idx 0xa
[Fri Nov  3 13:35:19 2017] al_eth 0000:00:01.0 eth0: free uncompleted tx skb qid 3 idx 0xb
[Fri Nov  3 13:35:19 2017] al_eth 0000:00:01.0 eth0: free uncompleted tx skb qid 3 idx 0xc
[Fri Nov  3 13:35:19 2017] al_eth 0000:00:01.0 eth0: free uncompleted tx skb qid 3 idx 0xd
[Fri Nov  3 13:35:20 2017] al_eth 0000:00:01.0 eth0: using MSI-X per Queue interrupt mode
[Fri Nov  3 13:35:20 2017] bond0: link status definitely down for interface eth0, disabling it

Logs for eth1:

root@viking-nas:~# dmesg -T | grep "eth1"
[Fri Nov  3 13:34:44 2017] al_eth 0000:00:03.0 eth1: AnnapurnaLabs unified 1Gbe/10Gbe found at mem fe020000, mac addr a0:63:91:9c:1b:fb
[Fri Nov  3 13:34:47 2017] al_eth 0000:00:03.0 eth1: using MSI-X per Queue interrupt mode
[Fri Nov  3 13:34:47 2017] al_eth 0000:00:03.0 eth1: phy[5]: device 18:05, driver Atheros 8035 ethernet
[Fri Nov  3 13:34:47 2017] al_eth 0000:00:03.0 eth1: phy[5]:supported 2ef adv 2ef
[Fri Nov  3 13:34:47 2017] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[Fri Nov  3 13:34:51 2017] al_eth 0000:00:03.0 eth1: Link is Up - 1Gbps/Full - flow control rx/tx
[Fri Nov  3 13:34:51 2017] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[Fri Nov  3 13:35:05 2017] al_eth 0000:00:03.0 eth1: al_eth_down
[Fri Nov  3 13:35:05 2017] bond0: Adding slave eth1
[Fri Nov  3 13:35:05 2017] al_eth 0000:00:03.0 eth1: using MSI-X per Queue interrupt mode
[Fri Nov  3 13:35:05 2017] al_eth 0000:00:03.0 eth1: phy[5]: device 18:05, driver Atheros 8035 ethernet
[Fri Nov  3 13:35:05 2017] al_eth 0000:00:03.0 eth1: phy[5]:supported 2ef adv 2ef
[Fri Nov  3 13:35:05 2017] bond0: Enslaving eth1 as a backup interface with a down link
[Fri Nov  3 13:35:09 2017] al_eth 0000:00:03.0 eth1: Link is Up - 1Gbps/Full - flow control rx/tx
[Fri Nov  3 13:35:09 2017] bond0: link status definitely up for interface eth1, 1000 Mbps full duplex

Firmware version:

root@viking-nas:~# dpkg -l |grep readynasos
ii  readynasos                              6.9.0+4                       armel        ReadyNASOS base system

Is someone had the same issue with a RN214 ? (and solved it ?)

Skywalker · ‎2017-11-03

@vikingfmo, your issue would be completely different from everything else in this thread, as it has nothing to do with Intel's driver. From your logs, your eth0 interface was not working from the beginning of that boot. Does unplugging and replugging the cable fix it? How about rebooting? Do you see any kernel messages about mdio reads failing?

mdgm-ntgr · ‎2017-11-04

davidr1, as we can reproduce the problem, if a newer version of the driver again is made available and if we decide to look into using that we can easily check to see if the problem still exists with that or not.

However there’s the possibility a newer driver version could have other issues.

davidr1 · ‎2017-11-05

Thanks mdgm.

I guess Netgear (and other vendors) are in a no-win situation here as you, as we do, assume drivers provided to you work properly.

I have never met such an animal as bug-free software.

You are right - future drivers (or updates) are taken on faith, not guarantee. I think anybody who expects bug-free softwre deludes themselves.

Anyway, at the end of the day, a sincere thanks for working so hard on this. That has been my experience for several years now - a good record ...

CLHatch · ‎2017-11-10

@Skywalker wrote:
Thanks for the detailed information. We are now able to duplicate the issue using a Nighthawk S8000 switch. After updating the firmware to the version I mentioned earlier, the problem went away. You can now download 6.9.1-T119 here, and it should resolve the issue. Please confirm if it works for you.

This beta did indeed fix the teaming issue in my case as well (LACP to an HP 1820-8G). Unfornately, I had to revert to 1.9.0 due to another issue. I was unable to get Docker to install with this beta, and I've been using Docker. So I had to revert to 6.9.0 for now and live with the network issues.

Skywalker · ‎2017-11-10

The official 6.9.1 Beta 1 has been posted, and should work fine for Docker.