Re: Keep getting NETWORK CHANGED message since upgrading to 6.9.0

amrob2 · ‎2017-10-31

FWIW, this is not showing up on my two RN52x systems. So it might not be that easy to detect (even though it is clearly impacting some users).

But the update should be tested on all NAS types that are not past end of life support. My RN316's are still available to buy, so I would expect Netgear to test all updates on RN316's prior to rolling out a major OS change.

Imagine if the OS update bricked all RN316's due to Netgear not having tested an update on them. That would be embarassing, damaging to their reputation and expensive if every 316 user lost all of their data....

I know that is quite a wild senario I have described there, but it is one that could come true if they don't test prior to going on full public release.

mdgm-ntgr · ‎2017-10-31

I think this is fixed in the latest hot-fix. You could try rebooting your NAS. The hot-fix should be installed within 15 minutes or so after that. If you do nothing (i.e. don't reboot your box) it'll be installed within a week.

We can't test every possible use case or network environment all the time as there's so many, but we do our best. Beta testing is an important supplement to that as user testing helps to cover some network environments that we can't cover. We do appreciate getting feedback on our beta releases as that helps us to make our production releases better for everyone.

We do test every update on all the supported systems.

We do review what we're doing to see where we can do better.

The new beta update channel should mean that more users are aware of beta firmware availability, which is a good thing.

Tinyhorns · ‎2017-10-31

Hi @mdgm-ntgr,

Good to hear, I will check out if my nas will get the hot-fix or not.

Can I force it somehow? apt-get upgrade?

Regarding beta-program, im pretty sure everyone has very important data on ther boxes, and dont really like dataloss dueto bugs in beta.

But, I will graldly beta-test your software, if you send me another NAS with some small disks, and I can do testing on it, no worries 🙂

Livetesting on my data, no thanks 🙂

// T

StephenB · ‎2017-10-31

@amrob2 wrote:

FWIW, this is not showing up on my two RN52x systems. So it might not be that easy to detect (even though it is clearly impacting some users).

But the update should be tested on all NAS types that are not past end of life support. My RN316's are still available to buy, so I would expect Netgear to test all updates on RN316's prior to rolling out a major OS change.

I agree that updates should be tested on OS-6 platforms. I don't know that the problem effects all RN31x platforms (or no RN52x platforms for that matter), since I don't know the root cause.

FWIW, I also don't think it's practical to run manual tests on all OS 6 platforms, so the tests need to be automated (and perhaps targeted at platform-specific variances).

amrob2 · ‎2017-10-31

Hi @mdgm-ntgr - I rebooted my NAS per your instructions over an hour ago. Is there any way that I would know if the hot fix has been applied? Does the NAS need to reboot to apply the hot fix or is it a patch like you get with Windows where you don't always need the machine to reboot after it has installed?

The reason I ask is because I am still getting the exact same error occuring on my box that started with the upgrade to 6.9.0 - so either I have not received the hot fix, or the hot fix does not work.

I notice on the "settings" page of the control panel that the "Update" section gives you the option to select the type of updates you want your box to install. My control panel is set to "Stable", would it need to be on a different setting in order to receive the hot fix?

mdgm-ntgr · ‎2017-10-31

@Tinyhorns wrote:

Livetesting on my data, no thanks

I understand. However making it as easy as possible for those who are willing to run betas to be aware of betas is a good thing.

Those who do test betas are helping those who don’t.

mdgm-ntgr · ‎2017-10-31

@amrob2 wrote:

Hi @mdgm-ntgr - I rebooted my NAS per your instructions over an hour ago. Is there any way that I would know if the hot fix has been applied? Does the NAS need to reboot to apply the hot fix or is it a patch like you get with Windows where you don't always need the machine to reboot after it has installed?

The hot-fix would automatically be installed in the background within about 15 minutes unless network issues prevent it from being downloaded. It’s not a firmware update. You shouldn’t need to reboot afterwards, I think.

You’re looking at the version of readynasos in packages.log

There could well be multiple issues being discussed in this thread which makes things difficult as a reply regarding one issue may be wrongly interpreted as a reply about another.

amrob2 · ‎2017-10-31

Thanks @mdgm-ntgr the packages log shows the following:

ii readynasos 6.9.0+2 amd64 ReadyNASOS base system

Does that mean that the hot fix has been applied or not? If it does, then it has not cured the problem that I am experiencing, which is the one in the opening post on this thread where the display changes to "NETWORK CHANGED", "No IP Address" then "192.168.0.22"

I have noticed that unlike before that I can do copying (reading) large chunks of data (300gb at a time) from the NAS to a different computer on my network without the problem kicking in, but trying to write files of more than 2gb at a time still results in the error.

bedlam1 · ‎2017-10-31

6.9.0+4 is where you should be at, just try another reboot or two and check back after 15-20 minutes

Tinyhorns · ‎2017-10-31

If one wants, its possible to use 'apt-get' to get the firmware in place.

but that requires ssh-access.

# apt-get update && apt-get upgrade readynasos
......
# dpkg -l |grep readynasos
ii  readynasos                     6.9.0+4                    amd64        ReadyNASOS base system

mdgm-ntgr · ‎2017-11-01

That's not the way we'd recommend doing it.

Tinyhorns · ‎2017-11-01

@mdgm-ntgr

So, I did upgrade, and after putting some load on the box.

My issue is still the same.

[Wed Nov  1 08:11:47 2017] e1000e: eth1 NIC Link is Down
[Wed Nov  1 08:11:47 2017] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:11:47 2017] e1000e: eth1 NIC Link is Down
[Wed Nov  1 08:11:47 2017] bond0: link status definitely down for interface eth1, disabling it
[Wed Nov  1 08:11:47 2017] bond0: first active interface up!
[Wed Nov  1 08:11:49 2017] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:11:49 2017] bond0: link status definitely up for interface eth0, 1000 Mbps full duplex
[Wed Nov  1 08:11:49 2017] bond0: first active interface up!
[Wed Nov  1 08:11:50 2017] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:11:51 2017] bond0: link status definitely up for interface eth1, 1000 Mbps full duplex
[Wed Nov  1 08:12:39 2017] e1000e: eth0 NIC Link is Down
[Wed Nov  1 08:12:39 2017] e1000e 0000:01:00.0 eth0: speed changed to 0 for port eth0
[Wed Nov  1 08:12:39 2017] bond0: link status definitely down for interface eth0, disabling it
[Wed Nov  1 08:12:39 2017] bond0: first active interface up!
[Wed Nov  1 08:12:41 2017] e1000e: eth1 NIC Link is Down
[Wed Nov  1 08:12:41 2017] e1000e 0000:04:00.0 eth1: speed changed to 0 for port eth1
[Wed Nov  1 08:12:41 2017] bond0: link status definitely down for interface eth1, disabling it
[Wed Nov  1 08:12:41 2017] bond0: first active interface up!
[Wed Nov  1 08:12:43 2017] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:12:43 2017] bond0: link status definitely up for interface eth0, 1000 Mbps full duplex
[Wed Nov  1 08:12:43 2017] bond0: first active interface up!
[Wed Nov  1 08:12:44 2017] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:12:44 2017] bond0: link status definitely up for interface eth1, 1000 Mbps full duplex
[Wed Nov  1 08:13:35 2017] e1000e: eth0 NIC Link is Down
[Wed Nov  1 08:13:35 2017] e1000e 0000:01:00.0 eth0: speed changed to 0 for port eth0
[Wed Nov  1 08:13:35 2017] bond0: link status definitely down for interface eth0, disabling it
[Wed Nov  1 08:13:35 2017] bond0: first active interface up!
[Wed Nov  1 08:13:38 2017] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:13:38 2017] bond0: link status definitely up for interface eth0, 1000 Mbps full duplex
[Wed Nov  1 08:13:40 2017] e1000e: eth1 NIC Link is Down
[Wed Nov  1 08:13:40 2017] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:13:40 2017] e1000e: eth1 NIC Link is Down
[Wed Nov  1 08:13:40 2017] bond0: link status definitely down for interface eth1, disabling it
[Wed Nov  1 08:13:40 2017] bond0: first active interface up!
[Wed Nov  1 08:13:40 2017] e1000e: eth0 NIC Link is Down
[Wed Nov  1 08:13:40 2017] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:13:40 2017] e1000e: eth0 NIC Link is Down
[Wed Nov  1 08:13:40 2017] bond0: link status definitely down for interface eth0, disabling it
[Wed Nov  1 08:13:40 2017] bond0: first active interface up!
[Wed Nov  1 08:13:43 2017] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:13:43 2017] bond0: link status definitely up for interface eth1, 1000 Mbps full duplex
[Wed Nov  1 08:13:43 2017] bond0: first active interface up!
[Wed Nov  1 08:13:43 2017] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Wed Nov  1 08:13:43 2017] bond0: link status definitely up for interface eth0, 1000 Mbps full duplex

amrob2 · ‎2017-11-01

Thanks @bedlam1 I have shutdown and rebooted three times now (with gaps of several hours between each cycle) but still am only on 6.9.0+2 - although it appears from @Tinyhorns post that the hot fix he received hasn't cured the problem anyway

amrob2 · ‎2017-11-01

That is most frustrating @Tinyhorns I still have not received the hot fix despite powering down and rebooting three times in the last 24 hours since we were told a hot fix is available, so I cannot even confirm if the fix would resume normal functionality on my 316

bugsplat · ‎2017-11-01

Rebooted both my RN314s last night, and I immediately check the ReadyNAS os to see where they were at ... lo and behold, they updated to the +4 hot-fixed immediately after reboot.

I'll do some heavy-duty file transfers when I get home tonight to see If I still have the problem (subject of this thread).

amrob2 · ‎2017-11-01

Well that is strange where everyone else is picking up the hot fix, but my 316 has not. It has full network connectivity, it can see the internet (it happily sends out email alerts when I reboot/shut it down). So I guess I am going to have to wait until next week until everyone receives it (per @mdgm-ntgr post on the subject yesterday)

StephenB · ‎2017-11-01

Perhaps PM mdgm, and ask if he can check the logs. There might be something there that explains why the hot-fix isn't being installed.

davidr1 · ‎2017-11-01

This is affecting me too - only since upgrading to OS 6.9.0

NAS314 with 3 x 4TB drives

Static IP address on NAS and reserved in router

The patch appears to have uploaded - package.log shows:

ii readynasos 6.9.0+4 amd64 ReadyNASOS base system

I don't know which log has the bond details ??

When transferring large file to the NAS or in a backup using Remote Rsync Server to an an attached USB3 drive (4TB):

'Network changed' is displayed periodically followed after abut 2 second the IP address - which is the correct IP address.

amrob2 · ‎2017-11-01

That is the exact same problem as what I am experiencing, but my box has not taken on the hot fix for some reason.

But it would appear that 6.9.0+4 has not resolved the problem from what you have said anyway

davidr1 · ‎2017-11-01

I am loathe to trust transfers and backups with this problem.

Checking the integrity of the transfers or backups (which are a transfer anyway) would be a mammoth task.

Or am I ignorant of a setting which can be used to md5sum(slow!!) ,or whatever, the original and the transferred files?

Is this relevant - there is nothing wrong with the cable connection at either end nor the usb3 cable:

Nov 02 09:19:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=configuration
Nov 02 09:19:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=disconnect
Nov 02 09:19:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=idle
Nov 02 09:19:23 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable removed
Nov 02 09:19:26 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable changed: State=ready
Nov 02 09:20:01 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=configuration
Nov 02 09:20:01 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=disconnect
Nov 02 09:20:01 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=idle
Nov 02 09:20:01 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable removed
Nov 02 09:20:03 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable changed: State=ready
Nov 02 09:20:08 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=configuration
Nov 02 09:20:08 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=disconnect
Nov 02 09:20:08 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=idle
Nov 02 09:20:09 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable removed
Nov 02 09:20:14 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable changed: State=ready
Nov 02 09:20:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=configuration
Nov 02 09:20:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=disconnect
Nov 02 09:20:23 NAS314 readynasd[2981]: PropertyChanged (/ethernet_08bd43f62740_cable): State=idle
Nov 02 09:20:23 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable removed
Nov 02 09:20:29 NAS314 readynasd[2981]: Service ethernet_08bd43f62740_cable changed: State=ready

bugsplat · ‎2017-11-01

@amrob2 wrote:
That is the exact same problem as what I am experiencing, but my box has not taken on the hot fix for some reason.

But it would appear that 6.9.0+4 has not resolved the problem from what you have said anyway

Yup, still hosed ... tried copying over a 10GB file tonight --- total disaster with 6.9.0+4

Tinyhorns · ‎2017-11-01

@davidr1 using ssh, you can see what the kernel thinks about bonding.

That requires enabled ssh tho, and Im not sure thats the recomended way.

But, my flapping bond shows up in the default log-view in the webgui.

if you have ssh enabled do the following.

# dmesg -T

dmesg will show the message buffer of the kernel, -T will show you timestamps instead of epoch-time.

// T

stubbsy1 · ‎2017-11-01

I too have this identical issue (Bond interface bond0 has slave interface eth0 offline) since updating to 6.9.0 and as others have said it seems to occur when moving multi Gb files. I also have the hotifx installed and am at ReadyNAS OS 6.9.0+4. So the hotfix did not fix the issue

evan2 · ‎2017-11-02

I reproduced the issue on my RN314, don't see the issue on RN628X.

but file transfer is still successfully when system log show interface offline,

the issue should be exist on older FW,

6.9.0 added fail log if there is enterface offline in bonding, so 6.9.0 can see the offline issue,

I filed a bug to track the issue.

Tinyhorns · ‎2017-11-02

@evan2 Hi,

I agree that the filetransfer do work, but, it will take longer time, and there is a lot of tcp retransmiting going on and dropped packets.

RX packets:87859001 errors:0 dropped:30499 overruns:0 frame:0

IO-wait on the connected clients are sky-high as well, as expected.

Thats how I did notice it in the first place.

But, I do disagree that it this bug exist on older firmware, I say this bug was introduced in 6.9.0.

I do continuous moninotring of the connected clients, and it was a clear spike in IO-wait/Load on the box.

Please see the graph attached.

Guess when I did the FW-upgrade to 6.9.0?

// T