ReadyNAS Ultra 4 is slowing down my entire network LAN & WAN

[IseWise]

Ive owned a ReadyNAS Ultra 4+ for several years now, with little to no problems, however I'm now having a very strange issue.

Every time my NAS is connected to the network, the entire network begins to crawl. And I do mean the entire network, both LAN and WAN. Page loads will slow down and even fail to load, accessing my mapped drives slows down, even wifi devices will slow down.
I am positive it's the NAS causing the slow down because I can unplug the 2 cables and the network returns to optimal performance.

My network:
Motorola Surfboard SB6141
Netgear R7000
Trendnet TEG-S80g siwtch <--- NAS is connected to switch with 2 Cat5e cables.

NAS is on the latest firmware, both network interfaces have static ip set inside frontview.

I tried trading the Trendnet switch for a Linksys SE2500 and that didn't work. I have purchased at the time of this writing a bunch of cat6 cables from Monoprice.com to replace every cable I can to see if that will help performance.

I have found a couple other threads that sound similar to the problem I'm experiencing but no answers or solutions:
http://www.readynas.com/forum/viewtopic.php?f=21&t=76765
http://www.readynas.com/forum/viewtopic.php?f=21&t=56182

Please help me out.

[StephenB]

What happens if you unplug one of the two ethernet cables to the NAS (leaving the other connected).

[IseWise]

Ive disconnected one of the ethernet cables and my network seems to be working properly.

However, won't this lower my read/write performance to the NAS?

[mdgm-ntgr]

Don't think so with that model or not that much.

[StephenB]

IseWise wrote:
Ive disconnected one of the ethernet cables and my network seems to be working properly.

That suggests the problem was some form of an ethernet/bridging loop. Generally when you connect two cables that way you want to use a link aggregation protocol (for instance LACP). I think you might not have been doing that. LACP needs the switch to support it to.

IseWise wrote:
However, won't this lower my read/write performance to the NAS?

Not for single user performance. If you have simultaneous users it might reduce performance a bit.

[IseWise]

StephenB wrote:
IseWise wrote: Ive disconnected one of the ethernet cables and my network seems to be working properly. That suggests the problem was some form of an ethernet/bridging loop. Generally when you connect two cables that way you want to use a link aggregation protocol (for instance LACP). I think you might not have been doing that. LACP needs the switch to support it to.

Perhaps, I spoke too soon. It appears that the problems have returned. I tried connectly the NAS directly to the router and that didnt seem to help either. Ive had both ethernet connections connected before for quite some time to the same switch and haven't had issues before.

IseWise wrote:
However, won't this lower my read/write performance to the NAS?

StephenB wrote:
Not for single user performance. If you have simultaneous users it might reduce performance a bit.

At this point, Ill take a small performance hit for reliability on my network.

Im wondering if this problem has anything to do with me recently switching ISPs. I recently upgraded from Centurylink DSL to Comcast cable.

I checked Network > Global Settings > DNS settings were empty. I filled them in with my DNS servers provided by my ISP, but I am still have slow down issues. A have ftp running on the NAS. Could this be related?

[StephenB]

IseWise wrote:
Im wondering if this problem has anything to do with me recently switching ISPs. I recently upgraded from Centurylink DSL to Comcast cable.

It seems unlikely that change would result in the NAS slowing down your entire LAN.

IseWise wrote:
...I checked Network > Global Settings > DNS settings were empty. I filled them in with my DNS servers provided by my ISP, but I am still have slow down issues

If the NAS is set to use DHCP, then those fields should be filled in by the NAS. Mine are set to the gateway address (though router behavior varies on this). Do you have jumbo frames enabled? If so, try turning them off. For Comcast the MTU should be set to 1500.

When you click on "show errors" are you seeing a lot of errors, collisions, or overruns?

IseWise wrote:
...A have ftp running on the NAS. Could this be related?

I don't see how.

Are you forwarding any ports in your router to the NAS?

[StephenB]

BTW, have you tried submitting a support request via support.netgear.com (using email)?

[FiremedicJM]

IseWise wrote:

Ive owned a ReadyNAS Ultra 4+ for several years now, with little to no problems, however I'm now having a very strange issue. Every time my NAS is connected to the network, the entire network begins to crawl. And I do mean the entire network, both LAN and WAN. Page loads will slow down and even fail to load, accessing my mapped drives slows down, even wifi devices will slow down. I am positive it's the NAS causing the slow down because I can unplug the 2 cables and the network returns to optimal performance. My network: Motorola Surfboard SB6141 Netgear R7000 Trendnet TEG-S80g siwtch <--- NAS is connected to switch with 2 Cat5e cables. NAS is on the latest firmware, both network interfaces have static ip set inside frontview. I tried trading the Trendnet switch for a Linksys SE2500 and that didn't work. I have purchased at the time of this writing a bunch of cat6 cables from Monoprice.com to replace every cable I can to see if that will help performance. I have found a couple other threads that sound similar to the problem I'm experiencing but no answers or solutions: http://www.readynas.com/forum/viewtopic.php?f=21&t=76765 http://www.readynas.com/forum/viewtopic.php?f=21&t=56182 Please help me out.

I am having the same problem..I have an Ultra 6 plus that has been working flawlessly for the last few years. The other day I noticed that my internet connection was intermittently down with 50% packet loss. I tracked the problem to the NAS. I unplugged the network cables and everything returned to normal. I rebooted the NAS and the problem remained. I tried each network port with different internal and external IP addresses(different gateways I have external and internal networks) and got the same results each time(Severe packet loss across the whole network) If I put in an internal static and leave the gateway blank then the NAS works with no packet loss. I can access it internally but of course not from the internet.

[StephenB]

IseWise - can you check for packet loss stats on your local LAN?

[IseWise]

I do have jumbo frames enabled. I will try disabling them. After disabling jumbo frames, the MTU defaults to 1500. Ive checked the errors page and I see 32 RX dropped. After disabling the jumbo frames and restarting 1 Rx dropped after about an hour on.

I recently realized that I am successfully pulling in a IPv6 address. In my router settings I have set up a DHCP reservation for the NAS.

The global settings are grayed out, but the DNS 1 is a long complicated IPv6 address and DNS 2 is the router gateway.

Ive disabled FTP for the time being and disabled forwarded port 21 for ftp.

I have not yet tried submitting a support request. Hopefully, I can fix it before then.

I tried the above settings for about an hour, then it went back to slowing down the entire network and disconnecting me from the internet.

[IseWise]

StephenB wrote:
IseWise - can you check for packet loss stats on your local LAN?

I checked my router and it displays Rx and Tx packets as well as collistions. No collisions across all interfaces.

[StephenB]

Can you try using the blocking controls in the R7000 to prevent any internet access for the NAS? Go into the "Advanced" tab of the router and choose "security", then "Block Services" Add a new service, specifying "ALL" in the pulldown and the NAS IP address. Set the schedule to "always", and then reboot the router. See if the performance symptoms change. I don't think this will impact local NAS access (though I haven't tried it either).

Don't block all IP addresses, just the NAS.

[StephenB]

FYI, I suspect that your NAS might also have gotten hacked, since the signature is so similar to FiremedicJM's. The timing (with Shellshock) is also about right.

[IseWise]

StephenB wrote:

Can you try using the blocking controls in the R7000 to prevent any internet access for the NAS? Go into the "Advanced" tab of the router and choose "security", then "Block Services" Add a new service, specifying "ALL" in the pulldown and the NAS IP address. Set the schedule to "always", and then reboot the router. See if the performance symptoms change. I don't think this will impact local NAS access (though I haven't tried it either). Don't block all IP addresses, just the NAS.

This seemed to do the trick. Been working fine. Ill have to find another way create a ftp server though.

[IseWise]

StephenB wrote:
FYI, I suspect that your NAS might also have gotten hacked, since the signature is so similar to FiremedicJM's. The timing (with Shellshock) is also about right.

How was it hacked and what can I do about fixing it?

[mdgm-ntgr]

Do you forward ports from the web to your NAS? If you don't need to forward ports you shouldn't.

What version of RAIDiator are you running?

The solution would be to backup your data, do a factory default (wipes all data, settings, everything), update to 4.2.27 beta (addresses ShellShock), then put your data back on.

[IseWise]

mdgm wrote:
Do you forward ports from the web to your NAS? If you don't need to forward ports you shouldn't.

I don't forward any ports explicitly to the NAS. I have Upnp enabled on my router howerver.

mdgm wrote:
What version of RAIDiator are you running?

4.2.26

mdgm wrote:
The solution would be to backup your data, do a factory default (wipes all data, settings, everything), update to 4.2.27 beta (addresses ShellShock), then put your data back on.

Ill have to prepare for that. Right now the instruction StephenB provided is working for me. Is there another way perhaps other than fully wiping it to fix the hack?

[mdgm-ntgr]

Could possibly wipe the OS partition and extract the OS onto that again, but would need remote access to your system to do that.

A factory default (wipes all data, settings, everything) would be best.

[IseWise]

mdgm wrote:
Could possibly wipe the OS partition and extract the OS onto that again, but would need remote access to your system to do that.

Is this the same as Reinstalling the OS via the Boot menu when the NAS is booting up? Because Ive tried that and it didnt work. By remote access do you mean SSH?

mdgm wrote:
A factory default (wipes all data, settings, everything) would be best.

What about updating to 4.2.27 beta right now?

[mdgm-ntgr]

IseWise wrote:
Is this the same as Reinstalling the OS via the Boot menu when the NAS is booting up? Because Ive tried that and it didnt work. By remote access do you mean SSH?

No, I mean remote access at a lower level.

IseWise wrote:
What about updating to 4.2.27 beta right now?

That would reduce the likelihood of being hacked in the future, but it is not likely to disinfect the OS partition of a system that has already been hacked.

[IseWise]

As an update, by preemptively blocking the blocks as instructed by StephenB, I was able to successfully reconnect the second ethernet jack. I tried enabling jumbo frames but that didn't seem to help, it didnt completely destroy the network as before, but accessing the NAS was slow. I think the jumbo frame issue is unrelated to the hacking matter and might be more of a cabling and hardware issue on my end.

Additionally, I viewed my routers "Block services" page after StephenB's instructions and noticed it was blocking access to ports that it categorized for use by the programs "Age of Empires" and "AIM". Neither are programs that I have installed and certainly not installed on my NAS. As of this post right now, I havent seen it reappear but Ill update if I do. I thought I would just mention that to see if it helps diagnose the problem of this hack.

[StephenB]

Of course there's no easy way to tell what the hack was doing. But given the amount of traffic, you should assume the your files were being uploaded, and take whatever precautions are needed for identity theft. I'd back up the NAS, zero the disks, and do a clean install (upgrading to 4.2.27 T5). Then restore the data from backup. It isn't clear that an OS reinstall is enough (though it might be).

If that simply isn't practical, you should still back up the NAS before you attempt the OS reinstall, since the reinstall might not be able to fully resolve the hack (and it is conceivable that you might end up bricked as a result).

On the services - not sure why the router is blocking services by default (my R6300v2 is not), but programs don't generally get exclusive rights to internet ports. A few older and very common programs (web browsers, ftp, email) are allocated their own ports by the internet authority people. But not "age of empires" and probably not AIM - they use ports, but don't "own" them. That long list of services was reverse engineered by Netgear, and is mostly a convenience.

So internet ports are often used for multiple services, and there might be other reasons to block those ports that are not apparent in the service name. In any event, blocking ports that aren't supposed to be used is a conservative choice - the catch is that it is often difficult to know what ports are used by the applications you install.

[StephenB]

BTW, manually setting your gateway IP address to 0.0.0.0 in the NAS might also improve performance - the network being slow indicates that the NAS is attempting to communicate outbound, and it might be succeeding to some degree.

As mentioned the issue here is not your network speed. Your network is slow because the NAS is almost certainly compromised, and it is generating a flood of network traffic. The big issue is that you don't know what it is sending.

[FiremedicJM]

OS reinstall does not work, Hack remains active after doing this from the boot menu.