AEBS: Bad Packets Unrecovered TCP Retransmits NV and NV+

[paikinator_yaho]

Bad Packets Unrecovered TCP Retransmits TCP Retransmits w ReadyNAS NV NV+ and Airport Extreme Base Station 5th Generation

I got an Airport Extreme Base Station (AEBS), 5th generation this past November to bring my entire network up to gigE standards and extend my wireless range a bit. I have really liked the AEBS 5th Gen overall since I got it 3 months ago. It was easy to setup and update and things have been noticeably faster. I can also easily take my netbook, laptop, and iPad out to my shop some 80 feet away from my AEBS and still surf rather well wirelessly too. I couldn't do this even with my previous D-Link wireless-n setup.

In December I picked up an old ReadyNAS NV+ to go along with my old ReadyNAS NV. When I finally started to setup the new network for my business in February, I noticed in the logs that I was getting a large numbers of bad packets, TCP retransmits, and unrecovered TCP retransmits. Previously, I had never had errors of any sort with any older network setups using linksys, dlink and netgear routers during the last 6 years.

Eventually cables, switches, and routers can go bad which results in the errors I have been getting so I went through the "Troubleshoot My Performance Problem/Check for Network Errors" section of the link below to try and figure out what was causing my problem.
(http://www.readynas.com/?p=310#Troubleshoot)

My Initial Setup:
Airport Extreme Base Station (AEBS):
- 1 incoming Cat5e from Motorola Surfboard 6120,
- 2 Cat5e connections to my wifes work network with her business computer and VOIP phone.
- 1 Cat 5e interconnecting the AEBS to my wired network on the HP Procurve Switch.

The AEBS manages addresses via DHCP on my wired network for all of my devices on the
HP Procurve 14008G Switch:
- 1 to NV
- 1 NV+
- 1 MacMini 2010
- 1 MacMini 2005
- 1 networked Epson R-3000 printer


1. Simple things first. Since I have always made all my own Cat 5e cable I figured I'd better get some better(?) cabling strait away. I went ordered new Cat 6 and Cat 6a cables to connect the NV, NV+, and my main Mac to the switch and connect the switch to the AEBS. The errors kept coming.

2. Then I separately connected each NAS unit directly to my computer by setting up the static IP on my computer first. Once I reconnected each ReadyNAS with the computer directly they defaulted to a static IP. Neither NAS had errors.
(http://sphardy.com/web/readynas/how-to-direct-connect-to-your-readynas/)

3. Errors were coming from either a bad Procurve 1400 switch or a bad AEBS. I hooked everything directly to my Airport Extreme and got lots of errors.

4. I decided to check the switch also. I altered each ReadyNAS's default static IP to one of my choosing individually. Then I hooked both ReadyNAS units to my HP Procurve 1400 switch. I hooked the switch directly to my computer and got no errors from either NAS. I continue to get no errors the next day.

5. When I first ran my tests I had Jumbo Frames turned Off. Currently, Jumbo Frames are On and there are still no errors and seems to be no slowness either.

Conclusion:
The Airport Extreme is the bottleneck causing the errors in my network.

Perhaps there is something I can alter which will rid me of the errors, but this type of error makes the AE seem to be pretty shoddy. A newer state-of-the-art router should not be bottlenecking my rather archaic slow network devices. Apparently, the 4th Gen AEBS's couldn't do Jumbo Frames, but this current model is supposed to, but I see no setting for changing the MTU.

Questions:
What could be causing the bottleneck?
Do I have a bad router?
How could I analyze this problem?
Has anyone else had similar issues and if so how were they resolved?

Current Setup:
All of my computers and NAS devices are hooked to my Procurve Switch each with its own Static IP.
The switch is working flawlessly with no Errors.
The network printer is now wireless and connected via the AEBS.
I am using wireless-n to connect to internet via the AEBS on my main machine.
I am sharing the internet connection with the other Mac Mini on the switch and any other machine I plug into the switch (not ideal).

Problem is:
I am not a systems administrator and don't really want to tinker with setting up Static IP's for the machines I hook and unhook to the switch.
I want the AEBS to manage addresses via DHCP so everything accessing my router will mindlessly be able to access all things on my network.
Can the AEBS be made to not produce errors or do I just have a lemon.

[StephenB]

When you are running wired, all your traffic stays on the ProCurve, it should never touch the AEBS. Your wife's business network would send traffic through the AEBS, internet traffec of course goes through the AEBS, but the wired connections to the ReadyNAS should not. Are you seeing these problems with purely wired connections? Or are you only seeing them on wireless?

BTW, jumbo frames can only work on the wired network. They won't work on 802.11n. You can probe the MTU size with ping if you want to diagnose it. Here is one how-to link: http://muzso.hu/2009/05/17/how-to-deter ... icmp-pings

I am sharing the internet connection with the other Mac Mini on the switch and any other machine I plug into the switch (not ideal).

I don't think this is an issue at all. Everything becomes shared when it hits the surfboard, as long as the local connections are faster they will not create a bottleneck. This appears to be the case for you. The wired connection between the AEBS and the ProCurve is gigabit correct? If so, it is 10-100x faster than your broadband.

[paikinator_yaho]

All of my network via static assigned IP's works on the Procurve.
All of my network isolated on the AEBS induces lots errors. I am not sure why this is.
Also… when I have my network all on the Procurve and I wire it to the AEBS via cat6a I get errors if the AEBS uses DHCP to assign addresses to my network devices on the procurve. This is what I want the AEBS to do so it is a problem for me. I don't know why the AEBS would cause errors when it is wired to my network on the procurve. Baffling.

Another person has told be that there AEBS does not give them errors with their ReadyNAS's attached to it. I am trying to understand if there is some setting or thing I can do to eliminate the errors or is this AEBS just a lemon.

Currently I can't wire the procurve together with the AEBS. I can only get to the AEBS by sharing my connection via the wireless-n from my main computer.

-------------------------

The info on the jumbo frames is great … thanks.
Normally I run my video_TS folders onto my main computer from my movie NAS and pipe the results to my HDTV.
But I suppose once I jailbreak it and try to do a similar thing with my Apple TV there will be a problem.
I will probably have to set Jumbo Frames to off.

[StephenB]

paikinator@yahoo.com wrote:
… when I have my network all on the Procurve and I wire it to the AEBS via cat6a I get errors if the AEBS uses DHCP to assign addresses to my network devices on the procurve. This is what I want the AEBS to do so it is a problem for me. I don't know why the AEBS would cause errors when it is wired to my network on the procurve. Baffling…

I think this is an important clue. Maybe try different cables, or using a different switch port on both devices. This should work, the fact that it fails suggests something is either broken in the switch or the router, or there is an incompability.

I would also look at the ethernet configuration for the switch port in the AEBS. Perhaps as an experiment set it manually to 100 mbits, and see if it works better.

[paikinator_yaho]

It is all new cable. I have lots of extra new cable I have been trying and the only time there is a problem is when I connect the switch to the router. I have connected it using various ports on the switch and various ports on the router as well. With all hookups with the router I start getting Bad Packets and Unrecovered TCP retransmits from the NAS devices.

The switch has never been a problem before with my other D-Link gigabit router I never got any errors with it, but the router itself was rather shoddy and slow and started dropping us off of the internet which is why I went ahead and got the AEBS. The AEBS has been rock solid and very very fast compared to the D-link. My only problem with it is that I can't have any ReadyNAS units attached to it without getting a bunch of errors.

Perhaps the idea of an incompatibility could be, but I don't really understand why this would be. I thought maybe there was a setting or way of changing something that might make a difference. I suppose I will have to take it in for a replacement and hope the incompatibility with the Procurve is not happening.

I have tried to set the NAS devices to 100mbits but that has not resulted in anything but errors as well when connected to the AEBS.

[paikinator_yaho]

I think your idea of something broken on the AEBS could be. That it seems to me that nothing is broken on any of the ports of the Procurve. Multiple different hookups have all been error free, that is until I connect with the AEBS.

[StephenB]

paikinator@yahoo.com wrote:
I have tried to set the NAS devices to 100mbits but that has not resulted in anything but errors as well when connected to the AEBS.

Sorry I was not clear. I meant try to set the AEBS connection to the switch to 100 mbits. You should be able to do this on the AEBS side.

[PapaBear1]

If you still have your old router, you could try putting it back in the network, temporarily replacing the AE. I did something like this when I replaced my old Netgear FS105 with a D-Link DGS-2205 switch and as long as I only had my old NV+, all was good. When I added an NVX, I could copy files to the NVX but not retrieve them. After doing a direct connection (using the same S.P. Hardy link you posted) I determined that the NVX was fine, so there was another suspect afoot. Replacing the new switch by temporarily reverting to my old slow FS105 solved all my problems (except for speed) while I awaited my new Netgear GS108 switch.

You might also temporarily disconnect your wife's computer and VOIP phone when it would be the least inconvenient for her. Throughout your testing, you have not commented on them, and while not directly in your sub-network, they are attached to the AE and there could be something in them that is affecting it. At this point all possibilities should be examined.

[paikinator_yaho]

Fixed I think.

Sorry for the delayed response. I have been traveling followed by being sick for the past couple weeks.

As it turns out I think my entire problem was all due to BAD CABLE.
I had some bad cable to begin with and so ordered a bunch of Cat 6 and Cat6a cable from Monoprice since I have always had great luck with them before. This time I ordered a 1' Cat6a, a 5' Cat6a, a 10' Cat6a, and 2 10'Cat 6 cables from them.
All 3 10' lengths turned out to produce bad packets and TCP Retransmits no matter how they were hooked up.

After wasting a weeks time trying to reconnect everything in different ways, I started from scratch and looked more specifically at the cables with greater depth. They weren't producing massive errors like my old bad cables. In fact it took a while for the Cat6a cable to produce errors…. I had to work at copying, transferrring files, playing various videos and audio files before I noticed it started spitting out errors on each of my servers (a ReadyNAS and then the ReadyNAS+) directly connected to the mac mini 2010 .

I have never had a bad cable from them before and got 3 this time.

So far it seems that the 1' Cat6a and the 5' Cat6a are performing well without any errors. It does make me sick to click on the Show Errors box as of now for sheer terror of seeing an error. But so far so good.

Anyhow…. it cost me a lot of time to figure this out.
I thought the AEBS or the Switch might be bad, but in the end it was simply the brand new cable.
So I don't need to change out my AEBS or my Procurve Switch.
I will need to pick up some new cables somewhere else since my packaging for the cables got tossed a bit ago.

-----------------------------
Here is my current setup:

Mac Mini 2010
HP Procurve 1400 8G switch
AEBS 5th Generation
ReadyNAS NV
ReadyNAS NV+

The mini is connected via the 1' Cat 6a cable to the HP Switch
The NV+ is connected via an apple Cat5e to the HP Switch
The NV is connected via the 5' Cat6a cable to the HP Switch

Mac Configuration
Ethernet Configured using DHCP with manual address; Static IP for the Network. (192.168.168.*)
Hardware - configured: automatic; speed 1000baseT; duplex: full duplex,flow control; MTU 1500


Wifi on mac mini enabled to connect via wireless N to the AEBS for internet connectivity and to share the internet connection with my wife's work network too.

NV and NV+
Both don't have jumbo frames enabled since I guess it isn't needed and doesn't help in wireless N
Both running Auto-negotiation at 1000baseT; MTU 1500
Both have a static IP for the Network (192.168.168.*)


Lesson learned… it can be the simplest thing leading to errors including new out of the package cable.

[StephenB]

Thanks for the update. Although annoying (and hard to track down), replacing the cables is inexpensive. You might contact the seller, even if you don't have the packaging and are beyond the return period.

[paikinator_yaho]

Drat.... went to load a giant photo shoot on my NAS NV.
TCP retransmits started showing up again.
I played a movie off of the NV+ yesterday and checked for errors at that time. No errors on the NV or NV+.
I started loading my photos today and all of the sudden 8 retransmits and 7 unrecovered. On both the NV and the NV+ ????

As I kept going loading more on the NV more retransmits on the NV and more unrecovered errors on the NV alone. No more bad packets though.
Eventually things seemed to drop connection and my Lightroom import froze.
I had to reboot the Procurve 1400 since neither NAS could be accessed and then was able to complete the import.
I am at a loss.
Could this be the switch then?
I have noticed that periodically over the last month and 1/2 that the procurve has needed to be rebooted before I could see or access the NAS devices again.

[StephenB]

sounds like it could be the switch

[paikinator_yaho]

I contacted HP about the switch being an issue.
The tech said that it was likely to be some sort of a loop that was causing the switch to shut down, but since I am not an Network admin and the procurve 1400 8g switch is unmanaged they would ship me another switch. Yeah for the procurve lifetime warranty.
The tech said that if the problem persists then it would be a loop in the network.

What would a loop in the network mean and how would I go about finding such a thing?

[StephenB]

A loop is a circular path, so it can only happen with devices that have multiple ethernet connections. For instance if you have two ethernet cables between the switch and the router.

[paikinator_yaho]

Ah... I asked the tech and he said as much, but then he seemed to think that the likely culprit is my NAS devices????
They don't seem to have the issue when direct connected to my Mac, but I would probably need to dump a bunch of stuff on them to max them out.
Last time when I tested them I dumped a 13 gig folder on each of them and didn't experience any errors, and I played some media files and dumped big files on them to see if I could cause a choke, but didn't have a problem.
Could there be a problem with the 2 NAS devices running on the same switch? I wouldn't think so since they both have separate static IP's

Anyhow HP is sending me a new switch except they didn't have any of the 1410 8G's on hand so they are shipping me an 1800 8G which is a managed switch.
Hopefully, this will do the trick and my error issues will be solved.

[StephenB]

paikinator@yahoo.com wrote:
Ah... I asked the tech and he said as much, but then he seemed to think that the likely culprit is my NAS devices????

That is only possible if the NAS has two lan ports connected. (It would also need to be bridging between them.)

[paikinator_yaho]

I think my problem is finally solved. Not the AEBS. Bad HP Procurve 1400-8G switch.
I have been running for 6 days with no errors.
Originally I do believe I had some bad cables which were contributing to my Bad Packets. Once I got the Cables all straitened out, the bad packets went away.
But then I had TCP retransmit problems. Now with the new switch, I get no more TCP retransmit errors.

Hewlett-Packard shipped replacement unit for my switch for free under their Procurve Lifetime Warranty. Awesome!

They didn't have any of the 1400 series on hand and so they shipped me a Procurve 1800-8G that looks to have been used, but is working beautifully. It is the managed version of what I had and would likely help me diagnose my problems more easily if something were to crop up in the future.

Thanks for all of the help.

[paikinator_yaho]

Drat... I was able to induce more TCP retransmit errors even with the new switch.
It has been pretty stable, but I went to import a large number of photos from a shoot and at the same time I started moving some other big files.
TCP retransmits happened. I got about 8, but none failed upon retransmission... Then I got another 8 with 7 errors.
I guess I am at a loss.
Cables are all new Cat6 or Cat 6a so no new bad packets. Switch is a replacement for the other one which seemed to produce consistent errors. The errors went away, but I was able to choke the switch and it hung and I had to reboot it.

So I am guessing that something must not be playing nicely with my NAS devices or my NAS devices have a problem of some sort.
What could be wrong and how would I test it?
If I tread lightly and don't do too much at the same time on either NAS they seem to not produce TCP retransmit errors.
Perhaps this is what I need to do is deal with the flaws in the system. I can't really afford to replace everything right now. Boo.

Anyhow any suggestions or tips from anyone would be great. I will try limping along for now.
I don't believe the TCP retransmits will affect my file integrity if I am correct.
Does anybody see a problem with living with the problem? I am getting tired of working on it all.
Will I risk losing data? Could this cause me significant headaches?

[StephenB]

TCP retransmits will not hurt file integrity, so you can rest easy on that one. They will slow transfer speeds, but occasional retransmits are probably something you can live with.

You might look at the switch stats - keeping in mind that they generally report from the switch perspective (tx on the switch is rx on the NAS).

One possible cause is buffer overrun in the switch - since you were pushing a lot of data when you got the error. Look to see if 802.3x flow control is enabled on the switch. Also, if there are QOS settings, you can possibly prioritize traffic to/from the NAS.

Another possibility (generally rare these days) is that the switch thinks the link is half-duplex, but the NAS or PC thinks it is full duplex. If that is the case, you can usually manually override the setting on the switch.