NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
[18373207] Duo Slow NFS Reads, Jumbo Frames and TCP window
My ReadyNAS Duo: RAIDiator 4.1.8 Two 2 TB Seagates, RAID 1 (Mirrored) The Duo shares movies to my xbmc server (ubuntu 11.10) via two nfs mounts. I am troubleshooting a problem with intermittant ve...
Sorry for not being more complete on the process of installing tcpdump. I really apprecaiate you taking the time to do this.
The first problem with Jumbo Frames is that they are not defined in any IETF (Internet Engineeding Task Force) standards document. The generally accepted (defacto-standard) Jumbo Frame size is 9000 bytes or less. Your NV+ is indeed running what is commonly referred to a Super Jumbo Frames (SJF). Read about them here: http://en.wikipedia.org/wiki/Jumbo_frame
TCP keeps the data in order by use of SEQuence numbers and ACKnowlegement numbers. The ACK number in a reply packet indicates to the NV+ that the client successfully recieved the data packet with a matching SEQ number. The TCP WINdow is the number of bytes that the sender of the packet (client) is willing to receive before it's buffers are full. This tells the NV+ how much data (number of packets X size of TCP data) to send before waiting for an ACK. There is also a TCP option called SelectiveACK, or SACK. This allows for a case where a packet is lost, or the connection is interrupted where a cllient can ACK a portion of the data sent, making it unnecessary to resend the entire stream of data packets. Only complete packets can be ACK'd, since all inbound packets must pass a checksum test before they are accepted.
When I look at the start of the trace, packets 1,2 and 3, it appears that the NV+, client and intermediate network devices are all handling the SJF. The NV+ sends a single Ethernet frame with 28735 bytes of data. The client replies with an ACKnowledgement for the entire data packet. Later we will see why this data probably arrived at the client as 19 separate 1500 MTU IP packets, with the client sending only one ACK for the last data packet.
As I look farther into the trace, things get interesting. In packet 50, the client sends a read request, with ACK for all previous data from the NV+, and advertises a TCP WIN of 65535 bytes. The NV+ responds with packets 51 (42,340 bytes) 52 (19,163 bytes), almost completly filling the client TCP WIN.
You can also see that the ACK numbers in packets 53, 54 and 55 do not match any of the SEQ numbers of the two NV+ data packets. Since only complete packets can pass checksum (and get ACK), we know with absolute certainty that the NV+ data packets have been split up by an intermediate network device (gigabit switch), and those new packets were recieved intact by the client.
In the above calculation, the value of 1448 represents the TCP payload of a 1500 MTU Ethernet packet. This proves that the total payload of the NV+ data packets 51 & 52 was divided by your gigabit switch into 42 separate 1500 byte Ethernet packets on the way to your client. This segmentation creates a load on that switch, which may result in lost or mal-formed packets, depending on other network activity. The SACK as explained earlier, puts extra load on the NV+, completing the calculations required to verify it's segmented data packets were all recieved by the client. I invite you to repeat your test, while capturing a trace with Wireshark on your client to verify my assertion about the 1514 byte segmented Ethernet packets arriving at your client.
Conclusion:
Your NV+ is most certainly sending Jumbo Frames, even though you have that function unchecked AND you have the MTU set to 1500! This looks like a BUG in the SPARC ReadyNAS software.
These are by far the largest Jumbo Frames I have ever seen in many years of network trace analysis. Your network would be more effient without the extra overhead created by the Super Jumbo Frames. I won't go through the math here but, modern hardware and a modern kernel would be able to send the data in packet 51 & 52 as 42 individual 1514 byte (plus interpacket gap, preamble and delimiter) Ethernet frames in 600 microseconds (uS). Allowing for the vintage hardware and kernel in the NV+, the total time with client read request (#50), NV+ data packets, Ethernet and TCP verification calculations and ACK packet times for this 60K byte transfer should be under 1000 uS if encapsulated in 1514 Byte Ethernet, 1500 MTU, 1448 Byte TCP packets.
The first problem with Jumbo Frames is that they are not defined in any IETF (Internet Engineeding Task Force) standards document. The generally accepted (defacto-standard) Jumbo Frame size is 9000 bytes or less. Your NV+ is indeed running what is commonly referred to a Super Jumbo Frames (SJF). Read about them here: http://en.wikipedia.org/wiki/Jumbo_frame
TCP keeps the data in order by use of SEQuence numbers and ACKnowlegement numbers. The ACK number in a reply packet indicates to the NV+ that the client successfully recieved the data packet with a matching SEQ number. The TCP WINdow is the number of bytes that the sender of the packet (client) is willing to receive before it's buffers are full. This tells the NV+ how much data (number of packets X size of TCP data) to send before waiting for an ACK. There is also a TCP option called SelectiveACK, or SACK. This allows for a case where a packet is lost, or the connection is interrupted where a cllient can ACK a portion of the data sent, making it unnecessary to resend the entire stream of data packets. Only complete packets can be ACK'd, since all inbound packets must pass a checksum test before they are accepted.
When I look at the start of the trace, packets 1,2 and 3, it appears that the NV+, client and intermediate network devices are all handling the SJF. The NV+ sends a single Ethernet frame with 28735 bytes of data. The client replies with an ACKnowledgement for the entire data packet. Later we will see why this data probably arrived at the client as 19 separate 1500 MTU IP packets, with the client sending only one ACK for the last data packet.
As I look farther into the trace, things get interesting. In packet 50, the client sends a read request, with ACK for all previous data from the NV+, and advertises a TCP WIN of 65535 bytes. The NV+ responds with packets 51 (42,340 bytes) 52 (19,163 bytes), almost completly filling the client TCP WIN.
Packet E. Frame Size SEQ NSEQ ACK TCP bytes Time uS
50 117 1629 1320614 [63] 281
51 42394 1320614 1702 42340 801
52 19217 1362954 1382117 1702 19163 140
53 60 1702 1326454 0 141
54 60 1702 1361494 0 561
55 60 1702 1382117 0 280
totals 61503 B 2204 uS
You can also see that the ACK numbers in packets 53, 54 and 55 do not match any of the SEQ numbers of the two NV+ data packets. Since only complete packets can pass checksum (and get ACK), we know with absolute certainty that the NV+ data packets have been split up by an intermediate network device (gigabit switch), and those new packets were recieved intact by the client.
ACK from # 53 1326454
SEQ from # 51 (-) 1320614
5820 Bytes of TCP data Ack'd 5820 / 1448 = 4 (1500 MTU) packets
ACK from # 54 1361494
ACK from # 53 (-) 1326454
35040 Bytes of TCP data Ack'd 35040 / 1448 = 24 (1500 MTU) packets
ACK from # 55 1382117
ACK from # 54 (-) 1361494
20623 Bytes of TCP data Ack'd 20623 / 1448 = 14 (1500 MTU) packets
In the above calculation, the value of 1448 represents the TCP payload of a 1500 MTU Ethernet packet. This proves that the total payload of the NV+ data packets 51 & 52 was divided by your gigabit switch into 42 separate 1500 byte Ethernet packets on the way to your client. This segmentation creates a load on that switch, which may result in lost or mal-formed packets, depending on other network activity. The SACK as explained earlier, puts extra load on the NV+, completing the calculations required to verify it's segmented data packets were all recieved by the client. I invite you to repeat your test, while capturing a trace with Wireshark on your client to verify my assertion about the 1514 byte segmented Ethernet packets arriving at your client.
Conclusion:
Your NV+ is most certainly sending Jumbo Frames, even though you have that function unchecked AND you have the MTU set to 1500! This looks like a BUG in the SPARC ReadyNAS software.
These are by far the largest Jumbo Frames I have ever seen in many years of network trace analysis. Your network would be more effient without the extra overhead created by the Super Jumbo Frames. I won't go through the math here but, modern hardware and a modern kernel would be able to send the data in packet 51 & 52 as 42 individual 1514 byte (plus interpacket gap, preamble and delimiter) Ethernet frames in 600 microseconds (uS). Allowing for the vintage hardware and kernel in the NV+, the total time with client read request (#50), NV+ data packets, Ethernet and TCP verification calculations and ACK packet times for this 60K byte transfer should be under 1000 uS if encapsulated in 1514 Byte Ethernet, 1500 MTU, 1448 Byte TCP packets.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
