[18373207] Duo Slow NFS Reads, Jumbo Frames and TCP window

I can't believe I have the only two ReadyNASs sending Jumbo Frames when they are disabled in Frontview. I have this problem on both my Duo and my NV+.
I would appreciate it if someone else here could check to see if their ReadyNAS Duo or NV+ is sending Jumbo Frames with that function unchecked in Frontview / Network / Interfaces / Ethernet (Make sure the MTU is also set to 1500). If interested, you can capture the packets with tcpdump and read the file with Wireshark. tcpdump is not native in the ReadyNAS. You will need to connect to the readyNAS with an ssh client, then:

apt-get install tcpdump

While still in the ssh terminal window, run this command to capture a network trace file. You will need to start a large data transfer from the ReadyNAS to the client first, then start tcpdump.

tcpdump -s 192 host <ip address of your client> -w /c/backup/tcpdump.cap

Let the capture run for a few seconds, then abort the capture. To stop the capture, you must press [Ctl]+[C]
The above will save only the first 192 bytes of each packet, which will capture all the headers up through application (nfs in my case), but not your payload data. It will write the trace to the backup share where you can view it with your windows client, and maybe post the results to this forum, should you choose. The host argument will limit the capture to traffic between the ReadyNAS and the clinet you are useing for your test.

Wireshark is certainly one of the best programs available to read your trace file. It is free and can be downloaded here: http://www.wireshark.org/download.html After you have Wireshark installed on your Windows PC, you can double-click the trace file and it will open to the summary screen. The sixth column is the Length of the packet. If you see any values in this column greater than 1514, the ReadyNAS is useing Jumbo Frames!

Well it was a bit more complicated than that - for those of us with factory standard software there is a need to install the ssh add-on, then install apt, then

apt-get update

to be able to get tcpdump.

Anyway figured that out and then the instructions above are perfect.

I have successfully captured some packets and have included the file here here for analysis (it's around 5mb).

I "think" I can see packet sizes greater than the 1500 MTU, as an example look at packet 6 which seems to be 42340 long

I can confirm than my NV+ is set up not to use jumbo frames - here's a picture to check. The host is connected by a 100mb network card to a gigabit switch, jumbo frames is not ticked on the readynas.



I'm typically getting around 12GB / hour on large scale transfers, whatever the protocol, which is equivalent to 3,333kb/s. I did see around 9000 kb/s after one setup change but it then fell back again

Hopefully this will help in getting those transfer rates up.

Hope someone can make sense of this

Sorry for not being more complete on the process of installing tcpdump. I really apprecaiate you taking the time to do this.

The first problem with Jumbo Frames is that they are not defined in any IETF (Internet Engineeding Task Force) standards document. The generally accepted (defacto-standard) Jumbo Frame size is 9000 bytes or less. Your NV+ is indeed running what is commonly referred to a Super Jumbo Frames (SJF). Read about them here: http://en.wikipedia.org/wiki/Jumbo_frame

TCP keeps the data in order by use of SEQuence numbers and ACKnowlegement numbers. The ACK number in a reply packet indicates to the NV+ that the client successfully recieved the data packet with a matching SEQ number. The TCP WINdow is the number of bytes that the sender of the packet (client) is willing to receive before it's buffers are full. This tells the NV+ how much data (number of packets X size of TCP data) to send before waiting for an ACK. There is also a TCP option called SelectiveACK, or SACK. This allows for a case where a packet is lost, or the connection is interrupted where a cllient can ACK a portion of the data sent, making it unnecessary to resend the entire stream of data packets. Only complete packets can be ACK'd, since all inbound packets must pass a checksum test before they are accepted.

When I look at the start of the trace, packets 1,2 and 3, it appears that the NV+, client and intermediate network devices are all handling the SJF. The NV+ sends a single Ethernet frame with 28735 bytes of data. The client replies with an ACKnowledgement for the entire data packet. Later we will see why this data probably arrived at the client as 19 separate 1500 MTU IP packets, with the client sending only one ACK for the last data packet.

As I look farther into the trace, things get interesting. In packet 50, the client sends a read request, with ACK for all previous data from the NV+, and advertises a TCP WIN of 65535 bytes. The NV+ responds with packets 51 (42,340 bytes) 52 (19,163 bytes), almost completly filling the client TCP WIN.

Packet  E. Frame Size         SEQ     NSEQ       ACK     TCP bytes  Time uS
   50          117            1629             1320614       [63]      281    
   51        42394         1320614                1702     42340       801
   52        19217         1362954  1382117       1702     19163       140
   53           60            1702             1326454         0       141
   54           60            1702             1361494         0       561
   55           60            1702             1382117         0       280
                                                   totals  61503 B    2204 uS

You can also see that the ACK numbers in packets 53, 54 and 55 do not match any of the SEQ numbers of the two NV+ data packets. Since only complete packets can pass checksum (and get ACK), we know with absolute certainty that the NV+ data packets have been split up by an intermediate network device (gigabit switch), and those new packets were recieved intact by the client.

ACK from # 53      1326454
SEQ from # 51  (-) 1320614
                      5820 Bytes of TCP data Ack'd   5820 / 1448 = 4 (1500 MTU) packets

ACK from # 54      1361494
ACK from # 53  (-) 1326454
                     35040 Bytes of TCP data Ack'd  35040 / 1448 = 24 (1500 MTU) packets

ACK from # 55      1382117
ACK from # 54  (-) 1361494
                     20623 Bytes of TCP data Ack'd  20623 / 1448 = 14 (1500 MTU) packets

In the above calculation, the value of 1448 represents the TCP payload of a 1500 MTU Ethernet packet. This proves that the total payload of the NV+ data packets 51 & 52 was divided by your gigabit switch into 42 separate 1500 byte Ethernet packets on the way to your client. This segmentation creates a load on that switch, which may result in lost or mal-formed packets, depending on other network activity. The SACK as explained earlier, puts extra load on the NV+, completing the calculations required to verify it's segmented data packets were all recieved by the client. I invite you to repeat your test, while capturing a trace with Wireshark on your client to verify my assertion about the 1514 byte segmented Ethernet packets arriving at your client.

Conclusion:
Your NV+ is most certainly sending Jumbo Frames, even though you have that function unchecked AND you have the MTU set to 1500! This looks like a BUG in the SPARC ReadyNAS software.

These are by far the largest Jumbo Frames I have ever seen in many years of network trace analysis. Your network would be more effient without the extra overhead created by the Super Jumbo Frames. I won't go through the math here but, modern hardware and a modern kernel would be able to send the data in packet 51 & 52 as 42 individual 1514 byte (plus interpacket gap, preamble and delimiter) Ethernet frames in 600 microseconds (uS). Allowing for the vintage hardware and kernel in the NV+, the total time with client read request (#50), NV+ data packets, Ethernet and TCP verification calculations and ACK packet times for this 60K byte transfer should be under 1000 uS if encapsulated in 1514 Byte Ethernet, 1500 MTU, 1448 Byte TCP packets.

godfreydanials wrote:
The first problem with Jumbo Frames is that they are not defined in any IETF (Internet Engineering Task Force) standards document.

One small mistake in a nice analysis. Ethernet and WiFi are standardized by the IEEE, not the IETF. (http://en.wikipedia.org/wiki/IEEE_802.3). The IETF doesn't standardize physical-layer, they work at layer 3 and above.

You are 100% correct in saying that Jumbo Frames are non standard, and more importantly that the ReadyNAS should not be sending packets of this size even if Jumbo Frames were turned on.

Sorry, my bad. I knew that.

OK, I have done another test, this time recording the data transferred on the client end, rather than the NAS. The log file is located here

Again I have double checked the settings and jumbo frames is off, and the NV+ is showing connection to a gigabit network, see below screen cap.



If someone could take a look at this and examine if jumboframes is the problem i'd be grateful - I believe the hardware should be capable of much faster rates than this.

Thanks

The key piece to determine from this most recent data captured at the client is that there are no Jumbo Frames (JF) on the wire. This is an upload of data from the client to the server. In this case the only packets originating from the NAS are simple TCP ACK segments. The NAS would have no reason to use a JF, since all traffic from that end is below the 1500 MTU.
The client is not sending JFs either.

I did measure client upload TCP data rate at 15.13 MBps with an MTU of 1500. This is not the SMB (CIFS) payload, which would be slightly lower and difficult to determine because of the header size and embedded Andx. Although this trace represents less that 6 seconds of the data upload, it is enough to see the bottlenecks and make some projections.

The largest delays in this transfer is caused by the times between the end of the client TCP stream and the ACK from the NAS.

		               From NAS          From Client
Packet count           3125               69657
ACK delay >1000uS      1317 (54%)            23 (< .01%)
delay >500 <1000uS     1317 (42%)            33 (< .01%)

The increased ACK times from the NAS are to be expected because this is an upload and the NAS must calculate chacksums before the ACK. However, since the data is originating from the client as a stream of 1500 MTU packets, and there are no packets retransmitted, the NAS does not have to employ SACK as we saw in you first trace of the download from the NAS. I would have expected these dealy times to be in the order of 300 to 500 microseconds.

I did some what-if scenarios to determine the influence of these ACK times. The first case is if all ACK delays were 1000 uS or less, for both the client and server. This increases the TCP data upload rate from 15.13 MBps to 22.20 MBps. If the ACK times were kept to 500 uS or less, your transfer rate would have been 32.17 MBps.