RAIDiator-x86 4.2.28 [T6] Teaming issue

[Bacelo]

Hello ,

I recognized a strange network speed performance lately with my ReadyNAS NVX Business Edition with teaming mode "transmit load balacing".

Szenario:

After a reboot I have around 3-5 MB/sec. transfer rate.

I need to go in the admin panel and swap back and forth the teaming mode (f.e. from "transmit load balacing" to "adaptive load balacing" and back to "transmit load balacing") to receive my normal max  transfer rate ~35MB/sec.

That happens after every reboot.

Has somebody noticed the same szenario or may has an idea, how to fix that network problem?

Regards,

Barcelo

[StephenB]

I use LACP myself.

I am mostly wondering why your normal max xmit rate is only 35 MB/s.  You should get much better than that with teaming off.

Perhaps try using NAStester to benchmark it?  http://www.808.dk/?code-csharp-nas-performance

[Bacelo]

Hi Stephen,

thank for your answer / help.

Really strange is the fact, that after a boot the speed is low (no matter which teaming setup).

I tracked as following:

Spoiler

Used switch -> Netgear GS105v3
GB-network with jumbo frames enabled

"transmit load balacing" -> after fresh boot
Running warmup...
Running a 400MB file write on N: 5 times...
Iteration 1:      7,09 MB/sec
Iteration 2:      6,23 MB/sec
Iteration 3:      8,15 MB/sec
Iteration 4:      7,62 MB/sec
Iteration 5:      6,28 MB/sec
-----------------------------
Average (W):      7,07 MB/sec
-----------------------------
Running a 400MB file read on N: 5 times...
Iteration 1:     27,98 MB/sec
Iteration 2:     25,40 MB/sec
Iteration 3:     25,03 MB/sec
Iteration 4:     27,60 MB/sec
Iteration 5:     25,80 MB/sec
-----------------------------
Average (R):     26,36 MB/sec
-----------------------------

"transmit load balacing" -> after switching back and forth
Running warmup...
Running a 400MB file write on N: 5 times...
Iteration 1:     32,94 MB/sec
Iteration 2:     32,23 MB/sec
Iteration 3:     31,45 MB/sec
Iteration 4:     25,31 MB/sec
Iteration 5:     30,83 MB/sec
-----------------------------
Average (W):     30,55 MB/sec
-----------------------------
Running a 400MB file read on N: 5 times...
Iteration 1:     93,71 MB/sec
Iteration 2:     93,44 MB/sec
Iteration 3:     87,97 MB/sec
Iteration 4:    102,47 MB/sec
Iteration 5:     93,34 MB/sec
-----------------------------
Average (R):     94,18 MB/sec
-----------------------------




"Active Backup" -> single connnection
Running warmup...
Running a 400MB file write on N: 5 times...
Iteration 1:     27,77 MB/sec
Iteration 2:     27,16 MB/sec
Iteration 3:     34,51 MB/sec
Iteration 4:     34,38 MB/sec
Iteration 5:     35,20 MB/sec
-----------------------------
Average (W):     31,80 MB/sec
-----------------------------
Running a 400MB file read on N: 5 times...
Iteration 1:    109,72 MB/sec
Iteration 2:    104,04 MB/sec
Iteration 3:    105,37 MB/sec
Iteration 4:    109,13 MB/sec
Iteration 5:     98,31 MB/sec
-----------------------------
Average (R):    105,31 MB/sec
-----------------------------

"IEEE 802.3ad LACP with Layer 2"
Running warmup...
Running a 400MB file write on N: 5 times...
Iteration 1:     31,05 MB/sec
Iteration 2:     40,90 MB/sec
Iteration 3:     37,65 MB/sec
Iteration 4:     30,48 MB/sec
Iteration 5:     32,37 MB/sec
-----------------------------
Average (W):     34,49 MB/sec
-----------------------------
Running a 400MB file read on N: 5 times...
Iteration 1:    111,86 MB/sec
Iteration 2:    110,62 MB/sec
Iteration 3:    109,71 MB/sec
Iteration 4:    102,07 MB/sec
Iteration 5:     94,30 MB/sec
-----------------------------
Average (R):    105,71 MB/sec
-----------------------------

"IEEE 802.3ad LACP with Layer 2+3"
Running warmup...
Running a 400MB file write on N: 5 times...
Iteration 1:     31,49 MB/sec
Iteration 2:     28,54 MB/sec
Iteration 3:     24,83 MB/sec
Iteration 4:     37,77 MB/sec
Iteration 5:     31,58 MB/sec
-----------------------------
Average (W):     30,84 MB/sec
-----------------------------
Running a 400MB file read on N: 5 times...
Iteration 1:     98,54 MB/sec
Iteration 2:    103,52 MB/sec
Iteration 3:     80,57 MB/sec
Iteration 4:    110,70 MB/sec
Iteration 5:    114,10 MB/sec
-----------------------------
Average (R):    101,48 MB/sec
-----------------------------

"IEEE 802.3ad LACP with Layer 3+4"
Running warmup...
Running a 400MB file write on N: 5 times...
Iteration 1:     34,79 MB/sec
Iteration 2:     27,49 MB/sec
Iteration 3:     34,08 MB/sec
Iteration 4:     31,82 MB/sec
Iteration 5:     25,96 MB/sec
-----------------------------
Average (W):     30,83 MB/sec
-----------------------------
Running a 400MB file read on N: 5 times...
Iteration 1:    111,52 MB/sec
Iteration 2:    107,12 MB/sec
Iteration 3:    102,39 MB/sec
Iteration 4:    108,38 MB/sec
Iteration 5:     94,76 MB/sec
-----------------------------
Average (R):    104,83 MB/sec
-----------------------------

You can see the different speeds specially when switching back and forth after a fresh boot.

-> after boot it's low ~7MB/s, after swithing back and forth it's ~30MB/s

Regards,
Barcelo

[StephenB]

---

@Bacelo wrote:

Hi Stephen,

thank for your answer / help.

Really strange is the fact, that after a boot the speed is low (no matter which teaming setup).

...

You can see the different speeds specially when switching back and forth after a fresh boot.

-> after boot it's low ~7MB/s, after swithing back and forth it's ~30MB/s

Regards,
Barcelo

---

There might be some background volume activity after reboot - it might be worth looking at your logs.

In addition to the slow speed on reboot, your write speeds look generally slow to me.  This is raid-5/single redundancy?  Are you using jumbo frames?  What is your test PC?

Teaming generally won't increase speeds when you have a single user (and actually often doesn't help much with 2-3 users). Its intended use was for trunking (connecting switches), so it comes into its own when there are lots of users simultaneously using the NAS.

[Bacelo]

Hi Stephen,

thanks again for your reply/ help.

There might be some background volume activity after reboot - it might be worth looking at your logs.
-> will check

Regarding your questions:

• This is raid-5/single redundancy?
  -> RAID-Level X-RAID2, 4 discs, Redundant

• Are you using jumbo frames?
  -> yes 9KB (of cource also set at the PC)

• What is your test PC?
  -> WIN 10 64-Bit
  -> AMD A10-7850K with 16GB on ASRock M2A88M Extreme4+
  -> Qualcomm Atheros AR8171/8175 PCI-E Gigabit Ethernet Controller (NDIS 6.30)
  -> fixed IP settings for IPv4
  -> CAT 5 cable
  -> IPv6 disabled
  -> QoS-Planer disabled
  -> all "Offload" settings disabled
  -> ECMA disabeld
  -> "Scaling" disabeld
  -> "Flow control" disabled
  -> "Interrupt moderation" disabeld
  -> Max. IRQ /s = 5000
  -> Buffer receive = 512
  -> Buffer send = 256
  -> RSS queues = 4
  -> Speed & duples = auto negotiation
  -> SWOI disabled

Regards,

Barcelo

[StephenB]

You should be using at least cat 5e with gigabit.

Try turning on flow control on the PC.  Also if there is a control for it on your switch, turn it on there too.  That could explain poor write speeds.

If flow control doesn't improve the write speeds, then try turning jumbo frames off on the PC and NAS as a test, and see what happens to the speeds.

JF aren't always faster, it depends on the specifics of the hardware.  The gains aren't on the network, they come from reducing the interrupt load on the clients.  There are offsetting losses (larger memory buffers, etc), so mileage does vary.

What switch are you using btw?  With the netgear ones (GS724T anyway) the MTU for the bond is set separately from the MTU for the port.

[Bacelo]

Cables:

Just checked the cables - all are CAT 5e

Switch:

-> is (as mentioned above) a Netgear Switch GS105v3

Flow control:
-> enabled this now and check performance -> no improvement

Jumbo frames:

-> disabled them and did a performance check -> no improvement

When check the logs the diagnostic log is:

Spoiler

Disks
-------------------------------
Passed diagnostics.

Memory
-------------------------------
Passed diagnostics.

Network
-------------------------------
Passed diagnostics.

Performance
-------------------------------
* Jumbo frames are disabled on interface 1.  If both your switch and clients support jumbo frames, you can enhance your write performance by enabling jumbo frames on this interface.
* Jumbo frames are disabled on interface 2.  If both your switch and clients support jumbo frames, you can enhance your write performance by enabling jumbo frames on this interface.

Volume
-------------------------------
Passed diagnostics.

-> strange that the logs says that JF's are disabled, because they are enabled in the backend settings

[StephenB]

The net here is that your write speeds seem quite sluggish for an x86 NAS with single redundancy.  I think you should be seeing something over 50 MB/s.  Smallnetbuilder measured 54MB/s in their (very old) NVX review, and I think performance might have been improved after that.   It's possible the PC you are using for testing is limiting the speeds (since the test file is read locally, then written over the network).

Are you seeing many errors on the ethernet stats (should be available on the NAS and the switch)?

Perhaps also try disconnecting one of the ethernet cables from the NAS, and see if performance changes.

-Did you set the MTU for the LAG bond also?  Again, that is not the same as the port MCU, it is a different setting (and unfortunately well hidden).  On the switch, try selecting the PORTS tab.  You should see a port configuration table, which includes three easy-to-miss controls at the top - ports, lags, and all.  Click on lags, and you will see LAG bond MTU settings (among other things).  They default to 1518 on my GS724T and GS108T.

[Bacelo]

Stephen, my Netgear GS105v3 is a non-managed GB-switch.

It supports jumbo frames but no the IEEE 802.3ad LACP protocol.

I'll try to get an managed GB switch with IEEE 802.3ad LACP support and see how that will work.

Yeah, the read speed is fine but the write speed is very bad. As I mentioned in the first post, it used to be even more slower at around 3-5 MB/sec... until I noticed that issue with back and forth switching after a boot, which increases the speed to ~30 MB/sec (read speed after boot is at poor ~23 MB/sec).

So there are two issues:

- very slow network speed (read & write) after a fresh boot, and why I need to back and forth change the setting

- slow write speed which could almost be double faster

More to follow

[StephenB]

Before you go buy a new switch, it might be worth sorting out whether teaming makes sense for you.  If a 5 port gigabit switch is enough for you, then the answer is almost certainly no.  How many ethernet connections do you use, and how often are they in use simultaneously?

Since the current switch is not managed, you can't use most of the teaming modes.  The two you can use are transmit load balancing and adaptive load balancing.  Adaptive load balancing is likely the better choice.

On the main issue, (slow write speeds) I'd suggest looking at the rx packet stats on the NAS next ("show errors" on the network interface").. RX dropped, overruns, frame, should all be very small percentages.  Collisions should be 0.   It's good to cut/paste the values before and after running NAStester, so you can see what was going on when it was running.

I'd also confirm the path MTU between the NAS and the PC with ping.  On a Windows machine, you open cmd (search for it in the taskbar search) and enter 

ping NASIPADDRESS -f -l 8972

This should succeed w/o giving you a fragmentation warning.  If it fails, find the largest -l value that does work (1472 is the value for normal ethernet).  Add 28 to that value, and that's the actual MTU of your network path.

[Bacelo]

Hi Stephen,

thank you again for helping me

- A 5 port gigabit switch is enough for me (but I'll borrow a newer managed one for testing)

- Most of the time I do only have one client connnection to the NAS -> so I will disable teaming as mentioned by you

-> Performance results (teaming disabled - jumbo frames enabled):

Spoiler

Running warmup...
Running a 400MB file write on M: 5 times...
Iteration 1:     34,27 MB/sec
Iteration 2:     31,60 MB/sec
Iteration 3:     41,79 MB/sec
Iteration 4:     31,17 MB/sec
Iteration 5:     32,74 MB/sec
-----------------------------
Average (W):     34,31 MB/sec
-----------------------------
Running a 400MB file read on M: 5 times...
Iteration 1:    112,78 MB/sec
Iteration 2:    121,33 MB/sec
Iteration 3:    118,52 MB/sec
Iteration 4:    113,78 MB/sec
Iteration 5:    123,08 MB/sec
-----------------------------
Average (R):    117,90 MB/sec
-----------------------------

Network card 1 stats (teaming disabled):

Spoiler

Sendepakete    462980
Sendefehler    0
Sendeverluste    0
Sendeüberschreitungen    0
TX-Carrier    0
Paket-Empfang    625250
Empfangsfehler    0
Empfangsverluste    2
Empfangsüberschreitung    0
Empfangs-Frame    0
Kollisionen    0

Network card 2 stats (when was teamed):

Spoiler

Sendepakete    119489
Sendefehler    0
Sendeverluste    0
Sendeüberschreitungen    0
TX-Carrier    0
Paket-Empfang    834976
Empfangsfehler    3
Empfangsverluste    0
Empfangsüberschreitung    0
Empfangs-Frame    3
Kollisionen    0

path MTU between the NAS and the PC:

Spoiler

C:\>ping 192.168.28.103 -f -l 8972

Ping wird ausgeführt für 192.168.28.103 mit 8972 Bytes Daten:
Antwort von 192.168.28.103: Bytes=8972 Zeit<1ms TTL=64
Antwort von 192.168.28.103: Bytes=8972 Zeit<1ms TTL=64
Antwort von 192.168.28.103: Bytes=8972 Zeit<1ms TTL=64
Antwort von 192.168.28.103: Bytes=8972 Zeit<1ms TTL=64

Ping-Statistik für 192.168.28.103:
    Pakete: Gesendet = 4, Empfangen = 4, Verloren = 0
    (0% Verlust),
Ca. Zeitangaben in Millisek.:
    Minimum = 0ms, Maximum = 0ms, Mittelwert = 0ms

-> successfully with jumbo frame settings

Regards,

Barcelo

[StephenB]

It really wants to write at ~35 MB/s.

I don't have any other thoughts on things to try with the write performance - the network stats look fine.   If something occurs to me, I'll post again.

[Bacelo]

Update:
I just swapped my switch to a brand new managed HP 1810-8g and didn't noticed any change at all - still around ~ 30 - 35 MB/sec write speed .

Then I checked the telemetry data (iStat) via "NAS Utils" and it looks like the NVX is doing very hard at the limit...

I made some screenshots:

[StephenB]

On the system->performance tab, do you have "enable write cache" checked?  What about "disable full data journaling"

[Bacelo]

Hi Stephen,

here are my performance tab settings:

[StephenB]

---

@Bacelo wrote:

Hi Stephen,

here are my performance tab settings:

...

---

 One more possibility excluded....

How full is the volume (% used, not the actual amount)?

[Bacelo]

[Bacelo]

Hi all,

besides my low network speed, I'd like to get back to the issue of my first post of this thread:

-> after a fresh boot, I need to change some option (doesn't mind which option) back and forth to get high network speed. Otherwise the network speed is around 3-5 MB/sec. transfer rate

Has anyone noticed anything like that?

Regards,
Bacelo

[StephenB]

---

@Bacelo wrote:

Hi all,

besides my low network speed, I'd like to get back to the issue of my first post of this thread:

-> after a fresh boot, I need to change some option (doesn't mind which option) back and forth to get high network speed. Otherwise the network speed is around 3-5 MB/sec. transfer rate

Has anyone noticed anything like that?

Regards,
Bacelo

---

I have not seen it.  I'd expect ALB to be better than TLB.  Does ALB also need you to toggle settings after reboot?

[Bacelo]

Currently I have configured "Backup" teaming.

The situation is independent from the teaming setup. I can choose any teaming config and the speed is always extreme slow after a fresh boot.

Once, any network setting is changed and applied, the ~10x speed is vailable

[StephenB]

---

@Bacelo wrote:

Currently I have configured "Backup" teaming.

The situation is independent from the teaming setup. I can choose any teaming config and the speed is always extreme slow after a fresh boot.

Once, any network setting is changed and applied, the ~10x speed is vailable

---

I figured that likely was the case.  And "backup teaming" means the additional NIC isn't even being used.

If this is specific to the beta firmware, probably Netgear would need to diagnose it.  They might be able to do that from the logs. 

[Bacelo]

Thanks Stephen

Providing logs is no problem.

The question is, if someone at Netgear is interestet to have a look at them

[StephenB]

Maybe PM skywalker, and ask.

[Bacelo]

I sent Skywalker a PM - let's see if and what he will answer.

[Bacelo]

A little update regarding my issue:

I haven't received any answer from Skywalker or any other Netgear stuff...

As a final conclusion for this circumstance, I guess I'll sell that last owned Netgear product and swap to another brand - the final end!