Forum Discussion

Guide

Jul 21, 2015

MPIO - slow speed

Hello! I have two ReadyNAS6 Pro boxes with the following configs: 1 box RDN6 pro/E7300/4gb/6*2Tb WD CB/4.2.27 FW 2 box RDN6 pro/E5300/2gb/6*2Tb WD CB/6.3.5 FW Other equipment: Juniper EX330...

iSCSI

MPIO

WingDog

Guide

Jul 23, 2015

I guess you must be running OS6 on the rnd6 because by default they have different OS than RN4220.

RND6 - 6.3.5 (thanks mdgm), RN4220 - 6.2.4 (latest for it)

I think the 4220 feezing, lagging pausing is a bgger concern. DO you have support case for that becuase there is some many things to discuss there hard to do here.

I HAD support case and while solving it I was need to reformat all 12*4Tb partition. funny?

thanks "Mateusz Janowicz NETGEAR Level 3 Technical Support Engineer" he was able to solve random halting RN4220. now it's only freesing.

What happens when no MPIO configured and using single network link? I assume it is all ok.

maybe, but ~100MB/sec is not enough.

From microsoft

"It is not necessary to have multiple subnets for iSCSI multi-pathing, but it's highly recommended, you can guarantee the paths it's going to use. You can use Multipath I/O (MPIO) on iSCSI connection, to deliver a high quality and reliable storage service with failover and load balancing capability."

So maybe we are both right but for me I have always tested anything with MPIO with seperate subnets. Vmware or Windows hypervisors

OK, now I have separate subnet. Also I've reformatted X-RAID to RAID0 (6*2Tb SATA drives).

here is results:

PS C:\> C:\SQLIO\sqlio.exe -s90 -kW -frandom -b8 -t8 -o16 -LS -BN D:\testfile.dat
sqlio v1.5.SG
using system counter for latency timings, 2474044 counts per second
8 threads writing for 90 secs to file D:\testfile.dat
        using 8KB random IOs
        enabling multiple I/Os per thread with 16 outstanding
        buffering set to not use file nor disk caches (as is SQL Server)
using current size: 1048576 MB for file: D:\testfile.dat
initialization done
CUMULATIVE DATA&colon;
throughput metrics:
IOs/sec:    43.59
MBs/sec:     0.34
latency metrics:
Min_Latency(ms): 750
Avg_Latency(ms): 2920
Max_Latency(ms): 6315
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 100

as you can see, even with RAID0 (!!!!) write speed and latency is unacceptable.

read speed equal to X-RAID (raid5) and limited to 2*1gbe.

PS C:\> C:\SQLIO\sqlio.exe -s90 -kR -frandom -b8 -t8 -o16 -LS -BN D:\testfile.dat
sqlio v1.5.SG
using system counter for latency timings, 2474044 counts per second
8 threads reading for 90 secs from file D:\testfile.dat
        using 8KB random IOs
        enabling multiple I/Os per thread with 16 outstanding
        buffering set to not use file nor disk caches (as is SQL Server)
using current size: 1048576 MB for file: D:\testfile.dat
initialization done
CUMULATIVE DATA&colon;
throughput metrics:
IOs/sec: 25469.41
MBs/sec:   198.97
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 4
Max_Latency(ms): 63
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 12  8  9 12 10 12 10  9 10  2  2  1  1  1  1  0  0  0  0  0  0  0  0  0  0

Of course I can never say we have no bugs but I can say we don't have an open one for MPIO at this time so we are happy to help if can.

Hoping for you, because after the third online chat escalltion I lose heart

MarcusF

NETGEAR Employee

Jul 23, 2015

Absolutley your results are not good howvere so no arguement there.

It wiill be hard to debug this here becaue I don't have an easy answer or configuartion change to make so will take a bit more back and forward.

If it is a software issue we need to have support case so can log issue.

Unfortunately we will not be able to log bug against a Pro6 running OS 6. So would need the data from the rn4220 if get that far to say is bug.

For now I would like to see reults with no MPIO and just single NIC so can start with basline with your intiator setup.

Just so you know I will be offline for a few days incase you think I am ignoring you.

mdgm or one of the others here may continue to help / provide advise

An internal request has already been logged to do some documentation around MPIO and I have asked could we try get some basline performance numbers from one of our labs with windows server 2012

WingDog
Guide
Jul 23, 2015
So would need the data from the rn4220 if get that far to say is bug.
4220 is in production, so why I don't want to make some hard experiments.
here is SQLio test at 4220 (one NIC)
PS C:\Windows\system32> C:\SQLIO\sqlio.exe -s90 -kW -frandom -b8 -t8 -o16 -LS -BN h:\testfile.dat sqlio v1.5.SG using system counter for latency timings, 2078193 counts per second 8 threads writing for 90 secs to file h:\testfile.dat using 8KB random IOs enabling multiple I/Os per thread with 16 outstanding buffering set to not use file nor disk caches (as is SQL Server) using current size: 102400 MB for file: h:\testfile.dat initialization done CUMULATIVE DATA&colon; throughput metrics: IOs/sec: 19.01 MBs/sec: 0.14 latency metrics: Min_Latency(ms): 834 Avg_Latency(ms): 6583 Max_Latency(ms): 7838 histogram: ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+ %: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 PS C:\Windows\system32>
it's X-raid (12*4Tb WD RE 4001-ffsx)
just after this test WEbUI stops working (3 minutes of "Connecting to ReadyNAS admin page.." and "readynas admin page is offline) and SMB shares were reconnected. FURY!!!!
it's freesing or lagging - as you wish to name it.
two minutes more - and it's alive again.
now I can confirm this bug - heavy iSCSI load makes 4220 unstable.
got fresh logs - can upload it or maybe you can create new case (previous was 25222344).

For now I would like to see reults with no MPIO and just single NIC so can start with basline with your intiator setup
look at my previous post (or this one).
WingDog
Guide
Jul 29, 2015
Hello!

Any news?
mdgm-ntgr
NETGEAR Employee Retired
Jul 29, 2015
Please open a new case and attach your logs. Let me know the case number.
WingDog
Guide
Jul 30, 2015
#25513848
mdgm-ntgr
NETGEAR Employee Retired
Jul 30, 2015
Can you update to 6.3.5-RC2 if not already running that?
WingDog
Guide
Jul 30, 2015
wich device? RN4220 or RDN6?
mdgm-ntgr
NETGEAR Employee Retired
Jul 30, 2015
Both
WingDog
Guide
Jul 30, 2015
now it's 6.4.0. T34 @ RN4220 =)
and still nothing ;)
WingDog
Guide
Aug 02, 2015
some fresh news (RN4220 NTGR highest device!!!!):
even with 6.4.0 T34 2*1gbe for Write is unavailable - only ~50-60MB/sec write speed with awesome latency (300-2000), so the storage is almost unusable.
w/o MPIO (1 NIC) it's a little faster and can work at 60-70MB/sec with the same shoking latency, but all other services like SMB or GUI is out of service during iSCSI high load.

conclusion is simple - at these days NTGR devices has no MPIO and very limited iSCSI support. and this is not Enterprise nor MID-Enterprise level devices.
I do not know what does your QA department do, but obviously something wrong.
WingDog
Guide
Aug 03, 2015
new case
25526961

RN4220 is inaccessible within default subnet
WingDog
Guide
Aug 03, 2015
incredible support team answer:
"if device is not halted now I can't escalate case, let's wait.

are Netgear still joking?
or Russian Support focused on home routers with "reset" option for any case???!!
BrendanM
NETGEAR Expert
Aug 07, 2015
I understand the case was escalated to L3 support and some recommendations were made regarding the network layout. Please let us know if you are seeing improvements now.
WingDog
Guide
Aug 07, 2015
Hello.
Yes, the case was escalated and some RDN network config changes were maid.

at this moment:
I've achived ~110MB/sec sequential read, and ~90MB/sec sequential write with one 1gbe NIC.
let it be the best for RN4220.

I still can't use MPIO at 4220 because of errors during initial connection (but five years old!!! RND6 pro work "well" with MPIO) and now I've lost any iscsi connetion to 4220 - will open new case. or maybe just reboot it several times? sad smile.

I will tell you honestly - I'm tired. every f*cking day I'm reading tons of email alerts from backup software which can't write/read/access 4220.
I'm opeing cases, chatting with L3 support, rebooting, reconfiguring, updating all around 4220.
I have eight (!!!!!) RDN6 boxes and it's 4-5 years old, but it's WORKING. EVERY SINGLE DAY TILL 5 YEARS!

anyway I finally convinced about NTRG enterprise devices quality - will never buy it again.
WingDog
Guide
Aug 07, 2015
new bug after reboot
endless "reconnecting to readynas admin page" and blue progress bar.
SMB shares are available, but iSCSI not (even one path).
RAIDar can't find any RDN at the network.

one more case?
OK!!
BrendanM
NETGEAR Expert
Aug 07, 2015
The existing case is still open, pending feedback from you, so you don't need to open a new case. Please continue to work with the L3 tech under that case.
WingDog
Guide
Aug 10, 2015
Hello.
72 hours uptime without issues!
it's new high score.

thanks everyone (especially L3 tech's Justin and Stefano).
BrendanM
NETGEAR Expert
Aug 10, 2015
That's good news, thanks for the update.

As OptimusPrime mentioned, we are working on getting an MPIO KB article published.

What do you think the solution was in your environment? It may help others who come across this thread.
WingDog
Guide
Aug 10, 2015
Hello BrendanM.

There were two misconfigs:
First
subnets for MPIO must be sepatated (i.e. 192.168.0.0/24 and 192.168.1.0/24).
other vendors allows one subnet for MPIO. it's usually separated VLAN for iSCSI traffic only.
Second
gateway at NICs config - GUI doesn't allow to save 0.0.0.0 at GW string (and that is correct!), but at the same time it's sets it (0.0.0.0) automatically.
it's weird and confusing. no GW is NO GW, nor 0.0.0.0 nor something else.
besause no one in sane mind will not route iSCSI traffic across routers.

other wishes:
- OOBM or other dedicated management
- VLANs