- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
Re: MPIO - slow speed
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
MPIO - slow speed
Hello!
I have two ReadyNAS6 Pro boxes with the following configs:
1 box
RDN6 pro/E7300/4gb/6*2Tb WD CB/4.2.27 FW
2 box
RDN6 pro/E5300/2gb/6*2Tb WD CB/6.3.5 FW
Other equipment:
Juniper EX3300 switch and Dell R510 server
R510 is Hyper-V 2012 R2 server for lightweight load with MPIO enabled.
I'm using SQLio to check iSCSI storage speed and latency.
while any tests I got only ~110MB/sec storage speed with 2*500mbps load at dedicated iSCSI NIC (taskmgr) with any NAS.
mpclaim show LB policy as RRWS (it seems to be correct, but RR only is unsupported??!) with two active paths, so MPIO configured correctly.
But PRTG monitor shows ~92-95% load at first NIC and ~10-15% at second NAS NIC, so why I guess there is some MPIO error with NAS config.
So, the question is: how to achive 2*1gbps speed using MPIO with ReadyNAS?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
Hi Wingdog,
How many active NICs are you using in your ReadyNAS system?
Kind regards,
BrianL
NETGEAR Community
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
Hello, mdgm.
Thanks for "the secret key" with two subnets - I've achived ~2*890mbps at READ with RND MPIO. Not so good as other vendors, but acceptable.
I think it will be better to add more MPIO specific information to ReadyNAS OS6 manual (and about supported LB policies too).
PS C:\> C:\SQLIO\sqlio.exe -s900 -kR -frandom -b8 -t8 -o16 -LS -BN D:\testfile.dat sqlio v1.5.SG using system counter for latency timings, 2474044 counts per second 8 threads reading for 900 secs from file D:\testfile.dat using 8KB random IOs enabling multiple I/Os per thread with 16 outstanding buffering set to not use file nor disk caches (as is SQL Server) using current size: 1048576 MB for file: D:\testfile.dat initialization done CUMULATIVE DATA: throughput metrics: IOs/sec: 25979.92 MBs/sec: 202.96 latency metrics: Min_Latency(ms): 0 Avg_Latency(ms): 4 Max_Latency(ms): 75 histogram: ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+ %: 10 9 11 12 11 13 11 9 9 2 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0
but what's the mess with WRITE??!!
PS C:\> C:\SQLIO\sqlio.exe -s900 -kW -frandom -b8 -t8 -o16 -LS -BN D:\testfile.dat sqlio v1.5.SG using system counter for latency timings, 2474044 counts per second 8 threads writing for 900 secs to file D:\testfile.dat using 8KB random IOs enabling multiple I/Os per thread with 16 outstanding buffering set to not use file nor disk caches (as is SQL Server) using current size: 1048576 MB for file: D:\testfile.dat initialization done CUMULATIVE DATA: throughput metrics: IOs/sec: 21.26 MBs/sec: 0.16 latency metrics: Min_Latency(ms): 268 Avg_Latency(ms): 6005 Max_Latency(ms): 13647 histogram: ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+ %: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 PS C:\>
that is unacceptable!
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
All MPIO performance is relate dto the client side not the readnas. to see good speed the client must plance the iscsi commands correctly.
The readynas just reponds to the reponse on whichver interface it sees. That is why 2 subnets important because otherwise the readynas will stick to one of the interfaces to send all traffic and so you get no ballancing
But now that you have subnets considered to ReadyNAs there is no extra configuration needed for MPIO. all decison over which interface to send the iscsi commands are on the client side.
I hope this makes sense
I do agreee we could do with some doicumentation in this area but it is not a common configurtation request we have sene but seems it is becoming more popopular.
On the read speed I would think at this point it is limited by interface speed and disk / volume maximum performane not really a readynas specific thing but I could be wrong. So Raid setup and disk performnce should be the limiting factors.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
All MPIO performance is relate dto the client side not the readnas. to see good speed the client must plance the iscsi commands correctly.
The readynas just reponds to the reponse on whichver interface it sees. That is why 2 subnets important because otherwise the readynas will stick to one of the interfaces to send all traffic and so you get no ballancing
That is not complete correct because if I'm using Windows Server as iSCSI target I can use any subnets, including one subnet for all traffic even with 8 NICs both sides.
I do agreee we could do with some doicumentation in this area but it is not a common configurtation request we have sene but seems it is becoming more popopular.
Unfortunally I also have 4220 box. it's freesing, pausing, lagging and It's absolutly equal to this RND6 pro by FW, but have powerful hardware. So if NTGR presents 4220 as Enterprise (however RND6 pro is MID-Ent for NTGR) manuals MUST have enough explanation of devices logic. And if you presents iSCSI and MPIO - it's must work without some strange config particularly if there is no Vlan Tags and "management" interfaces (or OOB).
On the read speed I would think at this point it is limited by interface speed and disk / volume maximum performane not really a readynas specific thing but I could be wrong. So Raid setup and disk performnce should be the limiting factors.
This is not correct at all. it's nonsense.
there is no any way to get 20 IOPS and 0MB/sec from one SATA drive.
this is software bug
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
here is results of the same test at one sata drive (it's second drive at my work PC)
PS C:\Windows\system32> C:\SQLIO\sqlio.exe -s90 -kW -frandom -b8 -t8 -o16 -LS -BN D:\testfile.dat sqlio v1.5.SG using system counter for latency timings, 2728190 counts per second 8 threads writing for 90 secs to file D:\testfile.dat using 8KB random IOs enabling multiple I/Os per thread with 16 outstanding buffering set to not use file nor disk caches (as is SQL Server) using current size: 1048576 MB for file: D:\testfile.dat initialization done CUMULATIVE DATA: throughput metrics: IOs/sec: 135.66 MBs/sec: 1.05 latency metrics: Min_Latency(ms): 0 Avg_Latency(ms): 939 Max_Latency(ms): 1828 histogram: ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+ %: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 99
135 IOPS - that is good for one sata drive with such load
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
I guess you must be running OS6 on the rnd6 because by default they have different OS than RN4220.
I think the 4220 feezing, lagging pausing is a bgger concern. DO you have support case for that becuase there is some many things to discuss there hard to do here.
What happens when no MPIO configured and using single network link? I assume it is all ok.
There is no special MPIO configuration needed on ReadyNAS. To ReadyNAS we are just recieving and transmitting using ISCSI protocol / commands. Just happens with MPIO those are recieved via seperate nics.
From microsoft
"It is not necessary to have multiple subnets for iSCSI multi-pathing, but it's highly recommended, you can guarantee the paths it's going to use. You can use Multipath I/O (MPIO) on iSCSI connection, to deliver a high quality and reliable storage service with failover and load balancing capability."
So maybe we are both right but for me I have always tested anything with MPIO with seperate subnets. Vmware or Windows hypervisors
Of course I can never say we have no bugs but I can say we don't have an open one for MPIO at this time so we are happy to help if can.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
I guess you must be running OS6 on the rnd6 because by default they have different OS than RN4220.
RND6 - 6.3.5 (thanks mdgm), RN4220 - 6.2.4 (latest for it)
I think the 4220 feezing, lagging pausing is a bgger concern. DO you have support case for that becuase there is some many things to discuss there hard to do here.
I HAD support case and while solving it I was need to reformat all 12*4Tb partition. funny?
thanks "Mateusz Janowicz NETGEAR Level 3 Technical Support Engineer" he was able to solve random halting RN4220. now it's only freesing.
What happens when no MPIO configured and using single network link? I assume it is all ok.
maybe, but ~100MB/sec is not enough.
From microsoft
"It is not necessary to have multiple subnets for iSCSI multi-pathing, but it's highly recommended, you can guarantee the paths it's going to use. You can use Multipath I/O (MPIO) on iSCSI connection, to deliver a high quality and reliable storage service with failover and load balancing capability."
So maybe we are both right but for me I have always tested anything with MPIO with seperate subnets. Vmware or Windows hypervisors
OK, now I have separate subnet. Also I've reformatted X-RAID to RAID0 (6*2Tb SATA drives).
here is results:
PS C:\> C:\SQLIO\sqlio.exe -s90 -kW -frandom -b8 -t8 -o16 -LS -BN D:\testfile.dat sqlio v1.5.SG using system counter for latency timings, 2474044 counts per second 8 threads writing for 90 secs to file D:\testfile.dat using 8KB random IOs enabling multiple I/Os per thread with 16 outstanding buffering set to not use file nor disk caches (as is SQL Server) using current size: 1048576 MB for file: D:\testfile.dat initialization done CUMULATIVE DATA: throughput metrics: IOs/sec: 43.59 MBs/sec: 0.34 latency metrics: Min_Latency(ms): 750 Avg_Latency(ms): 2920 Max_Latency(ms): 6315 histogram: ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+ %: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100
as you can see, even with RAID0 (!!!!) write speed and latency is unacceptable.
read speed equal to X-RAID (raid5) and limited to 2*1gbe.
PS C:\> C:\SQLIO\sqlio.exe -s90 -kR -frandom -b8 -t8 -o16 -LS -BN D:\testfile.dat sqlio v1.5.SG using system counter for latency timings, 2474044 counts per second 8 threads reading for 90 secs from file D:\testfile.dat using 8KB random IOs enabling multiple I/Os per thread with 16 outstanding buffering set to not use file nor disk caches (as is SQL Server) using current size: 1048576 MB for file: D:\testfile.dat initialization done CUMULATIVE DATA: throughput metrics: IOs/sec: 25469.41 MBs/sec: 198.97 latency metrics: Min_Latency(ms): 0 Avg_Latency(ms): 4 Max_Latency(ms): 63 histogram: ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+ %: 12 8 9 12 10 12 10 9 10 2 2 1 1 1 1 0 0 0 0 0 0 0 0 0 0
Of course I can never say we have no bugs but I can say we don't have an open one for MPIO at this time so we are happy to help if can.
Hoping for you, because after the third online chat escalltion I lose heart
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
iSCSI without MPIO (1 NIC)
PS C:\Windows\system32> C:\SQLIO\sqlio.exe -s90 -kW -frandom -b8 -t8 -o16 -LS -BN o:\testfile.dat sqlio v1.5.SG using system counter for latency timings, 2728190 counts per second 8 threads writing for 90 secs to file o:\testfile.dat using 8KB random IOs enabling multiple I/Os per thread with 16 outstanding buffering set to not use file nor disk caches (as is SQL Server) using current size: 409600 MB for file: o:\testfile.dat initialization done CUMULATIVE DATA: throughput metrics: IOs/sec: 19.60 MBs/sec: 0.15 latency metrics: Min_Latency(ms): 3217 Avg_Latency(ms): 6452 Max_Latency(ms): 7558 histogram: ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+ %: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
Absolutley your results are not good howvere so no arguement there.
It wiill be hard to debug this here becaue I don't have an easy answer or configuartion change to make so will take a bit more back and forward.
If it is a software issue we need to have support case so can log issue.
Unfortunately we will not be able to log bug against a Pro6 running OS 6. So would need the data from the rn4220 if get that far to say is bug.
For now I would like to see reults with no MPIO and just single NIC so can start with basline with your intiator setup.
Just so you know I will be offline for a few days incase you think I am ignoring you.
mdgm or one of the others here may continue to help / provide advise
An internal request has already been logged to do some documentation around MPIO and I have asked could we try get some basline performance numbers from one of our labs with windows server 2012
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
So would need the data from the rn4220 if get that far to say is bug.
4220 is in production, so why I don't want to make some hard experiments.
here is SQLio test at 4220 (one NIC)
PS C:\Windows\system32> C:\SQLIO\sqlio.exe -s90 -kW -frandom -b8 -t8 -o16 -LS -BN h:\testfile.dat sqlio v1.5.SG using system counter for latency timings, 2078193 counts per second 8 threads writing for 90 secs to file h:\testfile.dat using 8KB random IOs enabling multiple I/Os per thread with 16 outstanding buffering set to not use file nor disk caches (as is SQL Server) using current size: 102400 MB for file: h:\testfile.dat initialization done CUMULATIVE DATA: throughput metrics: IOs/sec: 19.01 MBs/sec: 0.14 latency metrics: Min_Latency(ms): 834 Avg_Latency(ms): 6583 Max_Latency(ms): 7838 histogram: ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+ %: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 PS C:\Windows\system32>
it's X-raid (12*4Tb WD RE 4001-ffsx)
just after this test WEbUI stops working (3 minutes of "Connecting to ReadyNAS admin page.." and "readynas admin page is offline) and SMB shares were reconnected. FURY!!!!
it's freesing or lagging - as you wish to name it.
two minutes more - and it's alive again.
now I can confirm this bug - heavy iSCSI load makes 4220 unstable.
got fresh logs - can upload it or maybe you can create new case (previous was 25222344).
For now I would like to see reults with no MPIO and just single NIC so can start with basline with your intiator setup
look at my previous post (or this one).
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
Please open a new case and attach your logs. Let me know the case number.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
Can you update to 6.3.5-RC2 if not already running that?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
now it's 6.4.0. T34 @ RN4220 😃
and still nothing 😉
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
some fresh news (RN4220 NTGR highest device!!!!):
even with 6.4.0 T34 2*1gbe for Write is unavailable - only ~50-60MB/sec write speed with awesome latency (300-2000), so the storage is almost unusable.
w/o MPIO (1 NIC) it's a little faster and can work at 60-70MB/sec with the same shoking latency, but all other services like SMB or GUI is out of service during iSCSI high load.
conclusion is simple - at these days NTGR devices has no MPIO and very limited iSCSI support. and this is not Enterprise nor MID-Enterprise level devices.
I do not know what does your QA department do, but obviously something wrong.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
new case
25526961
RN4220 is inaccessible within default subnet
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
incredible support team answer:
"if device is not halted now I can't escalate case, let's wait.
are Netgear still joking?
or Russian Support focused on home routers with "reset" option for any case???!!
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: MPIO - slow speed
I understand the case was escalated to L3 support and some recommendations were made regarding the network layout. Please let us know if you are seeing improvements now.