× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: Average transfer speed (read and write) plummeted after replacing hard drives

EstivalG
Aspirant

Average transfer speed (read and write) plummeted after replacing hard drives

Hello everyone

TL;DR: I just replaced my 3TB hard drive with some brand new 8TB hard drive, and the average speed plummeted, from 70MB (switch 1)/100MB (switch 2) to a steady (but slow) 40MB on both switches.

 

Long version:

I bought this ReadyNAS RN102 6 years ago along with two 1.5TB hard drives, and created an X-RAID array called data (yes, I know 😄 )

One year later, I replaced the 1.5TB had drives with brand new Western Digital RED WD30EFRX 3To hard drives, keeping the same array data, using the XRAID option to automatically expand the array size, which was 2.7To, and working perfectly well (70-100MB transfer speed, depending solely on which switch is used.)

The available space shrank dramatically recently, so I bought two brand new Seagate IronWolf 8 TB, ST8000VN0022 hard drives, tested it on windows (and speed was greater than 100MB/s on both drives). So I changed one drive, waited for resync, changed another drive, waited for resync and expand, and voila, 7.2TB available.

But I noticed a low speed between the RN102 and my linux computer (which use NFS and usually works at 50MB/s), about 20MB/s. Weird. I then try a 37GB file read from the NAS to my Windows 10 Desktop (CIFS/SMB) and got a slow 40MB/s transfer speed, instead of 70MB/s speed (switch 1)

I then tried the same 37GB file read from the NAS to my Windows 10 Laptop (CIFS/SMB), hooked on the switch 2, and got 40MB/s as well, instead of a whooping (and usual) 100MB/s transfer speed (I usually got 60-70MB on the dual ac wifi connection, but the current speed is now 40MB/s...).

I then tried every maintenance option I found: Test drive, defrag, balance, scrub. No change.

 

Do you have any idea to get back a standard speed?

The NAS has no apps, and antivirus is disabled.

 

Last chance, destroying the array and rebuild a brand new one, which is possible since the WD 3TB hard drives are readable on Linux, but will be extremely long and painful to transfer all the data from the old drive to the new ones. And the speed may stay at 40MB/s...

But before destroying the array, I'd like to try all other options I've got before.

 

Thanks in advance!

Model: RN102|ReadyNAS 100 Series
Message 1 of 13
StephenB
Guru

Re: Average transfer speed (read and write) plummeted after replacing hard drives

If this is still original RAID array, I do recommend doing a factory default and reloading it from backup.  I did that a couple of years ago, and saw a significant performance improvement.  

 

Message 2 of 13
Hopchen
Prodigy

Re: Average transfer speed (read and write) plummeted after replacing hard drives

Hi@EstivalG 

 

A RN102 NAS is the entry level NAS. I am actually surprised that you got 70MB/s + speed before. I had one a while ago for testing various things but I would typically be in the range of 30-50MB/s and that was with a 2TB RAID 1.


Expanding the volume size to 8TB is going to take more processing for the single core ARM CPU and 500MB of ram, housed in the NAS. It is not really unexpected that the speeds are going to drop as a result.

 

You can of course try to factory default the unit in order to get a clean filesystem. However, with 8TB you are probably starting to hit the upper limit of what the unit can realistically handle.

 

Perhaps it is time to upgrade NAS since it seems your data needs have increased? I reckon that if you monitor the NAS via top command (or similar) while transferring data you will probably see the unit being flat out.

 

 

Message 3 of 13
StephenB
Guru

Re: Average transfer speed (read and write) plummeted after replacing hard drives

FWIW, with my own RN102 I was seeing 70 MB/s read speeds after the reset.  I was using it as a backup NAS - configured as jbod with 6 TB and 8 TB disks.  That was a few years back though, and the memory footprint of the current firmware is bigger now than it was back then.

 

 

Message 4 of 13
EstivalG
Aspirant

Re: Average transfer speed (read and write) plummeted after replacing hard drives

Hi @Hopchen 

I was quite happy with a 70MB/s speed, but it seems that the speed is very dependant of:

_The computer. My previous desktop topped at 50MB/s

_The switch. With a less than 2 years gaming laptop and a new switch, I got 100MB/s with a big file.

 

I never monitored the unit through top before, and I now got a 70-80% system cpu usage (read or write), which is HUGE. I don't remember such a high system usage, even in my earliest linux days (in the 20th century!! Ho, my!)

 

Since the only option left is to rebuild a new array, I will try a JBOD setup first, as @StephenB suggested, and check for the system usage, and see if the transfer speed is better.

And I will post here the results in a few weeks.

 

edit: Just tried a direct write:

dd bs=1M count=2048 if=/dev/zero of=test2

2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 35.0169 s, 61.3 MB/s

No I/O wait (wa stays at 0.0%) but a 90% system cpu usage. I think JBOD setup will be better but I guess that RAID1 configuration will be quite slow, I will check if disabled XRAID is better.

 

Thank you everyone 😉

Message 5 of 13
Retired_Member
Not applicable

Re: Average transfer speed (read and write) plummeted after replacing hard drives

Hi @EstivalG , I can confirm the 40MB/s transfer speed. I get that from my retired RN104 with two volumes (RAID0 and RAID1), which I'm using as a guineapig for trying things out.

If you like you can try some hints in the below link and see, whether you can improve performance of your RN102 in general (In essence RN102 and RN104 do only differ in the number of bays they hold).

Should you find a way to boost the transfer speed back to 70MB/s using the more recent firmware releases, I will be more than happy to adopt your approach as soon as it will be available.

Kind regards

https://community.netgear.com/t5/ReadyNAS-Idea-Exchange/Improving-performance-and-stability-of-RN104...

Message 6 of 13
Hopchen
Prodigy

Re: Average transfer speed (read and write) plummeted after replacing hard drives

Good work on that article @Retired_Member  🙂

 

 

Message 7 of 13
StephenB
Guru

Re: Average transfer speed (read and write) plummeted after replacing hard drives


@Retired_Member wrote:

Hi @EstivalG , I can confirm the 40MB/s transfer speed. I get that from my retired RN104 with two volumes (RAID0 and RAID1), which I'm using as a guineapig for trying things out.

 


I decided to benchmark my own RN102, and I discovered that my speed depends on the SMB protocol version.  I tweaked the the max protocol with SMBPlus, and tested the results with NasTester 1.7.  The measured read speeds are:

  • SMB 1.0: 41 MB/sec
  • SMB 2.1: 70 MB/sec
  • SMB 3.1: 63 MB/sec

When I reported my 70 MB/sec results a couple of years back, I was still running Windows 7 (SMB 2.1) so the results haven't changed with newer firmware.

 

The drop off with SMB 3.1 isn't that surprising - I suspect Windows 10 is using transport encryption, and the RN102 has no hardware support for that.

 

The slow speed with SMB 1 was a bit surprising to me.

 

Test Details:

The RN102 is running 6.9.5, and the volume is XRAID (two 1 TB Ironwolf disks).  Bit-Rot protection (CoW) is off on the share; Volume Quota is enabled.   Strict sync is disabled, and SMB 3 transport encryption is set to "enabled".  Antivirus is off.   SMBPlus is the only app running on the NAS.

 

The PC is an older desktop running Windows 10; the PC is using 10 gigabit ethernet, but of course the NAS is using just gigabit.  The PC hardrive is an SSD.  

 

The NAStester results are:

 

SMB 1.0

Running a 400MB file write on \\10.0.0.13\Documents 5 times...
Iteration 1: 39.43 MB/sec
Iteration 2: 39.64 MB/sec
Iteration 3: 40.24 MB/sec
Iteration 4: 38.87 MB/sec
Iteration 5: 39.67 MB/sec
-----------------------------
Average (W): 39.57 MB/sec
-----------------------------
Running a 400MB file read on \\10.0.0.13\Documents 5 times...
Iteration 1: 40.36 MB/sec
Iteration 2: 40.71 MB/sec
Iteration 3: 39.82 MB/sec
Iteration 4: 41.65 MB/sec
Iteration 5: 40.67 MB/sec
-----------------------------
Average (R): 40.64 MB/sec

 

SMB 2.1

Running a 400MB file write on \\10.0.0.13\Documents 5 times...
Iteration 1:     68.44 MB/sec
Iteration 2:     67.45 MB/sec
Iteration 3:     66.73 MB/sec
Iteration 4:     66.87 MB/sec
Iteration 5:     68.45 MB/sec
-----------------------------
Average (W):     67.59 MB/sec
-----------------------------
Running a 400MB file read on \\10.0.0.13\Documents 5 times...
Iteration 1:     75.39 MB/sec
Iteration 2:     67.61 MB/sec
Iteration 3:     67.51 MB/sec
Iteration 4:     68.58 MB/sec
Iteration 5:     69.27 MB/sec
-----------------------------
Average (R):     69.67 MB/sec

 

SMB 3.1

Running a 400MB file write on \\10.0.0.13\Documents 5 times...
Iteration 1: 69.35 MB/sec
Iteration 2: 68.94 MB/sec
Iteration 3: 69.08 MB/sec
Iteration 4: 67.34 MB/sec
Iteration 5: 69.00 MB/sec
-----------------------------
Average (W): 68.74 MB/sec
-----------------------------
Running a 400MB file read on \\10.0.0.13\Documents 5 times...
Iteration 1: 63.87 MB/sec
Iteration 2: 63.06 MB/sec
Iteration 3: 61.22 MB/sec
Iteration 4: 63.89 MB/sec
Iteration 5: 64.10 MB/sec
-----------------------------
Average (R): 63.23 MB/sec

 

Message 8 of 13
StephenB
Guru

Re: Average transfer speed (read and write) plummeted after replacing hard drives


@EstivalG wrote:

 

dd bs=1M count=2048 if=/dev/zero of=test2

2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 35.0169 s, 61.3 MB/s

 


What disks are you using?

 

Message 9 of 13
Retired_Member
Not applicable

Re: Average transfer speed (read and write) plummeted after replacing hard drives

Thanks for the detailed analysis, @StephenB , very helpful.

Message 10 of 13
EstivalG
Aspirant

Re: Average transfer speed (read and write) plummeted after replacing hard drives

Hello everyone, and many thanks for the replies.

@StephenB I now use two IronWolf 8TB: ST8000VN0022, which are present on the RN102 compatibility list.

 

I used the unit solely as a dedicated storage with no apps or whatsoever, I didn't know there is some tweaks available. So, before destroying the array, I did some test by tweaking SMB daemon with SMBPlus, but it didn't goes well: still 35-40MB/s

So, the old 3TB was readable, all my data was safe, I did destroy the array and recreate a new one with only one disk, in JBOD mode.

the dd test was not the same:

dd bs=1M count=2048 if=/dev/zero of=test2
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 17.3837 s, 124 MB/s

top still shows a 90% system cpu usage, but it is now a normal value for a 7200 hard drive.

 

I then tried to write a 36GB file, and speed is back to 70MB/s, which is really great.

I tried to write the same 36GB file from a linux computer, through NFS, and like @Retired_Member said, this is bad: it spawns a lot of nfsd daemons, and the ARM single core can't cope: the speed is below 40MB/s

So, I tried CIFS mount on linux and got a not so bad 58-60MB/s.

And I tried to get back that file on a windows computer, copying on a SSD: 52-55MB/s. Aouch.

Since I have two switchs and was aware that one of them was not very efficient, I tried to hook up the linux computer to the second switch, to see the results: it's way better.

On Linux, switch 2, reading from the NAS and writing on a SSD, I got 70-80MB/s, with the NAS with 20-40% idle. Woa. The linux computer is a P5B Deluxe (yay, 10 years!!) so the NICs may be a little old. And slow.

But I tried the windows 10 computer hooked on the switch 2, and still got 52-55MB/s with a 70% system CPU Usage. I check SMBPlus and the setup was 2.0 minimum and 2.1 maximum, like I setup before destroying the array. On linux, the mount is set on version 2.1.

There is something with the windows 10 read performance, but...

Since the linux computer have a windows 10 system as well, I rebooted on windows 10 on this computer, and try to read the 36GB file: I got the EXACT same speed, with a 70% system CPU Usage. It looks like the way Windows 10 read the file on the NAS put a lot of system stress on it. There is something I don't understand on the way Windows 10 read the file on the NAS.

 

But, I need to have a RAID 1 NAS as soon as possible, so, having sorting speeds out, I have some data to compare between JBOD mode and RAID1 mode, let's try it! I will dig up the Windows 10 read performance later.

 

I created a new RAID 1 array, it is resyncing right now and it may take a while, since mdstat stats a 100MB/s speed, lead to a 22 hours syncing time at current speed. But speed usually plummets as it goes near the end of the disk.

As soon as the RAID 1 Array is ready, I will try it and post the performance results here.

 

Thanks again for the help!

Message 11 of 13
StephenB
Guru

Re: Average transfer speed (read and write) plummeted after replacing hard drives

Thx for updating us, and please do let us know the speed when the sync finishes.

 

Are you running 6.9.5 firmware?

Message 12 of 13
EstivalG
Aspirant

Re: Average transfer speed (read and write) plummeted after replacing hard drives

So, this is now the real deal

_OS 6.9.5

_RAID 1

_Seagate IronWolf 8 TB, ST8000VN0022 (both)

_X-RAID is OFF

_Quota and Antivirus is OFF

 

Tests on Linux

Although I planned to disable NFS server, I found out that you can set the number of nfsd forks. On a single core ARM CPU, the default number (8) of forks seems utterly stupid, so I changed this value to 1.

Unfortunately, it didn't go up to 40MB/s (write on NAS) (at start, the speed is 100MB/s, due to the speed of the linux hard drive and linux disk cache, so I assumed the linux hard drive is quick enough to go beyond 40MB/s). Average is 32-35MB/s.

Since there is no I/O Wait, I tried 2 forks and it seemed slightly better (36-38MB/s) but I don't think you can get something better with more forks.

However, reading from NAS (2 forks) is fairly good: 62-66MB/s (write on the linux computer's SSD)

In all cases, top shows a high sys CPU usage and si CPU usage (from 60-40 to 70-30) with no wa CPU usage.

 

Next, CIFS ver 2.1. I kept NAS CIFS's version to 2.1 maximum, to avoid encryption overhead on the NAS CPU. Mount point on linux is fixed as well: mount.cifs //192.168.1.250/Video /media/target/ -o user=foobar,vers=2.1

Write on NAS show a speed from 60 to 70MB/s, with an average of 62-64MB/s. Quite good!

Read from NAS show a speed from 70 to 80MB/s, with an average of 74-76MB/s. Excellent! And what's more, NAS is not at full CPU, showing a 10-20% idle, so I think the NAS may provide a better reading speed with a better Linux computer.

 

Tests on Windows 10

Write on NAS show a speed from 60 to 70MB/s, with an average of 62-64MB/s. Quite good!

Reading from NAS show a speed from 52 to 55MB/s, with an average of 54MB/s.

I'm quite surprised by this. CIFS Write speeds are the same on both OS - and top shows similar CPU usage - but read speeds are really different, and CPU usage shows differences as well. When a Windows 10 reads a file, there is no CPU idle (using my old linux computer shows CPU idle), CPU sys and CPU si is very high (usually 70-30)

I tried different Windows computers (2013 desktop and 2017 gaming laptop), with the same ethernet cable and switch, and it seems speed is locked at 54-55MB/s on Windows 10. I no longer have any Windows 7 computer.

I check NAS' SMBPlus settings, encryption is OFF and maximum version is 2.1.

 

Since Linux CIFS write is quite good, I will now start the backup restore from the old 3TB hard drive.

But if you have any clue to understand why Windows 10 reading speed is lower than Linux reading speed, feel free to post 😄

 

Thanks in advance!

Message 13 of 13
Top Contributors
Discussion stats
  • 12 replies
  • 3871 views
  • 3 kudos
  • 4 in conversation
Announcements