× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

RN214 - Extremely slow with zip files

Zurd
Star

RN214 - Extremely slow with zip files

Hi, I have a ReadyNAS 214 and it used to be fine before but since about 6-8 months or so, it is extremely painfully slow to compress / decompress zip files. Is it normal? Why would it be so slow? Anyone else has the same problem?

 

For example, a zip file of 836 MB which contain 1,597 folders and 25,230 files: if I try to decompress it in Linux (desktop, wired ethernet cable) in the GUI, after 2h and 30 minutes, it is still not finished, about 70% done. However, in a terminal it takes 48 minutes. If I log in SSH to the ReadyNAS214 and use the unzip command, it takes 49 seconds. If I copy the zip file to my local mechanical hard drive, it only takes 56 seconds. And finally, if I try to decompress it in Windows (laptop, wireless) it takes about 1h.

 

Here's a summary:

Desktop - Linux - Wired - GUI = over 3h

Desktop - Linux - Wired - Terminal = 48m

RN214 - SSH = 56s

Laptop - Windows - Local mechanical HD = 56s

Laptop - Windows - Wireless = about 1h

 

The CPU reach 52 celsius only when decompressing, all is fine. All drives are green, fans is spinning fine.

 

My ReadyNAS214 is encrypted. It says Status: Healthy, Antivirus: Disabled, File Search: Disable and I have the latest firmware 6.10.3 as of today. The network is in bond0 in Round-Robin with 2 network cables as it always been for a long time.

 

I only have one volume in READ 5. The speed performance with big files is fast, no problem with network.

 

There is no installed applications, no Cloud or Backup services configured. I have enabled the services of SMB, NFS, Rsync, HTTP, HTTPS and SSH.

 

I have 4 hard drives HGST HDN726040ALE614 in it, full specs here: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/pro...

Model: RN214|4 BAY Desktop ReadyNAS Storage
Message 1 of 13
StephenB
Guru

Re: RN214 - Extremely slow with zip files

It's not something I do much.  My setup is different (a 526x NAS using a 10gbit network - and no encryption).  If I have a chance, I'll try to duplicate your results on a Windows PC and see how that compares.  I use 7-zip (not sure if that will make a difference or not).

 

 

Message 2 of 13
StephenB
Guru

Re: RN214 - Extremely slow with zip files


@StephenB wrote:

 I'll try to duplicate your results on a Windows PC and see how that compares.  I use 7-zip (not sure if that will make a difference or not).


Turns out the 7-zip command line interface is a bit restrictive - it won't allow me to put an output zip on the SSD when the source is on the NAS.

 

However, I was able to compare SSD->SSD performance on my PC with NAS->NAS performance using a mapped drive on the PC.  While the NAS->NAS performance was slower, it wasn't anything like what you are seeing.  My test folder has 5884 files in 177 folders, and a total size of about 500 MB.

 

I suggest trying your test w/o the round-robin bonding - just to see if that is a factor.  You could also try to duplicate my results with 7za.  

 

Local SSD->Local SSD Zip Creation took about 12 seconds (Global Time is the actual wall clock time).

Test1:  Create Zip SSD->SSD
7za a -bt -tzip LocalTestSrc

Files read from disk: 5884
Archive size: 297220124 bytes (284 MiB)
Everything is Ok

Kernel  Time =     1.968 =   16%                 132515 MCycles
User    Time =    38.171 =  325%
Process Time =    40.140 =  342%    Virtual  Memory =    150 MB
Global  Time =    11.722 =  100%    Physical Memory =     90 MB

NAS->NAS Zip creation took about 40 seconds - definitely longer, but not minute/hours.

Test2:  Create Zip NAS->NAS (526x mapped drive)
7za a -bt -tzip NASTestSrc

Files read from disk: 5884
Archive size: 297195876 bytes (284 MiB)
Everything is Ok

Kernel  Time =     4.593 =   11%                 146766 MCycles
User    Time =    39.234 =   99%
Process Time =    43.828 =  111%    Virtual  Memory =    150 MB
Global  Time =    39.332 =  100%    Physical Memory =     48 MB

FWIW, the process time and cycle count is really quite close.  I'm guessing the difference is SMB latency.

 

Local SSD-SSD extraction also took about 12 seconds:

Test3:  Extract ZIP SSD->SSD
7za x -bt -tzip LocalTestSrc.zip

Folders: 178
Files: 5884
Size:       486495926
Compressed: 297220124

Kernel  Time =     6.875 =   58%                  35717 MCycles
User    Time =     3.843 =   32%
Process Time =    10.718 =   91%    Virtual  Memory =      5 MB
Global  Time =    11.736 =  100%    Physical Memory =      9 MB

 

 

NAS->NAS extraction took about 80 seconds

Test4:  Extract ZIP NAS->NAS (526x mapped drive)
7za x -bt -tzip NASTestSrc.zip

Folders: 178
Files: 5884
Size:       486495926
Compressed: 297195876

Kernel  Time =    14.796 =   18%                  62914 MCycles
User    Time =     3.453 =    4%
Process Time =    18.250 =   23%    Virtual  Memory =      5 MB
Global  Time =    78.821 =  100%    Physical Memory =      9 MB

Process time and cycle counts are higher than the local extraction test, but I think the main difference is again SMB latency.

 

I then repeated the NAS->NAS tests on an RN202.  I'm not using bonding (and I don't use volume encryption).  The share is on a jbod volume.

 

The NAS->NAS zip creation results were similar to the RN526 performance. The extraction test was quite a bit worse (165 seconds, compared with 79).  Still not minutes/hours though.

 

 

 

 

 

Message 3 of 13
Sandshark
Sensei

Re: RN214 - Extremely slow with zip files

Total guess, but try disabling strict sync on the share.  That causes Veracrypt to slow to a crawl on writes, and this may be related.  If you are running SMB Plus and have enabled pre-allocation, you should also try disabling that.

Message 4 of 13
StephenB
Guru

Re: RN214 - Extremely slow with zip files


@Sandshark wrote:

Total guess, but try disabling strict sync on the share.  That causes Veracrypt to slow to a crawl on writes, and this may be related.  If you are running SMB Plus and have enabled pre-allocation, you should also try disabling that.


Strict sync was enabled in my measurements, so that isn't it.

 

I don't see how pre-allocation would affect extraction to the local PC hard drive, since the NAS is only being read.  

 

Could the zip file (or the share) be heavily fragmented?

How full is the volume?

Message 5 of 13
Zurd
Star

Re: RN214 - Extremely slow with zip files

StephenB: I use 7zip too. Whatever you use will have a slight difference but that doesn't make much of a difference.

 

No 7zip in command line is not restrictive, I can output or zip with a command from and to a NAS, I've done it plenty of times on different computers, no idea what problem you're having but it should work.

 

You suggested that I disabled the bond so I did. I also removed one ethernet cable, so now I only have one ethernet cable plugged and here is the result with a 312.8 MB zipped file with 766 folders and 5517 files inside:

 

Laptop - Windows - From mechanical HD to mechanical HD = 16s

RN214 - SSH - From NAS to NAS =14.6s

Desktop - Linux - Wired - GUI - From NAS to NAS = 2m

Desktop - Linux - Wired - Terminal - From NAS to NAS =1m 54s

Laptop - Windows - Wireless - From NAS to NAS = 10m 14s

 

Ouch, that last one of 10m is quite something!

 

In my first test, if you compare the SSH extraction (56 seconds) to "Linux - Terminal" (48 minutes), the latter was 5143% slower.

 

In this test, the difference is 781% slower, that is much better than before! Looks like the bond0 I had wasn't good. It is true that I just plugged both cables in my router without any special configuration. However, it is still slow and still extremely slow in wireless (4206% slower), I'm having trouble wondering why that would be, why the network would make such a huge difference.

 

Sandshark: You suggested disabling "strict sync" which is in Share / Settings / Network Access / SMB / Advanced and here's the new results with the same file (I didn't rebooted the NAS, hopefully it doesn't matter):

 

Laptop - Windows - From mechanical HD to mechanical HD = 16s

RN214 - SSH - From NAS to NAS =14.6s

Desktop - Linux - Wired - GUI - From NAS to NAS = 2m

Desktop - Linux - Wired - Terminal - From NAS to NAS = 1m 54s

Laptop - Windows - Wireless - From NAS to NAS = 10m 11s

 

In this test, the strict sync did absolutely nothing unfortunately.

 

What is strist sync?

https://www.samba.org/samba/docs/using_samba/ch11.html

This share-level option determines whether Samba honors all requests to perform a disk sync when requested to do so by a client. Many Windows clients request a disk sync when they are really just trying to flush data to their own open files. In this case, a disk sync is generally unnecessary on Unix due to its high reliability, and it mostly has the effect of substantially reducing the performance of the Samba host system. The default value for this option is no, which allows the superfluous disk sync requests to be ignored.

 

Because of this, I think it's a good thing to not have it enabled. If you install Samba, it is not enabled by default. I wonder why Netgear would turn it on by default.

 

StephenB: About fragementation, there is an option in Volumes / Settings to Defrag. I will try it, it will take time so new results will be posted later.

 

About the free space, I have one share in RAID 5, 4.19 TB of data and 6.70 TB of free space total of 10.90 TB so I have enough free space.

 

So it is faster right now but I still think something is wrong, it should be faster. In your test of extraction (which are different tests that I did) from SSD to SSD (12s) compared to NAS to NAS (80 seconds), it is 667% slower, huge difference.

Model: RN214|4 BAY Desktop ReadyNAS Storage
Message 6 of 13
StephenB
Guru

Re: RN214 - Extremely slow with zip files


@Zurd wrote:

 

You suggested that I disabled the bond so I did. I also removed one ethernet cable, so now I only have one ethernet cable plugged and here is the result with a 312.8 MB zipped file with 766 folders and 5517 files inside:

The net here is that with bonding disabled we are getting similar results. That's assuming all your tests are extraction tests.  You could use -t to see how all the clocks compare.

 

One potential problem with round-robin bonding is that when you are going from 2 gbps->1gbps you create congestion (because the NAS can put data on the wire twice as fast as the client can accept it).  At some point the switch ends up dropping packets, because its buffers overun.  TCP congestion control then kicks in.  TCP congestion control is designed to back off the send rate sharply when packets are lost - to prevent the possibility of a network collapse.  So in this particular scenario, you end up underutilizing the network capacity.

 

You might want to check if you have ethernet flow control enabled in the switch (often it is not on by default).  If you enable that, you could try re-establishing the bond.

 


@Zurd wrote:

 

No 7zip in command line is not restrictive, I can output or zip with a command from and to a NAS, I've done it plenty of times on different computers, no idea what problem you're having but it should work.

The problem I ran into is that -o lets me extract to a different folder, but it is ignored when zipping (at least in windows).  No matter what I typed, the zip file was created in the folder I was zipping (not in the folder I was in).  So I could have extracted NAS->SSD, but I couldn't zip NAS->SSD. This was in windows.  I didn't spend a lot of time fiddling with it, as I normally don't use the 7za command line.  In any event, you can see the commands I actually tested, as they were included with my results.  

 


@Zurd wrote:

So it is faster right now but I still think something is wrong, it should be faster. In your test of extraction (which are different tests that I did) from SSD to SSD (12s) compared to NAS to NAS (80 seconds), it is 667% slower, huge difference.


This difference appears to be latency (you can see the huge gap between process times and global time in my slow extraction). 

 

So at the time I chalked that up to SMB.  Extraction presents a very different load from zipping (reading one large file -> writing lots of small files vs. reading lots of small files -> writing one large file).  

 

I'd expected that disabling strict sync would help on that particular test.  But you didn't find that, which is interesting. My suspicion is that SSD tiering on the NAS would fix it - something I've thought about using, but haven't gotten around to trying. 

 

If I get to it, I might try a NAS->SSD extraction test.

 


@Zurd wrote:

Because of this, I think it's a good thing to not have it [strict sync] enabled. If you install Samba, it is not enabled by default. I wonder why Netgear would turn it on by default.

 


For one thing, most NAS users aren't using linux clients. FWIW I think the Samba documentation might be over-touting "unix reliability". While my Windows systems often did crash many years back, I haven't found that to be the case with Windows 7 or Windows 10.

 

I suspect Netgear is prioritizing data safety over performance. With the setting disabled, more writes are cached in the NAS.  If the power on the NAS fails (or if it crashes) those writes are lost.  That can increase the odds that the RAID array will be out-of-sync (resulting in a corrupted file system or a RAID array that can't be mounted).   If the NAS is UPS protected, then you probably can safely disable it.

 

Message 7 of 13
Zurd
Star

Re: RN214 - Extremely slow with zip files

Defragmentation took only 3h for 4 TB, that was fast!

2020 Aug 04 23:05:18 Volume: Defragmentation complete for volume z.
2020 Aug 04 19:55:07 Volume: Defragmentation started for volume z.

 

And the results are exactly the same, so we can now rule out fragmentation:

Laptop - Windows - From mechanical HD to mechanical HD = 16s

RN214 - SSH - From NAS to NAS =14.6s

Desktop - Linux - Wired - GUI - From NAS to NAS = 2m

Desktop - Linux - Wired - Terminal - From NAS to NAS = 1m 54s

Laptop - Windows - Wireless - From NAS to NAS = 10m 11s

 

Thanks for the explanation of the round-robin bonding congestion problem, that explains the problem quite well. I kind of wonder if I really need bonding (which I think is called 802.3ad), I only have a few devices at home, all on 1 gpbs except the laptop which is wireless, it seems pretty useless in my case.

 

But it would be a good test to have it configured and do another extraction test. However, my router has OpenWrt 18.06.2 and it seems possible to do it but a bit tedious so I don't think I will try that. Reference: https://oldwiki.archive.openwrt.org/doc/howto/mwan3

 

I understand what you mean by process time and global time and SMB latency, it really makes sense. The RN214 has USB ports, if it is possible to plug it as an external storage in my desktop, then it should be much faster if we bypass the network then? I'm reading the documentation but I can't see anywhere if you can do that. Can a RN214 works as an external hard drive with USB?

 

Message 8 of 13
StephenB
Guru

Re: RN214 - Extremely slow with zip files


@Zurd wrote:

(which I think is called 802.3ad)

 


802.3ad is one method of bonding (also called LACP) - but that is not the method you were using.  Not all switches support LACP.

 


@Zurd wrote:

I understand what you mean by process time and global time and SMB latency, it really makes sense. The RN214 has USB ports, if it is possible to plug it as an external storage in my desktop, then it should be much faster if we bypass the network then? I'm reading the documentation but I can't see anywhere if you can do that. Can a RN214 works as an external hard drive with USB?

 


You have to use ethernet, there is no way to connect to your PC using USB.

 

You could test with NFS and see if that has less latency.

 

Message 9 of 13
Zurd
Star

Re: RN214 - Extremely slow with zip files

Too bad we cannot use the RN214 with USB as an external storage, that would have been a very interesting test.

 

For another test like you suggested, I just disabled SMB and enabled NFS, All squash and mapped to UID and GID 100 of my user in the ReadyNAS, then I mount it: sudo mount -t nfs 192.168.13.2:/VOLUME/SHARE /media/readynas/

 

For Windows, a bit more complex, you need to install "Service for NFS" in Control Panel, then modify the registry with AnonymousUid and AnonymousGid with a value of zero, reboot and run in terminal: mount -o anon \\192.168.13.2\VOLUME\SHARE Z: (there's is tutorial about this, just google it)

 

And here's the new results:

Laptop - Windows - From mechanical HD to mechanical HD = 16s

RN214 - SSH - From NAS to NAS =14.6s

Desktop - Linux - Wired - GUI - From NAS to NAS = 37s

Desktop - Linux - Wired - Terminal - From NAS to NAS = 31.5s

Laptop - Windows - Wireless - From NAS to NAS = 31m56

 

Wow! What a big difference in Linux! In terminal, it's now only 215% slower compared to extracting in SSH right in the NAS.

 

However, it is MUCH worse in Windows! It is 3 times slower, ouch. Seems like NFS doesn't play nicely in Windows. It was better with SMB.

 

In any case, it is more complex to use NFS than SMB and I read on the internet it is less secure than SMB and it's kind of deprecated today. Whatever the results was, I will still use SMB.

 

So, since it's really an issue of network latency and/or network protocol (SMB/NFS) overhead, that means that in the future, when my network is all 10 gbps (router, switch, ethernet card, cables), then it should be faster?

 

Message 10 of 13
StephenB
Guru

Re: RN214 - Extremely slow with zip files


@Zurd wrote:

So, since it's really an issue of network latency and/or network protocol (SMB/NFS) overhead, that means that in the future, when my network is all 10 gbps (router, switch, ethernet card, cables), then it should be faster?

 


My tests were using a 10 gbps network (though I didn't try disabling strict sync).

Message 11 of 13
Zurd
Star

Re: RN214 - Extremely slow with zip files

Interesting, I would have guess that a 10 gbps network would have made a much more noticeable difference.

 

It's not a big test but I just tried something new. Before, I had my desktop wired into a small unmanaged switch (Netgear GS308v2) which goes into my router then the modem and my RN214 was going into my router. Now, both desktop and RN214 goes into the switch.

 

Laptop - Windows - From mechanical HD to mechanical HD = 12s

RN214 - SSH - From NAS to NAS = 14.8s

Desktop - Linux - Wired - GUI - From NAS to NAS = 1m52

Desktop - Linux - Wired - Terminal - From NAS to NAS = 1m51

Laptop - Windows - Wireless - From NAS to NAS = 10m27s

 

Pretty much no changes, it was kind of to be expected. Interesting to note that 7z decompress a file in terminal in 1m51 and the unzip command on the same file takes 4m42.

 

I might have another test to do with a managed switch in the future (just for fun) but for now you could consider this resolved by removing the bond I had because I didn't had any ethernet flow control in my router, now I'll just use one ethernet cable. I still find it slow but I guess that's to be expected from a NAS 🙂

 

Thank you very much StephenB, you have been very patient and helpful!

Message 12 of 13
StephenB
Guru

Re: RN214 - Extremely slow with zip files

I'm glad I could help.

Message 13 of 13
Top Contributors
Discussion stats
  • 12 replies
  • 2472 views
  • 1 kudo
  • 3 in conversation
Announcements