Very Slow Read/Write and Time Machine Backup

---

SoundmanArt wrote:

 

Yes, I did get the Message "too long message".

Is it an issue with full-duplex and that's why it needed to be 1/2?

 

---

It's not a duplex issue - the MTU (maximum transmission unit) is the largest packet size your network can handle.  If you set it too large in the Mac or the NAS you'll end up with packet fragmentation and sometimes loss- both of which hurt performance. 

9000 is too large for your network.  There's something on the path that is limiting the size (probably a switch or router).

---

SoundmanArt wrote:

 

Yes, I did get the Message "too long message".

 

I also tried it back at MTU=1500 and changed -s to 1492.  I also got that message.

So I cut -s to 740 and it worked fine.

So I went back to MTU=9000 and tried -s 4492.  That worked fine.

---

Try setting the MTU back to 9000, and find the largest -s that doesn't get the message.  Add 28 to that value, and you'll get the biggest MTU your network supports.

Alternatively, set the MTU back to 1500 and try -s=1472.  That shouldn't give you the message (though -s=1492 will, as it is too big for that MTU).  

The reason for the mismatch between MTU size and -s:  The MTU size includes the 28 byte packet header for ping.  The -s value only includes the payload size.

If you have older devices on your network that use fast ethernet (100 mbps link speed), you should stick with MTU=1500, as those devices can't handle anything larger.

The main speed benefit from larger MTUs comes from reducing the interrupts per second in the devices - since there are fewer packets per second with the larger size.  There is a slight network efficiency gain, but it is inconsequential.  So once you find the max MTU for your network, you should do a speed test with both MTU=1500 and MTU=max value.  Just do a single large file transfer (500 MB for example).  You want a test that consistently uses the larger packets, not one uses the mixed sizes.

The answer is 8164.  But Ping tells me that it transmitted 8172, not 8192 as you would have thought.

---

SoundmanArt wrote:

The answer is 8164.  But Ping tells me that it transmitted 8172, not 8192 as you would have thought.

 

---

I don't have a Mac, and the output does vary by platform.  But I suspect that ping might be including the 8 byte ICMP header in it's output, but not the 20 byte IP header.

So if all your wired devices support jumbo frames you can test a single large file transfer (both read and write) with both MTU=8192 and MTU=1500 on both the NAS and the Mac, and see which gives you better performance.  It'd be helpful to post those results here.

FWIW, if it's close I'd just stick with MTU=1500 (which is the ethernet standard).

If you go with MTU=8192, you can also repeat the ping test with -s=8164, just to confirm that it's working correctly.

Sorry for the delay.  I had a 1TB folder with 1.2 million files that I needed to get off the server and onto my computer.  It took 18 hours to copy down.

After it copied, I tested the speed of  MTU=1500 vs MTU=8192.  Virtually the same.  So I went back to 1500.

I then copied a 1 GB file from Mac -> server.  Time = 10 sec.  I was fine with that.

Then 1GB file from server -> Mac.  Also 10 seconds.

Then I created a 100 GB file and copied that.

Mac -> Server = 16:40 (minutes/seconds)

Server  -> Mac = 16:48

It appears that the problem is NOT throughput, but rather, it seems that the problem is with directories and/or numbers of files.  That's why it takes sooooo long to find the files to copy.

As I said, I'm running 2 drives with JBOD.  Before that I had 4 drives with RAID 1 and it was slow.

Thoughts?

Art

---

SoundmanArt wrote:

 

I then copied a 1 GB file from Mac -> server.  Time = 10 sec.  I was fine with that.

Then 1GB file from server -> Mac.  Also 10 seconds.

 

Then I created a 100 GB file and copied that.

 

Mac -> Server = 16:40 (minutes/seconds)

Server  -> Mac = 16:48

 

It appears that the problem is NOT throughput,

---

I agree, you are getting 100 MB/s on large file transfers

---

SoundmanArt wrote:

but rather, it seems that the problem is with directories and/or numbers of files.  That's why it takes sooooo long to find the files to copy.

 

---

Yes.   To some degree this is inevitable.  Transfering lots of small files will require a lot more seek time, and also it's less efficient on the network.  But there might be some things that can be done to improve it.

Do you have strict sync enabled for the share?  (that's specific to AFP).    Look on the share settings wheel (Network Access->Advanced).  If it is enabled, try disabling it, and doing another test of small file transfer (looks for something less time consuming than ~ 1 million files).

A more expensive solution is to get some SSDs and use ReadyTier.

---

SoundmanArt wrote:

 

As I said, I'm running 2 drives with JBOD.  Before that I had 4 drives with RAID 1 and it was slow.

 

---

RAID-1 shouldn't be slower than JBOD.  Writes are done in parallel.  Reads also could be done in parallel (using both disk) - though I'm not sure if mdadm actually does that or not.

If you are interested in ReadyTier, probably the most robust approach would be to run RAID-1 with two mechanical disks, and use two SSDs in RAID-1 for metadata.  You would need to keep on eye on wear-levels for the SSDs (since both would reach their write limit at the same time).  That should substantially speed up small file transfers and directory browsing.

Do you have strict sync enabled for the share?  (that's specific to AFP).    Look on the share settings wheel (Network Access->Advanced).  If it is enabled, try disabling it, and doing another test of small file transfer (looks for something less time consuming than ~ 1 million files).

No.  I do not.  Also, for Time Machine, when you set up a Time Machine share, it now sets it up automatically as an SMB share.

I looked into ReadyTier.  Sounds interesting except 3 things...  1) I'm essentially going to lose 2 bays in my NAS to do this; 2) I'm going to have  to purchase 2 2TB SSD's for this to work with my 2 12 TB mechanical drives (it appears that it takes 1 MB for metadata for every 8 GB of data and 3) until about 6 months ago, this NAS worked just fine.

I'd hate to spend the money on 2 2TB drives, just to find out that it is still just as slow.

---

SoundmanArt wrote:

Do you have strict sync enabled for the share?  (that's specific to AFP).   

 

---

I mis-spoke here - strict sync is specific to SMB. It will kill performance for some applications.

---

SoundmanArt wrote:

 

I looked into ReadyTier.  Sounds interesting except 3 things...  1) I'm essentially going to lose 2 bays in my NAS to do this; 2) I'm going to have  to purchase 2 2TB SSD's for this to work with my 2 12 TB mechanical drives (it appears that it takes 1 MB for metadata for every 8 GB of data and 3) until about 6 months ago, this NAS worked just fine.

Yes, it does cost. two bays and two SSDs.

Where did you find the rule of thumb on metadata?  Page 11 of the optimization manual says something different: https://www.downloads.netgear.com/files/GDC/READYNAS-100/ReadyNAS_FlexRAID_Optimization_Guide.pdf

FWIW, I have an 18 TB volume with about 12.6 TiB of data.  My system shows about 15 GB of metadata - not the 1.5 TiB that your rule of thumb yields.

How much metadata does your system consume?  Note you can see this by hovering your mouse over the pie chart on system->volume.

---

SoundmanArt wrote:

 

I'd hate to spend the money on 2 2TB drives, just to find out that it is still just as slow.

 

---

Again, I don't see why they need to be that big.  But I agree it's an expense, and we don't really now how much it will help.

I mis-spoke here - strict sync is specific to SMB. It will kill performance for some applications.

I copied 58 GB to a share where I had strict sync turned off this a.m.  I got 48 MBS for that copy.  However, when you set up a Time Machine share, it doesn't give you the option of turning strict sync on or off.

Where did you find the rule of thumb on metadata? 

I was using my own data for the ratio.  Mine show 8.42 TB of data and 10.41 GB of metadata.  Duh!  You're right. I don't need nearly that much.  A misplaced decimal point.

I'll grab a couple today and try it.

---

SoundmanArt wrote:

I'll grab a couple today and try it.

 

---

Let us know how it turns out.

Will do. Just missed the cut off for same day Amazon shipping.  So it won't be until tomorrow until I get them.  But I've already changed the mechanical drives to RAID 0.  Just need to add the other 2 drives when I get them.

I did install an SSD into my rack and set up ReadyTier (it took me a couple of days because I thought it required 1 SSD for each mechanical drive rather than for each volume).  So that's now up and running.  I started up a Time Machine backup and just aborted it after 5-1/2 hours it has written 188 GB of data and 200 MB of metadata (or about 9.5 MB/second).  It took me 12 hours to backup the entire 8.8 TB of data to an attached storage RAID 1.

So I'm thinking this is not the solution.

---

SoundmanArt wrote:

 I started up a Time Machine backup and just aborted it after 5-1/2 hours it has written 188 GB of data and 200 MB of metadata (or about 9.5 MB/second).  It took me 12 hours to backup the entire 8.8 TB of data to an attached storage RAID 1.

 

---

Did you test straight SMB transfers (folders with lots of small files)?  Or just TM?

---

SoundmanArt wrote:

I did install an SSD into my rack and set up ReadyTier (it took me a couple of days because I thought it required 1 SSD for each mechanical drive rather than for each volume).  

 

---

Normally you'd use 2 SSDs in RAID-1 for the metadata RAID group.  That would give you redundancy if the drive fails.

However, using a single SSD shouldn't affect the speed much (if at all).

Did you test straight SMB transfers (folders with lots of small files)?  Or just TM?

I did not.  But I will now.  I have a 180 GB folder with 712k files.

I just got back to my desk.  After 22-1/2 hours, it had copied 160GB, so I stopped it.

I'm going to remove the 2 mechanical drives and put an SSD in there to see what happens with that.  That will at least potentially eliminate one area.

As a follow up, after 13 hours, it had only copied 22.5 GB to the SSD in the Rack.  That rules out issues with the mechanical drives.

I then took the SSD out of the rack and put it into a USB3 adapter that I had and copied it directly.  It copied all 180GB in 20 minutes.  I guess the SSD was ok, too.

---

SoundmanArt wrote:

As a follow up, after 13 hours, it had only copied 22.5 GB to the SSD in the Rack.  That rules out issues with the mechanical drives.

 

---

Was this with TM or SMB?

Do you have another PC you can test with?

Was this with TM or SMB?

SMB.

Do you have another PC you can test with?

Yes.  I will do that later today.  But as I recall, it was slow, too.

---

SoundmanArt wrote:

 

Yes.  I will do that later today.  But as I recall, it was slow, too.

 

---

Yes, but I think it's worth retesting it using a share on the SSD.

Ok.  I ran it 2 ways on a MacBook Pro.  Both tests ran for 3 hours.

180 GB folder

MacBook Pro -> Mechanical Drives = 15.59 GB

MacBook Pro -> SSD = 15.60 GB

It starts up fast on both, gettting the first 1-2 GB done in just a couple of minutes.  Then it comes to a halt.

---

SoundmanArt wrote:

Ok.  I ran it 2 ways on a MacBook Pro.  Both tests ran for 3 hours.

 

180 GB folder

 

MacBook Pro -> Mechanical Drives = 15.59 GB

MacBook Pro -> SSD = 15.60 GB

 

---

Certainly slower than it should be.

Perhaps try connecting the NAS directly to the MacBook???  That would require temporarily assigning a static IP address on both.

Perhaps try connecting the NAS directly to the MacBook???  That would require temporarily assigning a static IP address on both.

Did that with the NAS and the MacBook Pro connected directly.  I got the exact same amount copied -- 15.60 GB in 3 hours.

As with all of the other copies, it started out like gangbusters.  And then came came to a screaching halt.

So???  Is there a problem with my NAS that needs to be sent it under warranty?  Is there insufficient RAM?

This is crazy!

Art

---

SoundmanArt wrote:

Is there insufficient RAM?

---

It has 4 GB, which should be enough.

---

SoundmanArt wrote:

 

Did that with the NAS and the MacBook Pro connected directly.  I got the exact same amount copied -- 15.60 GB in 3 hours.

 

---

If it's still connected can you take a look at the network stats in the log zip?

I had disconnected it.  But here's the log file.

Were you using eth0 or eth1?

ETH0.  ETH1 was disconnected while I was doing the copy.