- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
Very Slow Read/Write and Time Machine Backup
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
Sorry for the delay. I had a 1TB folder with 1.2 million files that I needed to get off the server and onto my computer. It took 18 hours to copy down.
After it copied, I tested the speed of MTU=1500 vs MTU=8192. Virtually the same. So I went back to 1500.
I then copied a 1 GB file from Mac -> server. Time = 10 sec. I was fine with that.
Then 1GB file from server -> Mac. Also 10 seconds.
Then I created a 100 GB file and copied that.
Mac -> Server = 16:40 (minutes/seconds)
Server -> Mac = 16:48
It appears that the problem is NOT throughput, but rather, it seems that the problem is with directories and/or numbers of files. That's why it takes sooooo long to find the files to copy.
As I said, I'm running 2 drives with JBOD. Before that I had 4 drives with RAID 1 and it was slow.
Thoughts?
Art
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
@SoundmanArt wrote:
I then copied a 1 GB file from Mac -> server. Time = 10 sec. I was fine with that.
Then 1GB file from server -> Mac. Also 10 seconds.
Then I created a 100 GB file and copied that.
Mac -> Server = 16:40 (minutes/seconds)
Server -> Mac = 16:48
It appears that the problem is NOT throughput,
I agree, you are getting 100 MB/s on large file transfers
@SoundmanArt wrote:
but rather, it seems that the problem is with directories and/or numbers of files. That's why it takes sooooo long to find the files to copy.
Yes. To some degree this is inevitable. Transfering lots of small files will require a lot more seek time, and also it's less efficient on the network. But there might be some things that can be done to improve it.
Do you have strict sync enabled for the share? (that's specific to AFP). Look on the share settings wheel (Network Access->Advanced). If it is enabled, try disabling it, and doing another test of small file transfer (looks for something less time consuming than ~ 1 million files).
A more expensive solution is to get some SSDs and use ReadyTier.
@SoundmanArt wrote:
As I said, I'm running 2 drives with JBOD. Before that I had 4 drives with RAID 1 and it was slow.
RAID-1 shouldn't be slower than JBOD. Writes are done in parallel. Reads also could be done in parallel (using both disk) - though I'm not sure if mdadm actually does that or not.
If you are interested in ReadyTier, probably the most robust approach would be to run RAID-1 with two mechanical disks, and use two SSDs in RAID-1 for metadata. You would need to keep on eye on wear-levels for the SSDs (since both would reach their write limit at the same time). That should substantially speed up small file transfers and directory browsing.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
Do you have strict sync enabled for the share? (that's specific to AFP). Look on the share settings wheel (Network Access->Advanced). If it is enabled, try disabling it, and doing another test of small file transfer (looks for something less time consuming than ~ 1 million files).
No. I do not. Also, for Time Machine, when you set up a Time Machine share, it now sets it up automatically as an SMB share.
I looked into ReadyTier. Sounds interesting except 3 things... 1) I'm essentially going to lose 2 bays in my NAS to do this; 2) I'm going to have to purchase 2 2TB SSD's for this to work with my 2 12 TB mechanical drives (it appears that it takes 1 MB for metadata for every 8 GB of data and 3) until about 6 months ago, this NAS worked just fine.
I'd hate to spend the money on 2 2TB drives, just to find out that it is still just as slow.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
@SoundmanArt wrote:
Do you have strict sync enabled for the share? (that's specific to AFP).
I mis-spoke here - strict sync is specific to SMB. It will kill performance for some applications.
@SoundmanArt wrote:
I looked into ReadyTier. Sounds interesting except 3 things... 1) I'm essentially going to lose 2 bays in my NAS to do this; 2) I'm going to have to purchase 2 2TB SSD's for this to work with my 2 12 TB mechanical drives (it appears that it takes 1 MB for metadata for every 8 GB of data and 3) until about 6 months ago, this NAS worked just fine.
Yes, it does cost. two bays and two SSDs.
Where did you find the rule of thumb on metadata? Page 11 of the optimization manual says something different: https://www.downloads.netgear.com/files/GDC/READYNAS-100/ReadyNAS_FlexRAID_Optimization_Guide.pdf
FWIW, I have an 18 TB volume with about 12.6 TiB of data. My system shows about 15 GB of metadata - not the 1.5 TiB that your rule of thumb yields.
How much metadata does your system consume? Note you can see this by hovering your mouse over the pie chart on system->volume.
@SoundmanArt wrote:
I'd hate to spend the money on 2 2TB drives, just to find out that it is still just as slow.
Again, I don't see why they need to be that big. But I agree it's an expense, and we don't really now how much it will help.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
I mis-spoke here - strict sync is specific to SMB. It will kill performance for some applications.
I copied 58 GB to a share where I had strict sync turned off this a.m. I got 48 MBS for that copy. However, when you set up a Time Machine share, it doesn't give you the option of turning strict sync on or off.
Where did you find the rule of thumb on metadata?
I was using my own data for the ratio. Mine show 8.42 TB of data and 10.41 GB of metadata. Duh! You're right. I don't need nearly that much. A misplaced decimal point.
I'll grab a couple today and try it.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
@SoundmanArt wrote:
I'll grab a couple today and try it.
Let us know how it turns out.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
Will do. Just missed the cut off for same day Amazon shipping. So it won't be until tomorrow until I get them. But I've already changed the mechanical drives to RAID 0. Just need to add the other 2 drives when I get them.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
I did install an SSD into my rack and set up ReadyTier (it took me a couple of days because I thought it required 1 SSD for each mechanical drive rather than for each volume). So that's now up and running. I started up a Time Machine backup and just aborted it after 5-1/2 hours it has written 188 GB of data and 200 MB of metadata (or about 9.5 MB/second). It took me 12 hours to backup the entire 8.8 TB of data to an attached storage RAID 1.
So I'm thinking this is not the solution.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
@SoundmanArt wrote:
I started up a Time Machine backup and just aborted it after 5-1/2 hours it has written 188 GB of data and 200 MB of metadata (or about 9.5 MB/second). It took me 12 hours to backup the entire 8.8 TB of data to an attached storage RAID 1.
Did you test straight SMB transfers (folders with lots of small files)? Or just TM?
@SoundmanArt wrote:
I did install an SSD into my rack and set up ReadyTier (it took me a couple of days because I thought it required 1 SSD for each mechanical drive rather than for each volume).
Normally you'd use 2 SSDs in RAID-1 for the metadata RAID group. That would give you redundancy if the drive fails.
However, using a single SSD shouldn't affect the speed much (if at all).
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
Did you test straight SMB transfers (folders with lots of small files)? Or just TM?
I did not. But I will now. I have a 180 GB folder with 712k files.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
I just got back to my desk. After 22-1/2 hours, it had copied 160GB, so I stopped it.
I'm going to remove the 2 mechanical drives and put an SSD in there to see what happens with that. That will at least potentially eliminate one area.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
As a follow up, after 13 hours, it had only copied 22.5 GB to the SSD in the Rack. That rules out issues with the mechanical drives.
I then took the SSD out of the rack and put it into a USB3 adapter that I had and copied it directly. It copied all 180GB in 20 minutes. I guess the SSD was ok, too.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
@SoundmanArt wrote:
As a follow up, after 13 hours, it had only copied 22.5 GB to the SSD in the Rack. That rules out issues with the mechanical drives.
Was this with TM or SMB?
Do you have another PC you can test with?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
Was this with TM or SMB?
SMB.
Do you have another PC you can test with?
Yes. I will do that later today. But as I recall, it was slow, too.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
@SoundmanArt wrote:
Yes. I will do that later today. But as I recall, it was slow, too.
Yes, but I think it's worth retesting it using a share on the SSD.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
Ok. I ran it 2 ways on a MacBook Pro. Both tests ran for 3 hours.
180 GB folder
MacBook Pro -> Mechanical Drives = 15.59 GB
MacBook Pro -> SSD = 15.60 GB
It starts up fast on both, gettting the first 1-2 GB done in just a couple of minutes. Then it comes to a halt.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
@SoundmanArt wrote:
Ok. I ran it 2 ways on a MacBook Pro. Both tests ran for 3 hours.
180 GB folder
MacBook Pro -> Mechanical Drives = 15.59 GB
MacBook Pro -> SSD = 15.60 GB
Certainly slower than it should be.
Perhaps try connecting the NAS directly to the MacBook??? That would require temporarily assigning a static IP address on both.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
Perhaps try connecting the NAS directly to the MacBook??? That would require temporarily assigning a static IP address on both.
Did that with the NAS and the MacBook Pro connected directly. I got the exact same amount copied -- 15.60 GB in 3 hours.
As with all of the other copies, it started out like gangbusters. And then came came to a screaching halt.
So??? Is there a problem with my NAS that needs to be sent it under warranty? Is there insufficient RAM?
This is crazy!
Art
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
@SoundmanArt wrote:
Is there insufficient RAM?
It has 4 GB, which should be enough.
@SoundmanArt wrote:
Did that with the NAS and the MacBook Pro connected directly. I got the exact same amount copied -- 15.60 GB in 3 hours.
If it's still connected can you take a look at the network stats in the log zip?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
I had disconnected it. But here's the log file.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
Were you using eth0 or eth1?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
ETH0. ETH1 was disconnected while I was doing the copy.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
@SoundmanArt wrote:
ETH0. ETH1 was disconnected while I was doing the copy.
Ok. And no errors related to ethernet were reported in the log.
@SoundmanArt wrote:
I've changed hard drives, cables, switches. It's slow with ALL of my computers, not just one. I've tried Jumbo Frames and normal size frames. I've tried with/without antivirus.
I have no idea where to look now.
I'm certainly running out of ideas.
One or two more though:
- do you have file search enabled? If so, disable that.
- If you are running the new audit feature in 6.10.0, then disable that too.
Keep strict sync off (and test with "vanilla" SMB, not TimeMachine). And use with the new SSD volume
Not sure if the Mac will connect to the NAS if you turn SMB transport encryption off. But if it will, try disabling that too.
Maybe also disable any services you aren't using (for instance NFS), and turn off all apps.
BTW, I've been assuming that you aren't using the volume encryption feature of the NAS. Is that correct?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
I'm running a plain vanilla system. The only services running are SMB, UPnP, HTTP and HTTPS. Currently, I even have antivirus turned off.
File search is also off.
No apps are installed.
SMB3 Transport Encryption has 3 options -- Enable, Desired and Required. Right now it is Enabled.
Audit is also off.
I don't see where you turn volume encryption on or off.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Very Slow Read/Write and Time Machine Backup
@SoundmanArt wrote:
I don't see where you turn volume encryption on or off.
It's off by default (and you'd certainly know if you were using it).
@SoundmanArt wrote:
I'm running a plain vanilla system. The only services running are SMB, UPnP, HTTP and HTTPS. Currently, I even have antivirus turned off.
File search is also off.
No apps are installed.
SMB3 Transport Encryption has 3 options -- Enable, Desired and Required. Right now it is Enabled.
So it's not
- the load on the NAS (and even if it is encrypting the transport, it should be capable of faster speeds with small files).
- the disks since you switched over to ssd
- a volume issue, since the switch to ssd created a fresh volume
- cabling, since you replaced that (and aren't seeing issues with ethernet errors)
- your infrastructure, since direct connect gave the same performance
- the mac itelf, since you've tried more than one PC.
Looking once more at network_settings.log, you still have ipv6 enabled. That can create performance issues with Windows (especially when ipv6 isn't enabled in the router). I don't know if that's also an issue with Macs - but you probably should disable ipv6.
After that, I am out of ideas on things to try. So if that also makes no difference, then perhaps see if @JohnCM_S or @Hopchen are willing to analyze your full log zip (they might possibly see something there).
If you are conversant with linux, you could log in with ssh, and see if top (or one of the other linux tools) gives you any clues on where the performance bottleneck is.