× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: USB3 full backup slow to the point of unusable

dsnpevl
Virtuoso

Re: USB3 full backup slow to the point of unusable


@McRob1 wrote:

Wow you have a very fast backup there.  I have been backing up my 4.2 TB of data from my NAS516 to a 5 TB Toshiba drive for 8 days and it has finished 3.8 TB of data so far.  I am using the rear usb 3.0 so called faster port.

 

Must be caused by firmware 6.5.0-T338 that i'm running also.

 

And by the way I also agree with using NTFS so you can actually see the data on any PC or Mac computer without having to install special software.

 

Sure hope Netgear gets this figured out soon.


And I thought I had it bad. Well, it used to be even 10 times faster in 6.4.0 and earlier OS versions. Then 4.5T only toke 8 to 10 hours. I hope to get back to that situation (or faster) some time soon. The USB 3 specification has a much higher potential throughtput (4,8 Gbit/s) then what we are achieving now.

 

A few things you might want to check on this forum. I've seen a post about poor performance with a USB powered drive. Is that what you are using?

And of course there have been several posts on the AntiVirus causing delays as well. Switching AV off could help improve the situation according to these posts (didn't for me). Maybe you should look into that.

 

Message 26 of 58
McRob1
Luminary

Re: USB3 full backup slow to the point of unusable

Yes I am using a powered USB drive.

I have tried turning off anti-virus. No difference.

The production release of 6.4.2 almost stopped any access to the NAS until I had to purchase product support help.  They suggested I turn off anti-virus.

 

It's been quite a ride with Netgear.

Model: RN51664E|ReadyNAS 516 6-Bay
Message 27 of 58
dsnpevl
Virtuoso

Re: USB3 full backup slow to the point of unusable


@dsnpevl wrote:

@StephenB wrote:

It might make sense to copy a larger share, so you can see if the performance is sustained.


Yes, my thoughts exactly. Will compare a backup of almost 1TB of data from the Picture share to both Seagate and WD USB drives. The picture files come in many different sizes, ranging from a few kB to 300 MB. Should be a representative test.


The performance is not sustained. Backup of 979G to Seagate takes 23 hours, 5 minutes and 10 seconds, resulting in a low throughput of 12 MB/s.

 

Message 28 of 58
dsnpevl
Virtuoso

Re: USB3 full backup slow to the point of unusable


dsnpevl wrote: The performance is not sustained. Backup of 979G to Seagate takes 23 hours, 5 minutes and 10 seconds, resulting in a low throughput of 12 MB/s.

The performance on the other drive is in the same range. Backup of 979G to WD takes 1 day, 0 hours, 25 minutes and 31 seconds, resulting in a low throughput of

11,4 MB/s. So NTFS performance is not so good.

 

How can I change a drive to have EXT4 formatting? Then I'll compare with that also.

 

Edit: Never mind. Found how-to at: http://kb.netgear.com/app/answers/detail/a_id/29654/~/how-do-i-back-up-data-from-my-readynas-os-6-sy...

Message 29 of 58
McRob1
Luminary

Re: USB3 full backup slow to the point of unusable

You can format your drive using EXT4 by selecting  format from the System, Overview page.  With the drive connected there is a circula gear symbol that has pulldown options.  One of these options is format.

Model: RN51664E|ReadyNAS 516 6-Bay
Message 30 of 58
dsnpevl
Virtuoso

Re: USB3 full backup slow to the point of unusable

Formatted the Seagate to EXT 4 and this results in a very reasonable throughput of 112 MB/s.

I think this points to the problem being in the NTFS driver as suggested by OOM-9.

 

Test results for:
Copying approx 1TB from the Pictures share containing pictures,
videos and zip files ranging in filesize from a few kB to over 300 MB
to USB 3 drive on RN312. ***************** Seagate ST4000DM000-1F2168 (NTFS) ********************** 1050782796994 B / 83110 s 12643278 B / s 12 MB/s ****************** WD WDBFJK0060HBK-EESN (NTFS) **************************** 1050789792458 B / 87931 s 11950163,11037063 B / s 11,4 MB/s ***************** Seagate ST4000DM000-1F2168 (EXT4) ********************** 1053101189653 B / 8979 s 117284908,0802985 B / s 112 MB/s

 

Message 31 of 58
OOM-9
NETGEAR Expert

Re: USB3 full backup slow to the point of unusable

Those numbers are similar to what I am seeing in our lab.

I have been looking at alternative driver that you could use in the meantime that is available from the Debian repo (ntfs-3g). The performance on this driver is significantly better than the ufsd we are currently using. I am interested if you leave the USB drive plugged in our if you remove the drive frequently.

(I would want to go through the steps in our lab before posting complete steps.)


If you are feeling adventurous then you can run:

apt-get install ntfs-3g

Man page for reference: http://linux.die.net/man/8/ntfs-3g

 

Message 32 of 58
dsnpevl
Virtuoso

Re: USB3 full backup slow to the point of unusable


@OOM-9 wrote:

Those numbers are similar to what I am seeing in our lab.

I have been looking at alternative driver that you could use in the meantime that is available from the Debian repo (ntfs-3g). The performance on this driver is significantly better than the ufsd we are currently using. I am interested if you leave the USB drive plugged in our if you remove the drive frequently.

(I would want to go through the steps in our lab before posting complete steps.)


Will give the alternative driver a try with the same data as before.

I leave the USB drive connected to the NAS pretty much permanently on RN516.

On RN312, I only switched ports for testing to see if the issue was port or drive make/model related. I also intend to keep the USB drive connected permantly on RN312 when this issue is resolved.

Message 33 of 58
dsnpevl
Virtuoso

Re: USB3 full backup slow to the point of unusable

# lsblk
NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdd         8:48   0   5.5T  0 disk
└─sdd1      8:49   0   5.5T  0 part  /media/USB_HDD_1

# which ntfs-3g
/bin/ntfs-3g

Installed the driver. The USB drive is already mounted. Do I need to unmount and mount with ntfs-3g? If so, are these commands correct? Maybe I should mount under a different name as USB_HDD_1 is controlled by OS?

 

# umount /media/USB_HDD_1
# ntfs-3g /dev/sdd1 /media/USB_HDD_1
Message 34 of 58
OOM-9
NETGEAR Expert

Re: USB3 full backup slow to the point of unusable

Those commands look correct.

The OS should see the mount point is already used, so the conflict should not be an issue.
Message 35 of 58
dsnpevl
Virtuoso

Re: USB3 full backup slow to the point of unusable

Issued the two commands, but now the Admin pages don't show the mount any more. Therefore I can no longer issue a backup towards it. The backup listing has "no longer exists" where the USB destination USB_HDD_1 used to be.

Message 36 of 58
OOM-9
NETGEAR Expert

Re: USB3 full backup slow to the point of unusable

Did you confirm the volume was unmounted? I have found if there is another mount point referring to the USB device; it would not fulling unmount until the USB is no longer in use (including other mount points like NFS).


Are you able to navigate the data in the current state? What does the "mount" look like?


Note: I am running the tests from the UI after I mounted the ntfs-3g manually.
Message 37 of 58
dsnpevl
Virtuoso

Re: USB3 full backup slow to the point of unusable


@OOM-9 wrote:
Did you confirm the volume was unmounted? I have found if there is another mount point referring to the USB device; it would not fulling unmount until the USB is no longer in use (including other mount points like NFS).
Are you able to navigate the data in the current state? What does the "mount" look like?

Note: I am running the tests from the UI after I mounted the ntfs-3g manually.

It was mounted under two names,  /media/USB_HDD_1 and /run/nfs4/media/USB_HDD_1.

After unmounting both and mounting with ntfs-3g, the drive is visible again in the UI and the content can be listed with ls.

How can I be sure the ntfs-3g driver is used? Because at first glance I don't see much improvement in throughput.

 

 

# lsblk
NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sde         8:64   0   5.5T  0 disk
└─sde1      8:65   0   5.5T  0 part  /media/USB_HDD_1
# umount /media/USB_HDD_1
# lsblk
NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sde         8:64   0   5.5T  0 disk
└─sde1      8:65   0   5.5T  0 part  /run/nfs4/media/USB_HDD_1
# umount /run/nfs4/media/USB_HDD_1
# ntfs-3g /dev/sde1 /media/USB_HDD_1

 

Message 38 of 58
OOM-9
NETGEAR Expert

Re: USB3 full backup slow to the point of unusable

 

You will see the the type as fuseblk:

 

# mount
...
/dev/sdn1 on /media/USB_HDD_1 type fuseblk (rw,noatime,nodiratime,user_id=0,group_id=0,allow_other,blksize=4096)

Are you using a tool to check the disk io?

I have been iotop; it is not a light-wieght tool; which is why it is not installed the NAS by default, but it can help in cases like this to help monitor the disk io.

Message 39 of 58
dsnpevl
Virtuoso

Re: USB3 full backup slow to the point of unusable


OOM-9 wrote: You will see the the type as fuseblk:

 

# mount
...
/dev/sdn1 on /media/USB_HDD_1 type fuseblk (rw,noatime,nodiratime,user_id=0,group_id=0,allow_other,blksize=4096)

Are you using a tool to check the disk io?

I have been iotop; it is not a light-wieght tool; which is why it is not installed the NAS by default, but it can help in cases like this to help monitor the disk io.


OK. I can see fuseblk for USB_HDD_1. So that indicates that is it using ntfs-3g as a driver, right?

 

root@nas-27-5F-C4:~# mount -l
/dev/sdd1 on /media/USB_HDD_2 type ext4 (rw,noatime,nodiratime,data=ordered) [USB_HDD_2]
/dev/sdd1 on /run/nfs4/media/USB_HDD_2 type ext4 (rw,noatime,nodiratime,data=ordered) [USB_HDD_2]
/dev/sde1 on /media/USB_HDD_1 type fuseblk (rw,noatime,nodiratime,user_id=0,group_id=0,allow_other,blksize=4096) [My Book]

Never used iotop before. I am determining the overall throughput, taking into account the read time and any overhead from OS, drivers, ports, etc. It is based on the duration calculated from "Backup Start Time" and "Backup Finish Time" from the mail notification sent when the backup is finished. E.g the EXT4 calculation:

 

Backup Start Time: Mon Apr  4 2016 18:24:22
Backup Finish Time: Mon Apr  4 2016 20:54:01
The duration is 2 hours, 29 minutes and 39 seconds
The duration is 8979 seconds

Then I determine the number of bytes in the share I just copied to USB. E.g.

 

# cd /media/USB_HDD_2/data
# du -sbh Pictures
981G    Pictures
# du -sb Pictures
1053101189653   Pictures

From that I calculated the effective EXT4 throughput:

 

1053101189653 B / 8979 s
117284908,0802985 B / s
111.8516 MB/s
112 MB/s

This is the method I've been using to calculate the throughputs I mentioned in this post.

 

 

Message 40 of 58
dsnpevl
Virtuoso

Re: USB3 full backup slow to the point of unusable

Installed iotop on RN312:

 

# iotop
-bash: iotop: command not found
# apt-get install iotop
....
Do you want to continue? [Y/n] Y
....
Setting up python-minimal (2.7.3-4+deb7u1) ...
Setting up python (2.7.3-4+deb7u1) ...
Setting up iotop (0.4.4-4) ...

It is showing output like:

Total DISK READ:      14.74 M/s | Total DISK WRITE:      21.37 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
24885 be/4 root        0.00 B/s    0.00 B/s  0.00 % 17.49 % [kworker/u8:3]
 6165 be/4 root       14.74 M/s    0.00 B/s  0.00 %  9.92 % fvbackup -e 3
 5613 be/4 root        0.00 B/s   25.72 M/s  0.00 %  0.00 % ntfs-3g /dev/sde1 /media/USB_HDD_1

and this:

Total DISK READ:      27.58 M/s | Total DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 6165 be/4 root       27.58 M/s    0.00 B/s  0.00 % 19.38 % fvbackup -e 3
 5613 be/4 root        0.00 B/s   23.72 M/s  0.00 %  0.00 % ntfs-3g /dev/sde1 /media/USB_HDD_1
 1024 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [scsi_eh_7]

Even with this new driver, I'm still not achieving throughput that was available in 6.4.0 and 6.2.x. Will need to wait for the backup to complete before I can calculate the effective throughput.

Message 41 of 58
dsnpevl
Virtuoso

Re: USB3 full backup slow to the point of unusable


dsnpevl wrote: Even with this new driver, I'm still not achieving throughput that was available in 6.4.0 and 6.2.x. Will need to wait for the backup to complete before I can calculate the effective throughput.
1053099117277 B / 38408 s
27418743,94076755 B / s
26.1 MB/s

The backup with the ntfs-3g driver is a bit faster (about 2.1x) at 26.1 MB/s, compared to the old NTFS driver. That is in line with what iotop is showing. But still no where near to the througput I could achieve in older versions OS 6.2.x (and 6.4.0). I think Netgear should look into what has changed regarding backups and disk access between OS 6.4.0 and OS 6.4.1. This is when the backup speed on NTFS USB 3 drive dropped by a factor of 10! Maybe not only look at the drivers but also other processes that can can slow disk read/write down? Maybe also verify that the USB 3 driver is used (and not USB 2 driver)?

 

Message 42 of 58
McRob1
Luminary

Re: USB3 full backup slow to the point of unusable

I got tired of watching moss grow so I stopped my USB 3.0 backup that had been running for at least three days.  It had saved about 1.5 TB of data to a 5 TB drive.

The drive was formated using NTFS.

 

I reformatted the drive to EXT4 and started the backup over at 9:38 today.

It is now 12:00 and it has so far backed up 1.2 TB of data to the same drive.

 

Speed difference

NTFS=  2.0 Mb per hour

EXT4 = 50 Mb per hour

 

Huge difference in speed.

It's not hardware so it must be software.

I sure hope Netgear can find a solution using NTFS.

 

Model: RN51664E|ReadyNAS 516 6-Bay
Message 43 of 58
McRob1
Luminary

Re: USB3 full backup slow to the point of unusable

Wonder why ther is such a difference between NTFS formated USB drives and EXT4 formated USB drives.

 

What took over 8 days to backup using NTFS only took 10 hours using EXT4.

4.2 TB of data in 10 hours sounds pretty good to me.

 

I sure hope Netgear finds another driver for NTFS.

 

Model: RN51664E|ReadyNAS 516 6-Bay
Message 44 of 58
hawke84
Tutor

Re: USB3 full backup slow to the point of unusable

exactly the same issue for me, i dont do backups often, came to do my yearly backup and I couldnt believe how slow i was. 700GB data took 3 days to run.. WD 3TB drive formatte to NTFS. keeping my fingers crossed for a fix also!

Model: RN104|ReadyNAS 100 Series
Message 45 of 58
AlexPe
NETGEAR Expert

Re: USB3 full backup slow to the point of unusable

Hi Everyone,

 

I have a minor update. The issue has been identified and our engineering team is currently working with the team to resolve this issue. I cannot commit to a release as it depends on an external resource. I can tell you we are working very hard to get this released with the 6.5.0 firmware update.

 

Thanks for your patience,

 

AlexPe

Message 46 of 58
SirToby
Aspirant

Re: USB3 full backup slow to the point of unusable

Sorry but suggesting users fork out another 20 bucks to get ext4 support on a Windows box, just because something was muddled up during a firmware upgrade is a bit steep don't you think? Especially where NTFS is a supported file system and nowhere does it state that there are performance issues.

 

I have tested with Ext4 and instead of 100GB/hour, I get rates of approx. 120GB/hour. A slight improvement over NTFS but still nowhere near where it should be.

 

To me it looks more like USB3 is the problem, not NTFS, considering Ext4 is only marginally faster.

 

Cheers

 

 

Model: RN10400|ReadyNAS 100 Series 4- Bay (Diskless)
Message 47 of 58
mdgm-ntgr
NETGEAR Employee Retired

Re: USB3 full backup slow to the point of unusable

Thanks for your patience.

There is a fix in ReadyNASOS 6.5.0 (RC2). The production 6.5.0 firmware is currently expected to be released early/mid May.

Message 48 of 58
dsnpevl
Virtuoso

Re: USB3 full backup slow to the point of unusable

Thanks to everyone involved in identifying and solving the issue!

Upgraded RN312 to RC2. Now achieving 95 MB/s for a 1.1 TB backup to USB NTFS disk. That is a significant improvement from the 12 MB/s I saw earlier.

I still believe there is much more potential in the USB 3 spec that driver manufactures should inspire to utilize and achieve. As the size of data stored will only increase in the future, it's important to be able to have fast backup capability.

 

Tested with same test procedure as before on RN312 with MyBook on rear USB 3 port with standard NTFS driver.

 

root@nas-XX-XX-XX:/media/USB_HDD_1/data# mount -l
/dev/sdc1 on /media/USB_HDD_1 type ufsd (rw,noatime,nodiratime,nls=utf8,fmask=0,dmask=0) [My Book]
/dev/sdc1 on /run/nfs4/media/USB_HDD_1 type ufsd (rw,noatime,nodiratime,nls=utf8,fmask=0,dmask=0) [My Book]


Achieved better throughput. IOTop is consitently showing read and write around 100 MB/s.

Total DISK READ:     103.03 M/s | Total DISK WRITE:      98.03 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 1253 be/4 root        0.00 B/s   29.96 K/s  0.00 % 94.55 % [kworker/u8:10]
 6916 be/4 root      103.03 M/s   98.07 M/s  0.00 % 71.80 % fvbackup -e 3
 1536 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [btrfs-freespace]
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init


Overall throughput calculation

Successfully completed backup job Pictures to WD on USB.
-------- Backup Job Information -------- Backup Job Name: Pictures to WD on USB Backup Job Type: Full
Protocol: local
Backup Source: [Pictures]/
Backup Destination: [USB_HDD_1]/data/Pictures
Backup Start Time: Mon May  2 2016  7:03:52
Backup Finish Time: Mon May  2 2016 10:10:25
Backup Status: Success

 

3 hours, 6 minutes and 33 seconds can be converted to one of these units:

    11.193 seconds
    186 minutes (rounded down)
    3 hours (rounded down)
    0,04% of 2016
    12,95% of a 24 hour day


Number of seconds: 11193 s

root@nas-XX-XX-XX:/media/USB_HDD_1/data# du -sbh Pictures
1.1T    Pictures
root@nas-XX-XX-XX:/media/USB_HDD_1/data# du -sb Pictures
1111571127754   Pictures


Number of bytes: 1111571127754

1111571127754 B / 11193 s
99309491 B / s
95 MB/s

 

Message 49 of 58
mdgm-ntgr
NETGEAR Employee Retired

Re: USB3 full backup slow to the point of unusable

Thanks for sharing your feedback.

 

With USB I believe there is always the factor of the CPU involved.

 

Also there is the speed of the disks both in the NAS and in the external enclosure, amongst other things.

Message 50 of 58
Discussion stats
  • 57 replies
  • 8670 views
  • 10 kudos
  • 14 in conversation
Announcements