Ultra4 with OS 6.2.4 slow with RAID-5

[bgarwin]

Hi,

I added a third 4TB drive to the my ULTRA 4, the write performance degraded to 45MB/s. This is with (X-RAID/RAID5) with 2 drives(X-RAID/RAID-1) the write performance was close to 80MB/s. I would suspect it is a CPU bottleneck if it was close to 100% but it only use about 40-45% CPU.

All drive are identical ST4000DM000, ULTRA4 upgraded with 2GB of memory and SMART test shows everything OK.

I upgraded to 6.2.3/6.2.4 from 6.2.2 about the same time i added the third driver, i'm not sure if it is because of the OS upgrade or the addition of the driver.

Anyone have any idea?

[RX]

How did you test the write performance?
Is the PC you are using connected wired or wireless?
Have you tried connecting the NAS directly to the PC?

[bgarwin]

Test done from a PC Wired with Cat 5E cable, 1Gb connection through switch Jumbo frame enabled. A 5GB file used to test using copy/paste from windows 8.1 machine.

[StephenB]

RAID-5 write performance is likely slower than RAID-1. With RAID-1, you simply write everything to both disks. With RAID-5 you need to do at least one read along with the two writes (and the read needs to complete before you can issue the second write).

An old ultra-4 review (http://www.smallnetbuilder.com/nas/nas- ... l=&start=2) measured 65 MB/S write speed with ext. BTRFS would be slower - esp. if you have snapshots on. AV protection would also slow the write performance (since that was not tested in the review).

[bgarwin]

agreed, i didn't expect to have better performance. But I can't explain the 50% performance loss moving from RAID1 t o RIAD5 especially with CPU has free cycles. i do have snapshot enabled but not AV protection, but the same is true for RAID1 both with COW disabled.

[StephenB]

bgarwin wrote:
agreed, i didn't expect to have better performance. But I can't explain the 50% performance loss moving from RAID1 t o RIAD5 especially with CPU has free cycles. i do have snapshot enabled but not AV protection, but the same is true for RAID1 both with COW disabled.

Well, there is disk latency because every parity block update needs to wait for a read to complete. Most performance guides for RAID don't give specifics, but they all agree that RAID-1 has faster write times than RAID-5.

[itsjasper]

StephenB is on the money here. This is the RAID-5 write penalty.

A good article explaining it (with calculations in IOPS) here: http://theithollow.com/2012/03/understa ... d-penalty/

[bgarwin]

Thank you StephenB , makes sense.

[bgarwin]

but, if I tested the raid locally I get 148 MB/s

dd if=/dev/zero of=/data/test conv=fdatasync bs=384k count=10k; rm -f /data/test
10240+0 records in
10240+0 records out
4026531840 bytes (4.0 GB) copied, 27.166 s, 148 MB/s

[mdgm-ntgr]

What does your top output look like?

Fast sequential writes locally are quite a different thing to real world usage.

[bgarwin]

I'm running Plex full library scan right now, so results might be not accurate

Top with dd

top - 07:48:10 up  1:22,  2 users,  load average: 3.52, 3.20, 3.16
Tasks: 173 total,   1 running, 172 sleeping,   0 stopped,   0 zombie
%Cpu(s): 30.2 us, 39.0 sy,  0.0 ni, 10.5 id, 19.9 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem:   2041248 total,  1976140 used,    65108 free,      260 buffers
KiB Swap:   523708 total,       96 used,   523612 free,   377800 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
21598 root      20   0 1038m 1.0g  584 D  67.8 51.4   0:02.05 dd
10504 root      20   0  364m  36m 8312 S  61.5  1.8  38:15.70 Plex Media Scan
 9381 root      20   0  579m  33m 5120 S   3.3  1.7  10:21.67 Plex Media Serv
10549 root      20   0     0    0    0 S   0.7  0.0   0:13.40 btrfs-endio-2
10991 root      20   0     0    0    0 S   0.7  0.0   0:12.24 btrfs-endio-4
21583 root      20   0 28576 1688 1140 R   0.7  0.1   0:00.09 top
    4 root      20   0     0    0    0 S   0.3  0.0   0:02.83 kworker/0:0
  512 root      20   0     0    0    0 D   0.3  0.0   0:45.01 kswapd0
 1253 root      20   0     0    0    0 S   0.3  0.0   1:56.14 md127_raid5
 2055 root      20   0  798m 4576 1740 S   0.3  0.2   0:25.79 leafp2p
 2439 root      20   0     0    0    0 S   0.3  0.0   0:13.54 btrfs-endio-2
 2442 mysql     20   0  475m  36m  852 S   0.3  1.8   0:05.06 mysqld
 2672 root      19  -1 1267m  41m 3060 S   0.3  2.1   0:48.04 readynasd
10198 root      35  15 1261m 158m 1852 S   0.3  7.9   8:51.65 python
10550 root      20   0     0    0    0 S   0.3  0.0   0:13.27 btrfs-endio-3
21021 root      20   0  948m  30m 1804 S   0.3  1.5   0:07.30 python
    1 root      20   0 45700 3588  904 S   0.0  0.2   0:05.81 systemd
    2 root      20   0     0    0    0 S   0.0  0.0   0:00.03 kthreadd
    3 root      20   0     0    0    0 S   0.0  0.0   0:05.98 ksoftirqd/0
    6 root      rt   0     0    0    0 S   0.0  0.0   0:00.06 migration/0
    7 root      rt   0     0    0    0 S   0.0  0.0   0:00.01 watchdog/0
    8 root      rt   0     0    0    0 S   0.0  0.0   0:00.06 migration/1
   10 root      20   0     0    0    0 S   0.0  0.0   0:05.47 ksoftirqd/1
   12 root      rt   0     0    0    0 S   0.0  0.0   0:00.01 watchdog/1
   13 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 khelper
  264 root      20   0     0    0    0 S   0.0  0.0   0:00.00 sync_supers
  266 root      20   0     0    0    0 S   0.0  0.0   0:00.00 bdi-default
  268 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 kblockd
  374 root      20   0     0    0    0 S   0.0  0.0   0:00.00 khubd
  383 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 md
  485 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 rpciod
  507 root      20   0     0    0    0 S   0.0  0.0   0:00.00 khungtaskd
  577 root      20   0     0    0    0 S   0.0  0.0   0:00.00 fsnotify_mark
  594 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 nfsiod
  613 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 xfsalloc
  614 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 xfs_mru_cache
  615 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 xfslogd
  628 root      20   0     0    0    0 D   0.0  0.0   0:00.06 mdcsrepair
  629 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 crypto
  644 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 kthrotld
  744 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_0
  747 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_1
  750 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_2
  753 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_3
  756 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_4
  759 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_5
  766 root      20   0     0    0    0 S   0.0  0.0   0:00.06 kworker/u:6
  772 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 target_completi
  773 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 tmr
  774 root      20   0     0    0    0 S   0.0  0.0   0:00.00 iscsi_ttx
  775 root      20   0     0    0    0 S   0.0  0.0   0:00.00 iscsi_trx
  776 root      20   0     0    0    0 S   0.0  0.0   0:00.00 iscsi_ttx
  777 root      20   0     0    0    0 S   0.0  0.0   0:00.00 iscsi_trx
  778 root      20   0     0    0    0 S   0.0  0.0   0:00.00 iscsi_ttx

top with network copy

top - 07:49:38 up  1:23,  2 users,  load average: 3.32, 3.20, 3.17
Tasks: 170 total,   3 running, 167 sleeping,   0 stopped,   0 zombie
%Cpu(s): 35.0 us, 28.7 sy,  0.0 ni, 17.8 id,  1.3 wa,  0.0 hi, 17.1 si,  0.0 st
KiB Mem:   2041248 total,  1915688 used,   125560 free,      408 buffers
KiB Swap:   523708 total,       96 used,   523612 free,  1364204 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
10504 root      20   0  364m  36m 8324 S  77.8  1.8  39:17.31 Plex Media Scan
10683 guest     20   0  265m 6816 3244 R  29.4  0.3   2:03.91 smbd
 1253 root      20   0     0    0    0 R  19.5  0.0   2:02.25 md127_raid5
 2153 root      20   0     0    0    0 D  15.5  0.0   1:03.79 flush-btrfs-2
 9381 root      20   0  579m  34m 6060 S   2.6  1.7  10:24.24 Plex Media Serv
21583 root      20   0 28576 1696 1144 R   1.3  0.1   0:00.84 top
   10 root      20   0     0    0    0 S   1.0  0.0   0:05.60 ksoftirqd/1
 2055 root      20   0  798m 4576 1740 S   1.0  0.2   0:26.25 leafp2p
 2672 root      19  -1 1267m  41m 3104 S   1.0  2.1   0:48.41 readynasd
10991 root      20   0     0    0    0 S   1.0  0.0   0:12.97 btrfs-endio-4
    3 root      20   0     0    0    0 S   0.7  0.0   0:06.13 ksoftirqd/0
 2439 root      20   0     0    0    0 S   0.7  0.0   0:14.28 btrfs-endio-2
10198 root      35  15 1261m 158m 1852 S   0.7  7.9   8:51.89 python
10549 root      20   0     0    0    0 S   0.7  0.0   0:14.14 btrfs-endio-2
10550 root      20   0     0    0    0 S   0.7  0.0   0:13.99 btrfs-endio-3
21021 root      20   0  948m  30m 1804 S   0.7  1.5   0:07.36 python
    1 root      20   0 45700 3588  904 S   0.0  0.2   0:05.81 systemd
    2 root      20   0     0    0    0 S   0.0  0.0   0:00.03 kthreadd
    4 root      20   0     0    0    0 S   0.0  0.0   0:02.87 kworker/0:0
    6 root      rt   0     0    0    0 S   0.0  0.0   0:00.06 migration/0
    7 root      rt   0     0    0    0 S   0.0  0.0   0:00.01 watchdog/0
    8 root      rt   0     0    0    0 S   0.0  0.0   0:00.06 migration/1
   12 root      rt   0     0    0    0 S   0.0  0.0   0:00.02 watchdog/1
   13 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 khelper
  264 root      20   0     0    0    0 S   0.0  0.0   0:00.00 sync_supers
  266 root      20   0     0    0    0 S   0.0  0.0   0:00.00 bdi-default
  268 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 kblockd
  374 root      20   0     0    0    0 S   0.0  0.0   0:00.00 khubd
  383 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 md
  485 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 rpciod
  507 root      20   0     0    0    0 S   0.0  0.0   0:00.00 khungtaskd
  512 root      20   0     0    0    0 S   0.0  0.0   0:46.30 kswapd0
  577 root      20   0     0    0    0 S   0.0  0.0   0:00.00 fsnotify_mark
  594 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 nfsiod
  613 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 xfsalloc
  614 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 xfs_mru_cache
  615 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 xfslogd
  628 root      20   0     0    0    0 D   0.0  0.0   0:00.06 mdcsrepair
  629 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 crypto
  644 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 kthrotld
  744 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_0
  747 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_1
  750 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_2
  753 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_3
  756 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_4
  759 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_5
  766 root      20   0     0    0    0 S   0.0  0.0   0:00.06 kworker/u:6
  772 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 target_completi
  773 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 tmr
  774 root      20   0     0    0    0 S   0.0  0.0   0:00.00 iscsi_ttx
  775 root      20   0     0    0    0 S   0.0  0.0   0:00.00 iscsi_trx
  776 root      20   0     0    0    0 S   0.0  0.0   0:00.00 iscsi_ttx
  777 root      20   0     0    0    0 S   0.0  0.0   0:00.00 iscsi_trx
  778 root      20   0     0    0    0 S   0.0  0.0   0:00.00 iscsi_ttx

[mdgm-ntgr]

Plex using a lot of CPU for the library scan would have an impact on the performance of file transfers

[bgarwin]

I will post results without Plex scan running when I got home. I tried downgrading to 6.2.0 with no improvement, it got to be the RAID-5. but what I suspect is OS6 X86-64 is optimized for 3XX series and up, with review for the 314 scores in the 80MB/s.

I was quite happy with OS6 on the Ultra 4 on RAID-1, but now I'm considering to revert to official firmware.
There is this thread complaining about the performance drop from ROS4 to ROS6 http://www.readynas.com/forum/viewtopic.php?f=21&t=70847#p39348, but i want to hear more from other users about this RIAD-5 on ROS4 for the ULTRA4.

[mdgm-ntgr]

The Ultra 4 does have a single core CPU. It is the x86_64 ReadyNAS with the least powerful CPU

[bgarwin]

An update, this issue has nothing to do with netgear OS6 or netgear anything for that matter.

I remembered the CAT6 cable connected to the switch connected to the Readynas was connected to one of the Leviton SNAP on port, this port wasn't securely mounted to the wall hence couldn't firmly secure the port to it, it was moved when we were dusting the house :-S.

anyway, i can get 90MB/S write now after I secured the port.

Many thanks to StephenB and mdgm for their help.