Rsync slow

[joe_schmo]

I am rsyncing from one NAS on my LAN to another.

• If I go from NAS1 to NAS2(EDA500), the transfer goes at aout 25MB/sec. That's cool.
• If I go from NAS1 to NAS2, it transfers about that speed, but between files it takes literally forever to move to the next file, and the transfer speed drops to 100KB/sec.
• I've made sure I've done a defrag and a balance. I should also note that I get the same problem if I rsync between NAS2 to NAS2(EDA500), but not from NAS2(EDA500) to NAS2(OTHER EDA500).

This is what I see on NAS2 during this pause:

Cpu(s): 0.5 us, 3.8 sy, 0.0 ni, 14.1 id, 81.1 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem: 2030856 total, 1888888 used, 141968 free, 4484 buffers
KiB Swap: 1569788 total, 140760 used, 1429028 free. 939876 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2037 root 20 0 0 0 0 D 3.6 0.0 68:52.48 md125_raid5
2056 root 20 0 0 0 0 D 3.0 0.0 68:16.66 md124_raid5
2176 root 20 0 0 0 0 S 2.6 0.0 8:18.35 md118_raid5
5481 root 19 -1 1772156 15716 5008 S 1.3 0.8 36:39.50 readynasd
23662 root 20 0 315420 17712 2916 S 1.3 0.9 58:40.95 sshd
4469 root 20 0 0 0 0 S 1.0 0.0 4:46.09 btrfs-tran+
1723 root 0 -20 0 0 0 S 0.7 0.0 22:21.20 kworker/1:+

And this is what I see when the transfer is working OK:

Tasks: 342 total, 3 running, 339 sleeping, 0 stopped, 0 zombie
%Cpu(s): 17.0 us, 10.4 sy, 0.0 ni, 28.6 id, 39.3 wa, 0.0 hi, 4.7 si, 0.0 st
KiB Mem: 2030856 total, 1856096 used, 174760 free, 9188 buffers
KiB Swap: 1569788 total, 140756 used, 1429032 free. 969308 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23662 root 20 0 315420 17712 2916 R 79.9 0.9 58:52.42 sshd
23693 root 20 0 124520 2352 0 D 11.9 0.1 8:30.14 rsync
2037 root 20 0 0 0 0 D 9.6 0.0 68:53.99 md125_raid5
2594 root 20 0 0 0 0 S 3.3 0.0 0:00.43 kworker/u8+
1803 root 20 0 0 0 0 S 2.6 0.0 0:04.23 kworker/u8+
18347 root 20 0 0 0 0 S 2.6 0.0 0:47.71 kworker/u8+

[joe_schmo]

Cpu(s):  0.5 us,  3.8 sy,  0.0 ni, 14.1 id, 81.1 wa,  0.0 hi,  0.4 si,  0.0 st
KiB Mem:   2030856 total,  1888888 used,   141968 free,     4484 buffers
KiB Swap:  1569788 total,   140760 used,  1429028 free.   939876 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2037 root      20   0       0      0      0 D   3.6  0.0  68:52.48 md125_raid5
 2056 root      20   0       0      0      0 D   3.0  0.0  68:16.66 md124_raid5
 2176 root      20   0       0      0      0 S   2.6  0.0   8:18.35 md118_raid5
 5481 root      19  -1 1772156  15716   5008 S   1.3  0.8  36:39.50 readynasd
23662 root      20   0  315420  17712   2916 S   1.3  0.9  58:40.95 sshd
 4469 root      20   0       0      0      0 S   1.0  0.0   4:46.09 btrfs-tran+
 1723 root       0 -20       0      0      0 S   0.7  0.0  22:21.20 kworker/1:+

[joe_schmo]

Tasks: 342 total,   3 running, 339 sleeping,   0 stopped,   0 zombie
%Cpu(s): 17.0 us, 10.4 sy,  0.0 ni, 28.6 id, 39.3 wa,  0.0 hi,  4.7 si,  0.0 st
KiB Mem:   2030856 total,  1856096 used,   174760 free,     9188 buffers
KiB Swap:  1569788 total,   140756 used,  1429032 free.   969308 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
23662 root      20   0  315420  17712   2916 R  79.9  0.9  58:52.42 sshd
23693 root      20   0  124520   2352      0 D  11.9  0.1   8:30.14 rsync
 2037 root      20   0       0      0      0 D   9.6  0.0  68:53.99 md125_raid5
 2594 root      20   0       0      0      0 S   3.3  0.0   0:00.43 kworker/u8+
 1803 root      20   0       0      0      0 S   2.6  0.0   0:04.23 kworker/u8+
18347 root      20   0       0      0      0 S   2.6  0.0   0:47.71 kworker/u8+

[StephenB]

From the overall symptom, it looks like the issue is with the internal NAS2 data volume.

• Have you checked the disk health (via disk_info.log)?  
• How full is that data volume?

One option is to destroy/recreate that volume, reconfigure the shares and restore it from backup.