× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Abysmal CIFS performance, ReadyNAS pro 4

LeoW
Aspirant

Abysmal CIFS performance, ReadyNAS pro 4

I have had a bit of trouble with my ReadyNAS lately. Running Radiadtor 4.2.26 it would loose connection and I have been forced to downgrade to 4.2.25. *edit- 4.2.25*

Before this it had been working fine for several months.

Now it does not loose connection. However, speed over CIFS is very poor, 0-10MB/s when copying from NAS to PC. Speed over DLnA is good, ~100MB/s but usualy reaches an abrupt halt. Dropping to 0bytes/s after copying a few gig and then not resuming at all. Over CIFS the speed is as stated very poor, it sometimes drop to 0 but then resumes, unlike DLNA.

Speed is about the same on my wire connected PC as the wireless macbook.

The NAS and PC are wired to my ASUS RT-AC68U, the macbook is on AC wifi.

File transfer over CIFS between the PC and Macbook is fine, 50-100MB/s.

Any ideas as to why the NAS is so slow now?
Message 1 of 22
LeoW
Aspirant

Re: Abysmal CIFS performance, ReadyNAS pro 4

Anyone? Any ideas?

Its quite unusable as it stands now. Attempting to move or copy files on the nas itselft through regular windows acess is impossible with speeds of 10-50kbps at best...
Message 2 of 22
StephenB
Guru

Re: Abysmal CIFS performance, ReadyNAS pro 4

If you have ssh enabled, take a look at the fullness of the OS partition (not the data volume).
Message 3 of 22
LeoW
Aspirant

Re: Abysmal CIFS performance, ReadyNAS pro 4

Prometheus:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md0 4.0G 689M 3.2G 18% /
tmpfs 16K 0 16K 0% /USB
/dev/c/c 5.5T 2.4T 3.2T 43% /c



18%, should be ok?

And:

Prometheus:/# du -ch  --max-depth=1
156K ./root
16K ./lost+found
14M ./frontview
4.2M ./tmp
8.0K ./home
4.7M ./sbin
2.4T ./c
2.2M ./lib64
48M ./lib
165M ./var
564K ./dev
7.4M ./etc
365M ./usr
0 ./sys
du: cannot access `./proc/3095/task/3095/fd/4': No such file or directory
du: cannot access `./proc/3095/task/3095/fdinfo/4': No such file or directory
du: cannot access `./proc/3095/fd/4': No such file or directory
du: cannot access `./proc/3095/fdinfo/4': No such file or directory
0 ./proc
4.0K ./initrd
24K ./ramfs
4.0K ./mnt
0 ./USB
3.4M ./opt
3.5M ./bin
2.4T .
2.4T total
Message 4 of 22
StephenB
Guru

Re: Abysmal CIFS performance, ReadyNAS pro 4

18% should be ok. So that is not the cause.
Message 5 of 22
LeoW
Aspirant

Re: Abysmal CIFS performance, ReadyNAS pro 4

Ok. I really appreciate your help, as well as any other Ideas you might have.

One thing I have noticed are some RX drops. Could that mean anything? log from Network\Interfaces\Show errors

Network Errors [Ethernet 1+2]


TX packets 6843061
TX errors 0
TX dropped 0
TX overruns 0
TX carrier 0
RX packets 7895217
RX errors 8
RX dropped 61236
RX overruns 0
RX frame 8
Collisions 0
Message 6 of 22
StephenB
Guru

Re: Abysmal CIFS performance, ReadyNAS pro 4

if you are accessing the NAS over wireless, you can see drops (since there will be some packet loss).

Maybe try looking at the numbers before and after a transfer test.

Are you using teaming? Or Jumbo Frames?
Message 7 of 22
LeoW
Aspirant

Re: Abysmal CIFS performance, ReadyNAS pro 4

NAS is wired, 2Gbit/ Full duplex. Teaming - adaptive load balancing. No Jumboframes. PC is wired Gigabit ethernet.

Did a Transfer test NAS-PC, two ~5GB files. Speed was 0-12MB/s, mostly 5-10. The second one failed 4 times, but connection to NAS was never totally lost and it could resume and try again, after 4 times I gave up however.

Logs show:

(Im sorry, their in swedish but in the same order as before and I translated the relevant parts)

Pre:

Nätverksfel [Ethernet 1+2]

TX-paket 11864130
TX-fel 0
TX-nedstängningar 0
TX-överskridningar 0
TX-medium 0
RX-packets 12953093
RX-errors 10
RX dropped 76868
RX-överskridningar 0
RX-frame 10
Kollisioner 0

Post:

Nätverksfel [Ethernet 1+2]

TX-paket 20718586
TX-fel 0
TX-nedstängningar 0
TX-överskridningar 0
TX-medium 0
RX-packets 15148361
RX-errors 22
RX dropped 80574
RX-överskridningar 0
RX-frame 22
Kollisioner

Nätverksfel [Ethernet 1]

TX-paket 12248404
TX-fel 0
TX-nedstängningar 0
TX-överskridningar 0
TX-medium 0
RX-packets 14657848
RX-errors 22
RX dropped 0
RX-överskridningar 0
RX-frame 22
Kollisioner 0


Nätverksfel [Ethernet 2]

TX-paket 8489609
TX-fel 0
TX-nedstängningar 0
TX-överskridningar 0
TX-medium 0
RX-packets 510566
RX-errors 0
RX dropped 67477
RX-överskridningar 0
RX-frame 0
Kollisioner 0


So it would seem all the erorrs are on Ethernet 1 and all the drops on 2? Strange?
Message 8 of 22
fastfwd
Virtuoso

Re: Abysmal CIFS performance, ReadyNAS pro 4

LeoW wrote:
Before this it had been working fine for several months. .... The NAS and PC are wired to my ASUS RT-AC68U .... NAS is wired, 2Gbit/ Full duplex. Teaming - adaptive load balancing. No Jumboframes. PC is wired Gigabit ethernet.


When it was working fine, was teaming enabled? And were you using the same router, running the same router firmware as you are running now? And were you using the same Ethernet cables, plugged into the same ports on the router?
Message 9 of 22
StephenB
Guru

Re: Abysmal CIFS performance, ReadyNAS pro 4

My advice is to test performance w/o teaming next.
Message 10 of 22
LeoW
Aspirant

Re: Abysmal CIFS performance, ReadyNAS pro 4

I tried shutting of ethernet 1 and 2 sepparatley. No real effect. Then I tried connecting a new CAT 6 cable only to ethernet 1 with warying speed, then I tried the same calble in Ethernet 2 with Good results. 50-100MB/s. Great! Thanks! But why doesnt teaming work?

And a very strange problem occurs.

Since the speed was good and stable I thought I should try upgrading to firmware 4.2.26. I cklick uppdate. Wait. I get an email from the NAS claming there was an error with the upgrade. and now it claims that disk 1 is dead. Frontview claimed redundancy however, whilst the LED on the NAS stated unprotected, Previousley no Errors on any disks. I turned of the NAS, asked it to check FS at next boot, removed disk 1, reinserted it and booted. It didnt check FS for more than maybie one second. Then it said disk 1 and 3 (of 3) were dead, no redundancy and that disk 2 was restoring data. All the while I had full acess to files on the NAS with good speed. New reboot, check FS and now disk 1 is dead and 3 is restoring data. I should add that theese "restore data" took 1 and maybie 3 minutes respectivley. Seems suspiciously fast?

Right now it says disk 1 is dead, 2 and 3 are ok with no erros. Could it be a bad disk that caused theese problems? Even though I have checked for errors, chacked FS etc several times over the last week with no errors? Can I really trust the NAS right now with how it has been acting up? Should I trust the disk is dead, and if it is, would it be more probable that problems with the NAS killed the disk than a faulty disk causing theese errors?


*edit*

Check FS report:

Volume scan found no errors.

***** File system check forced at Wed Mar 26 22:18:38 CET 2014 *****
fsck 1.42.8 (20-Jun-2013)
e2fsck 1.42.8 (20-Jun-2013)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/c/c: 132184/91258880 files (5.5% non-contiguous), 626717031/1460142080 blocks


Shouldnt it find errors with one disk dead?
Message 11 of 22
LeoW
Aspirant

Re: Abysmal CIFS performance, ReadyNAS pro 4

fastfwd wrote:
LeoW wrote:
Before this it had been working fine for several months. .... The NAS and PC are wired to my ASUS RT-AC68U .... NAS is wired, 2Gbit/ Full duplex. Teaming - adaptive load balancing. No Jumboframes. PC is wired Gigabit ethernet.


When it was working fine, was teaming enabled? And were you using the same router, running the same router firmware as you are running now? And were you using the same Ethernet cables, plugged into the same ports on the router?



Teaming was enabled for several months without problems. A few months back I upgraded from the router my ISP gave me to the AC68U. I have tried switching the router to another AC68U, I have several in my network using the other as media bridges, so i swapped the router. No effect though. The firmware on the router has been uppgraded to the latest firmware. I have however also tried downgrading it to earlier, proven to work in my network, firmware without effect. I did use the same cables in the same ports to begin with. I have tried new cables as well. The latest attempt with yet another new cat 6 cable connected only to ethernet 2 seemed to work for speed. Then it was the disk dying.
Message 12 of 22
StephenB
Guru

Re: Abysmal CIFS performance, ReadyNAS pro 4

A failing disk can certainly hurt performance.

The main thing at this point is to replace it promptly. I strongly recommend updating (or making) a backup right away.

Then circle back to the teaming.
Message 13 of 22
LeoW
Aspirant

Re: Abysmal CIFS performance, ReadyNAS pro 4

StephenB wrote:
A failing disk can certainly hurt performance.

The main thing at this point is to replace it promptly. I strongly recommend updating (or making) a backup right away.

Then circle back to the teaming.


I did order a new disk yesterday, will keep the NAS off untill having replaced the disk. Then I´ll check if the disk is really broken. If it is ill RMA the disk. The question I am left with now is could the disk really be the problem? Hurting performance for more than a week without reporting eny errors whatsoever before beeing reported dead?

My plan right now should probably be like:
1. Replace the disk 1 with a new disk.
2. Test performance with teaming on, and if that shoud fail, off with only ethernet 2 connected.
3. Connect the disk to PC and run diagnostics, if it seems ok Ill reinsert it as disk 4 in the NAS and test performance again?
4. Repport back here with results, probably monday.
Message 14 of 22
StephenB
Guru

Re: Abysmal CIFS performance, ReadyNAS pro 4

According to the (somewhat dated) Google study, 36% of the drive failures they saw did not record any SMART errors prior to failure.
Message 15 of 22
LeoW
Aspirant

Re: Abysmal CIFS performance, ReadyNAS pro 4

StephenB wrote:
According to the (somewhat dated) Google study, 36% of the drive failures they saw did not record any SMART errors prior to failure.


Hmm. The drive is as old as the NAS, about a year. A WD RED 3TB, supposedly 24/7 NAS drive. Oh well I guess Il just RMA it without attempting to be to clever with it. Ill just:

1. Replace the disk 1 with a new disk.
2. Test performance with teaming on, and if that shoud fail, off with only ethernet 2 connected.
3. Repport back here with results, probably monday.
Message 16 of 22
StephenB
Guru

Re: Abysmal CIFS performance, ReadyNAS pro 4

Normally WDC would want you to run their lifeguard program (on a PC) to test the drive. Of course you can't do that unless you have a suitable PC.
Message 17 of 22
LeoW
Aspirant

Re: Abysmal CIFS performance, ReadyNAS pro 4

What I’ve done:
So, I’ve been out of town for work during the week. Before I left the new drive hadn’t arrived. I did want to attempt to have it running while away though. I kept the NAS powered of and removed the power chord. Then, Sunday evening a week ago, I powered it up and it rebuilt on the same drive (the faulty? Drive, now seemingly good as new). It took several hours, but since then it runs. It reports 0 Errors and speed is good.


Results, speed test:

Today I tested transfer performance. ~90MB/s using the settings that we worked out last week (No teaming, only Ethernet 2).
I then tried connecting Ethernet 1 as well, enabling teaming – balanced load adaptation, getting, good speeds, mostly 112MB/s but with some variation down to ~90 and (except for sometimes a slower first few seconds) one drop to about 20MB/s for ~10 seconds over the course of 10 3-10GB file transfers. Seems reasonable, if it keeps on like this, I will be very happy.

What to do?
What to do now? I still haven’t installed the new disk, and I haven’t dared upgrade to 4.2.26 again. The first time I upgraded was about the time the first problems started, and the last time I upgraded, disk 1 got reported bad. Can I trust disk 1? I cant shake the feeling of something going wrong in the update process hurting the FS integrity of disk 1, and it seems silly replacing it now. I could just use the new disk to expand volume. Also, I would like to run 4.2.26, if only for the TimeMachine fixes.

Unfortunately, work will bring me away this week as well, will be back next weekend. Any guidance on how to proceed?
Message 18 of 22
StephenB
Guru

Re: Abysmal CIFS performance, ReadyNAS pro 4

If the SMART stats are all good, then I'd say you are ok on the disks. Though I'd keep a close watch on the "failed" for a while, and keep the new one unused for a bit (so you can immediately switch to it if needed).

I'd run a disk scrub and a file system check next. After that completes, I'd try the 4.2.26 upgrade again.
Message 19 of 22
LeoW
Aspirant

Re: Abysmal CIFS performance, ReadyNAS pro 4

SMART stats looks fine, posting logs below (in english this time).

No errors as far as I can see. Im a little bit bothered that the stats are not all the same however, they all have 5437 Power On Hours, but the start stop count and some other stats differ slightly, normal?


SMART Information for Disk 1

Model: WDC WD30EFRX-68AX9N0
Serial: WD-WMC1T0863486
Firmware: 80.00A80
SMART Attribute
Raw Read Error Rate 0
Spin Up Time 6250
Start Stop Count 468
Reallocated Sector Count 0
Seek Error Rate 0
Power On Hours 5437
Spin Retry Count 0
Calibration Retry Count 0
Power Cycle Count 468
Power-Off Retract Count 5
Load Cycle Count 462
Temperature Celsius 33
Reallocated Event Count 0
Current Pending Sector 0
Offline Uncorrectable 0
UDMA CRC Error Count 0
Multi Zone Error Rate 0
ATA Error Count 0

SMART Information for Disk 2

Model: WDC WD30EFRX-68AX9N0
Serial: WD-WMC1T1143203
Firmware: 80.00A80
SMART Attribute
Raw Read Error Rate 0
Spin Up Time 5825
Start Stop Count 469
Reallocated Sector Count 0
Seek Error Rate 0
Power On Hours 5437
Spin Retry Count 0
Calibration Retry Count 0
Power Cycle Count 469
Power-Off Retract Count 5
Load Cycle Count 463
Temperature Celsius 34
Reallocated Event Count 0
Current Pending Sector 0
Offline Uncorrectable 0
UDMA CRC Error Count 0
Multi Zone Error Rate 0
ATA Error Count 0

SMART Information for Disk 3

Model: WDC WD30EFRX-68AX9N0
Serial: WD-WMC1T1497744
Firmware: 80.00A80
SMART Attribute
Raw Read Error Rate 0
Spin Up Time 5958
Start Stop Count 474
Reallocated Sector Count 0
Seek Error Rate 0
Power On Hours 5437
Spin Retry Count 0
Calibration Retry Count 0
Power Cycle Count 474
Power-Off Retract Count 10
Load Cycle Count 463
Temperature Celsius 32
Reallocated Event Count 0
Current Pending Sector 0
Offline Uncorrectable 0
UDMA CRC Error Count 0
Multi Zone Error Rate 0
ATA Error Count 0
Message 20 of 22
StephenB
Guru

Re: Abysmal CIFS performance, ReadyNAS pro 4

I agree they look fine. I see no need for concern on the slightly different start/stop counts.

The disk scrub (under volume maintenance) will access every sector in the raid array, so it is an easy way to ensure that every sector in the raid array (on every drive) is readable. I would schedule that next. Personally I schedule the disk scrub every 16 weeks and the on-line file system check every week.
Message 21 of 22
LeoW
Aspirant

Re: Abysmal CIFS performance, ReadyNAS pro 4

Success! And then further problems.

Ok, So I ran Disscrub and the file systemcheck. Seemingly without problems. Can't find a log for the disk-scrub though?

Now disk 3 is reported dead. At the same time the NAS says "Volume C, RAID-level: X2. Redundant" Disk 3 was also reported dead earlier for a short while as I stated in post 11 on page 1. At that time disk 1 was also dead, which should have meant the data was lost, but it was accessible. And if disk 3 is now dead, why does it state redundant?



FS check log:
***** Volume maintenance: Online file system check performed at Mon Apr 14 00:00:10 2014 *****
Warning! /dev/c/c_2014_04_14_00_00 is mounted.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/c/c_2014_04_14_00_00: 132423/91258880 files (5.5% non-contiguous), 634885341/1460142080 blocks
Message 22 of 22
Top Contributors
Discussion stats
  • 21 replies
  • 3084 views
  • 0 kudos
  • 3 in conversation
Announcements