× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

Synn
Aspirant

RN104 Poor network performance after upgrade to firmware 6.9.3

Hi

Recently had to update to FW 6.9.3.

The ReadyNAS is now extremely slow, network wise. Browsing from Windows Explorer seems to have to wake the drive up, and when I try and copy files to it I don't get any consistent speed, it moves up and down from 0kb/s to 10MB/s (ReadyNAS is located on a 100MB network)

Can't see any network settings to confirm duplex or anything, but worked fine on earlier FW versions (went from 6.2.0 to 6.5.x to 6.9.3)

The NAS is basicaly unusable at the minute

 

Any ideas?

Thanks

Model: RN104|ReadyNAS 100 Series 4- Bay
Message 1 of 28
mdgm-ntgr
NETGEAR Employee Retired

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

Have you checked network_settings.log for errors?

 

You may wish to get a gigabit switch and see if connecting the NAS to that and connecting the switch to your router works better.


Or maybe replace your router with one with gigabit ports.

Message 2 of 28
Synn
Aspirant

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

Where do I find that log as it’s not listed in the admin interface

It’s nothing to do with the port. I only have 100mb ports in the house, it’s inconsistent network after the firmware update

I rebooted the NAS and it seeems to work fine for a short period then degrades however I survived to power it off from the plug to reboot. If I try reboot from the admin site it sits at rebooting forever and doesn’t come back

This firmware definitely has bugs
Message 3 of 28
mdgm-ntgr
NETGEAR Employee Retired

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

It's in the logs zip file that you can download. You can send in your logs if you like (see the Sending Logs link in my sig).

Message 4 of 28
Synn
Aspirant

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

Hi

Thanks, I appreciate that - I'll send the logs over now

Message 5 of 28
mdgm-ntgr
NETGEAR Employee Retired

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

The logs are flooded with I/O errors for disk 4 (see e.g. systemd-journal.log). It looks like that disk is bad. smart_history.log shows that the ATA error count on that disk has recently climbed to 6.

 

Disk 3 has a current pending sector count of 1 though that was from way back in June 2015.

Message 6 of 28
Synn
Aspirant

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

Hi

Thanks for that notice, I'll have a look at replacing disk 4. Could the I/O errors be due to bad network throughput hampering disk performance?

The network performance and the network errors I get when copying to it started after the 6.9.3 update

I did notice that it seems to be trying to connect to my network at 1000 half duplex, when it's connected to a 100mb switch.

Is there a way to set the card speed and duplex to stop the mismatch that auto detect is doing?

Also, as I said, the system is super fast straight after a powerdown and reboot, but then degrades over a short period of time (not sure how long, but looks to be over a 4-6 hour period - rebooted last night around 11pm and it was great straight after, checked this morning and it's slow again)

Message 7 of 28
mdgm-ntgr
NETGEAR Employee Retired

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

The disk failure and the firmware update would be a coincidence.

 

eth1 is reporting 100Mb/s Full Duplex in network_settings.log. The info showing for eth0 is not relevant as you're not using that network port.

Message 8 of 28
Synn
Aspirant

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

So do you think that the performance issues are down to drive 4 looking to have read errors and nothing to do with the firmware?

If I pulled Drive 4 and left it as a single disk waiting to rebuild the array that would then confirm if the performance improved - but if I then put Drive 4 back in would it detect it as a newdrive and rebuild the array (while I purchased a new drive)

 

Do the tag#4 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE relate to potential drive failures?

When I look at the volume log I can see 1 pending sector count for sdb, but nothing for sda and both are passing the latest self test

I don't want to have to buy a new drive if that's not the cause, as you can appreciate. Happy to get one if it's pending a failure, but if that still doesn't fix the issue, I'm back to square one with a NAS that no longer performs properly

 

Really appreciate the help on this

Message 9 of 28
mdgm-ntgr
NETGEAR Employee Retired

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

smart_history.log shows that the current pending sector on sdb was years ago.

 

The ata errors on sda are much more recent.

 

The errors point either to a bad disk or much less likely to a hardware problem with the chassis. You may wish to test the disk using WD Data LifeGuard Diagnostics.

Message 10 of 28
Synn
Aspirant

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

Just done some reading and not sure that ata_errors are a worry point just yet, as they seem to coincide with a reboot

https://kb.netgear.com/19392/ATA-errors-increasing-on-disk-s-in-ReadyNAS

 

Still pretty sure that the fw is the cause of my network performance and availability issues.

I don't suppose you saw anything else in the logs that show any other types of issues, especially building after 11pm last night, which is when I did a last power off and power on?

Message 11 of 28
StephenB
Guru

Re: RN104 Poor network performance after upgrade to firmware 6.9.3


@Synn wrote:

Just done some reading and not sure that ata_errors are a worry point just yet, as they seem to coincide with a reboot

https://kb.netgear.com/19392/ATA-errors-increasing-on-disk-s-in-ReadyNAS

 


Perhaps run a disk test from the volume settings wheel (system->volume page).  Download the logs before and after the test, and then see if anything changes.

 

A failing disk will certainly hurt file transfer speed (as will removing one).

 

Did you also look at the ethernet stats in network_settings.log?

Message 12 of 28
Synn
Aspirant

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

I tried the WD tool and it wouldn't detect the NAS so will look at your alternative

Regarding the network stats, I did - and I can see 12801 dropped RX packets on eth1, as an example

No idea how to fix these as not sure if it's the NIC in the NAS that might be faulty now, or some other reason

 

Message 13 of 28
StephenB
Guru

Re: RN104 Poor network performance after upgrade to firmware 6.9.3


@Synn wrote:

I tried the WD tool and it wouldn't detect the NAS so will look at your alternative 


Diagnostics need direct access to the disk itself.  You have to remove the drive, and connect it to the PC (either with SATA or via a USB adapter/dock).  

 


@Synn wrote:

 

Regarding the network stats, I did - and I can see 12801 dropped RX packets on eth1, as an example 


That is worth pursuing.  However it could be a side effect (if the system can't write to the RAID array fast enough, eventually it will have to drop packets).

 

What about errors, frame, overruns, and carrier collisions?  You can cut/paste the stats and post here if you want us to take a look.  This is also a case where downloading the logs, doing some tests (e.g, file transfers), and then downloading the logs again can be useful.

 

You can also test basic connectivity from a PC.  Run the command prompt (CMD) and enter ping nas-ip-address -n 10000  Use the real nas-ip-address.  That will ping the NAS 10000 times, and give you a summary at the end.  (You can also use smaller counts of course).

 

BTW, is there a reason you still are running fast ethernet?

 

 

 

Message 14 of 28
Synn
Aspirant

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

These are the network stats

Link encap:Ethernet  HWaddr e8:fc:af:e5:46:37  
          inet addr:192.168.1.243  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fdaa:bbcc:ddee:0:eafc:afff:fee5:4637/64 Scope:Global
          inet6 addr: fe80::eafc:afff:fee5:4637/64 Scope:Link
          inet6 addr: 2a00:23c5:915:4a00:eafc:afff:fee5:4637/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST DYNAMIC  MTU:1500  Metric:1
          RX packets:3768537 errors:0 dropped:12801 overruns:0 frame:0
          TX packets:1507920 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:532
          RX bytes:376942335 (359.4 MiB)  TX bytes:28520415460 (26.5 GiB)
          Interrupt:27

 

No errors or overruns

 

Running fast ethernet as my home router only supported 100mb network ports, so I had 100mb network ports put in the house a long time ago. The switch that the NAS connects to is only 100mb.

 

100mb is fine for my needs, but it needs to work and not fluctuate

Message 15 of 28
StephenB
Guru

Re: RN104 Poor network performance after upgrade to firmware 6.9.3


@Synn wrote:

 

No errors or overruns

 


Yes, so there's nothing definitively pointing to the NIC card or the switch.

 

But I am seeing that you have ipv6 enabled on the NAS.  Can you try disabling that and retesting?

 


@Synn wrote:

 

Running fast ethernet as my home router only supported 100mb network ports, so I had 100mb network ports put in the house a long time ago. The switch that the NAS connects to is only 100mb.

 

Clearly the NAS should keep up with fast ethernet, so this is a sidebar.

 

But if your cabling is cat5 and connects all 4 twisted pairs end-to-end, then you should be able to use gigabit (assuming the distance isn't too far).  And if the NAS is directly connected to the switch, then there would be a performance benefit if the switch is gigabit, even if you set the house ports to fast ethernet.  For one thing, modern wifi is often faster than fast ethernet.  Another aspect is that multiple devices can access the NAS simultaneously.

 

 

Message 16 of 28
Synn
Aspirant

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

Disabling that now, but it's always been like this and didn't have an issue before these firmware updates

I did have to go from 6.2.0 to 6.5.x to 6.9.3

If I had confidence that it was disk 4 on it's way out I'd just replace it, but I'm not 100% on that

Message 17 of 28
mdgm-ntgr
NETGEAR Employee Retired

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

The combination of the ATA errors and the I/O error flood is pretty convincing to me that there's a hardware problem of some kind whether it's the disk or something else.

Message 18 of 28
StephenB
Guru

Re: RN104 Poor network performance after upgrade to firmware 6.9.3


@Synn wrote:

Disabling that now, but it's always been like this and didn't have an issue before these firmware updates

 


I understand your thinking on the firmware update theory.  But other people running 6.9.3 aren't seeing this.

 

BTW, if this is enabled on your router (as it appears to be), you might need to disable it there as well as the NAS.  Disabling in the router alone isn't enough, as the PCs might use the NAS link-local address.

 

What apps and services do you have enabled on the NAS?  Do you have quota enabled in the volume settings wheel?

 


@Synn wrote:

 

If I had confidence that it was disk 4 on it's way out I'd just replace it, but I'm not 100% on that


Can you post the SMART stats (and ATA count) for that drive here?

Message 19 of 28
Synn
Aspirant

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

I don't have any apps installed on the NAS and have the following services turned on

SMB

AFP

iTunes

ReadyDLNA

uPNP

HTTP

HTTPS

 

Here's the smart_history log

 

time                 model                 serial                realloc_sect  realloc_evnt  spin_retry_cnt  ioedc       cmd_timeouts  pending_sect  uncorrectable_err  ata_errors
-------------------  --------------------  --------------------  ------------  ------------  --------------  ----------  ------------  ------------  -----------------  ----------
2015-05-31 19:11:52  WDC WD40EFRX-68WT0N0 WD-WCC4YYYYYYYY       0             0             0               -1          -1            0             0                  0         
2015-05-31 19:11:53  WDC WD40EFRX-68WT0N0 WD-WCC4XXXXXXXX       0             0             0               -1          -1            0             0                  0         
2015-06-02 15:14:09  WDC WD40EFRX-68WT0N0 WD-WCC4YYYYYYYY       0             0             0               -1          -1            1             0                  0         
2018-08-13 16:13:13  WDC WD40EFRX-68WT0N0 WD-WCC4XXXXXXXX       0             0             0               -1          -1            0             0                  3         
2018-08-13 16:19:03  WDC WD40EFRX-68WT0N0 WD-WCC4XXXXXXXX       0             0             0               -1          -1            0             0                  4         
2018-08-13 16:39:37  WDC WD40EFRX-68WT0N0 WD-WCC4XXXXXXXX       0             0             0               -1          -1            0             0                  5         
2018-08-15 15:35:23  WDC WD40EFRX-68WT0N0 WD-WCC4XXXXXXXX       0             0             0               -1          -1            0             0                  6         
Message 20 of 28
mdgm-ntgr
NETGEAR Employee Retired

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

 

time                 model                 serial                realloc_sect  realloc_evnt  spin_retry_cnt  ioedc       cmd_timeouts  pending_sect  uncorrectable_err  ata_errors
-------------------  --------------------  --------------------  ------------  ------------  --------------  ----------  ------------  ------------  -----------------  ----------
2018-08-13 16:13:13  WDC WD40EFRX-68WT0N0  WD-WCC4XXXXXXXX       0             0             0               -1          -1            0             0                  3         
2018-08-13 16:19:03  WDC WD40EFRX-68WT0N0  WD-WCC4XXXXXXXX      0             0             0               -1          -1            0             0                  4         
2018-08-13 16:39:37  WDC WD40EFRX-68WT0N0  WD-WCC4XXXXXXXX       0             0             0               -1          -1            0             0                  5         
2018-08-15 15:35:23  WDC WD40EFRX-68WT0N0  WD-WCC4XXXXXXXX       0             0             0               -1          -1            0             0                  6     

It's not just thte ATA error count (6 is still pretty low), but the I/O errors in systemd-journal.log, kernel.log and dmesg.log:

Aug 22 08:45:33 ReadyNAS kernel: blk_update_request: I/O error, dev sda, sector 27152759
Aug 22 08:45:33 ReadyNAS kernel: md/raid1:md127: sda3: rescheduling sector 17452544
Aug 22 08:45:33 ReadyNAS kernel: ata1: EH complete
Aug 22 08:45:46 ReadyNAS kernel: md/raid1:md127: redirecting sector 17452544 to other mirror: sda3
Aug 22 08:45:53 ReadyNAS kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 22 08:45:53 ReadyNAS kernel: ata1.00: irq_stat 0x40000001
Aug 22 08:45:53 ReadyNAS kernel: ata1.00: failed command: READ DMA EXT
Aug 22 08:45:53 ReadyNAS kernel: ata1.00: cmd 25/00:40:40:4e:9e/00:05:01:00:00/e0 tag 0 dma 688128 in
                                          res 51/40:df:a0:51:9e/00:01:01:00:00/e0 Emask 0x9 (media error)
Aug 22 08:45:53 ReadyNAS kernel: ata1.00: status: { DRDY ERR }
Aug 22 08:45:53 ReadyNAS kernel: ata1.00: error: { UNC }
Aug 22 08:45:53 ReadyNAS kernel: ata1.00: configured for UDMA/133
Aug 22 08:45:53 ReadyNAS kernel: sd 0:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 22 08:45:53 ReadyNAS kernel: sd 0:0:0:0: [sda] tag#0 Sense Key : Medium Error [current] [descriptor] 
Aug 22 08:45:53 ReadyNAS kernel: sd 0:0:0:0: [sda] tag#0 Add. Sense: Unrecovered read error - auto reallocate failed
Aug 22 08:45:53 ReadyNAS kernel: sd 0:0:0:0: [sda] tag#0 CDB: Read(16) 88 00 00 00 00 00 01 9e 4e 40 00 00 05 40 00 00
Aug 22 08:45:53 ReadyNAS kernel: blk_update_request: I/O error, dev sda, sector 27152800
Aug 22 08:45:53 ReadyNAS kernel: md/raid1:md127: sda3: rescheduling sector 17452544
Aug 22 08:45:53 ReadyNAS kernel: ata1: EH complete

kernel.log is flooded with these errors so much that the section in the logs zip covers a period of time that's only just over 100 minutes. Seeing a flood of messages about I/O errors immediately suggests disk failure as the most likely situation and if it's not that the next most likely would be some other form of hardware failure.

 

Judging a failed/failing disk is subjective and you have to take different things into account: the SMART count, when the SMART count rose and by how much, any log flooding with messages about the disk, what problems are being seen that may be symptoms of the failure etc.

 

It's not necessarily the case that the NAS will give an alert that the disk is failing/failed when that is the case, as the point of failure can vary and is often a subjective call. Obviously if the quick self test indicated that the disk has failed then that's very clear, but a disk can pass a quick self test and still be bad.

 

Message 21 of 28
Synn
Aspirant

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

As this NAS is literally 3 years old, would it be worth me getting a 6 month Gearhead support engagement to cover the rest of the chassis? (assuming that it effectively provides an extended warranty)

It's not overly clear what the Gearhead contract does.

I'll look at getting a new disk based on this advice

 

Thanks 🙂

 

Message 22 of 28
StephenB
Guru

Re: RN104 Poor network performance after upgrade to firmware 6.9.3


@mdgm-ntgr wrote:

 

kernel.log is flooded with these errors so much that the section in the logs zip covers a period of time that's only just over 100 minutes. Seeing a flood of messages about I/O errors immediately suggests disk failure as the most likely situation and if it's not that the next most likely would be some other form of hardware failure.

 


 I agree.  SMART doesn't catch everything, and these errors clearly point to either the disk or (less likely) the SATA interface in the NAS.

 

The warranty on the NAS is three years (not transferrable) - you might check to see if you are covered.

 

Running lifeguard on the disk is the best way to test it.

 

If you have a spare disk (whether on the HCL or not), you can test the slot by 

(a) removing all disks, labeling them by slot

(b) insert the spare disk into slot 1 and do a factory install from the boot menu.  Check to see that disk is working properly by doing extended file transfers - tens or (better) hundreds of gigabytes.

(c) power down the NAS and move the disk into slot 4.  Reboot and retest the transfers.  Also download the logs, and look in kernel.log for the I/O errors.

 

A riskier way is to do the above test with disk 4.  If the i/o errors occur in slot 1, then it's the disk.  if they happen only in slot 4, then it's the system.  If they completely go away, then there's something very weird happening.

 

But doing the test with disk 4 risks data loss, since you will lose data redundancy on the RAID array.  If you have a full backup of the data, then you could of course take the risk.

 

Message 23 of 28
Synn
Aspirant

Re: RN104 Poor network performance after upgrade to firmware 6.9.3

No warranty 😞

 

Standard Product Warranty* :* :
Hardware Warranty:: Not Available
Phone Support:: Not Available
Email Support:: Not Available
Chat Support: : Not Available
Power Supply Warranty:: Not Available
Accessories Warranty:: Not Available
Advanced Support / Hardware Contracts
 
I'll need to look at a new disk either way then

Message 24 of 28
StephenB
Guru

Re: RN104 Poor network performance after upgrade to firmware 6.9.3


@Synn wrote:

 

I'll need to look at a new disk either way then


Odds are it is the disk.  But if it's the NAS slot, then replacing the disk won't fix it.

Message 25 of 28
Top Contributors
Discussion stats
  • 27 replies
  • 2337 views
  • 2 kudos
  • 3 in conversation
Announcements