RN104 Poor network performance after upgrade to firmware 6.9.3

Just done some reading and not sure that ata_errors are a worry point just yet, as they seem to coincide with a reboot

https://kb.netgear.com/19392/ATA-errors-increasing-on-disk-s-in-ReadyNAS

Still pretty sure that the fw is the cause of my network performance and availability issues.

I don't suppose you saw anything else in the logs that show any other types of issues, especially building after 11pm last night, which is when I did a last power off and power on?

---

Synn wrote:

Just done some reading and not sure that ata_errors are a worry point just yet, as they seem to coincide with a reboot

https://kb.netgear.com/19392/ATA-errors-increasing-on-disk-s-in-ReadyNAS

 

---

Perhaps run a disk test from the volume settings wheel (system->volume page).  Download the logs before and after the test, and then see if anything changes.

A failing disk will certainly hurt file transfer speed (as will removing one).

Did you also look at the ethernet stats in network_settings.log?

I tried the WD tool and it wouldn't detect the NAS so will look at your alternative

Regarding the network stats, I did - and I can see 12801 dropped RX packets on eth1, as an example

No idea how to fix these as not sure if it's the NIC in the NAS that might be faulty now, or some other reason

---

Synn wrote:

I tried the WD tool and it wouldn't detect the NAS so will look at your alternative 

---

Diagnostics need direct access to the disk itself.  You have to remove the drive, and connect it to the PC (either with SATA or via a USB adapter/dock).  

---

Synn wrote:

 

Regarding the network stats, I did - and I can see 12801 dropped RX packets on eth1, as an example 

---

That is worth pursuing.  However it could be a side effect (if the system can't write to the RAID array fast enough, eventually it will have to drop packets).

What about errors, frame, overruns, and carrier collisions?  You can cut/paste the stats and post here if you want us to take a look.  This is also a case where downloading the logs, doing some tests (e.g, file transfers), and then downloading the logs again can be useful.

You can also test basic connectivity from a PC.  Run the command prompt (CMD) and enter ping nas-ip-address -n 10000  Use the real nas-ip-address.  That will ping the NAS 10000 times, and give you a summary at the end.  (You can also use smaller counts of course).

BTW, is there a reason you still are running fast ethernet?

These are the network stats

Link encap:Ethernet  HWaddr e8:fc:af:e5:46:37  
          inet addr:192.168.1.243  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fdaa:bbcc:ddee:0:eafc:afff:fee5:4637/64 Scope:Global
          inet6 addr: fe80::eafc:afff:fee5:4637/64 Scope:Link
          inet6 addr: 2a00:23c5:915:4a00:eafc:afff:fee5:4637/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST DYNAMIC  MTU:1500  Metric:1
          RX packets:3768537 errors:0 dropped:12801 overruns:0 frame:0
          TX packets:1507920 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:532
          RX bytes:376942335 (359.4 MiB)  TX bytes:28520415460 (26.5 GiB)
          Interrupt:27

No errors or overruns

Running fast ethernet as my home router only supported 100mb network ports, so I had 100mb network ports put in the house a long time ago. The switch that the NAS connects to is only 100mb.

100mb is fine for my needs, but it needs to work and not fluctuate

---

Synn wrote:

 

No errors or overruns

 

---

Yes, so there's nothing definitively pointing to the NIC card or the switch.

But I am seeing that you have ipv6 enabled on the NAS.  Can you try disabling that and retesting?

---

Synn wrote:

 

Running fast ethernet as my home router only supported 100mb network ports, so I had 100mb network ports put in the house a long time ago. The switch that the NAS connects to is only 100mb.

 

Clearly the NAS should keep up with fast ethernet, so this is a sidebar.

But if your cabling is cat5 and connects all 4 twisted pairs end-to-end, then you should be able to use gigabit (assuming the distance isn't too far).  And if the NAS is directly connected to the switch, then there would be a performance benefit if the switch is gigabit, even if you set the house ports to fast ethernet.  For one thing, modern wifi is often faster than fast ethernet.  Another aspect is that multiple devices can access the NAS simultaneously.

Disabling that now, but it's always been like this and didn't have an issue before these firmware updates

I did have to go from 6.2.0 to 6.5.x to 6.9.3

If I had confidence that it was disk 4 on it's way out I'd just replace it, but I'm not 100% on that

The combination of the ATA errors and the I/O error flood is pretty convincing to me that there's a hardware problem of some kind whether it's the disk or something else.

---

Synn wrote:

Disabling that now, but it's always been like this and didn't have an issue before these firmware updates

 

---

I understand your thinking on the firmware update theory.  But other people running 6.9.3 aren't seeing this.

BTW, if this is enabled on your router (as it appears to be), you might need to disable it there as well as the NAS.  Disabling in the router alone isn't enough, as the PCs might use the NAS link-local address.

What apps and services do you have enabled on the NAS?  Do you have quota enabled in the volume settings wheel?

---

Synn wrote:

 

If I had confidence that it was disk 4 on it's way out I'd just replace it, but I'm not 100% on that

---

Can you post the SMART stats (and ATA count) for that drive here?

I don't have any apps installed on the NAS and have the following services turned on

SMB

AFP

iTunes

ReadyDLNA

uPNP

HTTP

HTTPS

Here's the smart_history log

time                 model                 serial                realloc_sect  realloc_evnt  spin_retry_cnt  ioedc       cmd_timeouts  pending_sect  uncorrectable_err  ata_errors
-------------------  --------------------  --------------------  ------------  ------------  --------------  ----------  ------------  ------------  -----------------  ----------
2015-05-31 19:11:52  WDC WD40EFRX-68WT0N0 WD-WCC4YYYYYYYY       0             0             0               -1          -1            0             0                  0         
2015-05-31 19:11:53  WDC WD40EFRX-68WT0N0 WD-WCC4XXXXXXXX       0             0             0               -1          -1            0             0                  0         
2015-06-02 15:14:09  WDC WD40EFRX-68WT0N0 WD-WCC4YYYYYYYY       0             0             0               -1          -1            1             0                  0         
2018-08-13 16:13:13  WDC WD40EFRX-68WT0N0 WD-WCC4XXXXXXXX       0             0             0               -1          -1            0             0                  3         
2018-08-13 16:19:03  WDC WD40EFRX-68WT0N0 WD-WCC4XXXXXXXX       0             0             0               -1          -1            0             0                  4         
2018-08-13 16:39:37  WDC WD40EFRX-68WT0N0 WD-WCC4XXXXXXXX       0             0             0               -1          -1            0             0                  5         
2018-08-15 15:35:23  WDC WD40EFRX-68WT0N0 WD-WCC4XXXXXXXX       0             0             0               -1          -1            0             0                  6         

time                 model                 serial                realloc_sect  realloc_evnt  spin_retry_cnt  ioedc       cmd_timeouts  pending_sect  uncorrectable_err  ata_errors
-------------------  --------------------  --------------------  ------------  ------------  --------------  ----------  ------------  ------------  -----------------  ----------
2018-08-13 16:13:13  WDC WD40EFRX-68WT0N0  WD-WCC4XXXXXXXX       0             0             0               -1          -1            0             0                  3         
2018-08-13 16:19:03  WDC WD40EFRX-68WT0N0  WD-WCC4XXXXXXXX      0             0             0               -1          -1            0             0                  4         
2018-08-13 16:39:37  WDC WD40EFRX-68WT0N0  WD-WCC4XXXXXXXX       0             0             0               -1          -1            0             0                  5         
2018-08-15 15:35:23  WDC WD40EFRX-68WT0N0  WD-WCC4XXXXXXXX       0             0             0               -1          -1            0             0                  6     

It's not just thte ATA error count (6 is still pretty low), but the I/O errors in systemd-journal.log, kernel.log and dmesg.log:

Aug 22 08:45:33 ReadyNAS kernel: blk_update_request: I/O error, dev sda, sector 27152759
Aug 22 08:45:33 ReadyNAS kernel: md/raid1:md127: sda3: rescheduling sector 17452544
Aug 22 08:45:33 ReadyNAS kernel: ata1: EH complete
Aug 22 08:45:46 ReadyNAS kernel: md/raid1:md127: redirecting sector 17452544 to other mirror: sda3
Aug 22 08:45:53 ReadyNAS kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 22 08:45:53 ReadyNAS kernel: ata1.00: irq_stat 0x40000001
Aug 22 08:45:53 ReadyNAS kernel: ata1.00: failed command: READ DMA EXT
Aug 22 08:45:53 ReadyNAS kernel: ata1.00: cmd 25/00:40:40:4e:9e/00:05:01:00:00/e0 tag 0 dma 688128 in
                                          res 51/40:df:a0:51:9e/00:01:01:00:00/e0 Emask 0x9 (media error)
Aug 22 08:45:53 ReadyNAS kernel: ata1.00: status: { DRDY ERR }
Aug 22 08:45:53 ReadyNAS kernel: ata1.00: error: { UNC }
Aug 22 08:45:53 ReadyNAS kernel: ata1.00: configured for UDMA/133
Aug 22 08:45:53 ReadyNAS kernel: sd 0:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 22 08:45:53 ReadyNAS kernel: sd 0:0:0:0: [sda] tag#0 Sense Key : Medium Error [current] [descriptor] 
Aug 22 08:45:53 ReadyNAS kernel: sd 0:0:0:0: [sda] tag#0 Add. Sense: Unrecovered read error - auto reallocate failed
Aug 22 08:45:53 ReadyNAS kernel: sd 0:0:0:0: [sda] tag#0 CDB: Read(16) 88 00 00 00 00 00 01 9e 4e 40 00 00 05 40 00 00
Aug 22 08:45:53 ReadyNAS kernel: blk_update_request: I/O error, dev sda, sector 27152800
Aug 22 08:45:53 ReadyNAS kernel: md/raid1:md127: sda3: rescheduling sector 17452544
Aug 22 08:45:53 ReadyNAS kernel: ata1: EH complete

kernel.log is flooded with these errors so much that the section in the logs zip covers a period of time that's only just over 100 minutes. Seeing a flood of messages about I/O errors immediately suggests disk failure as the most likely situation and if it's not that the next most likely would be some other form of hardware failure.

Judging a failed/failing disk is subjective and you have to take different things into account: the SMART count, when the SMART count rose and by how much, any log flooding with messages about the disk, what problems are being seen that may be symptoms of the failure etc.

It's not necessarily the case that the NAS will give an alert that the disk is failing/failed when that is the case, as the point of failure can vary and is often a subjective call. Obviously if the quick self test indicated that the disk has failed then that's very clear, but a disk can pass a quick self test and still be bad.

As this NAS is literally 3 years old, would it be worth me getting a 6 month Gearhead support engagement to cover the rest of the chassis? (assuming that it effectively provides an extended warranty)

It's not overly clear what the Gearhead contract does.

I'll look at getting a new disk based on this advice

Thanks :)

---

mdgm-ntgr wrote:

 

kernel.log is flooded with these errors so much that the section in the logs zip covers a period of time that's only just over 100 minutes. Seeing a flood of messages about I/O errors immediately suggests disk failure as the most likely situation and if it's not that the next most likely would be some other form of hardware failure.

 

---

 I agree.  SMART doesn't catch everything, and these errors clearly point to either the disk or (less likely) the SATA interface in the NAS.

The warranty on the NAS is three years (not transferrable) - you might check to see if you are covered.

Running lifeguard on the disk is the best way to test it.

If you have a spare disk (whether on the HCL or not), you can test the slot by 

(a) removing all disks, labeling them by slot

(b) insert the spare disk into slot 1 and do a factory install from the boot menu.  Check to see that disk is working properly by doing extended file transfers - tens or (better) hundreds of gigabytes.

(c) power down the NAS and move the disk into slot 4.  Reboot and retest the transfers.  Also download the logs, and look in kernel.log for the I/O errors.

A riskier way is to do the above test with disk 4.  If the i/o errors occur in slot 1, then it's the disk.  if they happen only in slot 4, then it's the system.  If they completely go away, then there's something very weird happening.

But doing the test with disk 4 risks data loss, since you will lose data redundancy on the RAID array.  If you have a full backup of the data, then you could of course take the risk.

No warranty :(

---

Synn wrote:

 

I'll need to look at a new disk either way then

---

Odds are it is the disk.  But if it's the NAS slot, then replacing the disk won't fix it.

New disk has arrived, now got a 20 hour wait while it rebuilds the RAID array, but will let you know if performance is restored when it's complete. Thanks guys

Thanks all - looks like you were right and it was disk in Bay 4. Looks like it was pure coincidence that the disks degraded at the exact same time I update the firmware, but performance seems great again.

Logs are still showing RX errors, however I think they haven't increased (and I'm not sure how to reset the counter), but the ATA count has dropped to 0

Thanks again all for the time to look at this with me :)

I'm glad we were able to help.

---

Synn wrote:

 

Logs are still showing RX errors, however I think they haven't increased (and I'm not sure how to reset the counter),

---

You'd need to reboot.  There's not a high percentage - I don't think they are affecting your performance.