I've been using ReadyNAS for years now, and I have had only good things to say, up until now that is.
This particular unit suddenly started to lag badly last week, out of the blue.
Configuration:
- ReadyNAS 316, about 4 years old.
- Firmware on the unit is 6.10.2 - it was updated a few months ago, so the lagging is not a result of a firmware update.
- System contains 3 x WD Red 4TB drives in RAID 5 configuration. All drive statuses appear as NEW, with no ATA errors.
- 1.63TB of free data on the volume. Defrag occurs weekly on a Friday after hours.
- Snapshots run hourly on the main share, and are configured as smart.
- Auditing is enabled to track all writes / deletions to the main share.
- Wasabi cloud backup is running, and file changes are pushed to the cloud on the fly - this has been running smoothly for months.
- Server is connected via GB Lan connection - 2 x LAN ports configured for Adaptive Load Balancing (is this an ideal configuration?).
- IPV6 is disabled (could this be an issue?).
Symptoms:
- Lag ranging from 5 seconds up to 20+ seconds while navigating the folder structure on the server.
- Lag while saving files (if working live on the server).
- During file copies, the initial calculation seems to lag, but then the copy process itself occurs at more or less normal speed.
- The GUI itself appears quite laggy and slow to respond in general.
- Symptoms are more severe on Macs (running Mojave - to the point where software will crash altogether while saving), but PC's also experience a degree of lag while navigating or saving files.
- The last 3 reboots / shutdowns were unable to complete gracefully - I eventually had to pull the plug on the NAS in order to force a reboot. I received a near heart attack after one of these reboots when the NAS got stuck booting at 27%, and then displayed: "kthread_data+7", and just hung there. I pulled the plug again and the NAS eventually came back to life after about 10 minutes booting (which is abnormally long).
- I shut it down today, and even after issuing the shutdown from the GUI, it remained available on the LAN (although slow), and the GUI was still up (but slow). The device showed "Shutting Down. Goodbye" on the display.
I tried to force 3 shutdowns from the crippled GUI, and on the 3rd try, the NAS eventually stopped responding to pings. I waited for the hard drive lights to stop flashing and remain stable, and then I pulled the plug. The next boot happened at about normal speed, so I was hopeful, but the symptoms persisted afterwards.
What I have tried:
- Connecting from the Macs via AFP, SMB and CIFS. CIFS seems to be slightly quicker somehow, but not by much.
- Disabling Quotas. This worked for me on a previous firmware version where the same symptoms were occurring on the hour during snapshot pruning.
- Disabling SMB Strict Sync.
- For SMB: enabling Enhance MacOS.
- Disabling Auditing and Clearing auditing logs in case these are causing an issue.
- Culling about 2 years of snapshots manually.
- SSH'ing into the device does not show any processes consistently using excess CPU or MEM, although rnotifyd spikes occasionally up to as high as 60%. What does rnotifyd do?
I can't imagine that the 1.63TB free space is a problem - surely this is sufficient?
I have had other ReadyNAS servers run fine with less free space.
I am also very wary of running Balances / Scrubs, as I know that these can take insanely long and often end up locking up the NAS.
Can a RAID simply "go bad" after a few years? Also, why would that lag the GUI?
I'm running out of things to try, and this previously excellent NAS is now becoming more of a headache... any ideas would be much appreciated.
Thanks all 🙂
Model: RN31600|ReadyNAS 300 Series 6- Bay (Diskless)
- SSH'ing into the device does not show any processes consistently using excess CPU or MEM, although rnotifyd spikes occasionally up to as high as 60%. What does rnotifyd do?
It's used by auditing - you'll find it disappears when you turn auditing off.
I can't imagine that the 1.63TB free space is a problem - surely this is sufficient?
You have an 8 TB data volume, so you should have about 18% free space. That should be fine (though I don't recommend letting the free space drop much below 15%.
I am also very wary of running Balances / Scrubs, as I know that these can take insanely long and often end up locking up the NAS.
They've never locked up my RN526x. As far as balance goes, if it doesn't have much to do then it will run pretty quickly. I run them every three months - the most recent two took about 30 minutes on a 26TB volume (about 14 TB of data). I do suggest running a balance, and see if that helps.
Note the GUI is run at a fairly low priority, so anything that takes a lot of CPU resources will add lag to the GUI.
- Server is connected via GB Lan connection - 2 x LAN ports configured for Adaptive Load Balancing (is this an ideal configuration?).
- IPV6 is disabled (could this be an issue?).
Definitely leave IPv6 off unless you use it on your home network. It can hurt performance.
There are only two modes of bonding that you can use without a smart or managed switch - ALB and TLB. ALB can sometimes misbehave, so you could try disconnecting the second LAN port and disabling the bond.
Generally not, as long as the disks remain healthy. You mention ATA errors specifically, but you might also look for reallocated or pending sectors and command timeouts.
Though if your volume is very old, you could see some performance improvements if you do a factory default and restore the files from a backup. One aspect is that Netgear did modify the on-disk setup for BTRFS a long time ago to improve performance - but that required a factory reset to apply those changes.
