NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Solved
ReadyNAS 214 BTRFS corruption - appears to be out of memory related
My ReadyNAS RN214, which has been reliably running for years, suddenly went read-only late yesterday afternoon. The UI log just had this ominous message: Jan 30, 2023 16:54:47 Volume: The volume...
Thanks for sharing this. I am wondering what apps and services you have running, and also what firmware version.
LimaAlphaHotel wrote:
- Is it possible to upgrade the memory in the RN 214, to prevent this happening again?
No - unfortunately it is not socketed, so it can not be upgraded.
Sorry for not replying quickly, I had a very long day at work so did not have chance to yesterday.
Replying to each question:
Was the volume still syncing?
No, that finished within a couple of days of me doing the factory reset:
03 Feb 2023 05:00:32 Volume: Volume data is resynced.
Also, did you have multiple threads running for the file restore?
Not sure what you mean, I copied each share back one-by-one from USB drives using a single rsync or cp command (over rsync or cifs respectively). This out of memory happened the same day but many hours (>10) after the restores had been finished - nothing was actively using the NAS at the time, as far as I know - this time there's nothing in the UI log (see below) so I have not yet worked out exactly what time it happened. As with last time, I only noticed when I tried to use it and found it was unresponsive (last time, I discovered it had failed when I found it was read-only).
This time I didn't get an email either, it just seems to have locked up with the message on the LCD. The first time (on 31st January) I had an email that said "The volume data encountered an error and was made read-only. It is recommended to backup your data.".
How large are the drives?
10TB
just searching for what did tip the memory usage over the top so you and others can avoid it
I've not had chance to examine the logs this time - will be doing that very shortly. Fortunately this time BTRFS seems to have survived, probably because nothing was accessing the NAS at the time so there were no writes "in flight" when it ran out of memory.
I am also wondering if the drives are SMR or CMR (since SMR can have very low write speeds, which might result in excessive memory buffering).
Errr... I had to google that one - they are Seagate IronWolf NAS drives, 7200RPM (all the same model but I sourced them from different retailers in the hope that reduce the risk of them being from the same batch) - according to scan's website they're CMR.
Drive health might also factor in. Did you test the drives?
Yes, I did that the first time and again - health checks are reporting the drives are all fine. All are reporting zero ATA errors, which was the indication of a faulty disk last time I had a drive fault. Two of the drives have single-digit numbers of reallocated sectors but they have been stable like that for a long time and counts have not increased while I've been having these issues - my understanding is modern drives will reallocate sectors automatically and unless the numbers start getting large or start growing in a shortish time that is probably nothing to worry about?
I've attached the kernel log (I'm afraid as a PDF as the forum won't let me attach a text file) - looks like the OOM reaper went after apache2 and syslog-journald (which, in the latter case, kept respawning).
But the processes are using relatively little memory (although, what is oath2-vault and how do I kill it? I assume it's related to ReadyNAS Vault but that is disabled and has never been enabled) - it looks to me like the majority of memory is used by buffers/cache (~1.2GB). I wonder what is causing that, and is there a way through the UI to tune it?
Here's a landscape version of the log, not much better but slightly more readable....
LimaAlphaHotel wrote:
(although, what is oath2-vault and how do I kill it? I assume it's related to ReadyNAS Vault but that is disabled and has never been enabled) - it looks to me like the majority of memory is used by buffers/cache (~1.2GB). I wonder what is causing that, and is there a way through the UI to tune it?
FWIW, it is not running on my RN526 (running 6.10.7), and I am not seeing it in the logs for my RN202 either. oauth2 generally is an authentication protocol.
Are you using any other cloud services on the cloud page? Also, what email provider are you using for alerts?
LimaAlphaHotel wrote:
I've attached the kernel log (I'm afraid as a PDF as the forum won't let me attach a text file)
Another option is to put the log into cloud storage (dropbox, etc) and include a download link.
There is some information leakage, so generally I advise people providing full logs to send them in a private message.
FWIW, it is not running on my RN526 (running 6.10.7), and I am not seeing it in the logs for my RN202 either. oauth2 generally is an authentication protocol. Are you using any other cloud services on the cloud page? Also, what email provider are you using for alerts?
Yes, I know oauth2 is an authentication protocol - I meant more why is it running on my ReadyNAS and how do I turn it off. There are no cloud services enabled (nor have ever been - screenshot attached). Email is direct to my own email server (plain old SMTP), authenticated (username/password) with the server directly - no oauth there either.
Enabling CIFS, RSYNC and NFS, recreating the shares, configuring SMTP for email alerts and configuring the UPS client are literally the only things I've changed since the rebuild after the initial failure until it fell over again. ReadyDLNA and uPnP were on by default and I've since turned them off. Nothing else has been enabled and no apps at all installed this time.
Another option is to put the log into cloud storage (dropbox, etc) and include a download link. There is some information leakage, so generally I advise people providing full logs to send them in a private message.
I don't use dropbox, one drive or anything like that.
I did a cursory check for anything sensitive before I posted it.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
