This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
RN212 with 2 12TB Seagate drives was working fine last night. Added a backup session to copy a folder from an RN528X to this RN212, which completed successfully overnight. Small transfer, only a few GB. This morning, the RN212 disk lights were active and I can hear the drives spinning constantly some 6 hours later at 2:40pm now. About 2 hours ago, accessed the Web GUI admin page and observed that access was incredibly slow, took well over 2 minutes for admin page to populate. Same issue on Windows Explorer trying to access shares - takes a very long time. Selected "Restart" which brought up the prompt "device is rebooting" and I could see the power button begin flashing. That power button has been flashing for 2 hours now, disks still constantly spinning, and now I can't access the GUI. Seems it's stuck in a loop. This is the same NAS that mysteriously lost all its data a few months ago, I restored it using a USB dongle which I commented on in another thread.
When a NAS is in this state, do I:
- Pull the AC plug to force power off, and then restart it
- Wait even longer for graceful shutdown to occur - if so, how much longer?
But it does show that having AV on, on those lower end units, is probably a bad idea.
As FileSearch brings you little value anyway, I would turn it off. I reckon when it starts to index new files you will probably see a spike in "tracker-miner" mem usage as well.
RN212 with 2 12TB Seagate drives was working fine last night. Added a backup session to copy a folder from an RN528X to this RN212, which completed successfully overnight. Small transfer, only a few GB. This morning, the RN212 disk lights were active and I can hear the drives spinning constantly some 6 hours later at 2:40pm now. About 2 hours ago, accessed the Web GUI admin page and observed that access was incredibly slow, took well over 2 minutes for admin page to populate. Same issue on Windows Explorer trying to access shares - takes a very long time. Selected "Restart" which brought up the prompt "device is rebooting" and I could see the power button begin flashing. That power button has been flashing for 2 hours now, disks still constantly spinning, and now I can't access the GUI. Seems it's stuck in a loop. This is the same NAS that mysteriously lost all its data a few months ago, I restored it using a USB dongle which I commented on in another thread.
When a NAS is in this state, do I:
- Pull the AC plug to force power off, and then restart it
- Wait even longer for graceful shutdown to occur - if so, how much longer?
- Other?
This can't be good for the HDD's...
Also tried holding power button down for 5 seconds per the user manual to force a shutdown - does nothing. Power button still blinking and activity light still rapidly blinking.
I think at this point, you need to just pull the plug.
We need to try and figure out why it happened though, that is probably important. I can help look at your logs when the NAS is back up an running. Go System > Logs > Download logs and it will give you all the logs in a zip file. Upload this zip file to Google Drive, Dropbox or similar and PM me a link to where I can grab then. Then I will take a look for you.
But at this stage, the unit seems totally locked up, not much support can do here either. Power cycle is likely the only way.
And you have verified that it's not in the middle of a re-sync, balance, scrub, or similar operation? Interrupting those processes with a forced power cycle is not recommended. If you have SSH enabled, top may give you a clue what is going on with regard to the activity light.
And you have verified that it's not in the middle of a re-sync, balance, scrub, or similar operation? Interrupting those processes with a forced power cycle is not recommended. If you have SSH enabled, top may give you a clue what is going on with regard to the activity light.
Hi @Sandshark , I did not do that as communications with the machine was very sluggish. I finally gulped and pulled power, realizing the risk. It rebooted fine, so after that I downloaded the log zip file per another request, and then upgraded the unit to the 6.10.5 beta. So far it's stable (12 hours later), although the AV version is still stuck on Sept 24, 2020 (something this beta allegedly fixed).
6.10.5 final is out, so you should go with that rather than the beta, but others have complained that the antivirus update isn't fixed in it, either.. If the NAS was busy doing something, there should have been an entry in the log saying what it started and when. The big thing is that you didn't harm anything by pulling the power.
With "6.10.5-T11 (Beta 1)" loaded, the system update checker says software is fully up to date. So if there's an official 6.10.5 the device doesn't want to fiind it.
In any case, thanks for the logs. Your raid, disks and filesystem appear healthy. The kernel log shows that the NAS ran into a low memory condition, early morning on the 8th of May at 1:14 am. The kernel started to kill process to try and free up memory, as a result.
It started with "mkdir", which is "make directory". It is a bit odd this would be killed but it might be related to the backups running over night? Regardless, the process is killed.
May 08 01:14:30 Kirkpatrick2016 kernel: mkdir invoked oom-killer: gfp_mask=0x24200ca, order=0, oom_score_adj=-1000
May 08 01:14:30 Kirkpatrick2016 kernel: mkdir cpuset=/ mems_allowed=0
May 08 01:14:30 Kirkpatrick2016 kernel: CPU: 2 PID: 10055 Comm: mkdir Tainted: P W O 4.4.190.alpine.1 #1
May 08 01:14:30 Kirkpatrick2016 kernel: Hardware name: Annapurna Labs Alpine
May 08 01:14:30 Kirkpatrick2016 kernel: [<c0014574>] (unwind_backtrace) from [<c0011460>] (show_stack+0x10/0x14)
May 08 01:14:30 Kirkpatrick2016 kernel: [<c0011460>] (show_stack) from [<c036ea80>] (dump_stack+0x7c/0x9c)
May 08 01:14:30 Kirkpatrick2016 kernel: [<c036ea80>] (dump_stack) from [<c00d679c>] (dump_header.constprop.5+0x44/0x174)
May 08 01:14:30 Kirkpatrick2016 kernel: [<c00d679c>] (dump_header.constprop.5) from [<c00a0c84>] (oom_kill_process+0xe8/0x4cc)
May 08 01:14:30 Kirkpatrick2016 kernel: [<c00a0c84>] (oom_kill_process) from [<c00a1338>] (out_of_memory+0x26c/0x2ec)
May 08 01:14:30 Kirkpatrick2016 kernel: [<c00a1338>] (out_of_memory) from [<c00a4d98>] (__alloc_pages_nodemask+0x710/0x7a8)
May 08 01:14:30 Kirkpatrick2016 kernel: [<c00a4d98>] (__alloc_pages_nodemask) from [<c00bdbbc>] (handle_mm_fault+0x848/0x10f8)
May 08 01:14:30 Kirkpatrick2016 kernel: [<c00bdbbc>] (handle_mm_fault) from [<c00166c4>] (do_page_fault+0x1a0/0x290)
May 08 01:14:30 Kirkpatrick2016 kernel: [<c00166c4>] (do_page_fault) from [<c0009260>] (do_DataAbort+0x2c/0xac)
May 08 01:14:30 Kirkpatrick2016 kernel: [<c0009260>] (do_DataAbort) from [<c0011fdc>] (__dabt_usr+0x3c/0x40)
May 08 01:14:30 Kirkpatrick2016 kernel: Exception stack(0xdf7a1fb0 to 0xdf7a1ff8)
Afterwards, we can see in the kernel log memory dump that it lists processes currently consuming memory. I have listed the biggest offenders here. Notice how clamd has a "oom_score_adj" score of 1000. This means the kernel will kill/sacrifice this process if the NAS gets memory started further. Higher oom_score_adj value = higher chance process will get sacrificed if needed. clamd (your antivirus) and tracker-miner (file search) are the biggest memory hogs it seems.
As clamd has a high oom_score_adj score, it is sacrificed next...
May 08 01:14:30 Kirkpatrick2016 kernel: Out of memory: Kill process 27644 (clamd) score 1363 or sacrifice child
May 08 01:14:30 Kirkpatrick2016 kernel: Killed process 27644 (clamd) total-vm:1027572kB, anon-rss:501228kB, file-rss:0kB
But it seems that even with the kernel killing processes, it still can't free up enough memory. ReadyCloud connection fails as well due to out of memory and soon after, at 3:03 am the NAS locks up and no more logging occurs.
May 08 02:54:32 Kirkpatrick2016 systemd[1]: radar.service: Failed with result 'start-limit-hit'.
May 08 02:54:34 Kirkpatrick2016 systemd[1]: systemd-journald-audit.socket: Cannot add dependency job, ignoring: Unit systemd-journald-audit.socket is masked.
May 08 02:54:35 Kirkpatrick2016 systemd[1]: radar.service: Start request repeated too quickly.
May 08 02:54:36 Kirkpatrick2016 systemd[1]: radar.service: Failed with result 'start-limit-hit'.
May 08 02:55:12 Kirkpatrick2016 systemd[1]: radar.service: Failed to fork: Cannot allocate memory
May 08 02:55:12 Kirkpatrick2016 systemd[1]: radar.service: Failed to run 'start' task: Cannot allocate memory
May 08 02:55:13 Kirkpatrick2016 systemd[1]: radar.service: Failed with result 'resources'.
I can see that you then rebooted the NAS at 19:25 the same day.
I went back and looked at previous kernel log entries. It is a frequent occurrence that the NAS kills processes to free up memory. Mostly, clamd (antivirus) gets sacrificed (due to its high oom_score_adj value).
My overall feeling is that the NAS is rather starved of memory, quite often. It does not always lead to a crash, but on the 8th of May, it did.
I don't think you can update memory on these models as it is soldered on to the motherboard. @Sandshark will know 🙂 But maybe consider if you really need things like AntiVirus and FileSearch enabled on your backup unit. Else, turn them off.
You can monitor memory usage on the NAS if you like, via the top command (over SSH - I can see you have SSH enabled so are probably familiar with it).
top -o %MEM
That will monitor highest memory consumers in real time. Might be a good idea to watch this over a period of time and see which processes are hogging all your memory.
The RN212 does not have upgradeable memory. AFAIK, no ReadyNAS with an ARM processor has upgradeable memory, though I've not seen them all. But it makes sense that a NAS with a soldered-on CPU to save money would also have soldered-on memory for the same reason, especially since Netgear considers a simple memory upgrade as voiding your warranty.
If your main access to the NAS is via SMB, then file search is doing little to nothing for you, so start by turning that off. But there don't seem to be a lot of folks reporting a similar problem, which makes me think you're running something unusual that's pushing it over the cliff.
I only used SMB sparingly, mainly in a failed attempt to increase bandwidth using dual-ethernet connections. So although familiar with it, I never use it really.
With that in mind, using the generic Web GUI Admin page, I looked through everything there. There is only 1 app loaded, "SMB Plus". There is only one Cloud app active, "ReadyCloud". Other cloud apps are loaded but all are switched off. There are no Backups in the scheduler. On System/Settings, the following are enabled: SMB, ReadyDLNA, Rsync, PnP, HTTP, HTTPS, SSH, Antivirus, FileSearch.
So I don't see what extraordinary process would be running that would cause the device to run out of RAM, other than maybe the indexing function. I (maybe stupidly) thought indexing was a feature that would enable the NAS to fetch files more quickly, similar to Windows indexing. If this is not the case - what is it for? It takes a very long time for indexing to complete on 5TB of JPEGs/etc. If I don't need it, and it's taking up valuable RAM, then I'll kill it. When I click on "Indexing" it says "Indexed: 712319 files and 82578 folders". Maybe this is the culprit???
FileSearch, which is the indexing service, does help as you state if you are using the NAS GUI to access files. But that's not the efficient way to use the NAS. Using SMB just means directly accessing from Windows, usually with one or more mapped drives. It has nothing to do with dual Ethernet connections and such.
82K files and folders no small amount to index, for a unit with an ARM processor and 2GB of RAM. I expect this probably takes a whole of memory usage to do and is quite possibly where the memory hog is.
The indexing helps in the GUI search for files but I would recommend using SMB to access your files instead, from the Windows explorer. Using the web interface for file handling, is far inferior to SMB.
I would turn of the FileSearch feature and use SMB as primary interaction with the files.
Sorry for poor wording - to be clear I enabled SSH and employed PowerShell to implement commands to run SMB multichannel protocol for ETH channel bonding. SSH was needed to get in there and hack the SMB folder on the NAS. You'll recall this thread, from a while back:
Thanks @rn_enthusiast , I didn't realize indexing is only useful for the GUI-based file system. You're right of course, I never use that. Who does??? Cumbersome to say the least. I always use Windows Explorer. So I will now turn off Indexing as well as AV. This may also explain why the NAS crashed a few months ago, forcing me to reinstall the OS from USB drive. Thankfully it restored everything, but I bet the root cause of that was this memory issue. Great catch, thanks very much for combing through the logs!!!
Disabled File Indexing and AV, and rebooted machine. Ran the "top -o %MEM" command, shows that about half the memory is being used now. This command is great, it refreshes the memory usage every few seconds. With that in mind, I turned AV back on (real-time scanning and OS protection check boxes) and watched the memory usage climb from 994M to nearly 1.9G within just a few seconds. After a minute or so of watching memory usage oscillate between 1.6-1.9 GB, I turned AV back off. The memory usage dropped all the way back down to around 650 MB or so. The 3 pictures below represent these 3 states (AV/Indexing off, rebooted; AV re-enabled; AV disabled again). Strange that memory usage fell so far after re-disabling AV, but it does seem to show that enabling AV burns a hell of a lot of RAM.
AV/Indexing disabled, fresh rebootAV re-enabledAV disabled again
But it does show that having AV on, on those lower end units, is probably a bad idea.
As FileSearch brings you little value anyway, I would turn it off. I reckon when it starts to index new files you will probably see a spike in "tracker-miner" mem usage as well.
