Re: ReadyNAS 2120 Still crashes after 6.8.1 update

JackyWong · ‎2017-10-22

Hello all,

Here's the issue I have. We bought a ReadyNAS 2120 in 2016. It has been running well, but it start crashes since this month. i do update the firmware to 6.8.1 in last week, but it didn't fix the issue.

Here's what happens when the crash occurs. The NAS stops responding to ping requests, you cannot ssh into it and the web interface is not accessible. Currently, we cannot gracefully reboot it, it has to be shut off via the power button on the back.

There are no Apps running on the NAS, it has a few shared folders and just one NIC card.

Do you guys have any idea what could be causing this, or do you have any advise on what else I could do to try and troubleshoot the issue?

mdgm-ntgr · ‎2017-10-22

So you use the NAS as a basic file server?

Have you checked the diagnostics log in RAIDar for errors?

Have you reviewed systemd-journal.log (or using ssh the output of # journalctl -a) from around when the crashes took place? Hopefully there would be some clues there as to why this is happening.

We do have newer firmware ReadyNAS OS 6.9.0 now available!

JackyWong · ‎2017-10-23

No we use it for Backup. it will crash the system in the middle time during backup

there are no log for errors that show in the admin page.

the attach is the systemd-journal log that from around the crashes time.

It looks like is out of memory problem.

Oct 22 02:06:30 ausy-nas1 kernel: Normal free:82140kB min:41284kB low:51604kB high:61924kB active_anon:48kB inactive_anon:88kB active_file:236772kB inactive_file:237112kB unevictable:0kB isolated(anon):0kB isolated(file):256kB present:786432kB managed:754676kB mlocked:0kB dirty:0kB writeback:0kB mapped:1320kB shmem:0kB slab_reclaimable:25592kB slab_unreclaimable:22848kB kernel_stack:1424kB pagetables:1244kB unstable:0kB bounce:0kB free_pcp:396kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:2878172 all_unreclaimable? yes
Oct 22 02:06:33 ausy-nas1 kernel: lowmem_reserve[]: 0 10240 10240
Oct 22 02:06:33 ausy-nas1 kernel: HighMem free:504kB min:512kB low:18436kB high:36360kB active_anon:0kB inactive_anon:256kB active_file:39936kB inactive_file:1269132kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:1310720kB managed:1310720kB mlocked:0kB dirty:0kB writeback:80kB mapped:28912kB shmem:4kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:252kB local_pcp:124kB free_cma:0kB writeback_tmp:0kB pages_scanned:10564996 all_unreclaimable? yes
Oct 22 02:06:33 ausy-nas1 kernel: lowmem_reserve[]: 0 0 0
Oct 22 02:06:33 ausy-nas1 kernel: Normal: 1015*4kB (UME) 3990*8kB (UME) 2703*16kB (UME) 89*32kB (UME) 1*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 82140kB
Oct 22 02:06:33 ausy-nas1 kernel: HighMem: 0*4kB 5*8kB (M) 7*16kB (UM) 3*32kB (UM) 4*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 504kB
Oct 22 02:06:33 ausy-nas1 kernel: 445914 total pagecache pages
Oct 22 02:06:33 ausy-nas1 kernel: 59 pages in swap cache
Oct 22 02:06:33 ausy-nas1 kernel: Swap cache stats: add 88738, delete 88679, find 53212/90265
Oct 22 02:06:33 ausy-nas1 kernel: Free swap  = 1000332kB
Oct 22 02:06:33 ausy-nas1 kernel: Total swap = 1047420kB
Oct 22 02:06:33 ausy-nas1 kernel: 524288 pages RAM
Oct 22 02:06:33 ausy-nas1 kernel: 327680 pages HighMem/MovableOnly
Oct 22 02:06:33 ausy-nas1 kernel: 7939 pages reserved
Oct 22 02:06:33 ausy-nas1 kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Oct 22 02:06:33 ausy-nas1 kernel: [ 1089]     0  1089     2420      458       6       0      170         -1000 systemd-udevd
Oct 22 02:06:33 ausy-nas1 kernel: [ 2205]    83  2205     1486      543       6       0      106          -900 dbus-daemon
Oct 22 02:06:33 ausy-nas1 kernel: [ 2207]     0  2207     1206      600       6       0       55             0 connmand
Oct 22 02:06:33 ausy-nas1 kernel: [ 2208]     0  2208      486      191       5       0       26             0 raidard
Oct 22 02:06:33 ausy-nas1 kernel: [ 2210]    84  2210      960      462       6       0       62             0 avahi-daemon
Oct 22 02:06:33 ausy-nas1 kernel: [ 2211]     0  2211     3145      362       6       0       36             0 mdcsrepaird
Oct 22 02:06:33 ausy-nas1 kernel: [ 2214]     0  2214      782      301       6       0       83             0 mdadm
Oct 22 02:06:33 ausy-nas1 kernel: [ 2217]     0  2217     1584      638       7       0       74             0 systemd-logind
Oct 22 02:06:33 ausy-nas1 kernel: [ 2220]    84  2220      960      215       5       0       54             0 avahi-daemon
Oct 22 02:06:33 ausy-nas1 kernel: [ 2221]     0  2221     1384      381       6       0       54             0 cron
Oct 22 02:06:33 ausy-nas1 kernel: [ 2259]     0  2259     1109      322       6       0       75             0 rpcbind
Oct 22 02:06:33 ausy-nas1 kernel: [ 2274]    79  2274     1155      305       5       0      124             0 rpc.statd
Oct 22 02:06:33 ausy-nas1 kernel: [ 2293]     0  2293     6327      778      15       0      340             0 nmbd
Oct 22 02:06:33 ausy-nas1 kernel: [ 2313]     0  2313     1071      272       6       0       29             0 agetty
Oct 22 02:06:33 ausy-nas1 kernel: [ 2317]     0  2317     1116      226       6       0       28             0 agetty
Oct 22 02:06:33 ausy-nas1 kernel: [ 2382]     0  2382     4143      943      12       0      667             0 apache2
Oct 22 02:06:33 ausy-nas1 kernel: [ 2390]     0  2390    58607     4418      56       0     7655         -1000 readynasd
Oct 22 02:06:33 ausy-nas1 kernel: [ 2494]    86  2494     3394      967      11       0      410             0 snmpd
Oct 22 02:06:33 ausy-nas1 kernel: [ 2553]     0  2553     2872      947       9       0      129             0 readynas-agent
Oct 22 02:06:33 ausy-nas1 kernel: [ 2558]     0  2558     6521      459      10       0       74             0 fvbackup-q
Oct 22 02:06:33 ausy-nas1 kernel: [ 2585]     0  2585      437      164       5       0       19             0 apache_log
Oct 22 02:06:33 ausy-nas1 kernel: [ 2586]     0  2586     4145      580      11       0      679             0 apache2
Oct 22 02:06:33 ausy-nas1 kernel: [ 2679]     0  2679      723      191       5       0       42             0 rpc.idmapd
Oct 22 02:06:33 ausy-nas1 kernel: [ 2680]     0  2680     1345      441       7       0      162             0 rpc.mountd
Oct 22 02:06:33 ausy-nas1 kernel: [ 3080]    98  3080     4223      650      12       0      736             0 apache2
Oct 22 02:06:33 ausy-nas1 kernel: [ 4655]    98  4655     4223      650      12       0      737             0 apache2
Oct 22 02:06:33 ausy-nas1 kernel: [24792]    98 24792     4225      633      12       0      739             0 apache2
Oct 22 02:06:33 ausy-nas1 kernel: [26000]     0 26000     5663      947       9       0       77             0 systemd-journal
Oct 22 02:06:33 ausy-nas1 kernel: [26019]    98 26019     4151      459      11       0      680             0 apache2
Oct 22 02:06:33 ausy-nas1 kernel: [26020]    98 26020     4151      476      11       0      680             0 apache2
Oct 22 02:06:33 ausy-nas1 kernel: [26021]    98 26021     4151      476      11       0      680             0 apache2
Oct 22 02:06:33 ausy-nas1 kernel: Out of memory: Kill process 2382 (apache2) score 2 or sacrifice child
Oct 22 02:06:33 ausy-nas1 kernel: Killed process 4655 (apache2) total-vm:16892kB, anon-rss:0kB, file-rss:2600kB
Oct 22 02:07:41 ausy-nas1 kernel: readynasd invoked oom-killer: gfp_mask=0x24200ca, order=0, oom_score_adj=-1000
Oct 22 02:08:01 ausy-nas1 kernel: readynasd cpuset=/ mems_allowed=0
Oct 22 02:08:01 ausy-nas1 kernel: CPU: 0 PID: 2390 Comm: readynasd Tainted: P           O    4.4.88.armada.1 #1
Oct 22 02:08:01 ausy-nas1 kernel: Hardware name: Marvell Armada 370/XP (Device Tree)
Oct 22 02:08:01 ausy-nas1 kernel: [<c0015f24>] (unwind_backtrace) from [<c00120dc>] (show_stack+0x10/0x18)
Oct 22 02:08:01 ausy-nas1 kernel: [<c00120dc>] (show_stack) from [<c0392c80>] (dump_stack+0x78/0x9c)
Oct 22 02:08:01 ausy-nas1 kernel: [<c0392c80>] (dump_stack) from [<c00d49a4>] (dump_header+0x48/0x1ac)
Oct 22 02:08:01 ausy-nas1 kernel: [<c00d49a4>] (dump_header) from [<c009d70c>] (oom_kill_process+0x1f8/0x478)
Oct 22 02:08:01 ausy-nas1 kernel: [<c009d70c>] (oom_kill_process) from [<c009dc68>] (out_of_memory+0x26c/0x358)
Oct 22 02:08:01 ausy-nas1 kernel: [<c009dc68>] (out_of_memory) from [<c00a1abc>] (__alloc_pages_nodemask+0x724/0x88c)
Oct 22 02:08:01 ausy-nas1 kernel: [<c00a1abc>] (__alloc_pages_nodemask) from [<c00cabc4>] (__read_swap_cache_async+0x108/0x1cc)
Oct 22 02:08:01 ausy-nas1 kernel: [<c00cabc4>] (__read_swap_cache_async) from [<c00cac9c>] (read_swap_cache_async+0x14/0x38)
Oct 22 02:08:01 ausy-nas1 kernel: [<c00cac9c>] (read_swap_cache_async) from [<c00cae00>] (swapin_readahead+0x140/0x198)
Oct 22 02:08:01 ausy-nas1 kernel: [<c00cae00>] (swapin_readahead) from [<c00bbfd0>] (handle_mm_fault+0x988/0xc88)
Oct 22 02:08:01 ausy-nas1 kernel: [<c00bbfd0>] (handle_mm_fault) from [<c0018b5c>] (do_page_fault+0x1c0/0x2b0)
Oct 22 02:08:01 ausy-nas1 kernel: [<c0018b5c>] (do_page_fault) from [<c00092d8>] (do_DataAbort+0x34/0xbc)
Oct 22 02:08:01 ausy-nas1 kernel: [<c00092d8>] (do_DataAbort) from [<c0012dfc>] (__dabt_usr+0x3c/0x40)
Oct 22 02:08:01 ausy-nas1 kernel: Exception stack(0xe069bfb0 to 0xe069bff8)
Oct 22 02:08:01 ausy-nas1 kernel: bfa0:                                     00000000 00000000 b4dbd084 000011e4
Oct 22 02:08:01 ausy-nas1 kernel: bfc0: 00163c28 01cd83d8 00000015 00200000 be944bc8 01bb3170 001e3308 001689f8
Oct 22 02:08:01 ausy-nas1 kernel: bfe0: 00000000 be9446c8 000c8e0c 000c8ad8 60000010 ffffffff
Oct 22 02:08:01 ausy-nas1 kernel: Mem-Info:
Oct 22 02:08:01 ausy-nas1 kernel: active_anon:11 inactive_anon:18 isolated_anon:0
                                   active_file:69229 inactive_file:376430 isolated_file:224
                                   unevictable:0 dirty:0 writeback:4 unstable:0
                                   slab_reclaimable:6397 slab_unreclaimable:5705
                                   mapped:7613 shmem:0 pagetables:306 bounce:0
                                   free:20626 free_pcp:185 free_cma:0
Oct 22 02:08:01 ausy-nas1 kernel: Normal free:82112kB min:41284kB low:51604kB high:61924kB active_anon:0kB inactive_anon:20kB active_file:236812kB inactive_file:236844kB unevictable:0kB isolated(anon):0kB isolated(file):640kB present:786432kB managed:754676kB mlocked:0kB dirty:0kB writeback:16kB mapped:1412kB shmem:0kB slab_reclaimable:25588kB slab_unreclaimable:22820kB kernel_stack:1432kB pagetables:1224kB unstable:0kB bounce:0kB free_pcp:436kB local_pcp:168kB free_cma:0kB writeback_tmp:0kB pages_scanned:2849532 all_unreclaimable? yes
-- Reboot --

mdgm-ntgr · ‎2017-10-23

Please don't tag the model as the RND2120. That's the ReadyNAS Duo, a 2-bay desktop device, and a very different product to your RN2120.

OOM errors suggest a memory leak of some kind.

You're looking for what it looked like right at the first OOM after the previous reboot to see where the problem started. kernel.log can go back further than systemd-journal.log. The journalctl command may go back further still.

Do you have quotas enabled on the volume? If so, you could try disabling those.

Which firmware did you update to 6.8.1 from (initrd.log will tell you)?

JackyWong · ‎2017-10-23

Sorry that for the wrong tag, since i am not a experience for it.

as Check on the Kernel log, it seems there are nothing too much can see. form my guessing, the NAS should be running backup and cause lack of memory.

the quotas did disabled already on the volume.

But the NAS is update from 6.5.1 to 6.8.1.

it seems not a good update right? anything i can do for fall back the firmware update and update it step by step?

thanks

mdgm-ntgr · ‎2017-10-24

You shouldn't downgrade to older firmware than 6.8.0 after upgrading to 6.8.1.

You could update to 6.9.0 (however you shouldn't downgrade from that firmware to older firmware).

JackyWong · ‎2017-10-24

will the problem solved if I upgrade to 6.9.0 ?

markwhit · ‎2017-11-27

I'm having the same issue on my ReadyNAS 2120. Using it just for backups, just has a few shares, no Apps installed, AV turned off, FTP and all other non-necessary services turned off, 1 NIC, IPV6 disabled, Firmware 6.9.0. It will work for a week or two and then becomes inaccessible via network. It worked fine without incident for over a year and now it keeps dropping connection. I agree and think that the issue is in the firmware update. Has there been any resolution to this?

StephenB · ‎2017-11-27

@markwhit wrote:

I'm having the same issue on my ReadyNAS 2120. Using it just for backups, just has a few shares, no Apps installed, AV turned off, FTP and all other non-necessary services turned off, 1 NIC, IPV6 disabled, Firmware 6.9.0. It will work for a week or two and then becomes inaccessible via network. It worked fine without incident for over a year and now it keeps dropping connection. I agree and think that the issue is in the firmware update. Has there been any resolution to this?

Your root cause might not be the same as the OP. You could try 6.9.1, as there are some network fixes in it. What backup protocol are you using?

A link to the manual install page is here: https://kb.netgear.com/000051531/ReadyNAS-OS-6-Software-Version-6-9-1 (I'm not sure if this has reached the Netgear update servers yet).

markwhit · ‎2017-11-27

Standard SMB Windows network shares, using NovaStor backup to copy files to the shares.

StephenB · ‎2017-11-27

@markwhit wrote:

Standard SMB Windows network shares, using NovaStor backup to copy files to the shares.

Thx. Try 6.9.1 then, and see if that cures it.