Re: ReadyNAS Pro Business - Freeze

mfinney · ‎2018-07-04

Hi,

My ReadyNAS Pro business running OS6.9.3 froze last night for the very first time - totally inaccessible, unable to 'ping'. When I checked my cabinet, the NAS was still running - fans spinning and power button lit, but when I pressed it (multiple times) there was no response. I eventually held the power button in and the NAS switched off. Powered back up again and everything is now working again.

Been running OS6 for months without issue. I've checked the logs through the admin page but it shows everything as normal. I've pressed the 'downloaded logs' button and looked at some of the logs in the zip file, but I'm way out of my depth. Can anyone suggest how I might go about isolating the issue causing the freeze, or provide some assistance - would really appreciate some help if someone knowledgeable has time.

Kind regards...

StephenB · ‎2018-07-04

What apps and services are you running?

You can look in system.log and kernel.log for crashes, and post any that you see here.

You should also look at the disk health (SMART statistics). They are in several places in the logs, including smart_history.log and disk_info.log.

mfinney · ‎2018-07-05

Hi, thanks for the fast reply and your direction.

To answer your questions:

- I'm running iStat NT and Plex apps

- I'm running ReadyCloud, ReadyNAS Replicate, SMB, Rsync, HTTP, HTTPS, File Search, uPnP

- Disk files show no errors at the moment

- system.log showed 'connmand[3058]: eth0... ignoring slave, index 2...' start being logged approx 1 hr before logging stopped - there's many of these

- kernel.log showed GPF immediately before logging stopped:

-- Logs begin at Sun 2018-07-01 01:25:01 GMT, end at Wed 2018-07-04 11:38:05 GMT. --
Jul 04 00:23:10 Dolphin kernel: general protection fault: 0000 [#1] SMP
Jul 04 00:23:10 Dolphin kernel: Modules linked in: nv6lcd(O) vpd(PO)
Jul 04 00:23:10 Dolphin kernel: CPU: 1 PID: 2876 Comm: btrfs-transacti Tainted: P O 4.4.116.x86_64.1 #1
Jul 04 00:23:10 Dolphin kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./To be filled by O.E.M., BIOS 080014 07/26/2010
Jul 04 00:23:10 Dolphin kernel: task: ffff88009f1c35c0 ti: ffff88009ed20000 task.ti: ffff88009ed20000
Jul 04 00:23:10 Dolphin kernel: RIP: 0010:[<ffffffff8833fd60>] [<ffffffff8833fd60>] process_one_buffer+0x20/0x110
Jul 04 00:23:10 Dolphin kernel: RSP: 0018:ffff88009ed23c80 EFLAGS: 00010206
Jul 04 00:23:10 Dolphin kernel: RAX: 0a0d0a0d323a6574 RBX: ffff8800a8d6f068 RCX: 0000000000004cf7
Jul 04 00:23:10 Dolphin kernel: RDX: ffff88009ed23d80 RSI: ffff8800a8d6f068 RDI: ffff88014944c800
Jul 04 00:23:10 Dolphin kernel: RBP: ffff88009ed23ca8 R08: 0000160000000000 R09: 0000000000000000
Jul 04 00:23:10 Dolphin kernel: R10: ffff8800a8d6f068 R11: 0000000000000000 R12: ffff88009ed23d24
Jul 04 00:23:10 Dolphin kernel: R13: ffff88014944c800 R14: ffff88009ed23d80 R15: 0000000000004cf7
Jul 04 00:23:10 Dolphin kernel: FS: 0000000000000000(0000) GS:ffff88014fc80000(0000) knlGS:0000000000000000
Jul 04 00:23:10 Dolphin kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jul 04 00:23:10 Dolphin kernel: CR2: 00007f49b2226000 CR3: 000000009e8ac000 CR4: 0000000000040670
Jul 04 00:23:10 Dolphin kernel: Stack:
Jul 04 00:23:10 Dolphin kernel: ffff8801199f8460 ffff88009ed23d24 ffff88014944c800 ffff880000000000
Jul 04 00:23:10 Dolphin kernel: ffff88009ed23d80 ffff88009ed23d08 ffffffff8833ff21 ffff8800ab6015a0
Jul 04 00:23:10 Dolphin kernel: fffffffffffffffa 0000160000000000 ffff8800a8de6000 000000014944c800
Jul 04 00:23:10 Dolphin kernel: Call Trace:
Jul 04 00:23:10 Dolphin kernel: [<ffffffff8833ff21>] walk_up_log_tree+0xd1/0x230
Jul 04 00:23:10 Dolphin kernel: [<ffffffff883404d2>] walk_log_tree+0x92/0x1a0
Jul 04 00:23:10 Dolphin kernel: [<ffffffff8834061c>] free_log_tree+0x3c/0x140
Jul 04 00:23:10 Dolphin kernel: [<ffffffff8833fd40>] ? wait_for_writer+0xb0/0xb0
Jul 04 00:23:10 Dolphin kernel: [<ffffffff88348499>] btrfs_free_log_root_tree+0x19/0x30
Jul 04 00:23:10 Dolphin kernel: [<ffffffff88301b7a>] btrfs_commit_transaction.part.25+0x45a/0xa80
Jul 04 00:23:10 Dolphin kernel: [<ffffffff882ec8e9>] ? btrfs_run_delayed_refs+0x209/0x2e0
Jul 04 00:23:10 Dolphin kernel: [<ffffffff883021d5>] btrfs_commit_transaction+0x35/0x60
Jul 04 00:23:10 Dolphin kernel: [<ffffffff882fcd36>] transaction_kthread+0x216/0x230
Jul 04 00:23:10 Dolphin kernel: [<ffffffff882fcb20>] ? btrfs_cleanup_transaction+0x4a0/0x4a0
Jul 04 00:23:10 Dolphin kernel: [<ffffffff88089356>] kthread+0xe6/0x100
Jul 04 00:23:10 Dolphin kernel: [<ffffffff88089270>] ? kthread_create_on_node+0x170/0x170
Jul 04 00:23:10 Dolphin kernel: [<ffffffff88a731c5>] ret_from_fork+0x55/0x80
Jul 04 00:23:10 Dolphin kernel: [<ffffffff88089270>] ? kthread_create_on_node+0x170/0x170
Jul 04 00:23:10 Dolphin kernel: Code: 5c 41 5d 5d c3 66 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 fd 53 48 8b 87 f0 01 00 00 48 89 f3 49 89 d6 49 89 cf <48> 8b 80 d0 02 00 00 f6 80 bc 00 00 00 04 75 6c 41 8b 76 0c 85
Jul 04 00:23:10 Dolphin kernel: RIP [<ffffffff8833fd60>] process_one_buffer+0x20/0x110
Jul 04 00:23:10 Dolphin kernel: RSP <ffff88009ed23c80>
Jul 04 00:23:10 Dolphin kernel: ---[ end trace 667a05548e8b3362 ]---

StephenB · ‎2018-07-05

Certainly there is evidence here of file system corruption, even though the disks are healthy.

It'd be good to make a full backup of your data. Then after that, perhaps try running a scrub using the volume settings wheel (on the volume page). A more conservative approach would be to do a factory reset, set the NAS up again, and restore the data from the backup.

mfinney · ‎2018-07-06

Understood.

Actually I did a factory default 2 weeks ago when I upgraded my CPU to an E7600 to help with Plex transcoding. A few days after that, I then upgraded to 4GB of Crucial PC2-6400 DDR2-800 memory - I'd been running OS6 on the 1GB factory default which came with the ReadyNAS when it was running OS4.

Could the root cause of this problem be related? If so, is it more likely to be one of the memory modules than the CPU?

StephenB · ‎2018-07-06

@mfinney wrote:

If so, is it more likely to be one of the memory modules than the CPU?

My guess is that the memory is more likely to be a problem than the CPU. Perhaps run the boot menu memory test for a while - if only to rule it out.

mfinney · ‎2018-07-06

Thanks Stephen - I'd forgotten about that boot menu test :-). The memory was my first suspicion too. Gut feel says I should have 'blown out' the previously unused memory slot before inserting the module - there'll have been a few years of dust in there.

Will have to try it next week though due to lack of opportunity. In the mean time I'll run the disk scrub. Then, before starting the mem test I'll remove the modules clean contacts and reseat them, test and report back.

Trouble with intermittent problems is it takes time to narrow things down, and at the end of the day you're never quite sure. Sound like if the test passes, I should factory default and start again to be sure?

Fortunately this is my backup NAS so nothing is being affected outright right now - not great though!

Really appreciate your help! If anything else comes to mind in the meantime I'd love to hear. Kind regards...

mfinney · ‎2018-07-16

Took some time to do the memory test - turns out my upgraded ReadyNAS Pro Business running 6.9.3 didn't have the necessary memory test files in the flash boot memory, but fortunately MDGM fixed this for me...

So, after removing both modules and cleaning the memory slots and modules with contact cleaner and then reseating them, the memory test past - successfully completed 40+ cycles over a 50+ hours.

Hopefully that will be the end of my problems.

Appreciate your help!