NAS Pro Pioneer Edition on OS 6.9.3 freezes/ network dropout

[firefly_242]

Thanks for this info. Much appreciated

[chsu83]

I've got the same Problem with my Pro 6 on 6.9.3 and had it several times. BUT actually I've got it less and lesser.. (don't know if I can write that in english like in german) :).

Don't know what the version was, when I started with v6 but in the beginning I had that problem something like once every 1-2 weeks. Now it mostly dropped to once 2-4 months. Actually I have the feeling thats it is often if a backup job or volume schedule runs.. but couldn't confirm that. In most time nas woked whole the whole day, and somewhen in early morning until 7 when I wake up it stopped.

no LACP

already 4GB since many years

[firefly_242]

Hi,

System has been pretty stable. As we went on holiday last week and nobody was at home I shutdown the storage.

When I started it again yesterday it froze 2 times after about 30 min uptime and a 3rd time after a few hours. After that I only got the READYNAS screen. I removed the newly added memory (2GB) and the system booted again. But it still froze a few hours later:(

It is currently resyncing.

Logs from 3GB and 1GB configuration are available if needed.

Regards,

Werner

[Alein]

After some research, I was able to catch some information, when NAS freezes.

Through serial port, I was able to monitor NAS even when it freezes.

As you may see btrfs-tran+ is consuming one core (thread) at 100%

So I google for it, and that is what you should be looking for

btrfs-transaction stuck and consuming 100% CPU

In general, it is a bug in btrfs, as I understand it.

what can freeze really fast NAS are:

-rsync backup (read write)

-iSCSI ( in general)

-NFS (v3 and v4) as datastore

-removing old snapshots

-creating snapshots in smart mode ( ~0:00 is the time when new Snaps. are created and deleted we have a lot of concurrent I/O)

-compressed shares, especially iSCSI 

-a huge amount of files,>1M

We should have possibilities to use different FS, not only BTRFS.

[Sandshark]

Both of my Pro BE's (same generation as your Pro Pioneer) have upgraded processors, so that could be why I never saw the problem.  Unfortunately, a CPU upgrade is a bit trickier than a simple RAM upgrade.  Core2 Duo E7500's can be had on eBay for around $3 these days, and that'll really up the throughput over the original Pentium E2160.

That upgrade does require the latest BIOS.  Check the forum to see if you have it and how to upgrade.  Or, stick to the Core2 E6600, which doesn't need it but is still a lot more CPU than stock.

[firefly_242]

Hi,

Experienced anotehr hang of my storage.
This time I was still able to login with putty via the second lan port.
top command shows the system is almost idle:
top - 08:33:56 up 3 days, 21:15,  2 users,  load average: 4.05, 4.03, 3.42
Tasks: 210 total,   1 running, 209 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   1011636 total,   959504 used,    52132 free,     2184 buffers
KiB Swap:  1308156 total,    81356 used,  1226800 free.   508880 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 4396 admin     38  18 1333688  34644      0 S   0.7  3.4 155:29.06 utserver
12128 root      20   0   28772   2944   2340 R   0.3  0.3   0:00.05 top
    1 root      20   0  136720   4268   2968 S   0.0  0.4   0:05.78 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.37 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:10.25 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      20   0       0      0      0 S   0.0  0.0   0:42.66 rcu_sched
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      rt   0       0      0      0 S   0.0  0.0   0:00.56 migration/0
   10 root      rt   0       0      0      0 S   0.0  0.0   0:00.70 watchdog/0

seems I am facing another issue that causes the system to hang.
Regards,
Werner

[firefly_242]

Hi, Still faceing system freezes fro mtime to time. Am on the latest FW (6.9.4).

Starting to suspect a HW issue for the errors I see in Kernel.log. But erros are not consistant. Last issue was on Oct 13 and I see this in the Kernel.log file before I rebooted:

"Oct 13 07:22:27 nas-storage kernel: alloc_fd: slot 3 not NULL!
Oct 13 07:22:27 nas-storage kernel: alloc_fd: slot 4 not NULL!
Oct 13 07:22:27 nas-storage kernel: general protection fault: 0000 [#1] SMP "

followed by some CPU data and:

"Oct 13 07:22:27 nas-storage kernel: Fixing recursive fault but reboot is needed!". 

However, other freezes show in Kernel.log:

"Sep 29 06:22:48 nas-storage kernel: eth1: hw csum failure"

and the same for eth0 on another occassion:

"Sep 19 06:12:44 nas-storage kernel: eth0: hw csum failure"

Confused with these contradicting mesages. Any  idea how I can find out what is causing this?

Thanks.

[StephenB]

I suggest running the memory test from the boot menu.

[firefly_242]

Ran 5 passes of memory test. all passed

[StephenB]

Perhaps try downgrading to 6.9.3 and see if that is more stable.

[firefly_242]

Hi,

issue is happeing since I upgrade to 6.9.2 from OS4.2. Also happens with 6.9.3 and all the released beta's.

Also tried reverting back to the latest stable OS4.2 (RAIDiator-x86-4.2.31), but issue also happens there. So, can only be a HW issue.

However, the alarms are not consistant. Maybe I just have to accept that the system is reaching it's end of life.

[StephenB]

---

@firefly_242 wrote:

Maybe I just have to accept that the system is reaching it's end of life.

---

Very possible, especially since it is happening on both OS 4.2 and OS 6.