SMB quits working

Hi!

Since yesterday I have increasing problems with my ReadyNAS 516. It runs 6 3TB harddisks, in X-RAID 6. It operates under OS 6.6.1. 

It has worked for quite a while since the update to 6.6.1 very reliable. But suddenly, I have problems with the SMB service, which suddenly seems to quit and cannot be reactivated via the GUI (if I try to deactivate and reactivate I just get an error message that the status of the service could not be changed).

Once I could no longer use the NAS with SMB and rebooted the NAS I usually got stuck diring reboot with 18% or 22% forcing me to shut down the NAS hard. When rebooting, it then seems to restart again.

Research in the log files (via ZIP download) indicates that the root file system might be full, while standard data volumes are having more than enough free space:

From btrfs.log:

Label: '7c6e460a:Main' uuid: 056dd4f9-8a7c-4dda-b584-02e6792b9c0a
Total devices 1 FS bytes used 6.34TiB
devid 1 size 10.90TiB used 6.38TiB path /dev/md127

=== filesystem /Main ===
Data, single: total=6.36TiB, used=6.33TiB
System, DUP: total=32.00MiB, used=736.00KiB
Metadata, DUP: total=9.50GiB, used=8.42GiB
GlobalReserve, single: total=512.00MiB, used=1.69MiB

From volume.log:

=== df -h ===
Filesystem Size Used Avail Use% Mounted on
udev 10M 4.0K 10M 1% /dev
/dev/md0 4.0G 2.2G 1.4G 62% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 16M 1.9G 1% /run
tmpfs 960M 1008K 959M 1% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/md127 11T 6.4T 4.6T 59% /Main
/dev/md127 11T 6.4T 4.6T 59% /home
/dev/md127 11T 6.4T 4.6T 59% /apps
=== df -i ===
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 490683 535 490148 1% /dev
/dev/md0 0 0 0 - /
tmpfs 491301 1 491300 1% /dev/shm
tmpfs 491301 766 490535 1% /run
tmpfs 491301 23 491278 1% /run/lock
tmpfs 491301 9 491292 1% /sys/fs/cgroup
/dev/md127 0 0 0 - /Main
/dev/md127 0 0 0 - /home
/dev/md127 0 0 0 - /apps

From smbd.log:

[2017/02/12 20:58:38.872148, 0] ../source3/lib/util.c:902(log_stack_trace)
#1 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f838db74a70]
#3 /usr/sbin/smbd(+0xb7c2) [0x5591a14527c2]
BACKTRACE: 12 stack frames:
#2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x1c) [0x7f83900172dc]
#4 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x9b13) [0x7f838c5a4b13]
#3 /usr/sbin/smbd(+0xb7c2) [0x5591a14527c2]
#0 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f838db745da]
#4 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x9b13) [0x7f838c5a4b13]
#6 /usr/lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f838c59f24d]
#7 /usr/lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f838c59f3eb]
#1 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f838db74a70]
#8 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x7ff7) [0x7f838c5a2ff7]
#2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x1c) [0x7f83900172dc]
#6 /usr/lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f838c59f24d]
#7 /usr/lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f838c59f3eb]
#3 /usr/sbin/smbd(+0xb7c2) [0x5591a14527c2]
#10 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f839028bb45]
#8 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x7ff7) [0x7f838c5a2ff7]
#4 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x9b13) [0x7f838c5a4b13]
#11 /usr/sbin/smbd(+0x7d7e) [0x5591a144ed7e]
#9 /usr/sbin/smbd(main+0x17da) [0x5591a144ebfa]
#5 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x8057) [0x7f838c5a3057]
#10 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f839028bb45]
#11 /usr/sbin/smbd(+0x7d7e) [0x5591a144ed7e]
[2017/02/12 20:58:40.526904, 0] ../source3/lib/dumpcore.c:303(dump_core)
dumping core in /var/cores
#6 /usr/lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f838c59f24d]
dumping core in /var/cores
#7 /usr/lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f838c59f3eb]
#8 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x7ff7) [0x7f838c5a2ff7]
#9 /usr/sbin/smbd(main+0x17da) [0x5591a144ebfa]
#10 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f839028bb45]
[2017/02/12 20:58:41.108191, 0] ../source3/lib/dumpcore.c:303(dump_core)
dumping core in /var/cores
[2017/02/12 21:03:54.018315, 0] ../source3/lib/util.c:478(reinit_after_fork)
[2017/02/12 21:03:54.073892, 0] ../source3/lib/util.c:478(reinit_after_fork)
[2017/02/12 21:03:54.163829, 0] ../source3/lib/util.c:478(reinit_after_fork)
messaging_reinit() failed: NT_STATUS_DISK_FULL
[2017/02/12 21:03:54.252399, 0] ../source3/smbd/server.c:758(smbd_accept_connection)
messaging_reinit() failed: NT_STATUS_DISK_FULL
[2017/02/12 21:03:54.385396, 0] ../source3/smbd/server.c:758(smbd_accept_connection)
reinit_after_fork() failed
[2017/02/12 21:03:54.499331, 0] ../source3/smbd/server.c:758(smbd_accept_connection)
reinit_after_fork() failed
[2017/02/12 21:03:54.499364, 0] ../source3/lib/util.c:791(smb_panic_s3)
PANIC (pid 15325): reinit_after_fork() failed
reinit_after_fork() failed
[2017/02/12 21:03:54.885632, 0] ../source3/lib/util.c:791(smb_panic_s3)
[2017/02/12 21:03:54.760125, 0] ../source3/lib/util.c:902(log_stack_trace)
PANIC (pid 15324): reinit_after_fork() failed
PANIC (pid 15323): reinit_after_fork() failed
BACKTRACE: 12 stack frames:
#0 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f838db745da]
[2017/02/12 21:03:55.049690, 0] ../source3/lib/util.c:902(log_stack_trace)
[2017/02/12 21:03:55.138056, 0] ../source3/lib/util.c:902(log_stack_trace)
#1 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f838db74a70]
BACKTRACE: 12 stack frames:
#0 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f838db745da]
#2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x1c) [0x7f83900172dc]
#3 /usr/sbin/smbd(+0xb7c2) [0x5591a14527c2]
#1 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f838db74a70]
#2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x1c) [0x7f83900172dc]
#4 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x9b13) [0x7f838c5a4b13]
#1 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f838db74a70]
#3 /usr/sbin/smbd(+0xb7c2) [0x5591a14527c2]
#5 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x8057) [0x7f838c5a3057]
#4 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x9b13) [0x7f838c5a4b13]
#2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x1c) [0x7f83900172dc]
#6 /usr/lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f838c59f24d]
#3 /usr/sbin/smbd(+0xb7c2) [0x5591a14527c2]
#5 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x8057) [0x7f838c5a3057]
#7 /usr/lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f838c59f3eb]
#4 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x9b13) [0x7f838c5a4b13]
#8 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x7ff7) [0x7f838c5a2ff7]
#9 /usr/sbin/smbd(main+0x17da) [0x5591a144ebfa]
#5 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x8057) [0x7f838c5a3057]
#7 /usr/lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f838c59f3eb]
#10 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f839028bb45]
#6 /usr/lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f838c59f24d]
#8 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x7ff7) [0x7f838c5a2ff7]
#11 /usr/sbin/smbd(+0x7d7e) [0x5591a144ed7e]
#7 /usr/lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f838c59f3eb]
#8 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x7ff7) [0x7f838c5a2ff7]
#9 /usr/sbin/smbd(main+0x17da) [0x5591a144ebfa]
[2017/02/12 21:03:56.859592, 0] ../source3/lib/dumpcore.c:303(dump_core)
#10 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f839028bb45]
#9 /usr/sbin/smbd(main+0x17da) [0x5591a144ebfa]
dumping core in /var/cores
#10 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f839028bb45]
#11 /usr/sbin/smbd(+0x7d7e) [0x5591a144ed7e]
[2017/02/12 21:03:57.263164, 0] ../source3/lib/dumpcore.c:303(dump_core)
[2017/02/12 21:03:57.206050, 0] ../source3/lib/dumpcore.c:303(dump_core)
dumping core in /var/cores
dumping core in /var/cores
[2017/02/12 21:08:08.620599, 0] ../source3/lib/util.c:478(reinit_after_fork)
messaging_reinit() failed: NT_STATUS_DISK_FULL
[2017/02/12 21:08:08.676184, 0] ../source3/lib/util.c:478(reinit_after_fork)
[2017/02/12 21:08:08.785429, 0] ../source3/smbd/server.c:758(smbd_accept_connection)
[2017/02/12 21:08:08.785388, 0] ../source3/lib/util.c:478(reinit_after_fork)
reinit_after_fork() failed
messaging_reinit() failed: NT_STATUS_DISK_FULL
messaging_reinit() failed: NT_STATUS_DISK_FULL
[2017/02/12 21:08:09.073466, 0] ../source3/lib/util.c:791(smb_panic_s3)
[2017/02/12 21:08:09.073482, 0] ../source3/smbd/server.c:758(smbd_accept_connection)
PANIC (pid 15392): reinit_after_fork() failed
[2017/02/12 21:08:09.165160, 0] ../source3/smbd/server.c:758(smbd_accept_connection)
reinit_after_fork() failed
[2017/02/12 21:08:09.487176, 0] ../source3/lib/util.c:791(smb_panic_s3)
reinit_after_fork() failed
[2017/02/12 21:08:09.369346, 0] ../source3/lib/util.c:902(log_stack_trace)
PANIC (pid 15394): reinit_after_fork() failed
[2017/02/12 21:08:09.645442, 0] ../source3/lib/util.c:791(smb_panic_s3)
BACKTRACE: 12 stack frames:
PANIC (pid 15393): reinit_after_fork() failed
[2017/02/12 21:08:09.767188, 0] ../source3/lib/util.c:902(log_stack_trace)
#0 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f838db745da]
#1 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f838db74a70]
BACKTRACE: 12 stack frames:
BACKTRACE: 12 stack frames:
#2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x1c) [0x7f83900172dc]
#0 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f838db745da]
#3 /usr/sbin/smbd(+0xb7c2) [0x5591a14527c2]
#0 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f838db745da]
#1 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f838db74a70]
#4 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x9b13) [0x7f838c5a4b13]
#1 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f838db74a70]
#2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x1c) [0x7f83900172dc]
#3 /usr/sbin/smbd(+0xb7c2) [0x5591a14527c2]
#5 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x8057) [0x7f838c5a3057]
#2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x1c) [0x7f83900172dc]
#6 /usr/lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f838c59f24d]
#7 /usr/lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f838c59f3eb]
#4 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x9b13) [0x7f838c5a4b13]
#8 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x7ff7) [0x7f838c5a2ff7]
#4 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x9b13) [0x7f838c5a4b13]
#5 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x8057) [0x7f838c5a3057]
#9 /usr/sbin/smbd(main+0x17da) [0x5591a144ebfa]
#6 /usr/lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f838c59f24d]
#7 /usr/lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f838c59f3eb]
#5 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x8057) [0x7f838c5a3057]
#8 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x7ff7) [0x7f838c5a2ff7]
#10 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f839028bb45]
#6 /usr/lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f838c59f24d]
#11 /usr/sbin/smbd(+0x7d7e) [0x5591a144ed7e]
#9 /usr/sbin/smbd(main+0x17da) [0x5591a144ebfa]
#7 /usr/lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f838c59f3eb]
#10 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f839028bb45]
[2017/02/12 21:08:11.631549, 0] ../source3/lib/dumpcore.c:303(dump_core)
#8 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x7ff7) [0x7f838c5a2ff7]
#11 /usr/sbin/smbd(+0x7d7e) [0x5591a144ed7e]
#9 /usr/sbin/smbd(main+0x17da) [0x5591a144ebfa]
dumping core in /var/cores
#10 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f839028bb45]
dumping core in /var/cores
#11 /usr/sbin/smbd(+0x7d7e) [0x5591a144ed7e]
[2017/02/12 21:08:12.250895, 0] ../source3/lib/dumpcore.c:303(dump_core)
[2017/02/12 21:08:19.031334, 0] ../source3/lib/util.c:478(reinit_after_fork)
messaging_reinit() failed: NT_STATUS_DISK_FULL
[2017/02/12 21:10:16.686429, 0] ../source3/lib/util.c:478(reinit_after_fork)
messaging_reinit() failed: NT_STATUS_DISK_FULL
[2017/02/12 21:10:16.760177, 0] ../source3/lib/util.c:478(reinit_after_fork)
[2017/02/12 21:10:16.827875, 0] ../source3/lib/util.c:478(reinit_after_fork)
[2017/02/12 21:10:16.827917, 0] ../source3/smbd/server.c:758(smbd_accept_connection)
messaging_reinit() failed: NT_STATUS_DISK_FULL
messaging_reinit() failed: NT_STATUS_DISK_FULL
reinit_after_fork() failed
[2017/02/12 21:10:17.026902, 0] ../source3/smbd/server.c:758(smbd_accept_connection)
[2017/02/12 21:10:17.026893, 0] ../source3/smbd/server.c:758(smbd_accept_connection)
reinit_after_fork() failed
[2017/02/12 21:10:17.083316, 0] ../source3/lib/util.c:791(smb_panic_s3)
reinit_after_fork() failed
[2017/02/12 21:10:17.249940, 0] ../source3/lib/util.c:791(smb_panic_s3)
PANIC (pid 16239): reinit_after_fork() failed
PANIC (pid 16241): reinit_after_fork() failed
PANIC (pid 16240): reinit_after_fork() failed
[2017/02/12 21:10:17.380678, 0] ../source3/lib/util.c:902(log_stack_trace)
[2017/02/12 21:10:17.443715, 0] ../source3/lib/util.c:902(log_stack_trace)
[2017/02/12 21:10:17.551820, 0] ../source3/lib/util.c:902(log_stack_trace)
BACKTRACE: 12 stack frames:
#0 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f838db745da]
BACKTRACE: 12 stack frames:
#0 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f838db745da]
#0 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f838db745da]
#2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x1c) [0x7f83900172dc]
#1 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f838db74a70]
#2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x1c) [0x7f83900172dc]
#1 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f838db74a70]
#4 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x9b13) [0x7f838c5a4b13]
#3 /usr/sbin/smbd(+0xb7c2) [0x5591a14527c2]
#2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x1c) [0x7f83900172dc]
#4 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x9b13) [0x7f838c5a4b13]
#5 /usr/lib/x86_64-linux-gnu/libtevent.so.0(+0x8057) [0x7f838c5a3057]

And very strange from system.log:

...

Feb 12 14:22:29 RasslerNAS mysqld[10070]:
Feb 12 14:22:29 RasslerNAS mysqld[10070]: 170212 14:22:29 [Note] /usr/sbin/mysqld: Shutdown complete
Feb 12 14:22:29 RasslerNAS mysqld[10070]:
-- Reboot --
Feb 12 21:19:12 RasslerNAS disk_event_handler[3181]: disk event handler called for add
Feb 12 21:19:12 RasslerNAS disk_event_handler[3182]: disk event handler called for add
Feb 12 21:19:12 RasslerNAS disk_event_handler[3184]: disk event handler called for add
Feb 12 21:19:12 RasslerNAS disk_event_handler[3185]: disk event handler called for add

...

The NAS was running the whole day. First reboot was this afternoon. But is seems that between the reboot this afternoon and the reboot this evening, logging into system.log was not working...

I have read, that the VirusScanner could be a reason for filling up the root file system. So I have not disabled the VirusScanner.  I will investigate further. 

Any idea, what could cause this problem.

Thanks and regards,

  Michael