× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: Firmware 6.9.4. makes RN104 hang up

Bruce2k18
Tutor

Firmware 6.9.4. makes RN104 hang up

Had to downgrade now two time to 6.93 because of hanging system while accessing samba share.

 

1 time with Android Filetransfer

1 time with Windows 10 Filetransfer

 

under 6.9.3 i can do it for hours. With 6.9.4 it starts hanging within minutes. No Reboot possible (Web, Button and ssh not working)- I have to Hard Reset

 

Error Feedback in Android: Transport1 timedout waiting for response to smbComCreatDirectory[command=SMB_COM_CREAT_DIRECTORY, received=false,errorCode=0,flags0x0018,flags2=0C803, signSeq=0,tid20837mpid8553mid=3616,mid=4362,wordCount=0,byteCount=17,directoryName="Foldername"

 

Sorry don't go again on 6.9.4 - it was costing me now 90+ Minutes of work. 6.9.3 is running stable for weeks.

 

Fix it!

Model: RN10400|ReadyNAS 100 Series 4- Bay (Diskless)
Message 1 of 28
StephenB
Guru

Re: Firmware 6.9.4. makes RN104 hang up

I've seen several Samba hangs with 6.9.4 in my RN526x as well.  I'm still trying to figure it out.  Strict sync was set, but changing that didn't help.  Resetting permissions didn't help either.  

 

However my issue seems to be different.  The web UI and ssh both still work after the hang.

Message 2 of 28
evan2
NETGEAR Expert

Re: Firmware 6.9.4. makes RN104 hang up

@Bruce2k18

Cloud do you please download logs and send to us?

How do I send all logs to ReadyNAS Community moderators?
https://kb.netgear.com/21543/How-do-I-send-all-logs-to-ReadyNAS-Community-moderators

 

 

Message 3 of 28
Ki_Adi_Mundi
NETGEAR Expert

Re: Firmware 6.9.4. makes RN104 hang up

I saw a bug fix for samba 4.7.10 :

https://bugzilla.samba.org/show_bug.cgi?id=13537

Not sure if it is releated,   can you  try to open ssh access and do 'echo "use sendfile = 0" > /etc/frontview/samba/smb.conf.overrides' , then 'systemctl restart smb' or reboot your box to see if that helps ?

Message 4 of 28
StephenB
Guru

Re: Firmware 6.9.4. makes RN104 hang up


@Ki_Adi_Mundi wrote:

I saw a bug fix for samba 4.7.10 :

https://bugzilla.samba.org/show_bug.cgi?id=13537

Not sure if it is releated,   can you  try to open ssh access and do 'echo "use sendfile = 0" > /etc/frontview/samba/smb.conf.overrides' , then 'systemctl restart smb' or reboot your box to see if that helps ?


I tried this and it didn't help.  I was able to reproduce the hang.  I rebooted the NAS and see this in kernel.log:

 

Sep 30 05:37:46 NAS kernel: INFO: task smbd:11847 blocked for more than 200 seconds.
Sep 30 05:37:46 NAS kernel:       Tainted: P           O    4.4.140.x86_64.1 #1
Sep 30 05:37:46 NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 30 05:37:46 NAS kernel: smbd            D ffff880168ed3d30     0 11847      1 0x00000004
Sep 30 05:37:46 NAS kernel:  ffff880168ed3d30 ffff88017ab9d240 ffff880134dcb8a0 ffff88017ab9d240
Sep 30 05:37:46 NAS kernel:  ffff880134dcb8a4 00000000ffffffff ffff880134dcb8a8 ffff880168ed3d40
Sep 30 05:37:46 NAS kernel:  ffffffff888e2ab0 ffff880168ed3d50 ffffffff888e2c93 ffff880168ed3db0
Sep 30 05:37:46 NAS kernel: Call Trace:
Sep 30 05:37:46 NAS kernel:  [<ffffffff888e2ab0>] schedule+0x69/0x7f
Sep 30 05:37:46 NAS kernel:  [<ffffffff888e2c93>] schedule_preempt_disabled+0x9/0xb
Sep 30 05:37:46 NAS kernel:  [<ffffffff888e3b75>] __mutex_lock_slowpath+0x98/0xf1
Sep 30 05:37:46 NAS kernel:  [<ffffffff888e3be0>] mutex_lock+0x12/0x22
Sep 30 05:37:46 NAS kernel:  [<ffffffff888e3be0>] ? mutex_lock+0x12/0x22
Sep 30 05:37:46 NAS kernel:  [<ffffffff882aeb9d>] btrfs_file_write_iter+0x6b/0x3c6
Sep 30 05:37:46 NAS kernel:  [<ffffffff880add97>] ? do_futex+0xdb/0x86e
Sep 30 05:37:46 NAS kernel:  [<ffffffff8811c9bd>] __vfs_write+0x9a/0xc3
Sep 30 05:37:46 NAS kernel:  [<ffffffff8811d0ce>] vfs_write+0xb6/0x160
Sep 30 05:37:46 NAS kernel:  [<ffffffff8811dc09>] SyS_pwrite64+0x5c/0x7b
Sep 30 05:37:46 NAS kernel:  [<ffffffff888e540a>] entry_SYSCALL_64_fastpath+0x1e/0x8e

 

Message 5 of 28
Bruce2k18
Tutor

Re: Firmware 6.9.4. makes RN104 hang up

Dear Community,

 

Hey guys,

 

thank you for the efford. I added the samba fix and I am waiting, what is happening. As soon as I get a hang up i will send the logs. I deleted them because it was 3000 pages of logs, which i considered is not necessary (While my 90 Minutes work) - So keep on - But I am lucky that the error could be reproduced.

 

cheers Bruce

 

 

Message 6 of 28
Bruce2k18
Tutor

Re: Firmware 6.9.4. makes RN104 hang up

Ok so first of all - Yes Samba Server (at my place with + Samba Plus App ) is hanging up.

 

:~# systemctl status smb.service
 smb.service - Samba SMB Daemon
   Loaded: loaded (/lib/systemd/system/smb.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-10-04 20:00:47 CEST; 16min ago
  Process: 4150 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
 Main PID: 19548 (smbd)
   Status: "smbd: ready to serve connections..."
   CGroup: /system.slice/smb.service
           ├─19044 /usr/sbin/smbd --foreground --no-process-group
           ├─19548 /usr/sbin/smbd --foreground --no-process-group
           ├─19550 /usr/sbin/smbd --foreground --no-process-group
           ├─19551 /usr/sbin/smbd --foreground --no-process-group
           ├─19555 /usr/sbin/smbd --foreground --no-process-group
           ├─19615 /usr/sbin/smbd --foreground --no-process-group
           └─19991 /usr/sbin/smbd --foreground --no-process-group
 
Oct 04 20:00:46 nas-bruce systemd[1]: Starting Samba SMB Daemon...
Oct 04 20:00:47 nas-bruce systemd[1]: Started Samba SMB Daemon.
Oct 04 20:00:47 nas-bruce smbd[19548]: [2018/10/04 20:00:47.519432,  0] ../lib/util/become_daemon.c:124(daemon_ready)
Oct 04 20:00:47 nas-bruce smbd[19548]:   STATUS=daemon 'smbd' finished starting up and ready to serve connections
Oct 04 20:15:02 nas-bruce smbd[19615]: [2018/10/04 20:15:02.937841,  0] ../source3/modules/vfs_snapper.c:578(snapper_list_confs_unpack)
Oct 04 20:15:02 nas-bruce smbd[19615]:   list_confs error response: org.freedesktop.DBus.Error.NoReply
Oct 04 20:15:02 nas-bruce smbd[19615]: [2018/10/04 20:15:02.948506,  0] ../source3/modules/vfs_default.c:1375(vfswrap_fsctl)
Oct 04 20:15:02 nas-bruce smbd[19615]:   FSCTL_GET_SHADOW_COPY_DATA: connectpath /data/Frackyload, failed - NT_STATUS_INVALID_PARAMETER.

Login to RN104 via ssh 

:~# systemctl restart smb.service

 

 worked to do it without Hardreset 🙂

 

:~# cat /etc/frontview/samba/smb.conf.overrides
 
oplocks = no
server min protocol = SMB2_02
use sendfile = 0

 Added the workarround a view days ago. And it still hangs. Reading out the logs now.

Message 7 of 28
Bruce2k18
Tutor

Re: Firmware 6.9.4. makes RN104 hang up

Btw: Frackyload User is using

Cyrillic

Language Documents 

 

The rest EN (Egnlish) or DE (German)

Message 8 of 28
Bruce2k18
Tutor

Re: Firmware 6.9.4. makes RN104 hang up

I cannot force this problem under 6.9.3 - So it is a 6.9.4 related bug on samba. 

I went back to 6.9.3 looking forward you fix it in 6.9.5 

 https://www.youtube.com/watch?v=1Isjgc0oX0s

Message 9 of 28
Ki_Adi_Mundi
NETGEAR Expert

Re: Firmware 6.9.4. makes RN104 hang up

I checked the logs,  looks like there may be 2 different cases here:

1.  Out of memory happened on Sep 29

Sep 29 09:32:59 nas-bruce kernel: ksoftirqd/0: page allocation failure: order:2, mode:0x2084020
Sep 29 09:32:59 nas-bruce kernel: CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: P           O    4.4.116.armada.1 #1
Sep 29 09:32:59 nas-bruce kernel: Hardware name: Marvell Armada 370/XP (Device Tree)
Sep 29 09:32:59 nas-bruce kernel: [<c0015f44>] (unwind_backtrace) from [<c00120fc>] (show_stack+0x10/0x18)
Sep 29 09:32:59 nas-bruce kernel: [<c00120fc>] (show_stack) from [<c03a6080>] (dump_stack+0x78/0x9c)
Sep 29 09:32:59 nas-bruce kernel: [<c03a6080>] (dump_stack) from [<c009f838>] (warn_alloc_failed+0xdc/0x120)
Sep 29 09:32:59 nas-bruce kernel: [<c009f838>] (warn_alloc_failed) from [<c00a2170>] (__alloc_pages_nodemask+0x818/0x8ec)
Sep 29 09:32:59 nas-bruce kernel: [<c00a2170>] (__alloc_pages_nodemask) from [<c00b5c50>] (kmalloc_order+0x10/0x28)
Sep 29 09:32:59 nas-bruce kernel: [<c00b5c50>] (kmalloc_order) from [<c04f524c>] (mvneta_rx_refill+0x140/0x160)
Sep 29 09:32:59 nas-bruce kernel: [<c04f524c>] (mvneta_rx_refill) from [<c04f8790>] (mvneta_poll+0x3d8/0x7a4)
Sep 29 09:32:59 nas-bruce kernel: [<c04f8790>] (mvneta_poll) from [<c05d8810>] (net_rx_action+0x14c/0x314)
Sep 29 09:32:59 nas-bruce kernel: [<c05d8810>] (net_rx_action) from [<c00276b0>] (__do_softirq+0xe0/0x230)
Sep 29 09:32:59 nas-bruce kernel: [<c00276b0>] (__do_softirq) from [<c0027838>] (run_ksoftirqd+0x38/0x68)
Sep 29 09:32:59 nas-bruce kernel: [<c0027838>] (run_ksoftirqd) from [<c0040a88>] (smpboot_thread_fn+0x18c/0x2e4)
Sep 29 09:32:59 nas-bruce kernel: [<c0040a88>] (smpboot_thread_fn) from [<c003d4b0>] (kthread+0x104/0x124)
Sep 29 09:32:59 nas-bruce kernel: [<c003d4b0>] (kthread) from [<c000f580>] (ret_from_fork+0x14/0x34)
Sep 29 09:32:59 nas-bruce kernel: Mem-Info:

2. The same error with StephenB's

Oct 04 19:44:49 nas-bruce kernel: INFO: task smbd:19030 blocked for more than 200 seconds.
Oct 04 19:44:49 nas-bruce kernel:       Tainted: P           O    4.4.140.armada.1 #1
Oct 04 19:44:49 nas-bruce kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 04 19:44:49 nas-bruce kernel: smbd            D c068129c     0 19030   2894 0x00000001
Oct 04 19:44:49 nas-bruce kernel: [<c068129c>] (__schedule) from [<c068155c>] (schedule+0xa4/0xd4)
Oct 04 19:44:49 nas-bruce kernel: [<c068155c>] (schedule) from [<c0681860>] (schedule_preempt_disabled+0x24/0x40)
Oct 04 19:44:49 nas-bruce kernel: [<c0681860>] (schedule_preempt_disabled) from [<c0683a1c>] (__mutex_lock_slowpath+0x1b4/0x374)
Oct 04 19:44:49 nas-bruce kernel: [<c0683a1c>] (__mutex_lock_slowpath) from [<c0683c00>] (mutex_lock+0x24/0x40)
Oct 04 19:44:49 nas-bruce kernel: [<c0683c00>] (mutex_lock) from [<c02a10cc>] (btrfs_file_write_iter+0x5c/0x524)
Oct 04 19:44:49 nas-bruce kernel: [<c02a10cc>] (btrfs_file_write_iter) from [<c00d366c>] (__vfs_write+0xa8/0xd0)
Oct 04 19:44:49 nas-bruce kernel: [<c00d366c>] (__vfs_write) from [<c00d3cdc>] (vfs_write+0xa8/0x130)
Oct 04 19:44:49 nas-bruce kernel: [<c00d3cdc>] (vfs_write) from [<c00d44ac>] (SyS_pwrite64+0x5c/0x7c)
Oct 04 19:44:49 nas-bruce kernel: [<c00d44ac>] (SyS_pwrite64) from [<c000eb20>] (ret_fast_syscall+0x0/0x40)

The second one may be the real issue.  we are trying to reproduce it. 

Guys, any special operation need to be done ?  Or just drag and drop files to a samba share?

Message 10 of 28
StephenB
Guru

Re: Firmware 6.9.4. makes RN104 hang up

FWIW, your second log snippet wasn't from my log.

 


@Ki_Adi_Mundi wrote:

 

Guys, any special operation need to be done ?  Or just drag and drop files to a samba share?


In my case, the failure always happens (every time) when I drag/drop a file from one share to another from Windows 10.  Note the entire data volume is mapped to a drive letter.  Generally I am doing this between two specific (public) shares.  Windows explorer completely locks up from that PC.  If I do net use * /delete and remap the drive, I can regain access, but the file I was copying remains locked on the destination share.

 

The strict sync setting doesn't matter, and neither share has auto defrag enabled.  I haven't seen it when I copy files between the same shares with SSH.  But once the problem happens with SMB, the destination file can't be deleted or moved with SSH either.  When I compare the locked destination file with the source, they do not match.   Once the problem occurs, I end up rebooting the NAS fo recover.

 

I haven't had the time to narrow it further (whether it happens between other shares, or whether I can reproduce it on an RN524, RN102, or RN202).  I believe the problem also will occur if I copy the file from the mapped data volume to a different mapped drive (mapping only the target share).  But I'm not 100% certain of that.

 

The workaround for now is to copy the file from the source share to the PC hard drive, and then copy it again from the hard drive to the target share.  That never fails.  Copying files within the source share, or within the target share also never fails.  So both shares can be read/written from the PC, the problem only happens when I drag/drop something directly from the source to the target share.

Message 11 of 28
Bruce2k18
Tutor

Re: Firmware 6.9.4. makes RN104 hang up

Testing rebuild of 6.9.4 FW by Netgear today. Stay up for news.

Message 12 of 28
Ki_Adi_Mundi
NETGEAR Expert

Re: Firmware 6.9.4. makes RN104 hang up

Thanks for trying that.  As StephenB said he can only saw this issue by copying files between shares,   may I know what's your way to reproduce this constantly?

Message 13 of 28
Bruce2k18
Tutor

Re: Firmware 6.9.4. makes RN104 hang up

Hey,

 

i reproduce the bug by using the NAS. 😉 No for real - NAS hang 20-60 Minutes after updating everytime. First 24 hours made. 72 hours following. "Device Uptime: 1 day 2 hours 12 minutes"

I have incremential backups - sometimes 160 GB big running every day here plus "normal" NAS traffic. 

And it always hanged when something downloaded to the public share / user folders .

Something was copied / backuped from Android to the NAS .

So i guess the rebuild is working like 6.9.3 only with the updates. 

 

BUT - i have another Problem so far with the rebiuld - I just checked the logs and it did not recognize the files system of the USB 3 attached Backup HDD 😞 - Gotta check on that now .

Cheers Bruce

Message 14 of 28
Bruce2k18
Tutor

Re: Firmware 6.9.4. makes RN104 hang up

CONFIRMED. After downgrading to 6.9.3 NTFS external storage was recognized again... I guess a rebuild of the rebuild is needed. 😉

Message 15 of 28
Ki_Adi_Mundi
NETGEAR Expert

Re: Firmware 6.9.4. makes RN104 hang up

Thanks for finding this,  looks like the driver is not rebuild. 

Does the rebuild fix your hung issue? 

Message 16 of 28
Bruce2k18
Tutor

Re: Firmware 6.9.4. makes RN104 hang up

Yes it did fix this. But as described there were other bugs.

Message 17 of 28
Karnicus
Aspirant

Re: Firmware 6.9.4. makes RN104 hang up

I'm in exactly the same position currently. 

 

If I copy any file from share to share with Windows 10, the copy instantly locks up and fails. Any attempt to use the destination file results in the process locking up in state "D" according to `ps` output. Including attempting to `rm` the file. This also includes any processes that scan the files such as Plex, resulting in Plex locking up requiring restarting the NAS. Even restarting the NAS fails and requries a hard-reset.

 

Once the device has reset, I am able to rm the offending files through SSH and everything seems fine again. 

 

Copying the files down to the local PC and back to the other share, or even copying from share to share via SSH works fine.

 

I've only encountered this issue since 6.9.4. Very much looking forward to a fix...

Model: RN31600|ReadyNAS 300 Series 6- Bay
Message 18 of 28
StephenB
Guru

Re: Firmware 6.9.4. makes RN104 hang up

Try downgrading back to 6.9.3, and see if those issues disappear.

Message 19 of 28
jbandes
Aspirant

Re: Firmware 6.9.4. makes RN104 hang up

I am fine in 6.9.3 but when I try to upgrade to 6.9.4 the webpage times out and just sits there. The connection to the Administrator Webpage fails. I let it sit for an hour but nothing happens. When I reboot it comes back up but on Firmware 6.9.3 and asks me again if I want to do an upgrade to 6.9.4. It was also crashing on loading the Administrator Webpage.

 

I ran a scub on the disks and today it did the same thing but at least now the Webpage does not crash, I just cant upgrae to 6.9.4 on completing the downloading and install the Administrator webpage hangs and disconnects. The only way to fix it is to reboot the NAS but then it comes back up on version 6.9.3. I am curious if anyone else if having problems upgrading to 6.9.4. I think I will just wait for the next firmware update.

Model: ReadyNAS RN12P1220|ReadyNAS 3200
Message 20 of 28
evan2
NETGEAR Expert

Re: Firmware 6.9.4. makes RN104 hang up

We find root cause and has fixed it in kernel, 

We will release a 6.9.5 beta FW.

Message 21 of 28
somervda
Aspirant

Re: Firmware 6.9.4. makes RN104 hang up

Same problem here after 6.9.4 updated - looks like it has been disgnosed so will wait for 6.9.5 unless there is an easy way to go back to 6.9.3

 

Thanks

Model: RN10400|ReadyNAS 100 Series 4-Bay (Diskless)
Message 22 of 28
Bruce2k18
Tutor

Re: Firmware 6.9.4. makes RN104 hang up

ReadyNASOS 6.9.5-T7 (Beta 1) solved all my problems so far and running smooth for 3 days now.

Message 23 of 28
bedlam1
Prodigy

Re: Firmware 6.9.4. makes RN104 hang up

OS 6.9.5 is out of Beta & there is a full version now (in fact 6.10 Beta is out now) 

Message 24 of 28
StephenB
Guru

Re: Firmware 6.9.4. makes RN104 hang up


@bedlam1 wrote:

OS 6.9.5 is out of Beta


I'm not seeing that, it still appears to be in Beta.

Message 25 of 28
Top Contributors
Discussion stats
  • 27 replies
  • 4557 views
  • 1 kudo
  • 9 in conversation
Announcements