fvbackup crashes, stalling further backup, disables graceful shutdown

Question

ReadyNAS 314 with Firmware 6.9.5 Hotfix 1&nbsp;I have a problem with backup jobs crashing and freezing ReadyNas from time to time. These jobs back up various ReadyNas shares to a portable USB drive.Once in a while, the jobs will get stuck. The timing of the problem seems completely random.I have periodically run DEFRAG and BALANCE without any issue.Please help me get some insight in this issue. How do I resolve this?Thank you.&nbsp;This is how it goes:&nbsp;(a) The first backup job will start(b) "/frontview/bin/fvbackup" will crash (segmentation fault) - see log below(c) The log will have a stack trace(d) In the stack trace, the RIP will have this string "ufsdapi_file_flush+0x253/0x2a0 [ufsd]"(e) The small status display screen will also display "ufsdapi_file_flush+0x253/0x2a0 [ufsd]"&nbsp;At this point, all future backups will stop.&nbsp;(a) The ReadyNas shares will remain accessible from my Windows PC, but(b) the admin web page will be inaccessible from a browser.(c) I shall be able to ssh into the ReadyNas and check logs etc.(d) Most significantly, once this happens, there is no way of gracefully shutting down the Nas(e) The only possibility is a forced shutdown by "Press and hold the Power button for 5 seconds"(f) Issuing a "shutdown -P" command after ssh'ing to the Nas does not shut it down.(g) The force shutdown makes the attached portable USB drive read-only.(h) It's probably not good for the file system either.&nbsp;******************** LOG SECTION - START ************************Mar 03 01:05:11 MITRANAS fvbackup-q[3515]: cmd=/frontview/bin/fvbackup -e 27Mar 03 01:05:11 MITRANAS fvbackup-q[3515]: Create a readonly snapshot of '/data/Documents' in '/data/._share/Documents/.snapshot/b_1583215511_21184'Mar 03 01:10:56 MITRANAS dbus[2832]: [system] Activating service name='org.opensuse.Snapper' (using servicehelper)Mar 03 01:10:58 MITRANAS dbus[2832]: [system] Successfully activated service 'org.opensuse.Snapper'Mar 03 01:14:25 MITRANAS clamd[32081]: SelfCheck: Database status OK.Mar 03 01:14:25 MITRANAS clamd[32081]: SelfCheck: Database status OK.Mar 03 01:17:03 MITRANAS CRON[21799]: pam_unix(cron:session): session opened for user root by (uid=0)Mar 03 01:17:03 MITRANAS CRON[21803]: (root) CMD ( cd / &amp;&amp; run-parts --report /etc/cron.hourly)Mar 03 01:17:03 MITRANAS CRON[21799]: pam_unix(cron:session): session closed for user rootMar 03 01:17:28 MITRANAS connmand[2999]: ntp: adjust (slew): -0.192048 secMar 03 01:24:25 MITRANAS clamd[32081]: SelfCheck: Database status OK.Mar 03 01:24:25 MITRANAS clamd[32081]: SelfCheck: Database status OK.Mar 03 01:28:58 MITRANAS kernel: general protection fault: 0000 [#1] SMPMar 03 01:28:58 MITRANAS kernel: Modules linked in: ufsd(PO) jnl(O) vpd(PO)Mar 03 01:28:58 MITRANAS kernel: CPU: 1 PID: 21184 Comm: fvbackup Tainted: P O 4.4.157.x86_64.1 #1Mar 03 01:28:58 MITRANAS kernel: Hardware name: NETGEAR ReadyNAS 314/ReadyNAS 314 , BIOS 4.6.5 01/07/2013Mar 03 01:28:58 MITRANAS kernel: task: ffff88007e665400 ti: ffff8800783ec000 task.ti: ffff8800783ec000Mar 03 01:28:58 MITRANAS kernel: RIP: 0010:[&lt;ffffffffa001e953&gt;] [&lt;ffffffffa001e953&gt;] ufsdapi_file_flush+0x253/0x2a0 [ufsd]Mar 03 01:28:58 MITRANAS kernel: RSP: 0018:ffff8800783efb80 EFLAGS: 00010202Mar 03 01:28:58 MITRANAS kernel: RAX: 0000000000000234 RBX: ffff88003a5fd618 RCX: ffff88003a5fd708Mar 03 01:28:58 MITRANAS kernel: RDX: 0000000000000234 RSI: ffff880063cee094 RDI: ffff88003a5fd618Mar 03 01:28:58 MITRANAS kernel: RBP: ffff8800783efbf0 R08: ffff880063cee094 R09: 0000000000000634Mar 03 01:28:58 MITRANAS kernel: R10: 0000000000000234 R11: 0000000000000634 R12: 0000000000000634Mar 03 01:28:58 MITRANAS kernel: R13: 0000000000000434 R14: 0000000000000014 R15: 0000000000000034Mar 03 01:28:58 MITRANAS kernel: FS: 00007f121bcb8a00(0000) GS:ffff88007ea80000(0000) knlGS:0000000000000000Mar 03 01:28:58 MITRANAS kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033Mar 03 01:28:58 MITRANAS kernel: CR2: 00007f64e00be000 CR3: 000000007ba6e000 CR4: 00000000000006f0Mar 03 01:28:58 MITRANAS kernel: Stack:Mar 03 01:28:58 MITRANAS kernel: 0000000000000000 0000000000000000 ffff880063cee000 0000000000000614Mar 03 01:28:58 MITRANAS kernel: 0000000000000214 0000000000000434 0000000000000634 0000000000000234Mar 03 01:28:58 MITRANAS kernel: dead000000000100 ffff88006a711000 ffff880010c7dc00 ffff88006a711220Mar 03 01:28:58 MITRANAS kernel: Call Trace:Mar 03 01:28:58 MITRANAS kernel: [&lt;ffffffffa0011329&gt;] do_delayed_tasks+0xdc/0x149 [ufsd]Mar 03 01:28:58 MITRANAS kernel: [&lt;ffffffffa001141f&gt;] _lock_ufsd+0x89/0x94 [ufsd]Mar 03 01:28:58 MITRANAS kernel: [&lt;ffffffffa00181d6&gt;] ufsd_file_open+0x10c/0x26e [ufsd]Mar 03 01:28:58 MITRANAS kernel: [&lt;ffffffffa00180ca&gt;] ? ufsd_open_by_id.isra.8+0xfd/0xfd [ufsd]Mar 03 01:28:58 MITRANAS kernel: [&lt;ffffffff881106ed&gt;] do_dentry_open+0x20f/0x2a8Mar 03 01:28:58 MITRANAS kernel: [&lt;ffffffff88111583&gt;] vfs_open+0x5e/0x65Mar 03 01:28:58 MITRANAS kernel: [&lt;ffffffff8811d6aa&gt;] path_openat+0xa20/0xcb4Mar 03 01:28:58 MITRANAS kernel: [&lt;ffffffff8811e82b&gt;] do_filp_open+0x39/0x7eMar 03 01:28:58 MITRANAS kernel: [&lt;ffffffff88128633&gt;] ? __alloc_fd+0xaa/0x167Mar 03 01:28:58 MITRANAS kernel: [&lt;ffffffff88111892&gt;] do_sys_open+0x14b/0x1dbMar 03 01:28:58 MITRANAS kernel: [&lt;ffffffff8811193b&gt;] SyS_open+0x19/0x1bMar 03 01:28:58 MITRANAS kernel: [&lt;ffffffff888cb38a&gt;] entry_SYSCALL_64_fastpath+0x1e/0x8eMar 03 01:28:58 MITRANAS kernel: Code: 7d c8 4c 89 f0 eb 93 90 83 e1 08 74 b5 4c 89 c8 48 8b 4d 98 48 8b 55 a0 4c 89 c648 89 df 48 89 8a 94 00 00 00 48 8b 0b 48 89 c2 &lt;ff&gt; 51 10 85 c0 75 95 48 83 7d 90 00 74 8e 48 8b 75 a0 4c 8b 75Mar 03 01:28:58 MITRANAS kernel: RIP [&lt;ffffffffa001e953&gt;] ufsdapi_file_flush+0x253/0x2a0 [ufsd]Mar 03 01:28:58 MITRANAS kernel: RSP &lt;ffff8800783efb80&gt;Mar 03 01:28:59 MITRANAS kernel: ---[ end trace c5f07c4159df64c5 ]---Mar 03 01:28:58 MITRANAS fvbackup-q[3515]: Error executing backup job (job_id=27). rc=35584 errorno=2******************** LOG SECTION - END ************************&nbsp;

StephenB · Answer

I believe the hightlighed error is linked to the USB drive.
&nbsp;
I'd test the USB drive in a PC.&nbsp; You can use vendor tools (seatools or lifeguard) and/or just try to copy files to and from the drive.&nbsp; Personally I'd do the full erase test in addition to the extended non-destructive test.
&nbsp;
&nbsp;

debumitra · Answer

Thank you for your response. That was my first hunch too. I might have forgotten to note in my original posting that I have been getting this for more than a error, sporadically.

So during a year of troubleshooting, I have tried out the following already, on the USB disk:

(1) As ReadyNAS renders the USB disk readonly, I would connect it to a windows PC and simply clear the code, reconnect it back to ReadyNAS. After that it will work fine for a while.

(2) Completely erase and reformat the USB disk using a Windows PC and then connect it back to ReadyNAS. After that it will work fine for a while.

(3) Replace the USB disk with a brand new one of a different brand. After that it will work fine for a while.

As I get the same symptoms back irrespective of what I do, I tend to think that it is not related to the USB disk.

Moreover, I have been running the same backup's for several years now on the same kind of USB disks. I started getting it only about a year back; after one ReadyNAS OS update. I am not sure which one.

If I may ask what exactly makes you think that the error is related to the USB drive? Is it because of the reference to ufsd?

StephenB · Answer

debumitra&nbsp;wrote:
&nbsp;
If I may ask what exactly makes you think that the error is related to the USB drive? Is it because of the reference to ufsd?
&nbsp;

Yes. Paragon's driver set is called "Universal File System Drivers", and they use ufsd in a variety of places.&nbsp;https://download.paragon-software.com/doc/Paragon_NTFS-HFS_for_Linux_9.5_User_Manual.pdf
&nbsp;
debumitra&nbsp;wrote:
&nbsp;
(1) As ReadyNAS renders the USB disk readonly, I would connect it to a windows PC and simply clear the code, reconnect it back to ReadyNAS. After that it will work fine for a while.
&nbsp;

This might be a result of an improper ejection (which of course is a given if the NAS crashes with the drive mounted).
&nbsp;
debumitra&nbsp;wrote:
(2) Completely erase and reformat the USB disk using a Windows PC and then connect it back to ReadyNAS. After that it will work fine for a while.
&nbsp;

I would definitely test it then.

debumitra · Answer

I am sorry, but I do not understand the diagnosis.&nbsp;I have tried a brand new USB drive - I get the same error on that too. Please note, the error doesn't happen frequently; only once in 2-3 months.&nbsp;Not just that, I have other backup jobs backing up other shares on the same USB disk. Those never fail. How can it be a disk issue if it shows up for only one backup job?&nbsp;This makes me think that the issue is not with the USB disk per se.&nbsp;Thanks.

StephenB · Answer

debumitra&nbsp;wrote:
I am sorry, but I do not understand the diagnosis.
&nbsp;

To be clear - it's at most a hypothesis that needs to be confirmed.&nbsp; If it were my own disk, I'd begin by running both the long non-destructive test and the full write-zeros/erase disk test using the vendor tools (WD's lifeguard program or Seagate's Seatools).
&nbsp;
If you aren't running the maintenance tasks (on the volume settings wheel) regularly, you might also run the disk test for the internal drives there.&nbsp; The scrub task is also a good diagnostic (as it either reads or writes every sector in the data volume).&nbsp; Though I'd hesitate to do a scrub before confirming that I had a solid backup.
&nbsp;
debumitra&nbsp;wrote:
&nbsp;
Not just that, I have other backup jobs backing up other shares on the same USB disk. Those never fail. How can it be a disk issue if it shows up for only one backup job?
&nbsp;
This makes me think that the issue is not with the USB disk per se.
&nbsp;

You could be correct - though the error definitely says the paragon drivers are crashing.
&nbsp;
I have certainly had USB drives that fail in a particular range of sectors.&nbsp; It is conceivable that the particular share just happens to wind up on a marginal area of the disk.&nbsp; If the share has more churn (files changing) then it might also be the case that the specific share requires more I/O than the others.
&nbsp;
FWIW, I have had at least one case where I found that many old files that hadn't changed since the initial backup couldn't be restored when I needed them - the copies failed when I restored from the USB drive.&nbsp; Personally I make sure I have at least two backups - in part to reduce the possibility that the backup hasn't silently failed.
&nbsp;
BTW, another strategy here is to connect the drive to a Windows PC and back up the NAS over ethernet.&nbsp; There are free tools out there that can do that on schedule (and I think some might also be able to verify).&nbsp; Then you could see if the problem moves to the PC or stays with the NAS.&nbsp;&nbsp;

Forum Discussion

fvbackup crashes, stalling further backup, disables graceful shutdown

7 Replies

Related Content

RAX200: Sporadische crashes

RAX54 v1.0 - Constant Crashing

ReadyNas Pro4 does not boot - at 85% fvbackup-q.service - Management service is offline

Stalls from WAC730

Orbi 970 - Crash after Speedtest

NETGEAR Academy

ProSupport for Business