fvbackup crashes, stalling further backup, disables graceful shutdown
ReadyNAS 314 with Firmware 6.9.5 Hotfix 1
I have a problem with backup jobs crashing and freezing ReadyNas from time to time. These jobs back up various ReadyNas shares to a portable USB drive.
Once in a while, the jobs will get stuck. The timing of the problem seems completely random.
I have periodically run DEFRAG and BALANCE without any issue.
Please help me get some insight in this issue. How do I resolve this?
Thank you.
This is how it goes:
(a) The first backup job will start (b) "/frontview/bin/fvbackup" will crash (segmentation fault) - see log below (c) The log will have a stack trace (d) In the stack trace, the RIP will have this string "ufsdapi_file_flush+0x253/0x2a0 [ufsd]" (e) The small status display screen will also display "ufsdapi_file_flush+0x253/0x2a0 [ufsd]"
At this point, all future backups will stop.
(a) The ReadyNas shares will remain accessible from my Windows PC, but (b) the admin web page will be inaccessible from a browser. (c) I shall be able to ssh into the ReadyNas and check logs etc. (d) Most significantly, once this happens, there is no way of gracefully shutting down the Nas (e) The only possibility is a forced shutdown by "Press and hold the Power button for 5 seconds" (f) Issuing a "shutdown -P" command after ssh'ing to the Nas does not shut it down. (g) The force shutdown makes the attached portable USB drive read-only. (h) It's probably not good for the file system either.
Re: fvbackup crashes, stalling further backup, disables graceful shutdown
I believe the hightlighed error is linked to the USB drive.
I'd test the USB drive in a PC. You can use vendor tools (seatools or lifeguard) and/or just try to copy files to and from the drive. Personally I'd do the full erase test in addition to the extended non-destructive test.
Re: fvbackup crashes, stalling further backup, disables graceful shutdown
Thank you for your response. That was my first hunch too. I might have forgotten to note in my original posting that I have been getting this for more than a error, sporadically.
So during a year of troubleshooting, I have tried out the following already, on the USB disk:
(1) As ReadyNAS renders the USB disk readonly, I would connect it to a windows PC and simply clear the code, reconnect it back to ReadyNAS. After that it will work fine for a while.
(2) Completely erase and reformat the USB disk using a Windows PC and then connect it back to ReadyNAS. After that it will work fine for a while.
(3) Replace the USB disk with a brand new one of a different brand. After that it will work fine for a while.
As I get the same symptoms back irrespective of what I do, I tend to think that it is not related to the USB disk.
Moreover, I have been running the same backup's for several years now on the same kind of USB disks. I started getting it only about a year back; after one ReadyNAS OS update. I am not sure which one.
If I may ask what exactly makes you think that the error is related to the USB drive? Is it because of the reference to ufsd?
Model: RN31400|ReadyNAS 300 Series 4- Bay (Diskless)
(1) As ReadyNAS renders the USB disk readonly, I would connect it to a windows PC and simply clear the code, reconnect it back to ReadyNAS. After that it will work fine for a while.
This might be a result of an improper ejection (which of course is a given if the NAS crashes with the drive mounted).
Re: fvbackup crashes, stalling further backup, disables graceful shutdown
I am sorry, but I do not understand the diagnosis.
I have tried a brand new USB drive - I get the same error on that too. Please note, the error doesn't happen frequently; only once in 2-3 months.
Not just that, I have other backup jobs backing up other shares on the same USB disk. Those never fail. How can it be a disk issue if it shows up for only one backup job?
This makes me think that the issue is not with the USB disk per se.
Thanks.
Model: RN31400|ReadyNAS 300 Series 4- Bay (Diskless)
I am sorry, but I do not understand the diagnosis.
To be clear - it's at most a hypothesis that needs to be confirmed. If it were my own disk, I'd begin by running both the long non-destructive test and the full write-zeros/erase disk test using the vendor tools (WD's lifeguard program or Seagate's Seatools).
If you aren't running the maintenance tasks (on the volume settings wheel) regularly, you might also run the disk test for the internal drives there. The scrub task is also a good diagnostic (as it either reads or writes every sector in the data volume). Though I'd hesitate to do a scrub before confirming that I had a solid backup.
Not just that, I have other backup jobs backing up other shares on the same USB disk. Those never fail. How can it be a disk issue if it shows up for only one backup job?
This makes me think that the issue is not with the USB disk per se.
You could be correct - though the error definitely says the paragon drivers are crashing.
I have certainly had USB drives that fail in a particular range of sectors. It is conceivable that the particular share just happens to wind up on a marginal area of the disk. If the share has more churn (files changing) then it might also be the case that the specific share requires more I/O than the others.
FWIW, I have had at least one case where I found that many old files that hadn't changed since the initial backup couldn't be restored when I needed them - the copies failed when I restored from the USB drive. Personally I make sure I have at least two backups - in part to reduce the possibility that the backup hasn't silently failed.
BTW, another strategy here is to connect the drive to a Windows PC and back up the NAS over ethernet. There are free tools out there that can do that on schedule (and I think some might also be able to verify). Then you could see if the problem moves to the PC or stays with the NAS.
Re: fvbackup crashes, stalling further backup, disables graceful shutdown
Although I do appreciate the suggestions made by StephenB, I still believe that we don't know what is causing the crash. We do know that the Paragon driver is crashing. But why?
Is there any direct way of knowing?
To be very clear, this crash is **not regular**. It happens once in a while, sometimes after 4 months of no crash, sometimes 6 months. No pattern. It only happens with this one backup job. But it goes through without crashing if I recycle the NAS and rerun the backup job.
I know it has been a long exchange. It becomes hard to follow when the chain gets long. So to summarize:
(1) I do regulalrly run the balancing and defragmentation on the internal disks
(2) I do occasionally run the scrubbing too
(3) I did run the internal disk test on the internal disks once in the recent past.
None of that indicated any error condition.
(4) I have done complete reformatting the USB disk
(5) I have replaced the USB disk with a brand new one.
But the random and irregular crash continues.
I can't simulate the problem. It happens when it happens.
Any thought on how to get to the root cause of this will be very much appreciated.
Thanks.
Model: RN31400|ReadyNAS 300 Series 4- Bay (Diskless)
Re: fvbackup crashes, stalling further backup, disables graceful shutdown
One thing you haven't tried (at least it's not on your list) is switching to a different USB port. If you're using the front port, then shift to one of the back ones (and vice versa).
