NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.

Forum Discussion

VolkerB's avatar
VolkerB
Aspirant
Sep 22, 2021
Solved

Admin page unavailable after cancelled backup job & hard reboot, shares are working

Hopefully someone can help me get out of this deadlock.

 

I recently added a rsync backup job to sync the local /media share with an USB drive that was connected via the rear USB socket of the RN214 box. After starting it, I found out that - by mistake - the NAS was creating a directory /media/media/... on the USB drive (essentially duplicating all my data), so I cancelled the backup job via the ReadyNAS admin page. After refreshing the page I was presented with  the progress bar and eventually a notification similar to the one described in https://kb.netgear.com/26883/ReadyNAS-OS-6-Admin-Page-is-offline. A graceful restart did not work, the LCD was not lighting up. So I pulled the plug, forcing a cold restart.

 

After this, the device came back up again, however boot progress notification is stuck at 95%, the upper line of the LCD says "fvbackup-q.servi". I can ping the device and the shares are accessible (RW). Unfortunately the admin page http://rn214/admin/ is still offline (same error as mentioned above), the power button never stops blinking.

 

I have not set up SSH access (dammit!), so I can't log on and view logs/running processes. NetGear RAIDar reported the management service to be offline and was unable to retrieve logfiles. Diagnostics yielded the snippet below (slightly abbreviated, there were a lot more entries saying "Failed to UPDATE dictionary".

 

I can reboot the device using RAIDar. In that case no progress bar is shown at all (only "Booting...") and the "fvbackup-q.servi" information if the power button is pressed once. Shares are available, admin page says "Connecting..." and then ends up with the offline notification mentioned above.

 

I then tried the boot menu's "OS Reinstall" option which successfully recovered the admin page. In the log, it says "Volume: System volume root's usage is 99%. This condition should not occur under normal conditions. Contact technical support.", downloading the logfiles (http://rn214/dbbroker) fails with this XML file:

 

<xs:nml xmlns:xs="http://www.netgear.com/protocol/transaction/NMLSchema-0.9" src="browser" dst="nas" locale="en-us">
<xs:transaction ref-id="" type="0">
<xs:response ref-id="opid" status="failure">
<xs:error>
<xs:error-code>
<![CDATA[ 12008010002 ]]>
</xs:error-code>
<xs:error-cause>
<![CDATA[ Can't create zipped log ]]>
</xs:error-cause>
<xs:error-details>
<![CDATA[ Error in dlowload log ]]>
</xs:error-details>
</xs:error>
</xs:response>
</xs:transaction>
</xs:nml>

Any advice on how to proceed?

 

Many thanks in advance!

 

Successfully completed diagnostics
System
No errors found.
Logs
2021-09-22 11:34:39: Assertion 'f' failed at ../src/journal/journal-file.c:1674, function journal_file_post_change(). Aborting.
2021-09-22 11:02:36: ufsd: "mount" (sda2): is mounted as NTFS at 2021-09-22 09:02:36
2021-09-22 11:00:19: ufsd: "umount" (sda1): is unmounted at 2021-09-22 09:00:19
2021-09-22 07:26:17: ufsd: "mount" (sda1): is mounted as NTFS at 2021-09-22 05:26:17
2021-09-22 07:25:57: ufsd: "umount" (sda1): is unmounted at 2021-09-22 05:25:57
2021-09-22 07:25:37: ufsd: "mount" (sda1): is mounted as NTFS at 2021-09-22 05:25:37
2021-09-22 07:20:02: ufsd: "umount" (sda2): is unmounted at 2021-09-22 05:20:02
2021-09-22 07:19:50: ufsd: "umount" (sda1): is unmounted at 2021-09-22 05:19:50
2021-09-22 07:19:31: ufsd: "mount" (sda1): is mounted as NTFS at 2021-09-22 05:19:31
2021-09-22 07:19:12: ufsd: "umount" (sda1): is unmounted at 2021-09-22 05:19:12
2021-09-22 07:18:02: ufsd: "mount" (sda1): is mounted as NTFS at 2021-09-22 05:18:02
2021-09-22 07:17:16: ufsd: "umount" (sda1): is unmounted at 2021-09-22 05:17:16
2021-09-22 07:05:39: ufsd: "mount" (sda1): is mounted as NTFS at 2021-09-22 05:05:39
2021-09-22 01:01:17: ufsd: "umount" (sda1): is unmounted at 2021-09-21 23:01:17
2021-09-21 07:44:33: ufsd: "mount" (sda1): is mounted as NTFS at 2021-09-21 05:44:33
2021-09-21 01:00:57: ufsd: "umount" (sda1): is unmounted at 2021-09-20 23:00:57
2021-09-20 10:22:24: ufsd: "mount" (sda1): is mounted as NTFS at 2021-09-20 08:22:24
2021-09-20 10:22:06: ufsd: "umount" (sda1): is unmounted at 2021-09-20 08:22:06
2021-09-20 10:19:31: ufsd: "mount" (sda1): is mounted as NTFS at 2021-09-20 08:19:31
2021-09-20 10:18:29: ufsd: "umount" (sda1): is unmounted at 2021-09-20 08:18:29
2021-09-20 08:13:42: ufsd: "mount" (sda1): is mounted as NTFS at 2021-09-20 06:13:42
2021-09-03 09:55:17: ufsd: "umount" (sdc1): is unmounted at 2021-09-03 07:55:17
System Management
2021-09-22 11:37:21: Failed to UPDATE dictionary
2021-09-22 11:37:21: Failed to UPDATE dictionary
2021-09-22 11:37:03: Failed to start ReadyNAS System Daemon.
2021-09-22 11:36:43: Failed to start ReadyNAS System Daemon.
2021-09-22 11:36:17: DB (main) schema version: new ==> 24
2021-09-22 11:36:17: DB (queue) schema version: new ==> 0
2021-09-22 11:36:16: DB sanity check failed! Trying backup readynasd_2021_09_22_110945.db.lz4.
2021-09-22 11:36:16: DB sanity check failed! Trying backup readynasd_2021_09_22_110945.db.
  • Did you at any point disconnect or power down the USB drive before the backup job said it was done?  I ask because I suspect that it continued after that.  But once the USB drive wasn't connected, it began to copy files to the mount point in the OS partion instead of the USB drive that should have been mounted there.  That'll fill the OS partition in a hurry.

     

    The message you are seeing is the location where the OS crashed, undoubtedly due to the too-full OS partition.  If you are now able to enable SSH after the OS re-install, you need to go in and clear out any files that were copied to the mount point directory.

     

    The fact that your files were copied to media/media is the way the rsync backup jobs are designed.  I disagree that's the way it shoud be, but it is.  The work-around is that you need to go back into the backup job configuration after it's created and put a single forward slash "/"  as the source path.

12 Replies

Replies have been turned off for this discussion
  • Sandshark's avatar
    Sandshark
    Sensei - Experienced User

    Did you at any point disconnect or power down the USB drive before the backup job said it was done?  I ask because I suspect that it continued after that.  But once the USB drive wasn't connected, it began to copy files to the mount point in the OS partion instead of the USB drive that should have been mounted there.  That'll fill the OS partition in a hurry.

     

    The message you are seeing is the location where the OS crashed, undoubtedly due to the too-full OS partition.  If you are now able to enable SSH after the OS re-install, you need to go in and clear out any files that were copied to the mount point directory.

     

    The fact that your files were copied to media/media is the way the rsync backup jobs are designed.  I disagree that's the way it shoud be, but it is.  The work-around is that you need to go back into the backup job configuration after it's created and put a single forward slash "/"  as the source path.

    • VolkerB's avatar
      VolkerB
      Aspirant

      Sandshark wrote:

      Did you at any point disconnect or power down the USB drive before the backup job said it was done?  I ask because I suspect that it continued after that.  But once the USB drive wasn't connected, it began to copy files to the mount point in the OS partion instead of the USB drive that should have been mounted there.  That'll fill the OS partition in a hurry.

      OMG. That is absolutely possible. rsync backup jobs on the RN214 always were a mystery to me, however still being the only choice to have a one-click incremental nobrainer backup to an attached USB3 device excluding btrfs snapshots and a couple of other unnecessary directories. Once I got aware of the /media/media problem, I wanted to avoid 4TB of redundant data being copied only to be deleted once the job was finished. Hence I canceled the job and did not really care about the HDD access  LED in the NAS. I then probably have ejected the USB device (don't really remember if I did) or it could be that at that point the admin page was already unresponsive.

       

      Since I was stupid enough not to enable SSH and the power button shutdown did not work, I had to pull the plug. Later I learned about RAIDar, which I could have tried to force a shutdown as well.

       

      After the forced restart, the admin GUI just showed the progress bar similar to https://kb.netgear.com/26883/ReadyNAS-OS-6-Admin-Page-is-offline, no way around that. I resorted to the OS reinstall boot menu option after which I could log onto the admin GUI successfully. Then I saw the "Volume: System volume root's usage is 99%. This condition should not occur under normal conditions. Contact technical support." log message but sure enough, I don't have any technical support contract.

       

      So the master plan was to enable SSH shell access for admin, which failed (blocked GUI again and all other kinds of weird behaviour). After a couple of OS reinstalls, suddently a "firmware image unpack failure" (or likewise) was displayed on the LCD panel and the admin GUI came up completely non-localized, with just the message tokens/placeholders instead of the real messages. Still no SSH access.

       

      That was the moment when I called it a day and performed a complete factory default reset, knowing that I had at least backed up all the data (without the snapshots unfortunately, since the NTFS partition on the external USB drive does not support them). Currently, the device is resyncing its data (at ~10%).

       


      Sandshark wrote:

      The message you are seeing is the location where the OS crashed, undoubtedly due to the too-full OS partition.  If you are now able to enable SSH after the OS re-install, you need to go in and clear out any files that were copied to the mount point directory.

      That was the plan. But it seems that my iterations and an OS partition filled up to the brim was not allowing this configuration change. Lesson learned. Enabling SSH was the veryfirst thing I did after the RN214 came up in it's pristine state.

       


      Sandshark wrote:

      The fact that your files were copied to media/media is the way the rsync backup jobs are designed.  I disagree that's the way it should be, but it is.  The work-around is that you need to go back into the backup job configuration after it's created and put a single forward slash "/"  as the source path.


      Usually I'm aware of those rsync subtleties but that's what happens, if you're copying files while working on other stuff in parallel...

       

      The big question is now: If I run an rsync backup job in the future and find out that things are going haywire: What is the recommended way of cancelling them in a safe way so that this disaster can't happen again? Does it suffice to wait for the job shown as "Cancelled"? Is there a logfile to have a look at?

       

      That was quite a miserable day to say the least. Nevertheless many thanks for your explanations, I learned a lot about rsync again.

      • Sandshark's avatar
        Sandshark
        Sensei - Experienced User

        For future aborts, I recommend you check the log (available inn the backup job menu) to see that it says the job was cancelled.  The log is only updated at completion (it doesn't show progress), so that should be a sure way to know it's done.

         

        There are other ways we could have helped you clean out the system, but it's complicated.  Since you had a full backup, that's really the road of least resistance in this kind of case.

    • VolkerB's avatar
      VolkerB
      Aspirant

       Sandshark wrote:

      Did you at any point disconnect or power down the USB drive before the backup job said it was done?  I ask because I suspect that it continued after that.  But once the USB drive wasn't connected, it began to copy files to the mount point in the OS partion instead of the USB drive that should have been mounted there.  That'll fill the OS partition in a hurry.

      OK, come to think of it and to avoid trouble in the future... Please apologize, if the follwing is a potentially stupid question:

       

      Assume, I want to backup a couple shares to an external USB drive connected to the ReadyNAS. I want to use rsync, because that allows hassle free incremental updates. So I connect the drive, enable RSYNC R/W access for that drive in the admin page share section like this:

      Now assume, I want to create a backup job for all home shares like this:

      As destination, I point the NAS to the remote rsync server on 127.0.0.1 (which is essentially the local machine accessing the USB drive via rsync):

      I exclude the /admin/snapshot directory, because this will confuse the NTFS filesystem and set the --delete option to remove remote files that don't exist locally anymore:

      So far, so good.

       

      This backup seems to be bound to the share HDS5C3020ALA632 for the destination. Now assume, that drive is not connected but I accidentially hit the backup button on the NAS. I hope, in this case it is NOT writing to the OS partition but rather failing the operation in the first place as any sane person (and all the Linux OSes I have worked with) would do.

       

      Am I right? Thanks again for your patience. I'm getting a bit paranoid after three sleepless nights. :smileywink:

       

      P.S.: I hope the screenshots show up in my post. I can see them while editing but after posting, there's just the yellow triangle...

      • Sandshark's avatar
        Sandshark
        Sensei - Experienced User

        I've not tried it, but I think the answer may be that it will write to the OS partition in that case.  I know rsync can't tell the difference when a directory that's intended as a mount point actually has something mounted to it or doesn't, but does somehting in the NAS unique software check first?

         

        The screen grabs are there.  A moderator has to approve them, but that's obviously already taken place.

         

        I think this is worth some time with my "sandbox" NAS, so I'll give it a try some time this weekend if I have the time.  You may have more fully explained why so many have issues with a too full OS partition.  We've seen instances that were clearly related to an unmounted USB drive, but I never thought about the backup button being a trigger.

         

        Or, if you want to try it, do so with a share that has very little in it, so the OS partition won't fill and you can still get in with SSH and check the content of the mount point with nothing mounted and delete anything that is there.

NETGEAR Academy

Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology! 

Join Us!

ProSupport for Business

Comprehensive support plans for maximum network uptime and business peace of mind.

 

Learn More