NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Rsync, hanging, OS 6,
I have a couple of old Sparc Duos that I am wanting to transfer the data to my new(to me) RN104. One of them has 0.2TB, the other 1.8TB data. I have previously transferred about 80% of the data from ...
The backup job is still running, though it now looks as though all files / folders have been copied.
Having enabled SSH, I used puTTY to do a find for files called *backup*
It looks like the backup jobs are stored under /etc/frontview/backup_jobs.conf. I used FTP to get the file to my PC and an example of a backup job is copied below: (IP address part obfuscated)
I am wondering if we can manually customise the rsync operation using the line highlighted in red ?
<backupJob>
<job_id>008</job_id>
<name>Sparenas media to Nas104 (all folders)</name>
<enabled>0</enabled>
<sch_every>24</sch_every>
<sch_start>0</sch_start>
<sch_end>23</sch_end>
<sch_dow>127</sch_dow>
<src_type>remote</src_type>
<src_share_proto>rsync</src_share_proto>
<src_share_vol></src_share_vol>
<src_path>172.16.x.xx::media</src_path>
<src_login></src_login>
<src_password></src_password>
<src_port>0</src_port>
<src_eject>0</src_eject>
<dst_type>share</dst_type>
<dst_share_proto>media</dst_share_proto>
<dst_share_vol>data</dst_share_vol>
<dst_path></dst_path>
<dst_login></dst_login>
<dst_password></dst_password>
<dst_port>0</dst_port>
<dst_eject>0</dst_eject>
<opt_full_backup_freq>FIRST_TIME</opt_full_backup_freq>
<opt_remove_previous_backup>0</opt_remove_previous_backup>
<opt_chown_to_share_owner>0</opt_chown_to_share_owner>
<opt_rsync_options>0</opt_rsync_options>
<opt_email_this_to_alert_email>0</opt_email_this_to_alert_email>
<opt_rsync_exclude></opt_rsync_exclude>
<opt_misc_option>0</opt_misc_option>
<wol_addr></wol_addr>
</backupJob>
That appears to only store the configurations you have chosen from those available rather than letting you add your own. Basically, it's what gets displayed on the various tabs of the backup options. On my job that deletes file on target that are deleted on source, opt_rsync_options is a one where yours is a zero.
Thanks for the reply Stephen. :smileyhappy:
I am a bit confused as I posted this message, went back to edit it to put the firmware versions in and it then seemed to disappear off the forum listing, so I made another https://community.netgear.com/t5/Using-your-ReadyNAS/Rsync-hanging-OS-6/m-p/1241702#M126140 which Sandshark has contributed to. I don't know what happened, so could I ask that you merge the threads.
Back on topic:
Yes I made sure the raid sync was complete before I started populating the Nas_104. When I bought it it had a single 2TB disk as JBOD. I added one disk which made it degraded, then Raid 1 which took about 3 hours. Adding the third disk made it degraded again, changed to Raid 5 and the rebuild took 17 hours following which a reboot expanded the volume.
As of now the job has been running for 124 hours, the backup job is still showing as in progress, all data / folders have been replicated and there is no backup log available.
Before running this job I did run another, smaller job which transferred around 120MB data through Rsync with a reported transfer rate of 4.8kb / s (bits, not Bytes, just to be clear)
Having decided I have sufficient patience to let the job continue running I have also created another rsync job on my second Duo, transferring 665MB from Duo1 to Duo2 (none of this date is on Duo2). As of now that job has been running for 30 hours, is 70% complete with a throughput rate of 4.5kb /s
All three NAS's are on the same Netgear switch.
This is a home setup so very simple from a security perspective as I only have one user group and one user, both admin as you have recommended.
I'm glad you have also seen the folder date problem. One the one hand it may not be critical, but on the other the whole idea of a copy is that it is an exact copy, not a copy with a few pieces of information changed. Running the second job from the Duo2 will also allow me to confirm that OS4 works as I remember.
I have done a bit of digging and found the backup_jobs.conf file and on Sandsharks suggestion I believe that OS 6 is specifying --omir-dir-times parameter in the rsync process, which is different to OS4. I was hoping to be able to manually force the desired flags into the conf file, but am now coming to the view that there is another file which references the conf file for backup variables, with -o being specified in that core file. I have a little experience with Linux / unix so gues that other file will be compiled and in the bin or sbin directories ??
It would also be nice if the backup jobs had some sort of progress indicator, and I see you have had that request in for some time already
Thanks for any further suggestions or advice
Thanks again. I also saw that the conf file stored the options from the GUI backup page, but I couldn't see the option which corresponded to the line I highlighted in red. I was maybe wishing / hoping that line allows the manual addition of variables, but having read up on rsync a little more the -o switch is something we want to remove from os6, not add as an additional parameter so that line probably won't achieve what we want.
There's obviously an executable file somewhere which references the conf file. Do you know where that file is and if it is compiled or not ?
The job is still running "in progress"
I was also confused about this thread, as I was sure I had answered it, then had to again. Maybe the database got corrupted and restored?
I don't know what process runs the rsync itself. Like you, I've searched for likely suspects and found none. I've seen some references to where it was in OS4.x, but that didn't pan out for 6.x.
How much CPU time does TOP show you've used since the process started? I have a long one as well, and the CPU time is abominable -- usually under 10 minutes of CPU time over a 3-day job, most of which is skipping over existing files. With lots of leftover CPU time, I don't understand why rsync is not using more and completing the job more quickly. It's at priority 30, but that really shouldn't matter when the CPU is nearly idle. Network and CPU usage spike when there is a file to copy, but are nearly non-existent otherwise.
I finally killed the job last night when I was trying to get rsync working Duo > RN104 so I cannot look at how much CPU time it took for the job, sorry.
I did some more work this morning and found a number of files to look at, but it's hard work to try and figure out what references which file and variables when you aren't a programmer.
Knowing the backup was complete and accurate I set the RN104 rsync job running again this morning. It completed in just 13 seconds.
I don't quite believe that, but the log is copied below.
I have another 2TB disk to add to the array now, so will leave the RN104 alone until it has completed the expansion, so it will probably be a couple of days before I can give another update.
Thanks for the help so far :smileyhappy:
********
Backup Job Name: Sparenas media to Nas104 (all folders)
Backup Job Type: Incremental
Protocol: rsync
Backup Source: [remote:rsync]/172.16.0.52::media
Backup Destination: [media]/
Backup Start Time: Wed Mar 8 2017 8:36:28
Backup Finish Time: Wed Mar 8 2017 8:36:41
Backup Status: Success
sd9/sent 1,586 bytes received 168,735 bytes 12,616.37 bytes/sec
total size is 1,648,225,306,642 speedup is 9,677,170.21*******************************
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
