ReadyNAS RN42200 hangs completely up during backup to RN31400

[Worli]

Hi everyone, I am trying now since already a year to do backups from my newer RN42200 (two 12TB Disks in RAID 1) to my older RN31400 (4 Disks, RAID 5). Both are on OS 6.10.3. I tried Backups via SMB, NFS, RSYNC, RSYNC over SSH....... The backup always crashed. It was also not possible to cancel the Backup-Job. After a longer time it always goes back from "cancelling" to "running". Only way is then to push the Power-Off button on the box.

Sometimes single files or Directories arrived at the RN31400. Finally it led to a file with ZERO bytes. E.g. a Backup from one of my Debian-Servers to the boxes is absolutely no problem. Also RSync works.

What I found today during another (unsuccessful) backup-test:

•  Backup Job didnt work.
•  Tried to cancel the job but after a while it went back to "running". BUT
•  The two disks went to sleeping mode.
•  Later on I connected via Terminal and a "ls -la" found also a file in the root "job_008" with permissions "d?????????", Group: "?", User "?", Size "?" and the message "ls: cannot access 'job_008': Host is down'.

Does somebody have an idea here?

Greetings, Worli

[Worli · Accepted solution]

Hi Stephen, I think the MTU size (at 9000) was the root cause! After changing it on both boxes back to 1500 and a reboot everything worked. I successfully tried via NFS, SMB and Rsync.

But first I had to delete the existing backup-jobs and had to create new ones. The old ones were not working! Whyever...

Wow, this is really absolutely unexpected. Strange, because from all other devices on the network it was always working! No matter which OS, Windows as well as Linux.

However, many thanks Stephen for your help and especially also for your patience which I really appreciate. May be I get ometimes to opportunity to help you too!!!

[StephenB]

Are the NAS on the same network?

I run daily backups - one for each share.  These are all set up for rsync.

Can you provide more information on the backup settings you are using?  One thing I suggest is using the IP address for the remote NAS - not the host name.

[Worli]

Thank you, yes they are both on same net. Static-IP: 192.168.1.8 and 192.168.1.20. Initially I had also an both of them the second NIC active in VLAN 192.168.50.0. But I deactivated them in the meantime.

I did a lot of tries. Also tried the Hostname and/or IP-Address. No change. I always deleted the faulty backup jobs after the tries. But to take the simple one (SMB):

• Created on both boxes new Share.
• I just copied a few files for testing into the source. (Via Webinterface)
• Only SMB active.
• No Snapshots, nothing else.
• User rights: Everbody: Read/Write.
• Source is Local. Target is Remote.
• No Schedule. Only one time run.
• Via Hostname: I searched for the other box via Admin/Password and it was found: NAS3-Worli.local. So chech of connection was always successful.
• Via IP: I entered it manually.
• Only when I tried to see the Shares on the Abkup-Box I didnt see one of the about 20 Shares.

Sometimes the Backup Job created several empty directories on the Back-Box and started in one of the Dirs with a file but it had always ZERO byte and stopped.

May be somethnig with the file permissions? (I did also Reset to Default on the Share.)

Kind regards, Worli

[StephenB]

Personally I use Rsync.  I run the backups on the destination NAS, and as noted earlier I use one backup job per share.

The settings I use are:

I use these on several NAS (running daily backups), and they run very reliably.

Can you tell us more about your network setup?

[Worli]

Hi Steven, Thank you. I tried now as you advised me to do. Negative. Tried to backup the share musik as in your example. Of course Rsync is activated on Source and I also added the IP of Target as allowed for Rsync. There was already the address of my Debian 10 Server in. With the Debian via Rsync it was always working fine.....

Also to say that before I got my RN4220 I had a ReadyNAS NVX and I did successfully Rsync-Backup from my RN31400 to the NVX!

On Network side I use four Ubiquity Switches, Unifi USG Secure Gateway and an Unifi Access Point. So nothing special, all locally in my appartment. Additionally a VLAN 50 for my Nextcloud Server which can be connected via Internet. The RN4220 is physically connected to first Switch after the USG. The RN31400 is on the last Switch. In the past I had both ReadyNAS connected to both VLAN (1+50). But during my tests I deactivated the second NIC on the ReadyNAS.

[Worli]

Tomorrow I will also connect both ReadyNAS to the same switch and see what happens. 

[Worli]

Hi Stephen, both NAS-boxes are now on the same Switch. Only one NIC active on both NAS-boxes.

But no change. Backup not successful. After 60 minutes:

• Rsync Backup terminated on Target-NAS with “Error during copying.”
• Target-Directory is still empty.
• Already a longer time before the backup job terminated the disks on Source-NAS went to „sleeping“, as configured. At this stage Backup-Job was still in status “Running” on Target-NAS.

Protocol of Backup-Job. (Shall I translate it?)

*-------- Informationen zum Sicherungsauftrag --------
Name des Sicherungsauftrags: job_001
Typ des Sicherungsauftrags: Inkrementell
Protokoll: rsync
Sicherungsquelle: [remote:rsync]/192.168.1.20::musik
Sicherungsziel: [temp]/
Startzeit der Sicherung: Do Apr 23 2020 8:56:35
Endzeit der Sicherung: Do Apr 23 2020 9:56:43
Sicherungsstatus: Fehlgeschlagen: Fehler während des Kopiervorgangs.

[StephenB]

The log looks like a normal backup job to me, so that part seems fine.

Are the file permissions set in Musik and Temp to allow full access? 

Have you tried setting up a test share on the source NAS that uses default settings, then enable rsync for it?  Then put a couple files into it for testing purposes?

[Worli]

Hi Stephen, yes, I created on both boxes a Directory called "temp" added a few files. I think in the meantime I did dozens of tests.  In the meantime I am somehow lost.  Will prepare screenshots with the current File-permissions.

What irritates me most is that this issues exists since the vary beginning. So also with the brand new installed box it wasnt working.

Thanks.

[Sandshark]

Make sure rsync is configured for read/write on the target NAS.  It's not by default.  And "Test Connection" does not check for that.

[StephenB]

Your symptoms are unusual, this is something that works literally every day for me on 5 ReadyNAS.

I do suggest starting over with a clean temp share on both machines.  Enable rsync as read/write on the both the source and destination shares to start with.  Maybe start with just one file, to keep things really simple.

What apps have you installed on the NAS?  Have you ever seen any warnings that the OS partition might be full?

[Worli]

Hi Stephen, today afternoon I created on both NAS a fresh new share „temp2“ and copied for testing only ten files into it, Rsync on both, read/write on both, etc. No change.

Never got errors that systen disk is full. I checked frequently. „df“ currently shows 34% usage.

[Worli]

Hi Stephen, this morning I started another try. Created a new admin key on both boxes. Created again new shares, enabled on both sides Rsync, on both boxes Read/Write under Rsync. System disk /dev/md0 reports 33% usage. (also during the first 25 min where the backup was running.)

I also saw on Source under System, Protocols, Connections that user "root" is connected via Rsync with the correct IP on the Target-Box.

Backup job is running now since already 25 min. Target-Dir still empty.

[Worli]

I waited up now for the end of the (unsuccessful) backup job. Can see now at the bottom line of protocol some more infornation.

Name des Sicherungsauftrags: job_001
Typ des Sicherungsauftrags: Inkrementell
Protokoll: rsync
Sicherungsquelle: [remote:rsync]/192.168.1.20::temp2
Sicherungsziel: [temp2]/
Startzeit der Sicherung: Fr Apr 24 2020 8:49:05
Endzeit der Sicherung: Fr Apr 24 2020 9:49:12
Sicherungsstatus: Fehlgeschlagen: Fehler während des Kopiervorgangs.

Failed to run rsync, return rc 30

[Receiver] io timeout after 1802 seconds -- exiting
rsync error: timeout in data send/receive (code 30) at io.c(204) [Receiver=3.1.3]

[StephenB]

---

@Worli wrote:

Hi Stephen, this morning I started another try. Created a new admin key on both boxes.

---

Do you mean an ssh key? Or do you mean the admin password?

I've never created an ssh key for admin on my own NAS, so that might be part of the puzzle.

[Worli]

Hi Stephen, it was not a SSH-Key but a new user which is in Admin-Group on the box. So for the connection via Rsync (in the backup job) I used this new admin-key instead of the standard-"admin". Just another try.

[StephenB]

---

@Worli wrote:

Hi Stephen, it was not a SSH-Key but a new user which is in Admin-Group on the box. So for the connection via Rsync (in the backup job) I used this new admin-key instead of the standard-"admin". Just another try.

---

Do you have password protection enabled on the remote share (for rsync)?  If you do, try disabling that setting.

If you have experience with the linux command line, you might also want to try accessing the NAS hosting the backup job with ssh,and try using rsync manually. That might give you some more clues.

Another test that might be useful:  you can "fake" the network connection by using 127.0.0.1 as the remote IP address.  Then you can try setting up two shares on the same NAS, and test the rsync part alone (taking the network out of the equation).  This can be done on both NAS.

[Worli]

Hi, Stephen, no I never activated never via Rsync Password (on the share, under Rsync). Double-checked, not activated.

May a good try to fin more out via ssh/rsync....

[StephenB]

---

@Worli wrote:

Hi, Stephen, no I never activated never via Rsync Password (on the share, under Rsync). Double-checked, not activated.

---

Then you shouldn't need a username/password in the backup job at all.  Though it normally does no harm to include it anyway.

---

@Worli wrote:

May a good try to fin more out via ssh/rsync....

---

When you try that, you will need to override the default behavior (which is to run rsync over ssh).  You to that by prepending "rsync:" to the remote path.

Still, I'd try the 127.0.0.1 tests first.  That will rule out the network, and it might also pinpoint one NAS that is misbehaving.

[Worli]

Stephen, fune, will do the 127.0.0.1 first.

[Worli]

Stephen, on both boxes the 127.0.0.1-test was successful. I Rsynced locally from Share Music to Share temp2. 15GB within a minimum of time.

[StephenB]

---

@Worli wrote:

Stephen, on both boxes the 127.0.0.1-test was successful. I Rsynced locally from Share Music to Share temp2. 15GB within a minimum of time.

---

That suggests there might be something going on with your network.  Anything on the path that might be blocking ports?  Are you using the standard ethernet MTU, or have you configured jumbo frames?

How comfortable are you with ssh?

[Worli]

Hi Stephen, since everything runs "internally" I do not block amything on my LAN. The MTU is set on both boxes to 9000.
Today I tried to manually (via SSH on the Target box) the shares from Soruce-Box. I tried via CIFS and alternatively via NFS:

1. Created a subdir /mnt/test
2. Mount via nfs (mount -t nfs Source-IP:/data/musik /mnt/test) failed with mount.nfs: access denied by server while mounting 192.168.1.20:/data/musik
3. Mount via CIFS (mount -t cifs -o user=nede,domain=NAS4-worli //Source-IP/musik /mnt/test) asked me correctly to Enter the Password for the user and finshed. But when I go to /mnt/test and do a "ls" nothing comes. After a couple of minutes I cancelled with Ctrl+C.

When I do same from my Debian Server (is on same Switch as the Target-Box) everything is working fine. NFS as well as CIFS. I didnt look into the NIC config of Debuan. But I am quite sure the  am not on Jumbo Frames with the MTU.

Will change this now back on the boxes to the default MTU and will try again.

[StephenB]

---

@Worli wrote:

Will change this now back on the boxes to the default MTU and will try again.

---

I agree that's the next step. 

[Worli · Accepted solution]

Hi Stephen, I think the MTU size (at 9000) was the root cause! After changing it on both boxes back to 1500 and a reboot everything worked. I successfully tried via NFS, SMB and Rsync.

But first I had to delete the existing backup-jobs and had to create new ones. The old ones were not working! Whyever...

Wow, this is really absolutely unexpected. Strange, because from all other devices on the network it was always working! No matter which OS, Windows as well as Linux.

However, many thanks Stephen for your help and especially also for your patience which I really appreciate. May be I get ometimes to opportunity to help you too!!!