This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
RN104 Resync Speed on 3rd Drive Replacement Very Slow
I am updating my RN104 from 4 8Tb drives to 4 10Tb drives. After replacing the first drive, the resync took a couple of days to complete. After replacing the second drive, about the same length of time. Now, I have replaced the third drive and the resync is going very, very slowly. After two days it is only at 53.7% and the estimate says it has 1240 hours to go. Is this normal? When I have updated drives in the past, each drive took about the same time to resync. This is the first time I can remember one drive taking so much longer than others.
Re: RN104 Resync Speed on 3rd Drive Replacement Very Slow
I rebooted the NAS and the resync speed increased to about 14 Mb/sec. I don't know what was causing the slow down before.
I do have backup jobs running Rsync. This particular NAS does not have any backups configured on it, but it is the NAS that recieves backups from another NAS.
After two days it is only at 53.7% and the estimate says it has 1240 hours to go. Is this normal?
That is unusually slow. Is the NAS doing something else (backup jobs, scheduled maintenance activities, or under a heavier usage then other times)? Resync is a lower priority task than other things, so it will run slower when there is more going on.
I suggest downloading the log zip file, and looking for disk errors in system.log and kernel.log
If ssh is enabled, you can get a better sense of the progress with cat /proc/mdstat and query the smart stats interactively with smartctl.
Re: RN104 Resync Speed on 3rd Drive Replacement Very Slow
No backups running at this time although there were some running during the sync so maybe that was slowing things some.
In the kernel.log I see a large number of ethernet errors:
May 13 00:36:55 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=512
May 13 00:36:55 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=896 May 13 00:39:03 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=448 May 13 00:39:04 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=448 May 13 00:39:04 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=640 May 13 00:39:09 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=896 May 13 00:39:10 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=448 May 13 00:39:10 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=640 May 13 00:39:11 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=448 May 13 00:39:12 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=640 May 13 00:39:12 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=448 May 13 00:39:12 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=768 May 13 00:39:13 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=448 May 13 00:39:13 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=1024 May 13 00:39:14 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=640 May 13 00:39:14 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=640 May 13 00:39:15 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=144 May 13 00:39:17 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=448 May 13 00:39:25 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=640 May 13 00:39:32 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=640 May 13 00:39:37 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=144 May 13 00:39:37 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=640 May 13 00:39:38 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=504 May 13 00:39:39 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=640 May 13 00:39:40 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=1024 May 13 00:39:40 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=1280 May 13 00:39:40 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=768 May 13 00:39:42 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0d830000 (overrun error), size=16 May 13 00:39:43 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=640 May 13 00:39:48 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=896 May 13 00:39:48 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=768 May 13 00:39:54 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=272 May 13 00:39:57 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=448 May 13 00:40:07 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=400 May 13 00:40:10 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=448 May 13 00:40:18 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=448 May 13 00:40:20 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0d830000 (overrun error), size=16 May 13 00:40:32 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=448 May 13 00:40:34 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=1152 May 13 00:40:37 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=272 May 13 00:40:47 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=896 May 13 00:40:57 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=448 May 13 00:40:57 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=440 May 13 00:40:58 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=1024 May 13 00:41:04 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=768 May 13 00:41:17 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=640 May 13 00:41:20 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=448 May 13 00:41:20 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=640 May 13 00:41:21 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=896 May 13 00:41:26 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=1024 May 13 00:41:28 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=1024 May 13 00:41:29 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=448 May 13 00:41:29 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=896 May 13 11:43:56 Media-Backup kernel: nr_pdflush_threads exported in /proc is scheduled for removal
I wouldn't think this would impact the sync speed, though.
In the system.log I see a number of failed log in attempts which is strange. Some of them say the attempt is coming from 192.168.1.1 which would be the gateway server, correct?
May 12 21:46:38 Media-Backup sshd[3419]: Invalid user Retford from 192.168.1.1 May 12 21:46:38 Media-Backup sshd[3419]: input_userauth_request: invalid user Retford [preauth] May 12 21:46:38 Media-Backup sshd[3419]: pam_unix(sshd:auth): check pass; user unknown May 12 21:46:38 Media-Backup sshd[3419]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=gateway May 12 21:46:38 Media-Backup sshd[3420]: SSH: Server;Ltype: Authname;Remote: 192.168.1.1-59610;Name: root [preauth] May 12 21:46:38 Media-Backup sshd[3420]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=gateway user=root May 12 21:46:39 Media-Backup sshd[3421]: SSH: Server;Ltype: Authname;Remote: 192.168.1.1-59611;Name: [preauth] May 12 21:46:39 Media-Backup sshd[3421]: Invalid user from 192.168.1.1 May 12 21:46:39 Media-Backup sshd[3421]: input_userauth_request: invalid user [preauth] May 12 21:46:39 Media-Backup sshd[3421]: Failed none for invalid user from 192.168.1.1 port 59611 ssh2 May 12 21:46:39 Media-Backup sshd[3421]: Connection closed by 192.168.1.1 [preauth]
Re: RN104 Resync Speed on 3rd Drive Replacement Very Slow
You have a multi-layer volume due to incremental drive size upgrades. Each layer is it's own RAID, and it can only sync one at a time. So the delayed one is waiting for the another to complete. The multiple layers may be contributing to the slower speed.
May 13 00:36:55 Media-Backup kernel: mvneta d0070000.ethernet eth0: bad rx status 0f830000 (overrun error), size=512
This is likely a consequence of the slowdown - the system isn't keeping up.
@Ken6432-1 wrote: May 12 21:46:38 Media-Backup sshd[3419]: Invalid user Retford from 192.168.1.1
This is a little scary. Do you have ports forwarded to the NAS in your router? OR perhaps put it in the router's DMZ? If you have, then you should disable the forwarding (or put the NAS out of the DMZ).
This is extremely slow (1 to 2 MB/s). Did you look at the smart stats with smartctl (smartctl -x /dev/sda (repeat for the other disks).
I'm not sure what I am looking for. Nothing jumps out to me on /dev/sda, but that appears to be the 8Tb drive in the rightmost drive location. What would the address of the drive that is the third from the left? That is the one that was just changed.
@Ken6432-1 wrote: May 12 21:46:38 Media-Backup sshd[3419]: Invalid user Retford from 192.168.1.1
This is a little scary. Do you have ports forwarded to the NAS in your router? OR perhaps put it in the router's DMZ? If you have, then you should disable the forwarding (or put the NAS out of the DMZ).
I have not set anything on my router that would have to do with any of my NAS's. In fact, I don't remember setting anything except the basic out-of-the-box settings. I agree it is concerning. I updated my passwords to be safe.
This is extremely slow (1 to 2 MB/s). Did you look at the smart stats with smartctl (smartctl -x /dev/sda (repeat for the other disks).
Is there something in particular I should be looking for? All four channels look similar to me. Nothing stands out on sdb as compared to the others.
@Ken6432-1 wrote: May 12 21:46:38 Media-Backup sshd[3419]: Invalid user Retford from 192.168.1.1
This is a little scary. Do you have ports forwarded to the NAS in your router? OR perhaps put it in the router's DMZ? If you have, then you should disable the forwarding (or put the NAS out of the DMZ).
I have not made any changes on the router specific to the NASs. In fact I think they are set pretty much out-of-the-box. I updated my passwords to be on the safe side. I did see a similar entry on one of the other NASs that I checked.
This is extremely slow (1 to 2 MB/s). Did you look at the smart stats with smartctl (smartctl -x /dev/sda (repeat for the other disks).
@Ken6432-1 wrote: May 12 21:46:38 Media-Backup sshd[3419]: Invalid user Retford from 192.168.1.1
This is a little scary. Do you have ports forwarded to the NAS in your router? OR perhaps put it in the router's DMZ? If you have, then you should disable the forwarding (or put the NAS out of the DMZ).
What should I look for in the smart stats? I don't see anything noticably different between the output for sdb (the drive most recently changed) as compared to the other three.
I have done nothing on the router with respect to the NASs. In fact, I think the settings are pretty much out of the box. I see and entry similar to the one above on one of my other NASs that I checked. Very concerning. I updated my passwords to be on the safe side.
This is extremely slow (1 to 2 MB/s). Did you look at the smart stats with smartctl (smartctl -x /dev/sda (repeat for the other disks).
What sort of things am I looking for in the output? The information for sdb does not seem any different than the information for the other three locations.
@Ken6432-1 wrote: May 12 21:46:38 Media-Backup sshd[3419]: Invalid user Retford from 192.168.1.1
This is a little scary. Do you have ports forwarded to the NAS in your router? OR perhaps put it in the router's DMZ? If you have, then you should disable the forwarding (or put the NAS out of the DMZ).
I have not made any settings on the router for my NASs. In fact, as far as I can tell, the router is set up pretty much with out of the box settings. I don't have any port forwarding or anything like that set. I updated my passwords on the NASs just to be sure.
I'm not sure what I am looking for. Nothing jumps out to me on /dev/sda, but that appears to be the 8Tb drive in the rightmost drive location. What would the address of the drive that is the third from the left? That is the one that was just changed.
Drive letters are normally sda->first slot; sdd-> fourth slot. But when you are upgrading disks, the system can assign letters differently.
One concern here is whether one of the drives (not necessarily the newest one) is failing. The sync speed will vary, but normally it'd be at least 30 MB/second - not 1.3 MB/sec.
If you are struggling understanding the smart stats, then perhaps instead download the full log zip file, and look for disk errors in system.log and kernel.log. Searching for sda, sdb, sdc, sdd would help locate them.
Re: RN104 Resync Speed on 3rd Drive Replacement Very Slow
I rebooted the NAS and the resync speed increased to about 14 Mb/sec. I don't know what was causing the slow down before.
I do have backup jobs running Rsync. This particular NAS does not have any backups configured on it, but it is the NAS that recieves backups from another NAS.
I do have backup jobs running Rsync. This particular NAS does not have any backups configured on it, but it is the NAS that recieves backups from another NAS.
Just rsync? Or rsync-over-ssh?
Is the other NAS on the local network? Or is it remote?
Re: RN104 Resync Speed on 3rd Drive Replacement Very Slow
My backups are Rsync Server mode. All NASs are on the local network, none are remote. I have SSH configured on as that is how I log in via PuTTY. That is how I was running the cat command.
I have rechecked my router and have no ports forwarding at all listed.
My backups are Rsync Server mode. All NASs are on the local network, none are remote. I have SSH configured on as that is how I log in via PuTTY. That is how I was running the cat command.
I have rechecked my router and have no ports forwarding at all listed.
Ok. So someone tried to use ssh to access the NAS, and that appeared to come from the router. Normally that means it was coming over the internet - which would require port forwarding.
If there are no router ports forwarded, and the NAS isn't listed in the router DMZ, then one other possibility is that uPNP is being used to open ports. So you could try turning uPNP off in your router. What router do you have? Is it's firmware up to date?
If the NAS is in fact compromised, you'd want to a factory default, and set it up fresh. That would require restoring the files from backup. I wouldn't try to save/restore configuration files, since that might propagate the problem.
My backups are Rsync Server mode. All NASs are on the local network, none are remote. I have SSH configured on as that is how I log in via PuTTY. That is how I was running the cat command.
I have rechecked my router and have no ports forwarding at all listed.
Ok. So someone tried to use ssh to access the NAS, and that appeared to come from the router. Normally that means it was coming over the internet - which would require port forwarding.
If there are no router ports forwarded, and the NAS isn't listed in the router DMZ, then one other possibility is that uPNP is being used to open ports. So you could try turning uPNP off in your router. What router do you have? Is it's firmware up to date?
If the NAS is in fact compromised, you'd want to a factory default, and set it up fresh. That would require restoring the files from backup. I wouldn't try to save/restore configuration files, since that might propagate the problem.
I have a Netgear Orbi with the firmware up to date. It defaults to UPnP on, though no ports are shown on that page as being forwarded/opened. I will try turning UPnP off. Hopefully that does not cause any problems with any of my other current devices.
I don't see anything that shows that a login was successfull, just attempts. I am hoping that means the NASs have not be compromised. I would hate to have to restore from backup on all four NASs.
I don't see anything that shows that a login was successfull, just attempts.
Which is of course good. But how the failed login requests are reaching the NAS isn't explained.
If I turn off SSH when I don't need it, would that help keep the NASs safe? I could turn it back on if I need to log in with PuTTY, then back off again.
Re: RN104 Resync Speed on 3rd Drive Replacement Very Slow
@Ken6432-1 wrote: If I turn off SSH when I don't need it, would that help keep the NASs safe? I could turn it back on if I need to log in with PuTTY, then back off again.
Probably would help - though it would get in the way if you wanted to troubleshoot the NAS if the web ui was non-responsive.
Also- when you said you changed your passwords, did you include your router password?
@Ken6432-1 wrote: If I turn off SSH when I don't need it, would that help keep the NASs safe? I could turn it back on if I need to log in with PuTTY, then back off again.
Probably would help - though it would get in the way if you wanted to troubleshoot the NAS if the web ui was non-responsive.
Also- when you said you changed your passwords, did you include your router password?
Good point on the downside of turning off SSH. I'll have to think about that a bit.
Yes, I changed my router password as well. I turned off UPnP on the router now. I'm going to monitor the NAS log files a bit closer now (as well as my firewall logs) and see if there are still login attempts.
