- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
Device starts dropping pings during backups
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Device starts dropping pings during backups
I have 2 Pro 6 business nas units. One is the central location for our vms and the other is for backups.
Firmware: 4.2.27
X-raid2
6 2tb drives
Under regular use and when idle pings are solid <1ms without a hiccup. But during intensive read/write operations (backup jobs), the device starts dropping pings. Once several pings in a row get dropped the jobs jail with Device Not Found type errors. The read/write operations are to CIFS shares. Backup jobs were from vmware vdr and vdp appliances and now from Veeam Backup & Replication
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Device starts dropping pings during backups
There is a 4.2.28 firmware release now available.
How many backups are you running at once? Have you staggered your backup jobs?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Device starts dropping pings during backups
Hello,
Just one backup. I run a ping logger.. everything is fine. I manually start a backup job... once the job begins transfering data pings start dropping. I may get 10-20 minutes into the job with one or two dropped pings every so many seconds but once it misses three in a row the backup job throws in the towel.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Device starts dropping pings during backups
Is this happening on both NAS?
Are you using NIC teaming?
Also, are you seeing errors in the ethernet stats on the two NAS?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Device starts dropping pings during backups
The other nas is running fine. But it also is not under the load of transferring hundreds of gigs worth of data plus whatever instructions go on during a backup job. I do not use nic teaming. I wanted to try that but having never done it before and because I only have the production environment to work with I am hesitant to experiment. Yes there are errors listed but the vm nas (not causing any disconnect issues) seems to have far more.
Backup NAS (drops pings under load of backup job)
TX packets | 3094012358 |
TX errors | 0 |
TX dropped | 0 |
TX overruns | 0 |
TX carrier | 0 |
RX packets | 2137332576 |
RX errors | 180867 |
RX dropped | 394264 |
RX overruns | 0 |
RX frame | 180867 |
Collisions | 0 |
VM NAS (not causing any noticable issues)
TX packets | 11032907002 |
TX errors | 0 |
TX dropped | 0 |
TX overruns | 0 |
TX carrier | 0 |
RX packets | 7154619993 |
RX errors | 4294967295 |
RX dropped | 3290316 |
RX overruns | 0 |
RX frame | 4294967295 |
Collisions | 0 |
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Device starts dropping pings during backups
I think you do need to resolve the high rx error rate on the VM NAS.
Just to clarify - these backups are being done from a third server? So data is being read from the VM NAS over the network, and then being written to the backup NAS over the network?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Device starts dropping pings during backups
Yes for the most part. I have a couple physical machines that are just backing up a few folders to the backup nas also. My personal physical pc is designated as the 'backup server' in veeam and the vm's from the vm nas are backed up to the backup nas. Before that a vm from vmware running on the vm nas was backing up the vms to the backup nas. We constantly had problems with it and decided to try a third party solution (veeam). For the most part it works but advanced functions like synthetic backups where it reads the incrementals from the backup nas and recreates the full will not work. The errors are always due to the backup nas disappearing from the network (ie: too many consecutive dropped pings)
How would I resolve high rx error rates?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Device starts dropping pings during backups
If you have a managed switch, then look at the stats there as well (and if possible look at the stats along the path between your desktop PC and the VM NAS, as well as between the VM server and the NAS).
If it is practical try changing the ethernet cable(s) connecting the VM NAS to the switch. They should be cat 5e or better.
Are you teaming the two NICs in the NAS? If so, what method are you using?
Also, are you running pingplotter from your desktop PC, or a different device?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Device starts dropping pings during backups
All switches on the network are Netgear GS716T
I am using premade cat6 cabling. I have spools of cat6 so I could definitely make up my own but I am doubtful that is the problem.
I am not using nic teaming.
I ran pingplotter on multiple machines cross checking to make sure when a device stops responding I could be sure which end was responsible.
There are no errors to speak of on the switch itself.
|
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Device starts dropping pings during backups
I did just notice that maximum frame size on the nas connected ports are 1518. Must have forgotten to set that to 9000 again when I replaced the switch a while ago in attempts to address these issues.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Device starts dropping pings during backups
@Rburn99 wrote:
I did just notice that maximum frame size on the nas connected ports are 1518. Must have forgotten to set that to 9000 again when I replaced the switch a while ago in attempts to address these issues.
Is this on the backup NAS or the VM NAS?
It does make sense to have everything use the same MTU, so maybe fix that first.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Device starts dropping pings during backups
Both.. all 16 ports on the switch were 1518. I have set the two nas box (vm and backups) to 9216. I don't want to run the backup during work as I don't want to field a bunch of why is my report taking so long phone calls 🙂 I don't know if this will fix things either though as I had swapped out the switch to address these issues (although at the time it was with vm's backup appliance periodically failing). And that switch I know I had set up with jumbo frames. Is there a way to reset the error count on nas boxes, obviously without doing factory resets 😛
And to my original questions.. Would maxing out the processing power of these readynas units be a likely cause for the units to stop responding to ping requests by connecting devices? And would maxing out the processing power to such an extent be a likely condition to begin with?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Device starts dropping pings during backups
The switch configuration is the ethernet (layer-2) max frame size, the NAS configuration is the IP payload (layer 3), which is smaller. The PC configuration is also usually the IP payload. So 1500 in the NAS corresponds to 1518 in the switch. If you use vlans, 1500 in the NAS corresponds to 1522 in the switch.
I'd set the NAS (and the PC and the VM server) to use 1500 for now. Alternatively you could set them to 9000 (with switch at 9022 or more).
I don't think you can clear the counts with 4.2.x firmware - what you can do is copy them to a spreadsheet, and then subtract.