- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
Re: Writes fall to zero bytes after a few seconds
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Writes fall to zero bytes after a few seconds
Every time I try to copy a file to the NAS, it starts out fine and then the writes fall to 0 bytes and then eventually errors out... on the latest firmware. A reboot seems to fix the issue for a while but it comes back after a few weeks of uptime..... what gives? Load average is low...
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
Hi @enkrypt3d , try disabling ipv6 in the nic config. Kind regards
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
@enkrypt3d wrote:
... on the latest firmware.
Is this 4.2.31? Or something else?
Do you have any add-ons installed on the NAS? Do you use ReadyDLNA?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
Yes running a Plex Server & Unifi controller and that's about it... and just the normal services that come with the readynas. Running OS 6.10.1... running a defrag right now which will take a long time - the odd thing is it doesn't show any logs that the defrag is running... weird..
The write performance issues happened even before I put plex on it.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
When you say a reboot resolves the issue temporarily, is that a reboot of NAS or computer?
Any chance you could pcap a failed file transfer? Ideally capturing the whole session setup, so close all windows explorer windows and delete all existing entries in 'net use', then start pcap -> open smb share -> start transfer -> once speed hits 0, wait 10 seconds (or until it errors out) and stop pcap. This should help narrowing down the issue.
If you want you can send the pcap to me in private as well.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
I ran a scrub which took almost a week to complete and it seems a bit better... I'll report back if I can reproduce the issue which typically happens after the nas has been up for more than 30 days or so...
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
FYI rebooting the NAS seems to fix the issue for some time....
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
Two things you might try:
- disable ipv6 in the NAS
- check the share settings, and if strict sync is enabled try disabling it.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
Just saw this issue pop up again - see screenshot.
Tried disabling ipv6 and strict sync and that had no change...
I think I found the issue though. Mongodb was running even though IO wait was not above a few %, it would stall out writes. I pruned the db and this has helped quite a bit. I'm planning on setting this prune job up as a schedule.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
I completely killed plex & mongo and the issue persists... so what I have found that enabling SMB pre-allocation, this seems to have fixed the issue although the throughput isn't great. I used to get 110-112MB/s but now it hovers around 40-80MB/s..... better than it failing all the time I guess.... I also set the max SMB version to 3.1.1 which seems to have helped as well.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
@enkrypt3d wrote:
so what I have found that enabling SMB pre-allocation, this seems to have fixed the issue although the throughput isn't great. I used to get 110-112MB/s but now it hovers around 40-80MB/s..... .
Interesting. That points to something on-disk - volume fragmentation for example. You could test that theory by measuring throughput locally in the NAS - eliminating the network (and SMB) from the equation.
How full is the volume?
Do you have snapshots enabled?
Has the volume ever been defragmented or balanced?
Is auto defragmentation enabled for the share?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
Capacity is at about 76% full...
I have run scrub and running a defrag on it now. I'm not sure how long it will take. I'm considering expanding the volume to 4 x 10TB disks or larger from the 4 x 8TB drives in it now... but that costs $ 🙂
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
Trying to defrag this thing was a bad idea..... now i cant load anything on the admin panel... and I can't seem to abort it via ssh... how do I cancel the defrag?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
Do you have snapshots enabled?
@enkrypt3d wrote:
Trying to defrag this thing was a bad idea..... now i cant load anything on the admin panel... and I can't seem to abort it via ssh...
What is the CPU utilitization shown by top?
Are you seeing any disk related errors in ssh?
- Journalctl gives you full access to the logs.
- Smartctl -x will give you extended smart info on each drive - which includes some history of disk i/o errors.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
Looking at expanding with this... Would this work? I'm on 6.10.0 on my pro 4... Not sure if the 512e / 4kn would be compatible?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
@enkrypt3d wrote:
Looking at expanding with this... Would this work? I'm on 6.10.0 on my pro 4... Not sure if the 512e / 4kn would be compatible?
It's on the HCL for the OS-6 desktop NAS, so it should work.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
thanks I ended up getting these - it's currently expanding now on disk #2... 2 more disks to go....
off topic question - once the expansion is done can I pop these 4 x 8TB drives into a new readynas and it should fire right up as long as they're in order? Or has the data set changed after expansion on the original 8TB drives? Debating on buying a 2nd pro 4 or thinking about building a home brew NAS as a backup to this one with the new drives in it....
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
Each time you insert a new drive and re-sync, the removed one gets out of sync with the others. So, no, you can't do that. The only way you could have is to put all the new ones in in place of the old, create a fresh volume, and then restore data from backup or the NAS in which you placed the old drives.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
one last question - why does it resync twice when expanding? When I replaced the first drive, it only resyned once, but every drive after that it has resynced twice which takes days.. once when it says degraded then again when the volume is green and healthy.
I'm on drive 3 of 4 and on my 5th resync... it's a bit insane. I'm guessing the 4th drive will do the same thing . why?
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
@enkrypt3d wrote:
one last question - why does it resync twice when expanding?
When you vertically expand, the NAS creates a second RAID group that uses the new space.
The original RAID group needs to be resynced every time you upgrade a drive - reconstructing the contents of the original drive on the new one.. The new RAID group needs to be extended to use that drive (which is actually reshaping, but the NAS UI calles it resyncing).
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
thanks for the input! last question - if I bought a newer readynas, I should be able to move these new 14TB drives over into the new one and it should work without an issue right? sorry for all the off topic quesitons.... just realized the pro 4 is only SATA II so now I'm thinking of getting a 2nd one that is a bit beefier... and use the pro 4 as a backup...
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
@enkrypt3d wrote:
just realized the pro 4 is only SATA II so now I'm thinking of getting a 2nd one that is a bit beefier... and use the pro 4 as a backup...
SATA II can handle 300 MB/sec - which is a lot faster than your disks (and your network). So you won't see any performance gain with SATA III - SATA isn't the bottleneck. Though of course the pro 4 is older, so you might consider repurposing it for backup anyway.
If you are looking for an upgrade, perhaps get a NAS with more slots, so you could add a couple of SSDs (either using ReadyTier or just creating a smaller SSD volume for selected shares).
If you go with an RN520 or RN620 series you could also look into getting a multigig switch with at least one 10 gb port. Then you could potentially upgrade a PC or two, and speed up your network. Those switches are expensive though.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
Yes I have it running with 2 x 1Gbps in LACP but it never uses the 2nd link even with my PC in a LAG on the same switch... I was looking at the
AR ReadyNAS 2304 as an upgrade but I didn't know if I could just swap the drives over and be all set....
The SATA speeds matter since I'm running things like plex, and other services that can thrash the drives local to the NAS... I'm going to test the throughput tonight with some bandwidth tests now that it's done syncing. I can tell a huge difference in plex performance over the old drives which were 5400... the new drives are 7200 RPM 🙂
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
Still seeing the same writing performance... what a bummer... sometimes it falls to 0 bytes and other times it spikes to 45MB/s which is terrible... like I said I used to be able to write 112MB/s steady without issue... Already ruled out the switch and PC networking issues......
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Re: Writes fall to zero bytes after a few seconds
@enkrypt3d wrote:
The SATA speeds matter since I'm running things like plex, and other services that can thrash the drives local to the NAS...
Mechanical disks are slower that 300 MB/s (SATA II).
@enkrypt3d wrote:
Yes I have it running with 2 x 1Gbps in LACP but it never uses the 2nd link even with my PC in a LAG on the same switch... )
Yes. That's how LACP works. Each flow is assigned to a NIC (depending on the xmit hash), so it won't split the flow across two links.
A static lag using round-robin in the NAS would use both NICs, but the packet loss might reduce your performance with other PCs.