× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: Writes fall to zero bytes after a few seconds

enkrypt3d
Tutor

Writes fall to zero bytes after a few seconds

Every time I try to copy a file to the NAS, it starts out fine and then the writes fall to 0 bytes and then eventually errors out... on the latest firmware. A reboot seems to fix the issue for a while but it comes back after a few weeks of uptime..... what gives? Load average is low... 

Model: RNDP4000|ReadyNAS Pro 4 Chassis only
Message 1 of 31
Retired_Member
Not applicable

Re: Writes fall to zero bytes after a few seconds

Hi @enkrypt3d , try disabling ipv6 in the nic config. Kind regards

Message 2 of 31
StephenB
Guru

Re: Writes fall to zero bytes after a few seconds


@enkrypt3d wrote:

... on the latest firmware. 


Is this 4.2.31?  Or something else?

 

Do you have any add-ons installed on the NAS?  Do you use ReadyDLNA?

Message 3 of 31
enkrypt3d
Tutor

Re: Writes fall to zero bytes after a few seconds

Yes running a Plex Server & Unifi controller and that's about it... and just the normal services that come with the readynas. Running OS 6.10.1... running a defrag right now which will take a long time - the odd thing is it doesn't show any logs that the defrag is running... weird.. 

 

The write performance issues happened even before I put plex on it.

Message 4 of 31
DEADDEADBEEF
Apprentice

Re: Writes fall to zero bytes after a few seconds

When you say a reboot resolves the issue temporarily, is that a reboot of NAS or computer?

 

Any chance you could pcap a failed file transfer? Ideally capturing the whole session setup, so close all windows explorer windows and delete all existing entries in 'net use', then start pcap -> open smb share -> start transfer -> once speed hits 0, wait 10 seconds (or until it errors out) and stop pcap. This should help narrowing down the issue.

 

If you want you can send the pcap to me in private as well.

 

 

Message 5 of 31
enkrypt3d
Tutor

Re: Writes fall to zero bytes after a few seconds

I ran a scrub which took almost a week to complete and it seems a bit better... I'll report back if I can reproduce the issue which typically happens after the nas has been up for more than 30 days or so...

Message 6 of 31
enkrypt3d
Tutor

Re: Writes fall to zero bytes after a few seconds

FYI rebooting the NAS seems to fix the issue for some time....

Message 7 of 31
StephenB
Guru

Re: Writes fall to zero bytes after a few seconds

Two things you might try:

  • disable ipv6 in the NAS
  • check the share settings, and if strict sync is enabled try disabling it.
Message 8 of 31
enkrypt3d
Tutor

Re: Writes fall to zero bytes after a few seconds

Just saw this issue pop up again - see screenshot.

 

Tried disabling ipv6 and strict sync and that had no change...

 

I think I found the issue though. Mongodb was running even though IO wait was not above a few %, it would stall out writes. I pruned the db and this has helped quite a bit. I'm planning on setting this prune job up as a schedule. 

Message 9 of 31
enkrypt3d
Tutor

Re: Writes fall to zero bytes after a few seconds

I completely killed plex & mongo and the issue persists... so what I have found that enabling SMB pre-allocation, this seems to have fixed the issue although the throughput isn't great. I used to get 110-112MB/s but now it hovers around 40-80MB/s..... better than it failing all the time I guess.... I also set the max SMB version to 3.1.1 which seems to have helped as well.

Message 10 of 31
StephenB
Guru

Re: Writes fall to zero bytes after a few seconds


@enkrypt3d wrote:

 so what I have found that enabling SMB pre-allocation, this seems to have fixed the issue although the throughput isn't great. I used to get 110-112MB/s but now it hovers around 40-80MB/s..... .


Interesting.  That points to something on-disk - volume fragmentation for example.  You could test that theory by measuring throughput locally in the NAS - eliminating the network (and SMB) from the equation.

 

How full is the volume?

Do you have snapshots enabled?

Has the volume ever been defragmented or balanced?

Is auto defragmentation enabled for the share?

Message 11 of 31
enkrypt3d
Tutor

Re: Writes fall to zero bytes after a few seconds

Capacity is at about 76% full...

 

I have run scrub and running a defrag on it now. I'm not sure how long it will take. I'm considering expanding the volume to 4 x 10TB disks or larger from the 4 x 8TB drives in it now... but that costs $ 🙂

Message 12 of 31
enkrypt3d
Tutor

Re: Writes fall to zero bytes after a few seconds

Trying to defrag this thing was a bad idea..... now i cant load anything on the admin panel... and I can't seem to abort it via ssh... how do I cancel the defrag? 

Message 13 of 31
StephenB
Guru

Re: Writes fall to zero bytes after a few seconds

Do you have snapshots enabled?

 


@enkrypt3d wrote:

Trying to defrag this thing was a bad idea..... now i cant load anything on the admin panel... and I can't seem to abort it via ssh... 


What is the CPU utilitization shown by top?

Are you seeing any disk related errors in ssh?

  • Journalctl gives you full access to the logs.
  • Smartctl -x will give you extended smart info on each drive - which includes some history of disk i/o errors.

 

 

Message 14 of 31
enkrypt3d
Tutor

Re: Writes fall to zero bytes after a few seconds

Seagate 14TB HDD Exos X14 7200 RPM 512e/4Kn SATA 6Gb/s 256MB Cache 3.5-Inch Enterprise Hard Drive (ST14000NM0018)

Looking at expanding with this... Would this work? I'm on 6.10.0 on my pro 4... Not sure if the 512e / 4kn would be compatible?
Message 15 of 31
StephenB
Guru

Re: Writes fall to zero bytes after a few seconds


@enkrypt3d wrote:
Looking at expanding with this... Would this work? I'm on 6.10.0 on my pro 4... Not sure if the 512e / 4kn would be compatible?

It's on the HCL for the OS-6 desktop NAS, so it should work.

Message 16 of 31
enkrypt3d
Tutor

Re: Writes fall to zero bytes after a few seconds

thanks I ended up getting these - it's currently expanding now on disk #2... 2 more disks to go....

 

off topic question - once the expansion is done can I pop these 4 x 8TB drives into a new readynas and it should fire right up as long as they're in order? Or has the data set changed after expansion on the original 8TB drives? Debating on buying a 2nd pro 4 or thinking about building a home brew NAS as a backup to this one with the new drives in it....

Message 17 of 31
Sandshark
Sensei

Re: Writes fall to zero bytes after a few seconds

Each time you insert a new drive and re-sync, the removed one gets out of sync with the others.  So, no, you can't do that.  The only way you could have is to put all the new ones in in place of the old, create a fresh volume, and then restore data from backup or the NAS in which you placed the old drives.

Message 18 of 31
enkrypt3d
Tutor

Re: Writes fall to zero bytes after a few seconds

one last question - why does it resync twice when expanding? When  I replaced the first drive, it only resyned once, but every drive after that it has resynced twice which takes days.. once when it says degraded then again when the volume is green and healthy.

 

I'm on drive 3 of 4 and on my 5th resync... it's a bit insane. I'm guessing the 4th drive will do the same thing . why?

Message 19 of 31
StephenB
Guru

Re: Writes fall to zero bytes after a few seconds


@enkrypt3d wrote:

one last question - why does it resync twice when expanding?


When you vertically expand, the NAS creates a second RAID group that uses the new space.  

 

The original RAID group needs to be resynced every time you upgrade a drive - reconstructing the contents of the original drive on the new one..  The new RAID group needs to be extended to use that drive (which is actually reshaping, but the NAS UI calles it resyncing).

Message 20 of 31
enkrypt3d
Tutor

Re: Writes fall to zero bytes after a few seconds

thanks for the input! last question - if I bought a newer readynas, I should be able to move these new 14TB drives over into the new one and it should work without an issue right? sorry for all the off topic quesitons.... just realized the pro 4 is only SATA II so now I'm thinking of getting a 2nd one that is a bit beefier...  and use the pro 4 as a backup...

Message 21 of 31
StephenB
Guru

Re: Writes fall to zero bytes after a few seconds


@enkrypt3d wrote:

 just realized the pro 4 is only SATA II so now I'm thinking of getting a 2nd one that is a bit beefier...  and use the pro 4 as a backup...


SATA II can handle 300 MB/sec - which is a lot faster than your disks (and your network).  So you won't see any performance gain with SATA III - SATA isn't the bottleneck.  Though of course the pro 4 is older, so you might consider repurposing it for backup anyway.

 

If you are looking for an upgrade, perhaps get a NAS with more slots, so you could add a couple of SSDs (either using ReadyTier or just creating a smaller SSD volume for selected shares). 

 

If you go with an RN520 or RN620 series you could also look into getting a multigig switch with at least one 10 gb port.  Then you could potentially upgrade a PC or two, and speed up your network.  Those switches are expensive though.

Message 22 of 31
enkrypt3d
Tutor

Re: Writes fall to zero bytes after a few seconds

Yes I have it running with 2 x 1Gbps in LACP but it never uses the 2nd link even with my PC in a LAG on the same switch... I was looking at the 

AR ReadyNAS 2304 as an upgrade but I didn't know if I could just swap the drives over and be all set.... 

 

The SATA speeds matter since I'm running things like plex, and other services that can thrash the drives local to the NAS... I'm going to test the throughput tonight with some bandwidth tests now that it's done syncing. I can tell a huge difference in plex performance over the old drives which were 5400... the new drives are 7200 RPM 🙂

Message 23 of 31
enkrypt3d
Tutor

Re: Writes fall to zero bytes after a few seconds

Still seeing the same writing performance... what a bummer... sometimes it falls to 0 bytes and other times it spikes to 45MB/s which is terrible... like I said I used to be able to write 112MB/s steady without issue... Already ruled out the switch and PC networking issues...... 

 

 

Message 24 of 31
StephenB
Guru

Re: Writes fall to zero bytes after a few seconds


@enkrypt3d wrote:

 

The SATA speeds matter since I'm running things like plex, and other services that can thrash the drives local to the NAS...


Mechanical disks are slower that 300 MB/s (SATA II).

 


@enkrypt3d wrote:

Yes I have it running with 2 x 1Gbps in LACP but it never uses the 2nd link even with my PC in a LAG on the same switch... )


Yes.  That's how LACP works.  Each flow is assigned to a NIC (depending on the xmit hash), so it won't split the flow across two links.

 

A static lag using round-robin in the NAS would use both NICs, but the packet loss might reduce your performance with other PCs.

Message 25 of 31
Top Contributors
Discussion stats
  • 30 replies
  • 3038 views
  • 0 kudos
  • 5 in conversation
Announcements