Very long file uploads using CIFS mount

[peku33]

Hi.

I use ReadyNas as IP camera storage.

I use dedicated software to transfer file from pc's hard-drive to storage.

Both readynas and hard-drives are pretty new.

Firmware version is 6.5.0 (updated a few days ago)

Files being transfered are about 20MB. There are ~10 files per minute to transfer (transfer is one by one, not paralell).

Computer and storage are connected with 1Gbps local network.

Most of time everything is ok, performance is superb, but:

Once every few recordings (maybe once every 5-10 minutes) transfer hangs. No data is transfered - mv command hangs. It usually takes up to 3 minutes to finalize transfer of single file.

During hangup stack trace of mv command is:

[<ffffffffa0f4d6df>] wait_for_free_request+0xcf/0x1b0 [cifs]
[<ffffffffa0f4e128>] cifs_call_async+0x48/0x1e0 [cifs]
[<ffffffffa0f2a7e8>] cifs_async_writev+0x1b8/0x2d0 [cifs]
[<ffffffffa0f43907>] cifs_writepages+0x547/0x7f0 [cifs]
[<ffffffff8114ef1e>] do_writepages+0x1e/0x30
[<ffffffff81143b2a>] __filemap_fdatawrite_range+0xaa/0xf0
[<ffffffff81143bec>] filemap_write_and_wait+0x2c/0x60
[<ffffffffa0f49a75>] cifs_setattr+0xb5/0x9d0 [cifs]
[<ffffffff811b77ed>] notify_change+0x20d/0x350
[<ffffffff811cd5d5>] utimes_common+0xc5/0x1b0
[<ffffffff811cd7cc>] do_utimes+0x10c/0x130
[<ffffffff811cd8b7>] SyS_utimensat+0x57/0x80
[<ffffffff8164b91b>] entry_SYSCALL_64_fastpath+0x16/0x6e

No other users access storage, antyvirus is off.

No messages in dmesg log.

What can cause the problem?

[peku33]

An update to my thread.

I logged into ReadyNAS via SSH, opened top, and i see, that

kworker/u2:7

 is consuming 100% CPU all the time

[JennC]

Hello peku33,

What's the model number of the NAS you have?

Can you also describe how the NAS and IP camera are setup in the network?

Regards,

[peku33]

It's ReadyNAS 102 with 2x WD 2GB drives.

8 ip cameras, PC computer and NAS are connected with 1Gbps switch

PC records video from cameras in 1-minute parts. They are stored in local temporary directory. After 10 parts are ready, they are transfered to NAS via linux mv command. NAS is mounted as CIFS. After transfer is complete, there is a cleanup routine that deletes 10 oldest files from NAS and so on.

I think that the major problem is the NAS uses btrfs that has to perform tree balancing after files beeing deleted.

However deleteing 10 files per minute shouldn't degradate performance to zero.

[JennC]

Hello peku33,

What apps are currently running on the NAS? Have you tried turning them off and see if there is any improvement?

Regards,

[peku33]

There are no apps running on NAS.

It's absolutely clean, just ssh + cifs enabled.

No apps, no other features, antyvirus is off.

I'm almost sure there is a problem with filesystem tree balancing.

[Retired_Member]

From your saying with temporary files qnd deleting, etc., your "issue" sounds like related to fragmentation / high metadata.
Do you copy-on-write enabled on the share? Do you have snapshots enable on the share? Did you ever run a balance? Did you ever run a defrag?
Can you post the result of the command "btrfs fi us /data"? (Replace /data with /<name_of_your_volume> if different)

[peku33]

root@nas:~# btrfs fi us /data
Overall:
    Device size:                   1.81TiB
    Device allocated:              1.81TiB
    Device unallocated:              0.00B
    Device missing:                  0.00B
    Used:                          1.80TiB
    Free (estimated):             11.22GiB      (min: 11.22GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.99
    Global reserve:               96.00MiB      (used: 0.00B)

Data,single: Size:1.81TiB, Used:1.80TiB
   /dev/md127      1.81TiB

Metadata,single: Size:8.00MiB, Used:0.00B
   /dev/md127      8.00MiB

Metadata,DUP: Size:1.00GiB, Used:221.53MiB
   /dev/md127      2.00GiB

System,single: Size:4.00MiB, Used:0.00B
   /dev/md127      4.00MiB

System,DUP: Size:8.00MiB, Used:256.00KiB
   /dev/md127     16.00MiB

Unallocated:
   /dev/md127        0.00B
root@nas:~#

[StephenB]

Your volume is completely full?

[peku33]

No, I always keep ~10G free space. This is done by the program from the first post.

[StephenB]

---

@peku33 wrote:

No, I always keep ~10G free space. This is done by the program from the first post.

---

So it is 99.5% full.  That's the problem I think.  If you keep the volume 80% full you should get much better performance.

[peku33]

Is there any technical explaination for that?

Loosing 20% (400GB!) of space just to keep the device running is radiculous.

[StephenB]

My personal experience says that you want at least 10% free space with any file system.  20% is Netgear's recommendation and I don't know all the reasons for it.

I do know that the BTRFS chunk size is 1 GB, and I suspect the file system is wanting to allocate a new chunk when your problem occurs (and doesn't have one).

[peku33]

Thank you for your reply.

Could you provide me a link to "Netgear's recommendation"?

[StephenB]

---

@peku33 wrote:

 Could you provide me a link to "Netgear's recommendation"?

---

Really?  You are somehow missing all those warnings in the log every time you restart your NAS?

I don't know if its in a knowledge base article or not, but it is routinely stated here.  For instance, mdgm's post on this thread: https://community.netgear.com/t5/ReadyNAS-Hardware-Compatibility/ReadyNas-3220-volume-degraded/td-p/...

[Retired_Member]

I personally never saw it written anywhere (except of course the warnings in the GUI)... but it's not the point, "NETGEAR recommends 20% free space" is a recommendation, not an obligation. That's probably why it's written not anywhere... to avoid confusion between recommandation and obligation.

That said, I understand this can look surprising. But if you consider the following, it actually makes sense:

It's common to ALL filesystems to require 10% free space to have optimum behavior, avoid issues and prevent too much fragmentation (of data and of free space).

BTRFS being a copy-on-write filesystem, it feels more confortable if there is a significant amount of free space. Unfortunately, 10GB is far from being enough.