I'm running 6.1.6 final - I STARTED with a fresh install and rebuild on a 6.1.6RC I had all my data offloaded; I started copying it back. I've since updated to 6.1.6 final. I am now copying it back. As I got into the 70% range, I started to have performance problems.
I would see some btrfs processes spiking CPU; and top reports most CPU usage going to 'wait' times. Load increases to 30+ and the system becomes unresponsive. Data copies to the system crawl to a halt. I stop transfers and the system eventually recovers (sometimes I have to reboot it). So long as I don't add more data; it works great -- but all efforts to add additional data seems to create problems.
I'm quite sure my btrfs block size should be the new; larger size -- but is there a command I can run to confirm? If I'm running the larger block size; it would appear there's still a major problem here!
Open to suggestions of what to do! I'd really like to use the last 3TB of my NAS! /dev/md127 11T 8.1T 2.9T 74% /data
I'm quite sure I upgraded to 6.1.6 final... but open to trying again if that does not appear to be the case.
no iSCSI
I have plex app installed
I use SMB, FTP, AFP, DLNA
I've managed to get to 80% usage; but my CPU usage spiked so high and never returned for hours -- I couldn't execute any commands and had to hard restart.
/dev/md127 11T 8.7T 2.3T 80% /data
mdgm wrote:
Are you sure that's the leafsize for /dev/md127 not /dev/md0. Can you PM the full contents of btrfs.log
In the logs what does os_version.log contain?
From your initrd.log it looks like you are still running a RC?
I can upgrade to final (but I'll wait to hear from you that I should since we're in the middle of troubleshooting)
Plex is often running and scanning -- but nothing too crazy.
I was just able to get it to almost lock up with a huge load due to lftp mirroring a directory to the server -- transfer started at 60MB/sec -- after a few seconds it slowed to kbs (halted, really) -- and I was SSH'd in -- wouldn't even execute my 'w' command to see the load. 😕
After about 10 minutes the load finally dropped below 20 and it executed I ran top to see a process md127_raid5 using a lot of CPU
Repeating the transfer with top running shows the following competing for CPU: proftpd md127_raid5 flush-btrfs-2 kswapd0
Load is holding around 5
Transfer going 30-35MB/sec
And now after a few minutes it locked up again. Here's what top looks like: https://www.dropbox.com/s/08bus9zeiss1lds/Screenshot%202014-02-17%2001.30.32.png Note the 99.5% wait Then it finally refreshed 1 time a few minutes later: https://www.dropbox.com/s/30r4ubjlxp9abzf/Screenshot%202014-02-17%2001.32.14.png
Many more minutes later as it finally recovered, it looked like: https://www.dropbox.com/s/k9u5xsgy65rr637/Screenshot%202014-02-17%2001.35.14.png (note the drastically reduced wait time, sky-high load which is dropping)
Well that's embarrassing.... I updated again to 6.1.6 final, and then when I went to look at the logs, realized I'd been reading out of an old download of the logs before.
I definitely was already on the latest: ReadyNASOS!!version=6.1.6,time=1391111179,arch=x86_64
[2014/02/02 22:37:04] Factory default initiated by Frontview! [2014/02/02 22:37:35] Defaulting to X-RAID2 mode, RAID level 5 [2014/02/02 22:37:58] Factory default initiated on ReadyNASOS 6.1.6. [2014/02/09 00:03:34] Updated from ReadyNASOS 6.1.6 to 6.1.6. [2014/02/17 00:23:18] Updated from ReadyNASOS 6.1.6 to 6.1.6.
Was able to immediately get it to do it again: https://www.dropbox.com/s/ztvaeatfmrhngvk/Screenshot%202014-02-17%2002.38.10.png
I have been noticing similiar behavious myself where my load will spike up to 30 on 6.1.6, but since I never did a factory reset yet I have just begun the process of debugging what is wrong although I think my problem is related to installed apps though. I am impressed that my NAS is not crashing though.
Have you tried turning off all addons while copying?
One thing I noticed in regards to your top imagaes you provided is that Plex Media Server is running when you have these problems. I have also noticed my Plex Media Server seems to be high in the top list when I get these spikes. When I turn off addons like plexmediaserver and nzbget my load immediately goes down.
I switched to nzbget since I thought sabnzb was taking too much processing time when idle but it may also be causing problems. I also switched from using deluged to utorrent and utorrent is way better on my system load. I also created custom init.d scripts for alot of my apps that set ionice, nice and io scheduler class. For me I have narrowed down my misbehaving apps to either plexmediaserver, nzbget or bitcasa.
I think my problem is with nzbget now though. 98-99% of the time my load is fine. It just seems to be shooting up to a load of 20-30 every once in a while. Last time I just shut off nzbget and all went back to normal.
I wish there was a was to run iotop on this kernel. Maybe I will have to compile a version of the kernel myself 🙂
# iotop Could not run iotop as some of the requirements are not met: - Linux >= 2.6.20 with - I/O accounting support (CONFIG_TASKSTATS, CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING)
