File system and / or Hard Drive diagnostics in OS6

[fream]

nasischijf wrote:

fream wrote: fream wrote: Future stuff I plan on testing to activate Jumbo frames to see if I can get better performance. And also to get a better network switch to be able to use network bonding (on both the Pro6 and the Mac Pro) to see if I can get better throughput. I also plan on maybe setting up iSCSI and move some of my Aperture libraries to the NAS. That requires to buy a iSCSI-driver for OSX though... So far a very happy upgrade and purchase! Looking forward to any feedback on the above and also new features in the coming releases of OS6! I've just configured iSCSI with CHAP on OS6 6.1.2 (RN102) using the free GlobalSan iSCSI initiator 4.1 on OSX 10.8.5 🙂 So, no need to purchase an expensive license 🙂

Nice! I didn't know that version was supported or worked on 10.8. Is it 64-bit? Would be good to hear your experiences going forward. Have you done any benchmarks or stress test of it?

[mangrove]

Is anyone else seeing absolutely abysmal write performance on iSCSI with OS6? Talking about 3-4 MB/s with sequential random data writes after some time. 7TB LUN, four-disk Ultra4, RAID5. Edit: related to TrueCrypt in some way.

Creating a new (1TB), fresh LUN and disabling "continuous protection" for this LUN improves speed, but still only to 30-40 MB/s, which is much, much slower than OS4. edit: this is false, but I have no real performance numbers as of now

Regarding this, what does the "Continuous protection" checkbox actually do? One could guess that it would disable copy-on-write for the iscsi container, but that is done anyway (as it really, really should!) according to lsattr for the file. Some kind of manual snapshotting?

[mangrove]

BTW, the performance problem is at its worst when using a TrueCrypt container on top of the iSCSI partition. Investigations are ongoing but my suspicion is that CoW isn't really disabled on the iSCSI container. Edit: doesn't matter if "Continuous protection" is unselected when creating a LUN, still useless performance. Found a bug though: even when you use thick provisioning, you can enable compression for the LUN afterwards in settings, although this is said to be impossible when creating a thick LUN. I have no idea if this actually tries to enable compression on the existing thick LUN.

[Skywalker]

Continuous protection == automatic scheduled snapshots. You can still take snapshots with CoW disabled on btrfs. It will still do the CoW as necessary if there is a snapshot reference.

[mangrove]

OK. Any theories on the extreme slowness with TrueCrypt?

[Skywalker]

Are there any lingering snapshots? If there are, then the first new write to any area of the LUN will result in a CoW operation. What iSCSI initiator?

btrfs sub list /data

[mangrove]

ID 256 gen 79 top level 5 path home
ID 258 gen 5705 top level 5 path .apps
ID 259 gen 7 top level 5 path .vault
ID 260 gen 4027 top level 5 path Pictures
ID 261 gen 4030 top level 5 path Videos
ID 262 gen 4024 top level 5 path Music
ID 263 gen 4021 top level 5 path Documents
ID 264 gen 4018 top level 5 path Backup
ID 265 gen 5936 top level 5 path ._share
ID 271 gen 5844 top level 5 path iscsi1
ID 8283 gen 5848 top level 5 path .purge
ID 8285 gen 6066 top level 5 path iscsi2

I use Microsofts iSCSI initiator on 32-bit win7, and the very same iSCSI partition gets 40 MB/s in writes just formatted as NTFS. So TrueCrypt means 1/10 write speed. Read speeds are the same. Machine speed is not an issue here.

[Skywalker]

No ideas off-hand. Is there an option to choose a larger block size on the client side?

[mangrove]

Hmmm this indeed looks like a problem with small blocks, XTS and MDRAID... NTGR's off the hook... for now. 😉 Tried upping stripe_cache_size but that didn't help at all.

BTW, was there ever talks (or even performance comparisons) of replacing the file-based iSCSI container with a device-based one?

[Skywalker]

It's been discussed. But you just lose so much flexibility (space must be pre-allocated at volume creation time, deletion immediately frees up space on the main volume) and features (snapshots, compression when using thin provisioning) that it was considered to be more trouble than it was worth. The performance gain in our testing was not significant. But I'm certain there are certain workloads where it would be beneficial.

[mangrove]

How do you solve the performance problem with your own encryption? Larger blocksize?

Where in the process does the decryption take place, and are there possibilities other than USB keys to unlock the volume? Specifically, how much of the OS boots before decrypting the volume?

[MueR]

Has anyone attempted to add a signed SSL certificate to OS6? I've been browsing through configuration files, but haven't really had much luck finding where the certificate is added.

[stevehaley]

Can someone summarise the pros and cons of upgrading a pro to OS 6.
Are there any speed benefits or specific enhancements that make it worthwhile?
If you do so is it fairly easy to drop back?
thanks

[mdgm-ntgr]

I'll let someone else summarise the differences but you can easily go back (though you do have to backup your data before downgrading as a factory default is required). I'll send you a PM with instructions on how to downgrade.

[MueR]

Something I noticed on my Ultra 6.. I't not using swap at all. The swap is always at 2094844 total, 0 used, while physical memory is pretty much always at 1009592 total, ~22000 free. Did I break something by chance? Could this be the cause of very slow copy actions? Load is currently at 11 for a simple par2 action, while under OS4 I never saw those numbers.. now the default load seems to be 3 instead of the ~0.7 I was used to.

[mdgm-ntgr]

If you were using swap that would be an indication that you don't have enough memory installed. Not using swap is a good thing. I assume a fair bit of your memory is currently being used for caching (this is good for performance and memory is freed up when needed)?

[MueR]

Well, if you look at this: https://dl.dropboxusercontent.com/u/208 ... nastop.PNG ... This is not normal behaviour for any linux machine I know. The problem lies in the huge amounts of waitstates the system is having. Just now I saw all 4 CPUs at 97+ wa. It does this fairly often, and always when those btrfs processes float up to the top.


edit:
... and the system just completely froze. Load averages at 20.37, 20.72, 14.26. Wait states at 100%, 100%, 100%, 96.4%. A large number of operations (such as an unrar or copying a large number of smaller files to the device) just causes the entire nas to become unresponsive. Netgear, why did you pick an experimental file system (and a dodgy one at that) for your new OS....

[mdgm-ntgr]

What about

# cat /proc/meminfo

The NAS uses free memory for caching files and frees it up when needed. The memory should be more than enough to avoid swapping.

The memory isn't the issue but rather something else. Those Load Averages are bad.

What version of ReadyNAS OS are you running?

[MueR]

I'm running 6.1.3

root@MueR-NAS:~# cat /proc/meminfo
MemTotal:        1009592 kB
MemFree:           23052 kB
Buffers:            1484 kB
Cached:           494400 kB
SwapCached:            0 kB
Active:           298428 kB
Inactive:         408992 kB
Active(anon):     108100 kB
Inactive(anon):   110208 kB
Active(file):     190328 kB
Inactive(file):   298784 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       2094844 kB
SwapFree:        2094844 kB
Dirty:            214456 kB
Writeback:             0 kB
AnonPages:        211676 kB
Mapped:            24780 kB
Shmem:              6632 kB
Slab:              55696 kB
SReclaimable:      32516 kB
SUnreclaim:        23180 kB
KernelStack:        1672 kB
PageTables:         8848 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     2599640 kB
Committed_AS:     809008 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        2776 kB
VmallocChunk:   34359727660 kB
DirectMap4k:        6720 kB
DirectMap2M:     1032192 kB

I just rebooted (had to, completely unresponsive). It restarts my unrar job and loads spike up again. Something about btrfs's transaction logging and unrar is causing very, very, very nasty IO lockups which will just result in the system grinding to a halt, because the FS can't handle normal work. Maybe I'm completely wrong here, but every time an unrar starts, it's trouble.

[ATCIS]

stevehaley wrote:
Can someone summarise the pros and cons of upgrading a pro to OS 6. Are there any speed benefits or specific enhancements that make it worthwhile? If you do so is it fairly easy to drop back? thanks

Well. . . Since I've PM'ed a similar response to about five others, perhaps I'll do a paradigm shift and post my experiences publicly this time:

ATCIS wrote:

It was relatively painless for the most part. I have two Pro 6's and one Ultra 4. If you have a Pro 6, make sure you upgrade the BIOS to v2 or later before you upgrade or you may run into what I refer to as the "Jet Engine Effect" where all the fans are blasting at maximum speed no matter what you do in software. Make sure you upgrade to the latest available BIOS for your specific platform BEFORE you convert to OS6. It's a lot easier to do from version 4.X.X PRO's: 1.) Running a modern version of Linux 2.) OS is very hackable. You can install virtually any software that is compatible with Debian (v7) Wheezy (I turned my Ultra 4 into a video surveillance system without having to pay NETGEAR $400 in licensing fees) 3.) The OS6 GUI looks cool and is 90% functional on legacy hardware 4.) Will continue to receive updates / and new features for the foreseeable future. Version 4.X.X will likely receive VERY FEW updates from this point forward CON's: 1.) If you are afraid to dabble a little at the Linux command line, don't bother making the conversion. You likely won't be happy with OS6 right out of the box (on legacy hardware). It requires a little bit of tweaking to get critical functions like fan speed control working properly 2.) Some GUI elements don't work at all (these seem to improve with each new version) on specific platforms 3.) OS6 on the Pro 6 - Front Panel Display is always on 4.) The necessity to back up ALL of your data somewhere else before attempting the conversion. When you have 6+TB's of data, that can be problematic

Of course your mileage may vary, but this have been my experience thus far. . . Other opinions / experiences always welcome!

[mdgm-ntgr]

Where are you unrar'ing to? The OS partition or the data volume?

[fastfwd]

MueR wrote:
Well, if you look at this: https://dl.dropboxusercontent.com/u/208 ... nastop.PNG ... This is not normal behaviour for any linux machine I know. The problem lies in the huge amounts of waitstates the system is having.

Any idea why the status of connmand, systemd, and kswapd0 is "D" (uninterruptible sleep)? If you sort your top display by process status (the "S" column), do you see more "D" processes? Is it normal for those processes to be in that state under OS6?

[MueR]

mdgm wrote:
Where are you unrar'ing to? The OS partition or the data volume?

The data volume. It's a rar archive on the data volume, that's being extracted to a different folder on the volume. I noticed the same behaviour when copying a large amount of smaller files to the data volume from a pc via SMB. FTP seems to be doing alright (read: still high loads, but less) for some reason.

[mdgm-ntgr]

Do you still have the issue on 6.1.4?

[MueR]

Less, but I still had to restart the device three times (hard reset via power cycle). I've nice'ed and ionice'ed sabnzbd and sickbeard in hopes of relieving it. It works to some extent. Nowhere near the performance that I had on OS4 though. I was able to download using sab at a comfy 12.5MB/s without slowing the system down one bit, combined with a running par2 or unrar job. Now, I'm lucky to make 6MB on an otherwise idle system. When attempting a download of a 100mb.bin file, the speed is good (10MB/s) and no noticeable load. It really does seem to be a problem of a lot of small files being written, and IO locking up.

Unfortunately, I can't run iotop since the kernel is too old or not configured for it, so I can't really track down the issue.

To illustrate, here's an iostat while sabnzbd is downloading:

root@MueR-NAS:/data# iostat -x 2 5
Linux 3.0.93.RNx86_64.2.1 (MueR-NAS)    10/25/2013      _x86_64_        (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.04    0.00    4.81   21.42    0.00   66.73

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda              73.69   241.75   38.65   44.24   787.48  1219.78    48.43     3.37   40.56   51.91   30.65   3.72  30.85
sdb              73.43   239.54   38.73   44.15   783.42  1210.85    48.12     3.41   41.00   50.77   32.44   3.80  31.46
sdc              73.64   238.67   38.57   43.99   790.17  1206.27    48.37     3.22   38.94   48.12   30.89   3.84  31.71
sdd              73.92   245.27   38.62   41.21   784.54  1224.76    50.34     5.05   62.84   56.66   68.63   4.69  37.44
sde              71.87   245.79   38.31   44.57   772.50  1237.41    48.51     3.26   39.20   50.63   29.37   3.69  30.60
sdf              73.22   245.97   38.77   44.84   793.42  1239.18    48.62     3.24   38.65   50.51   28.39   3.81  31.82
md0               0.00     0.00   12.01    5.23   341.03    94.84    50.55     0.00    0.00    0.00    0.00   0.00   0.00
md1               0.00     0.00    0.09    0.00     0.37     0.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
md127             0.00     0.00   76.57  139.64  2026.78  4995.38    64.96     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          23.23    0.00    5.93    1.77    0.00   69.07

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda              26.00   378.00    8.00   22.50   318.00  1602.75   125.95     0.48   15.34   12.75   16.27   4.77  14.55
sdb              46.00   378.50    9.00   17.50   368.00  1374.75   131.53     0.54   13.96    9.61   16.20   7.81  20.70
sdc              40.00   369.50   11.00   19.00   378.00  1444.75   121.52     0.43   13.20   13.55   13.00   5.22  15.65
sdd              56.00   362.00   11.50   20.00   484.00  1522.75   127.41     0.59   19.13   17.83   19.88   6.52  20.55
sde              46.50   367.00    9.50   18.00   378.00  1386.75   128.35     0.52   16.45   10.68   19.50   6.15  16.90
sdf              27.00   369.50    7.50   16.50   318.00  1176.75   124.56     0.58   19.46   18.00   20.12   6.94  16.65
md0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
md1               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
md127             0.00     0.00   25.50  145.00  1146.00  8272.00   110.48     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          22.40    0.00   23.83   10.29    0.00   43.49

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda             143.50  2264.50   18.00   58.50   680.00  9296.50   260.82     5.31   68.93   55.33   73.11   6.95  53.20
sdb             159.50  2217.00   23.50   61.50   800.00  9322.50   238.18     5.72   69.11   44.89   78.37   6.26  53.20
sdc             178.00  2179.50   31.00   57.00   868.00  9048.50   225.38     7.72   87.49   60.53  102.16   7.04  61.95
sdd             171.00  2169.00   25.50   53.00   804.00  8888.50   246.94     6.24   78.73   61.92   86.81   7.97  62.55
sde             157.50  2156.00   19.50   55.00   756.00  8992.50   261.70     4.68   63.52   58.87   65.17   6.81  50.75
sdf              96.50  2251.00   18.50   63.50   502.00  9622.50   246.94     4.36   54.55   39.22   59.02   6.71  55.00
md0               0.00     0.00    0.50    0.00     2.00     0.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
md1               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
md127             0.00     0.00   42.00  725.50   332.00 43556.00   114.37     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.32    0.00   11.73   11.10    0.00   68.85

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               3.00     0.00   29.00    0.50   148.00     0.25    10.05     0.25    8.32    8.45    1.00   3.71  10.95
sdb               1.00     0.00   36.50    0.50   160.00     0.25     8.66     0.26    7.00    6.96   10.00   4.00  14.80
sdc               0.50     0.00   34.50    1.00   140.00     2.25     8.01     0.23    7.92    5.91   77.00   3.62  12.85
sdd               1.50     0.00   30.50    1.00   134.00     0.25     8.52     0.23    9.02    6.90   73.50   3.95  12.45
sde               0.50     0.00   28.00    0.50   116.00     0.25     8.16     0.18    6.18    6.21    4.00   3.72  10.60
sdf               0.50     0.00   25.50    0.50   108.00     0.25     8.33     0.21    8.00    8.12    2.00   4.21  10.95
md0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
md1               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
md127             0.00     0.00  193.00    0.00   812.00     0.00     8.41     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.01    0.00    3.54   22.12    0.00   73.32

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda             166.50   236.50   35.00   34.50   806.00  1084.25    54.40     0.46    6.60    5.24    7.99   5.86  40.75
sdb             184.50   247.00   45.50   35.00   920.00  1132.25    50.99     0.55    6.84    5.11    9.10   5.91  47.55
sdc             189.50   258.00   46.00   38.50   942.00  1190.25    50.47     0.64    7.56    5.77    9.70   6.31  53.35
sdd             192.00   251.00   46.00   32.00   924.00  1134.25    52.78     0.71    9.12    6.57   12.78   7.42  57.85
sde             175.50   212.00   37.00   30.50   858.00   940.25    53.28     0.51    7.53    5.91    9.51   6.27  42.30
sdf             149.50   220.50   41.00   32.50   762.00  1012.25    48.28     0.51    6.94    5.51    8.74   5.85  43.00
md0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
md1               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
md127             0.00     0.00  136.50   82.00   550.00  4480.00    46.04     0.00    0.00    0.00    0.00   0.00   0.00

root@MueR-NAS:/data#