Anyone interested in compiling rtorrent for Duo V2?

[cosmos1]

Hello all,

I've been using rtorrent with specific setting on my previous NAS box, a meagre Maxtor Shared Storage II with great results. After obtaining my Readynas duo v2 box and being informed that I can use apt-get to download rtorrent I was more than cheerful! TBH, from transmission, enhanced-ctorrent and rtorrent I consider the last one to be the most balanced in performance as well as usability.

Unfortunately, it seems that the rtorrent (and its prerequisite, libtorrent) packages are compiled without any file preallocate functionality. Specifically, although I have enabled the file preallocation function of rtorrent (system.file_allocate.set = yes in ~/.rtorrent.rc), downloading a single 700Mb torrent took some 300+ extents on the ext4 filesystem... OTOH, the fact that the system.file_allocate.set is understood and rtorrent does not throw any errors about it at start, might indicate another issue.

Bottomline: I (and most likely, the rest of the community here) would be pretty excited if someone could compile (or crosscompile) rtorrent and libtorrent with the preallocate configure option, as discussed in https://wiki.archlinux.org/index.php/RT ... allocation ? Obviously, I would happily serve as a beta tester here!

[LrdShaper]

My NV+ v2 hasn't shipped yet so I cannot make an addon for you but I have more than a few ARM devices to compile this on. Is a deb package ok for you?

[cosmos1]

Thank you for your kind offer! I believe it would be just fine, although I am not sure what would be an alternative to a debian package here, or which are the merits of one way over the other.

What I do know is that it needs both the rtorrent and libtorrent packages. Also for the time being I have used apt-get to install ctorrent, which preallocates files and downloads just fine, so I could wait till your NV+ v2 arrives and you're cozy with it in order to compile directly on the same architecture.

In any case and whatever you prefer, a big THANK YOU!!! :worship:

[LrdShaper]

The alternative would be making it as an addon which lets you install/uninstall it via Frontview. Here they are:

libtorrent 0.13.1
rtorrent 0.9.1

I already have the sources so I didn't bother downloading the latest stable and just re-compiled with preallocate. 0.13.1 and 0.9.1 were still unstable when I downloaded them though but these are the same versions I'm using now and have had no problems so far. These will install to /usr/local so please uninstall any rtorrent and libtorrent packages you have installed via apt-get to avoid conflicts. Let me know how it goes, if it works as expected then I will download the latest stable sources. Cheers!

[cosmos1]

My apologies for the late reply, been busy.

I've uninstalled the other versions and installed the ones provided via "dpkg -i". When running rtorrent, it complained about a missing libtorrent14.so library, so I did a:

export LD_LIBRARY_PATH=/usr/local/lib

I will test it and get back to you!

[cosmos1]

Ok some initial testing. Download goes fine so far, however preallocation seems to not be working. Specifically, downloading a 700Mb file via enhanced-ctorrent (package "ctorrrent") and enabling preallocate for it (via a "-a" command line option) immediately creates a single 700Mb file, for which the following data are shown:

# filefrag -v file.mp4
File size of file.mp4 is 892549374 (217908 blocks, blocksize 4096)
 ext logical physical expected length flags
   0       0 369389568           32768
   1   32768 369422336           32768
   2   65536 369455104           32768
   3   98304 369487872           32768
   4  131072 369520640            2048
   5  133120 369524736 369522687  32768
   6  165888 369557504           32768
   7  198656 369590272           19252 eof
file.mp4: 2 extents found

ctorrent immediately allocates the entire file, before starting to download. On my old Maxtor Shared Storage II, that's how rtorrent also worked. I understood it worked that way because filefrag showed the exact same data, regardless if I run ctorrent with the preallocate option or rtorrent with the preallocate option.

Additionally, on my old platform, the files were not created with preallocation until the very first write access attempt. At that point, my rtorrent would pause for a while (probably because the preallocation routine was a blocker) and only after the file was completely created/allocated only then it would unfreeze and downloading would actually take place...

For some reason, this does not seem to be the case here, perhaps I'm missing out something. In the test file above, as time passes so does the output of filefrag changes. The extents used remain 2, but it seems that the file is heavily fragmented:

# filefrag -v file.mp4
 ext logical physical expected length flags
   0       0 369158144           30720 unwritten
   1   30720 369188864           30720 unwritten
   2   61440 369219584           30720 unwritten
   3   92160 369250304           30720 unwritten
   4  122880 369281024            2048 unwritten
   5  124928 369285120 369283071   9728 unwritten
   6  134656 369294848             101
   7  134757 369294949           27803 unwritten
   8  162560 369322752              16
   9  162576 369322768              12
  10  162588 369322780             164
  11  162752 369322944               4
  12  162756 369322948              60
  13  162816 369323008             256 unwritten
  14  163072 369323264             112
  15  163184 369323376             144 unwritten
  16  163328 369323520             198
  17  163526 369323718              10
  18  163536 369323728               5
  19  163541 369323733               3 unwritten
  20  163544 369323736             144
  21  163688 369323880               4 unwritten
  22  163692 369323884              36
  23  163728 369323920               4 unwritten
  24  163732 369323924              44
  25  163776 369323968               4
  26  163780 369323972              32
  27  163812 369324004               8 unwritten
  28  163820 369324012               5
  29  163825 369324017              15 unwritten
  30  163840 369324032              14
  31  163854 369324046            1266 unwritten
  32  165120 369325312              51
  33  165171 369325363              21 unwritten
  34  165192 369325384               6
  35  165198 369325390           13746 unwritten
  36  178944 369339136             256
  37  179200 369339392            6144 unwritten
  38  185344 369345536             349
  39  185693 369345885           27555 unwritten
  40  213248 369373440             256
  41  213504 369373696            4404 unwritten,eof
file.mp4: 2 extents found

I do know that my MSS used ext3, whereas this duo is using ext4 (not using Xraid here, just Flex-raid with JBOD). Could the preallocate function be that different in ext4?

[LrdShaper]

I don't know enough about this to answer your question. What I can do is test the same package on my raspberry pi and sheevaplug which is using ext3 and ext4 respectively. Both are running Debian Squeeze. Let me know what steps you want me to run and I'll post my results

[cosmos1]

Arghhhh, just lost firefox and the entire response I was writing!!!!

Anyways, thanks again for your awesome help mate, much appreciated. Okay, download and install both rtorrent as well as "enhanced-ctorrent" (if not available, install "ctorrent", they should be the same).

Do a port forward for ports 30000-30005 on your router, both TCP and UDP, to the IP of your test NAS. Create a ~/session and a ~/work directory. Create a ~/.rtorrent.rc file:

min_peers = 20
max_peers = 100
upload_rate = 60
directory = ~/
session = ~/session
ratio.enable=        ; Enables rtorrent to use ratio's.
ratio.min.set=110    ; Set min ratio to 1.0 / 1.0 upload / download.
port_range = 30001-30005
port_random = yes
use_udp_trackers = yes
encryption = allow_incoming,enable_retry,prefer_plaintext
dht = auto
dht_port = 30000
peer_exchange = yes
system.file_allocate.set = yes

Download a torrent file that has a single large (ie ~700Mbyte) file in it. Then use rtorrent to download it. You'll end up with a large file. Do a "filefrag -v <file>" on the resulting large file and post/check the result. If filefrag does not exist, then install package e2fsprogs.

Now do the same download, with the following command, on another directory:

mkdir ~/test
cd ~/test
ctorrent -a -E 1.1 -U 60 -p 30005 <sametorrentfile.torrent>

You'll end up with the same large file, albeit download from ctorrent. Do a "filefrag -v <file>" on the resulting file as well. Compare results.

[LrdShaper]

Sorry been busy, I'll test over the weekend

[cosmos1]

Don't worry at all mate, this is volunteer work after all, for which I am more than thankful! 🙂

[cosmos1]

Been some time since our last chat. I don't know if you are still interested (I am for certain) to throw a couple of tests. I am especially interested on whether certain symbols get defined or not. You see, the login for preallocation is in libtorrent in file src/data/socket_file.cc. Can you tell which of the following symbols get defined after running configure?

* HAVE_FALLOCATE
* USE_POSIX_FALLOCATE

I am interested in those two, because they define which method is used when the user selects preallocation. In that scenario, creation of a file happens through Socketfile::set_size (if I understand correctly). In the following, I've pasted this function, removing the SYS_DARWIN check:

bool
SocketFile::set_size(uint64_t size, int flags) const {
  if (!is_open())
    throw internal_error("SocketFile::set_size() called on a closed file");

#ifdef HAVE_FALLOCATE
  if (flags & flag_fallocate && fallocate(m_fd, 0, 0, size) == 0)
    return true;
#endif

#ifdef USE_POSIX_FALLOCATE
  if (flags & flag_fallocate &&
      flags & flag_fallocate_blocking &&
      posix_fallocate(m_fd, 0, size) == 0)
    return true;
#endif

#ifdef SYS_DARWIN
    [...] We are not interested here, does not get executed
#endif

  if (ftruncate(m_fd, size) == 0)
    return true;

  // Use workaround to resize files on vfat. It causes the whole
  // client to block while it is resizing the files, this really
  // should be in a seperate thread.
  if (size != 0 &&
      lseek(m_fd, size - 1, SEEK_SET) == (off_t)(size - 1) &&
      write(m_fd, &size, 1) == 1)
    return true;

  return false;
}

I am asking which of HAVE_FALLOCATE and USE_POSIX_FALLOCATE get used here, because I'd like to short circuit perhaps one or the other or both, in order for the preallocation to work properly... If HAVE_FALLOCATE is defined, then according to the piece of code above, preallocation always happens through a call to fallocate. Otherwise, if HAVE_FALLOCATE is not defined but USE_POSIX_FALLOCATE is, then preallocation happens through posix_fallocate... If neither exists, then allocation happens through the following catchall piece of code:

  if (ftruncate(m_fd, size) == 0)
    return true;

  // Use workaround to resize files on vfat. It causes the whole
  // client to block while it is resizing the files, this really
  // should be in a seperate thread.
  if (size != 0 &&
      lseek(m_fd, size - 1, SEEK_SET) == (off_t)(size - 1) &&
      write(m_fd, &size, 1) == 1)
    return true;

  return false;

[cosmos1]

By the way, I tried to allocate a 50Mb file using the fallocate command, and failed miserably:

# fallocate -l 50m test
fallocate: test: fallocate failed: Operation not supported

EDIT: Most likely, the fallocate call is supported.

[cosmos1]

Is it possible to provide the output of the configure step? I am interested to find which pragmas (like HAVE_FALLOCATE) are defined.

BTW, since I am definitely straining you here, is there a howto to compile packages for this platform?

[LrdShaper]

Received my ReadyNAS NV+ v2 a week ago. Still in the process of migrating everything over from my old ReadyNAS NV+ and should finish by Friday. I'll be more than happy to jump back on this once I'm done.