Re: ReadyNAS NV+ with slow networking, 114Mbit/sec @ GigE &
What version of RAIDiator?
What brand and model disks?
The NV+ (v1) is an evolutionary improvement over a product released back in February 2006 with almost the same hardware. It is faster than a 100Mbit ethernet device but it can go nowhere near saturating gigabit ethernet. The NV+ (v1) has a very slow CPU by today's standards.
Re: ReadyNAS NV+ with slow networking, 114Mbit/sec @ GigE &
It's very frustrating, there are few things as frustrating as watching an rsync copy from the NAS skip along at 12MB/sec.
If I mount the filesystem using AFP I can read 18MB/sec so about ~144Mbit which is an improvement but still not good. I'm not willing to install a beta firmware on this system to test, will likely just have to replace with a Synology or finally just give up and buy a JBOD again.
Re: ReadyNAS NV+ with slow networking, 114Mbit/sec @ GigE &
Presenting filesystems how? If the ReadyNAS isn't the Rsync client/server then what machines are? If NAS is computer A, then copying to computer C via computer B isn't going to be that quick.
Do look for that entry in the logs as depending on the value it may indicate a way performance can be improved.
The NV+ (v1) is pretty old hardware (evolutionary improvement over the NV which was released in Feb 2006). Considering the age of the hardware the performance isn't that surprising.
Newer models such as the ReadyNAS Pro Series as much faster.
Re: ReadyNAS NV+ with slow networking, 114Mbit/sec @ GigE &
emory wrote:
well, again, it isnt using rsync on the readynas at all. it isnt using rsync, it is just presenting filesystems.
Uhm. I'm afraid you are wrong.
rsync works at a client/server level.
an rsync client connects to an rsync server/service.
it is not a simple matter of 'presenting filesystems'.
it works by comparing checksums on both ends of the connection, this is a primary part that requires cpu processing and a limitation for low power systems.
it does not matter whether the rsync initiates the connection to your computer, or vice versa, there is processing going on at both ends.
Re: ReadyNAS NV+ with slow networking, 114Mbit/sec @ GigE &
I'm not a very good teacher, so I'm probably going to go about this all wrong.
When you read that Wikipedia page or the "any number" of other materials on rsync on Google, you're absolutely right that they are pretty consistent and that methods of using rsync are pretty much common knowledge. I'm a grizzly neck beard and I prefer to do things Ye Olde Fashioned way by reading the man page, so that's what my examples will refer to.
USAGE You use rsync in the same way you use rcp. You must specify a source and a destination, one of which may be remote.
One of the interesting things that apparently you're not aware of is that rsync doesn't need to be using another system at all. Rsync doesn't care if a group of files is local, presented by a ReadyNAS, living on an EMC iSCSI monstrosity or a microwave oven.
In some cases such as very high-latency low-bandwidth links between two locations, you will want to have that rsync "client/server" relationship, if not as a daemon or service on the remote end, at least to do the calculations and comparisons to avoid transferring the whole file back and forth over that slow connection. This is what rsync was really built for originally. Even if you don't connect over the rsync service, you can use ssh to connect to a remote host and then use the other end's rsync software and only shuffle over the bits that you need to.
You can also use rsync in local-only mode, where both the source and des- tination don't have a ':' in the name. In this case it behaves like an improved copy command.
In a situation like mine, where the CPU would be the limiting factor, I can (or rather, should be able to if the ReadyNAS worked as designed) move the contents of filesystems over the wire faster than the SPARC can compute hashes on large files (in my case backed up snapshots of a VMWare VM) and figure out which differences there are in rsync. My workstation will be the system conducting the comparisons of the two files. I have the filesystems presented by my ReadyNAS mounted locally, and as far as rsync would be concerned, they're a "local file" if I don't tell it otherwise.
This is pretty common use, since rsync can smartly copy things from Directory A to Directory B, respecting metadata, permissions, and only copying over the things it needs to. Doing it this way puts all of the computational load of the comparison of the SOURCE and DESTINATION, but my workstation doesn't even notice it since it's ample (i7 2600k @4GHz). By using --progress and --partial, I can recover from an interrupted copy and also see how much throughput I've got. That's how I know how slow it is to write to and read from my ReadyNAS over the network and how I can test multiple protocols (NFS/AFP) so easily.
I am very familiar with the computational limitations of the SPARC family of CPUs. I ported SETI@Home to sun4c and sun4m on two BSD distributions and was the port maintainer of those and m68k for a little while Back In The Day™. That is why I don't put the load of calculating file checksums on the ReadyNAS NV+, it clearly has enough to worry about.
Hopefully this illustrates things better and now you can go forth and use the mighty rsync in new ways, like the developers intended!
Re: ReadyNAS NV+ with slow networking, 114Mbit/sec @ GigE &
In any case, the readynas is still involved in all the IO that must take place and is still limited by the processing power of the cpu.
The CPU does not even have to be maxed out at 100% utilization for it to be the limit.
In the 5? some years I have had my nv+, I can tell you that you will not get more than about 12-15 max xfer rate with rsync on it, and often much less.
The generally fastest way to make transfers, is to use frontview to backup initially via NFS, then modify the backup job to use rsync for subsequent transfers.
Re: ReadyNAS NV+ with slow networking, 114Mbit/sec @ GigE &
emory wrote:
USAGE You use rsync in the same way you use rcp. You must specify a source and a destination, one of which may be remote.
One of the interesting things that apparently you're not aware of is that rsync doesn't need to be using another system at all.
Hey, the posters are only trying to help you. TeknoJnky was perfectly correct in what he said. Rsync is not a networked file system like NFS, CIFS or AFP. Your posts did not describe how you are using rsync very clearly. In your early posts you talked about sparc performance, which led folks to the reasonable conclusion that you were running it on the NAS. They responded accordingly.
I gather you are mounting the NAS folders with NFS, and using a workstation to run rsync - which forces the workstation to compute all the checksums by reading all the file data over the network. It's not immediately obvious to me if that is faster than running rsync on the NAS or not.
What I do know is that you get the best performance on the NAS by running an NFS copy first, and then follow that up with rsync to get incremental updates. Also, that NV+ V1 is not a very high performing NAS, you will get much better speeds with an ultra or pro. So if you choose to replace the NAS you should look at those products also.
Re: ReadyNAS NV+ with slow networking, 114Mbit/sec @ GigE &
Well you're right about one thing: the IO is taking place on the ReadyNAS even though rsync is not!
No disrespect intended, I genuinely assumed that TeknoJnky didn't realize you could use rsync in this manner since they said:
Uhm. I'm afraid you are wrong. rsync works at a client/server level.
I said repeatedly I wasn't using rsync on the ReadyNAS. I don't know how I can be more clear than that, but I'd welcome feedback. I would like to be better at teaching and communicating new concepts to technical people.
Re: ReadyNAS NV+ with slow networking, 114Mbit/sec @ GigE &
The way I understand it, even if you run rsync in 'local' mode, you are still in fact using a client server, the client and server being on the same device is irrelevant.
Re: ReadyNAS NV+ with slow networking, 114Mbit/sec @ GigE &
BLOCK SIZES
I don't see them referenced in the logs in /var/log, but I took to tune2fs to check it out.
Filesystem OS type: Linux Inode count: 128000 Block count: 128000 Reserved block count: 0 Free blocks: 66855 Free inodes: 105086 First block: 0 Block size: 16384 Fragment size: 16384 Blocks per group: 65528 Fragments per group: 65528 Inodes per group: 64000 Inode blocks per group: 500
That seems to be consistent across the disks' '1' slices (e.g. /dev/hdg1).
RSYNC NERDERY
There are "roles" and "processes" in rsync, and you'd be right that they're treated similar!
As a ROLE the "receiver" is a SYSTEM. As a PROCESS the "receiver" is the child process spun off to write things to disk. There is a third component too, the GENERATOR which actually negotiates the filesystem-level actions the other two processes take up front before it hits the SEND and RECEIVE portion. So yes, in a way you are in fact dealing with "client/server" as a relationship between the three processes that are running because one feeds data into the other, and they subsequently negotiate which end is up.
The only time rsync is communicating with a socket is when it is talking to a receiver ROLE i.e. an rsync daemon. In this use case it would be talking directly on an opened network socket like the "rsync service" on the ReadyNAS being configured for a "share". Otherwise it is a bit more complicated (and more interesting!) than that.
When you use rsync with a local source and destination, you are spawning a few processes in fact:
* one to act as GENERATOR process to determine filesystem activities on SENDER and RECEIVER * one to open as a SENDER process * one to open as a RECEIVER process
The system running the rsync will handle all of those, so the ReadyNAS only has to expend cycles handling your presentation of filesystems that are mounted via afp/cifs/nfsd.
If you use rsync on the ReadyNAS it will be doing those calculations and figuring out what to send/receive in addition to handling disk operations. I wasn't doing that precisely for that reason, which is why I was incredulous why people kept belaboring (my perception) the point of rsync being computationally expensive on a SPARC. No kidding it is, that's why I have avoided making it do as much work as I can entirely! :wink:
Re: ReadyNAS NV+ with slow networking, 114Mbit/sec @ GigE &
all your effort which, unfortuantely, does not seem to provide much benefit over running rsync directly on the readynas itself. 😞
seems any efficiency in running rsync on a more powerful workstation is lost in the overhead of the NFS protocol and latency of the readynas itself in processing the required IO.
I suggest you compare transfer rates with your rsync trials and then using
time cp src dst
via straight nfs
Also, possibly of interest to you may be the addons;
istatd if you have an iphone/touch/pad, you can utilize the istat app (not free) to monitor cpu/network bandwidth/disk usage, even remotely if you forward the applicable ports on your router/firewall.
Bonnie++ I thought there was a sparc addon for this, but I only see the x86 version now, perhaps you may find it worth while to compile/download for sparc and see what kind of non-network transfer rates you can get.
Re: ReadyNAS NV+ with slow networking, 114Mbit/sec @ GigE &
As expected, using 'cp' to copy is about the same, it's just a little bit slower.
Since even if I don't have any disk IO involved it isn't performing like I'd expect, adding disk into the equation makes it more pronounced. I have a couple of Synology units coming for evaluation and I'm considering repurposing a c2q 9550 micro ITX system in the garage. I can't find a way to speed up the NV+ and it seems nobody else can either, so it's best I don't spend any more time on it. I bought it from Infrant and obviously it's time to find something else to do with it. All these years and only a power supply replacement that included a Netgear chassis. It's just now as I'm storing larger and larger files that I even notice that it's slow.
Don't sweat the effort, the only effort I expended was explaining how rsync works and how I am using it. It wasn't like I was re-inventing the wheel or anything. Happy to teach someone new tricks.
Re: ReadyNAS NV+ with slow networking, 114Mbit/sec @ GigE &
Yeah the NV+ (v1) is an evolutionary improvement over a product released in Feb 2006. With that old hardware this performance is understandable. With much newer models such as the Ultra or Pro performance is much faster.
Re: ReadyNAS NV+ with slow networking, 114Mbit/sec @ GigE &
Greetings,
I am in a pretty nonstandard use-case at the moment but I am experiencing pathetic network performance w/my readynas nv+.
The hardware: Backblaze storage pod (BBSP) HP Procurve 2724 (24 port GigE switch) NV+ w/2 USB Seagate 3TB (Goflex) backup "drives" attached as /dev/md3 (raid0). (NV+ sparc) (Raidiator 4.1.8)
The why: I had initial power issues, vendor provided 20 gauge molex power extenders when I asked for 18 gauge. I didn't check when I had initially wired the thing, ran into power issues, etc etc...badness...teeth gnashing...etc. As such, when I had originally built the thing, I could only keep 10 drives powered. As such I created a 11T, 10drive raid 6, array /dev/md0 and put everything I had on the thing. Note that at the time, the NV+ was the only ReadyNAS I could afford and there wasn't a > 4 bay variant at the time. I thought it pure insanity to purchase another, at the time, $600 driveless box, when a storage pod would give me all the space I would ever need more cheaply.
At the time I had this all set up w/ubuntu 9.10. For _whatever_ reason, the chipset drivers happily see the port multipliers and all is happy and swell. But the drivers are not happy enough to recognize the 3TB drives hanging off of the any of the usb ports. Attempts at using a newer kernel result in the SATA chipset drivers becoming unhappy and I'm unable to rebuild, manual or otherwise, 11TB array. Suffice it to say it's all hanging together w/duct-tape and bailing twine as it were. I'm just.trying.to.get.my.data.off so I can wipe and rebuild it properly.
Moving right along, I stuck the goflex drives on the NV+, to my surprise it saw them in all their 3TB glory and I was even able to create a raid0 ~5TB, after formatting, array. All seemed super swell.
The issues: I'm getting _just_ over 10bT speeds. 1.8MB/s on the high end, ~1.3MB/s most of the time. The status LEDs on the switch show 1GigE, full duplex, link speed for both NV+ and BBSP, ethtool on both machines also show 1GigE, full duplex. It's not a drive bottleneck as hdparm, and dd, tests show I can get at least 30MB/s to NV+ /dev/md3 array. And the BBSP /dev/md0, source array, can do > 122MB/s sustained.
I've tried nfs mounting the NV+ drives to the BBSP and doing a straight rsync, in local mode, from the BBSP and straight rsync, from the BBSP, to pull files from the NV+. I'm still get terrible speeds. I've also even swapped cables.
No jumbo frames are enabled, no crazy addons on the NV+, and no other network traffic. It's a private net just for these two boxes.
Re: ReadyNAS NV+ with slow networking, 114Mbit/sec @ GigE &
As an update, I replaced the NV+ w/a sony vaio-p I had lying around. I just stuck ubuntu 11.10 on it, installed mdadm, rebuilt the array, and started pulling data from the BBSP. Now I'm getting ~33MB/s. A substantial increase in throughput.
I guess the NV+ really _is_ _that_ anemic. I would not have figured it would be mostly worthless. Oh well.
Re: ReadyNAS NV+ with slow networking, 114Mbit/sec @ GigE &
I've found this thread very interesting. emory I think you did a decent job of explaining that concept to people like me that didn't realise that setup was possible.
If the CPU is the bottleneck in this situation (ie, if the observed transfer speed is the max write speed of the device because of the CPU), can we find the CPU level in the Readynas logs anywhere? I've had a quick look and can't see it.
