Running concurrent backups to rsync servers - questions

[davids355]

Hi guys,
just wondering if there is anyone else here that runs rsync backups to their own servers using backup assist?

I am just wondering how many concurrent backups you guys can run?

Currently have several separate rsync servers receiving backups but thinking about moving to a single server and wanted some feedback on how different hardware can handle receiving backups via rsync...

[Mike D]

It would depend on your Rsync destination on how many different sites can be running to that server at one time. The CWRsync Server we have had customers go up to 20 to 25 concurrent backups but once they get over that 25 there was issues. Again this depends on the Rsync destination server and what it supports.

[davids355]

Currently I am running backups to several servers - a mixture of i3 and i7 machines.
I am looking at a hexacore Xeon - e5 2620 v2, 48GB ram.
When I look at CPU comparison between that and the i7's I currently have, there isnt much difference in performance.

I seem to be able to get about 6 concurrent backups runinng on the i7 currently.

But I dont know if the Xeon setup would afford more - it has dedicated SATA controller card and a few other bits that might help.

I dont know what the biggest demand is on rsync (whether its CPU or disk writes etc)?

What sort of spec devices did you have when you were running 25 concurrent tasks?

[davids355]

^Oh sorry, just realized you were talking about other customers that had up to 25 concurrent tasks running.

I would really like to know what sort of hardware was required in those scenarios, and what in particular I need to get that level of performance.

I would definitely rather have one server if possible, as its easier to manage. I think in total we have about 50 backups running per night, but they can be spread out throughout the evening to produce around 10-15 concurrent tasks.

[davids355]

I have read that reducing the encryption of ssh when using Rsync can reduce cpu load on destination, is it possible to do that in backup assist?

[TimN]

Hi,

It's Tim from BackupAssist.

We have not tested reducing the level of SSH encryption to reduce the CPU load.

You can test this in BackupAssist via the 'Rsync options > Extra SSH options' section in BackupAssist.

[davids355]

Hi Tim,

thanks for the reply.
OK, I think I am going to set some backups to run concurrently each night, then I will start playing around with some of the encryption and priority settings for rsync/ssh (just noticed that space to add additional options to those commands).

For everyone else, I would really appreciate hearing experiences of how many concurrent tasks can be run to a remote rsync server and what specs those servers are.
I will report back my results as well.

[davids355]

Just a note of what I have set up for tonight:

Have set up 8 concurrent backups to run tonight.
All going via rsync/ssh to single server.
spec of server is:

i7-4770
32gb ram
2x2tb sata 7200 rpm - no raid, just separate drives
1gbps port
200mbps guranteed bandwidth

Will post back results of how well it runs but I have a feeling that many concurrent tasks will cause problems.

If it does I will lower compression and/or encryption and try again.

Wondering also whether processor clockspeed or cores is more beneficial - as the server I am looking to move to has lower clockspeed but more cores...

[davids355]

UPDATE:

OK so I had 8 concurrent tasks running tonight. each backup usually lasts between 1-3 hours so fair amounts of data being backed up.
All were backing up to the same remote rsync server (running CentOS).
5 backups were going to one physical harddrive and 3 backups to another.

Whilst the backups were running, CPU was sitting at 2-3 percent.
As soon as I saw some "bash -c rsync" commands, on the remote server, processor for that PID was around 30%, I noticed the --delete command there as well - would that be rsync tidying up old backups?? (See screenshot for example).

As soon as the clients moved to the last stage - compiling usage reports, then CPU for that PID went to constant 100% (Screenshot also attached) - looked like each PID doing compile of usage report was using 100% of a single core on the remote backup system.

Interestingly, once I canceled the job on the corresponding client, it did not stop that process from running on the remote server.


Based on that I am going to run all 8 backups together again tomorrow evening, with usage report feature turned off (i think i saw that in settings), in the hope that the demand on remote server will be greatly decreased...

Screenshot showing several processes on remote backup server, created by backup assist jobs:

[img]bloggtech.com/backup%20assist%20remote%20rsync%20command.png[/img]

[davids355]

Sorry, posted the screenshot link incorrectly:

[davids355]

Wow, turned off media usage reports and it runs much smoother.
Cpu usage gone from 100% to 5%.