Two way sync with rsync

Question

I have a folder a/ and a remote folder A/. I now run something like this on a Makefile:

get-music:
 rsync -avzru server:/media/10001/music/ /media/Incoming/music/

put-music:
 rsync -avzru /media/Incoming/music/ server:/media/10001/music/

sync-music: get-music put-music

when I make sync-music, it first gets all the diffs from server to local and then the opposite, sending all the diffs from local to server.

This works very well only if there are just updates or new files on the future. If there are deletions, it doesn't do anything.

In rsync there is --delete and --delete-after options to help accomplish what I want but thing is, it doesn't work on a 2-way-sync.

If I want to delete server files on a syn, when local files have been deleted, it works, but if, for some reason (explained after) I have some files that aren't in the server but exist locally and they were deleted, I want locally to remove them and not server copied (as it happens).

Thing is I have 3 machines in context:

1. desktop
2. notebook
3. home-server

So, sometimes, server will have files that were deleted with a notebook sync, for example and then, when I run a sync with my desktop (where the deleted server files still exist on) I want these files to be deleted and not to be copied again to the server.

I guess this is only possible with a database and track of operations :P

Any simpler solutions? Thank you.

Connaught

I have this same problem. Did you find a solution, mwm? – Countdown

i actually did now. i was trying to achieve a dropbox like experience and came up with a solution that involves unison (that uses rsync) and lsyncd for monitoring file changes. i also found other people with this solution and more tricks on top » cerebralmastication.com/2011/04/fast-two-way-sync-in-ubuntu – Connaught

nowadays I'm using nextcloud locally on my home server. perfect "dropbox" sollution 100% managed by me. has android/ios/win/macos/linux/web clients. – Connaught

@Connaught : as I have already noted at superuser.com/questions/34697/… , this is a misunderstanding : unison does not "use" rsync in any normally meaningful way. It is surprising how many people keep believing this. – Fingerbreadth

Answer 1

Try Unison: http://www.cis.upenn.edu/~bcpierce/unison/

Syntax:

unison dirA/ dirB/

Unison asks what to do when files are different, but you can automate the process by using the following which accepts default (nonconflicting) options:

unison -auto dirA/ dirB/

unison -batch dirA/ dirB/ asks no questions at all, and writes to output how many files were ignored (because they conflicted).

Note: I am no longer using Unison (I use NextCloud, which doesn't address the original use case). However, note that rsync is not designed for bidirectional sync, while unison is. unison may have its bugs (as any other piece of software) and its wrinkles. I am surprised it seems to be actively maintained now (last time I looked I think I thought it looked dead), but I'm not sure what's the state nowadays. I haven't had the need to have a two-way file synchronizer, so there may be better options, though.

Note 2: I am now currently using Mutagen for some purposes. It still doesn't cover the question's use case completely, but it does implement two-way sync.

Oakland

By the way, if you think, that you're limited to rsync, because you don't have enough privileges to install unison - you still may install unison using prefix . – Delogu

Unison is pretty easy to use for small directories. But it is excruciatingly SLOW for directories with a large number of files. Where rsync will start copying almost right away, unison sometimes takes over 12 hours to start copying, because it scans every file before copying anything. – Aoristic

It git good at this ?At least git is easy to use and transport is fast.for blob there is an extension developed by github called Large File Storage. – Coequal

Main problems of git are large files and that it is really designed for explicit commits. Both can be reasonably improved upon by add-ons such as git-annex, large file storage, etc. – Oakland

@Coequal Don't use a hammer for a screw, or a screwdriver for a nail. :) – Dubbing

If you're synchronizing e.g. a music library where you care about 2-way file deletion but don't care about file content conflicts, you can use Unison -fastercheckUNSAFE flag to considerably speed up Unison. It will not compute the file content hash but just look at timestamp/length and overwite if newer. See the manual – Cylindroid

@ADTC, if hammering the screw in is going to work feel free to hammer away. – Conjugation

unison is great - but also very unreliable. In about 50% of my use cases it simple has not worked. The most common cause of this is poor inter version capability. I sync sometimes with content on a computer I have no privileges on. If that server happens to have different version than I do, unison fails. In addition, I've recently had it fail between unison clients of the same version, because they were using different compiled versions of some library (this was between an ubuntu computer and a raspberry pi). – Dugout

This isn't a scalable solution as there's no -dryrun and no man page. It's better to use simpler, known tools such as rsync than reinvent the wheel with a million options that you don't need. – Wild

The unison version needs to be equal on both sides, otherwise unison revues to work. That's why I am looking for an alternative. – Gurule

AVOID USING UNISON!! I've had it running for 2 months now on a cron, haven't changed versions on either side. today i noticed some folders were missing. turns out sometimes it fails and simply deletes folders it has issues with. it ended up deleting about half of nested folders. luckily i had backups and restored everything. rsync is 100x more reliable than unison. – Parallelize

Unison seems to be written in OCaml. I doubt I could find a binary for my ancient Ubuntu server, and I surely can't compiler there. – Sterilization

@Parallelize I disagree. I'm using unison for a dozen of years, and you have to read docs and not just guess, because there are caveats, but generally it's completely reliable, aside from disk read failures. I'd like to see the logs, just for self-education and out of my curiosity. – Scopula

@EdwardFalk unison pre-dates Ubuntu. Unison first released in 1998, while first Ubuntu only in 2004. Which ubuntu do you have? I guess unison might be in the repo. – Scopula

Answer 2

Since the original question also involves a desktop and laptop and example involving music files (hence he's probably using a GUI), I'd also mention one of the best bi-directional, multi-platform, free and open source programs to date: FreeFileSync.

It's GUI based, very fast and intuitive, comes with filtering and many other options, including the ability to remote connect, to view and interactively manage "collisions" (in example, files with similar timestamps) and to switch between bidirectional transfer, mirroring and so on.

FreeFileSync can easily sync two computers on the same network and also sync two computers on different and remote networks.

• On same network: have FreeFileSync use the local file system on one side and a shared network drive / path on the other. On Windows systems you enable file / disk sharing on one computer and access that share from the other. I use FreeFileSync this way to keep my main development PC source code synced with my 2 laptops. I have also synced one of these laptops with a Linux server with Samba installed and sharing one of its directories.
• Across networks: create a VPN and do the same as above. FreeFileSync will see the remote disk as it was on the local network. Or buy one router that allows you to connect a USB disk to it and share over the internet. I have installed a VPN on a remote Linux server and used it through the OpenVPN Windows client.

Leopardi

Does not work over a network. Why would I want to sync two local folders? – Wristband

@Wristband I have added the answer to the above reply – Leopardi

Answer 3

I'm surprised no one has mentioned Syncthing yet. I have been using it for years to synchronize my phone, my tablet and my two laptops. One time I also used it to send 10 GB of photos to my family ~600 km away, straight from my machine to their machine, and it was incredibly fast (despite the data getting routed through Syncthing's discovery server to work around NAT issues). I also tried OwnCloud/NextCloud at some point but Syncthing has been much more reliable and, also, much faster.

Swastika

Answer 4

You could also try bitpocket: https://github.com/sickill/bitpocket

Ngocnguyen

Answer 5

Try this,

get-music:
 rsync -avzru --delete-excluded server:/media/10001/music/ /media/Incoming/music/

put-music:
 rsync -avzru --delete-excluded /media/Incoming/music/ server:/media/10001/music/

sync-music: get-music put-music

I just test this and it worked for me. I'm doing a 2-way sync between Windows7 (using cygwin with the rsync package installed) and FreeNAS fileserver (FreeNAS runs on FreeBSD with rsync package pre-installed).

Janeanjaneczka

Won't this delete new local files? -u skips files newer on the destination, but rsync doesn't have a way of tracking the age of a delete, so how could this script differentiate between when a local file is new (and should be kept) or old (and should be deleted)? Or am I misunderstanding? – Saccharine

Does --delete-excluded delete any files that exist on destination directories but not source directories? I thought you had to define --exclude=<filename> so won't rsync only delete these excludes? Or does rsync treat all non-source directory files as "excludes"? – Geotaxis

Found the answer: In addition to deleting the files on the receiving side that are not on the sending side, this tells rsync to also delete any files on the receiving side that are excluded (see --exclude) rsync.samba.org/ftp/rsync/rsync.html – Geotaxis

From what I understand, if I create a file on the destination and then execute the rsync command to copy from source to destination, it will delete the file at the destination, since no time stamps are used. This isn't what you want right? – Pictorial

@Lübnah: Correct; this solution is not a full 2-way sync: Because rsync with --delete-excluded is run with the server as the source first: files added locally will be deleted during that run (whereas files added to the server are copied to the local target). The only local files synced back to the server are files that also exist on the server and were modified locally (more recently than the server versions) - which is not likely in the OP's scenario. As you hint at, methinks true 2-way syncing can't be done with rsync, because saving state between runs is required. – Thomsen

Answer 6

You might use Osync: http://www.netpower.fr/osync , which is rsync based with intelligent deletion propagation. it has also multiple options like resuming a halted execution, soft deletion, and time control.

Frager

Answer 7

You could try csync, it is the sync engine under the hood of owncloud.

Martlet

As of 2018, csync development appears to have stalled and owncloud look like they are forking it. See: github.com/owncloud/client/issues/4661 for more details – Liggett

Answer 8

I'm now using SparkleShare https://www.sparkleshare.org/

works on mac, linux and windows.

Connaught

as time goes by, I keep updating my solution. I'm now using nextcloud on my home server. syncing with the nextcloud client on linux, mac, windows, android and ios. – Connaught

Answer 9

Rclone is what you are looking for. Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers including local filesystems. Rclone was previously known as Swiftsync and has been available since 2013.

Fleam

It does not support two way sync: rclone.org/commands/rclone_sync – Tripura

Looks like rclone does support two-way (bidirectional) sync now: rclone.org/commands/rclone_bisync – Espinosa

Aye, rclone seems to work quite well in its bidirectional mode! I'm a fan of all those tools — rsync, unison, rclone — but lately I've been using rclone most, since I can pretty much do bidirectional sync with basically any protocol out there... – Spathe

Answer 10

I'm not sure whether it works with two syncing but for the --delete to work you also need to add the --recursive parameter as well.

Taxicab

The short equivalent of option --recurse is -r, which the OP is using (embedded in compressed-options group -avzru). – Thomsen