How to keep two folders automatically synchronized?

Question

I would like to have a synchronized copy of one folder with all its subtree.

It should work automatically in this way: whenever I create, modify, or delete stuff from the original folder those changes should be automatically applied to the sync-folder.

Which is the best approach to this task?

BTW: I'm on Ubuntu 12.04

Final goal is to have a separated real-time backup copy, without the use of symlinks or mount. I used Ubuntu One to synchronize data between my computers, and after a while something went wrong and all my data was lost during a synchronization.

So I thought to add a step further to keep a backup copy of my data:

• I keep my data stored on a "folder A"
• I need the answer of my current question to create a one-way sync of "folder A" to "folder B" (cron a script with rsync? could be?). I need it to be one-way only from A to B any changes to B must not be applied to A.
• The I simply keep synchronized "folder B" with Ubuntu One
  
  In this manner any change in A will be appled to B, which will be detected from U1 and synchronized to the cloud. If anything goes wrong and U1 delete my data on B, I always have them on A.

Inspired by lanzz's comments, another idea could be to run rsync at startup to backup the content of a folder under Ubuntu One, and start Ubuntu One only after rsync is completed.

What do you think about that? How to know when rsync ends?

Monarch

What are you trying to accomplish? Do the folders truly need to be separate, can't you symlink one name to the other, or bind-mount the directory to the secondary location? Also, take a look at these search results. – Simonize

I'd advise against actual live synchronization (i.e. watching continuously for file changes in the source directory) in favor of periodic rsync via cron. – Simonize

Answer 1

You can use inotifywait (with the modify,create,delete,move flags enabled) and rsync.

while inotifywait -r -e modify,create,delete,move /directory; do
    rsync -avz /directory /target
done

If you don't have inotifywait on your system, run sudo apt-get install inotify-tools

Dmitri

You might wanna consider to add -e ssh user@remote:/target to rsync if you want to use SSH as transport to access your remote machine. – Califate

Thanks @Falcon, I usually use rsync -avz --delete --exclude-from=.rsyncignore /directory user@server:path. However I wanted to write a simple answer. – Dmitri

to clarify: is this a one-time-run thing, or would you add this to a bashrc or something? – Hexavalent

You can consider the whole while loop as a process that synchronizes when needed. I usually add it to the Makefile of the project I'm working on. This way I can choose the server and the directory I want it to be saved. One of my use cases is when I program deep learning algorithms in my laptop and I synchronize it to perform the heavy process in a dedicated computer with better CPU and a GPU. I also exclude files that I don't need, such as the .git folder. – Dmitri

This doesn't handle file renames, use modify,create,delete,move – Repose

You're completely correct, thanks for the comment @Repose , I updated the answer ;) – Dmitri

Just 2 tips I found helpful when trying this: 1. If inotifywait is not installed, you need to apt-get install inotify-tools (or dnf, yum, pacman ... etc). 2. To run this on startup, add it to /etc/rc.local before the exit line. – Bookrest

This is one-way. It's like a backup. – Scroggins

2 more "perfectionist" corrections to the example script: 1) if we don't use rsync --delete, there is no need to listen for the delete event, 2) inotifywait, at least in its current version, listens to all events by default. – Campanula

Answer 2

You need something like this: https://github.com/axkibe/lsyncd It is a tool which combines rsync and inotify - the former is a tool that mirrors, with the correct options set, a directory to the last bit. The latter tells the kernel to notify a program of changes to a directory ot file. It says:

It aggregates and combines events for a few seconds and then spawns one (or more) process(es) to synchronize the changes.

But - according to Digital Ocean at https://www.digitalocean.com/community/tutorials/how-to-mirror-local-and-remote-directories-on-a-vps-with-lsyncd - it ought to be in the Ubuntu repository!

I have similar requirements, and this tool, which I have yet to try, seems suitable for the task.

Spotless

lsyncd at first seemed like a silver bullet, but it has lots of open issues and apparently the macOS implementation is "old and outdated." I couldn't get it to work on 10.12. – Iciness

@apaidnerd To be fair, it's 90 issues open vs 420 closed. Though, only 10/40 issues were closed in the last year. – Trimetrogon

Answer 3

Just simple modification of @silgon answer:

while true; do 
  inotifywait -r -e modify,create,delete /directory
  rsync -avz /directory /target
done

(@silgon version sometimes crashes on Ubuntu 16 if you run it in cron)

Feminism

Why would you run it in cron? – Propagandism

@Propagandism For a specific task, after Ubuntu rebooting the sync process should be run automatically – Feminism

I see.. Nevertheless, since this normally should be started once and be running as long as the server is up; I would consider something inlined within rc.local instead. – Propagandism

I inclued sleep 15m soyou can see logged changes – Salomon

Answer 4

Using the cross-platform fswatch and rsync:

fswatch -o /src | xargs -n1 -I{} rsync -a /src /dest

Trimetrogon

beautiful!! combing back here for the 2nd time! – Darden

Answer 5

You can take advantage of fschange. It’s a Linux filesystem change notification. The source code is downloadable from the above link, you can compile it yourself. fschange can be used to keep track of file changes by reading data from a proc file (/proc/fschange). When data is written to a file, fschange reports the exact interval that has been modified instead of just saying that the file has been changed. If you are looking for the more advanced solution, I would suggest checking Resilio Connect. It is cross-platform, provides extended options for use and monitoring. Since it’s BitTorrent-based, it is faster than any other existing sync tool. It was written on their behalf.

Corporator

Answer 6

I use this free program to synchronize local files and directories: https://github.com/Fitus/Zaloha.sh. The repository contains a simple demo as well.

The good point: It is a bash shell script (one file only). Not a black box like other programs. Documentation is there as well. Also, with some technical talents, you can "bend" and "integrate" it to create the final solution you like.

Creamer