How to setup mixed (S)FTP + Git workflow for website

Question

I have a lot of virtualhosted websites that are currently being updated via chrooted SFTP. Updates are done by clients (Typically using Dreamweaver and CuteFTP/Filezilla) and staff at my company (typically using Eclipse Team Syncronisation with JCraft SFTP). This setup works ok unless the clients are editing their site at the same time we are. In that case we have to constantly sync to check for changes which is both slow and unreliable. Also SFTP transfers are slow given the huge number of files in each site, slow directory traversal and lack of delta-compression.

I want to move to a (partial) Git workflow mostly to benefit from delta-compression but also to introduce some rudimentary revision control. I say "rudimentary" because the typical git workflow of developing and testing locally before bundling changes into git commits is a poor fit for our needs because:

• Some of our clients will need to use SFTP, not Git for convenience or compatibility
• I don't want to force a particular SFTP client (like git-ftp). Our clients are comfortable with their own choice (Dreamweaver or CuteFTP for example). They may be on any OS and they don't use shells.
• None of our clients can test locally, every small change must be uploaded to the webserver and go "live" immediately (we have test sites so "live" doesn't necessarily mean "production" or "public" in this case)
• About 90% of changes will be small 1-line CSS/HTML changes making commit messages time consuming.
• Some changes are done directly on the server webroot via a shell or scripts.

I guess what I want is for the remote git repo to use the webroot as a working directory and auto-commit any file changed there with a generic message indicating which file(s) changed. However I keep reading that the primary repo for a project shouldn't have a working directory (git init --bare) and even if it does it wouldn't normally commit changes made there. I don't care if this setup loses ownership details of commits (since it wont know who changed the files and I typically don't care).

Failing that I basically want to use Git as an alternative to Rsync-over-SSH (which does not appear to be supported by Eclipse Team Sync). Any answer suggesting another technology needs to support Eclipse (my tool of choice) and Dreamweaver (my clients main tool of choice). I realise this is not ideal but it isn't negotiable. If I could force use of Git I would and this issue would be irrelevant. I have to deal with hundreds of clients, mostly Graphic Designers on Macs.

PS. I realise there will be issues when the clients don't resync their local files regularly enough. Any suggestions on handling that would be appreciated.

Can anybody provide guidance on this setup (Centos 6.4 linux).

Engrossing

If you want more reliable and easy to use set up, I think the only way is to allow clients develop locally and then commit/deploy when all is ready. Local development is easier and more manageable. – Personalty

@kan: Sorry, impossible. Server software is linux only, complicated setup. Absolutely NOT an option. – Engrossing

Answer 1

For changes made directly on the server (or via SFTP):

Take a look at Gitwatch which is "a bash script to watch a file or folder and commit changes to a git repo." It seems to be just the thing you are looking for.

You will probably want to have your webroot not as the primary repo, but as a remote cloned repo. Setup your primary somewhere else (like github or bitbucket), clone it on the webserver. Use gitwatch to monitor changes, auto-commit, and push to your primary/github repo.

A big benefit of using a separate primary repo like github/bitbucket is that you can easily see/review changes that are being made, and pull them down locally yourself if you want, without needing direct access to the git repo on the webserver.

For changes made via git:

Are you looking to also make changes via git and have them auto-pulled on the webserver? In theory you could hook up a post-receive hook in git to do this for you. However you could easily get merge conflicts if there are changes being made directly on the server at the same time.

On second thought, avoid this. Stick with a single direction webserver -> local commit -> push to primary git repo.

Insert disclaimer here about how this is not a proper user of git, breaks all kinds of rules, and violates best practices. You seem well aware of this already, so I'll skip it. =)

Wilterdink

Gitwatch seems like a good start. Will look at it and get back to you. As for github I use it for free software but see no point in buying a private repo when I already have git on the webserver and fully own/control the server. And thanks for skipping the lecture, there's too much dogma around here already. The real problem here basically starts and ends with Adobe and their shit products and lame "RDS" protocol. They could add Git support but they won't. – Engrossing

Bitbucket lets you create private repos for free, so that might be better than github if you need to avoid costs. Or you might not need it at all, just thought it would be a helpful piece. – Wilterdink

@Engrossing any luck with gitwatch? – Wilterdink

I've only looked at the source code but I'm confident it'll do what I need (haven't had the time for testing). Unless a better suggestion comes up by tommorow the bounty is yours. – Engrossing

I've done some testing but the results are mixed. gitwatch is pushing changes to the (non-bare) repo but if I checkout changes from the repo after a push (post-receive hook with git checkout -f) gitwatch immediately deletes the file from the working directory! I'll try again using a bare repo and GIT_WORK_TREE set to the web root but this behaviour is a head-scratcher. – Engrossing

Answer 2

To me, you should setup a continuous integration tool such as Jenkins, CruiseControl or one of the many other alternatives, configured to deploy at the web server.

You need:

1. a repository for each website, accessible via sftp and watched with gitwatch to commit changes (like jszobody suggests) and a githook to pull / push / ask_for_help_via_email_to_a_developer_if_merge_fails
2. a "central" repository where developers and the hook will push each commit.
3. a CI tool of your choice that, at each push recieved by the central repository, wakes-up and sends the updated contents to the webserver (sftp, direct fs access... what is better for you).
4. a web server that, now, don't need to provide public access neither via git nor via sftp.

Pros:

• covers all your requirements
• no need for a working directory in the central repository
• no need to open more ports at the webserver (except for http/https, obviously)
• no need to install git on the web server
• no need to overload the web server's filesystem with the repository history (that is contained in the .git/ folder)
• quite flexible
• easy to evolve to a full featured CI solution (automated tests, code metrics and so on)
• copes with clients out of sync, because the hook won't be able to merge and a developer will be warned

Cons:

• complex, thus expensive to setup
• you will have to inform your clients about the new sftp address to use (unless you had wisely used a different subdomain and you can just wait for DNSs to update).

Herniotomy

Thanks for the response. This is overkill for my needs and seems better suited to software development than web development. I don't require my images and CSS to build nor are there any automated tests for visual quality. I'm going to go with the simpler solution. – Engrossing

That's fine. But let me guess that you will finally move to this solution. I've tested simpler ones in a web development agency and we had finally to switch to Jenkins, since the others proved to be unreliable. Unfortunatelly, we wasted a lot of time and we had to face with customers' complains in the process. – Herniotomy

Answer 3

I'm not seeing the sense in having a hundred developers deploying to a single web server and trying to track their changes there, but the quick and dirty way is to use a cron job to git add . and git push --force the server clone to an appropriate mirror somewhere else where you can better manage the merges and push back to the server clone (which should be on a distinct branch). The server clone (for the designers to whom the git is invisible) will be dirty almost all the time from the looks of it, so you might want to ease up on the cron interval.

Prerequisite

Your opening sentence is confusing. Are you suggesting I have 100 physical servers or ignoring the point about them not being able to deploy locally? – Engrossing

You don't seem to be worried that your clients might be overwriting one another's changes while sharing what is effectively the same scratchpad? – Prerequisite

The clients are on different websites, they are not all editing the same one, hence the "chroot". – Engrossing

Answer 4

Maybe something custom built on top of dulwich would suit your need, some kind of python based webserver for managing the repos with it. https://github.com/jelmer/dulwich

Inset