Git: Ignore files for public repository, but not for private

Question

I'm deploying a Rails app on Heroku (for now) via git, and would also like to have a public version for people to look at. Some files are sensitive and should only be committed and pushed in the "heroku" branch, but not the "public" branch. What is the best way to go about this?

(I do know about Heroku's Config variables, which is great as a temporary solution, but not fun if and when I need to switch hosts.)

The two branches don't need to be synced at all times - I'm okay with periodically merging the "master" branch into the "public" branch and pushing it to github separately.

I have tried various things:

• separate .gitignore files and an "ours" merge strategy - this didn't work at first, and after messing with it for a while I decided it was getting too complicated just so I could achieve a seemingly simple task

• using a custom exclude file, and adding the following to .git/config... this simply did not work:

.git/config

[branch "public"]
  excludesfile = +info/exclude_from_public

What is the best way to have a private and public repository share the same code, but ignore sensitive files in the public repository?

You can assume that no code has been committed or pushed, i.e. this is a freshly initialized repository.

(This question has been asked before in various forms, but none of the answers were straight-forward or the answers seemed really hacky. I'm just here to ask this in a very simple manner, and hopefully receive a very simple response.)

Stigmasterol

Can you delete the files in the public branch? – Libava

Yes, but I will get merge conflicts with .gitignore this way, and the files will be added back into the public branch when I run git merge master – Stigmasterol

If you are confident using multiple version control systems, you could use darcs/mercurial/svn/bzr/whatever for the private branch, and selectively push to git. – Blastocoel

What are the specific files you need to hide? Perhaps we can suggest a simpler way to approach this, such as using Heroku config variables. – Isiahisiahi

@Isiahisiahi initializers/secret_token.rb, and a few YAML files containing some encrypted information (admin login & s3 creds) – Stigmasterol

Answer 1

I will second the submodule answer, but try to provide some clarification. First, git does not deal with files but with commits. There is no way to filter files or paths in a branch because a branch is really a pointer to a commit. When you exclude or ignore you are just keeping files from being added to your repository. none of the 'sensitive files' files are even in the repository, just in your working directory.

The submodule is just a reference to another repository stored in your repository, and a specific commit that that checked out repository is tracking. you can say update using

git submodule update --recursive sensitive-files

In order to simplify things, you can commit symlinks in the proper place pointing to the submodule path.

ln -sf sensitive-files/shadow passwd

Then add the symlink as you would any other file..

Remember the submodule is just a checked out git repository, you can easily restrict access to that actual repository and make the main one public.

Updated:

Sorry I missed the notification, if you are still working on this.

You can have multiple symlinks in your private repository referencing the private repository (submodule) which is checked out in a subdirectory.Each of the databases or whatever used by the Rails instance could be a symlink into that private subdirectory.

Also, you don' t need a remote pointing to the private repository, just an entry in the .gitmodules file which is maintained automatically by git submodule. You would still need to protect the private repository so that only your Heroku instance could access it. For that I would suggest installing gitosis on a server if you can or use some other private git hosting solution. Add the public ssh key matching your instances private key tothe list of allowed users. (I'm not familiar with how to do this in Heroku.)

When you push your changes to heroku it should recursive download all the submodules mentioned in the repository.

Weatherly

Thank you for clarifying. I will probably accept your answer, however I'd like just a bit more clarification. I will need a private git repository in order to do this, correct? Also, the sensitive information is in various places in my Rails directory structure... Could I check out the "private repo" into the proper locations, or would the private repo have to basically just have a sort of database file that the Rails files could reference? Lastly, would my repo on Heroku have a remote reference the "private repo", or could I check it all into Heroku as one? Thanks! – Stigmasterol

Answer 2

You could create a pre-commit hook in your local repo, in here you can write a script to check the currently checked out branch and delete the offending files if they are present before the commit is processed. This avoids the files ever being recorded in the Git history of the wrong branch.

#!/bin/bash
current_branch="$(git branch | sed -e 's/^*//')"
if [ $current_branch != "heroku" ]; then 
    // Delete sensitive files before commit
    rm -f dir1/dir2/exclude_from_public
    rm -f dir1/dir2/exclude_from_public_also
fi
exit 0

Alternatively, the script could just check for the files and return exit code "1", notifying you that the commit cannot proceed because it contains sensitive files.

The caveat is that you will need to hand this script to anyone who is working on the "privileged" heroku branch, and always have it included in your own local repo.

Ideally you would have this check done server-side as well; but unfortunately GitHub only offers a Web variant of the post-receive hook, so unless you do you're own repo hosting this approach can only be performed locally.

Corollary

Answer 3

One way to do this would be to put your private file(s) in a submodule, and refer to that module from your public repo. (Alternately, you could put your public files in a submodule, and refer to that repo from your private repo.)

Len

I see how this could work, but it seems like an overly-complex way to just ignore two files. I will wait to see what others have to say. Thanks for your response. – Stigmasterol

This strategy does not work if your files in question are in the same directory as the regular files which you want to keep a is. Submodules are meant for retrenching other projects, libraries etc. - not for treating a small subset of files within one repo. – Palua

Answer 4

Here are some other StackOverflow questions and answers along the line of "how do you do a merge while ignoring some files":

• Git: merging public and private branches while while keeping certain files intact in both branches
• git merge should ignore one directory
• How do you merge selective files with git-merge?
• How do you make Git ignore files without using .gitignore?
• Public and private code in a single Git repository
• How do I tell git to always select my local version for conflicted merges on a specific file?

The simplest I can think of, is to use an alias'ed merge that will remove the private files before doing the merge commit. This would work if you're willing to live with non-fast-forward merges. Here's the alias:

git config alias.merge-master-exclude-private '!git merge --no-commit --no-ff master && (git diff --name-only HEAD..master | grep -f private_files | while read f; do git reset HEAD -- "$f"; rm -f "$f"; done; git commit -m "Merge master, excluding private files.")'

Then edit the private_files file and add file patterns that are private; for example secret_file.*$. You could replace private_files in the alias with "$(git rev-parse --show-toplevel)"/private_files to read private_files from the top-level directory.

Use git merge-master-exclude-private to do the merge. This will execute a non-fast-forward merge without committing, find files matching patterns in the private_files file, reset the index of any private files found, remove private files in the working directory, then commit. This should handle files that have whitespace in their names.

If you don't want to do the commit, giving you a chance to edit the commit message, remove -m "Merge master, excluding private files." from the alias.

Bitterroot

Answer 5

A guy named David Albert wrote a tool called Junk to solve almost exactly this problem. It lets files from a separate “junk drawer” repository live alongside the ones in your main repository.

The private files will have separate commits from the public ones, but it might do the job.

Muirhead

Answer 6

Create 2 branches. The one branch that has the private files will not be pushed to the public repo. After a merge, restore the files in question with git checkout HEAD^ -- files that should not have been merged, rm other files, git add -A and git commit --amend -C HEAD. I'm not sure what the difference in the files in question is but you get the idea. Make a small script for this and you're good to go. You could even commit a sensitive file list that you commit at the root and the script could act off of that.

Palua

I don't quite understand this method... could you explain it a little further? Will this keep the sensitive files completely out of the commit history for the public repo? – Stigmasterol

just don't push the private branch – Palua

Answer 7

I know this sidesteps the question, but I would simply have two git repositories. Then you can just add them permanently to the ignore list on the public repository.

You can have a second repository for the private files, and a small script to copy the changes to the correct location on the production system, when doing your deployment.

This reduces the risk that when you go on vacation and the new intern updates the public repo, your private information will accidentally get leaked. ;-)

Sike

Answer 8

Looks like a you could use mine.

Basically, it tells git to steer clear of stuff following the convention <file or directory>_mine_ and the tool itself gives you a snapshot, clean, and restore function, not full-fledged versioning, but for personal stuff it does the trick nicely.

The whole thing's pretty concise.

Agna