Git checkout to external work tree and remove deleted files

Question

We want to use Git to deploy code on our webserver. Therefore, we have a initialized a bare repository on our production server. Whenever we release a new version, we perform a git checkout into the DocumentRoot of the website:

git --work-tree=/path/to/webroot/ checkout -f master

In the subdirectories of webroot, there are several files which are not tracked by Git (Cache files, user-uploaded files etc.). These must of course not be deleted by Git when performing the checkout (and this part works fine so far).

However, Git also does not delete files which were previously tracked, but have been removed in the meantime (e.g. they were deleted in the development process because they are no longer needed). Such files currently survive the checkout process, leading to a steadily increasing number of "dead" files. Is there a way to make Git delete such files when performing the checkout?

EDIT - steps to reproduce:

# create dirs and repos
cd /base/path
mkdir repo.git
mkdir target
cd repo.git && git init

# create, add and commit two files
touch test1.txt
touch test2.txt
git add test1.txt test2.txt
git commit -m testcommit

# checkout to target directory
git --work-tree=../target checkout master -f
# target directory now contains both files

# remove one file from file system and git repo, commit
rm test2.txt
git rm test2.txt
git commit -m deletecommit

# checkout to target again
git --work-tree=../target checkout master -f
# target still contains both files

Wolpert

I tried to do the same thing as you did, and it works for me. Can you create a minimal example and show how to reproduce it? – Underfeed

When you say the files were removed, do you mean you just deleted them from the work tree or did you actually delete them from the git index? E.g. in this case, you want the latter, accomplished via git rm – Geoffreygeoffry

I deleted them from both the file system and the repository. I added a minimal working example to illustrate the problem. – Wolpert

The index tracks worktree content. To track separate worktrees, keep an index file for each of them. – Hemielytron

"However, Git also does not delete files which were previously tracked, but have been removed in the meantime": Yes, with Git 2.22 (Q2 2019) and git checkout --overlay. See my answer below. – Bergquist

I had the same issue and reading all answers here made me even more confused, the only solution I find super simple is adding: git --work-tree=../target --git-dir=/path/to/repo.git clean -f – Kinshasa

Answer 1

However, Git also does not delete files which were previously tracked, but have been removed in the meantime

Yes, with Git 2.22 (Q2 2019) and git checkout --overlay, it does.
"git checkout --no-overlay" can be used to trigger a new mode of checking out paths out of the tree-ish, that allows paths that match the pathspec that are in the current index and working tree and are not in the tree-ish.

See commit e92aa0e (04 Feb 2019), commit 1495ff7, commit 091e04b (08 Jan 2019), and commit b7033e7, commit 5160fa0, commit 6fdc205, commit 536ec18, commit b702dd1, commit a0cc584 (20 Dec 2018) by Thomas Gummerer (tgummerer).
Suggested-by: Jonathan Nieder (artagnon).
^{(Merged by Junio C Hamano -- gitster -- in commit 7d0c1f4, 07 Mar 2019)}

checkout: introduce --{,no-}overlay option

Currently 'git checkout' is defined as an overlay operation, which means that if in 'git checkout <tree-ish> -- [<pathspec>]' we have an entry in the index that matches <pathspec>, but that doesn't exist in <tree-ish>, that entry will not be removed from the index or the working tree.

Introduce a new --{,no-}overlay option, which allows using 'git checkout' in non-overlay mode, thus removing files from the working tree if they do not exist in <tree-ish> but match <pathspec>.

Note that 'git checkout -p <tree-ish> -- [<pathspec>]' already works this way, so no changes are needed for the patch mode.
We disallow 'git checkout --overlay -p' to avoid confusing users who would expect to be able to force overlay mode in 'git checkout -p' this way.

Untracked files are not affected by this change, so 'git checkout --no-overlay HEAD -- untracked' will not remove untracked from the working tree.
This is so e.g. 'git checkout --no-overlay HEAD -- dir/' doesn't delete all untracked files in dir/, but rather just resets the state of files that are known to git.

And you have a new git config setting:

checkout.overlayMode:

In the default overlay mode, git checkout never removes files from the index or the working tree.
When setting checkout.overlayMode to false, files that appear in the index and working tree, but not in <tree-ish> are removed, to make them match <tree-ish> exactly.

---

"`git restore/checkout --no-overlay" with wildcarded pathspec mistakenly removed matching paths in subdirectories, which has been corrected with Git 2.29 (Q4 2020).

See commit bfda204 (22 Aug 2020) by René Scharfe (rscharfe).
^{(Merged by Junio C Hamano -- gitster -- in commit c57afd7, 31 Aug 2020)}

checkout, restore: make pathspec recursive

^{Reported-by: Sergii Shkarnikov
Initial-test-by: Sergii Shkarnikov
Helped-by: Jeff King
Signed-off-by: René Scharfe}

The pathspec given to git checkout and git restore^(man) and is used with both tree_entry_interesting (via read_tree_recursive) and match_pathspec (via ce_path_match).

The latter effectively only supports recursive matching regardless of the value of the pathspec flag "recursive", which is unset here.

That causes different match results for pathspecs with wildcards, and can lead checkout and restore in no-overlay mode to remove entries instead of modifying them.

Enable recursive matching for both checkout and restore to make matching consistent.

Setting the flag in checkout_main() technically also affects git switch^(man), but since that command doesn't accept pathspecs at all this has no actual consequence.

Bergquist

Answer 2

By using one working directory for some of the work and then another for the rest you are getting them out of sync with the rest of the repository. Git does not seem to be intended to be used that way.

If you want to use multiple working directories with one git repository there are some solutions available. See the stackoverflow question here.

Otherwise you can either:

• Clone the repository directly into the webroot. Though that would imply starting a fresh webroot.
• Clone a copy for but not in webroot and stick to consistently using it with the webroot as its only working directory (once you've got it in-sync with what you already have there). You can use git config core.worktree ../target to set it as the repositories default.

Churinga

This is not wrong, but in my deployment environment, I don't want the .git directory to be there (in the working directory of the web server). – Evesham

Answer 3

I have used git-clean to do something similar. If the files you don't want to remove are in gitignore, you can delete all unwanted files.

Matey