How to do submodule sparse-checkout with Git?

K

2

14

There are a lot of articles and SO questions about sparse-checkout. Unfortunately I did not find concrete examples. I would like to get this following example work:

Create a submodule

cd ~
mkdir sub && cd $_
git init 
mkdir foo && touch $_/foo
mkdir bar && touch $_/bar
git add .
git commit -am "Initial commit"

Create a project

cd ~
mkdir project && cd $_
git init
git submodule add ../sub sub
git config -f .gitmodules submodule.sub.shallow true
git config -f .gitmodules submodule.sub.sparsecheckout true
echo foo/* > .git/modules/sub/info/sparse-checkout
git commit -am "Initial commit"
git submodule update
cd sub
git checkout .

A this point I am expecting sub folder to only contain foo/foo not bar. Unfortunately it doesn't work:

$ ls
bar/ foo/

How can I make it work?

Kellda answered 15/8, 2017 at 7:19 Comment(0)

B

18

git submodule add itself checks out the submodule.

What succeeded for me was:

git init
# I did not find a way to add submodule in 1 step without checking out
git clone --depth=1 --no-checkout ../sub sub
git submodule add ../sub sub
git submodule absorbgitdirs
# note there is no "submodule.sub.sparsecheckout" key
git -C sub config core.sparseCheckout true
# note quoted wildcards to avoid their expansion by shell
echo 'foo/*' >>.git/modules/sub/info/sparse-checkout
git submodule update --force --checkout sub

Or from a git URL:

# Initialize main repo (if needed)
git init

# Checkout the to-be submodule
# I did not find a way to add submodule in 1 step without checking out
git clone --depth=1 --no-checkout [email protected]:username/some-repo.git path/some-repo

# Add as a submodule
git submodule add [email protected]:username/some-repo.git path/some-repo

# Move the .git dir from path/some-repo/.git into parent repo's .git 
git submodule absorbgitdirs

# Note there is no "submodule.sub.sparsecheckout" key
git -C sub config core.sparseCheckout true

# This pattern determines which files within some-repo.git get checked out. 
# Note quoted wildcards to avoid their expansion by shell
echo 'foo/*'  >> .git/modules/path/some-repo/info/sparse-checkout

# Actually do the checkout
git submodule update --force --checkout sub

Besprent answered 15/8, 2017 at 9:1 Comment(6)

Have you updated echo bar >>.git/modules/sub/info/sparse-checkout with your pattern? – Besprent 15/8, 2017 at 9:10

Thanks, worked perfectly! Even though I think in this case absorbgitdirs does nothing because submodule add already puts the submodules git directory inside to root .git. Calling it doesn't hurt though – Muzzy 7/11, 2018 at 8:41

in the second example (with URL) there are inconsistencies between path/some-repo and sub, it should obviously be the same thing – Least 14/12, 2022 at 9:56

Is there a way to save this configuration so these commands don't have to be ran every time the parent repository is checked out? – Bracteate 29/12, 2022 at 14:13

What is the git submodule absorbgitdirs line about? Can a real example be given? – Thoer 17/4, 2023 at 21:41

what would be the sequence of commands for someone who clones the containing repo after it has been set up and pushed with the contained submodules? – Weese 23/7 at 11:3

P

8

To add to max630's answer:

since Git 2.25 (Q1 2020), you would use the new command git sparse-checkout
And with Git 2.28 (Q3 2020), The effect of sparse checkout settings on submodules is documented.

Meaning if you make the main repository itself sparsely checked out, in addition of the submodule (already sparse, as in max630's answer), using git sparse-checkout on the main repository will not affect the submodule negatively (ie. removing it by mistake while there is work in progress in it).

See commit e7d7c73 (10 Jun 2020) by Elijah Newren (newren).
^{(Merged by Junio C Hamano -- gitster -- in commit 81be89e, 22 Jun 2020)}

git-sparse-checkout: clarify interactions with submodules

^{Signed-off-by: Elijah Newren}
^{Reviewed-by: Derrick Stolee}

Ignoring the sparse-checkout feature momentarily, if one has a submodule and creates local branches within it with unpushed changes and maybe adds some untracked files to it, then we would want to avoid accidentally removing such a submodule.

So, for example with git.git, if you run
git checkout v2.13.0
then the sha1collisiondetection/ submodule is NOT removed even though it did not exist as a submodule until v2.14.0.

Similarly, if you only had v2.13.0 checked out previously and ran
git checkout v2.14.0
the sha1collisiondetection/ submodule would NOT be automatically initialized despite being part of v2.14.0.

In both cases, git requires submodules to be initialized or deinitialized separately.

Further, we also have special handling for submodules in other commands such as clean, which requires two --force flags to delete untracked submodules, and some commands have a --recurse-submodules flag.

sparse-checkout is very similar to checkout, as evidenced by the similar name -- it adds and removes files from the working copy.

However, for the same avoid-data-loss reasons we do not want to remove a submodule from the working copy with checkout, we do not want to do it with sparse-checkout either.

So submodules need to be separately initialized or deinitialized; changing sparse-checkout rules should not automatically trigger the removal or vivification of submodules.

I believe the previous wording in git sparse-checkout about submodules was only about this particular issue.

Unfortunately, the previous wording could be interpreted to imply that submodules should be considered active regardless of sparsity patterns.

Update the wording to avoid making such an implication.

It may be helpful to consider two example situations where the differences in wording become important:

In the future, we want users to be able to run commands like
git clone --sparse=moduleA --recurse-submodules $REPO_URL
and have sparsity paths automatically set up and have submodules within the sparsity paths be automatically initialized.

We do not want all submodules in any path to be automatically initialized with that command.

Similarly, we want to be able to do things like
git -c sparse.restrictCmds grep --recurse-submodules $REV $PATTERN
and search through $REV for $PATTERN within the recorded sparsity patterns.

We want it to recurse into submodules within those sparsity patterns, but do not want to recurse into directories that do not match the sparsity patterns in search of a possible submodule.

So the documentation now includes:

If your repository contains one or more submodules, then submodules are populated based on interactions with the git submodule command.
Specifically, git submodule init -- <path> will ensure the submodule at <path> is present, while git submodule deinit [-f] -- <path> will remove the files for the submodule at <path> (including any untracked files, uncommitted changes, and unpushed history).
Similar to how sparse-checkout removes files from the working tree but still leaves entries in the index, deinitialized submodules are removed from the working directory but still have an entry in the index.

Since submodules may have unpushed changes or untracked files, removing them could result in data loss.
Thus, changing sparse inclusion/exclusion rules will not cause an already checked out submodule to be removed from the working copy.
Said another way, just as checkout will not cause submodules to be automatically removed or initialized even when switching between branches that remove or add submodules, using sparse-checkout to reduce or expand the scope of "interesting" files will not cause submodules to be automatically deinitialized or initialized either.

Further, the above facts mean that there are multiple reasons that "tracked" files might not be present in the working copy: sparsity pattern application from sparse-checkout, and submodule initialization state.
Thus, commands like git grep that work on tracked files in the working copy may return results that are limited by either or both of these restrictions.

With Git 2.31 (Q1 2021), "git grep"^(man) has been tweaked to be limited to the sparse checkout paths.

Since you might need git grep in sparse-checkedout submodules, this is important.

See commit 42d906b (09 Feb 2021) by Matheus Tavares (matheustavares).
^{(Merged by Junio C Hamano -- gitster -- in commit 628c13c, 25 Feb 2021)}

grep: honor sparse-checkout on working tree searches

^{Suggested-by: Elijah Newren}
^{Signed-off-by: Matheus Tavares}
^{Reviewed-by: Elijah Newren}

On a sparse checked out repository, git grep^(man) (without --cached) ends up searching the cache when an entry matches the search pathspec and has the SKIP_WORKTREE bit set.

This is confusing both because the sparse paths are not expected to be in a working tree search (as they are not checked out), and because the output mixes working tree and cache results without distinguishing them.
(Note that grep also resorts to the cache on working tree searches that include --assume-unchanged paths.
But the whole point in that case is to assume that the contents of the index entry and the file are the same.
This does not apply to the case of sparse paths, where the file isn't even expected to be present.)

Fix that by teaching grep to honor the sparse-checkout rules for working tree searches.
If the user wants to grep paths outside the current sparse-checkout definition, they may either update the sparsity rules to materialize the files, or use --cached to search all blobs registered in the index.

And, still for sparsed-checked out submodules:

With Git 2.39 (Q4 2022), "git grep"^(man) learned to expand the sparse-index more lazily and on demand in a sparse checkout.

See commit 7cae762 (22 Sep 2022) by Shaoxuan Yuan (ffyuanda).
^{(Merged by Junio C Hamano -- gitster -- in commit 67bf4a8, 10 Oct 2022)}

builtin/grep.c: integrate with sparse index

^{Suggested-by: Derrick Stolee}
^{Helped-by: Derrick Stolee}
^{Helped-by: Victoria Dye}
^{Helped-by: Elijah Newren}
^{Signed-off-by: Shaoxuan Yuan}

Turn on sparse index and remove ensure_full_index().

Before this patch, git-grep utilizes the ensure_full_index() method to expand the index and search all the entries.
Because this method requires walking all the trees and constructing the index, it is the slow part within the whole command.

To achieve better performance, this patch uses grep_tree() to search the sparse directory entries and get rid of the ensure_full_index() method.

Why grep_tree() is a better choice over ensure_full_index()?

grep_tree() is as correct as ensure_full_index().
grep_tree() looks into every sparse-directory entry (represented by a tree) recursively when looping over the index, and the result of doing so matches the result of expanding the index.

grep_tree() utilizes pathspecs to limit the scope of searching.
ensure_full_index() always expands the index, which means it will always walk all the trees and blobs in the repo without caring if the user only wants a subset of the content, i.e. using a pathspec.
On the other hand, grep_tree() will only search the contents that match the pathspec, and thus possibly walking fewer trees.

grep_tree() does not construct and copy back a new index, while ensure_full_index() does. This also saves some time.

Piccadilly answered 28/6, 2020 at 1:1 Comment(5)

Hi there! I didn't get how Git 2.28 would simplify/improve @Besprent answer. Would you mind post/describe the equivalent commands to achieve something like this https://mcmap.net/q/804179/-how-to-do-submodule-sparse-checkout-with-git ? Thx! – Ticktock 15/11, 2021 at 16:43

@GerardBosch This is only to complement max630's answer by describe how the new git sparse-checkout command interact with submodules. – Piccadilly 15/11, 2021 at 17:18

Thanks, I thought it was a full answer! Maybe you can update it just to state what you said in this comment :) Thanks anyway!! – Ticktock 15/11, 2021 at 19:23

@GerardBosch Sure thing. I have edited the introduction of this answer, to make its purpose clearer. – Piccadilly 15/11, 2021 at 20:56

although this looks a bit useful, it is not that useful to not have example commands that clarify the documentation. Thanks for putting that together though. – Lip 8/5 at 8:41

Create a submodule

Create a project

`git-sparse-checkout`: clarify interactions with submodules

`grep`: honor sparse-checkout on working tree searches

`builtin/grep.c`: integrate with sparse index

Recommended topics

Hot tags

Create a submodule

Create a project

git-sparse-checkout: clarify interactions with submodules

grep: honor sparse-checkout on working tree searches

builtin/grep.c: integrate with sparse index

Recommended topics

Hot tags

`git-sparse-checkout`: clarify interactions with submodules

`grep`: honor sparse-checkout on working tree searches

`builtin/grep.c`: integrate with sparse index