Re: [PATCH v2 3/3] grep: prefetch necessary blobs
- From
Derrick Stolee <stolee@gmail.com>
- Date
- Apr 27, 2026, 12:59 UTC
- Message-ID
- <31763514-2602-4d8e-ac25-70590f090947@gmail.com>
- In-Reply-To
- <8fbfe69bc4d0c6166967986f24861ffa393ed7cf.1776472347.git.gitgitgadget@gmail.com>
On 4/17/2026 8:32 PM, Elijah Newren via GitGitGadget wrote:
Show 6 quoted lines
> From: Elijah Newren <newren@gmail.com> > > In partial clones, `git grep` fetches necessary blobs on-demand one > at a time, which can be very slow. In partial clones, add an extra > preliminary walk over the tree similar to grep_tree() which collects > the blobs of interest, and then prefetches them.
A log of the code is about walking trees to find blobs matching the input pathspec, with this being the core method:
Show 6 quoted lines
> +static void collect_blob_oids_for_tree(struct repository *repo, > + const struct pathspec *pathspec, > + struct tree_desc *tree, > + struct strbuf *base, > + int tn_len, > + struct oidset *blob_oids)
And in your test, you set up a repo to have three blobs with matches in two of the files:
Show 15 quoted lines
> +test_expect_success 'grep of revision in partial clone does bulk prefetch' ' > + test_when_finished "rm -rf grep-partial-src grep-partial" && > + > + git init grep-partial-src && > + ( > + cd grep-partial-src && > + git config uploadpack.allowfilter 1 && > + git config uploadpack.allowanysha1inwant 1 && > + echo "needle in haystack" >searchme && > + echo "no match here" >other && > + mkdir subdir && > + echo "needle again" >subdir/deep && > + git add . && > + git commit -m "initial" > + ) &&
But then the command downloads all of the blobs, not using a pathspec:
Show 10 quoted lines
> + # grep HEAD should batch-prefetch all blobs in one request. > + GIT_TRACE2_EVENT="$(pwd)/grep-trace" \ > + git -C grep-partial grep -c "needle" HEAD >result && > + > + # Should find matches in two files. > + test_line_count = 2 result && > + > + # Should have prefetched all 3 objects at once > + test_trace2_data promisor fetch_count 3 <grep-trace > +'
I think your code is correct, but I'd like to see a test here that demonstrates a pathspec filter on the 'grep' command to help filter out a blob that has a matching string.
Perhaps something like:
* matches.txt (has needle) * nomatch.txt (does not have needle) * matches.md (has needle)
and then 'git grep -c "needle" HEAD -- *.txt' would download two blobs and find one match. A second run without the pathspec would download one blob and find two matches.
Does that make sense as a test?
Thanks, -Stolee