From: Elijah Newren Date: Wed, 13 May 2026 19:21:39 GMT Subject: Re: [PATCH v2 3/3] grep: prefetch necessary blobs Message-ID: In-Reply-To: <31763514-2602-4d8e-ac25-70590f090947@gmail.com> On Mon, Apr 27, 2026 at 5:59 AM Derrick Stolee wrote: > > On 4/17/2026 8:32 PM, Elijah Newren via GitGitGadget wrote: > > From: Elijah Newren > > > > In partial clones, `git grep` fetches necessary blobs on-demand one > > at a time, which can be very slow. In partial clones, add an extra > > preliminary walk over the tree similar to grep_tree() which collects > > the blobs of interest, and then prefetches them. > > A log of the code is about walking trees to find blobs matching > the input pathspec, with this being the core method: > > > +static void collect_blob_oids_for_tree(struct repository *repo, > > + const struct pathspec *pathspec, > > + struct tree_desc *tree, > > + struct strbuf *base, > > + int tn_len, > > + struct oidset *blob_oids) > > And in your test, you set up a repo to have three blobs with > matches in two of the files: > > > +test_expect_success 'grep of revision in partial clone does bulk prefetch' ' > > + test_when_finished "rm -rf grep-partial-src grep-partial" && > > + > > + git init grep-partial-src && > > + ( > > + cd grep-partial-src && > > + git config uploadpack.allowfilter 1 && > > + git config uploadpack.allowanysha1inwant 1 && > > + echo "needle in haystack" >searchme && > > + echo "no match here" >other && > > + mkdir subdir && > > + echo "needle again" >subdir/deep && > > + git add . && > > + git commit -m "initial" > > + ) && > > But then the command downloads all of the blobs, not using a > pathspec: > > > + # grep HEAD should batch-prefetch all blobs in one request. > > + GIT_TRACE2_EVENT="$(pwd)/grep-trace" \ > > + git -C grep-partial grep -c "needle" HEAD >result && > > + > > + # Should find matches in two files. > > + test_line_count = 2 result && > > + > > + # Should have prefetched all 3 objects at once > > + test_trace2_data promisor fetch_count 3 > +' > I think your code is correct, but I'd like to see a test > here that demonstrates a pathspec filter on the 'grep' > command to help filter out a blob that has a matching string. > > Perhaps something like: > > * matches.txt (has needle) > * nomatch.txt (does not have needle) > * matches.md (has needle) > > and then 'git grep -c "needle" HEAD -- *.txt' would > download two blobs and find one match. A second run without > the pathspec would download one blob and find two matches. > > Does that make sense as a test? Yes, absolutely. And thanks for suggesting it; although I was handling pathspecs correctly, I discovered that I was unconditionally requesting to download whatever objects matched the pathspecs (or all blobs in the commit if no pathspec given), even if the blobs were already local. I'll send an updated test in v2, along with the fix.