git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 2/3] builtin/log: prefetch necessary blobs for `git cherry`

From
Elijah Newren <newren@gmail.com>
Date
Apr 21, 2026, 21:28 UTC
Message-ID
<CABPp-BF4woakYQ5RZ32J8SzDs_VpvT2Wv+Y2WaHTnFnM=96Kzg@mail.gmail.com>
In-Reply-To
<a010a4ad-403a-4b6f-9a92-a33323eca0f2@gmail.com>
Hi Phillip,
On Sun, Apr 19, 2026 at 7:04 AM Phillip Wood <phillip.wood123@gmail.com> wrote:
Show 15 quoted lines
>
> Hi Elijah
>
> On 18/04/2026 01:32, Elijah Newren via GitGitGadget wrote:
> > From: Elijah Newren <newren@gmail.com>
> >
> > In partial clones, `git cherry` fetches necessary blobs on-demand one
> > at a time, which can be very slow.  We would like to prefetch all
> > necessary blobs upfront.  To do so, we need to be able to first figure
> > out which blobs are needed.
>
> "git rebase" without "--reapply-cherry-picks" suffers from this problem
> as well as it does the equivalent of "git log --cherry-pick". Is there
> any way to share prefetch_cherry_blobs() with the cherry-pick detection
> in revision.c?

Yes, you're right; git rebase without --reapply-cherry-picks and git log --cherry-pick both go through cherry_pick_list() in revision.c, which has exactly the same shape as the patch-ids loop in cmd_cherry(): build a hashmap of one side via add_commit_patch_id(), then look up the other side via patch_id_iter_first(). The on-demand blob fetches come from the same patch_id_neq() callback.

After poking around, I think the approximate scope of the fix would be: Move collect_diff_blob_oids(), always_match(), and prefetch_cherry_blobs() from builtin/log.c to patch-ids.c and expose the last one in patch-ids.h. In cherry_pick_list(), between the add_commit_patch_id loop and the comparison loop, build a temporary list of just the lookup-side commits (filtering by SYMMETRIC_LEFT/BOUNDARY as the existing loop already does) and call prefetch_cherry_blobs() on it.

That said, I'd rather leave this out of the current series. The bigger picture is that I have reservations about expanding partial-clone support further into this area. git cherry, git log --cherry-pick, and the default cherry-pick detection in git rebase all exist to answer "has this patch already landed upstream?" -- a question that, in repositories large enough to need partial clones, I feel is rarely worth the cost of computing patch-ids across arbitrary amounts of history. The honest guidance I would probably give for users on a large repo is "pass --reapply-cherry-picks (with rebase) and skip this entirely" or to narrow the range under consideration. The omission of a --no-reapply-cherry-picks option in git-replay wasn't a lack of effort or oversight, but a deliberate choice where I'd rather hold off (possibly indefinitely) on implementing it. So I'm a bit reluctant to make the performance hazard less visible without also asking whether we should even be doing that piece of the operation.

I only implemented the git cherry fix because of a specific customer situation where the operation was already baked into tooling, and prefetching at least makes the worst case tolerable. I don't want to hold myself to doing the same for the cherry_pick_list() path, but I'm fairly confident the code here can be re-used for those other cases and I'd help review a patch from anyone who wants to carry it forward.

Anyway, you are making the right connection, it's just that my personal answer is to let some other interested individual do it.

Previous: Phillip WoodNext: Phillip Wood
Message 11 of 29 in “Batch prefetching”
  1. 0/3 Batch prefetchingElijah Newren via GitGitGadget, Apr 16, 2026
  2. 1/3 patch-ids.h: add missing trailing parenthesis in documentation commentElijah Newren via GitGitGadget, Apr 16, 2026
  3. 2/3 builtin/log: prefetch necessary blobs for `git cherry`Elijah Newren via GitGitGadget, Apr 16, 2026
  4. Junio C HamanoApr 17, 2026
  5. Elijah NewrenApr 17, 2026
  6. 3/3 grep: prefetch necessary blobsElijah Newren via GitGitGadget, Apr 16, 2026
  7. 0/3 Batch prefetchingElijah Newren via GitGitGadget, Apr 18, 2026
  8. 1/3 patch-ids.h: add missing trailing parenthesis in documentation commentElijah Newren via GitGitGadget, Apr 18, 2026
  9. 2/3 builtin/log: prefetch necessary blobs for `git cherry`Elijah Newren via GitGitGadget, Apr 18, 2026
  10. Phillip WoodApr 19, 2026
  11. Elijah NewrenApr 21, 2026
  12. Phillip WoodApr 23, 2026
  13. Elijah NewrenApr 23, 2026
  14. Derrick StoleeApr 27, 2026
  15. Junio C HamanoMay 11, 2026
  16. Elijah NewrenMay 11, 2026
  17. Elijah NewrenMay 13, 2026
  18. Derrick StoleeMay 18, 2026
  19. 3/3 grep: prefetch necessary blobsElijah Newren via GitGitGadget, Apr 18, 2026
  20. Derrick StoleeApr 27, 2026
  21. Elijah NewrenMay 13, 2026
  22. 0/4 Batch prefetchingElijah Newren via GitGitGadget, May 14, 2026
  23. 1/4 promisor-remote: document caller filtering contractElijah Newren via GitGitGadget, May 14, 2026
  24. 2/4 patch-ids.h: add missing trailing parenthesis in documentation commentElijah Newren via GitGitGadget, May 14, 2026
  25. 3/4 builtin/log: prefetch necessary blobs for `git cherry`Elijah Newren via GitGitGadget, May 14, 2026
  26. 4/4 grep: prefetch necessary blobsElijah Newren via GitGitGadget, May 14, 2026
  27. Derrick StoleeMay 18, 2026
  28. Junio C HamanoMay 18, 2026
  29. Elijah NewrenMay 18, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.