git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 2/3] builtin/log: prefetch necessary blobs for `git cherry`

From
Elijah Newren <newren@gmail.com>
Date
Apr 23, 2026, 17:38 UTC
Message-ID
<CABPp-BGQkN0ZeDAR4NzuyBakJHLM1AuqkdSGbb0YQfgWh2dWFg@mail.gmail.com>
In-Reply-To
<2abdc8ba-e361-492c-88b7-0c807ee9fb4d@gmail.com>
Hi Phillip,
On Thu, Apr 23, 2026 at 8:15 AM Phillip Wood <phillip.wood123@gmail.com> wrote:
> On 21/04/2026 22:28, Elijah Newren wrote:
> > On Sun, Apr 19, 2026 at 7:04 AM Phillip Wood <phillip.wood123@gmail.com> wrote:
> >> On 18/04/2026 01:32, Elijah Newren via GitGitGadget wrote:
> "--reapply-cherry-picks --empty=drop" is certainly more efficient. When
> we're computing patch ids do we do it for every upstream commit or just
> the ones that modify the set of paths that are modified in the branch
> we're rebasing?

You are correct that the patch id computations won't look at file contents of commits unless they modify the same set of files as one of the commits in our topic branch, but in order to determine the set of commits which modify the same paths as commits in the branch we're rebasing, we have to walk the upstream commits and do a tree-diff for every one of them. Yes, commits and trees tend to be much smaller than blobs, but the number of trees/commits we have to look at may be far larger than the number of blobs. The biggest repositories are constantly pushing so many commits that they are at a size where even a merge-base operation can start to feel expensive.

> It is a shame that we don't have a config setting for
> "-reapply-cherry-picks" as it is easy to forget to pass that option.
> Unfortunately it is not supported by the apply backend which makes such
> a setting potentially confusing.
Indeed.
Show 13 quoted lines
> >  The omission of
> > a --no-reapply-cherry-picks option in git-replay wasn't a lack of
> > effort or oversight, but a deliberate choice where I'd rather hold off
> > (possibly indefinitely) on implementing it.  So I'm a bit reluctant to
> > make the performance hazard less visible without also asking whether
> > we should even be doing that piece of the operation.
> >
> > I only implemented the git cherry fix because of a specific customer
> > situation where the operation was already baked into tooling, and
> > prefetching at least makes the worst case tolerable.
>
> I'm a bit surprised customers aren't complaining about tools that use
> "git rebase" being slow.
Are you sure they aren't complaining?

The merging parts of a rebase operation do have batch prefetching already (up to 3 batches per commit; done that way to minimize the number of objects downloaded because sometimes 2 or more of those batches can be skipped entirely and trying to combine them into a single batch would only be doable by downloading far more than needed). But, as you're alluding to, the --no-reapply-cherry-picks part does not.

I'll note that GitHub tends to focus far more on the server side; it's just that in this particular case with a special customer, they had me dig a little closer to their client side operations. In their case, they were using git-replay rather than git-rebase, so they'd have no reason to complain about rebase. git-replay shares the same batch prefetching for merge operations that rebase has, and doesn't have a --no-reapply-cherry-picks behavior that can even be selected. Honestly, I think the main reason this customer was also using git-cherry was because I didn't get the drop-commits-that-become-empty logic in the early versions of git-replay. You added that to git-replay (thanks again!), but after they had already built their tooling. This is only a guess on my part; they may have other reasons for actively wanting git-cherry, but I think it might be worthwhile for me to ask them if they can upgrade git versions (to get your fixes for empty commits in replay) and then drop the calls to git-cherry. However, I didn't want it to sound like I was pushing them to change their workflows at my convenience, and hence this patch so that things can be fast even if they keep the git-cherry in there.

Show 9 quoted lines
> > I don't want to
> > hold myself to doing the same for the cherry_pick_list() path, but I'm
> > fairly confident the code here can be re-used for those other cases
> > and I'd help review a patch from anyone who wants to carry it forward.
> >
> > Anyway, you are making the right connection, it's just that my
> > personal answer is to let some other interested individual do it.
>
> Fair enough
Thanks for taking a look and asking interesting questions.
Elijah
Previous: Phillip WoodNext: Derrick Stolee
Message 13 of 29 in “Batch prefetching”
  1. 0/3 Batch prefetchingElijah Newren via GitGitGadget, Apr 16, 2026
  2. 1/3 patch-ids.h: add missing trailing parenthesis in documentation commentElijah Newren via GitGitGadget, Apr 16, 2026
  3. 2/3 builtin/log: prefetch necessary blobs for `git cherry`Elijah Newren via GitGitGadget, Apr 16, 2026
  4. Junio C HamanoApr 17, 2026
  5. Elijah NewrenApr 17, 2026
  6. 3/3 grep: prefetch necessary blobsElijah Newren via GitGitGadget, Apr 16, 2026
  7. 0/3 Batch prefetchingElijah Newren via GitGitGadget, Apr 18, 2026
  8. 1/3 patch-ids.h: add missing trailing parenthesis in documentation commentElijah Newren via GitGitGadget, Apr 18, 2026
  9. 2/3 builtin/log: prefetch necessary blobs for `git cherry`Elijah Newren via GitGitGadget, Apr 18, 2026
  10. Phillip WoodApr 19, 2026
  11. Elijah NewrenApr 21, 2026
  12. Phillip WoodApr 23, 2026
  13. Elijah NewrenApr 23, 2026
  14. Derrick StoleeApr 27, 2026
  15. Junio C HamanoMay 11, 2026
  16. Elijah NewrenMay 11, 2026
  17. Elijah NewrenMay 13, 2026
  18. Derrick StoleeMay 18, 2026
  19. 3/3 grep: prefetch necessary blobsElijah Newren via GitGitGadget, Apr 18, 2026
  20. Derrick StoleeApr 27, 2026
  21. Elijah NewrenMay 13, 2026
  22. 0/4 Batch prefetchingElijah Newren via GitGitGadget, May 14, 2026
  23. 1/4 promisor-remote: document caller filtering contractElijah Newren via GitGitGadget, May 14, 2026
  24. 2/4 patch-ids.h: add missing trailing parenthesis in documentation commentElijah Newren via GitGitGadget, May 14, 2026
  25. 3/4 builtin/log: prefetch necessary blobs for `git cherry`Elijah Newren via GitGitGadget, May 14, 2026
  26. 4/4 grep: prefetch necessary blobsElijah Newren via GitGitGadget, May 14, 2026
  27. Derrick StoleeMay 18, 2026
  28. Junio C HamanoMay 18, 2026
  29. Elijah NewrenMay 18, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.