git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 3/4] builtin/remote: rework how remote refs get renamed

From
Patrick Steinhardt <ps@pks.im>
Date
Jul 29, 2025, 12:24 UTC
Message-ID
<aIi9mAoAb1v9igQ6@pks.im>
In-Reply-To
<20250729081658.GA1842689@coredump.intra.peff.net>
On Tue, Jul 29, 2025 at 04:16:58AM -0400, Jeff King wrote:
Show 16 quoted lines
> On Mon, Jul 28, 2025 at 03:08:47PM +0200, Patrick Steinhardt wrote:
> 
> > The next-best thing is to do it in two transactions: one to delete all
> > the references, and one to recreate the references and their reflogs.
> > This signicantly speeds up the operation with the "files" backend. The
> > following benchmark renames a remote with 10000 references:
> 
> Hmm. I was surprised to see so much reflog code here. It looks like
> you're replaying the old reflog entry by entry. But the old code was
> leaning on refs_rename_ref() to do the individual renames, which just
> asks the backend to handle that for us (so e.g., the files backend just
> copies/moves the log files).
> 
> So it feels like ideally we'd be able to create a transaction element
> for renaming, and then the backends could similarly do what makes sense
> for them (and we wouldn't need a bunch of reflog code here).

I think overall the ref transaction code is in for a bit of a redesign. The current one-size-fits-all `struct ref_update` doesn't make a lot of sense anymore. The requirements have shifted because of the reftable backend, where we want to redesign interfaces so that we batch updates together to the best extent possible.

And this patch series here demonstrates that not only the "reftable" backend benefits.

In any case, the current infrastructure is extremely brittle and every change one does is a bit like threading a needle. More likely than not you missed this one edge case where changing a flag for one of the updates has a consequence in a completely unrelated place.

I think we should start splitting this up more and introduce different update types:

  - Forced updates where we don't very the old state.
  - Raceless updates where we do.
  - Reflog updates that only write a message as well as an old/new
    state.

If we had such an infrastructure it would also be feasible to introduce more types, like deletes or renames.

Show 9 quoted lines
> I guess that does not work with the two delete/create transactions you
> end up with here, though. And you need those to worry about D/F
> conflicts. But then...how did the original handle D/F conflicts? It kind
> of looks like it didn't, as it is doing a mass ref-by-ref rename in the
> middle.
> 
> If the refs code learned how to order things to handle the D/F conflicts
> within a transaction, then we could do a single transaction. And it
> could learn about rename primitives.

True. But as I already mentioned to Junio I don't really know how to backfill D/F conflict handling in the "files" backend. The problem is that with preexisting "refs/heads/parent" and "refs/heads/parent/child" you cannot create the latter ".lock" file. Sure, we can hack our way around that in some manner. But is that backwards compatible if another Git client were to operate in the same repository? I dunno.

Show 44 quoted lines
> I dunno. I think that would be nicer, but it's probably not worth
> holding up this topic. Your perf numbers are very nice. I guess the
> possible flip-side is that the existing code could be faster when
> renaming a single ref (so no quadratic behavior) with a pathological
> reflog (so moving the file is faster than re-writing all of those logs).
> 
> Hmm, yeah. Something like this:
> 
> 	cat >setup <<-\EOF
> 	#!/bin/sh
> 
> 	rm -rf repo
> 	git init repo
> 	cd repo
> 
> 	git init server
> 	git -C server commit --allow-empty -m foo
> 
> 	git remote add origin server
> 	git fetch
> 
> 	# make the reflog gigantic
> 	perl -i -ne 'for my $i (1..10**5) { print }' .git/logs/refs/remotes/origin/main
> 	EOF
> 
> 	hyperfine -p ./setup -L v old,new './git.{v} -C repo remote rename origin foo'
> 
> results in:
>   
>   Benchmark 1: ./git.old -C repo remote rename origin foo
>     Time (mean ± σ):       5.5 ms ±   1.1 ms    [User: 1.5 ms, System: 1.3 ms]
>     Range (min … max):     3.6 ms …   9.7 ms    58 runs
>   
>   Benchmark 2: ./git.new -C repo remote rename origin foo
>     Time (mean ± σ):     476.3 ms ±   9.8 ms    [User: 203.6 ms, System: 268.0 ms]
>     Range (min … max):   467.8 ms … 498.7 ms    10 runs
>   
>   Summary
>     ./git.old -C repo remote rename origin foo ran
>      86.43 ± 16.61 times faster than ./git.new -C repo remote rename origin foo
> 
> It's hard to bring myself to care, though. This is a stupidly
> pathological reflog, and the absolute time change is peanuts compared to
> the per-ref cost you're fixing here.

For the "files" backend performance is worse, for the "reftable" backend I'd expect that this might even be faster. Mostly because there is no way to trivially rename a reflog -- we basically do the same on a rename as we are doing with this patch series now.

Overall I don't care too much about this edge case. By default we never write reflogs for remote references anyway, and I doubt that you'll ever end up with a remote reflog that has thousands of entries. So I'd rather make the general case fast even if the esoteric case becomes slower.

But ideally we're able to lift such limitations in the future if we were to do the above rework.

Patrick
Previous: Jeff KingNext: Jeff King
Message 17 of 37 in “builtin/remote: rework how remote refs get renamed”
  1. 0/4 builtin/remote: rework how remote refs get renamedPatrick Steinhardt, Jul 28, 2025
  2. 1/4 refs: pass refname when invoking reflog entry callbackPatrick Steinhardt, Jul 28, 2025
  3. Justin ToblerJul 28, 2025
  4. Junio C HamanoJul 28, 2025
  5. Karthik NayakJul 29, 2025
  6. Patrick SteinhardtJul 31, 2025
  7. 2/4 refs: simplify logic when migrating reflog entriesPatrick Steinhardt, Jul 28, 2025
  8. Justin ToblerJul 28, 2025
  9. Junio C HamanoJul 28, 2025
  10. 3/4 builtin/remote: rework how remote refs get renamedPatrick Steinhardt, Jul 28, 2025
  11. Junio C HamanoJul 28, 2025
  12. Patrick SteinhardtJul 29, 2025
  13. Justin ToblerJul 28, 2025
  14. Junio C HamanoJul 28, 2025
  15. Patrick SteinhardtJul 29, 2025
  16. Jeff KingJul 29, 2025
  17. Patrick SteinhardtJul 29, 2025
  18. Jeff KingAug 2, 2025
  19. 4/4 builtin/remote: only iterate through refs that are to be renamedPatrick Steinhardt, Jul 28, 2025
  20. Junio C HamanoJul 28, 2025
  21. Karthik NayakJul 30, 2025
  22. Patrick SteinhardtJul 31, 2025
  23. Junio C HamanoJul 28, 2025
  24. 0/6 builtin/remote: rework how remote refs get renamedPatrick Steinhardt, Jul 31, 2025
  25. 1/6 refs: pass refname when invoking reflog entry callbackPatrick Steinhardt, Jul 31, 2025
  26. 2/6 refs: simplify logic when migrating reflog entriesPatrick Steinhardt, Jul 31, 2025
  27. 3/6 builtin/remote: fix sign comparison warningsPatrick Steinhardt, Jul 31, 2025
  28. 4/6 builtin/remote: determine whether refs need renaming early onPatrick Steinhardt, Jul 31, 2025
  29. 5/6 builtin/remote: rework how remote refs get renamedPatrick Steinhardt, Jul 31, 2025
  30. Jeff KingAug 2, 2025
  31. Patrick SteinhardtAug 4, 2025
  32. 6/6 builtin/remote: only iterate through refs that are to be renamedPatrick Steinhardt, Jul 31, 2025
  33. Junio C HamanoJul 31, 2025
  34. Patrick SteinhardtAug 1, 2025
  35. Junio C HamanoAug 1, 2025
  36. Patrick SteinhardtAug 4, 2025
  37. Junio C HamanoAug 4, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.