git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 4/4] connected: add incremental connectivity check via rev-list

From
Kristofer Karlsson <krka@spotify.com>
Date
Oct 16, 2026, 11:50 UTC
Message-ID
<CAL71e4NA3SH8V_haxJFeYCBmKk8+96j7ckxLDZA7+XMNcVfdKg@mail.gmail.com>
In-Reply-To
<atC-GVBnQrfr3cQ0@pks.im>
On Thu, 15 Oct 2026 at 13:50, Patrick Steinhardt <ps@pks.im> wrote:
Show 25 quoted lines
> >
> > I think this is where it gets tricky to reason about.  The win
> > from from larger trees pays off without bound -- the larger
> > trees you have, the larger the win.
> >
> > Scaling with commits on the other hand we have ceiling of how
> > much it can regress.  I am thinking that could be reasonable
> > that anyone who enables the config flag for this mode could
> > simply accept that it would likely speed things up in most
> > cases but in other cases they could see a ~1.5x regression.
> >
> > I am hoping that this tradeoff would then be acceptable,
> > otherwise we would indeed need to add some way of switching,
> > though I don't think there's a single good indicator value
> > for when it's better to switch.
> >
> > I could add a new config flag like
> > transfer.connectivityCheckIncrementalMaxCommits to control it
> > perhaps.
>
> Yeah, making this configurable may be reasonable. The way I understand
> it, the new connectivity check is most likely to be usable for smaller,
> incremental pushes. And based on the above table you have, we see that
> up to 1k commits the new algorithm is faster. Sure, it's an artificial
> benchmark, but maybe this is a good enough cutoff for now?
I looked into this a bit more and I have two main notes:

First of all, I benchmarked multiple repos, both with small object trees (such as git.git), medium (linux.git) and some really big ones (proprietary).

    repo          files   commits   full    incr.  full/incr
    internal A    552K       1000   1.69s   0.66s    2.6x
    internal B    331K       1000   1.48s   0.57s    2.6x
    linux          95K        609   0.34s   0.12s    2.8x
    linux          95K      13965   1.86s   1.90s    1.0x
    git.git        4.9K       356   0.09s   0.13s    0.7x
    git.git        4.9K       976   0.23s   0.31s    0.7x

So for large and medium repos the cutoff of 1000 is conservative. Incremental mode is faster but the full mode is also good enough here.

For small repos like git, the incremental mode is slightly slower, but since the number of files is so small, both modes are still fast enough that it does not matter.

(I could not get an exact 1000 commit test synthetically by looking back in the history for all repos, since it depends a bit on the shape of the graph of commits.)

Thus, I think having a cutoff of 1000 is indeed a valid strategy here.

Second of all (and with the previous in mind), I think I would want to change the configuration into three modes:

  - full (default): work as it does today.
  - incremental:    only use the incremental mode
  - auto:           decide on full or incremental depending
                    number of commits to process.

This would make things safe to roll out and start gaining more real life confidence before potentially changing the default to auto instead.

We could also make the cutoff configurable, but I am not sure that is needed in practice.

Show 6 quoted lines
> > > These probably should be `size_t`.
> >
> > Sure, I can change that.  Or intmax_t since we'll need to cast
> > to that later anyway for the trace2 output?
>
> Hm, okay. Not a fan of using signed integers as counters, but so be it.
No worries, I will switch to size_t instead.
Show 27 quoted lines
>
> > > > +     if (type >= 0)
> > > > +             die(_("object %s is a %s, not a %s"),
> > > > +                 oid_to_hex(oid), type_name(type), "blob");
> > >
> > > I'd love not having all these calls to `die()`, as this logic is
> > > essentially part of a library, and dying inside a library is a bad
> > > smell. We do this everywhere else in Git, too, but we're slowly getting
> > > rid of this as part of our libification efforts.
> >
> > I think this would be a bigger refactor in general --
> > the existing connectivity check also dies as the way to
> > report problems.  This works since it's running as a subprocess.
> >
> > If this is something we want to change, I think it would be
> > good to do that refactoring separately and make the connectivity
> > check subprocess only die() at the very edge, and making all of
> > the internal library code support not dying.  My first prototype
> > tried to do this but the amount of changes to internal code got
> > out of hand, so this version was the simplest I could make it.
> >
> > But I'm happy to take a step back and revisit that approach if
> > this is a blocker.
>
> I mostly aim at not making _new_ library code die anymore. In any case,
> I don't think it's a blocker, but I nowadays consider this to be a code
> smell.

I have spent some more time thinking about this and I have changed my mind -- I think I was too stuck in the idea of all-or-nothing here, but I agree I can make this work in a mixed way -- not introducing any new die() but accept the cases where it already dies.

I will include this in the reroll (likely next week, don't want to rush things).

Thanks, Kristofer

Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 36 of 60 in “connected: add incremental connectivity check”
  1. 0/2 connected: add incremental connectivity checkKristofer Karlsson via GitGitGadget, Sep 14, 2026
  2. 1/2 Documentation: describe connectivity checkingKristofer Karlsson via GitGitGadget, Sep 14, 2026
  3. 2/2 connected: add incremental connectivity check via rev-listKristofer Karlsson via GitGitGadget, Sep 14, 2026
  4. Junio C HamanoSep 14, 2026
  5. Kristofer KarlssonSep 14, 2026
  6. Junio C HamanoSep 14, 2026
  7. 0/2 connected: add incremental connectivity checkKristofer Karlsson via GitGitGadget, Sep 28, 2026
  8. 1/2 Documentation: describe connectivity checkingKristofer Karlsson via GitGitGadget, Sep 28, 2026
  9. Patrick SteinhardtOct 5, 2026
  10. Junio C HamanoOct 5, 2026
  11. Patrick SteinhardtOct 6, 2026
  12. Kristofer KarlssonOct 6, 2026
  13. Kristofer KarlssonOct 6, 2026
  14. 2/2 connected: add incremental connectivity check via rev-listKristofer Karlsson via GitGitGadget, Sep 28, 2026
  15. Patrick SteinhardtOct 5, 2026
  16. Kristofer KarlssonOct 6, 2026
  17. Patrick SteinhardtOct 6, 2026
  18. Kristofer KarlssonOct 6, 2026
  19. 0/4 connected: add incremental connectivity checkKristofer Karlsson via GitGitGadget, Oct 12, 2026
  20. 1/4 Documentation: describe connectivity checkingKristofer Karlsson via GitGitGadget, Oct 12, 2026
  21. Patrick SteinhardtOct 15, 2026
  22. Kristofer KarlssonOct 15, 2026
  23. 2/4 p5413: add perf tests for connectivity checkKristofer Karlsson via GitGitGadget, Oct 12, 2026
  24. Patrick SteinhardtOct 15, 2026
  25. Kristofer KarlssonOct 15, 2026
  26. 3/4 connected: add transfer.connectivityCheck config and rev-list flagKristofer Karlsson via GitGitGadget, Oct 12, 2026
  27. Junio C HamanoOct 12, 2026
  28. Kristofer KarlssonOct 13, 2026
  29. Junio C HamanoOct 13, 2026
  30. Patrick SteinhardtOct 15, 2026
  31. Kristofer KarlssonOct 15, 2026
  32. 4/4 connected: add incremental connectivity check via rev-listKristofer Karlsson via GitGitGadget, Oct 12, 2026
  33. Patrick SteinhardtOct 15, 2026
  34. Kristofer KarlssonOct 15, 2026
  35. Patrick SteinhardtOct 15, 2026
  36. Kristofer KarlssonOct 16, 2026
  37. Patrick SteinhardtOct 19, 2026
  38. Junio C HamanoOct 12, 2026
  39. Kristofer KarlssonOct 13, 2026
  40. 0/6 connected: add incremental connectivity checkKristofer Karlsson via GitGitGadget, Oct 21, 2026
  41. 1/6 Documentation: describe connectivity checkingKristofer Karlsson via GitGitGadget, Oct 21, 2026
  42. 2/6 p5413: add perf tests for connectivity checkKristofer Karlsson via GitGitGadget, Oct 21, 2026
  43. 3/6 connected: drop dependency on `the_repository`Kristofer Karlsson via GitGitGadget, Oct 21, 2026
  44. 4/6 connected: add transfer.connectivityCheck configKristofer Karlsson via GitGitGadget, Oct 21, 2026
  45. 5/6 connected: add incremental connectivity check via rev-listKristofer Karlsson via GitGitGadget, Oct 21, 2026
  46. Junio C HamanoOct 21, 2026
  47. Kristofer KarlssonOct 22, 2026
  48. Kristofer KarlssonOct 22, 2026
  49. Junio C HamanoOct 22, 2026
  50. Kristofer KarlssonOct 22, 2026
  51. Junio C HamanoOct 23, 2026
  52. 6/6 connected: add "auto" connectivity check modeKristofer Karlsson via GitGitGadget, Oct 21, 2026
  53. 0/6 connected: add incremental connectivity checkKristofer Karlsson via GitGitGadget, Oct 23, 2026
  54. 1/6 Documentation: describe connectivity checkingKristofer Karlsson via GitGitGadget, Oct 23, 2026
  55. 2/6 p5413: add perf tests for connectivity checkKristofer Karlsson via GitGitGadget, Oct 23, 2026
  56. 3/6 connected: drop dependency on `the_repository`Kristofer Karlsson via GitGitGadget, Oct 23, 2026
  57. 4/6 connected: add transfer.connectivityCheck configKristofer Karlsson via GitGitGadget, Oct 23, 2026
  58. 5/6 connected: add incremental connectivity check via rev-listKristofer Karlsson via GitGitGadget, Oct 23, 2026
  59. 6/6 connected: add "auto" connectivity check modeKristofer Karlsson via GitGitGadget, Oct 23, 2026
  60. Junio C HamanoOct 23, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.