git/list[1] front-page[2] threads[3] people[4] search[5] about
 

commit-graph is cool (overcoming add_missing_tags() perf issues)

From
Elijah Newren <newren@gmail.com>
Date
Oct 17, 2018, 18:00 UTC
Message-ID
<CABPp-BECpSOxudovjbDG_3W9wus102RW+E+qPmd4g3Qyd-QDKQ@mail.gmail.com>
Hi,
Just wanted to give a shout-out for the commit-graph work and how
impressive it is.  I had an internal report from a user that git
pushes containing only one new tiny commit were taking over a minute
(in a moderate size repo with good network connectivity). After
digging for a while, I noticed three unusual things about the repo[1]:
  * he had push.followTags set to true
  * upstream repo had about 20k tags (despite only 55k commits)
  * his repo had an additional 2.5k tags, but none of these were in
    the history of the branches he was pushing and thus would not be
    included in any pushes.

Digging in, almost all the time was CPU-bound and spent in add_missing_tags()[2]. If I'm reading the code correctly, it appears that function loops over each tag, calling in_merge_bases_many() once per tag. Thus, for his case, we were potentially walking all of history of the main branch 2.5k times. That seemed rather suboptimal.

Before attempting to optimize, I decided to try out the commit-graph with a version of git from pu. While I expected a speed-up, I was a bit suprised that it was a factor of over 100; dropping the time for local dry-run push[2] to sub-second. A quick look suggests that commit-graph doesn't fix the fact that we call in_merge_bases_many() N times from add_missing_tags() and thus likely need to do N merge base computations, it just makes each of the N much faster. So, perhaps there's still another scaling issue we'll eventually need to address, but for now, I'm pretty excited about commit-graph.

(And in the mean time I gave the user a one-liner to nuke his local-only tags that I suspect he doesn't need.)

Thanks, Elijah

[1] lerna seems to scale horribly, especially when you suddenly transition dozens of web developers and even more independent repositories into a single large monorepo. Usage of lerna was thankfully ripped out at some point, but the crazy number of historical tags remain. Also, this user did a bunch of the filter-branch'ing to suck extra repos into the monorepo, likely involved somehow in the many extra tags he had.

[2] In fact, I still had timings of over a minute when adjusting the command to:
  git push --follow-tags --dry-run /PATH/TO/LOCAL-MIRROR $BRANCH
Next: Derrick Stolee
Message 1 of 7 in “commit-graph is cool (overcoming add_missing_tags() perf issues)”
  1. Elijah NewrenOct 17, 2018
  2. Derrick StoleeOct 17, 2018
  3. Jeff KingOct 17, 2018
  4. Derrick StoleeOct 30, 2018
  5. Elijah NewrenOct 31, 2018
  6. Ævar Arnfjörð BjarmasonOct 30, 2018
  7. Elijah NewrenOct 31, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.