git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Making split commit graphs pick up new options (namely --changed-paths)

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Jun 10, 2021, 10:40 UTC
Message-ID
<871r9a2dol.fsf@evledraar.gmail.com>
On Wed, Sep 16 2020, Taylor Blau wrote:

Replying to http://lore.kernel.org/git/ccb6482feb8d8606d82b5ab97e33184f26d6c5b6.1600279373.git.me@ttaylorr.com as a start-off point for discussion;

Show 40 quoted lines
> Introduce a command-line flag to specify the maximum number of new Bloom
> filters that a 'git commit-graph write' is willing to compute from
> scratch.
>
> Prior to this patch, a commit-graph write with '--changed-paths' would
> compute Bloom filters for all selected commits which haven't already
> been computed (i.e., by a previous commit-graph write with '--split'
> such that a roll-up or replacement is performed).
>
> This behavior can cause prohibitively-long commit-graph writes for a
> variety of reasons:
>
>   * There may be lots of filters whose diffs take a long time to
>     generate (for example, they have close to the maximum number of
>     changes, diffing itself takes a long time, etc).
>
>   * Old-style commit-graphs (which encode filters with too many entries
>     as not having been computed at all) cause us to waste time
>     recomputing filters that appear to have not been computed only to
>     discover that they are too-large.
>
> This can make the upper-bound of the time it takes for 'git commit-graph
> write --changed-paths' to be rather unpredictable.
>
> To make this command behave more predictably, introduce
> '--max-new-filters=<n>' to allow computing at most '<n>' Bloom filters
> from scratch. This lets "computing" already-known filters proceed
> quickly, while bounding the number of slow tasks that Git is willing to
> do.
> [...]
> @@ -67,6 +67,11 @@ this option is given, future commit-graph writes will automatically assume
>  that this option was intended. Use `--no-changed-paths` to stop storing this
>  data.
>  +
> +With the `--max-new-filters=<n>` option, generate at most `n` new Bloom
> +filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is
> +enforced. Commits whose filters are not calculated are stored as a
> +length zero Bloom filter.
> ++
> [...]

Is there any way with an existing --split setup that introduces a --changed-paths to make the "add bloom filters to the graph" eventually consistent, or is some one-off --split=replace the only way to grandfather in such a feature?

Reading the code there seems to be no way to do that, and we have the "chunk_bloom_data" in the graph, as well as "bloom_filter_settings".

I'd expect some way to combine the "max_new_filters" and --split with some eventual-consistency logic so that graphs not matching our current settings are replaced, or replaced some <limit> at a time.

Also, am I reading the expire_commit_graphs() logic correctly that we first write the split graph, and then unlink() things that are too old? I.e. if you rely on the commit-graph to optimize things this will make things slower until the next run of writing the graph?

I expected to find something more gentle there, i.e. marking that file as obsolete, not making it part of the new chain (replacing it), and then unlinking only things not part of the current chain of data that are too old. But perhaps I'm just misreading or misunderstanding the behavior...

Next: Taylor Blau
Message 1 of 10 in “Making split commit graphs pick up new options (namely --changed-paths)”
  1. Ævar Arnfjörð BjarmasonJun 10, 2021
  2. Taylor BlauJun 10, 2021
  3. Derrick StoleeJun 10, 2021
  4. Ævar Arnfjörð BjarmasonJun 10, 2021
  5. Taylor BlauJun 11, 2021
  6. Derrick StoleeJun 11, 2021
  7. Taylor BlauJun 11, 2021
  8. Derrick StoleeJun 15, 2021
  9. Ævar Arnfjörð BjarmasonJun 15, 2021
  10. Junio C HamanoJun 16, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.