git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 00/13] PATH WALK II: Add --path-walk option to 'git pack-objects'

From
Junio C Hamano <gitster@pobox.com>
Date
May 2, 2025, 21:24 UTC
Message-ID
<xmqqbjsau2nl.fsf@gitster.g>
In-Reply-To
<pull.1819.v2.git.1742829769.gitgitgadget@gmail.com>
"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
Show 50 quoted lines
> This patch series does the following:
>
>  1. Add a new '--path-walk' option to 'git pack-objects' that uses the
>     path-walk API instead of the revision API to collect objects for delta
>     compression.
>
>  2. Add a new '--path-walk' option to 'git repack' to pass this option along
>     to 'git pack-objects'.
>
>  3. Add a new 'pack.usePathWalk' config option to opt into this option
>     implicitly, such as in 'git push'.
>
>  4. Optimize the '--path-walk' option using threading so it better competes
>     with the existing multi-threaded delta compression mechanism.
>
>  5. Update the path-walk API with a new 'edge_aggressive' option that pairs
>     close to the --edge-aggressive option in the revision API. This is
>     useful when creating thin packs inside shallow clones.
>
> This feature works by using the path-walk API to emit groups of objects that
> appear at the same path. These groups are tracked so they can be tested for
> delta compression with each other, and then after those groups are tested a
> second pass using the name-hash attempts to find better (or first time)
> deltas across path boundaries. This second pass is much faster than a fresh
> pass since the existing deltas are used as a limit for the size of
> potentially new deltas, short-circuiting the checks when the delta size
> exceeds the current-best.
> ...
> This feature was shipped with similar features in microsoft/git as of
> v2.47.0.vfs.0.3 [4]. This was used in CI machines for an internal monorepo
> that had significant repository growth due to constructing a batch of
> beachball [5] CHANGELOG.[md|json] files and pushing them to a release
> branch. These pushes were frequently 70-200 MB due to poor delta
> compression. Using the 'pack.usePathWalk=true' config, these pushes dropped
> in size by 100x while improving performance. Since these CI machines were
> working with a shallow clone, the 'edge_aggressive' changes were required to
> enable the path-walk option.
>
> [4] https://github.com/microsoft/git/releases/tag/v2.47.0.vfs.0.3
>
> [5] https://github.com/microsoft/beachball
>
>
> Updates in v2
> =============
>
>  * Re-added a dropped comment when moving code in patch 1.
>  * Updated documentation to include interaction with --use-bitmap-index.
>  * An UNUSED parameter is now used, reducing the use of global variables
>    slightly.

The iteration saw no comments from anybody, so I (naturally) forgot about it for quite a long time. Let me mark it for 'next'.

Thanks.
Previous: Derrick Stolee via GitGitGadgetNext: Taylor Blau
Message 57 of 75 in “PATH WALK II: Add --path-walk option to 'git pack-objects'”
  1. 00/13 PATH WALK II: Add --path-walk option to 'git pack-objects'Derrick Stolee via GitGitGadget, Mar 10, 2025
  2. 01/13 pack-objects: extract should_attempt_deltas()Derrick Stolee via GitGitGadget, Mar 10, 2025
  3. Taylor BlauMar 12, 2025
  4. Derrick StoleeMar 20, 2025
  5. 03/13 pack-objects: update usage to match docsDerrick Stolee via GitGitGadget, Mar 10, 2025
  6. Taylor BlauMar 12, 2025
  7. 02/13 pack-objects: add --path-walk optionDerrick Stolee via GitGitGadget, Mar 10, 2025
  8. Taylor BlauMar 12, 2025
  9. Derrick StoleeMar 20, 2025
  10. 04/13 p5313: add performance tests for --path-walkDerrick Stolee via GitGitGadget, Mar 10, 2025
  11. 05/13 pack-objects: introduce GIT_TEST_PACK_PATH_WALKDerrick Stolee via GitGitGadget, Mar 10, 2025
  12. 06/13 t5538: add tests to confirm deltas in shallow pushesDerrick Stolee via GitGitGadget, Mar 10, 2025
  13. 07/13 repack: add --path-walk optionDerrick Stolee via GitGitGadget, Mar 10, 2025
  14. 08/13 pack-objects: enable --path-walk via configDerrick Stolee via GitGitGadget, Mar 10, 2025
  15. 09/13 scalar: enable path-walk during push via configDerrick Stolee via GitGitGadget, Mar 10, 2025
  16. 10/13 pack-objects: refactor path-walk delta phaseDerrick Stolee via GitGitGadget, Mar 10, 2025
  17. Taylor BlauMar 12, 2025
  18. Derrick StoleeMar 20, 2025
  19. 11/13 pack-objects: thread the path-based compressionDerrick Stolee via GitGitGadget, Mar 10, 2025
  20. 12/13 path-walk: add new 'edge_aggressive' optionDerrick Stolee via GitGitGadget, Mar 10, 2025
  21. 13/13 pack-objects: allow --shallow and --path-walkDerrick Stolee via GitGitGadget, Mar 10, 2025
  22. Junio C HamanoMar 10, 2025
  23. Taylor BlauMar 12, 2025
  24. Derrick StoleeMar 20, 2025
  25. 00/13 PATH WALK II: Add --path-walk option to 'git pack-objects'Derrick Stolee via GitGitGadget, Mar 24, 2025
  26. 01/13 pack-objects: extract should_attempt_deltas()Derrick Stolee via GitGitGadget, Mar 24, 2025
  27. Taylor BlauMay 2, 2025
  28. 02/13 pack-objects: add --path-walk optionDerrick Stolee via GitGitGadget, Mar 24, 2025
  29. Taylor BlauMay 2, 2025
  30. Derrick StoleeMay 6, 2025
  31. Derrick StoleeMay 16, 2025
  32. 03/13 pack-objects: update usage to match docsDerrick Stolee via GitGitGadget, Mar 24, 2025
  33. 04/13 p5313: add performance tests for --path-walkDerrick Stolee via GitGitGadget, Mar 24, 2025
  34. Taylor BlauMay 2, 2025
  35. 06/13 t5538: add tests to confirm deltas in shallow pushesDerrick Stolee via GitGitGadget, Mar 24, 2025
  36. Taylor BlauMay 2, 2025
  37. Derrick StoleeMay 16, 2025
  38. 05/13 pack-objects: introduce GIT_TEST_PACK_PATH_WALKDerrick Stolee via GitGitGadget, Mar 24, 2025
  39. Taylor BlauMay 2, 2025
  40. Derrick StoleeMay 6, 2025
  41. 07/13 repack: add --path-walk optionDerrick Stolee via GitGitGadget, Mar 24, 2025
  42. Taylor BlauMay 2, 2025
  43. 08/13 pack-objects: enable --path-walk via configDerrick Stolee via GitGitGadget, Mar 24, 2025
  44. Taylor BlauMay 2, 2025
  45. Derrick StoleeMay 6, 2025
  46. Derrick StoleeMay 16, 2025
  47. 09/13 scalar: enable path-walk during push via configDerrick Stolee via GitGitGadget, Mar 24, 2025
  48. Taylor BlauMay 7, 2025
  49. 10/13 pack-objects: refactor path-walk delta phaseDerrick Stolee via GitGitGadget, Mar 24, 2025
  50. Taylor BlauMay 7, 2025
  51. Derrick StoleeMay 16, 2025
  52. Taylor BlauMay 29, 2025
  53. 12/13 path-walk: add new 'edge_aggressive' optionDerrick Stolee via GitGitGadget, Mar 24, 2025
  54. 11/13 pack-objects: thread the path-based compressionDerrick Stolee via GitGitGadget, Mar 24, 2025
  55. Taylor BlauMay 7, 2025
  56. 13/13 pack-objects: allow --shallow and --path-walkDerrick Stolee via GitGitGadget, Mar 24, 2025
  57. Junio C HamanoMay 2, 2025
  58. Taylor BlauMay 2, 2025
  59. Taylor BlauMay 2, 2025
  60. Taylor BlauMay 7, 2025
  61. 00/13 PATH WALK II: Add --path-walk option to 'git pack-objects'Derrick Stolee via GitGitGadget, May 16, 2025
  62. 01/13 pack-objects: extract should_attempt_deltas()Derrick Stolee via GitGitGadget, May 16, 2025
  63. 02/13 pack-objects: add --path-walk optionDerrick Stolee via GitGitGadget, May 16, 2025
  64. 03/13 pack-objects: update usage to match docsDerrick Stolee via GitGitGadget, May 16, 2025
  65. 04/13 p5313: add performance tests for --path-walkDerrick Stolee via GitGitGadget, May 16, 2025
  66. 05/13 pack-objects: introduce GIT_TEST_PACK_PATH_WALKDerrick Stolee via GitGitGadget, May 16, 2025
  67. 06/13 t5538: add tests to confirm deltas in shallow pushesDerrick Stolee via GitGitGadget, May 16, 2025
  68. 07/13 repack: add --path-walk optionDerrick Stolee via GitGitGadget, May 16, 2025
  69. 08/13 pack-objects: enable --path-walk via configDerrick Stolee via GitGitGadget, May 16, 2025
  70. 09/13 scalar: enable path-walk during push via configDerrick Stolee via GitGitGadget, May 16, 2025
  71. 10/13 pack-objects: refactor path-walk delta phaseDerrick Stolee via GitGitGadget, May 16, 2025
  72. 11/13 pack-objects: thread the path-based compressionDerrick Stolee via GitGitGadget, May 16, 2025
  73. 12/13 path-walk: add new 'edge_aggressive' optionDerrick Stolee via GitGitGadget, May 16, 2025
  74. 13/13 pack-objects: allow --shallow and --path-walkDerrick Stolee via GitGitGadget, May 16, 2025
  75. Taylor BlauMay 29, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.