{"thread":{"id":"63083","subject":"[PATCH 00/13] PATH WALK II: Add --path-walk option to 'git pack-objects'","startedAt":"2025-03-10T01:50:58Z","lastAt":"2025-05-29T00:20:27Z","messageCount":75,"participants":["Derrick Stolee via GitGitGadget","Junio C Hamano","Taylor Blau","Derrick Stolee"],"isPatch":true,"patchVersion":1,"patchTotal":13},"messages":[{"id":"513854","messageId":"pull.1819.git.1741571455.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":null,"subject":"[PATCH 00/13] PATH WALK II: Add --path-walk option to 'git pack-objects'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-10T01:50:42Z","receivedAt":"2025-03-10T01:50:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Here is a full submission of the --path-walk feature for 'git pack-objects'\nand 'git repack'. It's been discussed in an RFC [1], as a future application\nfor the path walk API [2], and is updated now that --name-hash-version=2\nexists (as a replacement for the --full-name-hash option from the RFC) [3].\n\n[1]\nhttps://lore.kernel.org/git/pull.1813.v2.git.1729431810.gitgitgadget@gmail.com/\n\n[2]\nhttps://lore.kernel.org/git/pull.1818.git.1730356023.gitgitgadget@gmail.com\n\n[3]\nhttps://lore.kernel.org/git/pull.1813.git.1728396723.gitgitgadget@gmail.com\n\nThis patch series does the following:\n\n 1. Add a new '--path-walk' option to 'git pack-objects' that uses the\n    path-walk API instead of the revision API to collect objects for delta\n    compression.\n\n 2. Add a new '--path-walk' option to 'git repack' to pass this option along\n    to 'git pack-objects'.\n\n 3. Add a new 'pack.usePathWalk' config option to opt into this option\n    implicitly, such as in 'git push'.\n\n 4. Optimize the '--path-walk' option using threading so it better competes\n    with the existing multi-threaded delta compression mechanism.\n\n 5. Update the path-walk API with a new 'edge_aggressive' option that pairs\n    close to the --edge-aggressive option in the revision API. This is\n    useful when creating thin packs inside shallow clones.\n\nThis feature works by using the path-walk API to emit groups of objects that\nappear at the same path. These groups are tracked so they can be tested for\ndelta compression with each other, and then after those groups are tested a\nsecond pass using the name-hash attempts to find better (or first time)\ndeltas across path boundaries. This second pass is much faster than a fresh\npass since the existing deltas are used as a limit for the size of\npotentially new deltas, short-circuiting the checks when the delta size\nexceeds the current-best.\n\nThe benefits of the --path-walk feature first come into play when the name\nhash functions have many collisions, so sorting by name hash value leads to\nunhelpful groupings of objects. Many of these benefits are improved by\n--name-hash-version=2, but collisions still exist with any hash-based\napproach. There are also performance benefits in some cases due to the\nisolation of delta compression testing within path groups.\n\nAll of the benefits of the --path-walk feature are less dramatic when\ncompared to --name-hash-version=2, but they can still exist in many cases. I\nhave also seen some cases where --name-hash-version=2 compresses better than\n--path-walk with --name-hash-version=1, but these options can be combined to\nget the best of both worlds.\n\nDetailed statistics are provided within patch messages, but a few are\nhighlighted here:\n\nThe microsoft/fluentui is a public Javascript repo that suffers from many of\nthe name hash collisions as internal repositories I've worked with. Here is\na comparison of the compressed size and end-to-end time of the repack:\n\nRepack Method    Pack Size       Time\n---------------------------------------\nHash v1             439.4M      87.24s\nHash v2             161.7M      21.51s\nPath Walk           142.5M      28.16s\n\n\nLess dramatic, but perhaps more standardly structured is the nodejs/node\nrepository, with these stats:\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                739.9M      71.18s\nHash v2                764.6M      67.82s\nPath Walk              698.0M      75.10s\n\n\nEven the Linux kernel repository gains some benefits, even though the number\nof hash collisions is relatively low due to a preference for short\nfilenames:\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                  2.5G     554.41s\nHash v2                  2.5G     549.62s\nPath Walk                2.2G     559.00s\n\n\nThe drawbacks of the --path-walk feature is that it will be harder to\nintegrate it with bitmap features, specifically delta islands. This is not\ninsurmountable, but would require more work, such as a revision walk to\npaint objects with reachability information before using that during delta\ncomputations.\n\nHowever, there should still be significant benefits to Git clients trying to\nsave space and improve local performance.\n\nThis feature was shipped with similar features in microsoft/git as of\nv2.47.0.vfs.0.3 [4]. This was used in CI machines for an internal monorepo\nthat had significant repository growth due to constructing a batch of\nbeachball [5] CHANGELOG.[md|json] files and pushing them to a release\nbranch. These pushes were frequently 70-200 MB due to poor delta\ncompression. Using the 'pack.usePathWalk=true' config, these pushes dropped\nin size by 100x while improving performance. Since these CI machines were\nworking with a shallow clone, the 'edge_aggressive' changes were required to\nenable the path-walk option.\n\n[4] https://github.com/microsoft/git/releases/tag/v2.47.0.vfs.0.3\n\n[5] https://github.com/microsoft/beachball\n\nThis version incorporates feedback from previous RFCs and reviewed patch\nseries whenever possible. It also benefits from learned experience, much of\nwhich was already applied in the original path-walk API submission.\n\nThanks, -Stolee\n\nDerrick Stolee (13):\n  pack-objects: extract should_attempt_deltas()\n  pack-objects: add --path-walk option\n  pack-objects: update usage to match docs\n  p5313: add performance tests for --path-walk\n  pack-objects: introduce GIT_TEST_PACK_PATH_WALK\n  t5538: add tests to confirm deltas in shallow pushes\n  repack: add --path-walk option\n  pack-objects: enable --path-walk via config\n  scalar: enable path-walk during push via config\n  pack-objects: refactor path-walk delta phase\n  pack-objects: thread the path-based compression\n  path-walk: add new 'edge_aggressive' option\n  pack-objects: allow --shallow and --path-walk\n\n Documentation/config/feature.adoc          |   4 +\n Documentation/config/pack.adoc             |   8 +\n Documentation/git-pack-objects.adoc        |  25 +-\n Documentation/git-repack.adoc              |  14 +-\n Documentation/technical/api-path-walk.adoc |   9 +\n builtin/pack-objects.c                     | 411 +++++++++++++++++++--\n builtin/repack.c                           |   7 +-\n pack-objects.h                             |  12 +\n path-walk.c                                |   6 +-\n path-walk.h                                |   7 +\n repo-settings.c                            |   3 +\n repo-settings.h                            |   1 +\n scalar.c                                   |   1 +\n t/README                                   |   4 +\n t/helper/test-path-walk.c                  |   2 +\n t/perf/p5313-pack-objects.sh               |  37 +-\n t/t0411-clone-from-partial.sh              |   6 +\n t/t0450/adoc-help-mismatches               |   1 -\n t/t5300-pack-object.sh                     |  19 +\n t/t5306-pack-nobase.sh                     |   5 +\n t/t5310-pack-bitmaps.sh                    |  13 +-\n t/t5316-pack-delta-depth.sh                |   9 +-\n t/t5332-multi-pack-reuse.sh                |   7 +\n t/t5538-push-shallow.sh                    |  34 ++\n t/t6601-path-walk.sh                       |  20 +\n t/t7406-submodule-update.sh                |   3 +\n 26 files changed, 601 insertions(+), 67 deletions(-)\n\n\nbase-commit: a36e024e989f4d35f35987a60e3af8022cac3420\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1819%2Fderrickstolee%2Fpath-walk-upstream-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1819/derrickstolee/path-walk-upstream-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1819\n-- \ngitgitgadget\n"},{"id":"513855","messageId":"a2ed1f2d4e3946c563f934fcaf149050d50f255f.1741571455.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.git.1741571455.gitgitgadget@gmail.com","subject":"[PATCH 01/13] pack-objects: extract should_attempt_deltas()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-10T01:50:43Z","receivedAt":"2025-03-10T01:50:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThis will be helpful in a future change, which will reuse this logic.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c | 53 +++++++++++++++++++++++-------------------\n 1 file changed, 29 insertions(+), 24 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 58a9b161262..1d0992a8dac 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3196,6 +3196,33 @@ static int add_ref_tag(const char *tag UNUSED, const char *referent UNUSED, cons\n \treturn 0;\n }\n \n+static int should_attempt_deltas(struct object_entry *entry)\n+{\n+\tif (DELTA(entry))\n+\t\treturn 0;\n+\n+\tif (!entry->type_valid ||\n+\t    oe_size_less_than(&to_pack, entry, 50))\n+\t\treturn 0;\n+\n+\tif (entry->no_try_delta)\n+\t\treturn 0;\n+\n+\tif (!entry->preferred_base) {\n+\t\tif (oe_type(entry) < 0)\n+\t\t\tdie(_(\"unable to get type of object %s\"),\n+\t\t\t\toid_to_hex(&entry->idx.oid));\n+\t} else if (oe_type(entry) < 0) {\n+\t\t/*\n+\t\t * This object is not found, but we\n+\t\t * don't have to include it anyway.\n+\t\t */\n+\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+}\n+\n static void prepare_pack(int window, int depth)\n {\n \tstruct object_entry **delta_list;\n@@ -3226,33 +3253,11 @@ static void prepare_pack(int window, int depth)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = to_pack.objects + i;\n \n-\t\tif (DELTA(entry))\n-\t\t\t/* This happens if we decided to reuse existing\n-\t\t\t * delta from a pack.  \"reuse_delta &&\" is implied.\n-\t\t\t */\n-\t\t\tcontinue;\n-\n-\t\tif (!entry->type_valid ||\n-\t\t    oe_size_less_than(&to_pack, entry, 50))\n+\t\tif (!should_attempt_deltas(entry))\n \t\t\tcontinue;\n \n-\t\tif (entry->no_try_delta)\n-\t\t\tcontinue;\n-\n-\t\tif (!entry->preferred_base) {\n+\t\tif (!entry->preferred_base)\n \t\t\tnr_deltas++;\n-\t\t\tif (oe_type(entry) < 0)\n-\t\t\t\tdie(_(\"unable to get type of object %s\"),\n-\t\t\t\t    oid_to_hex(&entry->idx.oid));\n-\t\t} else {\n-\t\t\tif (oe_type(entry) < 0) {\n-\t\t\t\t/*\n-\t\t\t\t * This object is not found, but we\n-\t\t\t\t * don't have to include it anyway.\n-\t\t\t\t */\n-\t\t\t\tcontinue;\n-\t\t\t}\n-\t\t}\n \n \t\tdelta_list[n++] = entry;\n \t}\n-- \ngitgitgadget\n\n"},{"id":"513856","messageId":"bc678acb109c34ac96c641c5e6fbfb418e447010.1741571455.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.git.1741571455.gitgitgadget@gmail.com","subject":"[PATCH 03/13] pack-objects: update usage to match docs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-10T01:50:45Z","receivedAt":"2025-03-10T01:51:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThe t0450 test script verifies that builtin usage matches the synopsis\nin the documentation. Adjust the builtin to match and then remove 'git\npack-objects' from the exception list.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-pack-objects.adoc | 14 +++++++-------\n builtin/pack-objects.c              | 10 ++++++++--\n t/t0450/adoc-help-mismatches        |  1 -\n 3 files changed, 15 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc\nindex 7dbbe6d54d2..ad765334729 100644\n--- a/Documentation/git-pack-objects.adoc\n+++ b/Documentation/git-pack-objects.adoc\n@@ -10,13 +10,13 @@ SYNOPSIS\n --------\n [verse]\n 'git pack-objects' [-q | --progress | --all-progress] [--all-progress-implied]\n-\t[--no-reuse-delta] [--delta-base-offset] [--non-empty]\n-\t[--local] [--incremental] [--window=<n>] [--depth=<n>]\n-\t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n-\t[--cruft] [--cruft-expiration=<time>]\n-\t[--stdout [--filter=<filter-spec>] | <base-name>]\n-\t[--shallow] [--keep-true-parents] [--[no-]sparse]\n-\t[--name-hash-version=<n>] [--path-walk] < <object-list>\n+\t\t   [--no-reuse-delta] [--delta-base-offset] [--non-empty]\n+\t\t   [--local] [--incremental] [--window=<n>] [--depth=<n>]\n+\t\t   [--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n+\t\t   [--cruft] [--cruft-expiration=<time>]\n+\t\t   [--stdout [--filter=<filter-spec>] | <base-name>]\n+\t\t   [--shallow] [--keep-true-parents] [--[no-]sparse]\n+\t\t   [--name-hash-version=<n>] [--path-walk] < <object-list>\n \n \n DESCRIPTION\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5596c409927..fe23db3c5d8 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -187,8 +187,14 @@ static inline void oe_set_delta_size(struct packing_data *pack,\n #define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n \n static const char *pack_usage[] = {\n-\tN_(\"git pack-objects --stdout [<options>] [< <ref-list> | < <object-list>]\"),\n-\tN_(\"git pack-objects [<options>] <base-name> [< <ref-list> | < <object-list>]\"),\n+\tN_(\"git pack-objects [-q | --progress | --all-progress] [--all-progress-implied]\\n\"\n+\t   \"                 [--no-reuse-delta] [--delta-base-offset] [--non-empty]\\n\"\n+\t   \"                 [--local] [--incremental] [--window=<n>] [--depth=<n>]\\n\"\n+\t   \"                 [--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\\n\"\n+\t   \"                 [--cruft] [--cruft-expiration=<time>]\\n\"\n+\t   \"                 [--stdout [--filter=<filter-spec>] | <base-name>]\\n\"\n+\t   \"                 [--shallow] [--keep-true-parents] [--[no-]sparse]\\n\"\n+\t   \"                 [--name-hash-version=<n>] [--path-walk] < <object-list>\"),\n \tNULL\n };\n \ndiff --git a/t/t0450/adoc-help-mismatches b/t/t0450/adoc-help-mismatches\nindex c4a15fd0cb8..06b469bdee2 100644\n--- a/t/t0450/adoc-help-mismatches\n+++ b/t/t0450/adoc-help-mismatches\n@@ -38,7 +38,6 @@ merge-one-file\n multi-pack-index\n name-rev\n notes\n-pack-objects\n push\n range-diff\n rebase\n-- \ngitgitgadget\n\n"},{"id":"513857","messageId":"9b31dc87bb61f4d73eced02a24baea58bc51aa5e.1741571455.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.git.1741571455.gitgitgadget@gmail.com","subject":"[PATCH 02/13] pack-objects: add --path-walk option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-10T01:50:44Z","receivedAt":"2025-03-10T01:51:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nIn order to more easily compute delta bases among objects that appear at\nthe exact same path, add a --path-walk option to 'git pack-objects'.\n\nThis option will use the path-walk API instead of the object walk given\nby the revision machinery. Since objects will be provided in batches\nrepresenting a common path, those objects can be tested for delta bases\nimmediately instead of waiting for a sort of the full object list by\nname-hash. This has multiple benefits, including avoiding collisions by\nname-hash.\n\nThe objects marked as UNINTERESTING are included in these batches, so we\nare guaranteeing some locality to find good delta bases.\n\nAfter the individual passes are done on a per-path basis, the default\nname-hash is used to find other opportunistic delta bases that did not\nmatch exactly by the full path name.\n\nThe current implementation performs delta calculations while walking\nobjects, which is not ideal for a few reasons. First, this will cause\nthe \"Enumerating objects\" phase to be much longer than usual. Second, it\ndoes not take advantage of threading during the path-scoped delta\ncalculations. Even with this lack of threading, the path-walk option is\nsometimes faster than the usual approach. Future changes will refactor\nthis code to allow for threading, but that complexity is deferred until\nlater to keep this patch as simple as possible.\n\nThis new walk is incompatible with some features and is ignored by\nothers:\n\n * Object filters are not currently integrated with the path-walk API,\n   such as sparse-checkout or tree depth. A blobless packfile could be\n   integrated easily, but that is deferred for later.\n\n * Server-focused features such as delta islands, shallow packs, and\n   using a bitmap index are incompatible with the path-walk API.\n\n * The path walk API is only compatible with the --revs option, not\n   taking object lists or pack lists over stdin. These alternative ways\n   to specify the objects currently ignores the --path-walk option\n   without even a warning.\n\nFuture changes will create performance tests that demonstrate the power\nof this approach.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-pack-objects.adoc        |  13 +-\n Documentation/technical/api-path-walk.adoc |   1 +\n builtin/pack-objects.c                     | 147 +++++++++++++++++++--\n t/t5300-pack-object.sh                     |  15 +++\n 4 files changed, 166 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc\nindex 7f69ae4855f..7dbbe6d54d2 100644\n--- a/Documentation/git-pack-objects.adoc\n+++ b/Documentation/git-pack-objects.adoc\n@@ -16,7 +16,7 @@ SYNOPSIS\n \t[--cruft] [--cruft-expiration=<time>]\n \t[--stdout [--filter=<filter-spec>] | <base-name>]\n \t[--shallow] [--keep-true-parents] [--[no-]sparse]\n-\t[--name-hash-version=<n>] < <object-list>\n+\t[--name-hash-version=<n>] [--path-walk] < <object-list>\n \n \n DESCRIPTION\n@@ -375,6 +375,17 @@ many different directories. At the moment, this version is not allowed\n when writing reachability bitmap files with `--write-bitmap-index` and it\n will be automatically changed to version `1`.\n \n+--path-walk::\n+\tBy default, `git pack-objects` walks objects in an order that\n+\tpresents trees and blobs in an order unrelated to the path they\n+\tappear relative to a commit's root tree. The `--path-walk` option\n+\tenables a different walking algorithm that organizes trees and\n+\tblobs by path. This has the potential to improve delta compression\n+\tespecially in the presence of filenames that cause collisions in\n+\tGit's default name-hash algorithm. Due to changing how the objects\n+\tare walked, this option is not compatible with `--delta-islands`,\n+\t`--shallow`, or `--filter`.\n+\n \n DELTA ISLANDS\n -------------\ndiff --git a/Documentation/technical/api-path-walk.adoc b/Documentation/technical/api-path-walk.adoc\nindex 3e089211fb4..e522695dd9f 100644\n--- a/Documentation/technical/api-path-walk.adoc\n+++ b/Documentation/technical/api-path-walk.adoc\n@@ -69,4 +69,5 @@ Examples\n \n See example usages in:\n \t`t/helper/test-path-walk.c`,\n+\t`builtin/pack-objects.c`,\n \t`builtin/backfill.c`\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1d0992a8dac..5596c409927 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -41,6 +41,9 @@\n #include \"promisor-remote.h\"\n #include \"pack-mtimes.h\"\n #include \"parse-options.h\"\n+#include \"blob.h\"\n+#include \"tree.h\"\n+#include \"path-walk.h\"\n \n /*\n  * Objects we are going to pack are collected in the `to_pack` structure.\n@@ -217,6 +220,7 @@ static int delta_search_threads;\n static int pack_to_stdout;\n static int sparse;\n static int thin;\n+static int path_walk;\n static int num_preferred_base;\n static struct progress *progress_state;\n \n@@ -4188,6 +4192,105 @@ static void mark_bitmap_preferred_tips(void)\n \t}\n }\n \n+static inline int is_oid_interesting(struct repository *repo,\n+\t\t\t\t     struct object_id *oid)\n+{\n+\tstruct object *o = lookup_object(repo, oid);\n+\treturn o && !(o->flags & UNINTERESTING);\n+}\n+\n+static int add_objects_by_path(const char *path,\n+\t\t\t       struct oid_array *oids,\n+\t\t\t       enum object_type type,\n+\t\t\t       void *data)\n+{\n+\tstruct object_entry **delta_list;\n+\tsize_t oe_start = to_pack.nr_objects;\n+\tsize_t oe_end;\n+\tunsigned int sub_list_size;\n+\tunsigned int *processed = data;\n+\n+\t/*\n+\t * First, add all objects to the packing data, including the ones\n+\t * marked UNINTERESTING (translated to 'exclude') as they can be\n+\t * used as delta bases.\n+\t */\n+\tfor (size_t i = 0; i < oids->nr; i++) {\n+\t\tint exclude;\n+\t\tstruct object_info oi = OBJECT_INFO_INIT;\n+\t\tstruct object_id *oid = &oids->oid[i];\n+\n+\t\t/* Skip objects that do not exist locally. */\n+\t\tif (exclude_promisor_objects &&\n+\t\t    oid_object_info_extended(the_repository, oid, &oi,\n+\t\t\t\t\t     OBJECT_INFO_FOR_PREFETCH) < 0)\n+\t\t\tcontinue;\n+\n+\t\texclude = !is_oid_interesting(the_repository, oid);\n+\n+\t\tif (exclude && !thin)\n+\t\t\tcontinue;\n+\n+\t\tadd_object_entry(oid, type, path, exclude);\n+\t}\n+\n+\toe_end = to_pack.nr_objects;\n+\n+\t/* We can skip delta calculations if it is a no-op. */\n+\tif (oe_end == oe_start || !window)\n+\t\treturn 0;\n+\n+\tsub_list_size = 0;\n+\tALLOC_ARRAY(delta_list, oe_end - oe_start);\n+\n+\tfor (size_t i = 0; i < oe_end - oe_start; i++) {\n+\t\tstruct object_entry *entry = to_pack.objects + oe_start + i;\n+\n+\t\tif (!should_attempt_deltas(entry))\n+\t\t\tcontinue;\n+\n+\t\tdelta_list[sub_list_size++] = entry;\n+\t}\n+\n+\t/*\n+\t * Find delta bases among this list of objects that all match the same\n+\t * path. This causes the delta compression to be interleaved in the\n+\t * object walk, which can lead to confusing progress indicators. This is\n+\t * also incompatible with threaded delta calculations. In the future,\n+\t * consider creating a list of regions in the full to_pack.objects array\n+\t * that could be picked up by the threaded delta computation.\n+\t */\n+\tif (sub_list_size && window) {\n+\t\tQSORT(delta_list, sub_list_size, type_size_sort);\n+\t\tfind_deltas(delta_list, &sub_list_size, window, depth, processed);\n+\t}\n+\n+\tfree(delta_list);\n+\treturn 0;\n+}\n+\n+static void get_object_list_path_walk(struct rev_info *revs)\n+{\n+\tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n+\tunsigned int processed = 0;\n+\n+\tinfo.revs = revs;\n+\tinfo.path_fn = add_objects_by_path;\n+\tinfo.path_fn_data = &processed;\n+\trevs->tag_objects = 1;\n+\n+\t/*\n+\t * Allow the --[no-]sparse option to be interesting here, if only\n+\t * for testing purposes. Paths with no interesting objects will not\n+\t * contribute to the resulting pack, but only create noisy preferred\n+\t * base objects.\n+\t */\n+\tinfo.prune_all_uninteresting = sparse;\n+\n+\tif (walk_objects_by_path(&info))\n+\t\tdie(_(\"failed to pack objects via path-walk\"));\n+}\n+\n static void get_object_list(struct rev_info *revs, int ac, const char **av)\n {\n \tstruct setup_revision_opt s_r_opt = {\n@@ -4234,7 +4337,7 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n \n \twarn_on_object_refname_ambiguity = save_warning;\n \n-\tif (use_bitmap_index && !get_object_list_from_bitmap(revs))\n+\tif (use_bitmap_index && !path_walk && !get_object_list_from_bitmap(revs))\n \t\treturn;\n \n \tif (use_delta_islands)\n@@ -4243,15 +4346,19 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n \tif (write_bitmap_index)\n \t\tmark_bitmap_preferred_tips();\n \n-\tif (prepare_revision_walk(revs))\n-\t\tdie(_(\"revision walk setup failed\"));\n-\tmark_edges_uninteresting(revs, show_edge, sparse);\n-\n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n-\ttraverse_commit_list(revs,\n-\t\t\t     show_commit, fn_show_object,\n-\t\t\t     NULL);\n+\n+\tif (path_walk) {\n+\t\tget_object_list_path_walk(revs);\n+\t} else {\n+\t\tif (prepare_revision_walk(revs))\n+\t\t\tdie(_(\"revision walk setup failed\"));\n+\t\tmark_edges_uninteresting(revs, show_edge, sparse);\n+\t\ttraverse_commit_list(revs,\n+\t\t\t\tshow_commit, fn_show_object,\n+\t\t\t\tNULL);\n+\t}\n \n \tif (unpack_unreachable_expiration) {\n \t\trevs->ignore_missing_links = 1;\n@@ -4461,6 +4568,8 @@ int cmd_pack_objects(int argc,\n \t\t\t N_(\"use the sparse reachability algorithm\")),\n \t\tOPT_BOOL(0, \"thin\", &thin,\n \t\t\t N_(\"create thin packs\")),\n+\t\tOPT_BOOL(0, \"path-walk\", &path_walk,\n+\t\t\t N_(\"use the path-walk API to walk objects when possible\")),\n \t\tOPT_BOOL(0, \"shallow\", &shallow,\n \t\t\t N_(\"create packs suitable for shallow fetches\")),\n \t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep_on_disk,\n@@ -4546,7 +4655,27 @@ int cmd_pack_objects(int argc,\n \t\twindow = 0;\n \n \tstrvec_push(&rp, \"pack-objects\");\n-\tif (thin) {\n+\n+\tif (path_walk && filter_options.choice) {\n+\t\twarning(_(\"cannot use --filter with --path-walk\"));\n+\t\tpath_walk = 0;\n+\t}\n+\tif (path_walk && use_delta_islands) {\n+\t\twarning(_(\"cannot use delta islands with --path-walk\"));\n+\t\tpath_walk = 0;\n+\t}\n+\tif (path_walk && shallow) {\n+\t\twarning(_(\"cannot use --shallow with --path-walk\"));\n+\t\tpath_walk = 0;\n+\t}\n+\tif (path_walk) {\n+\t\tstrvec_push(&rp, \"--boundary\");\n+\t\t /*\n+\t\t  * We must disable the bitmaps because we are removing\n+\t\t  * the --objects / --objects-edge[-aggressive] options.\n+\t\t  */\n+\t\tuse_bitmap_index = 0;\n+\t} else if (thin) {\n \t\tuse_internal_rev_list = 1;\n \t\tstrvec_push(&rp, shallow\n \t\t\t\t? \"--objects-edge-aggressive\"\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 5ac8d39094b..16420d12863 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -723,4 +723,19 @@ test_expect_success '--name-hash-version=2 and --write-bitmap-index are incompat\n \t! test_grep \"currently, --write-bitmap-index requires --name-hash-version=1\" err\n '\n \n+test_expect_success '--path-walk pack everything' '\n+\tgit -C server rev-parse HEAD >in &&\n+\tgit -C server pack-objects --stdout --revs --path-walk <in >out.pack &&\n+\tgit -C server index-pack --stdin <out.pack\n+'\n+\n+test_expect_success '--path-walk thin pack' '\n+\tcat >in <<-EOF &&\n+\t$(git -C server rev-parse HEAD)\n+\t^$(git -C server rev-parse HEAD~2)\n+\tEOF\n+\tgit -C server pack-objects --thin --stdout --revs --path-walk <in >out.pack &&\n+\tgit -C server index-pack --fix-thin --stdin <out.pack\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"513858","messageId":"de848ebff74b3ea461d6b750454d4b50af23bedc.1741571455.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.git.1741571455.gitgitgadget@gmail.com","subject":"[PATCH 04/13] p5313: add performance tests for --path-walk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-10T01:50:46Z","receivedAt":"2025-03-10T01:51:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThe previous change added a --path-walk option to 'git pack-objects'.\nCreate a performance test that demonstrates the time and space benefits\nof the feature.\n\nIn order to get an appropriate comparison, we need to avoid reusing\ndeltas and recompute them from scratch.\n\nCompare the creation of a thin pack representing a small push and the\ncreation of a relatively large non-thin pack.\n\nRunning on my copy of the Git repository results in this data (removing\nthe repack tests for --name-hash-version):\n\nTest                                                     this tree\n------------------------------------------------------------------------\n5313.2: thin pack with --name-hash-version=1             0.02(0.01+0.01)\n5313.3: thin pack size with --name-hash-version=1                   1.6K\n5313.4: big pack with --name-hash-version=1              2.55(4.20+0.26)\n5313.5: big pack size with --name-hash-version=1                   16.4M\n5313.6: shallow fetch pack with --name-hash-version=1    1.24(2.03+0.08)\n5313.7: shallow pack size with --name-hash-version=1               12.2M\n5313.10: thin pack with --name-hash-version=2            0.03(0.01+0.01)\n5313.11: thin pack size with --name-hash-version=2                  1.6K\n5313.12: big pack with --name-hash-version=2             1.91(3.23+0.20)\n5313.13: big pack size with --name-hash-version=2                  16.4M\n5313.14: shallow fetch pack with --name-hash-version=2   1.06(1.57+0.10)\n5313.15: shallow pack size with --name-hash-version=2              12.5M\n5313.18: thin pack with --path-walk                      0.03(0.01+0.01)\n5313.19: thin pack size with --path-walk                            1.6K\n5313.20: big pack with --path-walk                       2.05(3.24+0.27)\n5313.21: big pack size with --path-walk                            16.3M\n5313.22: shallow fetch pack with --path-walk             1.08(1.66+0.07)\n5313.23: shallow pack size with --path-walk                        12.4M\n\nThis can be reformatted as follows:\n\nPack Type            Hash v1   Hash v2     Path Walk\n---------------------------------------------------\nthin pack    (time)    0.02s      0.03s      0.03s\n             (size)    1.6K       1.6K       1.6K\nbig pack     (time)    2.55s      1.91s      2.05s\n             (size)   16.4M      16.4M      16.3M\nshallow pack (time)    1.24s      1.06s      1.08s\n             (size)   12.2M      12.5M      12.4M\n\nNote that the timing is slower because there is no threading in the\n--path-walk case (yet). Also, the shallow pack cases are really not\nusing the --path-walk logic right now because it is disabled until some\nadditions are made to the path walk API.\n\nThe cases where the --path-walk option really shines is when the default\nname-hash is overwhelmed with collisions. An open source example can be\nfound in the microsoft/fluentui repo [1] at a certain commit [2].\n\n[1] https://github.com/microsoft/fluentui\n[2] e70848ebac1cd720875bccaa3026f4a9ed700e08\n\nRunning the tests on this repo results in the following comparison table:\n\nPack Type            Hash v1    Hash v2    Path Walk\n---------------------------------------------------\nthin pack    (time)    0.36s      0.12s      0.08s\n             (size)    1.2M      22.0K      18.4K\nbig pack     (time)    2.00s      2.90s      2.21s\n             (size)   20.4M      25.9M      19.5M\nshallow pack (time)    1.41s      1.80s      1.65s\n             (size)   34.4M      33.7M      33.6M\n\nNotice in particular that in the small thin pack, the time performance\nhas improved from 0.36s for --name-hash-version=1 to 0.08s and this is\nlikely due to the improved size of the resulting pack: 18.4K instead of\n1.2M.  The relatively new --name-hash-version=2 is competitive with\n--path-walk (0.12s and 22.0K) but not quite as successful.\n\nFinally, running this on a copy of the Linux kernel repository results\nin these data points:\n\nPack Type            Hash v1    Hash v2    Path Walk\n---------------------------------------------------\nthin pack    (time)    0.03s      0.13s      0.03s\n             (size)    4.6K       4.6K       4.6K\nbig pack     (time)   15.29s     12.32s     13.92s\n             (size)  201.1M     159.1M     158.5M\nshallow pack (time)   10.88s     22.93s     22.74s\n             (size)  269.2M     273.8M     267.7M\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n t/perf/p5313-pack-objects.sh | 37 ++++++++++++++++++++++--------------\n 1 file changed, 23 insertions(+), 14 deletions(-)\n\ndiff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh\nindex be5229a0ecd..cd6dd3abb71 100755\n--- a/t/perf/p5313-pack-objects.sh\n+++ b/t/perf/p5313-pack-objects.sh\n@@ -25,46 +25,55 @@ test_expect_success 'create rev input' '\n \tEOF\n '\n \n-for version in 1 2\n-do\n-\texport version\n+test_all_with_args () {\n+\tparameter=$1\n+\texport parameter\n \n-\ttest_perf \"thin pack with version $version\" '\n+\ttest_perf \"thin pack with $parameter\" '\n \t\tgit pack-objects --thin --stdout --revs --sparse \\\n-\t\t\t--name-hash-version=$version <in-thin >out\n+\t\t\t$parameter <in-thin >out\n \t'\n \n-\ttest_size \"thin pack size with version $version\" '\n+\ttest_size \"thin pack size with $parameter\" '\n \t\ttest_file_size out\n \t'\n \n-\ttest_perf \"big pack with version $version\" '\n+\ttest_perf \"big pack with $parameter\" '\n \t\tgit pack-objects --stdout --revs --sparse \\\n-\t\t\t--name-hash-version=$version <in-big >out\n+\t\t\t$parameter <in-big >out\n \t'\n \n-\ttest_size \"big pack size with version $version\" '\n+\ttest_size \"big pack size with $parameter\" '\n \t\ttest_file_size out\n \t'\n \n-\ttest_perf \"shallow fetch pack with version $version\" '\n+\ttest_perf \"shallow fetch pack with $parameter\" '\n \t\tgit pack-objects --stdout --revs --sparse --shallow \\\n-\t\t\t--name-hash-version=$version <in-shallow >out\n+\t\t\t$parameter <in-shallow >out\n \t'\n \n-\ttest_size \"shallow pack size with version $version\" '\n+\ttest_size \"shallow pack size with $parameter\" '\n \t\ttest_file_size out\n \t'\n+}\n \n-\ttest_perf \"repack with version $version\" '\n+for version in 1 2\n+do\n+\texport version\n+\n+\ttest_all_with_args --name-hash-version=$version\n+\n+\ttest_perf \"repack with --name-hash-version=$version\" '\n \t\tgit repack -adf --name-hash-version=$version\n \t'\n \n-\ttest_size \"repack size with version $version\" '\n+\ttest_size \"repack size with --name-hash-version=$version\" '\n \t\tgitdir=$(git rev-parse --git-dir) &&\n \t\tpack=$(ls $gitdir/objects/pack/pack-*.pack) &&\n \t\ttest_file_size \"$pack\"\n \t'\n done\n \n+test_all_with_args --path-walk\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"513860","messageId":"4bfd438b5fa3c08ae639f8b4d1daf7a5f2d3514d.1741571455.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.git.1741571455.gitgitgadget@gmail.com","subject":"[PATCH 05/13] pack-objects: introduce GIT_TEST_PACK_PATH_WALK","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-10T01:50:47Z","receivedAt":"2025-03-10T01:51:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThere are many tests that validate whether 'git pack-objects' works as\nexpected. Instead of duplicating these tests, add a new test environment\nvariable, GIT_TEST_PACK_PATH_WALK, that implies --path-walk by default\nwhen specified.\n\nThis was useful in testing the implementation of the --path-walk\nimplementation, especially in conjunction with test such as:\n\n - t0411-clone-from-partial.sh : One test fetches from a repo that does\n   not have the boundary objects. This causes the path-based walk to\n   fail. Disable the variable for this test.\n\n - t5306-pack-nobase.sh : Similar to t0411, one test fetches from a repo\n   without a boundary object.\n\n - t5310-pack-bitmaps.sh : One test compares the case when packing with\n   bitmaps to the case when packing without them. Since we disable the\n   test variable when writing bitmaps, this causes a difference in the\n   object list (the --path-walk option adds an extra object). Specify\n   --no-path-walk in both processes for the comparison. Another test\n   checks for a specific delta base, but when computing dynamically\n   without using bitmaps, the base object it too small to be considered\n   in the delta calculations so no base is used.\n\n - t5316-pack-delta-depth.sh : This script cares about certain delta\n   choices and their chain lengths. The --path-walk option changes how\n   these chains are selected, and thus changes the results of this test.\n\n - t5322-pack-objects-sparse.sh : This demonstrates the effectiveness of\n   the --sparse option and how it combines with --path-walk.\n\n - t5332-multi-pack-reuse.sh : This test verifies that the preferred\n   pack is used for delta reuse when possible. The --path-walk option is\n   not currently aware of the preferred pack at all, so finds a\n   different delta base.\n\n - t7406-submodule-update.sh : When using the variable, the --depth\n   option collides with the --path-walk feature, resulting in a warning\n   message. Disable the variable so this warning does not appear.\n\nI want to call out one specific test change that is only temporary:\n\n - t5530-upload-pack-error.sh : One test cares specifically about an\n   \"unable to read\" error message. Since the current implementation\n   performs delta calculations within the path-walk API callback, a\n   different \"unable to get size\" error message appears. When this\n   is changed in a future refactoring, this test change can be reverted.\n\nSimilar to GIT_TEST_NAME_HASH_VERSION, we do not add this option to the\nlinux-TEST-vars CI build as that's already an overloaded build.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c        | 12 ++++++++++--\n t/README                      |  4 ++++\n t/t0411-clone-from-partial.sh |  6 ++++++\n t/t5306-pack-nobase.sh        |  5 +++++\n t/t5310-pack-bitmaps.sh       | 13 +++++++++++--\n t/t5316-pack-delta-depth.sh   |  9 ++++++---\n t/t5332-multi-pack-reuse.sh   |  7 +++++++\n t/t5530-upload-pack-error.sh  |  6 ++++++\n t/t7406-submodule-update.sh   |  3 +++\n 9 files changed, 58 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex fe23db3c5d8..120202b05e9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -226,7 +226,7 @@ static int delta_search_threads;\n static int pack_to_stdout;\n static int sparse;\n static int thin;\n-static int path_walk;\n+static int path_walk = -1;\n static int num_preferred_base;\n static struct progress *progress_state;\n \n@@ -4227,7 +4227,7 @@ static int add_objects_by_path(const char *path,\n \t\tstruct object_id *oid = &oids->oid[i];\n \n \t\t/* Skip objects that do not exist locally. */\n-\t\tif (exclude_promisor_objects &&\n+\t\tif ((exclude_promisor_objects || arg_missing_action != MA_ERROR) &&\n \t\t    oid_object_info_extended(the_repository, oid, &oi,\n \t\t\t\t\t     OBJECT_INFO_FOR_PREFETCH) < 0)\n \t\t\tcontinue;\n@@ -4645,6 +4645,14 @@ int cmd_pack_objects(int argc,\n \tif (pack_to_stdout != !base_name || argc)\n \t\tusage_with_options(pack_usage, pack_objects_options);\n \n+\tif (path_walk < 0) {\n+\t\tif (use_bitmap_index > 0 ||\n+\t\t    !use_internal_rev_list)\n+\t\t\tpath_walk = 0;\n+\t\telse\n+\t\t\tpath_walk = git_env_bool(\"GIT_TEST_PACK_PATH_WALK\", 0);\n+\t}\n+\n \tif (depth < 0)\n \t\tdepth = 0;\n \tif (depth >= (1 << OE_DEPTH_BITS)) {\ndiff --git a/t/README b/t/README\nindex 53e5b4a7107..ae06e628815 100644\n--- a/t/README\n+++ b/t/README\n@@ -415,6 +415,10 @@ GIT_TEST_PACK_SPARSE=<boolean> if disabled will default the pack-objects\n builtin to use the non-sparse object walk. This can still be overridden by\n the --sparse command-line argument.\n \n+GIT_TEST_PACK_PATH_WALK=<boolean> if enabled will default the pack-objects\n+builtin to use the path-walk API for the object walk. This can still be\n+overridden by the --no-path-walk command-line argument.\n+\n GIT_TEST_PRELOAD_INDEX=<boolean> exercises the preload-index code path\n by overriding the minimum number of cache entries required per thread.\n \ndiff --git a/t/t0411-clone-from-partial.sh b/t/t0411-clone-from-partial.sh\nindex 196fc617843..9e6bca56255 100755\n--- a/t/t0411-clone-from-partial.sh\n+++ b/t/t0411-clone-from-partial.sh\n@@ -59,6 +59,12 @@ test_expect_success 'pack-objects should fetch from promisor remote and execute\n \n test_expect_success 'clone from promisor remote does not lazy-fetch by default' '\n \trm -f script-executed &&\n+\n+\t# The --path-walk feature of \"git pack-objects\" is not\n+\t# compatible with this kind of fetch from an incomplete repo.\n+\tGIT_TEST_PACK_PATH_WALK=0 &&\n+\texport GIT_TEST_PACK_PATH_WALK &&\n+\n \ttest_must_fail git clone evil no-lazy 2>err &&\n \ttest_grep \"lazy fetching disabled\" err &&\n \ttest_path_is_missing script-executed\ndiff --git a/t/t5306-pack-nobase.sh b/t/t5306-pack-nobase.sh\nindex 805d60ff317..609399d54fb 100755\n--- a/t/t5306-pack-nobase.sh\n+++ b/t/t5306-pack-nobase.sh\n@@ -59,6 +59,11 @@ test_expect_success 'indirectly clone patch_clone' '\n \t git pull ../.git &&\n \t test $(git rev-parse HEAD) = $B &&\n \n+\t# The --path-walk feature of \"git pack-objects\" is not\n+\t# compatible with this kind of fetch from an incomplete repo.\n+\tGIT_TEST_PACK_PATH_WALK=0 &&\n+\texport GIT_TEST_PACK_PATH_WALK &&\n+\n \t git pull ../patch_clone/.git &&\n \t test $(git rev-parse HEAD) = $C\n \t)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 621bbbdd26e..e01df807a62 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -158,8 +158,9 @@ test_bitmap_cases () {\n \t\tls .git/objects/pack/ | grep bitmap >output &&\n \t\ttest_line_count = 1 output &&\n \t\t# verify equivalent packs are generated with/without using bitmap index\n-\t\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa </dev/null) &&\n-\t\tpackbsha1=$(git pack-objects --use-bitmap-index --all packb </dev/null) &&\n+\t\t# Be careful to not use the path-walk option in either case.\n+\t\tpackasha1=$(git pack-objects --no-use-bitmap-index --no-path-walk --all packa </dev/null) &&\n+\t\tpackbsha1=$(git pack-objects --use-bitmap-index --no-path-walk --all packb </dev/null) &&\n \t\tlist_packed_objects packa-$packasha1.idx >packa.objects &&\n \t\tlist_packed_objects packb-$packbsha1.idx >packb.objects &&\n \t\ttest_cmp packa.objects packb.objects\n@@ -388,6 +389,14 @@ test_bitmap_cases () {\n \t\tgit init --bare client.git &&\n \t\t(\n \t\t\tcd client.git &&\n+\n+\t\t\t# This test relies on reusing a delta, but if the\n+\t\t\t# path-walk machinery is engaged, the base object\n+\t\t\t# is considered too small to use during the\n+\t\t\t# dynamic computation, so is not used.\n+\t\t\tGIT_TEST_PACK_PATH_WALK=0 &&\n+\t\t\texport GIT_TEST_PACK_PATH_WALK &&\n+\n \t\t\tgit config transfer.unpackLimit 1 &&\n \t\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n \t\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\ndiff --git a/t/t5316-pack-delta-depth.sh b/t/t5316-pack-delta-depth.sh\nindex 32cf4227451..167c3a35234 100755\n--- a/t/t5316-pack-delta-depth.sh\n+++ b/t/t5316-pack-delta-depth.sh\n@@ -89,15 +89,18 @@ max_chain() {\n # adjusted (or scrapped if the heuristics have become too unreliable)\n test_expect_success 'packing produces a long delta' '\n \t# Use --window=0 to make sure we are seeing reused deltas,\n-\t# not computing a new long chain.\n-\tpack=$(git pack-objects --all --window=0 </dev/null pack) &&\n+\t# not computing a new long chain. (Also avoid the --path-walk\n+\t# option as it may break delta chains.)\n+\tpack=$(git pack-objects --all --window=0 --no-path-walk </dev/null pack) &&\n \techo 9 >expect &&\n \tmax_chain pack-$pack.pack >actual &&\n \ttest_cmp expect actual\n '\n \n test_expect_success '--depth limits depth' '\n-\tpack=$(git pack-objects --all --depth=5 </dev/null pack) &&\n+\t# Avoid --path-walk to avoid breaking delta chains across path\n+\t# boundaries.\n+\tpack=$(git pack-objects --all --depth=5 --no-path-walk </dev/null pack) &&\n \techo 5 >expect &&\n \tmax_chain pack-$pack.pack >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t5332-multi-pack-reuse.sh b/t/t5332-multi-pack-reuse.sh\nindex 57cad7708f8..395d09444ce 100755\n--- a/t/t5332-multi-pack-reuse.sh\n+++ b/t/t5332-multi-pack-reuse.sh\n@@ -7,6 +7,13 @@ test_description='pack-objects multi-pack reuse'\n \n GIT_TEST_MULTI_PACK_INDEX=0\n GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n+\n+# The --path-walk option does not consider the preferred pack\n+# at all for reusing deltas, so this variable changes the\n+# behavior of this test, if enabled.\n+GIT_TEST_PACK_PATH_WALK=0\n+export GIT_TEST_PACK_PATH_WALK\n+\n objdir=.git/objects\n packdir=$objdir/pack\n \ndiff --git a/t/t5530-upload-pack-error.sh b/t/t5530-upload-pack-error.sh\nindex 558eedf25a4..8eb6fea839a 100755\n--- a/t/t5530-upload-pack-error.sh\n+++ b/t/t5530-upload-pack-error.sh\n@@ -34,6 +34,12 @@ test_expect_success 'upload-pack fails due to error in pack-objects packing' '\n \thexsz=$(test_oid hexsz) &&\n \tprintf \"%04xwant %s\\n00000009done\\n0000\" \\\n \t\t$(($hexsz + 10)) $head >input &&\n+\n+\t# The current implementation of path-walk causes a different\n+\t# error message. This will be changed by a future refactoring.\n+\tGIT_TEST_PACK_PATH_WALK=0 &&\n+\texport GIT_TEST_PACK_PATH_WALK &&\n+\n \ttest_must_fail git upload-pack . <input >/dev/null 2>output.err &&\n \ttest_grep \"unable to read\" output.err &&\n \ttest_grep \"pack-objects died\" output.err\ndiff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh\nindex c562bad042a..ab76d4b6dc4 100755\n--- a/t/t7406-submodule-update.sh\n+++ b/t/t7406-submodule-update.sh\n@@ -1095,12 +1095,15 @@ test_expect_success 'submodule update --quiet passes quietness to fetch with a s\n \t(cd super5 &&\n \t # This test var can mess with the stderr output checked in this test.\n \t GIT_TEST_NAME_HASH_VERSION=1 \\\n+\t GIT_TEST_PACK_PATH_WALK=0 \\\n \t\tgit submodule update --quiet --init --depth=1 submodule3 >out 2>err &&\n \t test_must_be_empty out &&\n \t test_must_be_empty err\n \t) &&\n \tgit clone super4 super6 &&\n \t(cd super6 &&\n+\t # This test variable will create a \"warning\" message to stderr\n+\t GIT_TEST_PACK_PATH_WALK=0 \\\n \t git submodule update --init --depth=1 submodule3 >out 2>err &&\n \t test_file_not_empty out &&\n \t test_file_not_empty err\n-- \ngitgitgadget\n\n"},{"id":"513859","messageId":"cfee2136e92832cb3fb4e90e016d5d71034ea9a6.1741571455.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.git.1741571455.gitgitgadget@gmail.com","subject":"[PATCH 06/13] t5538: add tests to confirm deltas in shallow pushes","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-10T01:50:48Z","receivedAt":"2025-03-10T01:51:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nIt can be notoriously difficult to detect if delta bases are being\ncomputed properly during 'git push'. Construct an example where it will\nmake a kilobyte worth of difference when a delta base is not found. We\ncan then use the progress indicators to distinguish between bytes and\nKiB depending on whether the delta base is found and used.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n t/t5538-push-shallow.sh | 34 ++++++++++++++++++++++++++++++++++\n 1 file changed, 34 insertions(+)\n\ndiff --git a/t/t5538-push-shallow.sh b/t/t5538-push-shallow.sh\nindex e91fcc173e8..11b85cca9e8 100755\n--- a/t/t5538-push-shallow.sh\n+++ b/t/t5538-push-shallow.sh\n@@ -123,4 +123,38 @@ EOF\n \tgit cat-file blob $(echo 1|git hash-object --stdin) >/dev/null\n \t)\n '\n+\n+test_expect_success 'push new commit from shallow clone has correct object count' '\n+\tgit init origin &&\n+\ttest_commit -C origin a &&\n+\ttest_commit -C origin b &&\n+\n+\tgit clone --depth=1 \"file://$(pwd)/origin\" client &&\n+\tgit -C client checkout -b topic &&\n+\tgit -C client commit --allow-empty -m \"empty\" &&\n+\tGIT_PROGRESS_DELAY=0 git -C client push --progress origin topic 2>err &&\n+\ttest_grep \"Enumerating objects: 1, done.\" err\n+'\n+\n+test_expect_success 'push new commit from shallow clone has good deltas' '\n+\tgit init base &&\n+\ttest_seq 1 999 >base/a &&\n+\ttest_commit -C base initial &&\n+\tgit -C base add a &&\n+\tgit -C base commit -m \"big a\" &&\n+\n+\tgit clone --depth=1 \"file://$(pwd)/base\" deltas &&\n+\tgit -C deltas checkout -b deltas &&\n+\ttest_seq 1 1000 >deltas/a &&\n+\tgit -C deltas commit -a -m \"bigger a\" &&\n+\tGIT_TRACE2_PERF=\"$(pwd)/trace.txt\" \\\n+\tGIT_PROGRESS_DELAY=0 git -C deltas push --progress origin deltas 2>err &&\n+\n+\ttest_grep \"Enumerating objects: 5, done\" err &&\n+\n+\t# If the delta base is found, then this message uses \"bytes\".\n+\t# If the delta base is not found, then this message uses \"KiB\".\n+\ttest_grep \"Writing objects: .* bytes\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"513861","messageId":"1e75f0682810da8881e38f489e9a06e78b6a1abf.1741571455.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.git.1741571455.gitgitgadget@gmail.com","subject":"[PATCH 07/13] repack: add --path-walk option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-10T01:50:49Z","receivedAt":"2025-03-10T01:51:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nSince 'git pack-objects' supports a --path-walk option, allow passing it\nthrough in 'git repack'. This presents interesting testing opportunities for\ncomparing the different repacking strategies against each other.\n\nAdd the --path-walk option to the performance tests in p5313.\n\nFor the microsoft/fluentui repo [1] checked out at a specific commit [2],\nthe --path-walk tests in p5313 look like this:\n\nTest                                                     this tree\n-------------------------------------------------------------------------\n5313.18: thin pack with --path-walk                      0.08(0.06+0.02)\n5313.19: thin pack size with --path-walk                           18.4K\n5313.20: big pack with --path-walk                       2.10(7.80+0.26)\n5313.21: big pack size with --path-walk                            19.8M\n5313.22: shallow fetch pack with --path-walk             1.62(3.38+0.17)\n5313.23: shallow pack size with --path-walk                        33.6M\n5313.24: repack with --path-walk                         81.29(96.08+0.71)\n5313.25: repack size with --path-walk                             142.5M\n\n[1] https://github.com/microsoft/fluentui\n[2] e70848ebac1cd720875bccaa3026f4a9ed700e08\n\nAlong with the earlier tests in p5313, I'll instead reformat the\ncomparison as follows:\n\nRepack Method    Pack Size       Time\n---------------------------------------\nHash v1             439.4M      87.24s\nHash v2             161.7M      21.51s\nPath Walk           142.5M      81.29s\n\nThere are a few things to notice here:\n\n 1. The benefits of --name-hash-version=2 over --name-hash-version=1 are\n    significant, but --path-walk still compresses better than that\n    option.\n\n 2. The --path-walk command is still using --name-hash-version=1 for the\n    second pass of delta computation, using the increased name hash\n    collisions as a potential method for opportunistic compression on\n    top of the path-focused compression.\n\n 3. The --path-walk algorithm is currently sequential and does not use\n    multiple threads for delta compression. Threading will be\n    implemented in a future change so the computation time will improve\n    to better compete in this metric.\n\nThere are small benefits in size for my copy of the Git repository:\n\nRepack Method    Pack Size       Time\n---------------------------------------\nHash v1             248.8M      30.44s\nHash v2             249.0M      30.15s\nPath Walk           213.2M     142.50s\n\nAs well as in the nodejs/node repository [3]:\n\nRepack Method    Pack Size       Time\n---------------------------------------\nHash v1             739.9M      71.18s\nHash v2             764.6M      67.82s\nPath Walk           698.1M     208.10s\n\n[3] https://github.com/nodejs/node\n\nThis benefit also repeats in my copy of the Linux kernel repository:\n\nRepack Method    Pack Size       Time\n---------------------------------------\nHash v1               2.5G     554.41s\nHash v2               2.5G     549.62s\nPath Walk             2.2G    1562.36s\n\nIt is important to see that even when the repository shape does not have\nmany name-hash collisions, there is a slight space boost to be found\nusing this method.\n\nAs this repacking strategy was released in Git for Windows 2.47.0, some\nusers have reported cases where the --path-walk compression is slightly\nworse than the --name-hash-version=2 option. In those cases, it may be\nbeneficial to combine the two options. However, there has not been a\nreleased version of Git that has both options and I don't have access to\nthese repos for testing.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-repack.adoc | 14 +++++++++++++-\n builtin/repack.c              |  7 ++++++-\n t/perf/p5313-pack-objects.sh  | 18 ++++++++----------\n 3 files changed, 27 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/git-repack.adoc b/Documentation/git-repack.adoc\nindex 5852a5c9736..2199eb3b94f 100644\n--- a/Documentation/git-repack.adoc\n+++ b/Documentation/git-repack.adoc\n@@ -11,7 +11,7 @@ SYNOPSIS\n [verse]\n 'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n \t[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n-\t[--write-midx] [--name-hash-version=<n>]\n+\t[--write-midx] [--name-hash-version=<n>] [--path-walk]\n \n DESCRIPTION\n -----------\n@@ -255,6 +255,18 @@ linkgit:git-multi-pack-index[1]).\n \tProvide this argument to the underlying `git pack-objects` process.\n \tSee linkgit:git-pack-objects[1] for full details.\n \n+--path-walk::\n+\tThis option passes the `--path-walk` option to the underlying\n+\t`git pack-options` process (see linkgit:git-pack-objects[1]).\n+\tBy default, `git pack-objects` walks objects in an order that\n+\tpresents trees and blobs in an order unrelated to the path they\n+\tappear relative to a commit's root tree. The `--path-walk` option\n+\tenables a different walking algorithm that organizes trees and\n+\tblobs by path. This has the potential to improve delta compression\n+\tespecially in the presence of filenames that cause collisions in\n+\tGit's default name-hash algorithm. Due to changing how the objects\n+\tare walked, this option is not compatible with `--delta-islands`\n+\tor `--filter`.\n \n CONFIGURATION\n -------------\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 75e3752353a..d7f798280c0 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -43,7 +43,7 @@ static char *packdir, *packtmp_name, *packtmp;\n static const char *const git_repack_usage[] = {\n \tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n \t   \"[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\\n\"\n-\t   \"[--write-midx] [--name-hash-version=<n>]\"),\n+\t   \"[--write-midx] [--name-hash-version=<n>] [--path-walk]\"),\n \tNULL\n };\n \n@@ -63,6 +63,7 @@ struct pack_objects_args {\n \tint quiet;\n \tint local;\n \tint name_hash_version;\n+\tint path_walk;\n \tstruct list_objects_filter_options filter_options;\n };\n \n@@ -313,6 +314,8 @@ static void prepare_pack_objects(struct child_process *cmd,\n \t\tstrvec_pushf(&cmd->args, \"--no-reuse-object\");\n \tif (args->name_hash_version)\n \t\tstrvec_pushf(&cmd->args, \"--name-hash-version=%d\", args->name_hash_version);\n+\tif (args->path_walk)\n+\t\tstrvec_pushf(&cmd->args, \"--path-walk\");\n \tif (args->local)\n \t\tstrvec_push(&cmd->args,  \"--local\");\n \tif (args->quiet)\n@@ -1212,6 +1215,8 @@ int cmd_repack(int argc,\n \t\t\t\tN_(\"pass --no-reuse-object to git-pack-objects\")),\n \t\tOPT_INTEGER(0, \"name-hash-version\", &po_args.name_hash_version,\n \t\t\t\tN_(\"specify the name hash version to use for grouping similar objects by path\")),\n+\t\tOPT_BOOL(0, \"path-walk\", &po_args.path_walk,\n+\t\t\t\tN_(\"pass --path-walk to git-pack-objects\")),\n \t\tOPT_NEGBIT('n', NULL, &run_update_server_info,\n \t\t\t\tN_(\"do not run git-update-server-info\"), 1),\n \t\tOPT__QUIET(&po_args.quiet, N_(\"be quiet\")),\ndiff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh\nindex cd6dd3abb71..98748b0e203 100755\n--- a/t/perf/p5313-pack-objects.sh\n+++ b/t/perf/p5313-pack-objects.sh\n@@ -55,23 +55,21 @@ test_all_with_args () {\n \ttest_size \"shallow pack size with $parameter\" '\n \t\ttest_file_size out\n \t'\n-}\n-\n-for version in 1 2\n-do\n-\texport version\n-\n-\ttest_all_with_args --name-hash-version=$version\n \n-\ttest_perf \"repack with --name-hash-version=$version\" '\n-\t\tgit repack -adf --name-hash-version=$version\n+\ttest_perf \"repack with $parameter\" '\n+\t\tgit repack -adf $parameter\n \t'\n \n-\ttest_size \"repack size with --name-hash-version=$version\" '\n+\ttest_size \"repack size with $parameter\" '\n \t\tgitdir=$(git rev-parse --git-dir) &&\n \t\tpack=$(ls $gitdir/objects/pack/pack-*.pack) &&\n \t\ttest_file_size \"$pack\"\n \t'\n+}\n+\n+for version in 1 2\n+do\n+\ttest_all_with_args --name-hash-version=$version\n done\n \n test_all_with_args --path-walk\n-- \ngitgitgadget\n\n"},{"id":"513862","messageId":"073cb44a47c904b26df613bf6312027076cc7d68.1741571455.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.git.1741571455.gitgitgadget@gmail.com","subject":"[PATCH 08/13] pack-objects: enable --path-walk via config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-10T01:50:50Z","receivedAt":"2025-03-10T01:51:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nUsers may want to enable the --path-walk option for 'git pack-objects' by\ndefault, especially underneath commands like 'git push' or 'git repack'.\n\nThis should be limited to client repositories, since the --path-walk option\ndisables bitmap walks, so would be bad to include in Git servers when\nserving fetches and clones. There is potential that it may be helpful to\nconsider when repacking the repository, to take advantage of improved deltas\nacross historical versions of the same files.\n\nMuch like how \"pack.useSparse\" was introduced and included in\n\"feature.experimental\" before being enabled by default, use the repository\nsettings infrastructure to make the new \"pack.usePathWalk\" config enabled by\n\"feature.experimental\" and \"feature.manyFiles\".\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/config/feature.adoc | 4 ++++\n Documentation/config/pack.adoc    | 8 ++++++++\n builtin/pack-objects.c            | 3 +++\n repo-settings.c                   | 3 +++\n repo-settings.h                   | 1 +\n 5 files changed, 19 insertions(+)\n\ndiff --git a/Documentation/config/feature.adoc b/Documentation/config/feature.adoc\nindex f061b64b748..cb49ff2604a 100644\n--- a/Documentation/config/feature.adoc\n+++ b/Documentation/config/feature.adoc\n@@ -20,6 +20,10 @@ walking fewer objects.\n +\n * `pack.allowPackReuse=multi` may improve the time it takes to create a pack by\n reusing objects from multiple packs instead of just one.\n++\n+* `pack.usePathWalk` may speed up packfile creation and make the packfiles be\n+significantly smaller in the presence of certain filename collisions with Git's\n+default name-hash.\n \n feature.manyFiles::\n \tEnable config options that optimize for repos with many files in the\ndiff --git a/Documentation/config/pack.adoc b/Documentation/config/pack.adoc\nindex da527377faf..08d06271177 100644\n--- a/Documentation/config/pack.adoc\n+++ b/Documentation/config/pack.adoc\n@@ -155,6 +155,14 @@ pack.useSparse::\n \tcommits contain certain types of direct renames. Default is\n \t`true`.\n \n+pack.usePathWalk::\n+\tWhen true, git will default to using the '--path-walk' option in\n+\t'git pack-objects' when the '--revs' option is present. This\n+\talgorithm groups objects by path to maximize the ability to\n+\tcompute delta chains across historical versions of the same\n+\tobject. This may disable other options, such as using bitmaps to\n+\tenumerate objects.\n+\n pack.preferBitmapTips::\n \tWhen selecting which commits will receive bitmaps, prefer a\n \tcommit at the tip of any reference that is a suffix of any value\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 120202b05e9..c756ce50dd7 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4649,6 +4649,9 @@ int cmd_pack_objects(int argc,\n \t\tif (use_bitmap_index > 0 ||\n \t\t    !use_internal_rev_list)\n \t\t\tpath_walk = 0;\n+\t\telse if (the_repository->gitdir &&\n+\t\t\t the_repository->settings.pack_use_path_walk)\n+\t\t\tpath_walk = 1;\n \t\telse\n \t\t\tpath_walk = git_env_bool(\"GIT_TEST_PACK_PATH_WALK\", 0);\n \t}\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 67e9cfd2e63..9b5595c708e 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -47,11 +47,13 @@ void prepare_repo_settings(struct repository *r)\n \t\tr->settings.fetch_negotiation_algorithm = FETCH_NEGOTIATION_SKIPPING;\n \t\tr->settings.pack_use_bitmap_boundary_traversal = 1;\n \t\tr->settings.pack_use_multi_pack_reuse = 1;\n+\t\tr->settings.pack_use_path_walk = 1;\n \t}\n \tif (manyfiles) {\n \t\tr->settings.index_version = 4;\n \t\tr->settings.index_skip_hash = 1;\n \t\tr->settings.core_untracked_cache = UNTRACKED_CACHE_WRITE;\n+\t\tr->settings.pack_use_path_walk = 1;\n \t}\n \n \t/* Commit graph config or default, does not cascade (simple) */\n@@ -66,6 +68,7 @@ void prepare_repo_settings(struct repository *r)\n \n \t/* Boolean config or default, does not cascade (simple)  */\n \trepo_cfg_bool(r, \"pack.usesparse\", &r->settings.pack_use_sparse, 1);\n+\trepo_cfg_bool(r, \"pack.usepathwalk\", &r->settings.pack_use_path_walk, 0);\n \trepo_cfg_bool(r, \"core.multipackindex\", &r->settings.core_multi_pack_index, 1);\n \trepo_cfg_bool(r, \"index.sparse\", &r->settings.sparse_index, 0);\n \trepo_cfg_bool(r, \"index.skiphash\", &r->settings.index_skip_hash, r->settings.index_skip_hash);\ndiff --git a/repo-settings.h b/repo-settings.h\nindex ddc11967e01..a31decad221 100644\n--- a/repo-settings.h\n+++ b/repo-settings.h\n@@ -56,6 +56,7 @@ struct repo_settings {\n \tenum untracked_cache_setting core_untracked_cache;\n \n \tint pack_use_sparse;\n+\tint pack_use_path_walk;\n \tenum fetch_negotiation_setting fetch_negotiation_algorithm;\n \n \tint core_multi_pack_index;\n-- \ngitgitgadget\n\n"},{"id":"513863","messageId":"fd9a335329e9932131a33104c5a136583fad5d23.1741571455.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.git.1741571455.gitgitgadget@gmail.com","subject":"[PATCH 09/13] scalar: enable path-walk during push via config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-10T01:50:51Z","receivedAt":"2025-03-10T01:51:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nRepositories registered with Scalar are expected to be client-only\nrepositories that are rather large. This means that they are more likely to\nbe good candidates for using the --path-walk option when running 'git\npack-objects', especially under the hood of 'git push'. Enable this config\nin Scalar repositories.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n scalar.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/scalar.c b/scalar.c\nindex da42b4be0cc..bf638fa34b8 100644\n--- a/scalar.c\n+++ b/scalar.c\n@@ -170,6 +170,7 @@ static int set_recommended_config(int reconfigure)\n \t\t{ \"core.autoCRLF\", \"false\" },\n \t\t{ \"core.safeCRLF\", \"false\" },\n \t\t{ \"fetch.showForcedUpdates\", \"false\" },\n+\t\t{ \"pack.usePathWalk\", \"true\" },\n \t\t{ NULL, NULL },\n \t};\n \tint i;\n-- \ngitgitgadget\n\n"},{"id":"513867","messageId":"c047fbd7f275ef79695b5e1356075705e6fd7bc8.1741571455.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.git.1741571455.gitgitgadget@gmail.com","subject":"[PATCH 10/13] pack-objects: refactor path-walk delta phase","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-10T01:50:52Z","receivedAt":"2025-03-10T01:51:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nPreviously, the --path-walk option to 'git pack-objects' would compute\ndeltas inline with the path-walk logic. This would make the progress\nindicator look like it is taking a long time to enumerate objects, and\nthen very quickly computed deltas.\n\nInstead of computing deltas on each region of objects organized by tree,\nstore a list of regions corresponding to these groups. These can later\nbe pulled from the list for delta compression before doing the \"global\"\ndelta search.\n\nThis presents a new progress indicator that can be used in tests to\nverify that this stage is happening.\n\nThe current implementation is not integrated with threads, but could be\ndone in a future update.\n\nSince we do not attempt to sort objects by size until after exploring\nall trees, we can remove the previous change to t5530 due to a different\nerror message appearing first.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c       | 82 +++++++++++++++++++++++++-----------\n pack-objects.h               | 12 ++++++\n t/t5300-pack-object.sh       |  8 +++-\n t/t5530-upload-pack-error.sh |  6 ---\n 4 files changed, 75 insertions(+), 33 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex c756ce50dd7..c5a3129c88e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3233,6 +3233,51 @@ static int should_attempt_deltas(struct object_entry *entry)\n \treturn 1;\n }\n \n+static void find_deltas_for_region(struct object_entry *list UNUSED,\n+\t\t\t\t   struct packing_region *region,\n+\t\t\t\t   unsigned int *processed)\n+{\n+\tstruct object_entry **delta_list;\n+\tuint32_t delta_list_nr = 0;\n+\n+\tALLOC_ARRAY(delta_list, region->nr);\n+\tfor (uint32_t i = 0; i < region->nr; i++) {\n+\t\tstruct object_entry *entry = to_pack.objects + region->start + i;\n+\t\tif (should_attempt_deltas(entry))\n+\t\t\tdelta_list[delta_list_nr++] = entry;\n+\t}\n+\n+\tQSORT(delta_list, delta_list_nr, type_size_sort);\n+\tfind_deltas(delta_list, &delta_list_nr, window, depth, processed);\n+\tfree(delta_list);\n+}\n+\n+static void find_deltas_by_region(struct object_entry *list,\n+\t\t\t\t  struct packing_region *regions,\n+\t\t\t\t  uint32_t start, uint32_t nr)\n+{\n+\tunsigned int processed = 0;\n+\tuint32_t progress_nr;\n+\n+\tif (!nr)\n+\t\treturn;\n+\n+\tprogress_nr = regions[nr - 1].start + regions[nr - 1].nr;\n+\n+\tif (progress)\n+\t\tprogress_state = start_progress(the_repository,\n+\t\t\t\t\t\t_(\"Compressing objects by path\"),\n+\t\t\t\t\t\tprogress_nr);\n+\n+\twhile (nr--)\n+\t\tfind_deltas_for_region(list,\n+\t\t\t\t       &regions[start++],\n+\t\t\t\t       &processed);\n+\n+\tdisplay_progress(progress_state, progress_nr);\n+\tstop_progress(&progress_state);\n+}\n+\n static void prepare_pack(int window, int depth)\n {\n \tstruct object_entry **delta_list;\n@@ -3257,6 +3302,10 @@ static void prepare_pack(int window, int depth)\n \tif (!to_pack.nr_objects || !window || !depth)\n \t\treturn;\n \n+\tif (path_walk)\n+\t\tfind_deltas_by_region(to_pack.objects, to_pack.regions,\n+\t\t\t\t      0, to_pack.nr_regions);\n+\n \tALLOC_ARRAY(delta_list, to_pack.nr_objects);\n \tnr_deltas = n = 0;\n \n@@ -4210,10 +4259,8 @@ static int add_objects_by_path(const char *path,\n \t\t\t       enum object_type type,\n \t\t\t       void *data)\n {\n-\tstruct object_entry **delta_list;\n \tsize_t oe_start = to_pack.nr_objects;\n \tsize_t oe_end;\n-\tunsigned int sub_list_size;\n \tunsigned int *processed = data;\n \n \t/*\n@@ -4246,32 +4293,17 @@ static int add_objects_by_path(const char *path,\n \tif (oe_end == oe_start || !window)\n \t\treturn 0;\n \n-\tsub_list_size = 0;\n-\tALLOC_ARRAY(delta_list, oe_end - oe_start);\n-\n-\tfor (size_t i = 0; i < oe_end - oe_start; i++) {\n-\t\tstruct object_entry *entry = to_pack.objects + oe_start + i;\n+\tALLOC_GROW(to_pack.regions,\n+\t\t   to_pack.nr_regions + 1,\n+\t\t   to_pack.nr_regions_alloc);\n \n-\t\tif (!should_attempt_deltas(entry))\n-\t\t\tcontinue;\n-\n-\t\tdelta_list[sub_list_size++] = entry;\n-\t}\n+\tto_pack.regions[to_pack.nr_regions].start = oe_start;\n+\tto_pack.regions[to_pack.nr_regions].nr = oe_end - oe_start;\n+\tto_pack.nr_regions++;\n \n-\t/*\n-\t * Find delta bases among this list of objects that all match the same\n-\t * path. This causes the delta compression to be interleaved in the\n-\t * object walk, which can lead to confusing progress indicators. This is\n-\t * also incompatible with threaded delta calculations. In the future,\n-\t * consider creating a list of regions in the full to_pack.objects array\n-\t * that could be picked up by the threaded delta computation.\n-\t */\n-\tif (sub_list_size && window) {\n-\t\tQSORT(delta_list, sub_list_size, type_size_sort);\n-\t\tfind_deltas(delta_list, &sub_list_size, window, depth, processed);\n-\t}\n+\t*processed += oids->nr;\n+\tdisplay_progress(progress_state, *processed);\n \n-\tfree(delta_list);\n \treturn 0;\n }\n \ndiff --git a/pack-objects.h b/pack-objects.h\nindex d73e3843c92..1b01304f9c4 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -119,11 +119,23 @@ struct object_entry {\n \tunsigned ext_base:1; /* delta_idx points outside packlist */\n };\n \n+/**\n+ * A packing region is a section of the packing_data.objects array\n+ * as given by a starting index and a number of elements.\n+ */\n+struct packing_region {\n+\tuint32_t start;\n+\tuint32_t nr;\n+};\n+\n struct packing_data {\n \tstruct repository *repo;\n \tstruct object_entry *objects;\n \tuint32_t nr_objects, nr_alloc;\n \n+\tstruct packing_region *regions;\n+\tuint32_t nr_regions, nr_regions_alloc;\n+\n \tint32_t *index;\n \tuint32_t index_size;\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 16420d12863..c8df6afd784 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -725,7 +725,9 @@ test_expect_success '--name-hash-version=2 and --write-bitmap-index are incompat\n \n test_expect_success '--path-walk pack everything' '\n \tgit -C server rev-parse HEAD >in &&\n-\tgit -C server pack-objects --stdout --revs --path-walk <in >out.pack &&\n+\tGIT_PROGRESS_DELAY=0 git -C server pack-objects \\\n+\t\t--stdout --revs --path-walk --progress <in >out.pack 2>err &&\n+\tgrep \"Compressing objects by path\" err &&\n \tgit -C server index-pack --stdin <out.pack\n '\n \n@@ -734,7 +736,9 @@ test_expect_success '--path-walk thin pack' '\n \t$(git -C server rev-parse HEAD)\n \t^$(git -C server rev-parse HEAD~2)\n \tEOF\n-\tgit -C server pack-objects --thin --stdout --revs --path-walk <in >out.pack &&\n+\tGIT_PROGRESS_DELAY=0 git -C server pack-objects \\\n+\t\t--thin --stdout --revs --path-walk --progress <in >out.pack 2>err &&\n+\tgrep \"Compressing objects by path\" err &&\n \tgit -C server index-pack --fix-thin --stdin <out.pack\n '\n \ndiff --git a/t/t5530-upload-pack-error.sh b/t/t5530-upload-pack-error.sh\nindex 8eb6fea839a..558eedf25a4 100755\n--- a/t/t5530-upload-pack-error.sh\n+++ b/t/t5530-upload-pack-error.sh\n@@ -34,12 +34,6 @@ test_expect_success 'upload-pack fails due to error in pack-objects packing' '\n \thexsz=$(test_oid hexsz) &&\n \tprintf \"%04xwant %s\\n00000009done\\n0000\" \\\n \t\t$(($hexsz + 10)) $head >input &&\n-\n-\t# The current implementation of path-walk causes a different\n-\t# error message. This will be changed by a future refactoring.\n-\tGIT_TEST_PACK_PATH_WALK=0 &&\n-\texport GIT_TEST_PACK_PATH_WALK &&\n-\n \ttest_must_fail git upload-pack . <input >/dev/null 2>output.err &&\n \ttest_grep \"unable to read\" output.err &&\n \ttest_grep \"pack-objects died\" output.err\n-- \ngitgitgadget\n\n"},{"id":"513864","messageId":"cf990b90b34e96428d6b20e0a1334139fe8d3af4.1741571455.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.git.1741571455.gitgitgadget@gmail.com","subject":"[PATCH 11/13] pack-objects: thread the path-based compression","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-10T01:50:53Z","receivedAt":"2025-03-10T01:51:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nAdapting the implementation of ll_find_deltas(), create a threaded\nversion of the --path-walk compression step in 'git pack-objects'.\n\nThis involves adding a 'regions' member to the thread_params struct,\nallowing each thread to own a section of paths. We can simplify the way\njobs are split because there is no value in extending the batch based on\nname-hash the way sections of the object entry array are attempted to be\ngrouped. We re-use the 'list_size' and 'remaining' items for the purpose\nof borrowing work in progress from other \"victim\" threads when a thread\nhas finished its batch of work more quickly.\n\nUsing the Git repository as a test repo, the p5313 performance test\nshows that the resulting size of the repo is the same, but the threaded\nimplementation gives gains of varying degrees depending on the number of\nobjects being packed. (This was tested on a 16-core machine.)\n\nTest                        HEAD~1      HEAD\n---------------------------------------------------\n5313.20: big pack             2.38      1.99 -16.4%\n5313.21: big pack size       16.1M     16.0M  -0.2%\n5313.24: repack             107.32     45.41 -57.7%\n5313.25: repack size        213.3M    213.2M  -0.0%\n\n(Test output is formatted to better fit in message.)\n\nThis ~60% reduction in 'git repack --path-walk' time is typical across\nall repos I used for testing. What is interesting is to compare when the\noverall time improves enough to outperform the --name-hash-version=1\ncase. These time improvements correlate with repositories with data\nshapes that significantly improve their data size as well. The\n--path-walk feature frequently takes longer than --name-hash-verison=2,\ntrading some extrac computation for some additional compression. The\nnatural place where this additional computation comes from is the two\ncompression passes that --path-walk takes, though the first pass is\nnaturally faster due to the path boundaries avoiding a number of delta\ncompression attempts.\n\nFor example, the microsoft/fluentui repo has significant size reduction\nfrom --name-hash-version=1 to --name-hash-version=2 followed by further\nimprovements with --path-walk. The threaded computation makes\n--path-walk more competitive in time compared to --name-hash-version=2,\nthough still ~31% more expensive in that metric.\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                439.4M      87.24s\nHash v2                161.7M      21.51s\nPath Walk (Before)     142.5M      81.29s\nPath Walk (After)      142.5M      28.16s\n\nSimilar results hold for the Git repository:\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                248.8M      30.44s\nHash v2                249.0M      30.15s\nPath Walk (Before)     213.2M     142.50s\nPath Walk (After)      213.3M      45.41s\n\n...as well as the nodejs/node repository:\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                739.9M      71.18s\nHash v2                764.6M      67.82s\nPath Walk (Before)     698.1M     208.10s\nPath Walk (After)      698.0M      75.10s\n\nFinally, the Linux kernel repository is a good test for this repacking\ntime change, even though the space savings is more subtle:\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                  2.5G     554.41s\nHash v2                  2.5G     549.62s\nPath Walk (before)       2.2G    1562.36s\nPath Walk (before)       2.2G     559.00s\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c | 163 ++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 161 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex c5a3129c88e..e8b1b057ec3 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2964,6 +2964,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n struct thread_params {\n \tpthread_t thread;\n \tstruct object_entry **list;\n+\tstruct packing_region *regions;\n \tunsigned list_size;\n \tunsigned remaining;\n \tint window;\n@@ -3278,6 +3279,164 @@ static void find_deltas_by_region(struct object_entry *list,\n \tstop_progress(&progress_state);\n }\n \n+static void *threaded_find_deltas_by_path(void *arg)\n+{\n+\tstruct thread_params *me = arg;\n+\n+\tprogress_lock();\n+\twhile (me->remaining) {\n+\t\twhile (me->remaining) {\n+\t\t\tprogress_unlock();\n+\t\t\tfind_deltas_for_region(to_pack.objects,\n+\t\t\t\t\t       me->regions,\n+\t\t\t\t\t       me->processed);\n+\t\t\tprogress_lock();\n+\t\t\tme->remaining--;\n+\t\t\tme->regions++;\n+\t\t}\n+\n+\t\tme->working = 0;\n+\t\tpthread_cond_signal(&progress_cond);\n+\t\tprogress_unlock();\n+\n+\t\t/*\n+\t\t * We must not set ->data_ready before we wait on the\n+\t\t * condition because the main thread may have set it to 1\n+\t\t * before we get here. In order to be sure that new\n+\t\t * work is available if we see 1 in ->data_ready, it\n+\t\t * was initialized to 0 before this thread was spawned\n+\t\t * and we reset it to 0 right away.\n+\t\t */\n+\t\tpthread_mutex_lock(&me->mutex);\n+\t\twhile (!me->data_ready)\n+\t\t\tpthread_cond_wait(&me->cond, &me->mutex);\n+\t\tme->data_ready = 0;\n+\t\tpthread_mutex_unlock(&me->mutex);\n+\n+\t\tprogress_lock();\n+\t}\n+\tprogress_unlock();\n+\t/* leave ->working 1 so that this doesn't get more work assigned */\n+\treturn NULL;\n+}\n+\n+static void ll_find_deltas_by_region(struct object_entry *list,\n+\t\t\t\t     struct packing_region *regions,\n+\t\t\t\t     uint32_t start, uint32_t nr)\n+{\n+\tstruct thread_params *p;\n+\tint i, ret, active_threads = 0;\n+\tunsigned int processed = 0;\n+\tuint32_t progress_nr;\n+\tinit_threaded_search();\n+\n+\tif (!nr)\n+\t\treturn;\n+\n+\tprogress_nr =  regions[nr - 1].start + regions[nr - 1].nr;\n+\tif (delta_search_threads <= 1) {\n+\t\tfind_deltas_by_region(list, regions, start, nr);\n+\t\tcleanup_threaded_search();\n+\t\treturn;\n+\t}\n+\n+\tif (progress > pack_to_stdout)\n+\t\tfprintf_ln(stderr, _(\"Path-based delta compression using up to %d threads\"),\n+\t\t\t   delta_search_threads);\n+\tCALLOC_ARRAY(p, delta_search_threads);\n+\n+\tif (progress)\n+\t\tprogress_state = start_progress(the_repository,\n+\t\t\t\t\t\t_(\"Compressing objects by path\"),\n+\t\t\t\t\t\tprogress_nr);\n+\t/* Partition the work amongst work threads. */\n+\tfor (i = 0; i < delta_search_threads; i++) {\n+\t\tunsigned sub_size = nr / (delta_search_threads - i);\n+\n+\t\tp[i].window = window;\n+\t\tp[i].depth = depth;\n+\t\tp[i].processed = &processed;\n+\t\tp[i].working = 1;\n+\t\tp[i].data_ready = 0;\n+\n+\t\tp[i].regions = regions;\n+\t\tp[i].list_size = sub_size;\n+\t\tp[i].remaining = sub_size;\n+\n+\t\tregions += sub_size;\n+\t\tnr -= sub_size;\n+\t}\n+\n+\t/* Start work threads. */\n+\tfor (i = 0; i < delta_search_threads; i++) {\n+\t\tif (!p[i].list_size)\n+\t\t\tcontinue;\n+\t\tpthread_mutex_init(&p[i].mutex, NULL);\n+\t\tpthread_cond_init(&p[i].cond, NULL);\n+\t\tret = pthread_create(&p[i].thread, NULL,\n+\t\t\t\t     threaded_find_deltas_by_path, &p[i]);\n+\t\tif (ret)\n+\t\t\tdie(_(\"unable to create thread: %s\"), strerror(ret));\n+\t\tactive_threads++;\n+\t}\n+\n+\t/*\n+\t * Now let's wait for work completion.  Each time a thread is done\n+\t * with its work, we steal half of the remaining work from the\n+\t * thread with the largest number of unprocessed objects and give\n+\t * it to that newly idle thread.  This ensure good load balancing\n+\t * until the remaining object list segments are simply too short\n+\t * to be worth splitting anymore.\n+\t */\n+\twhile (active_threads) {\n+\t\tstruct thread_params *target = NULL;\n+\t\tstruct thread_params *victim = NULL;\n+\t\tunsigned sub_size = 0;\n+\n+\t\tprogress_lock();\n+\t\tfor (;;) {\n+\t\t\tfor (i = 0; !target && i < delta_search_threads; i++)\n+\t\t\t\tif (!p[i].working)\n+\t\t\t\t\ttarget = &p[i];\n+\t\t\tif (target)\n+\t\t\t\tbreak;\n+\t\t\tpthread_cond_wait(&progress_cond, &progress_mutex);\n+\t\t}\n+\n+\t\tfor (i = 0; i < delta_search_threads; i++)\n+\t\t\tif (p[i].remaining > 2*window &&\n+\t\t\t    (!victim || victim->remaining < p[i].remaining))\n+\t\t\t\tvictim = &p[i];\n+\t\tif (victim) {\n+\t\t\tsub_size = victim->remaining / 2;\n+\t\t\ttarget->regions = victim->regions + victim->remaining - sub_size;\n+\t\t\tvictim->list_size -= sub_size;\n+\t\t\tvictim->remaining -= sub_size;\n+\t\t}\n+\t\ttarget->list_size = sub_size;\n+\t\ttarget->remaining = sub_size;\n+\t\ttarget->working = 1;\n+\t\tprogress_unlock();\n+\n+\t\tpthread_mutex_lock(&target->mutex);\n+\t\ttarget->data_ready = 1;\n+\t\tpthread_cond_signal(&target->cond);\n+\t\tpthread_mutex_unlock(&target->mutex);\n+\n+\t\tif (!sub_size) {\n+\t\t\tpthread_join(target->thread, NULL);\n+\t\t\tpthread_cond_destroy(&target->cond);\n+\t\t\tpthread_mutex_destroy(&target->mutex);\n+\t\t\tactive_threads--;\n+\t\t}\n+\t}\n+\tcleanup_threaded_search();\n+\tfree(p);\n+\n+\tdisplay_progress(progress_state, progress_nr);\n+\tstop_progress(&progress_state);\n+}\n+\n static void prepare_pack(int window, int depth)\n {\n \tstruct object_entry **delta_list;\n@@ -3303,8 +3462,8 @@ static void prepare_pack(int window, int depth)\n \t\treturn;\n \n \tif (path_walk)\n-\t\tfind_deltas_by_region(to_pack.objects, to_pack.regions,\n-\t\t\t\t      0, to_pack.nr_regions);\n+\t\tll_find_deltas_by_region(to_pack.objects, to_pack.regions,\n+\t\t\t\t\t 0, to_pack.nr_regions);\n \n \tALLOC_ARRAY(delta_list, to_pack.nr_objects);\n \tnr_deltas = n = 0;\n-- \ngitgitgadget\n\n"},{"id":"513865","messageId":"bc9383ef11cd31c2a39f7d626ac57060891dcd2c.1741571455.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.git.1741571455.gitgitgadget@gmail.com","subject":"[PATCH 12/13] path-walk: add new 'edge_aggressive' option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-10T01:50:54Z","receivedAt":"2025-03-10T01:51:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nIn preparation for allowing both the --shallow and --path-walk options\nin the 'git pack-objects' builtin, create a new 'edge_aggressive' option\nin the path-walk API. This option will help walk the boundary more\nthoroughly and help avoid sending extra objects during fetches and\npushes.\n\nThe only use of the 'edge_hint_aggressive' option in the revision API is\nwithin mark_edges_uninteresting(), which is usually called before\nbetween prepare_revision_walk() and before visiting commits with\nget_revision(). In prepare_revision_walk(), the UNINTERESTING commits\nare walked until a boundary is found.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/technical/api-path-walk.adoc |  8 ++++++++\n path-walk.c                                |  6 +++++-\n path-walk.h                                |  7 +++++++\n t/helper/test-path-walk.c                  |  2 ++\n t/t6601-path-walk.sh                       | 20 ++++++++++++++++++++\n 5 files changed, 42 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/technical/api-path-walk.adoc b/Documentation/technical/api-path-walk.adoc\nindex e522695dd9f..34c905eb9c3 100644\n--- a/Documentation/technical/api-path-walk.adoc\n+++ b/Documentation/technical/api-path-walk.adoc\n@@ -56,6 +56,14 @@ better off using the revision walk API instead.\n \tthe revision walk so that the walk emits commits marked with the\n \t`UNINTERESTING` flag.\n \n+`edge_aggressive`::\n+\tFor performance reasons, usually only the boundary commits are\n+\texplored to find UNINTERESTING objects. However, in the case of\n+\tshallow clones it can be helpful to mark all trees and blobs\n+\treachable from UNINTERESTING tip commits as UNINTERESTING. This\n+\tmatches the behavior of `--objects-edge-aggressive` in the\n+\trevision API.\n+\n `pl`::\n \tThis pattern list pointer allows focusing the path-walk search to\n \ta set of patterns, only emitting paths that match the given\ndiff --git a/path-walk.c b/path-walk.c\nindex 341bdd2ba4e..2d4ddbadd50 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -503,7 +503,11 @@ int walk_objects_by_path(struct path_walk_info *info)\n \tif (prepare_revision_walk(info->revs))\n \t\tdie(_(\"failed to setup revision walk\"));\n \n-\t/* Walk trees to mark them as UNINTERESTING. */\n+\t/*\n+\t * Walk trees to mark them as UNINTERESTING.\n+\t * This is particularly important when 'edge_aggressive' is set.\n+\t */\n+\tinfo->revs->edge_hint_aggressive = info->edge_aggressive;\n \tedge_repo = info->revs->repo;\n \tedge_tree_list = root_tree_list;\n \tmark_edges_uninteresting(info->revs, show_edge,\ndiff --git a/path-walk.h b/path-walk.h\nindex 473ee9d361c..5ef5a8440e6 100644\n--- a/path-walk.h\n+++ b/path-walk.h\n@@ -50,6 +50,13 @@ struct path_walk_info {\n \t */\n \tint prune_all_uninteresting;\n \n+\t/**\n+\t * When 'edge_aggressive' is set, then the revision walk will use\n+\t * the '--object-edge-aggressive' option to mark even more objects\n+\t * as uninteresting.\n+\t */\n+\tint edge_aggressive;\n+\n \t/**\n \t * Specify a sparse-checkout definition to match our paths to. Do not\n \t * walk outside of this sparse definition. If the patterns are in\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nindex 61e845e5ec2..fe63002c2be 100644\n--- a/t/helper/test-path-walk.c\n+++ b/t/helper/test-path-walk.c\n@@ -82,6 +82,8 @@ int cmd__path_walk(int argc, const char **argv)\n \t\t\t N_(\"toggle inclusion of tree objects\")),\n \t\tOPT_BOOL(0, \"prune\", &info.prune_all_uninteresting,\n \t\t\t N_(\"toggle pruning of uninteresting paths\")),\n+\t\tOPT_BOOL(0, \"edge-aggressive\", &info.edge_aggressive,\n+\t\t\t N_(\"toggle aggressive edge walk\")),\n \t\tOPT_BOOL(0, \"stdin-pl\", &stdin_pl,\n \t\t\t N_(\"read a pattern list over stdin\")),\n \t\tOPT_END(),\ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex c89b0f1e19d..785c2f22373 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -378,6 +378,26 @@ test_expect_success 'topic, not base, boundary with pruning' '\n \ttest_cmp_sorted expect out\n '\n \n+test_expect_success 'topic, not base, --edge-aggressive with pruning' '\n+\ttest-tool path-walk --prune --edge-aggressive -- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:commit::$(git rev-parse topic)\n+\t1:tree::$(git rev-parse topic^{tree})\n+\t1:tree::$(git rev-parse base^{tree}):UNINTERESTING\n+\t2:tree:right/:$(git rev-parse topic:right)\n+\t2:tree:right/:$(git rev-parse base:right):UNINTERESTING\n+\t3:blob:right/c:$(git rev-parse base:right/c):UNINTERESTING\n+\t3:blob:right/c:$(git rev-parse topic:right/c)\n+\tblobs:2\n+\tcommits:1\n+\ttags:0\n+\ttrees:4\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n test_expect_success 'trees are reported exactly once' '\n \ttest_when_finished \"rm -rf unique-trees\" &&\n \ttest_create_repo unique-trees &&\n-- \ngitgitgadget\n\n"},{"id":"513866","messageId":"2eb92507213fca6dc7630ace20878eb288440bdf.1741571455.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.git.1741571455.gitgitgadget@gmail.com","subject":"[PATCH 13/13] pack-objects: allow --shallow and --path-walk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-10T01:50:55Z","receivedAt":"2025-03-10T01:51:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThere does not appear to be anything particularly incompatible about the\n--shallow and --path-walk options of 'git pack-objects'. If shallow\ncommits are to be handled differently, then it is by the revision walk\nthat defines the commit set and which are interesting or uninteresting.\n\nHowever, before the previous change, a trivial removal of the warning\nwould cause a failure in t5500-fetch-pack.sh when\nGIT_TEST_PACK_PATH_WALK is enabled. The shallow fetch would provide more\nobjects than we desired, due to some incorrect behavior of the path-walk\nAPI, especially around walking uninteresting objects.\n\nThe recently-added tests in t5538-push-shallow.sh help to confirm this\nbehavior is working with the --path-walk option if\nGIT_TEST_PACK_PATH_WALK is enabled. These tests passed previously due to\nthe --path-walk feature being disabled in the presence of a shallow\nclone.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c | 7 ++-----\n 1 file changed, 2 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e8b1b057ec3..4bd943728b4 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -209,6 +209,7 @@ static int keep_unreachable, unpack_unreachable, include_tag;\n static timestamp_t unpack_unreachable_expiration;\n static int pack_loose_unreachable;\n static int cruft;\n+static int shallow = 0;\n static timestamp_t cruft_expiration;\n static int local;\n static int have_non_local_packs;\n@@ -4483,6 +4484,7 @@ static void get_object_list_path_walk(struct rev_info *revs)\n \t * base objects.\n \t */\n \tinfo.prune_all_uninteresting = sparse;\n+\tinfo.edge_aggressive = shallow;\n \n \tif (walk_objects_by_path(&info))\n \t\tdie(_(\"failed to pack objects via path-walk\"));\n@@ -4684,7 +4686,6 @@ int cmd_pack_objects(int argc,\n \t\t     struct repository *repo UNUSED)\n {\n \tint use_internal_rev_list = 0;\n-\tint shallow = 0;\n \tint all_progress_implied = 0;\n \tstruct strvec rp = STRVEC_INIT;\n \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n@@ -4872,10 +4873,6 @@ int cmd_pack_objects(int argc,\n \t\twarning(_(\"cannot use delta islands with --path-walk\"));\n \t\tpath_walk = 0;\n \t}\n-\tif (path_walk && shallow) {\n-\t\twarning(_(\"cannot use --shallow with --path-walk\"));\n-\t\tpath_walk = 0;\n-\t}\n \tif (path_walk) {\n \t\tstrvec_push(&rp, \"--boundary\");\n \t\t /*\n-- \ngitgitgadget\n"},{"id":"513926","messageId":"xmqqwmcw7q2x.fsf@gitster.g","threadId":"63083","inReplyTo":"pull.1819.git.1741571455.gitgitgadget@gmail.com","subject":"Re: [PATCH 00/13] PATH WALK II: Add --path-walk option to 'git pack-objects'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-03-10T17:28:22Z","receivedAt":"2025-03-10T17:28:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> ... deltas across path boundaries. This second pass is much faster than a fresh\n> pass since the existing deltas are used as a limit for the size of\n> potentially new deltas, short-circuiting the checks when the delta size\n> exceeds the current-best.\n\nVery nice.\n\n> The microsoft/fluentui is a public Javascript repo that suffers from many of\n> the name hash collisions as internal repositories I've worked with. Here is\n> a comparison of the compressed size and end-to-end time of the repack:\n>\n> Repack Method    Pack Size       Time\n> ---------------------------------------\n> Hash v1             439.4M      87.24s\n> Hash v2             161.7M      21.51s\n> Path Walk           142.5M      28.16s\n>\n>\n> Less dramatic, but perhaps more standardly structured is the nodejs/node\n> repository, with these stats:\n>\n> Repack Method       Pack Size       Time\n> ------------------------------------------\n> Hash v1                739.9M      71.18s\n> Hash v2                764.6M      67.82s\n> Path Walk              698.0M      75.10s\n>\n>\n> Even the Linux kernel repository gains some benefits, even though the number\n> of hash collisions is relatively low due to a preference for short\n> filenames:\n>\n> Repack Method       Pack Size       Time\n> ------------------------------------------\n> Hash v1                  2.5G     554.41s\n> Hash v2                  2.5G     549.62s\n> Path Walk                2.2G     559.00s\n\nThis third one, v2 not performing much better than v1, is quite\nsurprising.\n\n> The drawbacks of the --path-walk feature is that it will be harder to\n> integrate it with bitmap features, specifically delta islands. This is not\n> insurmountable, but would require more work, such as a revision walk to\n> paint objects with reachability information before using that during delta\n> computations.\n>\n> However, there should still be significant benefits to Git clients trying to\n> save space and improve local performance.\n\nSure.  More experiments and more approaches will eventually give us\noverall improvement.  I am hoping that we will be able to condense\nthe result of these different approaches and their combinations into\neasy-to-choose-from canned choices (as opposed to a myriad of little\nknobs the users need to futz with without really understanding what\nthey are tweaking).\n\n> This feature was shipped with similar features in microsoft/git as of\n> v2.47.0.vfs.0.3 [4]. This was used in CI machines for an internal monorepo\n> that had significant repository growth due to constructing a batch of\n> beachball [5] CHANGELOG.[md|json] files and pushing them to a release\n> branch. These pushes were frequently 70-200 MB due to poor delta\n> compression. Using the 'pack.usePathWalk=true' config, these pushes dropped\n> in size by 100x while improving performance. Since these CI machines were\n> working with a shallow clone, the 'edge_aggressive' changes were required to\n> enable the path-walk option.\n\nNice, thanks.\n"},{"id":"514109","messageId":"Z9Hy6Yk2XM1RCsNC@nand.local","threadId":"63083","inReplyTo":"xmqqwmcw7q2x.fsf@gitster.g","subject":"Re: [PATCH 00/13] PATH WALK II: Add --path-walk option to 'git pack-objects'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-12T20:47:37Z","receivedAt":"2025-03-12T20:47:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 10, 2025 at 10:28:22AM -0700, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n>\n> > ... deltas across path boundaries. This second pass is much faster than a fresh\n> > pass since the existing deltas are used as a limit for the size of\n> > potentially new deltas, short-circuiting the checks when the delta size\n> > exceeds the current-best.\n>\n> Very nice.\n>\n> > The microsoft/fluentui is a public Javascript repo that suffers from many of\n> > the name hash collisions as internal repositories I've worked with. Here is\n> > a comparison of the compressed size and end-to-end time of the repack:\n> >\n> > Repack Method    Pack Size       Time\n> > ---------------------------------------\n> > Hash v1             439.4M      87.24s\n> > Hash v2             161.7M      21.51s\n> > Path Walk           142.5M      28.16s\n\nOK, so microsoft/fluentui benefits from the path-walk approach in the\nsize of the resulting pack, but at the cost of additional time to\ngenerate it.\n\n> > Less dramatic, but perhaps more standardly structured is the nodejs/node\n> > repository, with these stats:\n> >\n> > Repack Method       Pack Size       Time\n> > ------------------------------------------\n> > Hash v1                739.9M      71.18s\n> > Hash v2                764.6M      67.82s\n> > Path Walk              698.0M      75.10s\n\nSame here.\n\n> > Even the Linux kernel repository gains some benefits, even though the number\n> > of hash collisions is relatively low due to a preference for short\n> > filenames:\n> >\n> > Repack Method       Pack Size       Time\n> > ------------------------------------------\n> > Hash v1                  2.5G     554.41s\n> > Hash v2                  2.5G     549.62s\n> > Path Walk                2.2G     559.00s\n\nOK, so here the savings are a little more substantial, and the\nperformance hit isn't too bad.\n\n> This third one, v2 not performing much better than v1, is quite\n> surprising.\n\nI'm not sure... I think Stolee's \"the number of hash collisions is\nrelatively low due to preference for short filenames\" is why v2 behaves\nso similarly to v1 here.\n\n> > The drawbacks of the --path-walk feature is that it will be harder to\n> > integrate it with bitmap features, specifically delta islands. This is not\n> > insurmountable, but would require more work, such as a revision walk to\n> > paint objects with reachability information before using that during delta\n> > computations.\n> >\n> > However, there should still be significant benefits to Git clients trying to\n> > save space and improve local performance.\n>\n> Sure.  More experiments and more approaches will eventually give us\n> overall improvement.  I am hoping that we will be able to condense\n> the result of these different approaches and their combinations into\n> easy-to-choose-from canned choices (as opposed to a myriad of little\n> knobs the users need to futz with without really understanding what\n> they are tweaking).\n\nIn the above three examples we see some trade-offs between pack size and\nthe time it took to generate it. I think it's worth discussing whether\nor not the potential benefit of such a trade-off is worth the\nsignificant complexity and code that this feature will introduce. (To be\nclear, I don't have a strong opinion here one way or the other, but I do\nthink that it's at least worth discussing).\n\nI wonder how much of the benefits of path-walk over the hash v2 approach\ncould be had by simply widening the pack.window during delta selection?\n\nI tried to run a similar experiment as you did above on the\nmicrosoft/fluentui repository and got the following:\n\n    Repack Method       Pack Size       Time\n    ------------------------------------------\n    Hash v1              447.2MiB      932.41s\n    Hash v2              154.1MiB      404.35s\n    Hash v2 (window=20)  146.7MiB      472.66s\n    Hash v2 (window=50)  138.3MiB      622.13s\n    Path Walk            140.8MiB      168.86s\n\nIn your experiment above on the same repository, the path walk feature\nrepresents an 11.873% reduction in pack size, but at the cost of a 30.9%\nregression in runtime.\n\nWhen I set pack.window to \"50\" (over the default value of \"10\"), I get a\n~10.3% reduction in pack size at the cost of a 54% increase in runtime\n(relative to just --name-hash-version=2 with the default pack.window\nsettings).\n\nBut when I set the pack.window to \"20\", the relative values (again\ncomparing against --name-hash-version=2 with the default pack.window)\nare 4.8% reduction in pack size and a 16.9% increase in runtime.\n\nBut these numbers are pretty confusing to me, TBH. The reduction in pack\nsizes makes sense, and here I see numbers that are on-par with what you\nnoted above for the same repository. But the runtimes are wildly\ndifferent (e.g., hash v1 takes you just 87s while mine takes 932s).\n\nThere must be something in our environment that is different. I'm\nstarting with a bare clone of microsoft/fluentui from GitHub, and made\nseveral 'cp -al' copies of it for the different experiments. In the\npenultimate one, I ran:\n\n    $ time git.compile -c pack.window=50 repack --name-hash-version=2 \\\n        -adF --no-write-bitmap-index\n\n, and similarly for the other experiments with appropriate values for\npack.window, --name-hash-version, and --path-walk, when applicable. All\nof this was done on a -O2 build of Git with your patches on top.\n\nSo I'm not sure what to make of these results. Clearly on my machine\nsomething is different that makes path-walk much faster than hash v2.\nBut on your machine it's slower, so I don't know how much I trust the\ntiming results from either machine.\n\nIn any event, it seems like at least in this example we can get\nperformance that is on-par with path-walk by simply widening the\npack.window when using hash v2. On my machine that seems to cost more\ntime than it does for you to the point where it's slower than my\npath-walk. But I think I need to understand what the differences are\nhere before we can draw any conclusions on the size or timing.\n\nIf the overwhelming majority of cases where the --path-walk feature\npresents a significant benefit over hash v2 at various pack.window sizes\n(where we could get approximately the same reduction in pack size with\napproximately the same end-to-end runtime of 'git repack'), then I feel\nwe might want to reconsider whether or not the complexity of this feature\nis worthwhile.\n\nBut if the --path-walk feature either gives us a significant size\nbenefit that we can't get with hash v2 and a wider pack.window without\npaying a significant runtime cost (or vice-versa), then this feature\nwould indeed be worthwhile.\n\nI also have no idea how representative the above is of your intended\nuse-case, which seems much more oriented around pushes than from-scratch\nrepacks, which would also affect our conclusions here.\n\nThanks,\nTaylor\n"},{"id":"514111","messageId":"Z9H2E9hEWgaS9NnP@nand.local","threadId":"63083","inReplyTo":"a2ed1f2d4e3946c563f934fcaf149050d50f255f.1741571455.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 01/13] pack-objects: extract should_attempt_deltas()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-12T21:01:07Z","receivedAt":"2025-03-12T21:01:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 10, 2025 at 01:50:43AM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <stolee@gmail.com>\n>\n> This will be helpful in a future change, which will reuse this logic.\n>\n> Signed-off-by: Derrick Stolee <stolee@gmail.com>\n> ---\n>  builtin/pack-objects.c | 53 +++++++++++++++++++++++-------------------\n>  1 file changed, 29 insertions(+), 24 deletions(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 58a9b161262..1d0992a8dac 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3196,6 +3196,33 @@ static int add_ref_tag(const char *tag UNUSED, const char *referent UNUSED, cons\n>  \treturn 0;\n>  }\n>\n> +static int should_attempt_deltas(struct object_entry *entry)\n> +{\n> +\tif (DELTA(entry))\n> +\t\treturn 0;\n> +\n> +\tif (!entry->type_valid ||\n> +\t    oe_size_less_than(&to_pack, entry, 50))\n> +\t\treturn 0;\n> +\n> +\tif (entry->no_try_delta)\n> +\t\treturn 0;\n> +\n> +\tif (!entry->preferred_base) {\n> +\t\tif (oe_type(entry) < 0)\n> +\t\t\tdie(_(\"unable to get type of object %s\"),\n> +\t\t\t\toid_to_hex(&entry->idx.oid));\n> +\t} else if (oe_type(entry) < 0) {\n> +\t\t/*\n> +\t\t * This object is not found, but we\n> +\t\t * don't have to include it anyway.\n> +\t\t */\n> +\t\treturn 0;\n> +\t}\n> +\n> +\treturn 1;\n> +}\n> +\n>  static void prepare_pack(int window, int depth)\n>  {\n>  \tstruct object_entry **delta_list;\n> @@ -3226,33 +3253,11 @@ static void prepare_pack(int window, int depth)\n>  \tfor (i = 0; i < to_pack.nr_objects; i++) {\n>  \t\tstruct object_entry *entry = to_pack.objects + i;\n>\n> -\t\tif (DELTA(entry))\n> -\t\t\t/* This happens if we decided to reuse existing\n> -\t\t\t * delta from a pack.  \"reuse_delta &&\" is implied.\n> -\t\t\t */\n\nIt looks like this comment went away when this part of prepare_pack()\nwas extracted into should_attempt_deltas().\n\n> -\t\t\tcontinue;\n> -\n> -\t\tif (!entry->type_valid ||\n> -\t\t    oe_size_less_than(&to_pack, entry, 50))\n> +\t\tif (!should_attempt_deltas(entry))\n>  \t\t\tcontinue;\n>\n> -\t\tif (entry->no_try_delta)\n> -\t\t\tcontinue;\n> -\n> -\t\tif (!entry->preferred_base) {\n> +\t\tif (!entry->preferred_base)\n>  \t\t\tnr_deltas++;\n\nMakes sense; should_attempt_deltas() doesn't itself change nr_deltas, so\nwe want to do it ourselves here. Looking good!\n\nThanks,\nTaylor\n"},{"id":"514112","messageId":"Z9H5JsicyLWXagxS@nand.local","threadId":"63083","inReplyTo":"9b31dc87bb61f4d73eced02a24baea58bc51aa5e.1741571455.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 02/13] pack-objects: add --path-walk option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-12T21:14:14Z","receivedAt":"2025-03-12T21:14:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 10, 2025 at 01:50:44AM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <stolee@gmail.com>\n>\n> In order to more easily compute delta bases among objects that appear at\n> the exact same path, add a --path-walk option to 'git pack-objects'.\n>\n> This option will use the path-walk API instead of the object walk given\n> by the revision machinery. Since objects will be provided in batches\n> representing a common path, those objects can be tested for delta bases\n> immediately instead of waiting for a sort of the full object list by\n> name-hash. This has multiple benefits, including avoiding collisions by\n> name-hash.\n>\n> The objects marked as UNINTERESTING are included in these batches, so we\n> are guaranteeing some locality to find good delta bases.\n>\n> After the individual passes are done on a per-path basis, the default\n> name-hash is used to find other opportunistic delta bases that did not\n> match exactly by the full path name.\n>\n> The current implementation performs delta calculations while walking\n> objects, which is not ideal for a few reasons. First, this will cause\n> the \"Enumerating objects\" phase to be much longer than usual. Second, it\n> does not take advantage of threading during the path-scoped delta\n> calculations. Even with this lack of threading, the path-walk option is\n> sometimes faster than the usual approach. Future changes will refactor\n> this code to allow for threading, but that complexity is deferred until\n> later to keep this patch as simple as possible.\n>\n> This new walk is incompatible with some features and is ignored by\n> others:\n>\n>  * Object filters are not currently integrated with the path-walk API,\n>    such as sparse-checkout or tree depth. A blobless packfile could be\n>    integrated easily, but that is deferred for later.\n>\n>  * Server-focused features such as delta islands, shallow packs, and\n>    using a bitmap index are incompatible with the path-walk API.\n>\n>  * The path walk API is only compatible with the --revs option, not\n>    taking object lists or pack lists over stdin. These alternative ways\n>    to specify the objects currently ignores the --path-walk option\n>    without even a warning.\n>\n> Future changes will create performance tests that demonstrate the power\n> of this approach.\n>\n> Signed-off-by: Derrick Stolee <stolee@gmail.com>\n> ---\n>  Documentation/git-pack-objects.adoc        |  13 +-\n>  Documentation/technical/api-path-walk.adoc |   1 +\n>  builtin/pack-objects.c                     | 147 +++++++++++++++++++--\n>  t/t5300-pack-object.sh                     |  15 +++\n>  4 files changed, 166 insertions(+), 10 deletions(-)\n>\n> diff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc\n> index 7f69ae4855f..7dbbe6d54d2 100644\n> --- a/Documentation/git-pack-objects.adoc\n> +++ b/Documentation/git-pack-objects.adoc\n> @@ -16,7 +16,7 @@ SYNOPSIS\n>  \t[--cruft] [--cruft-expiration=<time>]\n>  \t[--stdout [--filter=<filter-spec>] | <base-name>]\n>  \t[--shallow] [--keep-true-parents] [--[no-]sparse]\n> -\t[--name-hash-version=<n>] < <object-list>\n> +\t[--name-hash-version=<n>] [--path-walk] < <object-list>\n>\n>\n>  DESCRIPTION\n> @@ -375,6 +375,17 @@ many different directories. At the moment, this version is not allowed\n>  when writing reachability bitmap files with `--write-bitmap-index` and it\n>  will be automatically changed to version `1`.\n>\n> +--path-walk::\n> +\tBy default, `git pack-objects` walks objects in an order that\n> +\tpresents trees and blobs in an order unrelated to the path they\n> +\tappear relative to a commit's root tree. The `--path-walk` option\n> +\tenables a different walking algorithm that organizes trees and\n> +\tblobs by path. This has the potential to improve delta compression\n> +\tespecially in the presence of filenames that cause collisions in\n> +\tGit's default name-hash algorithm. Due to changing how the objects\n> +\tare walked, this option is not compatible with `--delta-islands`,\n> +\t`--shallow`, or `--filter`.\n\nI think from reading further below that this feature is somewhat\nincompatible with --use-bitmap-index, at least in the sense that we\nimplicitly disable the latter whenever we see the former. Would that be\nworth mentioning here?\n\n> +static int add_objects_by_path(const char *path,\n> +\t\t\t       struct oid_array *oids,\n> +\t\t\t       enum object_type type,\n> +\t\t\t       void *data)\n> +{\n> +\tstruct object_entry **delta_list;\n> +\tsize_t oe_start = to_pack.nr_objects;\n> +\tsize_t oe_end;\n> +\tunsigned int sub_list_size;\n> +\tunsigned int *processed = data;\n> +\n> +\t/*\n> +\t * First, add all objects to the packing data, including the ones\n> +\t * marked UNINTERESTING (translated to 'exclude') as they can be\n> +\t * used as delta bases.\n> +\t */\n> +\tfor (size_t i = 0; i < oids->nr; i++) {\n> +\t\tint exclude;\n> +\t\tstruct object_info oi = OBJECT_INFO_INIT;\n> +\t\tstruct object_id *oid = &oids->oid[i];\n> +\n> +\t\t/* Skip objects that do not exist locally. */\n> +\t\tif (exclude_promisor_objects &&\n> +\t\t    oid_object_info_extended(the_repository, oid, &oi,\n> +\t\t\t\t\t     OBJECT_INFO_FOR_PREFETCH) < 0)\n> +\t\t\tcontinue;\n> +\n> +\t\texclude = !is_oid_interesting(the_repository, oid);\n> +\n> +\t\tif (exclude && !thin)\n> +\t\t\tcontinue;\n> +\n> +\t\tadd_object_entry(oid, type, path, exclude);\n> +\t}\n> +\n> +\toe_end = to_pack.nr_objects;\n> +\n> +\t/* We can skip delta calculations if it is a no-op. */\n> +\tif (oe_end == oe_start || !window)\n> +\t\treturn 0;\n> +\n> +\tsub_list_size = 0;\n> +\tALLOC_ARRAY(delta_list, oe_end - oe_start);\n\nMakes sense, and seems all reasonable.\n\n> +\tfor (size_t i = 0; i < oe_end - oe_start; i++) {\n> +\t\tstruct object_entry *entry = to_pack.objects + oe_start + i;\n> +\n> +\t\tif (!should_attempt_deltas(entry))\n> +\t\t\tcontinue;\n> +\n> +\t\tdelta_list[sub_list_size++] = entry;\n> +\t}\n> +\n> +\t/*\n> +\t * Find delta bases among this list of objects that all match the same\n> +\t * path. This causes the delta compression to be interleaved in the\n> +\t * object walk, which can lead to confusing progress indicators. This is\n> +\t * also incompatible with threaded delta calculations. In the future,\n> +\t * consider creating a list of regions in the full to_pack.objects array\n> +\t * that could be picked up by the threaded delta computation.\n> +\t */\n> +\tif (sub_list_size && window) {\n> +\t\tQSORT(delta_list, sub_list_size, type_size_sort);\n> +\t\tfind_deltas(delta_list, &sub_list_size, window, depth, processed);\n> +\t}\n\nInteresting. I like the \"regions in to_pack.objects\" idea as a way of\nthreading the delta selection process in the future.\n\nThanks,\nTaylor\n"},{"id":"514113","messageId":"Z9H5Ng3niRodVp9T@nand.local","threadId":"63083","inReplyTo":"bc678acb109c34ac96c641c5e6fbfb418e447010.1741571455.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 03/13] pack-objects: update usage to match docs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-12T21:14:30Z","receivedAt":"2025-03-12T21:14:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 10, 2025 at 01:50:45AM +0000, Derrick Stolee via GitGitGadget wrote:\n> ---\n>  Documentation/git-pack-objects.adoc | 14 +++++++-------\n>  builtin/pack-objects.c              | 10 ++++++++--\n>  t/t0450/adoc-help-mismatches        |  1 -\n>  3 files changed, 15 insertions(+), 10 deletions(-)\n\nThanks for cleaning these up.\n\nThanks,\nTaylor\n"},{"id":"514114","messageId":"Z9H65fCcv/cybKRa@nand.local","threadId":"63083","inReplyTo":"c047fbd7f275ef79695b5e1356075705e6fd7bc8.1741571455.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 10/13] pack-objects: refactor path-walk delta phase","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-03-12T21:21:41Z","receivedAt":"2025-03-12T21:21:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 10, 2025 at 01:50:52AM +0000, Derrick Stolee via GitGitGadget wrote:\n> ---\n>  builtin/pack-objects.c       | 82 +++++++++++++++++++++++++-----------\n>  pack-objects.h               | 12 ++++++\n>  t/t5300-pack-object.sh       |  8 +++-\n>  t/t5530-upload-pack-error.sh |  6 ---\n>  4 files changed, 75 insertions(+), 33 deletions(-)\n\nAh, nice :-). This is where you implement the idea that you were\nmentioning back in\n\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index c756ce50dd7..c5a3129c88e 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3233,6 +3233,51 @@ static int should_attempt_deltas(struct object_entry *entry)\n>  \treturn 1;\n>  }\n>\n> +static void find_deltas_for_region(struct object_entry *list UNUSED,\n\nInteresting, it looks like \"list\" here is UNUSED in this patch. On first\nread I figured that you were going to make use of it in later patches,\nbut it looks like it remains UNUSED at the tip of my local copy of this\nseries.\n\nIs this a stray change from when you were writing this? Something else?\n\n> +\t\t\t\t   struct packing_region *region,\n> +\t\t\t\t   unsigned int *processed)\n> +{\n> +\tstruct object_entry **delta_list;\n> +\tuint32_t delta_list_nr = 0;\n\nI know that we have a lot of \"_nr\" and \"_alloc\" variables in the\npack-objects code that use uint32_t as their type, but we should prefer\nsize_t for these in the future, even if it breaks the existing pattern.\n\nAs much as I can grok of the implementation through the rest of the\npatch makes sense to me and looks good.\n\nThanks,\nTaylor\n"},{"id":"514803","messageId":"2649010e-e4c7-4c39-b697-ecf89549c0d4@gmail.com","threadId":"63083","inReplyTo":"Z9H5JsicyLWXagxS@nand.local","subject":"Re: [PATCH 02/13] pack-objects: add --path-walk option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-03-20T19:46:05Z","receivedAt":"2025-03-20T19:46:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/12/2025 5:14 PM, Taylor Blau wrote:\n> On Mon, Mar 10, 2025 at 01:50:44AM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee <stolee@gmail.com>\n\n>> +--path-walk::\n>> +\tBy default, `git pack-objects` walks objects in an order that\n>> +\tpresents trees and blobs in an order unrelated to the path they\n>> +\tappear relative to a commit's root tree. The `--path-walk` option\n>> +\tenables a different walking algorithm that organizes trees and\n>> +\tblobs by path. This has the potential to improve delta compression\n>> +\tespecially in the presence of filenames that cause collisions in\n>> +\tGit's default name-hash algorithm. Due to changing how the objects\n>> +\tare walked, this option is not compatible with `--delta-islands`,\n>> +\t`--shallow`, or `--filter`.\n> \n> I think from reading further below that this feature is somewhat\n> incompatible with --use-bitmap-index, at least in the sense that we\n> implicitly disable the latter whenever we see the former. Would that be\n> worth mentioning here?\n\nWhile it is not incompatible and does not even include a warning, the\n--use-bitmap-index option gets silently ignored. This matches how\n--shallow does similar things. I'll still add to this doc as I agree\nit would be helpful.\n\nThanks,\n-Stolee\n"},{"id":"514804","messageId":"afa6b6a2-c301-45e5-a931-66d146392ad1@gmail.com","threadId":"63083","inReplyTo":"Z9H2E9hEWgaS9NnP@nand.local","subject":"Re: [PATCH 01/13] pack-objects: extract should_attempt_deltas()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-03-20T19:48:55Z","receivedAt":"2025-03-20T19:48:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\nOn 3/12/2025 5:01 PM, Taylor Blau wrote:\n> On Mon, Mar 10, 2025 at 01:50:43AM +0000, Derrick Stolee via GitGitGadget wrote:\n\n>> +static int should_attempt_deltas(struct object_entry *entry)\n>> +{\n>> +\tif (DELTA(entry))\n>> +\t\treturn 0;\n>> +\n...>> -\t\tif (DELTA(entry))\n>> -\t\t\t/* This happens if we decided to reuse existing\n>> -\t\t\t * delta from a pack.  \"reuse_delta &&\" is implied.\n>> -\t\t\t */\n> \n> It looks like this comment went away when this part of prepare_pack()\n> was extracted into should_attempt_deltas().\n> \n>> -\t\t\tcontinue;\n\nGood catch. \n\nThanks,\n-Stolee\n"},{"id":"514805","messageId":"58750d8c-fb1e-4c6d-a356-7cd0e59c621d@gmail.com","threadId":"63083","inReplyTo":"Z9H65fCcv/cybKRa@nand.local","subject":"Re: [PATCH 10/13] pack-objects: refactor path-walk delta phase","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-03-20T19:57:09Z","receivedAt":"2025-03-20T19:57:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/12/2025 5:21 PM, Taylor Blau wrote:\n> On Mon, Mar 10, 2025 at 01:50:52AM +0000, Derrick Stolee via GitGitGadget wrote:\n\n>> +static void find_deltas_for_region(struct object_entry *list UNUSED,\n> \n> Interesting, it looks like \"list\" here is UNUSED in this patch. On first\n> read I figured that you were going to make use of it in later patches,\n> but it looks like it remains UNUSED at the tip of my local copy of this\n> series.\n> \n> Is this a stray change from when you were writing this? Something else?\n\nActually, the intention was to make this method less focused on globals,\nbut it in fact uses 'to_pack.objects' when it should use 'list'. Thanks\nfor the careful eye here. \n>> +\t\t\t\t   struct packing_region *region,\n>> +\t\t\t\t   unsigned int *processed)\n>> +{\n>> +\tstruct object_entry **delta_list;\n>> +\tuint32_t delta_list_nr = 0;\n> \n> I know that we have a lot of \"_nr\" and \"_alloc\" variables in the\n> pack-objects code that use uint32_t as their type, but we should prefer\n> size_t for these in the future, even if it breaks the existing pattern.\nUnfortunately, the fact that these are used in the find_deltas() method\nas an 'unsigned *' reference, these types need to be as they are. I'm\nhesitant to use 64-bit integers that will eventually cast down to 32-bit\nin these instances.\n\nThanks,\n-Stolee\n\n\n"},{"id":"514807","messageId":"192714d0-d351-405b-9186-367c5212aeca@gmail.com","threadId":"63083","inReplyTo":"Z9Hy6Yk2XM1RCsNC@nand.local","subject":"Re: [PATCH 00/13] PATH WALK II: Add --path-walk option to 'git pack-objects'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-03-20T20:18:32Z","receivedAt":"2025-03-20T20:18:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/12/2025 4:47 PM, Taylor Blau wrote:\n> On Mon, Mar 10, 2025 at 10:28:22AM -0700, Junio C Hamano wrote:\n>> \"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> In the above three examples we see some trade-offs between pack size and\n> the time it took to generate it. I think it's worth discussing whether\n> or not the potential benefit of such a trade-off is worth the\n> significant complexity and code that this feature will introduce. (To be\n> clear, I don't have a strong opinion here one way or the other, but I do\n> think that it's at least worth discussing).\n> \n> I wonder how much of the benefits of path-walk over the hash v2 approach\n> could be had by simply widening the pack.window during delta selection?\n> \n> I tried to run a similar experiment as you did above on the\n> microsoft/fluentui repository and got the following:\n> \n>     Repack Method       Pack Size       Time\n>     ------------------------------------------\n>     Hash v1              447.2MiB      932.41s\n>     Hash v2              154.1MiB      404.35s\n>     Hash v2 (window=20)  146.7MiB      472.66s\n>     Hash v2 (window=50)  138.3MiB      622.13s\n>     Path Walk            140.8MiB      168.86s\n> \n> In your experiment above on the same repository, the path walk feature\n> represents an 11.873% reduction in pack size, but at the cost of a 30.9%\n> regression in runtime.\n> \n> When I set pack.window to \"50\" (over the default value of \"10\"), I get a\n> ~10.3% reduction in pack size at the cost of a 54% increase in runtime\n> (relative to just --name-hash-version=2 with the default pack.window\n> settings).\n> \n> But when I set the pack.window to \"20\", the relative values (again\n> comparing against --name-hash-version=2 with the default pack.window)\n> are 4.8% reduction in pack size and a 16.9% increase in runtime.\n\nYou're right that I wasn't including data around the --window option in\nmy analysis. This option presents folks with the opportunity to add CPU\ntime in order to improve the possibility of better compression due to\nconsidering more object pairs.\n\nBut it's also important to note that that option still works with\n--path-walk, except that the --path-walk option is focused on improving\nthe quality of objects being considered within a window. There's also the\naspect that there are two passes (one path-based and one name-hash-based)\nso increasing the --window size has a larger impact on the --path-walk\noption.\n\nWith regards to the microsoft/fluentui repo, I had previously been using\nan old clone using --bare. The size changes if I use --mirror as well,\nsince it will get the fork hint refs corresponding to objects in public\nforks that are not actually in the core repo. This changes the clone size\nas well as the repacked size.\n\n(To save time, I didn't repeat the --window option tests for name hash\nv1 as name hash v2 is clearly superior to that option in this repo.)\n\nCloned with --bare:\n\n| Type         | Window: 10     | Window: 20     | Window: 50     |\n|--------------|----------------|----------------|----------------|\n| name hash v1 | 451 M | 1m 42s |       |        |       |        |\n| name hash v2 | 160 M | 35.4 s | 151 M | 25.4 s | 141 M | 31.0 s |\n| --path-walk  | 141 M | 31.0 s | 136 M | 35.7 s | 129 M | 49.3 s |\n\nCloned with --mirror:\n\n| Type         | Window: 10     | Window: 20     | Window: 50      |\n|--------------|----------------|----------------|-----------------|\n| name hash v1 | 882 M | 3m 27s |       |        |       |         |\n| name hash v2 | 584 M | 70.4 s | 554 M | 54.6 s | 530 M | 69.4 s  |\n| --path-walk  | 548 M | 79.8 s | 523 M | 93.9 s | 507 M | 126.2 s |\n\nRunning on a slightly-larger Javascript repo with the same CHANGLOG\nfilename issue, I get these results:\n\n| Type         | Window: 10     | Window: 20     | Window: 50     |\n|--------------|----------------|----------------|----------------|\n| name hash v1 | 6.4 G | 36m 9s |       |        |       |        |\n| name hash v2 | 920 M | 7m 39s | 767 M | 5m 49s | 665 M | 6m 12s |\n| --path-walk  | 834 M | 4m 48s | 697 M | 7m 39s | 615 M | 8m 42s |\n\n> But these numbers are pretty confusing to me, TBH. The reduction in pack\n> sizes makes sense, and here I see numbers that are on-par with what you\n> noted above for the same repository. But the runtimes are wildly\n> different (e.g., hash v1 takes you just 87s while mine takes 932s).\n\nI wonder if it's related to threading? I'm using as many cores as I can.\n\n> There must be something in our environment that is different. I'm\n> starting with a bare clone of microsoft/fluentui from GitHub, and made\n> several 'cp -al' copies of it for the different experiments. In the\n> penultimate one, I ran:\n> \n>     $ time git.compile -c pack.window=50 repack --name-hash-version=2 \\\n>         -adF --no-write-bitmap-index\n\nThere's also some strange things with my numbers because I'm not copying\nthe same data into multiple places but instead running the test on the\nsame repo. Thus, the \"input size\" is changing with each run and this is\nprobably a big factor in the larger tests.\n\nSo the order in my tables is left-to-right, top-to-bottom, like reading\na page in English. Thus, the short time for --path-walk --window=10 in\nthe last example is maybe a bit faster because it is starting from the\n665 M from the --name-hash-version=2 --window=50 example. \n> In any event, it seems like at least in this example we can get\n> performance that is on-par with path-walk by simply widening the\n> pack.window when using hash v2. On my machine that seems to cost more\n> time than it does for you to the point where it's slower than my\n> path-walk. But I think I need to understand what the differences are\n> here before we can draw any conclusions on the size or timing.\n\nI'd be very curious to see if more folks have bandwidth to do similar\ntesting. My default mode is that I like giving users more options to\nexplore which may work better for them. \n> If the overwhelming majority of cases where the --path-walk feature\n> presents a significant benefit over hash v2 at various pack.window sizes\n> (where we could get approximately the same reduction in pack size with\n> approximately the same end-to-end runtime of 'git repack'), then I feel\n> we might want to reconsider whether or not the complexity of this feature\n> is worthwhile.\n> \n> But if the --path-walk feature either gives us a significant size\n> benefit that we can't get with hash v2 and a wider pack.window without\n> paying a significant runtime cost (or vice-versa), then this feature\n> would indeed be worthwhile.\n> \n> I also have no idea how representative the above is of your intended\n> use-case, which seems much more oriented around pushes than from-scratch\n> repacks, which would also affect our conclusions here.The push story is valuable, but I'm also interested in helping users shrink their local repositories in whatever means they are\nwilling to wait for.\n\n---\n\nMeta-response to your patch review: I have made adjustments to my local\nbranch in response to the points you brought up. I'll hold off on v2\nfor a few more days to give more opportunity for review.\n\nThanks,\n-Stolee\n\n"},{"id":"514938","messageId":"pull.1819.v2.git.1742829769.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.git.1741571455.gitgitgadget@gmail.com","subject":"[PATCH v2 00/13] PATH WALK II: Add --path-walk option to 'git pack-objects'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-24T15:22:36Z","receivedAt":"2025-03-24T15:22:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Here is a full submission of the --path-walk feature for 'git pack-objects'\nand 'git repack'. It's been discussed in an RFC [1], as a future application\nfor the path walk API [2], and is updated now that --name-hash-version=2\nexists (as a replacement for the --full-name-hash option from the RFC) [3].\n\n[1]\nhttps://lore.kernel.org/git/pull.1813.v2.git.1729431810.gitgitgadget@gmail.com/\n\n[2]\nhttps://lore.kernel.org/git/pull.1818.git.1730356023.gitgitgadget@gmail.com\n\n[3]\nhttps://lore.kernel.org/git/pull.1813.git.1728396723.gitgitgadget@gmail.com\n\nThis patch series does the following:\n\n 1. Add a new '--path-walk' option to 'git pack-objects' that uses the\n    path-walk API instead of the revision API to collect objects for delta\n    compression.\n\n 2. Add a new '--path-walk' option to 'git repack' to pass this option along\n    to 'git pack-objects'.\n\n 3. Add a new 'pack.usePathWalk' config option to opt into this option\n    implicitly, such as in 'git push'.\n\n 4. Optimize the '--path-walk' option using threading so it better competes\n    with the existing multi-threaded delta compression mechanism.\n\n 5. Update the path-walk API with a new 'edge_aggressive' option that pairs\n    close to the --edge-aggressive option in the revision API. This is\n    useful when creating thin packs inside shallow clones.\n\nThis feature works by using the path-walk API to emit groups of objects that\nappear at the same path. These groups are tracked so they can be tested for\ndelta compression with each other, and then after those groups are tested a\nsecond pass using the name-hash attempts to find better (or first time)\ndeltas across path boundaries. This second pass is much faster than a fresh\npass since the existing deltas are used as a limit for the size of\npotentially new deltas, short-circuiting the checks when the delta size\nexceeds the current-best.\n\nThe benefits of the --path-walk feature first come into play when the name\nhash functions have many collisions, so sorting by name hash value leads to\nunhelpful groupings of objects. Many of these benefits are improved by\n--name-hash-version=2, but collisions still exist with any hash-based\napproach. There are also performance benefits in some cases due to the\nisolation of delta compression testing within path groups.\n\nAll of the benefits of the --path-walk feature are less dramatic when\ncompared to --name-hash-version=2, but they can still exist in many cases. I\nhave also seen some cases where --name-hash-version=2 compresses better than\n--path-walk with --name-hash-version=1, but these options can be combined to\nget the best of both worlds.\n\nDetailed statistics are provided within patch messages, but a few are\nhighlighted here:\n\nThe microsoft/fluentui is a public Javascript repo that suffers from many of\nthe name hash collisions as internal repositories I've worked with. Here is\na comparison of the compressed size and end-to-end time of the repack:\n\nRepack Method    Pack Size       Time\n---------------------------------------\nHash v1             439.4M      87.24s\nHash v2             161.7M      21.51s\nPath Walk           142.5M      28.16s\n\n\nLess dramatic, but perhaps more standardly structured is the nodejs/node\nrepository, with these stats:\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                739.9M      71.18s\nHash v2                764.6M      67.82s\nPath Walk              698.0M      75.10s\n\n\nEven the Linux kernel repository gains some benefits, even though the number\nof hash collisions is relatively low due to a preference for short\nfilenames:\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                  2.5G     554.41s\nHash v2                  2.5G     549.62s\nPath Walk                2.2G     559.00s\n\n\nThe drawbacks of the --path-walk feature is that it will be harder to\nintegrate it with bitmap features, specifically delta islands. This is not\ninsurmountable, but would require more work, such as a revision walk to\npaint objects with reachability information before using that during delta\ncomputations.\n\nHowever, there should still be significant benefits to Git clients trying to\nsave space and improve local performance.\n\nThis feature was shipped with similar features in microsoft/git as of\nv2.47.0.vfs.0.3 [4]. This was used in CI machines for an internal monorepo\nthat had significant repository growth due to constructing a batch of\nbeachball [5] CHANGELOG.[md|json] files and pushing them to a release\nbranch. These pushes were frequently 70-200 MB due to poor delta\ncompression. Using the 'pack.usePathWalk=true' config, these pushes dropped\nin size by 100x while improving performance. Since these CI machines were\nworking with a shallow clone, the 'edge_aggressive' changes were required to\nenable the path-walk option.\n\n[4] https://github.com/microsoft/git/releases/tag/v2.47.0.vfs.0.3\n\n[5] https://github.com/microsoft/beachball\n\n\nUpdates in v2\n=============\n\n * Re-added a dropped comment when moving code in patch 1.\n * Updated documentation to include interaction with --use-bitmap-index.\n * An UNUSED parameter is now used, reducing the use of global variables\n   slightly.\n\nThanks, -Stolee\n\nDerrick Stolee (13):\n  pack-objects: extract should_attempt_deltas()\n  pack-objects: add --path-walk option\n  pack-objects: update usage to match docs\n  p5313: add performance tests for --path-walk\n  pack-objects: introduce GIT_TEST_PACK_PATH_WALK\n  t5538: add tests to confirm deltas in shallow pushes\n  repack: add --path-walk option\n  pack-objects: enable --path-walk via config\n  scalar: enable path-walk during push via config\n  pack-objects: refactor path-walk delta phase\n  pack-objects: thread the path-based compression\n  path-walk: add new 'edge_aggressive' option\n  pack-objects: allow --shallow and --path-walk\n\n Documentation/config/feature.adoc          |   4 +\n Documentation/config/pack.adoc             |   8 +\n Documentation/git-pack-objects.adoc        |  26 +-\n Documentation/git-repack.adoc              |  14 +-\n Documentation/technical/api-path-walk.adoc |   9 +\n builtin/pack-objects.c                     | 414 +++++++++++++++++++--\n builtin/repack.c                           |   7 +-\n pack-objects.h                             |  12 +\n path-walk.c                                |   6 +-\n path-walk.h                                |   7 +\n repo-settings.c                            |   3 +\n repo-settings.h                            |   1 +\n scalar.c                                   |   1 +\n t/README                                   |   4 +\n t/helper/test-path-walk.c                  |   2 +\n t/perf/p5313-pack-objects.sh               |  37 +-\n t/t0411-clone-from-partial.sh              |   6 +\n t/t0450/adoc-help-mismatches               |   1 -\n t/t5300-pack-object.sh                     |  19 +\n t/t5306-pack-nobase.sh                     |   5 +\n t/t5310-pack-bitmaps.sh                    |  13 +-\n t/t5316-pack-delta-depth.sh                |   9 +-\n t/t5332-multi-pack-reuse.sh                |   7 +\n t/t5538-push-shallow.sh                    |  34 ++\n t/t6601-path-walk.sh                       |  20 +\n t/t7406-submodule-update.sh                |   3 +\n 26 files changed, 605 insertions(+), 67 deletions(-)\n\n\nbase-commit: a36e024e989f4d35f35987a60e3af8022cac3420\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1819%2Fderrickstolee%2Fpath-walk-upstream-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1819/derrickstolee/path-walk-upstream-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1819\n\nRange-diff vs v1:\n\n  1:  a2ed1f2d4e3 !  1:  57c1cc20de0 pack-objects: extract should_attempt_deltas()\n     @@ builtin/pack-objects.c: static int add_ref_tag(const char *tag UNUSED, const cha\n      +static int should_attempt_deltas(struct object_entry *entry)\n      +{\n      +\tif (DELTA(entry))\n     ++\t\t/* This happens if we decided to reuse existing\n     ++\t\t * delta from a pack. \"reuse_delta &&\" is implied.\n     ++\t\t */\n      +\t\treturn 0;\n      +\n      +\tif (!entry->type_valid ||\n  2:  9b31dc87bb6 !  2:  a271d6245c2 pack-objects: add --path-walk option\n     @@ Documentation/git-pack-objects.adoc: many different directories. At the moment,\n      +\tespecially in the presence of filenames that cause collisions in\n      +\tGit's default name-hash algorithm. Due to changing how the objects\n      +\tare walked, this option is not compatible with `--delta-islands`,\n     -+\t`--shallow`, or `--filter`.\n     ++\t`--shallow`, or `--filter`. The `--use-bitmap-index` option will\n     ++\tbe ignored in the presence of `--path-walk.`\n      +\n       \n       DELTA ISLANDS\n  3:  bc678acb109 =  3:  dcff01392ff pack-objects: update usage to match docs\n  4:  de848ebff74 =  4:  97a0b52ccee p5313: add performance tests for --path-walk\n  5:  4bfd438b5fa =  5:  0d49bb3d30a pack-objects: introduce GIT_TEST_PACK_PATH_WALK\n  6:  cfee2136e92 =  6:  ddf804e606a t5538: add tests to confirm deltas in shallow pushes\n  7:  1e75f068281 =  7:  11767e7653e repack: add --path-walk option\n  8:  073cb44a47c =  8:  dd66a5b46f2 pack-objects: enable --path-walk via config\n  9:  fd9a335329e =  9:  e5624c379d5 scalar: enable path-walk during push via config\n 10:  c047fbd7f27 ! 10:  622439d7855 pack-objects: refactor path-walk delta phase\n     @@ builtin/pack-objects.c: static int should_attempt_deltas(struct object_entry *en\n       \treturn 1;\n       }\n       \n     -+static void find_deltas_for_region(struct object_entry *list UNUSED,\n     ++static void find_deltas_for_region(struct object_entry *list,\n      +\t\t\t\t   struct packing_region *region,\n      +\t\t\t\t   unsigned int *processed)\n      +{\n     @@ builtin/pack-objects.c: static int should_attempt_deltas(struct object_entry *en\n      +\n      +\tALLOC_ARRAY(delta_list, region->nr);\n      +\tfor (uint32_t i = 0; i < region->nr; i++) {\n     -+\t\tstruct object_entry *entry = to_pack.objects + region->start + i;\n     ++\t\tstruct object_entry *entry = list + region->start + i;\n      +\t\tif (should_attempt_deltas(entry))\n      +\t\t\tdelta_list[delta_list_nr++] = entry;\n      +\t}\n     @@ pack-objects.h: struct object_entry {\n       \tuint32_t nr_objects, nr_alloc;\n       \n      +\tstruct packing_region *regions;\n     -+\tuint32_t nr_regions, nr_regions_alloc;\n     ++\tuint64_t nr_regions, nr_regions_alloc;\n      +\n       \tint32_t *index;\n       \tuint32_t index_size;\n 11:  cf990b90b34 = 11:  ae73d26319a pack-objects: thread the path-based compression\n 12:  bc9383ef11c = 12:  c45f35c204e path-walk: add new 'edge_aggressive' option\n 13:  2eb92507213 = 13:  d5484ebd942 pack-objects: allow --shallow and --path-walk\n\n-- \ngitgitgadget\n"},{"id":"514939","messageId":"57c1cc20de0c80b84ad11a6546763826fe4b1a09.1742829770.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v2.git.1742829769.gitgitgadget@gmail.com","subject":"[PATCH v2 01/13] pack-objects: extract should_attempt_deltas()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-24T15:22:37Z","receivedAt":"2025-03-24T15:22:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThis will be helpful in a future change, which will reuse this logic.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c | 56 ++++++++++++++++++++++++------------------\n 1 file changed, 32 insertions(+), 24 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 58a9b161262..7805429f5d1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3196,6 +3196,36 @@ static int add_ref_tag(const char *tag UNUSED, const char *referent UNUSED, cons\n \treturn 0;\n }\n \n+static int should_attempt_deltas(struct object_entry *entry)\n+{\n+\tif (DELTA(entry))\n+\t\t/* This happens if we decided to reuse existing\n+\t\t * delta from a pack. \"reuse_delta &&\" is implied.\n+\t\t */\n+\t\treturn 0;\n+\n+\tif (!entry->type_valid ||\n+\t    oe_size_less_than(&to_pack, entry, 50))\n+\t\treturn 0;\n+\n+\tif (entry->no_try_delta)\n+\t\treturn 0;\n+\n+\tif (!entry->preferred_base) {\n+\t\tif (oe_type(entry) < 0)\n+\t\t\tdie(_(\"unable to get type of object %s\"),\n+\t\t\t\toid_to_hex(&entry->idx.oid));\n+\t} else if (oe_type(entry) < 0) {\n+\t\t/*\n+\t\t * This object is not found, but we\n+\t\t * don't have to include it anyway.\n+\t\t */\n+\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+}\n+\n static void prepare_pack(int window, int depth)\n {\n \tstruct object_entry **delta_list;\n@@ -3226,33 +3256,11 @@ static void prepare_pack(int window, int depth)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = to_pack.objects + i;\n \n-\t\tif (DELTA(entry))\n-\t\t\t/* This happens if we decided to reuse existing\n-\t\t\t * delta from a pack.  \"reuse_delta &&\" is implied.\n-\t\t\t */\n-\t\t\tcontinue;\n-\n-\t\tif (!entry->type_valid ||\n-\t\t    oe_size_less_than(&to_pack, entry, 50))\n+\t\tif (!should_attempt_deltas(entry))\n \t\t\tcontinue;\n \n-\t\tif (entry->no_try_delta)\n-\t\t\tcontinue;\n-\n-\t\tif (!entry->preferred_base) {\n+\t\tif (!entry->preferred_base)\n \t\t\tnr_deltas++;\n-\t\t\tif (oe_type(entry) < 0)\n-\t\t\t\tdie(_(\"unable to get type of object %s\"),\n-\t\t\t\t    oid_to_hex(&entry->idx.oid));\n-\t\t} else {\n-\t\t\tif (oe_type(entry) < 0) {\n-\t\t\t\t/*\n-\t\t\t\t * This object is not found, but we\n-\t\t\t\t * don't have to include it anyway.\n-\t\t\t\t */\n-\t\t\t\tcontinue;\n-\t\t\t}\n-\t\t}\n \n \t\tdelta_list[n++] = entry;\n \t}\n-- \ngitgitgadget\n\n"},{"id":"514940","messageId":"a271d6245c243d50d95ea6c7e624b3165f7da558.1742829770.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v2.git.1742829769.gitgitgadget@gmail.com","subject":"[PATCH v2 02/13] pack-objects: add --path-walk option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-24T15:22:38Z","receivedAt":"2025-03-24T15:22:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nIn order to more easily compute delta bases among objects that appear at\nthe exact same path, add a --path-walk option to 'git pack-objects'.\n\nThis option will use the path-walk API instead of the object walk given\nby the revision machinery. Since objects will be provided in batches\nrepresenting a common path, those objects can be tested for delta bases\nimmediately instead of waiting for a sort of the full object list by\nname-hash. This has multiple benefits, including avoiding collisions by\nname-hash.\n\nThe objects marked as UNINTERESTING are included in these batches, so we\nare guaranteeing some locality to find good delta bases.\n\nAfter the individual passes are done on a per-path basis, the default\nname-hash is used to find other opportunistic delta bases that did not\nmatch exactly by the full path name.\n\nThe current implementation performs delta calculations while walking\nobjects, which is not ideal for a few reasons. First, this will cause\nthe \"Enumerating objects\" phase to be much longer than usual. Second, it\ndoes not take advantage of threading during the path-scoped delta\ncalculations. Even with this lack of threading, the path-walk option is\nsometimes faster than the usual approach. Future changes will refactor\nthis code to allow for threading, but that complexity is deferred until\nlater to keep this patch as simple as possible.\n\nThis new walk is incompatible with some features and is ignored by\nothers:\n\n * Object filters are not currently integrated with the path-walk API,\n   such as sparse-checkout or tree depth. A blobless packfile could be\n   integrated easily, but that is deferred for later.\n\n * Server-focused features such as delta islands, shallow packs, and\n   using a bitmap index are incompatible with the path-walk API.\n\n * The path walk API is only compatible with the --revs option, not\n   taking object lists or pack lists over stdin. These alternative ways\n   to specify the objects currently ignores the --path-walk option\n   without even a warning.\n\nFuture changes will create performance tests that demonstrate the power\nof this approach.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-pack-objects.adoc        |  14 +-\n Documentation/technical/api-path-walk.adoc |   1 +\n builtin/pack-objects.c                     | 147 +++++++++++++++++++--\n t/t5300-pack-object.sh                     |  15 +++\n 4 files changed, 167 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc\nindex 7f69ae4855f..7065758eddf 100644\n--- a/Documentation/git-pack-objects.adoc\n+++ b/Documentation/git-pack-objects.adoc\n@@ -16,7 +16,7 @@ SYNOPSIS\n \t[--cruft] [--cruft-expiration=<time>]\n \t[--stdout [--filter=<filter-spec>] | <base-name>]\n \t[--shallow] [--keep-true-parents] [--[no-]sparse]\n-\t[--name-hash-version=<n>] < <object-list>\n+\t[--name-hash-version=<n>] [--path-walk] < <object-list>\n \n \n DESCRIPTION\n@@ -375,6 +375,18 @@ many different directories. At the moment, this version is not allowed\n when writing reachability bitmap files with `--write-bitmap-index` and it\n will be automatically changed to version `1`.\n \n+--path-walk::\n+\tBy default, `git pack-objects` walks objects in an order that\n+\tpresents trees and blobs in an order unrelated to the path they\n+\tappear relative to a commit's root tree. The `--path-walk` option\n+\tenables a different walking algorithm that organizes trees and\n+\tblobs by path. This has the potential to improve delta compression\n+\tespecially in the presence of filenames that cause collisions in\n+\tGit's default name-hash algorithm. Due to changing how the objects\n+\tare walked, this option is not compatible with `--delta-islands`,\n+\t`--shallow`, or `--filter`. The `--use-bitmap-index` option will\n+\tbe ignored in the presence of `--path-walk.`\n+\n \n DELTA ISLANDS\n -------------\ndiff --git a/Documentation/technical/api-path-walk.adoc b/Documentation/technical/api-path-walk.adoc\nindex 3e089211fb4..e522695dd9f 100644\n--- a/Documentation/technical/api-path-walk.adoc\n+++ b/Documentation/technical/api-path-walk.adoc\n@@ -69,4 +69,5 @@ Examples\n \n See example usages in:\n \t`t/helper/test-path-walk.c`,\n+\t`builtin/pack-objects.c`,\n \t`builtin/backfill.c`\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7805429f5d1..4f934847558 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -41,6 +41,9 @@\n #include \"promisor-remote.h\"\n #include \"pack-mtimes.h\"\n #include \"parse-options.h\"\n+#include \"blob.h\"\n+#include \"tree.h\"\n+#include \"path-walk.h\"\n \n /*\n  * Objects we are going to pack are collected in the `to_pack` structure.\n@@ -217,6 +220,7 @@ static int delta_search_threads;\n static int pack_to_stdout;\n static int sparse;\n static int thin;\n+static int path_walk;\n static int num_preferred_base;\n static struct progress *progress_state;\n \n@@ -4191,6 +4195,105 @@ static void mark_bitmap_preferred_tips(void)\n \t}\n }\n \n+static inline int is_oid_interesting(struct repository *repo,\n+\t\t\t\t     struct object_id *oid)\n+{\n+\tstruct object *o = lookup_object(repo, oid);\n+\treturn o && !(o->flags & UNINTERESTING);\n+}\n+\n+static int add_objects_by_path(const char *path,\n+\t\t\t       struct oid_array *oids,\n+\t\t\t       enum object_type type,\n+\t\t\t       void *data)\n+{\n+\tstruct object_entry **delta_list;\n+\tsize_t oe_start = to_pack.nr_objects;\n+\tsize_t oe_end;\n+\tunsigned int sub_list_size;\n+\tunsigned int *processed = data;\n+\n+\t/*\n+\t * First, add all objects to the packing data, including the ones\n+\t * marked UNINTERESTING (translated to 'exclude') as they can be\n+\t * used as delta bases.\n+\t */\n+\tfor (size_t i = 0; i < oids->nr; i++) {\n+\t\tint exclude;\n+\t\tstruct object_info oi = OBJECT_INFO_INIT;\n+\t\tstruct object_id *oid = &oids->oid[i];\n+\n+\t\t/* Skip objects that do not exist locally. */\n+\t\tif (exclude_promisor_objects &&\n+\t\t    oid_object_info_extended(the_repository, oid, &oi,\n+\t\t\t\t\t     OBJECT_INFO_FOR_PREFETCH) < 0)\n+\t\t\tcontinue;\n+\n+\t\texclude = !is_oid_interesting(the_repository, oid);\n+\n+\t\tif (exclude && !thin)\n+\t\t\tcontinue;\n+\n+\t\tadd_object_entry(oid, type, path, exclude);\n+\t}\n+\n+\toe_end = to_pack.nr_objects;\n+\n+\t/* We can skip delta calculations if it is a no-op. */\n+\tif (oe_end == oe_start || !window)\n+\t\treturn 0;\n+\n+\tsub_list_size = 0;\n+\tALLOC_ARRAY(delta_list, oe_end - oe_start);\n+\n+\tfor (size_t i = 0; i < oe_end - oe_start; i++) {\n+\t\tstruct object_entry *entry = to_pack.objects + oe_start + i;\n+\n+\t\tif (!should_attempt_deltas(entry))\n+\t\t\tcontinue;\n+\n+\t\tdelta_list[sub_list_size++] = entry;\n+\t}\n+\n+\t/*\n+\t * Find delta bases among this list of objects that all match the same\n+\t * path. This causes the delta compression to be interleaved in the\n+\t * object walk, which can lead to confusing progress indicators. This is\n+\t * also incompatible with threaded delta calculations. In the future,\n+\t * consider creating a list of regions in the full to_pack.objects array\n+\t * that could be picked up by the threaded delta computation.\n+\t */\n+\tif (sub_list_size && window) {\n+\t\tQSORT(delta_list, sub_list_size, type_size_sort);\n+\t\tfind_deltas(delta_list, &sub_list_size, window, depth, processed);\n+\t}\n+\n+\tfree(delta_list);\n+\treturn 0;\n+}\n+\n+static void get_object_list_path_walk(struct rev_info *revs)\n+{\n+\tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n+\tunsigned int processed = 0;\n+\n+\tinfo.revs = revs;\n+\tinfo.path_fn = add_objects_by_path;\n+\tinfo.path_fn_data = &processed;\n+\trevs->tag_objects = 1;\n+\n+\t/*\n+\t * Allow the --[no-]sparse option to be interesting here, if only\n+\t * for testing purposes. Paths with no interesting objects will not\n+\t * contribute to the resulting pack, but only create noisy preferred\n+\t * base objects.\n+\t */\n+\tinfo.prune_all_uninteresting = sparse;\n+\n+\tif (walk_objects_by_path(&info))\n+\t\tdie(_(\"failed to pack objects via path-walk\"));\n+}\n+\n static void get_object_list(struct rev_info *revs, int ac, const char **av)\n {\n \tstruct setup_revision_opt s_r_opt = {\n@@ -4237,7 +4340,7 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n \n \twarn_on_object_refname_ambiguity = save_warning;\n \n-\tif (use_bitmap_index && !get_object_list_from_bitmap(revs))\n+\tif (use_bitmap_index && !path_walk && !get_object_list_from_bitmap(revs))\n \t\treturn;\n \n \tif (use_delta_islands)\n@@ -4246,15 +4349,19 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n \tif (write_bitmap_index)\n \t\tmark_bitmap_preferred_tips();\n \n-\tif (prepare_revision_walk(revs))\n-\t\tdie(_(\"revision walk setup failed\"));\n-\tmark_edges_uninteresting(revs, show_edge, sparse);\n-\n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n-\ttraverse_commit_list(revs,\n-\t\t\t     show_commit, fn_show_object,\n-\t\t\t     NULL);\n+\n+\tif (path_walk) {\n+\t\tget_object_list_path_walk(revs);\n+\t} else {\n+\t\tif (prepare_revision_walk(revs))\n+\t\t\tdie(_(\"revision walk setup failed\"));\n+\t\tmark_edges_uninteresting(revs, show_edge, sparse);\n+\t\ttraverse_commit_list(revs,\n+\t\t\t\tshow_commit, fn_show_object,\n+\t\t\t\tNULL);\n+\t}\n \n \tif (unpack_unreachable_expiration) {\n \t\trevs->ignore_missing_links = 1;\n@@ -4464,6 +4571,8 @@ int cmd_pack_objects(int argc,\n \t\t\t N_(\"use the sparse reachability algorithm\")),\n \t\tOPT_BOOL(0, \"thin\", &thin,\n \t\t\t N_(\"create thin packs\")),\n+\t\tOPT_BOOL(0, \"path-walk\", &path_walk,\n+\t\t\t N_(\"use the path-walk API to walk objects when possible\")),\n \t\tOPT_BOOL(0, \"shallow\", &shallow,\n \t\t\t N_(\"create packs suitable for shallow fetches\")),\n \t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep_on_disk,\n@@ -4549,7 +4658,27 @@ int cmd_pack_objects(int argc,\n \t\twindow = 0;\n \n \tstrvec_push(&rp, \"pack-objects\");\n-\tif (thin) {\n+\n+\tif (path_walk && filter_options.choice) {\n+\t\twarning(_(\"cannot use --filter with --path-walk\"));\n+\t\tpath_walk = 0;\n+\t}\n+\tif (path_walk && use_delta_islands) {\n+\t\twarning(_(\"cannot use delta islands with --path-walk\"));\n+\t\tpath_walk = 0;\n+\t}\n+\tif (path_walk && shallow) {\n+\t\twarning(_(\"cannot use --shallow with --path-walk\"));\n+\t\tpath_walk = 0;\n+\t}\n+\tif (path_walk) {\n+\t\tstrvec_push(&rp, \"--boundary\");\n+\t\t /*\n+\t\t  * We must disable the bitmaps because we are removing\n+\t\t  * the --objects / --objects-edge[-aggressive] options.\n+\t\t  */\n+\t\tuse_bitmap_index = 0;\n+\t} else if (thin) {\n \t\tuse_internal_rev_list = 1;\n \t\tstrvec_push(&rp, shallow\n \t\t\t\t? \"--objects-edge-aggressive\"\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 5ac8d39094b..16420d12863 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -723,4 +723,19 @@ test_expect_success '--name-hash-version=2 and --write-bitmap-index are incompat\n \t! test_grep \"currently, --write-bitmap-index requires --name-hash-version=1\" err\n '\n \n+test_expect_success '--path-walk pack everything' '\n+\tgit -C server rev-parse HEAD >in &&\n+\tgit -C server pack-objects --stdout --revs --path-walk <in >out.pack &&\n+\tgit -C server index-pack --stdin <out.pack\n+'\n+\n+test_expect_success '--path-walk thin pack' '\n+\tcat >in <<-EOF &&\n+\t$(git -C server rev-parse HEAD)\n+\t^$(git -C server rev-parse HEAD~2)\n+\tEOF\n+\tgit -C server pack-objects --thin --stdout --revs --path-walk <in >out.pack &&\n+\tgit -C server index-pack --fix-thin --stdin <out.pack\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"514941","messageId":"dcff01392ffba718cb3cce86e63b9e959190ccf7.1742829770.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v2.git.1742829769.gitgitgadget@gmail.com","subject":"[PATCH v2 03/13] pack-objects: update usage to match docs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-24T15:22:39Z","receivedAt":"2025-03-24T15:22:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThe t0450 test script verifies that builtin usage matches the synopsis\nin the documentation. Adjust the builtin to match and then remove 'git\npack-objects' from the exception list.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-pack-objects.adoc | 14 +++++++-------\n builtin/pack-objects.c              | 10 ++++++++--\n t/t0450/adoc-help-mismatches        |  1 -\n 3 files changed, 15 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc\nindex 7065758eddf..7c666a14277 100644\n--- a/Documentation/git-pack-objects.adoc\n+++ b/Documentation/git-pack-objects.adoc\n@@ -10,13 +10,13 @@ SYNOPSIS\n --------\n [verse]\n 'git pack-objects' [-q | --progress | --all-progress] [--all-progress-implied]\n-\t[--no-reuse-delta] [--delta-base-offset] [--non-empty]\n-\t[--local] [--incremental] [--window=<n>] [--depth=<n>]\n-\t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n-\t[--cruft] [--cruft-expiration=<time>]\n-\t[--stdout [--filter=<filter-spec>] | <base-name>]\n-\t[--shallow] [--keep-true-parents] [--[no-]sparse]\n-\t[--name-hash-version=<n>] [--path-walk] < <object-list>\n+\t\t   [--no-reuse-delta] [--delta-base-offset] [--non-empty]\n+\t\t   [--local] [--incremental] [--window=<n>] [--depth=<n>]\n+\t\t   [--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n+\t\t   [--cruft] [--cruft-expiration=<time>]\n+\t\t   [--stdout [--filter=<filter-spec>] | <base-name>]\n+\t\t   [--shallow] [--keep-true-parents] [--[no-]sparse]\n+\t\t   [--name-hash-version=<n>] [--path-walk] < <object-list>\n \n \n DESCRIPTION\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 4f934847558..75a6545cca1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -187,8 +187,14 @@ static inline void oe_set_delta_size(struct packing_data *pack,\n #define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n \n static const char *pack_usage[] = {\n-\tN_(\"git pack-objects --stdout [<options>] [< <ref-list> | < <object-list>]\"),\n-\tN_(\"git pack-objects [<options>] <base-name> [< <ref-list> | < <object-list>]\"),\n+\tN_(\"git pack-objects [-q | --progress | --all-progress] [--all-progress-implied]\\n\"\n+\t   \"                 [--no-reuse-delta] [--delta-base-offset] [--non-empty]\\n\"\n+\t   \"                 [--local] [--incremental] [--window=<n>] [--depth=<n>]\\n\"\n+\t   \"                 [--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\\n\"\n+\t   \"                 [--cruft] [--cruft-expiration=<time>]\\n\"\n+\t   \"                 [--stdout [--filter=<filter-spec>] | <base-name>]\\n\"\n+\t   \"                 [--shallow] [--keep-true-parents] [--[no-]sparse]\\n\"\n+\t   \"                 [--name-hash-version=<n>] [--path-walk] < <object-list>\"),\n \tNULL\n };\n \ndiff --git a/t/t0450/adoc-help-mismatches b/t/t0450/adoc-help-mismatches\nindex c4a15fd0cb8..06b469bdee2 100644\n--- a/t/t0450/adoc-help-mismatches\n+++ b/t/t0450/adoc-help-mismatches\n@@ -38,7 +38,6 @@ merge-one-file\n multi-pack-index\n name-rev\n notes\n-pack-objects\n push\n range-diff\n rebase\n-- \ngitgitgadget\n\n"},{"id":"514942","messageId":"97a0b52ccee8844c1c91232c625c2c6444873d85.1742829770.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v2.git.1742829769.gitgitgadget@gmail.com","subject":"[PATCH v2 04/13] p5313: add performance tests for --path-walk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-24T15:22:40Z","receivedAt":"2025-03-24T15:22:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThe previous change added a --path-walk option to 'git pack-objects'.\nCreate a performance test that demonstrates the time and space benefits\nof the feature.\n\nIn order to get an appropriate comparison, we need to avoid reusing\ndeltas and recompute them from scratch.\n\nCompare the creation of a thin pack representing a small push and the\ncreation of a relatively large non-thin pack.\n\nRunning on my copy of the Git repository results in this data (removing\nthe repack tests for --name-hash-version):\n\nTest                                                     this tree\n------------------------------------------------------------------------\n5313.2: thin pack with --name-hash-version=1             0.02(0.01+0.01)\n5313.3: thin pack size with --name-hash-version=1                   1.6K\n5313.4: big pack with --name-hash-version=1              2.55(4.20+0.26)\n5313.5: big pack size with --name-hash-version=1                   16.4M\n5313.6: shallow fetch pack with --name-hash-version=1    1.24(2.03+0.08)\n5313.7: shallow pack size with --name-hash-version=1               12.2M\n5313.10: thin pack with --name-hash-version=2            0.03(0.01+0.01)\n5313.11: thin pack size with --name-hash-version=2                  1.6K\n5313.12: big pack with --name-hash-version=2             1.91(3.23+0.20)\n5313.13: big pack size with --name-hash-version=2                  16.4M\n5313.14: shallow fetch pack with --name-hash-version=2   1.06(1.57+0.10)\n5313.15: shallow pack size with --name-hash-version=2              12.5M\n5313.18: thin pack with --path-walk                      0.03(0.01+0.01)\n5313.19: thin pack size with --path-walk                            1.6K\n5313.20: big pack with --path-walk                       2.05(3.24+0.27)\n5313.21: big pack size with --path-walk                            16.3M\n5313.22: shallow fetch pack with --path-walk             1.08(1.66+0.07)\n5313.23: shallow pack size with --path-walk                        12.4M\n\nThis can be reformatted as follows:\n\nPack Type            Hash v1   Hash v2     Path Walk\n---------------------------------------------------\nthin pack    (time)    0.02s      0.03s      0.03s\n             (size)    1.6K       1.6K       1.6K\nbig pack     (time)    2.55s      1.91s      2.05s\n             (size)   16.4M      16.4M      16.3M\nshallow pack (time)    1.24s      1.06s      1.08s\n             (size)   12.2M      12.5M      12.4M\n\nNote that the timing is slower because there is no threading in the\n--path-walk case (yet). Also, the shallow pack cases are really not\nusing the --path-walk logic right now because it is disabled until some\nadditions are made to the path walk API.\n\nThe cases where the --path-walk option really shines is when the default\nname-hash is overwhelmed with collisions. An open source example can be\nfound in the microsoft/fluentui repo [1] at a certain commit [2].\n\n[1] https://github.com/microsoft/fluentui\n[2] e70848ebac1cd720875bccaa3026f4a9ed700e08\n\nRunning the tests on this repo results in the following comparison table:\n\nPack Type            Hash v1    Hash v2    Path Walk\n---------------------------------------------------\nthin pack    (time)    0.36s      0.12s      0.08s\n             (size)    1.2M      22.0K      18.4K\nbig pack     (time)    2.00s      2.90s      2.21s\n             (size)   20.4M      25.9M      19.5M\nshallow pack (time)    1.41s      1.80s      1.65s\n             (size)   34.4M      33.7M      33.6M\n\nNotice in particular that in the small thin pack, the time performance\nhas improved from 0.36s for --name-hash-version=1 to 0.08s and this is\nlikely due to the improved size of the resulting pack: 18.4K instead of\n1.2M.  The relatively new --name-hash-version=2 is competitive with\n--path-walk (0.12s and 22.0K) but not quite as successful.\n\nFinally, running this on a copy of the Linux kernel repository results\nin these data points:\n\nPack Type            Hash v1    Hash v2    Path Walk\n---------------------------------------------------\nthin pack    (time)    0.03s      0.13s      0.03s\n             (size)    4.6K       4.6K       4.6K\nbig pack     (time)   15.29s     12.32s     13.92s\n             (size)  201.1M     159.1M     158.5M\nshallow pack (time)   10.88s     22.93s     22.74s\n             (size)  269.2M     273.8M     267.7M\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n t/perf/p5313-pack-objects.sh | 37 ++++++++++++++++++++++--------------\n 1 file changed, 23 insertions(+), 14 deletions(-)\n\ndiff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh\nindex be5229a0ecd..cd6dd3abb71 100755\n--- a/t/perf/p5313-pack-objects.sh\n+++ b/t/perf/p5313-pack-objects.sh\n@@ -25,46 +25,55 @@ test_expect_success 'create rev input' '\n \tEOF\n '\n \n-for version in 1 2\n-do\n-\texport version\n+test_all_with_args () {\n+\tparameter=$1\n+\texport parameter\n \n-\ttest_perf \"thin pack with version $version\" '\n+\ttest_perf \"thin pack with $parameter\" '\n \t\tgit pack-objects --thin --stdout --revs --sparse \\\n-\t\t\t--name-hash-version=$version <in-thin >out\n+\t\t\t$parameter <in-thin >out\n \t'\n \n-\ttest_size \"thin pack size with version $version\" '\n+\ttest_size \"thin pack size with $parameter\" '\n \t\ttest_file_size out\n \t'\n \n-\ttest_perf \"big pack with version $version\" '\n+\ttest_perf \"big pack with $parameter\" '\n \t\tgit pack-objects --stdout --revs --sparse \\\n-\t\t\t--name-hash-version=$version <in-big >out\n+\t\t\t$parameter <in-big >out\n \t'\n \n-\ttest_size \"big pack size with version $version\" '\n+\ttest_size \"big pack size with $parameter\" '\n \t\ttest_file_size out\n \t'\n \n-\ttest_perf \"shallow fetch pack with version $version\" '\n+\ttest_perf \"shallow fetch pack with $parameter\" '\n \t\tgit pack-objects --stdout --revs --sparse --shallow \\\n-\t\t\t--name-hash-version=$version <in-shallow >out\n+\t\t\t$parameter <in-shallow >out\n \t'\n \n-\ttest_size \"shallow pack size with version $version\" '\n+\ttest_size \"shallow pack size with $parameter\" '\n \t\ttest_file_size out\n \t'\n+}\n \n-\ttest_perf \"repack with version $version\" '\n+for version in 1 2\n+do\n+\texport version\n+\n+\ttest_all_with_args --name-hash-version=$version\n+\n+\ttest_perf \"repack with --name-hash-version=$version\" '\n \t\tgit repack -adf --name-hash-version=$version\n \t'\n \n-\ttest_size \"repack size with version $version\" '\n+\ttest_size \"repack size with --name-hash-version=$version\" '\n \t\tgitdir=$(git rev-parse --git-dir) &&\n \t\tpack=$(ls $gitdir/objects/pack/pack-*.pack) &&\n \t\ttest_file_size \"$pack\"\n \t'\n done\n \n+test_all_with_args --path-walk\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"514943","messageId":"ddf804e606a5560639e4018709da33cd80c74012.1742829770.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v2.git.1742829769.gitgitgadget@gmail.com","subject":"[PATCH v2 06/13] t5538: add tests to confirm deltas in shallow pushes","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-24T15:22:42Z","receivedAt":"2025-03-24T15:22:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nIt can be notoriously difficult to detect if delta bases are being\ncomputed properly during 'git push'. Construct an example where it will\nmake a kilobyte worth of difference when a delta base is not found. We\ncan then use the progress indicators to distinguish between bytes and\nKiB depending on whether the delta base is found and used.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n t/t5538-push-shallow.sh | 34 ++++++++++++++++++++++++++++++++++\n 1 file changed, 34 insertions(+)\n\ndiff --git a/t/t5538-push-shallow.sh b/t/t5538-push-shallow.sh\nindex e91fcc173e8..11b85cca9e8 100755\n--- a/t/t5538-push-shallow.sh\n+++ b/t/t5538-push-shallow.sh\n@@ -123,4 +123,38 @@ EOF\n \tgit cat-file blob $(echo 1|git hash-object --stdin) >/dev/null\n \t)\n '\n+\n+test_expect_success 'push new commit from shallow clone has correct object count' '\n+\tgit init origin &&\n+\ttest_commit -C origin a &&\n+\ttest_commit -C origin b &&\n+\n+\tgit clone --depth=1 \"file://$(pwd)/origin\" client &&\n+\tgit -C client checkout -b topic &&\n+\tgit -C client commit --allow-empty -m \"empty\" &&\n+\tGIT_PROGRESS_DELAY=0 git -C client push --progress origin topic 2>err &&\n+\ttest_grep \"Enumerating objects: 1, done.\" err\n+'\n+\n+test_expect_success 'push new commit from shallow clone has good deltas' '\n+\tgit init base &&\n+\ttest_seq 1 999 >base/a &&\n+\ttest_commit -C base initial &&\n+\tgit -C base add a &&\n+\tgit -C base commit -m \"big a\" &&\n+\n+\tgit clone --depth=1 \"file://$(pwd)/base\" deltas &&\n+\tgit -C deltas checkout -b deltas &&\n+\ttest_seq 1 1000 >deltas/a &&\n+\tgit -C deltas commit -a -m \"bigger a\" &&\n+\tGIT_TRACE2_PERF=\"$(pwd)/trace.txt\" \\\n+\tGIT_PROGRESS_DELAY=0 git -C deltas push --progress origin deltas 2>err &&\n+\n+\ttest_grep \"Enumerating objects: 5, done\" err &&\n+\n+\t# If the delta base is found, then this message uses \"bytes\".\n+\t# If the delta base is not found, then this message uses \"KiB\".\n+\ttest_grep \"Writing objects: .* bytes\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"514944","messageId":"0d49bb3d30add66676280ec7fabed12351d5b3ac.1742829770.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v2.git.1742829769.gitgitgadget@gmail.com","subject":"[PATCH v2 05/13] pack-objects: introduce GIT_TEST_PACK_PATH_WALK","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-24T15:22:41Z","receivedAt":"2025-03-24T15:22:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThere are many tests that validate whether 'git pack-objects' works as\nexpected. Instead of duplicating these tests, add a new test environment\nvariable, GIT_TEST_PACK_PATH_WALK, that implies --path-walk by default\nwhen specified.\n\nThis was useful in testing the implementation of the --path-walk\nimplementation, especially in conjunction with test such as:\n\n - t0411-clone-from-partial.sh : One test fetches from a repo that does\n   not have the boundary objects. This causes the path-based walk to\n   fail. Disable the variable for this test.\n\n - t5306-pack-nobase.sh : Similar to t0411, one test fetches from a repo\n   without a boundary object.\n\n - t5310-pack-bitmaps.sh : One test compares the case when packing with\n   bitmaps to the case when packing without them. Since we disable the\n   test variable when writing bitmaps, this causes a difference in the\n   object list (the --path-walk option adds an extra object). Specify\n   --no-path-walk in both processes for the comparison. Another test\n   checks for a specific delta base, but when computing dynamically\n   without using bitmaps, the base object it too small to be considered\n   in the delta calculations so no base is used.\n\n - t5316-pack-delta-depth.sh : This script cares about certain delta\n   choices and their chain lengths. The --path-walk option changes how\n   these chains are selected, and thus changes the results of this test.\n\n - t5322-pack-objects-sparse.sh : This demonstrates the effectiveness of\n   the --sparse option and how it combines with --path-walk.\n\n - t5332-multi-pack-reuse.sh : This test verifies that the preferred\n   pack is used for delta reuse when possible. The --path-walk option is\n   not currently aware of the preferred pack at all, so finds a\n   different delta base.\n\n - t7406-submodule-update.sh : When using the variable, the --depth\n   option collides with the --path-walk feature, resulting in a warning\n   message. Disable the variable so this warning does not appear.\n\nI want to call out one specific test change that is only temporary:\n\n - t5530-upload-pack-error.sh : One test cares specifically about an\n   \"unable to read\" error message. Since the current implementation\n   performs delta calculations within the path-walk API callback, a\n   different \"unable to get size\" error message appears. When this\n   is changed in a future refactoring, this test change can be reverted.\n\nSimilar to GIT_TEST_NAME_HASH_VERSION, we do not add this option to the\nlinux-TEST-vars CI build as that's already an overloaded build.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c        | 12 ++++++++++--\n t/README                      |  4 ++++\n t/t0411-clone-from-partial.sh |  6 ++++++\n t/t5306-pack-nobase.sh        |  5 +++++\n t/t5310-pack-bitmaps.sh       | 13 +++++++++++--\n t/t5316-pack-delta-depth.sh   |  9 ++++++---\n t/t5332-multi-pack-reuse.sh   |  7 +++++++\n t/t5530-upload-pack-error.sh  |  6 ++++++\n t/t7406-submodule-update.sh   |  3 +++\n 9 files changed, 58 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 75a6545cca1..a6b8a78d42a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -226,7 +226,7 @@ static int delta_search_threads;\n static int pack_to_stdout;\n static int sparse;\n static int thin;\n-static int path_walk;\n+static int path_walk = -1;\n static int num_preferred_base;\n static struct progress *progress_state;\n \n@@ -4230,7 +4230,7 @@ static int add_objects_by_path(const char *path,\n \t\tstruct object_id *oid = &oids->oid[i];\n \n \t\t/* Skip objects that do not exist locally. */\n-\t\tif (exclude_promisor_objects &&\n+\t\tif ((exclude_promisor_objects || arg_missing_action != MA_ERROR) &&\n \t\t    oid_object_info_extended(the_repository, oid, &oi,\n \t\t\t\t\t     OBJECT_INFO_FOR_PREFETCH) < 0)\n \t\t\tcontinue;\n@@ -4648,6 +4648,14 @@ int cmd_pack_objects(int argc,\n \tif (pack_to_stdout != !base_name || argc)\n \t\tusage_with_options(pack_usage, pack_objects_options);\n \n+\tif (path_walk < 0) {\n+\t\tif (use_bitmap_index > 0 ||\n+\t\t    !use_internal_rev_list)\n+\t\t\tpath_walk = 0;\n+\t\telse\n+\t\t\tpath_walk = git_env_bool(\"GIT_TEST_PACK_PATH_WALK\", 0);\n+\t}\n+\n \tif (depth < 0)\n \t\tdepth = 0;\n \tif (depth >= (1 << OE_DEPTH_BITS)) {\ndiff --git a/t/README b/t/README\nindex 53e5b4a7107..ae06e628815 100644\n--- a/t/README\n+++ b/t/README\n@@ -415,6 +415,10 @@ GIT_TEST_PACK_SPARSE=<boolean> if disabled will default the pack-objects\n builtin to use the non-sparse object walk. This can still be overridden by\n the --sparse command-line argument.\n \n+GIT_TEST_PACK_PATH_WALK=<boolean> if enabled will default the pack-objects\n+builtin to use the path-walk API for the object walk. This can still be\n+overridden by the --no-path-walk command-line argument.\n+\n GIT_TEST_PRELOAD_INDEX=<boolean> exercises the preload-index code path\n by overriding the minimum number of cache entries required per thread.\n \ndiff --git a/t/t0411-clone-from-partial.sh b/t/t0411-clone-from-partial.sh\nindex 196fc617843..9e6bca56255 100755\n--- a/t/t0411-clone-from-partial.sh\n+++ b/t/t0411-clone-from-partial.sh\n@@ -59,6 +59,12 @@ test_expect_success 'pack-objects should fetch from promisor remote and execute\n \n test_expect_success 'clone from promisor remote does not lazy-fetch by default' '\n \trm -f script-executed &&\n+\n+\t# The --path-walk feature of \"git pack-objects\" is not\n+\t# compatible with this kind of fetch from an incomplete repo.\n+\tGIT_TEST_PACK_PATH_WALK=0 &&\n+\texport GIT_TEST_PACK_PATH_WALK &&\n+\n \ttest_must_fail git clone evil no-lazy 2>err &&\n \ttest_grep \"lazy fetching disabled\" err &&\n \ttest_path_is_missing script-executed\ndiff --git a/t/t5306-pack-nobase.sh b/t/t5306-pack-nobase.sh\nindex 805d60ff317..609399d54fb 100755\n--- a/t/t5306-pack-nobase.sh\n+++ b/t/t5306-pack-nobase.sh\n@@ -59,6 +59,11 @@ test_expect_success 'indirectly clone patch_clone' '\n \t git pull ../.git &&\n \t test $(git rev-parse HEAD) = $B &&\n \n+\t# The --path-walk feature of \"git pack-objects\" is not\n+\t# compatible with this kind of fetch from an incomplete repo.\n+\tGIT_TEST_PACK_PATH_WALK=0 &&\n+\texport GIT_TEST_PACK_PATH_WALK &&\n+\n \t git pull ../patch_clone/.git &&\n \t test $(git rev-parse HEAD) = $C\n \t)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 621bbbdd26e..e01df807a62 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -158,8 +158,9 @@ test_bitmap_cases () {\n \t\tls .git/objects/pack/ | grep bitmap >output &&\n \t\ttest_line_count = 1 output &&\n \t\t# verify equivalent packs are generated with/without using bitmap index\n-\t\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa </dev/null) &&\n-\t\tpackbsha1=$(git pack-objects --use-bitmap-index --all packb </dev/null) &&\n+\t\t# Be careful to not use the path-walk option in either case.\n+\t\tpackasha1=$(git pack-objects --no-use-bitmap-index --no-path-walk --all packa </dev/null) &&\n+\t\tpackbsha1=$(git pack-objects --use-bitmap-index --no-path-walk --all packb </dev/null) &&\n \t\tlist_packed_objects packa-$packasha1.idx >packa.objects &&\n \t\tlist_packed_objects packb-$packbsha1.idx >packb.objects &&\n \t\ttest_cmp packa.objects packb.objects\n@@ -388,6 +389,14 @@ test_bitmap_cases () {\n \t\tgit init --bare client.git &&\n \t\t(\n \t\t\tcd client.git &&\n+\n+\t\t\t# This test relies on reusing a delta, but if the\n+\t\t\t# path-walk machinery is engaged, the base object\n+\t\t\t# is considered too small to use during the\n+\t\t\t# dynamic computation, so is not used.\n+\t\t\tGIT_TEST_PACK_PATH_WALK=0 &&\n+\t\t\texport GIT_TEST_PACK_PATH_WALK &&\n+\n \t\t\tgit config transfer.unpackLimit 1 &&\n \t\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n \t\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\ndiff --git a/t/t5316-pack-delta-depth.sh b/t/t5316-pack-delta-depth.sh\nindex 32cf4227451..167c3a35234 100755\n--- a/t/t5316-pack-delta-depth.sh\n+++ b/t/t5316-pack-delta-depth.sh\n@@ -89,15 +89,18 @@ max_chain() {\n # adjusted (or scrapped if the heuristics have become too unreliable)\n test_expect_success 'packing produces a long delta' '\n \t# Use --window=0 to make sure we are seeing reused deltas,\n-\t# not computing a new long chain.\n-\tpack=$(git pack-objects --all --window=0 </dev/null pack) &&\n+\t# not computing a new long chain. (Also avoid the --path-walk\n+\t# option as it may break delta chains.)\n+\tpack=$(git pack-objects --all --window=0 --no-path-walk </dev/null pack) &&\n \techo 9 >expect &&\n \tmax_chain pack-$pack.pack >actual &&\n \ttest_cmp expect actual\n '\n \n test_expect_success '--depth limits depth' '\n-\tpack=$(git pack-objects --all --depth=5 </dev/null pack) &&\n+\t# Avoid --path-walk to avoid breaking delta chains across path\n+\t# boundaries.\n+\tpack=$(git pack-objects --all --depth=5 --no-path-walk </dev/null pack) &&\n \techo 5 >expect &&\n \tmax_chain pack-$pack.pack >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t5332-multi-pack-reuse.sh b/t/t5332-multi-pack-reuse.sh\nindex 57cad7708f8..395d09444ce 100755\n--- a/t/t5332-multi-pack-reuse.sh\n+++ b/t/t5332-multi-pack-reuse.sh\n@@ -7,6 +7,13 @@ test_description='pack-objects multi-pack reuse'\n \n GIT_TEST_MULTI_PACK_INDEX=0\n GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n+\n+# The --path-walk option does not consider the preferred pack\n+# at all for reusing deltas, so this variable changes the\n+# behavior of this test, if enabled.\n+GIT_TEST_PACK_PATH_WALK=0\n+export GIT_TEST_PACK_PATH_WALK\n+\n objdir=.git/objects\n packdir=$objdir/pack\n \ndiff --git a/t/t5530-upload-pack-error.sh b/t/t5530-upload-pack-error.sh\nindex 558eedf25a4..8eb6fea839a 100755\n--- a/t/t5530-upload-pack-error.sh\n+++ b/t/t5530-upload-pack-error.sh\n@@ -34,6 +34,12 @@ test_expect_success 'upload-pack fails due to error in pack-objects packing' '\n \thexsz=$(test_oid hexsz) &&\n \tprintf \"%04xwant %s\\n00000009done\\n0000\" \\\n \t\t$(($hexsz + 10)) $head >input &&\n+\n+\t# The current implementation of path-walk causes a different\n+\t# error message. This will be changed by a future refactoring.\n+\tGIT_TEST_PACK_PATH_WALK=0 &&\n+\texport GIT_TEST_PACK_PATH_WALK &&\n+\n \ttest_must_fail git upload-pack . <input >/dev/null 2>output.err &&\n \ttest_grep \"unable to read\" output.err &&\n \ttest_grep \"pack-objects died\" output.err\ndiff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh\nindex c562bad042a..ab76d4b6dc4 100755\n--- a/t/t7406-submodule-update.sh\n+++ b/t/t7406-submodule-update.sh\n@@ -1095,12 +1095,15 @@ test_expect_success 'submodule update --quiet passes quietness to fetch with a s\n \t(cd super5 &&\n \t # This test var can mess with the stderr output checked in this test.\n \t GIT_TEST_NAME_HASH_VERSION=1 \\\n+\t GIT_TEST_PACK_PATH_WALK=0 \\\n \t\tgit submodule update --quiet --init --depth=1 submodule3 >out 2>err &&\n \t test_must_be_empty out &&\n \t test_must_be_empty err\n \t) &&\n \tgit clone super4 super6 &&\n \t(cd super6 &&\n+\t # This test variable will create a \"warning\" message to stderr\n+\t GIT_TEST_PACK_PATH_WALK=0 \\\n \t git submodule update --init --depth=1 submodule3 >out 2>err &&\n \t test_file_not_empty out &&\n \t test_file_not_empty err\n-- \ngitgitgadget\n\n"},{"id":"514945","messageId":"11767e7653e5212622ed7f82cad3b1a118621e8c.1742829770.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v2.git.1742829769.gitgitgadget@gmail.com","subject":"[PATCH v2 07/13] repack: add --path-walk option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-24T15:22:43Z","receivedAt":"2025-03-24T15:23:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nSince 'git pack-objects' supports a --path-walk option, allow passing it\nthrough in 'git repack'. This presents interesting testing opportunities for\ncomparing the different repacking strategies against each other.\n\nAdd the --path-walk option to the performance tests in p5313.\n\nFor the microsoft/fluentui repo [1] checked out at a specific commit [2],\nthe --path-walk tests in p5313 look like this:\n\nTest                                                     this tree\n-------------------------------------------------------------------------\n5313.18: thin pack with --path-walk                      0.08(0.06+0.02)\n5313.19: thin pack size with --path-walk                           18.4K\n5313.20: big pack with --path-walk                       2.10(7.80+0.26)\n5313.21: big pack size with --path-walk                            19.8M\n5313.22: shallow fetch pack with --path-walk             1.62(3.38+0.17)\n5313.23: shallow pack size with --path-walk                        33.6M\n5313.24: repack with --path-walk                         81.29(96.08+0.71)\n5313.25: repack size with --path-walk                             142.5M\n\n[1] https://github.com/microsoft/fluentui\n[2] e70848ebac1cd720875bccaa3026f4a9ed700e08\n\nAlong with the earlier tests in p5313, I'll instead reformat the\ncomparison as follows:\n\nRepack Method    Pack Size       Time\n---------------------------------------\nHash v1             439.4M      87.24s\nHash v2             161.7M      21.51s\nPath Walk           142.5M      81.29s\n\nThere are a few things to notice here:\n\n 1. The benefits of --name-hash-version=2 over --name-hash-version=1 are\n    significant, but --path-walk still compresses better than that\n    option.\n\n 2. The --path-walk command is still using --name-hash-version=1 for the\n    second pass of delta computation, using the increased name hash\n    collisions as a potential method for opportunistic compression on\n    top of the path-focused compression.\n\n 3. The --path-walk algorithm is currently sequential and does not use\n    multiple threads for delta compression. Threading will be\n    implemented in a future change so the computation time will improve\n    to better compete in this metric.\n\nThere are small benefits in size for my copy of the Git repository:\n\nRepack Method    Pack Size       Time\n---------------------------------------\nHash v1             248.8M      30.44s\nHash v2             249.0M      30.15s\nPath Walk           213.2M     142.50s\n\nAs well as in the nodejs/node repository [3]:\n\nRepack Method    Pack Size       Time\n---------------------------------------\nHash v1             739.9M      71.18s\nHash v2             764.6M      67.82s\nPath Walk           698.1M     208.10s\n\n[3] https://github.com/nodejs/node\n\nThis benefit also repeats in my copy of the Linux kernel repository:\n\nRepack Method    Pack Size       Time\n---------------------------------------\nHash v1               2.5G     554.41s\nHash v2               2.5G     549.62s\nPath Walk             2.2G    1562.36s\n\nIt is important to see that even when the repository shape does not have\nmany name-hash collisions, there is a slight space boost to be found\nusing this method.\n\nAs this repacking strategy was released in Git for Windows 2.47.0, some\nusers have reported cases where the --path-walk compression is slightly\nworse than the --name-hash-version=2 option. In those cases, it may be\nbeneficial to combine the two options. However, there has not been a\nreleased version of Git that has both options and I don't have access to\nthese repos for testing.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-repack.adoc | 14 +++++++++++++-\n builtin/repack.c              |  7 ++++++-\n t/perf/p5313-pack-objects.sh  | 18 ++++++++----------\n 3 files changed, 27 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/git-repack.adoc b/Documentation/git-repack.adoc\nindex 5852a5c9736..2199eb3b94f 100644\n--- a/Documentation/git-repack.adoc\n+++ b/Documentation/git-repack.adoc\n@@ -11,7 +11,7 @@ SYNOPSIS\n [verse]\n 'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n \t[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n-\t[--write-midx] [--name-hash-version=<n>]\n+\t[--write-midx] [--name-hash-version=<n>] [--path-walk]\n \n DESCRIPTION\n -----------\n@@ -255,6 +255,18 @@ linkgit:git-multi-pack-index[1]).\n \tProvide this argument to the underlying `git pack-objects` process.\n \tSee linkgit:git-pack-objects[1] for full details.\n \n+--path-walk::\n+\tThis option passes the `--path-walk` option to the underlying\n+\t`git pack-options` process (see linkgit:git-pack-objects[1]).\n+\tBy default, `git pack-objects` walks objects in an order that\n+\tpresents trees and blobs in an order unrelated to the path they\n+\tappear relative to a commit's root tree. The `--path-walk` option\n+\tenables a different walking algorithm that organizes trees and\n+\tblobs by path. This has the potential to improve delta compression\n+\tespecially in the presence of filenames that cause collisions in\n+\tGit's default name-hash algorithm. Due to changing how the objects\n+\tare walked, this option is not compatible with `--delta-islands`\n+\tor `--filter`.\n \n CONFIGURATION\n -------------\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 75e3752353a..d7f798280c0 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -43,7 +43,7 @@ static char *packdir, *packtmp_name, *packtmp;\n static const char *const git_repack_usage[] = {\n \tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n \t   \"[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\\n\"\n-\t   \"[--write-midx] [--name-hash-version=<n>]\"),\n+\t   \"[--write-midx] [--name-hash-version=<n>] [--path-walk]\"),\n \tNULL\n };\n \n@@ -63,6 +63,7 @@ struct pack_objects_args {\n \tint quiet;\n \tint local;\n \tint name_hash_version;\n+\tint path_walk;\n \tstruct list_objects_filter_options filter_options;\n };\n \n@@ -313,6 +314,8 @@ static void prepare_pack_objects(struct child_process *cmd,\n \t\tstrvec_pushf(&cmd->args, \"--no-reuse-object\");\n \tif (args->name_hash_version)\n \t\tstrvec_pushf(&cmd->args, \"--name-hash-version=%d\", args->name_hash_version);\n+\tif (args->path_walk)\n+\t\tstrvec_pushf(&cmd->args, \"--path-walk\");\n \tif (args->local)\n \t\tstrvec_push(&cmd->args,  \"--local\");\n \tif (args->quiet)\n@@ -1212,6 +1215,8 @@ int cmd_repack(int argc,\n \t\t\t\tN_(\"pass --no-reuse-object to git-pack-objects\")),\n \t\tOPT_INTEGER(0, \"name-hash-version\", &po_args.name_hash_version,\n \t\t\t\tN_(\"specify the name hash version to use for grouping similar objects by path\")),\n+\t\tOPT_BOOL(0, \"path-walk\", &po_args.path_walk,\n+\t\t\t\tN_(\"pass --path-walk to git-pack-objects\")),\n \t\tOPT_NEGBIT('n', NULL, &run_update_server_info,\n \t\t\t\tN_(\"do not run git-update-server-info\"), 1),\n \t\tOPT__QUIET(&po_args.quiet, N_(\"be quiet\")),\ndiff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh\nindex cd6dd3abb71..98748b0e203 100755\n--- a/t/perf/p5313-pack-objects.sh\n+++ b/t/perf/p5313-pack-objects.sh\n@@ -55,23 +55,21 @@ test_all_with_args () {\n \ttest_size \"shallow pack size with $parameter\" '\n \t\ttest_file_size out\n \t'\n-}\n-\n-for version in 1 2\n-do\n-\texport version\n-\n-\ttest_all_with_args --name-hash-version=$version\n \n-\ttest_perf \"repack with --name-hash-version=$version\" '\n-\t\tgit repack -adf --name-hash-version=$version\n+\ttest_perf \"repack with $parameter\" '\n+\t\tgit repack -adf $parameter\n \t'\n \n-\ttest_size \"repack size with --name-hash-version=$version\" '\n+\ttest_size \"repack size with $parameter\" '\n \t\tgitdir=$(git rev-parse --git-dir) &&\n \t\tpack=$(ls $gitdir/objects/pack/pack-*.pack) &&\n \t\ttest_file_size \"$pack\"\n \t'\n+}\n+\n+for version in 1 2\n+do\n+\ttest_all_with_args --name-hash-version=$version\n done\n \n test_all_with_args --path-walk\n-- \ngitgitgadget\n\n"},{"id":"514946","messageId":"dd66a5b46f2ef52eedea323df891a2e24fcd48c1.1742829770.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v2.git.1742829769.gitgitgadget@gmail.com","subject":"[PATCH v2 08/13] pack-objects: enable --path-walk via config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-24T15:22:44Z","receivedAt":"2025-03-24T15:23:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nUsers may want to enable the --path-walk option for 'git pack-objects' by\ndefault, especially underneath commands like 'git push' or 'git repack'.\n\nThis should be limited to client repositories, since the --path-walk option\ndisables bitmap walks, so would be bad to include in Git servers when\nserving fetches and clones. There is potential that it may be helpful to\nconsider when repacking the repository, to take advantage of improved deltas\nacross historical versions of the same files.\n\nMuch like how \"pack.useSparse\" was introduced and included in\n\"feature.experimental\" before being enabled by default, use the repository\nsettings infrastructure to make the new \"pack.usePathWalk\" config enabled by\n\"feature.experimental\" and \"feature.manyFiles\".\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/config/feature.adoc | 4 ++++\n Documentation/config/pack.adoc    | 8 ++++++++\n builtin/pack-objects.c            | 3 +++\n repo-settings.c                   | 3 +++\n repo-settings.h                   | 1 +\n 5 files changed, 19 insertions(+)\n\ndiff --git a/Documentation/config/feature.adoc b/Documentation/config/feature.adoc\nindex f061b64b748..cb49ff2604a 100644\n--- a/Documentation/config/feature.adoc\n+++ b/Documentation/config/feature.adoc\n@@ -20,6 +20,10 @@ walking fewer objects.\n +\n * `pack.allowPackReuse=multi` may improve the time it takes to create a pack by\n reusing objects from multiple packs instead of just one.\n++\n+* `pack.usePathWalk` may speed up packfile creation and make the packfiles be\n+significantly smaller in the presence of certain filename collisions with Git's\n+default name-hash.\n \n feature.manyFiles::\n \tEnable config options that optimize for repos with many files in the\ndiff --git a/Documentation/config/pack.adoc b/Documentation/config/pack.adoc\nindex da527377faf..08d06271177 100644\n--- a/Documentation/config/pack.adoc\n+++ b/Documentation/config/pack.adoc\n@@ -155,6 +155,14 @@ pack.useSparse::\n \tcommits contain certain types of direct renames. Default is\n \t`true`.\n \n+pack.usePathWalk::\n+\tWhen true, git will default to using the '--path-walk' option in\n+\t'git pack-objects' when the '--revs' option is present. This\n+\talgorithm groups objects by path to maximize the ability to\n+\tcompute delta chains across historical versions of the same\n+\tobject. This may disable other options, such as using bitmaps to\n+\tenumerate objects.\n+\n pack.preferBitmapTips::\n \tWhen selecting which commits will receive bitmaps, prefer a\n \tcommit at the tip of any reference that is a suffix of any value\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a6b8a78d42a..0ea85754c52 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4652,6 +4652,9 @@ int cmd_pack_objects(int argc,\n \t\tif (use_bitmap_index > 0 ||\n \t\t    !use_internal_rev_list)\n \t\t\tpath_walk = 0;\n+\t\telse if (the_repository->gitdir &&\n+\t\t\t the_repository->settings.pack_use_path_walk)\n+\t\t\tpath_walk = 1;\n \t\telse\n \t\t\tpath_walk = git_env_bool(\"GIT_TEST_PACK_PATH_WALK\", 0);\n \t}\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 67e9cfd2e63..9b5595c708e 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -47,11 +47,13 @@ void prepare_repo_settings(struct repository *r)\n \t\tr->settings.fetch_negotiation_algorithm = FETCH_NEGOTIATION_SKIPPING;\n \t\tr->settings.pack_use_bitmap_boundary_traversal = 1;\n \t\tr->settings.pack_use_multi_pack_reuse = 1;\n+\t\tr->settings.pack_use_path_walk = 1;\n \t}\n \tif (manyfiles) {\n \t\tr->settings.index_version = 4;\n \t\tr->settings.index_skip_hash = 1;\n \t\tr->settings.core_untracked_cache = UNTRACKED_CACHE_WRITE;\n+\t\tr->settings.pack_use_path_walk = 1;\n \t}\n \n \t/* Commit graph config or default, does not cascade (simple) */\n@@ -66,6 +68,7 @@ void prepare_repo_settings(struct repository *r)\n \n \t/* Boolean config or default, does not cascade (simple)  */\n \trepo_cfg_bool(r, \"pack.usesparse\", &r->settings.pack_use_sparse, 1);\n+\trepo_cfg_bool(r, \"pack.usepathwalk\", &r->settings.pack_use_path_walk, 0);\n \trepo_cfg_bool(r, \"core.multipackindex\", &r->settings.core_multi_pack_index, 1);\n \trepo_cfg_bool(r, \"index.sparse\", &r->settings.sparse_index, 0);\n \trepo_cfg_bool(r, \"index.skiphash\", &r->settings.index_skip_hash, r->settings.index_skip_hash);\ndiff --git a/repo-settings.h b/repo-settings.h\nindex ddc11967e01..a31decad221 100644\n--- a/repo-settings.h\n+++ b/repo-settings.h\n@@ -56,6 +56,7 @@ struct repo_settings {\n \tenum untracked_cache_setting core_untracked_cache;\n \n \tint pack_use_sparse;\n+\tint pack_use_path_walk;\n \tenum fetch_negotiation_setting fetch_negotiation_algorithm;\n \n \tint core_multi_pack_index;\n-- \ngitgitgadget\n\n"},{"id":"514947","messageId":"e5624c379d5804bf4fc44163c14a85ef55f89c97.1742829770.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v2.git.1742829769.gitgitgadget@gmail.com","subject":"[PATCH v2 09/13] scalar: enable path-walk during push via config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-24T15:22:45Z","receivedAt":"2025-03-24T15:23:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nRepositories registered with Scalar are expected to be client-only\nrepositories that are rather large. This means that they are more likely to\nbe good candidates for using the --path-walk option when running 'git\npack-objects', especially under the hood of 'git push'. Enable this config\nin Scalar repositories.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n scalar.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/scalar.c b/scalar.c\nindex da42b4be0cc..bf638fa34b8 100644\n--- a/scalar.c\n+++ b/scalar.c\n@@ -170,6 +170,7 @@ static int set_recommended_config(int reconfigure)\n \t\t{ \"core.autoCRLF\", \"false\" },\n \t\t{ \"core.safeCRLF\", \"false\" },\n \t\t{ \"fetch.showForcedUpdates\", \"false\" },\n+\t\t{ \"pack.usePathWalk\", \"true\" },\n \t\t{ NULL, NULL },\n \t};\n \tint i;\n-- \ngitgitgadget\n\n"},{"id":"514948","messageId":"622439d78557d94da899d21444920c27768d3c67.1742829770.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v2.git.1742829769.gitgitgadget@gmail.com","subject":"[PATCH v2 10/13] pack-objects: refactor path-walk delta phase","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-24T15:22:46Z","receivedAt":"2025-03-24T15:23:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nPreviously, the --path-walk option to 'git pack-objects' would compute\ndeltas inline with the path-walk logic. This would make the progress\nindicator look like it is taking a long time to enumerate objects, and\nthen very quickly computed deltas.\n\nInstead of computing deltas on each region of objects organized by tree,\nstore a list of regions corresponding to these groups. These can later\nbe pulled from the list for delta compression before doing the \"global\"\ndelta search.\n\nThis presents a new progress indicator that can be used in tests to\nverify that this stage is happening.\n\nThe current implementation is not integrated with threads, but could be\ndone in a future update.\n\nSince we do not attempt to sort objects by size until after exploring\nall trees, we can remove the previous change to t5530 due to a different\nerror message appearing first.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c       | 82 +++++++++++++++++++++++++-----------\n pack-objects.h               | 12 ++++++\n t/t5300-pack-object.sh       |  8 +++-\n t/t5530-upload-pack-error.sh |  6 ---\n 4 files changed, 75 insertions(+), 33 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 0ea85754c52..d4e05ca4434 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3236,6 +3236,51 @@ static int should_attempt_deltas(struct object_entry *entry)\n \treturn 1;\n }\n \n+static void find_deltas_for_region(struct object_entry *list,\n+\t\t\t\t   struct packing_region *region,\n+\t\t\t\t   unsigned int *processed)\n+{\n+\tstruct object_entry **delta_list;\n+\tuint32_t delta_list_nr = 0;\n+\n+\tALLOC_ARRAY(delta_list, region->nr);\n+\tfor (uint32_t i = 0; i < region->nr; i++) {\n+\t\tstruct object_entry *entry = list + region->start + i;\n+\t\tif (should_attempt_deltas(entry))\n+\t\t\tdelta_list[delta_list_nr++] = entry;\n+\t}\n+\n+\tQSORT(delta_list, delta_list_nr, type_size_sort);\n+\tfind_deltas(delta_list, &delta_list_nr, window, depth, processed);\n+\tfree(delta_list);\n+}\n+\n+static void find_deltas_by_region(struct object_entry *list,\n+\t\t\t\t  struct packing_region *regions,\n+\t\t\t\t  uint32_t start, uint32_t nr)\n+{\n+\tunsigned int processed = 0;\n+\tuint32_t progress_nr;\n+\n+\tif (!nr)\n+\t\treturn;\n+\n+\tprogress_nr = regions[nr - 1].start + regions[nr - 1].nr;\n+\n+\tif (progress)\n+\t\tprogress_state = start_progress(the_repository,\n+\t\t\t\t\t\t_(\"Compressing objects by path\"),\n+\t\t\t\t\t\tprogress_nr);\n+\n+\twhile (nr--)\n+\t\tfind_deltas_for_region(list,\n+\t\t\t\t       &regions[start++],\n+\t\t\t\t       &processed);\n+\n+\tdisplay_progress(progress_state, progress_nr);\n+\tstop_progress(&progress_state);\n+}\n+\n static void prepare_pack(int window, int depth)\n {\n \tstruct object_entry **delta_list;\n@@ -3260,6 +3305,10 @@ static void prepare_pack(int window, int depth)\n \tif (!to_pack.nr_objects || !window || !depth)\n \t\treturn;\n \n+\tif (path_walk)\n+\t\tfind_deltas_by_region(to_pack.objects, to_pack.regions,\n+\t\t\t\t      0, to_pack.nr_regions);\n+\n \tALLOC_ARRAY(delta_list, to_pack.nr_objects);\n \tnr_deltas = n = 0;\n \n@@ -4213,10 +4262,8 @@ static int add_objects_by_path(const char *path,\n \t\t\t       enum object_type type,\n \t\t\t       void *data)\n {\n-\tstruct object_entry **delta_list;\n \tsize_t oe_start = to_pack.nr_objects;\n \tsize_t oe_end;\n-\tunsigned int sub_list_size;\n \tunsigned int *processed = data;\n \n \t/*\n@@ -4249,32 +4296,17 @@ static int add_objects_by_path(const char *path,\n \tif (oe_end == oe_start || !window)\n \t\treturn 0;\n \n-\tsub_list_size = 0;\n-\tALLOC_ARRAY(delta_list, oe_end - oe_start);\n-\n-\tfor (size_t i = 0; i < oe_end - oe_start; i++) {\n-\t\tstruct object_entry *entry = to_pack.objects + oe_start + i;\n+\tALLOC_GROW(to_pack.regions,\n+\t\t   to_pack.nr_regions + 1,\n+\t\t   to_pack.nr_regions_alloc);\n \n-\t\tif (!should_attempt_deltas(entry))\n-\t\t\tcontinue;\n-\n-\t\tdelta_list[sub_list_size++] = entry;\n-\t}\n+\tto_pack.regions[to_pack.nr_regions].start = oe_start;\n+\tto_pack.regions[to_pack.nr_regions].nr = oe_end - oe_start;\n+\tto_pack.nr_regions++;\n \n-\t/*\n-\t * Find delta bases among this list of objects that all match the same\n-\t * path. This causes the delta compression to be interleaved in the\n-\t * object walk, which can lead to confusing progress indicators. This is\n-\t * also incompatible with threaded delta calculations. In the future,\n-\t * consider creating a list of regions in the full to_pack.objects array\n-\t * that could be picked up by the threaded delta computation.\n-\t */\n-\tif (sub_list_size && window) {\n-\t\tQSORT(delta_list, sub_list_size, type_size_sort);\n-\t\tfind_deltas(delta_list, &sub_list_size, window, depth, processed);\n-\t}\n+\t*processed += oids->nr;\n+\tdisplay_progress(progress_state, *processed);\n \n-\tfree(delta_list);\n \treturn 0;\n }\n \ndiff --git a/pack-objects.h b/pack-objects.h\nindex d73e3843c92..7ba9f3448fe 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -119,11 +119,23 @@ struct object_entry {\n \tunsigned ext_base:1; /* delta_idx points outside packlist */\n };\n \n+/**\n+ * A packing region is a section of the packing_data.objects array\n+ * as given by a starting index and a number of elements.\n+ */\n+struct packing_region {\n+\tuint32_t start;\n+\tuint32_t nr;\n+};\n+\n struct packing_data {\n \tstruct repository *repo;\n \tstruct object_entry *objects;\n \tuint32_t nr_objects, nr_alloc;\n \n+\tstruct packing_region *regions;\n+\tuint64_t nr_regions, nr_regions_alloc;\n+\n \tint32_t *index;\n \tuint32_t index_size;\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 16420d12863..c8df6afd784 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -725,7 +725,9 @@ test_expect_success '--name-hash-version=2 and --write-bitmap-index are incompat\n \n test_expect_success '--path-walk pack everything' '\n \tgit -C server rev-parse HEAD >in &&\n-\tgit -C server pack-objects --stdout --revs --path-walk <in >out.pack &&\n+\tGIT_PROGRESS_DELAY=0 git -C server pack-objects \\\n+\t\t--stdout --revs --path-walk --progress <in >out.pack 2>err &&\n+\tgrep \"Compressing objects by path\" err &&\n \tgit -C server index-pack --stdin <out.pack\n '\n \n@@ -734,7 +736,9 @@ test_expect_success '--path-walk thin pack' '\n \t$(git -C server rev-parse HEAD)\n \t^$(git -C server rev-parse HEAD~2)\n \tEOF\n-\tgit -C server pack-objects --thin --stdout --revs --path-walk <in >out.pack &&\n+\tGIT_PROGRESS_DELAY=0 git -C server pack-objects \\\n+\t\t--thin --stdout --revs --path-walk --progress <in >out.pack 2>err &&\n+\tgrep \"Compressing objects by path\" err &&\n \tgit -C server index-pack --fix-thin --stdin <out.pack\n '\n \ndiff --git a/t/t5530-upload-pack-error.sh b/t/t5530-upload-pack-error.sh\nindex 8eb6fea839a..558eedf25a4 100755\n--- a/t/t5530-upload-pack-error.sh\n+++ b/t/t5530-upload-pack-error.sh\n@@ -34,12 +34,6 @@ test_expect_success 'upload-pack fails due to error in pack-objects packing' '\n \thexsz=$(test_oid hexsz) &&\n \tprintf \"%04xwant %s\\n00000009done\\n0000\" \\\n \t\t$(($hexsz + 10)) $head >input &&\n-\n-\t# The current implementation of path-walk causes a different\n-\t# error message. This will be changed by a future refactoring.\n-\tGIT_TEST_PACK_PATH_WALK=0 &&\n-\texport GIT_TEST_PACK_PATH_WALK &&\n-\n \ttest_must_fail git upload-pack . <input >/dev/null 2>output.err &&\n \ttest_grep \"unable to read\" output.err &&\n \ttest_grep \"pack-objects died\" output.err\n-- \ngitgitgadget\n\n"},{"id":"514949","messageId":"c45f35c204eb0a2d188226d0d62faa7c7453ae67.1742829770.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v2.git.1742829769.gitgitgadget@gmail.com","subject":"[PATCH v2 12/13] path-walk: add new 'edge_aggressive' option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-24T15:22:48Z","receivedAt":"2025-03-24T15:23:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nIn preparation for allowing both the --shallow and --path-walk options\nin the 'git pack-objects' builtin, create a new 'edge_aggressive' option\nin the path-walk API. This option will help walk the boundary more\nthoroughly and help avoid sending extra objects during fetches and\npushes.\n\nThe only use of the 'edge_hint_aggressive' option in the revision API is\nwithin mark_edges_uninteresting(), which is usually called before\nbetween prepare_revision_walk() and before visiting commits with\nget_revision(). In prepare_revision_walk(), the UNINTERESTING commits\nare walked until a boundary is found.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/technical/api-path-walk.adoc |  8 ++++++++\n path-walk.c                                |  6 +++++-\n path-walk.h                                |  7 +++++++\n t/helper/test-path-walk.c                  |  2 ++\n t/t6601-path-walk.sh                       | 20 ++++++++++++++++++++\n 5 files changed, 42 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/technical/api-path-walk.adoc b/Documentation/technical/api-path-walk.adoc\nindex e522695dd9f..34c905eb9c3 100644\n--- a/Documentation/technical/api-path-walk.adoc\n+++ b/Documentation/technical/api-path-walk.adoc\n@@ -56,6 +56,14 @@ better off using the revision walk API instead.\n \tthe revision walk so that the walk emits commits marked with the\n \t`UNINTERESTING` flag.\n \n+`edge_aggressive`::\n+\tFor performance reasons, usually only the boundary commits are\n+\texplored to find UNINTERESTING objects. However, in the case of\n+\tshallow clones it can be helpful to mark all trees and blobs\n+\treachable from UNINTERESTING tip commits as UNINTERESTING. This\n+\tmatches the behavior of `--objects-edge-aggressive` in the\n+\trevision API.\n+\n `pl`::\n \tThis pattern list pointer allows focusing the path-walk search to\n \ta set of patterns, only emitting paths that match the given\ndiff --git a/path-walk.c b/path-walk.c\nindex 341bdd2ba4e..2d4ddbadd50 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -503,7 +503,11 @@ int walk_objects_by_path(struct path_walk_info *info)\n \tif (prepare_revision_walk(info->revs))\n \t\tdie(_(\"failed to setup revision walk\"));\n \n-\t/* Walk trees to mark them as UNINTERESTING. */\n+\t/*\n+\t * Walk trees to mark them as UNINTERESTING.\n+\t * This is particularly important when 'edge_aggressive' is set.\n+\t */\n+\tinfo->revs->edge_hint_aggressive = info->edge_aggressive;\n \tedge_repo = info->revs->repo;\n \tedge_tree_list = root_tree_list;\n \tmark_edges_uninteresting(info->revs, show_edge,\ndiff --git a/path-walk.h b/path-walk.h\nindex 473ee9d361c..5ef5a8440e6 100644\n--- a/path-walk.h\n+++ b/path-walk.h\n@@ -50,6 +50,13 @@ struct path_walk_info {\n \t */\n \tint prune_all_uninteresting;\n \n+\t/**\n+\t * When 'edge_aggressive' is set, then the revision walk will use\n+\t * the '--object-edge-aggressive' option to mark even more objects\n+\t * as uninteresting.\n+\t */\n+\tint edge_aggressive;\n+\n \t/**\n \t * Specify a sparse-checkout definition to match our paths to. Do not\n \t * walk outside of this sparse definition. If the patterns are in\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nindex 61e845e5ec2..fe63002c2be 100644\n--- a/t/helper/test-path-walk.c\n+++ b/t/helper/test-path-walk.c\n@@ -82,6 +82,8 @@ int cmd__path_walk(int argc, const char **argv)\n \t\t\t N_(\"toggle inclusion of tree objects\")),\n \t\tOPT_BOOL(0, \"prune\", &info.prune_all_uninteresting,\n \t\t\t N_(\"toggle pruning of uninteresting paths\")),\n+\t\tOPT_BOOL(0, \"edge-aggressive\", &info.edge_aggressive,\n+\t\t\t N_(\"toggle aggressive edge walk\")),\n \t\tOPT_BOOL(0, \"stdin-pl\", &stdin_pl,\n \t\t\t N_(\"read a pattern list over stdin\")),\n \t\tOPT_END(),\ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex c89b0f1e19d..785c2f22373 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -378,6 +378,26 @@ test_expect_success 'topic, not base, boundary with pruning' '\n \ttest_cmp_sorted expect out\n '\n \n+test_expect_success 'topic, not base, --edge-aggressive with pruning' '\n+\ttest-tool path-walk --prune --edge-aggressive -- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:commit::$(git rev-parse topic)\n+\t1:tree::$(git rev-parse topic^{tree})\n+\t1:tree::$(git rev-parse base^{tree}):UNINTERESTING\n+\t2:tree:right/:$(git rev-parse topic:right)\n+\t2:tree:right/:$(git rev-parse base:right):UNINTERESTING\n+\t3:blob:right/c:$(git rev-parse base:right/c):UNINTERESTING\n+\t3:blob:right/c:$(git rev-parse topic:right/c)\n+\tblobs:2\n+\tcommits:1\n+\ttags:0\n+\ttrees:4\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n test_expect_success 'trees are reported exactly once' '\n \ttest_when_finished \"rm -rf unique-trees\" &&\n \ttest_create_repo unique-trees &&\n-- \ngitgitgadget\n\n"},{"id":"514950","messageId":"ae73d26319ac5db338507988eb23f1ecaac67671.1742829770.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v2.git.1742829769.gitgitgadget@gmail.com","subject":"[PATCH v2 11/13] pack-objects: thread the path-based compression","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-24T15:22:47Z","receivedAt":"2025-03-24T15:23:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nAdapting the implementation of ll_find_deltas(), create a threaded\nversion of the --path-walk compression step in 'git pack-objects'.\n\nThis involves adding a 'regions' member to the thread_params struct,\nallowing each thread to own a section of paths. We can simplify the way\njobs are split because there is no value in extending the batch based on\nname-hash the way sections of the object entry array are attempted to be\ngrouped. We re-use the 'list_size' and 'remaining' items for the purpose\nof borrowing work in progress from other \"victim\" threads when a thread\nhas finished its batch of work more quickly.\n\nUsing the Git repository as a test repo, the p5313 performance test\nshows that the resulting size of the repo is the same, but the threaded\nimplementation gives gains of varying degrees depending on the number of\nobjects being packed. (This was tested on a 16-core machine.)\n\nTest                        HEAD~1      HEAD\n---------------------------------------------------\n5313.20: big pack             2.38      1.99 -16.4%\n5313.21: big pack size       16.1M     16.0M  -0.2%\n5313.24: repack             107.32     45.41 -57.7%\n5313.25: repack size        213.3M    213.2M  -0.0%\n\n(Test output is formatted to better fit in message.)\n\nThis ~60% reduction in 'git repack --path-walk' time is typical across\nall repos I used for testing. What is interesting is to compare when the\noverall time improves enough to outperform the --name-hash-version=1\ncase. These time improvements correlate with repositories with data\nshapes that significantly improve their data size as well. The\n--path-walk feature frequently takes longer than --name-hash-verison=2,\ntrading some extrac computation for some additional compression. The\nnatural place where this additional computation comes from is the two\ncompression passes that --path-walk takes, though the first pass is\nnaturally faster due to the path boundaries avoiding a number of delta\ncompression attempts.\n\nFor example, the microsoft/fluentui repo has significant size reduction\nfrom --name-hash-version=1 to --name-hash-version=2 followed by further\nimprovements with --path-walk. The threaded computation makes\n--path-walk more competitive in time compared to --name-hash-version=2,\nthough still ~31% more expensive in that metric.\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                439.4M      87.24s\nHash v2                161.7M      21.51s\nPath Walk (Before)     142.5M      81.29s\nPath Walk (After)      142.5M      28.16s\n\nSimilar results hold for the Git repository:\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                248.8M      30.44s\nHash v2                249.0M      30.15s\nPath Walk (Before)     213.2M     142.50s\nPath Walk (After)      213.3M      45.41s\n\n...as well as the nodejs/node repository:\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                739.9M      71.18s\nHash v2                764.6M      67.82s\nPath Walk (Before)     698.1M     208.10s\nPath Walk (After)      698.0M      75.10s\n\nFinally, the Linux kernel repository is a good test for this repacking\ntime change, even though the space savings is more subtle:\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                  2.5G     554.41s\nHash v2                  2.5G     549.62s\nPath Walk (before)       2.2G    1562.36s\nPath Walk (before)       2.2G     559.00s\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c | 163 ++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 161 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex d4e05ca4434..2a6246c1e78 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2964,6 +2964,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n struct thread_params {\n \tpthread_t thread;\n \tstruct object_entry **list;\n+\tstruct packing_region *regions;\n \tunsigned list_size;\n \tunsigned remaining;\n \tint window;\n@@ -3281,6 +3282,164 @@ static void find_deltas_by_region(struct object_entry *list,\n \tstop_progress(&progress_state);\n }\n \n+static void *threaded_find_deltas_by_path(void *arg)\n+{\n+\tstruct thread_params *me = arg;\n+\n+\tprogress_lock();\n+\twhile (me->remaining) {\n+\t\twhile (me->remaining) {\n+\t\t\tprogress_unlock();\n+\t\t\tfind_deltas_for_region(to_pack.objects,\n+\t\t\t\t\t       me->regions,\n+\t\t\t\t\t       me->processed);\n+\t\t\tprogress_lock();\n+\t\t\tme->remaining--;\n+\t\t\tme->regions++;\n+\t\t}\n+\n+\t\tme->working = 0;\n+\t\tpthread_cond_signal(&progress_cond);\n+\t\tprogress_unlock();\n+\n+\t\t/*\n+\t\t * We must not set ->data_ready before we wait on the\n+\t\t * condition because the main thread may have set it to 1\n+\t\t * before we get here. In order to be sure that new\n+\t\t * work is available if we see 1 in ->data_ready, it\n+\t\t * was initialized to 0 before this thread was spawned\n+\t\t * and we reset it to 0 right away.\n+\t\t */\n+\t\tpthread_mutex_lock(&me->mutex);\n+\t\twhile (!me->data_ready)\n+\t\t\tpthread_cond_wait(&me->cond, &me->mutex);\n+\t\tme->data_ready = 0;\n+\t\tpthread_mutex_unlock(&me->mutex);\n+\n+\t\tprogress_lock();\n+\t}\n+\tprogress_unlock();\n+\t/* leave ->working 1 so that this doesn't get more work assigned */\n+\treturn NULL;\n+}\n+\n+static void ll_find_deltas_by_region(struct object_entry *list,\n+\t\t\t\t     struct packing_region *regions,\n+\t\t\t\t     uint32_t start, uint32_t nr)\n+{\n+\tstruct thread_params *p;\n+\tint i, ret, active_threads = 0;\n+\tunsigned int processed = 0;\n+\tuint32_t progress_nr;\n+\tinit_threaded_search();\n+\n+\tif (!nr)\n+\t\treturn;\n+\n+\tprogress_nr =  regions[nr - 1].start + regions[nr - 1].nr;\n+\tif (delta_search_threads <= 1) {\n+\t\tfind_deltas_by_region(list, regions, start, nr);\n+\t\tcleanup_threaded_search();\n+\t\treturn;\n+\t}\n+\n+\tif (progress > pack_to_stdout)\n+\t\tfprintf_ln(stderr, _(\"Path-based delta compression using up to %d threads\"),\n+\t\t\t   delta_search_threads);\n+\tCALLOC_ARRAY(p, delta_search_threads);\n+\n+\tif (progress)\n+\t\tprogress_state = start_progress(the_repository,\n+\t\t\t\t\t\t_(\"Compressing objects by path\"),\n+\t\t\t\t\t\tprogress_nr);\n+\t/* Partition the work amongst work threads. */\n+\tfor (i = 0; i < delta_search_threads; i++) {\n+\t\tunsigned sub_size = nr / (delta_search_threads - i);\n+\n+\t\tp[i].window = window;\n+\t\tp[i].depth = depth;\n+\t\tp[i].processed = &processed;\n+\t\tp[i].working = 1;\n+\t\tp[i].data_ready = 0;\n+\n+\t\tp[i].regions = regions;\n+\t\tp[i].list_size = sub_size;\n+\t\tp[i].remaining = sub_size;\n+\n+\t\tregions += sub_size;\n+\t\tnr -= sub_size;\n+\t}\n+\n+\t/* Start work threads. */\n+\tfor (i = 0; i < delta_search_threads; i++) {\n+\t\tif (!p[i].list_size)\n+\t\t\tcontinue;\n+\t\tpthread_mutex_init(&p[i].mutex, NULL);\n+\t\tpthread_cond_init(&p[i].cond, NULL);\n+\t\tret = pthread_create(&p[i].thread, NULL,\n+\t\t\t\t     threaded_find_deltas_by_path, &p[i]);\n+\t\tif (ret)\n+\t\t\tdie(_(\"unable to create thread: %s\"), strerror(ret));\n+\t\tactive_threads++;\n+\t}\n+\n+\t/*\n+\t * Now let's wait for work completion.  Each time a thread is done\n+\t * with its work, we steal half of the remaining work from the\n+\t * thread with the largest number of unprocessed objects and give\n+\t * it to that newly idle thread.  This ensure good load balancing\n+\t * until the remaining object list segments are simply too short\n+\t * to be worth splitting anymore.\n+\t */\n+\twhile (active_threads) {\n+\t\tstruct thread_params *target = NULL;\n+\t\tstruct thread_params *victim = NULL;\n+\t\tunsigned sub_size = 0;\n+\n+\t\tprogress_lock();\n+\t\tfor (;;) {\n+\t\t\tfor (i = 0; !target && i < delta_search_threads; i++)\n+\t\t\t\tif (!p[i].working)\n+\t\t\t\t\ttarget = &p[i];\n+\t\t\tif (target)\n+\t\t\t\tbreak;\n+\t\t\tpthread_cond_wait(&progress_cond, &progress_mutex);\n+\t\t}\n+\n+\t\tfor (i = 0; i < delta_search_threads; i++)\n+\t\t\tif (p[i].remaining > 2*window &&\n+\t\t\t    (!victim || victim->remaining < p[i].remaining))\n+\t\t\t\tvictim = &p[i];\n+\t\tif (victim) {\n+\t\t\tsub_size = victim->remaining / 2;\n+\t\t\ttarget->regions = victim->regions + victim->remaining - sub_size;\n+\t\t\tvictim->list_size -= sub_size;\n+\t\t\tvictim->remaining -= sub_size;\n+\t\t}\n+\t\ttarget->list_size = sub_size;\n+\t\ttarget->remaining = sub_size;\n+\t\ttarget->working = 1;\n+\t\tprogress_unlock();\n+\n+\t\tpthread_mutex_lock(&target->mutex);\n+\t\ttarget->data_ready = 1;\n+\t\tpthread_cond_signal(&target->cond);\n+\t\tpthread_mutex_unlock(&target->mutex);\n+\n+\t\tif (!sub_size) {\n+\t\t\tpthread_join(target->thread, NULL);\n+\t\t\tpthread_cond_destroy(&target->cond);\n+\t\t\tpthread_mutex_destroy(&target->mutex);\n+\t\t\tactive_threads--;\n+\t\t}\n+\t}\n+\tcleanup_threaded_search();\n+\tfree(p);\n+\n+\tdisplay_progress(progress_state, progress_nr);\n+\tstop_progress(&progress_state);\n+}\n+\n static void prepare_pack(int window, int depth)\n {\n \tstruct object_entry **delta_list;\n@@ -3306,8 +3465,8 @@ static void prepare_pack(int window, int depth)\n \t\treturn;\n \n \tif (path_walk)\n-\t\tfind_deltas_by_region(to_pack.objects, to_pack.regions,\n-\t\t\t\t      0, to_pack.nr_regions);\n+\t\tll_find_deltas_by_region(to_pack.objects, to_pack.regions,\n+\t\t\t\t\t 0, to_pack.nr_regions);\n \n \tALLOC_ARRAY(delta_list, to_pack.nr_objects);\n \tnr_deltas = n = 0;\n-- \ngitgitgadget\n\n"},{"id":"514951","messageId":"d5484ebd9428f4d7dce05e43c5067d741dc37a9d.1742829770.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v2.git.1742829769.gitgitgadget@gmail.com","subject":"[PATCH v2 13/13] pack-objects: allow --shallow and --path-walk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-03-24T15:22:49Z","receivedAt":"2025-03-24T15:23:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThere does not appear to be anything particularly incompatible about the\n--shallow and --path-walk options of 'git pack-objects'. If shallow\ncommits are to be handled differently, then it is by the revision walk\nthat defines the commit set and which are interesting or uninteresting.\n\nHowever, before the previous change, a trivial removal of the warning\nwould cause a failure in t5500-fetch-pack.sh when\nGIT_TEST_PACK_PATH_WALK is enabled. The shallow fetch would provide more\nobjects than we desired, due to some incorrect behavior of the path-walk\nAPI, especially around walking uninteresting objects.\n\nThe recently-added tests in t5538-push-shallow.sh help to confirm this\nbehavior is working with the --path-walk option if\nGIT_TEST_PACK_PATH_WALK is enabled. These tests passed previously due to\nthe --path-walk feature being disabled in the presence of a shallow\nclone.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c | 7 ++-----\n 1 file changed, 2 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 2a6246c1e78..7db2ebc7962 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -209,6 +209,7 @@ static int keep_unreachable, unpack_unreachable, include_tag;\n static timestamp_t unpack_unreachable_expiration;\n static int pack_loose_unreachable;\n static int cruft;\n+static int shallow = 0;\n static timestamp_t cruft_expiration;\n static int local;\n static int have_non_local_packs;\n@@ -4486,6 +4487,7 @@ static void get_object_list_path_walk(struct rev_info *revs)\n \t * base objects.\n \t */\n \tinfo.prune_all_uninteresting = sparse;\n+\tinfo.edge_aggressive = shallow;\n \n \tif (walk_objects_by_path(&info))\n \t\tdie(_(\"failed to pack objects via path-walk\"));\n@@ -4687,7 +4689,6 @@ int cmd_pack_objects(int argc,\n \t\t     struct repository *repo UNUSED)\n {\n \tint use_internal_rev_list = 0;\n-\tint shallow = 0;\n \tint all_progress_implied = 0;\n \tstruct strvec rp = STRVEC_INIT;\n \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n@@ -4875,10 +4876,6 @@ int cmd_pack_objects(int argc,\n \t\twarning(_(\"cannot use delta islands with --path-walk\"));\n \t\tpath_walk = 0;\n \t}\n-\tif (path_walk && shallow) {\n-\t\twarning(_(\"cannot use --shallow with --path-walk\"));\n-\t\tpath_walk = 0;\n-\t}\n \tif (path_walk) {\n \t\tstrvec_push(&rp, \"--boundary\");\n \t\t /*\n-- \ngitgitgadget\n"},{"id":"517133","messageId":"xmqqbjsau2nl.fsf@gitster.g","threadId":"63083","inReplyTo":"pull.1819.v2.git.1742829769.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/13] PATH WALK II: Add --path-walk option to 'git pack-objects'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-05-02T21:24:30Z","receivedAt":"2025-05-02T21:24:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> This patch series does the following:\n>\n>  1. Add a new '--path-walk' option to 'git pack-objects' that uses the\n>     path-walk API instead of the revision API to collect objects for delta\n>     compression.\n>\n>  2. Add a new '--path-walk' option to 'git repack' to pass this option along\n>     to 'git pack-objects'.\n>\n>  3. Add a new 'pack.usePathWalk' config option to opt into this option\n>     implicitly, such as in 'git push'.\n>\n>  4. Optimize the '--path-walk' option using threading so it better competes\n>     with the existing multi-threaded delta compression mechanism.\n>\n>  5. Update the path-walk API with a new 'edge_aggressive' option that pairs\n>     close to the --edge-aggressive option in the revision API. This is\n>     useful when creating thin packs inside shallow clones.\n>\n> This feature works by using the path-walk API to emit groups of objects that\n> appear at the same path. These groups are tracked so they can be tested for\n> delta compression with each other, and then after those groups are tested a\n> second pass using the name-hash attempts to find better (or first time)\n> deltas across path boundaries. This second pass is much faster than a fresh\n> pass since the existing deltas are used as a limit for the size of\n> potentially new deltas, short-circuiting the checks when the delta size\n> exceeds the current-best.\n> ...\n> This feature was shipped with similar features in microsoft/git as of\n> v2.47.0.vfs.0.3 [4]. This was used in CI machines for an internal monorepo\n> that had significant repository growth due to constructing a batch of\n> beachball [5] CHANGELOG.[md|json] files and pushing them to a release\n> branch. These pushes were frequently 70-200 MB due to poor delta\n> compression. Using the 'pack.usePathWalk=true' config, these pushes dropped\n> in size by 100x while improving performance. Since these CI machines were\n> working with a shallow clone, the 'edge_aggressive' changes were required to\n> enable the path-walk option.\n>\n> [4] https://github.com/microsoft/git/releases/tag/v2.47.0.vfs.0.3\n>\n> [5] https://github.com/microsoft/beachball\n>\n>\n> Updates in v2\n> =============\n>\n>  * Re-added a dropped comment when moving code in patch 1.\n>  * Updated documentation to include interaction with --use-bitmap-index.\n>  * An UNUSED parameter is now used, reducing the use of global variables\n>    slightly.\n\nThe iteration saw no comments from anybody, so I (naturally) forgot\nabout it for quite a long time.  Let me mark it for 'next'.\n\nThanks.\n"},{"id":"517137","messageId":"aBVLC57bMJKjygyi@nand.local","threadId":"63083","inReplyTo":"xmqqbjsau2nl.fsf@gitster.g","subject":"Re: [PATCH v2 00/13] PATH WALK II: Add --path-walk option to 'git pack-objects'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-02T22:45:31Z","receivedAt":"2025-05-02T22:45:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, May 02, 2025 at 02:24:30PM -0700, Junio C Hamano wrote:\n> > Updates in v2\n> > =============\n> >\n> >  * Re-added a dropped comment when moving code in patch 1.\n> >  * Updated documentation to include interaction with --use-bitmap-index.\n> >  * An UNUSED parameter is now used, reducing the use of global variables\n> >    slightly.\n>\n> The iteration saw no comments from anybody, so I (naturally) forgot\n> about it for quite a long time.  Let me mark it for 'next'.\n\nSorry, this fell off of my to-do list. I don't want to hold things up\nfurther, but would appreciate a chance to review this before it hits\n'next'.\n\nI can take a look now.\n\nThanks,\nTaylor\n"},{"id":"517138","messageId":"aBVL16rCzFZGjw89@nand.local","threadId":"63083","inReplyTo":"57c1cc20de0c80b84ad11a6546763826fe4b1a09.1742829770.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 01/13] pack-objects: extract should_attempt_deltas()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-02T22:48:55Z","receivedAt":"2025-05-02T22:48:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 24, 2025 at 03:22:37PM +0000, Derrick Stolee via GitGitGadget wrote:\n> @@ -3226,33 +3256,11 @@ static void prepare_pack(int window, int depth)\n>  \tfor (i = 0; i < to_pack.nr_objects; i++) {\n>  \t\tstruct object_entry *entry = to_pack.objects + i;\n>\n> -\t\tif (DELTA(entry))\n> -\t\t\t/* This happens if we decided to reuse existing\n> -\t\t\t * delta from a pack.  \"reuse_delta &&\" is implied.\n> -\t\t\t */\n> -\t\t\tcontinue;\n> -\n> -\t\tif (!entry->type_valid ||\n> -\t\t    oe_size_less_than(&to_pack, entry, 50))\n> +\t\tif (!should_attempt_deltas(entry))\n>  \t\t\tcontinue;\n>\n> -\t\tif (entry->no_try_delta)\n> -\t\t\tcontinue;\n> -\n> -\t\tif (!entry->preferred_base) {\n> +\t\tif (!entry->preferred_base)\n>  \t\t\tnr_deltas++;\n\nOK, this part didn't move since nr_deltas is local to prepare_pack().\nThe transformation looks equivalent to me, with the only minor\ndifference being an inconsequential one. Before, we would increment\nnr_deltas if oe_type() failed, but now we don't. This doesn't matter\nsince the failure mode is die(), so I think this is all OK.\n\nThanks,\nTaylor\n"},{"id":"517139","messageId":"aBVTcvhW+lW0GLpQ@nand.local","threadId":"63083","inReplyTo":"a271d6245c243d50d95ea6c7e624b3165f7da558.1742829770.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 02/13] pack-objects: add --path-walk option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-02T23:21:22Z","receivedAt":"2025-05-02T23:21:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 24, 2025 at 03:22:38PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <stolee@gmail.com>\n>\n> In order to more easily compute delta bases among objects that appear at\n> the exact same path, add a --path-walk option to 'git pack-objects'.\n>\n> This option will use the path-walk API instead of the object walk given\n> by the revision machinery. Since objects will be provided in batches\n> representing a common path, those objects can be tested for delta bases\n> immediately instead of waiting for a sort of the full object list by\n> name-hash. This has multiple benefits, including avoiding collisions by\n> name-hash.\n>\n> The objects marked as UNINTERESTING are included in these batches, so we\n> are guaranteeing some locality to find good delta bases.\n>\n> After the individual passes are done on a per-path basis, the default\n> name-hash is used to find other opportunistic delta bases that did not\n> match exactly by the full path name.\n\nThanks for the clear explanation here.\n\n> The current implementation performs delta calculations while walking\n> objects, which is not ideal for a few reasons. First, this will cause\n> the \"Enumerating objects\" phase to be much longer than usual. Second, it\n> does not take advantage of threading during the path-scoped delta\n> calculations. Even with this lack of threading, the path-walk option is\n> sometimes faster than the usual approach. Future changes will refactor\n> this code to allow for threading, but that complexity is deferred until\n> later to keep this patch as simple as possible.\n\nI can't remember if I called this out in my previous round of review,\nbut I definitely appreciate you separating out the threading\nimplementation. When I first started to read this paragraph, I thought\n\"huh, I bet this is parallelizeable\", and I was glad to see that you\nboth (a) called out the same thing, and (b) deferred it to a later patch\nto reduce the complexity of this one. Thanks.\n\n> This new walk is incompatible with some features and is ignored by\n> others:\n>\n>  * Object filters are not currently integrated with the path-walk API,\n>    such as sparse-checkout or tree depth. A blobless packfile could be\n>    integrated easily, but that is deferred for later.\n>\n>  * Server-focused features such as delta islands, shallow packs, and\n>    using a bitmap index are incompatible with the path-walk API.\n>\n>  * The path walk API is only compatible with the --revs option, not\n>    taking object lists or pack lists over stdin. These alternative ways\n>    to specify the objects currently ignores the --path-walk option\n>    without even a warning.\n\nMuch appreciated to likewise call out the (current) limitations here as\nwell.\n\n> Future changes will create performance tests that demonstrate the power\n> of this approach.\n>\n> Signed-off-by: Derrick Stolee <stolee@gmail.com>\n> ---\n>  Documentation/git-pack-objects.adoc        |  14 +-\n>  Documentation/technical/api-path-walk.adoc |   1 +\n>  builtin/pack-objects.c                     | 147 +++++++++++++++++++--\n>  t/t5300-pack-object.sh                     |  15 +++\n>  4 files changed, 167 insertions(+), 10 deletions(-)\n>\n> diff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc\n> index 7f69ae4855f..7065758eddf 100644\n> --- a/Documentation/git-pack-objects.adoc\n> +++ b/Documentation/git-pack-objects.adoc\n> @@ -16,7 +16,7 @@ SYNOPSIS\n>  \t[--cruft] [--cruft-expiration=<time>]\n>  \t[--stdout [--filter=<filter-spec>] | <base-name>]\n>  \t[--shallow] [--keep-true-parents] [--[no-]sparse]\n> -\t[--name-hash-version=<n>] < <object-list>\n> +\t[--name-hash-version=<n>] [--path-walk] < <object-list>\n>\n>\n>  DESCRIPTION\n> @@ -375,6 +375,18 @@ many different directories. At the moment, this version is not allowed\n>  when writing reachability bitmap files with `--write-bitmap-index` and it\n>  will be automatically changed to version `1`.\n>\n> +--path-walk::\n> +\tBy default, `git pack-objects` walks objects in an order that\n> +\tpresents trees and blobs in an order unrelated to the path they\n> +\tappear relative to a commit's root tree. The `--path-walk` option\n> +\tenables a different walking algorithm that organizes trees and\n> +\tblobs by path. This has the potential to improve delta compression\n> +\tespecially in the presence of filenames that cause collisions in\n> +\tGit's default name-hash algorithm. Due to changing how the objects\n> +\tare walked, this option is not compatible with `--delta-islands`,\n> +\t`--shallow`, or `--filter`. The `--use-bitmap-index` option will\n> +\tbe ignored in the presence of `--path-walk.`\n> +\n\nThis is perhaps a stylistic difference, but I don't think in general it\nis necessary to say \"by default [...]\" to explain some existing\nbehavior, unless the alternative behavior enabled by the flag is\nimpossible to understand without contrasting.\n\nIn this particular instance, I think that \"--path-walk\" can be\nexplained in isolation rather than in contrast to the default behavior.\n\nI would probably write something like:\n\n    --path-walk::\n    Walk objects in batches grouped by a common path. This can improve\n    delta compression, especially in the case where filenames cause\n    collisions in the default name-hash algorithm.\n\n    Incompatible with `--delta-islands`, `--shallow`, and `--filter`. If\n    provided, `--use-bitmap-index` will be ignored.\n\n> +static inline int is_oid_interesting(struct repository *repo,\n> +\t\t\t\t     struct object_id *oid)\n> +{\n> +\tstruct object *o = lookup_object(repo, oid);\n\nThis function and its caller tripped me up a little bit while reading,\nbut I was able to un-confuse myself. Here are some notes that I took\nwhile reading, which perhaps you can validate to tell me whether or not\nI'm on the right track.\n\nHere we allow lookup_object() to return NULL, and then immediately\nreturn 0 if it does. But...\n\n> +\treturn o && !(o->flags & UNINTERESTING);\n> +}\n> +\n> +static int add_objects_by_path(const char *path,\n> +\t\t\t       struct oid_array *oids,\n> +\t\t\t       enum object_type type,\n> +\t\t\t       void *data)\n> +{\n> +\tstruct object_entry **delta_list;\n> +\tsize_t oe_start = to_pack.nr_objects;\n> +\tsize_t oe_end;\n> +\tunsigned int sub_list_size;\n\nCould you help me understand why sub_list_size is an unsigned int and\nnot a size_t here?\n\n> +\tunsigned int *processed = data;\n> +\n> +\t/*\n> +\t * First, add all objects to the packing data, including the ones\n> +\t * marked UNINTERESTING (translated to 'exclude') as they can be\n> +\t * used as delta bases.\n> +\t */\n> +\tfor (size_t i = 0; i < oids->nr; i++) {\n> +\t\tint exclude;\n> +\t\tstruct object_info oi = OBJECT_INFO_INIT;\n> +\t\tstruct object_id *oid = &oids->oid[i];\n> +\n> +\t\t/* Skip objects that do not exist locally. */\n> +\t\tif (exclude_promisor_objects &&\n> +\t\t    oid_object_info_extended(the_repository, oid, &oi,\n> +\t\t\t\t\t     OBJECT_INFO_FOR_PREFETCH) < 0)\n> +\t\t\tcontinue;\n> +\n> +\t\texclude = !is_oid_interesting(the_repository, oid);\n> +\n> +\t\tif (exclude && !thin)\n> +\t\t\tcontinue;\n\n...here we only skip calling add_object_entry() if the object is\nexcluded (which lookup_object() returning NULL would cause above) *and*\nwe're not using thin packs. That makes sense, since if we have a missing\nobject, but we're using thin packs, then it's OK to skip.\n\nSo I think all of this looks good. However, I find the interface here a\nlittle awkward. The function is called \"is_oid_interesting()\", but it\ndetermines that fact by checking whether or not the object has the\nUNINTERSTING bit set on its flag, and then negating the result. But then\nwe negate it again here at the caller to obtain whether or not we should\n\"exclude\" the object.\n\nI was wondering if there were other callers of is_oid_interesting() that\ndon't negate its result. But it doesn't look like there are (at least in\nthis patch). I wonder if it would be cleaner just to inline this instead\nof having the result negated at multiple levels.\n\n> +\t\tadd_object_entry(oid, type, path, exclude);\n> +\t}\n> +\n> +\toe_end = to_pack.nr_objects;\n> +\n> +\t/* We can skip delta calculations if it is a no-op. */\n> +\tif (oe_end == oe_start || !window)\n> +\t\treturn 0;\n> +\n> +\tsub_list_size = 0;\n\nI'm nit-picking, but I would imagine that sub_list_nr is probably more\nconventional, but this is textbook bike-shedding ;-).\n\n> +\tALLOC_ARRAY(delta_list, oe_end - oe_start);\n\nALLOC_ARRAY() will barf if oe_start > oe_end and we wrap around the\nrange of size_t, but it might be nice to have a sanity check here just\ntheir equality.\n\n> +\n> +\tfor (size_t i = 0; i < oe_end - oe_start; i++) {\n> +\t\tstruct object_entry *entry = to_pack.objects + oe_start + i;\n> +\n> +\t\tif (!should_attempt_deltas(entry))\n> +\t\t\tcontinue;\n> +\n> +\t\tdelta_list[sub_list_size++] = entry;\n> +\t}\n\nMakes sense.\n\n> +\t/*\n> +\t * Find delta bases among this list of objects that all match the same\n> +\t * path. This causes the delta compression to be interleaved in the\n> +\t * object walk, which can lead to confusing progress indicators. This is\n> +\t * also incompatible with threaded delta calculations. In the future,\n> +\t * consider creating a list of regions in the full to_pack.objects array\n> +\t * that could be picked up by the threaded delta computation.\n> +\t */\n> +\tif (sub_list_size && window) {\n\nDo we need to check that window is non-zero here? It looks like that\ncheck already happens above.\n\n> +\t\tQSORT(delta_list, sub_list_size, type_size_sort);\n> +\t\tfind_deltas(delta_list, &sub_list_size, window, depth, processed);\n> +\t}\n> +\n> +\tfree(delta_list);\n\nIs there a return path through this function that doesn't allocate\ndelta_list? I think the answer is \"no\", but it might be a nice guard\nagainst future refactorings to initialize this field to NULL just in\ncase.\n\n> +\treturn 0;\n> +}\n> +\n> +static void get_object_list_path_walk(struct rev_info *revs)\n> +{\n> +\tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n> +\tunsigned int processed = 0;\n> +\n> +\tinfo.revs = revs;\n> +\tinfo.path_fn = add_objects_by_path;\n> +\tinfo.path_fn_data = &processed;\n> +\trevs->tag_objects = 1;\n\nWhat is the purpose of setting revs->tag_objects to 1 here? Do we need\nto restore it back to its original value before returning? A comment\nhere would be useful, I think.\n\n> +\n> +\t/*\n> +\t * Allow the --[no-]sparse option to be interesting here, if only\n> +\t * for testing purposes. Paths with no interesting objects will not\n> +\t * contribute to the resulting pack, but only create noisy preferred\n> +\t * base objects.\n> +\t */\n> +\tinfo.prune_all_uninteresting = sparse;\n> +\n> +\tif (walk_objects_by_path(&info))\n> +\t\tdie(_(\"failed to pack objects via path-walk\"));\n> +}\n> +\n>  static void get_object_list(struct rev_info *revs, int ac, const char **av)\n>  {\n>  \tstruct setup_revision_opt s_r_opt = {\n> @@ -4237,7 +4340,7 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n>\n>  \twarn_on_object_refname_ambiguity = save_warning;\n>\n> -\tif (use_bitmap_index && !get_object_list_from_bitmap(revs))\n> +\tif (use_bitmap_index && !path_walk && !get_object_list_from_bitmap(revs))\n\nI was going to suggest something like:\n\n    if (path_walk && use_bitmap_index) {\n        warning(_(\"cannot use --use-bitmap-index with --path-walk, ignoring\"));\n        use_bitmap_index = 0;\n    }\n\n, which would avoid the need for this check here and insulate us against\nhaving to make similar changes to future conditionals that care about\nuse_bitmap_index.\n\nBut it looks like such a check *does* already exist below, so I am not\nsure I understand why we need to check for path_walk here.\n\n>  \t\treturn;\n>\n>  \tif (use_delta_islands)\n> @@ -4246,15 +4349,19 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n>  \tif (write_bitmap_index)\n>  \t\tmark_bitmap_preferred_tips();\n>\n> -\tif (prepare_revision_walk(revs))\n> -\t\tdie(_(\"revision walk setup failed\"));\n> -\tmark_edges_uninteresting(revs, show_edge, sparse);\n> -\n>  \tif (!fn_show_object)\n>  \t\tfn_show_object = show_object;\n> -\ttraverse_commit_list(revs,\n> -\t\t\t     show_commit, fn_show_object,\n> -\t\t\t     NULL);\n> +\n> +\tif (path_walk) {\n> +\t\tget_object_list_path_walk(revs);\n> +\t} else {\n> +\t\tif (prepare_revision_walk(revs))\n> +\t\t\tdie(_(\"revision walk setup failed\"));\n> +\t\tmark_edges_uninteresting(revs, show_edge, sparse);\n> +\t\ttraverse_commit_list(revs,\n> +\t\t\t\tshow_commit, fn_show_object,\n> +\t\t\t\tNULL);\n> +\t}\n>\n>  \tif (unpack_unreachable_expiration) {\n>  \t\trevs->ignore_missing_links = 1;\n> @@ -4464,6 +4571,8 @@ int cmd_pack_objects(int argc,\n>  \t\t\t N_(\"use the sparse reachability algorithm\")),\n>  \t\tOPT_BOOL(0, \"thin\", &thin,\n>  \t\t\t N_(\"create thin packs\")),\n> +\t\tOPT_BOOL(0, \"path-walk\", &path_walk,\n> +\t\t\t N_(\"use the path-walk API to walk objects when possible\")),\n>  \t\tOPT_BOOL(0, \"shallow\", &shallow,\n>  \t\t\t N_(\"create packs suitable for shallow fetches\")),\n>  \t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep_on_disk,\n> @@ -4549,7 +4658,27 @@ int cmd_pack_objects(int argc,\n>  \t\twindow = 0;\n>\n>  \tstrvec_push(&rp, \"pack-objects\");\n> -\tif (thin) {\n> +\n> +\tif (path_walk && filter_options.choice) {\n> +\t\twarning(_(\"cannot use --filter with --path-walk\"));\n> +\t\tpath_walk = 0;\n> +\t}\n> +\tif (path_walk && use_delta_islands) {\n> +\t\twarning(_(\"cannot use delta islands with --path-walk\"));\n> +\t\tpath_walk = 0;\n> +\t}\n> +\tif (path_walk && shallow) {\n> +\t\twarning(_(\"cannot use --shallow with --path-walk\"));\n> +\t\tpath_walk = 0;\n> +\t}\n\nHere's a tiny idea I had while reading this code.\n\n    if (path_walk) {\n        const char *option = NULL;\n        if (filter_options.choice)\n            option = \"--filter\";\n        else if (use_delta_islands)\n            option = \"--delta-islands\";\n        else if (shallow)\n            option = \"--shallow\";\n\n        if (option) {\n            warning(_(\"cannot use %s with --path-walk\"), option);\n            path_walk = 0;\n        }\n    }\n\nThis may be too DRY for your taste, and it does have the disadvantage of\nreducing the grep-ability of the error messages. I don't have a strong\nfeeling about it one way or another, but I figured I'd at least write it\ndown.\n\nThanks,\nTaylor\n"},{"id":"517140","messageId":"aBVUeRFKTaYNDOmc@nand.local","threadId":"63083","inReplyTo":"97a0b52ccee8844c1c91232c625c2c6444873d85.1742829770.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 04/13] p5313: add performance tests for --path-walk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-02T23:25:45Z","receivedAt":"2025-05-02T23:25:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 24, 2025 at 03:22:40PM +0000, Derrick Stolee via GitGitGadget wrote:\n> The cases where the --path-walk option really shines is when the default\n> name-hash is overwhelmed with collisions. An open source example can be\n> found in the microsoft/fluentui repo [1] at a certain commit [2].\n>\n> [1] https://github.com/microsoft/fluentui\n> [2] e70848ebac1cd720875bccaa3026f4a9ed700e08\n\nA meta-comment that occurred to me at this point while reading through\nthe series. I typically think of the namehash function here as wanting\nto have some amount of collisions so that objects which are related but\nhave slightly different paths (e.g., due to a rename, or similar) are\ndelta'd against each other.\n\nIn my mind, the problem isn't that the hash function has collisions, but\nthat it has *bad* collisions between two paths where the content of each\nobject is completely unrelated to the other, and any effort to find\ndeltas here would be wasted at best.\n\nI'm not sure I have a concrete suggestion for how to reword this, but I\nfigured it was worth mentioning since not all collisions are bad or\ncreated equal ;-).\n\n> Running the tests on this repo results in the following comparison table:\n>\n> Pack Type            Hash v1    Hash v2    Path Walk\n> ---------------------------------------------------\n> thin pack    (time)    0.36s      0.12s      0.08s\n>              (size)    1.2M      22.0K      18.4K\n> big pack     (time)    2.00s      2.90s      2.21s\n>              (size)   20.4M      25.9M      19.5M\n> shallow pack (time)    1.41s      1.80s      1.65s\n>              (size)   34.4M      33.7M      33.6M\n\nThanks for reformatting. As a side-note (unrelated to your series), I do\nthink it would be nice to have a t/perf-native way of doing this sort of\nmulti-dimensional comparison.\n\n> ---\n>  t/perf/p5313-pack-objects.sh | 37 ++++++++++++++++++++++--------------\n>  1 file changed, 23 insertions(+), 14 deletions(-)\n\nThe patch itself makes perfect sense to me.\n\nThanks,\nTaylor\n"},{"id":"517141","messageId":"aBVVwCZ0NZT3KLB/@nand.local","threadId":"63083","inReplyTo":"0d49bb3d30add66676280ec7fabed12351d5b3ac.1742829770.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 05/13] pack-objects: introduce GIT_TEST_PACK_PATH_WALK","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-02T23:31:12Z","receivedAt":"2025-05-02T23:31:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 24, 2025 at 03:22:41PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <stolee@gmail.com>\n>\n> There are many tests that validate whether 'git pack-objects' works as\n> expected. Instead of duplicating these tests, add a new test environment\n> variable, GIT_TEST_PACK_PATH_WALK, that implies --path-walk by default\n> when specified.\n\nI normally dislike adding new GIT_TEST_* environment variables, because\nI don't think we (myself included) are great about remembering to get\nrid of them. But in this case I think it does make sense to add one.\n\n> This was useful in testing the implementation of the --path-walk\n> implementation, especially in conjunction with test such as:\n\nI am not quite following... this sentence sets up a list of tests which\nI was expecting to show some demonstration of how they uniquely\nexercised the path-walk feature. But instead it looks like it describes\ntests that are sensitive to object ordering within packs and thus had to\nhave the new GIT_TEST_PACK_PATH_WALK variable unset/set to zero.\n\n>  - t0411-clone-from-partial.sh : One test fetches from a repo that does\n>    not have the boundary objects. This causes the path-based walk to\n>    fail. Disable the variable for this test.\n>\n> [...]\n\nThanks,\nTaylor\n"},{"id":"517143","messageId":"aBVWfJUFNdfwS0PO@nand.local","threadId":"63083","inReplyTo":"ddf804e606a5560639e4018709da33cd80c74012.1742829770.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 06/13] t5538: add tests to confirm deltas in shallow pushes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-02T23:34:20Z","receivedAt":"2025-05-02T23:34:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 24, 2025 at 03:22:42PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <stolee@gmail.com>\n>\n> It can be notoriously difficult to detect if delta bases are being\n> computed properly during 'git push'. Construct an example where it will\n> make a kilobyte worth of difference when a delta base is not found. We\n> can then use the progress indicators to distinguish between bytes and\n> KiB depending on whether the delta base is found and used.\n>\n> Signed-off-by: Derrick Stolee <stolee@gmail.com>\n> ---\n>  t/t5538-push-shallow.sh | 34 ++++++++++++++++++++++++++++++++++\n>  1 file changed, 34 insertions(+)\n>\n> diff --git a/t/t5538-push-shallow.sh b/t/t5538-push-shallow.sh\n> index e91fcc173e8..11b85cca9e8 100755\n> --- a/t/t5538-push-shallow.sh\n> +++ b/t/t5538-push-shallow.sh\n> @@ -123,4 +123,38 @@ EOF\n>  \tgit cat-file blob $(echo 1|git hash-object --stdin) >/dev/null\n>  \t)\n>  '\n> +\n> +test_expect_success 'push new commit from shallow clone has correct object count' '\n> +\tgit init origin &&\n> +\ttest_commit -C origin a &&\n> +\ttest_commit -C origin b &&\n> +\n> +\tgit clone --depth=1 \"file://$(pwd)/origin\" client &&\n> +\tgit -C client checkout -b topic &&\n> +\tgit -C client commit --allow-empty -m \"empty\" &&\n> +\tGIT_PROGRESS_DELAY=0 git -C client push --progress origin topic 2>err &&\n> +\ttest_grep \"Enumerating objects: 1, done.\" err\n\nWhy is a full 'git push' necessary here? Could you instead directly\ninvoke pack-objects as git push / send-pack does? That test_grep to\nassert on the size of the pack seems very fragile to me.\n\n> +\n> +test_expect_success 'push new commit from shallow clone has good deltas' '\n> +\tgit init base &&\n> +\ttest_seq 1 999 >base/a &&\n> +\ttest_commit -C base initial &&\n> +\tgit -C base add a &&\n> +\tgit -C base commit -m \"big a\" &&\n\nI don't think it really matters, but you may want to write a test_tick\nhere before committing.\n\n> +\n> +\tgit clone --depth=1 \"file://$(pwd)/base\" deltas &&\n> +\tgit -C deltas checkout -b deltas &&\n> +\ttest_seq 1 1000 >deltas/a &&\n> +\tgit -C deltas commit -a -m \"bigger a\" &&\n> +\tGIT_TRACE2_PERF=\"$(pwd)/trace.txt\" \\\n> +\tGIT_PROGRESS_DELAY=0 git -C deltas push --progress origin deltas 2>err &&\n\nSame note here.\n\n> +\n> +\ttest_grep \"Enumerating objects: 5, done\" err &&\n> +\n> +\t# If the delta base is found, then this message uses \"bytes\".\n> +\t# If the delta base is not found, then this message uses \"KiB\".\n> +\ttest_grep \"Writing objects: .* bytes\" err\n\nIf we had the raw pack that was generated, could you use verify-pack\ninstead to check that the desired delta/base relationship was\ndiscovered?\n\nThanks,\nTaylor\n"},{"id":"517144","messageId":"aBVXbSPYxhajUI8A@nand.local","threadId":"63083","inReplyTo":"11767e7653e5212622ed7f82cad3b1a118621e8c.1742829770.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 07/13] repack: add --path-walk option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-02T23:38:21Z","receivedAt":"2025-05-02T23:38:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 24, 2025 at 03:22:43PM +0000, Derrick Stolee via GitGitGadget wrote:\n> diff --git a/Documentation/git-repack.adoc b/Documentation/git-repack.adoc\n> index 5852a5c9736..2199eb3b94f 100644\n> --- a/Documentation/git-repack.adoc\n> +++ b/Documentation/git-repack.adoc\n> @@ -11,7 +11,7 @@ SYNOPSIS\n>  [verse]\n>  'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n>  \t[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n> -\t[--write-midx] [--name-hash-version=<n>]\n> +\t[--write-midx] [--name-hash-version=<n>] [--path-walk]\n>\n>  DESCRIPTION\n>  -----------\n> @@ -255,6 +255,18 @@ linkgit:git-multi-pack-index[1]).\n>  \tProvide this argument to the underlying `git pack-objects` process.\n>  \tSee linkgit:git-pack-objects[1] for full details.\n>\n> +--path-walk::\n> +\tThis option passes the `--path-walk` option to the underlying\n> +\t`git pack-options` process (see linkgit:git-pack-objects[1]).\n> +\tBy default, `git pack-objects` walks objects in an order that\n> +\tpresents trees and blobs in an order unrelated to the path they\n> +\tappear relative to a commit's root tree. The `--path-walk` option\n> +\tenables a different walking algorithm that organizes trees and\n> +\tblobs by path. This has the potential to improve delta compression\n> +\tespecially in the presence of filenames that cause collisions in\n> +\tGit's default name-hash algorithm. Due to changing how the objects\n> +\tare walked, this option is not compatible with `--delta-islands`\n> +\tor `--filter`.\n\nI was going to make a similar comment here as I did in an earlier commit\nmessage about describing the direct effect of a command-line flag\ninstead of describing it in contrast to the default behavior. But I\nthink here that I would recommend instead abbreviating the description\nto just:\n\n    --path-walk::\n    Pass the `--path-walk` option to `git pack-objects`, see\n    linkgit:git-pack-objects[1].\n\n, which is consistent with options like `-l`, `-f`, -F`, etc.\n\nDoes it makes sense to have a separate `--[no-]cruft-path-walk` here to\nlet callers determine separate behavior when repack is tasked with\ngenerating cruft packs?\n\nThanks,\nTaylor\n"},{"id":"517145","messageId":"aBVYa9SzXk7eguIz@nand.local","threadId":"63083","inReplyTo":"dd66a5b46f2ef52eedea323df891a2e24fcd48c1.1742829770.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 08/13] pack-objects: enable --path-walk via config","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-02T23:42:35Z","receivedAt":"2025-05-02T23:42:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 24, 2025 at 03:22:44PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <stolee@gmail.com>\n>\n> Users may want to enable the --path-walk option for 'git pack-objects' by\n> default, especially underneath commands like 'git push' or 'git repack'.\n>\n> This should be limited to client repositories, since the --path-walk option\n> disables bitmap walks, so would be bad to include in Git servers when\n> serving fetches and clones. There is potential that it may be helpful to\n> consider when repacking the repository, to take advantage of improved deltas\n> across historical versions of the same files.\n>\n> Much like how \"pack.useSparse\" was introduced and included in\n> \"feature.experimental\" before being enabled by default, use the repository\n> settings infrastructure to make the new \"pack.usePathWalk\" config enabled by\n> \"feature.experimental\" and \"feature.manyFiles\".\n>\n> Signed-off-by: Derrick Stolee <stolee@gmail.com>\n> ---\n>  Documentation/config/feature.adoc | 4 ++++\n>  Documentation/config/pack.adoc    | 8 ++++++++\n>  builtin/pack-objects.c            | 3 +++\n>  repo-settings.c                   | 3 +++\n>  repo-settings.h                   | 1 +\n>  5 files changed, 19 insertions(+)\n>\n> diff --git a/Documentation/config/feature.adoc b/Documentation/config/feature.adoc\n> index f061b64b748..cb49ff2604a 100644\n> --- a/Documentation/config/feature.adoc\n> +++ b/Documentation/config/feature.adoc\n> @@ -20,6 +20,10 @@ walking fewer objects.\n>  +\n>  * `pack.allowPackReuse=multi` may improve the time it takes to create a pack by\n>  reusing objects from multiple packs instead of just one.\n> ++\n> +* `pack.usePathWalk` may speed up packfile creation and make the packfiles be\n> +significantly smaller in the presence of certain filename collisions with Git's\n> +default name-hash.\n>\n>  feature.manyFiles::\n>  \tEnable config options that optimize for repos with many files in the\n> diff --git a/Documentation/config/pack.adoc b/Documentation/config/pack.adoc\n> index da527377faf..08d06271177 100644\n> --- a/Documentation/config/pack.adoc\n> +++ b/Documentation/config/pack.adoc\n> @@ -155,6 +155,14 @@ pack.useSparse::\n>  \tcommits contain certain types of direct renames. Default is\n>  \t`true`.\n>\n> +pack.usePathWalk::\n> +\tWhen true, git will default to using the '--path-walk' option in\n> +\t'git pack-objects' when the '--revs' option is present. This\n> +\talgorithm groups objects by path to maximize the ability to\n> +\tcompute delta chains across historical versions of the same\n> +\tobject. This may disable other options, such as using bitmaps to\n> +\tenumerate objects.\n> +\n\nSame note here as in the previous patch, I think it's fine to refer\nreaders to the git-pack-objects[1] documentation instead of repeating\nyourself here. (And it removes the risk of these three descriptions\nfalling out of sync with one another).\n\n>  pack.preferBitmapTips::\n>  \tWhen selecting which commits will receive bitmaps, prefer a\n>  \tcommit at the tip of any reference that is a suffix of any value\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index a6b8a78d42a..0ea85754c52 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -4652,6 +4652,9 @@ int cmd_pack_objects(int argc,\n>  \t\tif (use_bitmap_index > 0 ||\n>  \t\t    !use_internal_rev_list)\n>  \t\t\tpath_walk = 0;\n> +\t\telse if (the_repository->gitdir &&\n> +\t\t\t the_repository->settings.pack_use_path_walk)\n> +\t\t\tpath_walk = 1;\n>  \t\telse\n>  \t\t\tpath_walk = git_env_bool(\"GIT_TEST_PACK_PATH_WALK\", 0);\n>  \t}\n\nThe limited diff context makes it hard for me to tell for sure, but this\ntakes place after git_config(), right? If so, I think we can avoid using\nthe repository settings machinery here and just use the config API\ndirectly.\n\n(FWIW, I typically think of repository settings as a way to expose\nconfig information to some part of the codebase that doesn't otherwise\nhave easy access to, e.g., a static field that was set by a git_config()\ncallback).\n\nSeparate from the above: should this have a sanity test to makes sure\nthat we read the config setting correctly?\n\nThanks,\nTaylor\n"},{"id":"517146","messageId":"aBVY3u58R23bI/Uo@nand.local","threadId":"63083","inReplyTo":"aBVLC57bMJKjygyi@nand.local","subject":"Re: [PATCH v2 00/13] PATH WALK II: Add --path-walk option to 'git pack-objects'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-02T23:44:30Z","receivedAt":"2025-05-02T23:44:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, May 02, 2025 at 06:45:31PM -0400, Taylor Blau wrote:\n> On Fri, May 02, 2025 at 02:24:30PM -0700, Junio C Hamano wrote:\n> > > Updates in v2\n> > > =============\n> > >\n> > >  * Re-added a dropped comment when moving code in patch 1.\n> > >  * Updated documentation to include interaction with --use-bitmap-index.\n> > >  * An UNUSED parameter is now used, reducing the use of global variables\n> > >    slightly.\n> >\n> > The iteration saw no comments from anybody, so I (naturally) forgot\n> > about it for quite a long time.  Let me mark it for 'next'.\n>\n> Sorry, this fell off of my to-do list. I don't want to hold things up\n> further, but would appreciate a chance to review this before it hits\n> 'next'.\n>\n> I can take a look now.\n\nOK, I was able to get through the first 8 or so patches in the series,\nand left a handful of comments throughout. I'm running out of time ATM\nto finish reviewing, but I should be able to pick it up next Tuesday (I\nam out of office on Monday).\n\nAgain, I am sorry for the delay here. I know that I am holding things up\nhere, but this genuinely fell off of my radar, and Junio's nudge above\nreminded me. I'll finish reviewing this promptly next week.\n\nThanks,\nTaylor\n"},{"id":"517382","messageId":"052c534a-c2a4-4ff6-9130-4c1e96e1ef72@gmail.com","threadId":"63083","inReplyTo":"aBVTcvhW+lW0GLpQ@nand.local","subject":"Re: [PATCH v2 02/13] pack-objects: add --path-walk option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-05-06T19:39:31Z","receivedAt":"2025-05-06T19:39:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/2/25 7:21 PM, Taylor Blau wrote:\n> On Mon, Mar 24, 2025 at 03:22:38PM +0000, Derrick Stolee via GitGitGadget wrote:\n\n\n>> +--path-walk::\n>> +\tBy default, `git pack-objects` walks objects in an order that\n>> +\tpresents trees and blobs in an order unrelated to the path they\n>> +\tappear relative to a commit's root tree. The `--path-walk` option\n>> +\tenables a different walking algorithm that organizes trees and\n>> +\tblobs by path. This has the potential to improve delta compression\n>> +\tespecially in the presence of filenames that cause collisions in\n>> +\tGit's default name-hash algorithm. Due to changing how the objects\n>> +\tare walked, this option is not compatible with `--delta-islands`,\n>> +\t`--shallow`, or `--filter`. The `--use-bitmap-index` option will\n>> +\tbe ignored in the presence of `--path-walk.`\n>> +\n> \n> This is perhaps a stylistic difference, but I don't think in general it\n> is necessary to say \"by default [...]\" to explain some existing\n> behavior, unless the alternative behavior enabled by the flag is\n> impossible to understand without contrasting.\n> \n> In this particular instance, I think that \"--path-walk\" can be\n> explained in isolation rather than in contrast to the default behavior.\n> \n> I would probably write something like:\n> \n>      --path-walk::\n>      Walk objects in batches grouped by a common path. This can improve\n>      delta compression, especially in the case where filenames cause\n>      collisions in the default name-hash algorithm.\n> \n>      Incompatible with `--delta-islands`, `--shallow`, and `--filter`. If\n>      provided, `--use-bitmap-index` will be ignored.\n\nSure. I'll give this a shot.\n\n>> +static inline int is_oid_interesting(struct repository *repo,\n>> +\t\t\t\t     struct object_id *oid)\n>> +{\n>> +\tstruct object *o = lookup_object(repo, oid);\n> \n> This function and its caller tripped me up a little bit while reading,\n> but I was able to un-confuse myself. Here are some notes that I took\n> while reading, which perhaps you can validate to tell me whether or not\n> I'm on the right track.\n> \n> Here we allow lookup_object() to return NULL, and then immediately\n> return 0 if it does. But...\n> \n>> +\treturn o && !(o->flags & UNINTERESTING);\n>> +}\n>> +\n>> +static int add_objects_by_path(const char *path,\n>> +\t\t\t       struct oid_array *oids,\n>> +\t\t\t       enum object_type type,\n>> +\t\t\t       void *data)\n>> +{\n>> +\tstruct object_entry **delta_list;\n>> +\tsize_t oe_start = to_pack.nr_objects;\n>> +\tsize_t oe_end;\n>> +\tunsigned int sub_list_size;\n> \n> Could you help me understand why sub_list_size is an unsigned int and\n> not a size_t here?\n\nThis is based on a similar \"list_size\" used in ll_find_deltas(), but\nsince we are starting from scratch here it would be better to use\nsize_t.\n\n>> +\tunsigned int *processed = data;\n>> +\n>> +\t/*\n>> +\t * First, add all objects to the packing data, including the ones\n>> +\t * marked UNINTERESTING (translated to 'exclude') as they can be\n>> +\t * used as delta bases.\n>> +\t */\n>> +\tfor (size_t i = 0; i < oids->nr; i++) {\n>> +\t\tint exclude;\n>> +\t\tstruct object_info oi = OBJECT_INFO_INIT;\n>> +\t\tstruct object_id *oid = &oids->oid[i];\n>> +\n>> +\t\t/* Skip objects that do not exist locally. */\n>> +\t\tif (exclude_promisor_objects &&\n>> +\t\t    oid_object_info_extended(the_repository, oid, &oi,\n>> +\t\t\t\t\t     OBJECT_INFO_FOR_PREFETCH) < 0)\n>> +\t\t\tcontinue;\n>> +\n>> +\t\texclude = !is_oid_interesting(the_repository, oid);\n>> +\n>> +\t\tif (exclude && !thin)\n>> +\t\t\tcontinue;\n> \n> ...here we only skip calling add_object_entry() if the object is\n> excluded (which lookup_object() returning NULL would cause above) *and*\n> we're not using thin packs. That makes sense, since if we have a missing\n> object, but we're using thin packs, then it's OK to skip.\n> \n> So I think all of this looks good. However, I find the interface here a\n> little awkward. The function is called \"is_oid_interesting()\", but it\n> determines that fact by checking whether or not the object has the\n> UNINTERSTING bit set on its flag, and then negating the result. But then\n> we negate it again here at the caller to obtain whether or not we should\n> \"exclude\" the object.\n> \n> I was wondering if there were other callers of is_oid_interesting() that\n> don't negate its result. But it doesn't look like there are (at least in\n> this patch). I wonder if it would be cleaner just to inline this instead\n> of having the result negated at multiple levels.\n\n\n\n>> +\t\tadd_object_entry(oid, type, path, exclude);\n>> +\t}\n>> +\n>> +\toe_end = to_pack.nr_objects;\n>> +\n>> +\t/* We can skip delta calculations if it is a no-op. */\n>> +\tif (oe_end == oe_start || !window)\n>> +\t\treturn 0;\n>> +\n>> +\tsub_list_size = 0;\n> \n> I'm nit-picking, but I would imagine that sub_list_nr is probably more\n> conventional, but this is textbook bike-shedding ;-).\n\nSure.\n\n>> +\tALLOC_ARRAY(delta_list, oe_end - oe_start);\n> \n> ALLOC_ARRAY() will barf if oe_start > oe_end and we wrap around the\n> range of size_t, but it might be nice to have a sanity check here just\n> their equality.\n\nI'll look for similar examples.\n\n>> +\t/*\n>> +\t * Find delta bases among this list of objects that all match the same\n>> +\t * path. This causes the delta compression to be interleaved in the\n>> +\t * object walk, which can lead to confusing progress indicators. This is\n>> +\t * also incompatible with threaded delta calculations. In the future,\n>> +\t * consider creating a list of regions in the full to_pack.objects array\n>> +\t * that could be picked up by the threaded delta computation.\n>> +\t */\n>> +\tif (sub_list_size && window) {\n> \n> Do we need to check that window is non-zero here? It looks like that\n> check already happens above.\n\nIf we've already checked, then\n\n>> +\t\tQSORT(delta_list, sub_list_size, type_size_sort);\n>> +\t\tfind_deltas(delta_list, &sub_list_size, window, depth, processed);\n>> +\t}\n>> +\n>> +\tfree(delta_list);\n> \n> Is there a return path through this function that doesn't allocate\n> delta_list? I think the answer is \"no\", but it might be a nice guard\n> against future refactorings to initialize this field to NULL just in\n> case.\n\nOK.\n\n>> +\treturn 0;\n>> +}\n>> +\n>> +static void get_object_list_path_walk(struct rev_info *revs)\n>> +{\n>> +\tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n>> +\tunsigned int processed = 0;\n>> +\n>> +\tinfo.revs = revs;\n>> +\tinfo.path_fn = add_objects_by_path;\n>> +\tinfo.path_fn_data = &processed;\n>> +\trevs->tag_objects = 1;\n> \n> What is the purpose of setting revs->tag_objects to 1 here? Do we need\n> to restore it back to its original value before returning? A comment\n> here would be useful, I think.\n\n\n\n>> @@ -4237,7 +4340,7 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n>>\n>>   \twarn_on_object_refname_ambiguity = save_warning;\n>>\n>> -\tif (use_bitmap_index && !get_object_list_from_bitmap(revs))\n>> +\tif (use_bitmap_index && !path_walk && !get_object_list_from_bitmap(revs))\n> \n> I was going to suggest something like:\n> \n>      if (path_walk && use_bitmap_index) {\n>          warning(_(\"cannot use --use-bitmap-index with --path-walk, ignoring\"));\n>          use_bitmap_index = 0;\n>      }\n> \n> , which would avoid the need for this check here and insulate us against\n> having to make similar changes to future conditionals that care about\n> use_bitmap_index.\n> \n> But it looks like such a check *does* already exist below, so I am not\n> sure I understand why we need to check for path_walk here.\n\nThis is probably cruft from an earlier version of the check. Thanks.\n\n>> +\tif (path_walk && filter_options.choice) {\n>> +\t\twarning(_(\"cannot use --filter with --path-walk\"));\n>> +\t\tpath_walk = 0;\n>> +\t}\n>> +\tif (path_walk && use_delta_islands) {\n>> +\t\twarning(_(\"cannot use delta islands with --path-walk\"));\n>> +\t\tpath_walk = 0;\n>> +\t}\n>> +\tif (path_walk && shallow) {\n>> +\t\twarning(_(\"cannot use --shallow with --path-walk\"));\n>> +\t\tpath_walk = 0;\n>> +\t}\n> \n> Here's a tiny idea I had while reading this code.\n> \n>      if (path_walk) {\n>          const char *option = NULL;\n>          if (filter_options.choice)\n>              option = \"--filter\";\n>          else if (use_delta_islands)\n>              option = \"--delta-islands\";\n>          else if (shallow)\n>              option = \"--shallow\";\n> \n>          if (option) {\n>              warning(_(\"cannot use %s with --path-walk\"), option);\n>              path_walk = 0;\n>          }\n>      }\n> \n> This may be too DRY for your taste, and it does have the disadvantage of\n> reducing the grep-ability of the error messages. I don't have a strong\n> feeling about it one way or another, but I figured I'd at least write it\n> down.\nI like the reduction of work on the translators.\n\nThanks,\n-Stolee\n\n"},{"id":"517383","messageId":"7a919f7e-1e3e-4c30-9f57-5a536dfb3697@gmail.com","threadId":"63083","inReplyTo":"aBVVwCZ0NZT3KLB/@nand.local","subject":"Re: [PATCH v2 05/13] pack-objects: introduce GIT_TEST_PACK_PATH_WALK","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-05-06T19:43:54Z","receivedAt":"2025-05-06T19:43:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/2/25 7:31 PM, Taylor Blau wrote:\n> On Mon, Mar 24, 2025 at 03:22:41PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee <stolee@gmail.com>\n>>\n>> There are many tests that validate whether 'git pack-objects' works as\n>> expected. Instead of duplicating these tests, add a new test environment\n>> variable, GIT_TEST_PACK_PATH_WALK, that implies --path-walk by default\n>> when specified.\n> \n> I normally dislike adding new GIT_TEST_* environment variables, because\n> I don't think we (myself included) are great about remembering to get\n> rid of them. But in this case I think it does make sense to add one.\n> \n>> This was useful in testing the implementation of the --path-walk\n>> implementation, especially in conjunction with test such as:\n> \n> I am not quite following... this sentence sets up a list of tests which\n> I was expecting to show some demonstration of how they uniquely\n> exercised the path-walk feature. But instead it looks like it describes\n> tests that are sensitive to object ordering within packs and thus had to\n> have the new GIT_TEST_PACK_PATH_WALK variable unset/set to zero.\nI guess I mean that having the test variable helped gain confidence in\nthe feature since the only test failures that occurred were due to\ncases where the tests relied too much on specifics of the object walk.\n\nI'll think of better ways to say this.\n\nThanks,\n-Stolee\n\n\n"},{"id":"517384","messageId":"ea342443-2c71-476d-ab45-86cbe711fec7@gmail.com","threadId":"63083","inReplyTo":"aBVYa9SzXk7eguIz@nand.local","subject":"Re: [PATCH v2 08/13] pack-objects: enable --path-walk via config","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-05-06T19:46:43Z","receivedAt":"2025-05-06T19:46:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/2/25 7:42 PM, Taylor Blau wrote:\n> On Mon, Mar 24, 2025 at 03:22:44PM +0000, Derrick Stolee via GitGitGadget wrote:\n\n>> +pack.usePathWalk::\n>> +\tWhen true, git will default to using the '--path-walk' option in\n>> +\t'git pack-objects' when the '--revs' option is present. This\n>> +\talgorithm groups objects by path to maximize the ability to\n>> +\tcompute delta chains across historical versions of the same\n>> +\tobject. This may disable other options, such as using bitmaps to\n>> +\tenumerate objects.\n>> +\n> \n> Same note here as in the previous patch, I think it's fine to refer\n> readers to the git-pack-objects[1] documentation instead of repeating\n> yourself here. (And it removes the risk of these three descriptions\n> falling out of sync with one another).\n\nSure.\n\n>>   pack.preferBitmapTips::\n>>   \tWhen selecting which commits will receive bitmaps, prefer a\n>>   \tcommit at the tip of any reference that is a suffix of any value\n>> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n>> index a6b8a78d42a..0ea85754c52 100644\n>> --- a/builtin/pack-objects.c\n>> +++ b/builtin/pack-objects.c\n>> @@ -4652,6 +4652,9 @@ int cmd_pack_objects(int argc,\n>>   \t\tif (use_bitmap_index > 0 ||\n>>   \t\t    !use_internal_rev_list)\n>>   \t\t\tpath_walk = 0;\n>> +\t\telse if (the_repository->gitdir &&\n>> +\t\t\t the_repository->settings.pack_use_path_walk)\n>> +\t\t\tpath_walk = 1;\n>>   \t\telse\n>>   \t\t\tpath_walk = git_env_bool(\"GIT_TEST_PACK_PATH_WALK\", 0);\n>>   \t}\n> \n> The limited diff context makes it hard for me to tell for sure, but this\n> takes place after git_config(), right? If so, I think we can avoid using\n> the repository settings machinery here and just use the config API\n> directly.\n\nAs I'm paging this thread back into my memory, I internally thought\n\"obviously the test demonstrates that it works...\"\n\n> (FWIW, I typically think of repository settings as a way to expose\n> config information to some part of the codebase that doesn't otherwise\n> have easy access to, e.g., a static field that was set by a git_config()\n> callback).\n> \n> Separate from the above: should this have a sanity test to makes sure\n> that we read the config setting correctly?\n...but of course I neglected to create such a test. Will fix.\n\nThanks,\n-Stolee\n\n"},{"id":"517411","messageId":"aBqwNaF9UNT7VK+1@nand.local","threadId":"63083","inReplyTo":"e5624c379d5804bf4fc44163c14a85ef55f89c97.1742829770.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 09/13] scalar: enable path-walk during push via config","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-07T00:58:29Z","receivedAt":"2025-05-07T00:58:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 24, 2025 at 03:22:45PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <stolee@gmail.com>\n>\n> Repositories registered with Scalar are expected to be client-only\n> repositories that are rather large. This means that they are more likely to\n> be good candidates for using the --path-walk option when running 'git\n> pack-objects', especially under the hood of 'git push'. Enable this config\n> in Scalar repositories.\n\nMakes sense, and feels well-justified.\n\nFrom my understanding of the series thus far, it seems like path-walk is\n(currently) a better default choice for clients than for servers. And\nScalar is a perfect candidate, since its users are likely using it with\nlarge repositories.\n\nThanks,\nTaylor\n"},{"id":"517415","messageId":"aBqz+VfSvecJdWHw@nand.local","threadId":"63083","inReplyTo":"622439d78557d94da899d21444920c27768d3c67.1742829770.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 10/13] pack-objects: refactor path-walk delta phase","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-07T01:14:33Z","receivedAt":"2025-05-07T01:14:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 24, 2025 at 03:22:46PM +0000, Derrick Stolee via GitGitGadget wrote:\n> The current implementation is not integrated with threads, but could be\n> done in a future update.\n\nI think that \"in a future update\" may be worth replacing with \"the\nfollowing commit\", as the former suggests (to me) that it may be perused\noutside of this series.\n\n> Since we do not attempt to sort objects by size until after exploring\n> all trees, we can remove the previous change to t5530 due to a different\n> error message appearing first.\n\nMakes sense.\n\n> Signed-off-by: Derrick Stolee <stolee@gmail.com>\n> ---\n>  builtin/pack-objects.c       | 82 +++++++++++++++++++++++++-----------\n>  pack-objects.h               | 12 ++++++\n>  t/t5300-pack-object.sh       |  8 +++-\n>  t/t5530-upload-pack-error.sh |  6 ---\n>  4 files changed, 75 insertions(+), 33 deletions(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 0ea85754c52..d4e05ca4434 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3236,6 +3236,51 @@ static int should_attempt_deltas(struct object_entry *entry)\n>  \treturn 1;\n>  }\n>\n> +static void find_deltas_for_region(struct object_entry *list,\n> +\t\t\t\t   struct packing_region *region,\n> +\t\t\t\t   unsigned int *processed)\n> +{\n> +\tstruct object_entry **delta_list;\n> +\tuint32_t delta_list_nr = 0;\n> +\n> +\tALLOC_ARRAY(delta_list, region->nr);\n> +\tfor (uint32_t i = 0; i < region->nr; i++) {\n\nI know that these values are all unsigned, and very unlikely to get near\nthe 32-bit maximum, but I do think we should use size_t here (and\nlikewise when dealing with values that deal with how many entries in a\nlist we've allocated and indexes into that list).\n\n> +\t\tstruct object_entry *entry = list + region->start + i;\n> +\t\tif (should_attempt_deltas(entry))\n> +\t\t\tdelta_list[delta_list_nr++] = entry;\n> +\t}\n> +\n> +\tQSORT(delta_list, delta_list_nr, type_size_sort);\n> +\tfind_deltas(delta_list, &delta_list_nr, window, depth, processed);\n> +\tfree(delta_list);\n> +}\n\nThe rest of this function (modulo the inline comment removal, which I\nwrote about below) appear to be a straightforward copy of the previous\nhome of this function.\n\n> +static void find_deltas_by_region(struct object_entry *list,\n> +\t\t\t\t  struct packing_region *regions,\n> +\t\t\t\t  uint32_t start, uint32_t nr)\n\nI imagine that \"start\" here is setup for the following commit which will\nparallelize this task across multiple threads, with each thread starting\nat a different position.\n\nI wonder if (as an alternative) we could get away with passing in a\n(struct packing_region *, size_t) pair and drop \"start\". I think this\ncan work if you pass in the result of adding whatever the value of\n\"start\" would be to to_pack.regions here.\n\nThat somewhat hides the fact that this code is meant to be run across\nmultiple threads, but in a way that I think is worth doing. It lets the\nfunction avoid having to do things like:\n\n  for (size_t i = start; i < start + nr; i++)\n\nand instead do the simpler:\n\n  for (size_t i = 0; i < nr; i++)\n\nsince the caller already adjusted the regions pointer for us. As a\nside-effect, it also means that the call below in prepare_pack() can\navoid passing a literal zero.\n\n> +{\n> +\tunsigned int processed = 0;\n> +\tuint32_t progress_nr;\n\nThis uses a mix of uint32_t and unsigned int types. Should these all be\nthe same (and/or size_t's)?\n\n> -\t/*\n> -\t * Find delta bases among this list of objects that all match the same\n> -\t * path. This causes the delta compression to be interleaved in the\n> -\t * object walk, which can lead to confusing progress indicators. This is\n> -\t * also incompatible with threaded delta calculations. In the future,\n> -\t * consider creating a list of regions in the full to_pack.objects array\n> -\t * that could be picked up by the threaded delta computation.\n> -\t */\n\nNice, it is very satisfying to see this comment go away ;-).\n\n> -\tif (sub_list_size && window) {\n> -\t\tQSORT(delta_list, sub_list_size, type_size_sort);\n> -\t\tfind_deltas(delta_list, &sub_list_size, window, depth, processed);\n> -\t}\n> +\t*processed += oids->nr;\n> +\tdisplay_progress(progress_state, *processed);\n>\n> -\tfree(delta_list);\n>  \treturn 0;\n>  }\n>\n> diff --git a/pack-objects.h b/pack-objects.h\n> index d73e3843c92..7ba9f3448fe 100644\n> --- a/pack-objects.h\n> +++ b/pack-objects.h\n> @@ -119,11 +119,23 @@ struct object_entry {\n>  \tunsigned ext_base:1; /* delta_idx points outside packlist */\n>  };\n>\n> +/**\n\nThis is an extreme nitpick, but I think that \"/**\" (as opposed to \"/*\")\nis preferred, as the former is used for Doxygen-style comments. I feel\nlike I have seen Junio comment on this before, but searching 'f:Junio\n\"/**\"' yields a measly 83,000+ results, so I am unlikely to find it ;-).\n\n(Not worth rerolling on its own, but if you are rerolling anyway, which\nis what I gathered from some of your earlier replies, it may be worth\npicking up along the way.)\n\n> + * A packing region is a section of the packing_data.objects array\n> + * as given by a starting index and a number of elements.\n> + */\n> +struct packing_region {\n> +\tuint32_t start;\n> +\tuint32_t nr;\n> +};\n\nSame note here about the uint32_t versus size_t.\n> +\n>  struct packing_data {\n>  \tstruct repository *repo;\n>  \tstruct object_entry *objects;\n>  \tuint32_t nr_objects, nr_alloc;\n>\n> +\tstruct packing_region *regions;\n> +\tuint64_t nr_regions, nr_regions_alloc;\n\nDitto, but this one jumps to uint64_t. What differentiates these two?\n\nThanks,\nTaylor\n"},{"id":"517423","messageId":"aBq4b2R0pAS5PiZm@nand.local","threadId":"63083","inReplyTo":"ae73d26319ac5db338507988eb23f1ecaac67671.1742829770.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 11/13] pack-objects: thread the path-based compression","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-07T01:33:35Z","receivedAt":"2025-05-07T01:33:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 24, 2025 at 03:22:47PM +0000, Derrick Stolee via GitGitGadget wrote:\n> Using the Git repository as a test repo, the p5313 performance test\n> shows that the resulting size of the repo is the same, but the threaded\n> implementation gives gains of varying degrees depending on the number of\n> objects being packed. (This was tested on a 16-core machine.)\n>\n> Test                        HEAD~1      HEAD\n> ---------------------------------------------------\n> 5313.20: big pack             2.38      1.99 -16.4%\n> 5313.21: big pack size       16.1M     16.0M  -0.2%\n> 5313.24: repack             107.32     45.41 -57.7%\n> 5313.25: repack size        213.3M    213.2M  -0.0%\n\nVery nice indeed!\n\n> This ~60% reduction in 'git repack --path-walk' time is typical across\n> all repos I used for testing. What is interesting is to compare when the\n> overall time improves enough to outperform the --name-hash-version=1\n> case. These time improvements correlate with repositories with data\n> shapes that significantly improve their data size as well. The\n\nI had to read this sentence a couple of times to wrap my head around it,\nbut perhaps you can double-check my understanding. Are you saying that\nreducing the size of the resulting packs suggests that our runtime with\nthe path-walk algorithm is improved enough to be competitive with (or\nbetter than) the non-path-walk implementation with name-hash v1?\n\nI think that's what you mean, and I admit that I don't find my own\nrestatement to be all that much clearer. So I think it's fine to leave\nthis as-is, though it is interesting to think about why the statement\nis true.\n\nI suspect (without having tested it thoroughly) that a significant\nproportion of the time saved on the path-walk side is because we found\nbetter deltas that are easier to compress, as a result of a more\nwell-tuned search technique. I don't think you necessarily have to get\ntoo far into the details here, but I was curious (not for commit message\npolishing, but purely for my own curiosity) if you had any thoughts or\nmeasurements of why this might be the case.\n\nBut...\n\n> --path-walk feature frequently takes longer than --name-hash-verison=2,\n> trading some extrac computation for some additional compression. The\n\ns/extrac/extra/\n\n> natural place where this additional computation comes from is the two\n> compression passes that --path-walk takes, though the first pass is\n> naturally faster due to the path boundaries avoiding a number of delta\n> compression attempts.\n\n... I should have kept reading before commenting the above! ;-)\n\n> Signed-off-by: Derrick Stolee <stolee@gmail.com>\n> ---\n>  builtin/pack-objects.c | 163 ++++++++++++++++++++++++++++++++++++++++-\n>  1 file changed, 161 insertions(+), 2 deletions(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index d4e05ca4434..2a6246c1e78 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -2964,6 +2964,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n>  struct thread_params {\n>  \tpthread_t thread;\n>  \tstruct object_entry **list;\n> +\tstruct packing_region *regions;\n>  \tunsigned list_size;\n>  \tunsigned remaining;\n>  \tint window;\n> @@ -3281,6 +3282,164 @@ static void find_deltas_by_region(struct object_entry *list,\n>  \tstop_progress(&progress_state);\n>  }\n>\n> +static void *threaded_find_deltas_by_path(void *arg)\n> +{\n> +\tstruct thread_params *me = arg;\n> +\n> +\tprogress_lock();\n> +\twhile (me->remaining) {\n> +\t\twhile (me->remaining) {\n\nI am not quite following the double-while loop here. Could you help\nclarify what's going on here? Is this due to the work-stealing below?\n\n> +static void ll_find_deltas_by_region(struct object_entry *list,\n> +\t\t\t\t     struct packing_region *regions,\n> +\t\t\t\t     uint32_t start, uint32_t nr)\n> +{\n> +\tstruct thread_params *p;\n> +\tint i, ret, active_threads = 0;\n> +\tunsigned int processed = 0;\n> +\tuint32_t progress_nr;\n> +\tinit_threaded_search();\n> +\n> +\tif (!nr)\n> +\t\treturn;\n> +\n> +\tprogress_nr =  regions[nr - 1].start + regions[nr - 1].nr;\n> +\tif (delta_search_threads <= 1) {\n> +\t\tfind_deltas_by_region(list, regions, start, nr);\n> +\t\tcleanup_threaded_search();\n> +\t\treturn;\n> +\t}\n> +\n> +\tif (progress > pack_to_stdout)\n> +\t\tfprintf_ln(stderr, _(\"Path-based delta compression using up to %d threads\"),\n> +\t\t\t   delta_search_threads);\n\nThis looks a copy-and-paste from the non-path-walk code, but I wonder if\nit might be worth using Q_() here instead of _() to provide better\noutput in the case that delta_search_threads is 1.\n\nThe rest of the implementation of ll_find_deltas_by_region() looks good\nto me.\n\nThanks,\nTaylor\n"},{"id":"517424","messageId":"aBq48Dog53bn5bJb@nand.local","threadId":"63083","inReplyTo":"aBVY3u58R23bI/Uo@nand.local","subject":"Re: [PATCH v2 00/13] PATH WALK II: Add --path-walk option to 'git pack-objects'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-07T01:35:44Z","receivedAt":"2025-05-07T01:35:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, May 02, 2025 at 07:44:30PM -0400, Taylor Blau wrote:\n> OK, I was able to get through the first 8 or so patches in the series,\n> and left a handful of comments throughout. I'm running out of time ATM\n> to finish reviewing, but I should be able to pick it up next Tuesday (I\n> am out of office on Monday).\n\nAll set, and thanks again to Stolee for their patience. I think I got\nthrough everything I wanted to, and I think in total there is enough to\nmake a reroll worthwhile here.\n\nThanks,\nTaylor\n"},{"id":"518253","messageId":"6087094c-2639-4262-8026-89e4d8aa8c8f@gmail.com","threadId":"63083","inReplyTo":"052c534a-c2a4-4ff6-9130-4c1e96e1ef72@gmail.com","subject":"Re: [PATCH v2 02/13] pack-objects: add --path-walk option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-05-16T15:27:17Z","receivedAt":"2025-05-16T15:27:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/6/25 3:39 PM, Derrick Stolee wrote:\n> On 5/2/25 7:21 PM, Taylor Blau wrote:\n>> On Mon, Mar 24, 2025 at 03:22:38PM +0000, Derrick Stolee via GitGitGadget wrote:\n\n>>> +static int add_objects_by_path(const char *path,\n>>> +                   struct oid_array *oids,\n>>> +                   enum object_type type,\n>>> +                   void *data)\n>>> +{\n>>> +    struct object_entry **delta_list;\n>>> +    size_t oe_start = to_pack.nr_objects;\n>>> +    size_t oe_end;\n>>> +    unsigned int sub_list_size;\n>>\n>> Could you help me understand why sub_list_size is an unsigned int and\n>> not a size_t here?\n> \n> This is based on a similar \"list_size\" used in ll_find_deltas(), but\n> since we are starting from scratch here it would be better to use\n> size_t.\n\nSorry for the delay, but the find_deltas() method called in both\nplaces expects unsigned ints, so we are stuck with this until we\ndecide to upgrade it in both places.\n\n\n>>> +    unsigned int *processed = data;\n>>> +\n>>> +    /*\n>>> +     * First, add all objects to the packing data, including the ones\n>>> +     * marked UNINTERESTING (translated to 'exclude') as they can be\n>>> +     * used as delta bases.\n>>> +     */\n>>> +    for (size_t i = 0; i < oids->nr; i++) {\n>>> +        int exclude;\n>>> +        struct object_info oi = OBJECT_INFO_INIT;\n>>> +        struct object_id *oid = &oids->oid[i];\n>>> +\n>>> +        /* Skip objects that do not exist locally. */\n>>> +        if (exclude_promisor_objects &&\n>>> +            oid_object_info_extended(the_repository, oid, &oi,\n>>> +                         OBJECT_INFO_FOR_PREFETCH) < 0)\n>>> +            continue;\n>>> +\n>>> +        exclude = !is_oid_interesting(the_repository, oid);\n>>> +\n>>> +        if (exclude && !thin)\n>>> +            continue;\n>>\n>> ...here we only skip calling add_object_entry() if the object is\n>> excluded (which lookup_object() returning NULL would cause above) *and*\n>> we're not using thin packs. That makes sense, since if we have a missing\n>> object, but we're using thin packs, then it's OK to skip.\n>>\n>> So I think all of this looks good. However, I find the interface here a\n>> little awkward. The function is called \"is_oid_interesting()\", but it\n>> determines that fact by checking whether or not the object has the\n>> UNINTERSTING bit set on its flag, and then negating the result. But then\n>> we negate it again here at the caller to obtain whether or not we should\n>> \"exclude\" the object.\n>>\n>> I was wondering if there were other callers of is_oid_interesting() that\n>> don't negate its result. But it doesn't look like there are (at least in\n>> this patch). I wonder if it would be cleaner just to inline this instead\n>> of having the result negated at multiple levels.\n\nDouble-checked now and there seem to be no other callers, so this will be\nrenamed to the opposite case.\n\n>>> +    return 0;\n>>> +}\n>>> +\n>>> +static void get_object_list_path_walk(struct rev_info *revs)\n>>> +{\n>>> +    struct path_walk_info info = PATH_WALK_INFO_INIT;\n>>> +    unsigned int processed = 0;\n>>> +\n>>> +    info.revs = revs;\n>>> +    info.path_fn = add_objects_by_path;\n>>> +    info.path_fn_data = &processed;\n>>> +    revs->tag_objects = 1;\n>>\n>> What is the purpose of setting revs->tag_objects to 1 here? Do we need\n>> to restore it back to its original value before returning? A comment\n>> here would be useful, I think.\n\nI think this is not necessary and should be removed.\n\nThanks,\n-Stolee\n\n"},{"id":"518255","messageId":"f0f1ebed-06af-4d73-9068-a08956f01e48@gmail.com","threadId":"63083","inReplyTo":"aBVWfJUFNdfwS0PO@nand.local","subject":"Re: [PATCH v2 06/13] t5538: add tests to confirm deltas in shallow pushes","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-05-16T15:32:42Z","receivedAt":"2025-05-16T15:32:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/2/25 7:34 PM, Taylor Blau wrote:\n> On Mon, Mar 24, 2025 at 03:22:42PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee <stolee@gmail.com>\n>>\n>> It can be notoriously difficult to detect if delta bases are being\n>> computed properly during 'git push'. Construct an example where it will\n>> make a kilobyte worth of difference when a delta base is not found. We\n>> can then use the progress indicators to distinguish between bytes and\n>> KiB depending on whether the delta base is found and used.\n>>\n>> Signed-off-by: Derrick Stolee <stolee@gmail.com>\n>> ---\n>>   t/t5538-push-shallow.sh | 34 ++++++++++++++++++++++++++++++++++\n>>   1 file changed, 34 insertions(+)\n>>\n>> diff --git a/t/t5538-push-shallow.sh b/t/t5538-push-shallow.sh\n>> index e91fcc173e8..11b85cca9e8 100755\n>> --- a/t/t5538-push-shallow.sh\n>> +++ b/t/t5538-push-shallow.sh\n>> @@ -123,4 +123,38 @@ EOF\n>>   \tgit cat-file blob $(echo 1|git hash-object --stdin) >/dev/null\n>>   \t)\n>>   '\n>> +\n>> +test_expect_success 'push new commit from shallow clone has correct object count' '\n>> +\tgit init origin &&\n>> +\ttest_commit -C origin a &&\n>> +\ttest_commit -C origin b &&\n>> +\n>> +\tgit clone --depth=1 \"file://$(pwd)/origin\" client &&\n>> +\tgit -C client checkout -b topic &&\n>> +\tgit -C client commit --allow-empty -m \"empty\" &&\n>> +\tGIT_PROGRESS_DELAY=0 git -C client push --progress origin topic 2>err &&\n>> +\ttest_grep \"Enumerating objects: 1, done.\" err\n> \n> Why is a full 'git push' necessary here? Could you instead directly\n> invoke pack-objects as git push / send-pack does? That test_grep to\n> assert on the size of the pack seems very fragile to me.\n\nHere, I think we _need_ to test at the 'git push' level, since that is the\nuser behavior we want to protect. Earlier testing using 'git pack-objects'\ndirectly had not discovered this subtle issue in how it was used in the\nshallow case.\n\nIt does make the test more fragile, but it's testing \"the right thing\"\ninstead of locking in an internal detail.\n\n>> +\n>> +test_expect_success 'push new commit from shallow clone has good deltas' '\n>> +\tgit init base &&\n>> +\ttest_seq 1 999 >base/a &&\n>> +\ttest_commit -C base initial &&\n>> +\tgit -C base add a &&\n>> +\tgit -C base commit -m \"big a\" &&\n> \n> I don't think it really matters, but you may want to write a test_tick\n> here before committing.\n\nI don't understand why. Is the point to get a different commit time?\n\n>> +\ttest_grep \"Enumerating objects: 5, done\" err &&\n>> +\n>> +\t# If the delta base is found, then this message uses \"bytes\".\n>> +\t# If the delta base is not found, then this message uses \"KiB\".\n>> +\ttest_grep \"Writing objects: .* bytes\" err\n> \n> If we had the raw pack that was generated, could you use verify-pack\n> instead to check that the desired delta/base relationship was\n> discovered?\n\nTrue, we could do that with a lot of effort, but what we really care\nabout is \"the push was small\" not \"this specific delta base was\npicked\" (though, the delta base is how we got a small packfile).\n\nThanks,\n-Stolee\n\n"},{"id":"518256","messageId":"c849a4e1-b409-4a2e-ae32-b72f53137133@gmail.com","threadId":"63083","inReplyTo":"ea342443-2c71-476d-ab45-86cbe711fec7@gmail.com","subject":"Re: [PATCH v2 08/13] pack-objects: enable --path-walk via config","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-05-16T15:41:57Z","receivedAt":"2025-05-16T15:41:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/6/25 3:46 PM, Derrick Stolee wrote:\n> On 5/2/25 7:42 PM, Taylor Blau wrote:\n>>>   pack.preferBitmapTips::\n>>>       When selecting which commits will receive bitmaps, prefer a\n>>>       commit at the tip of any reference that is a suffix of any value\n>>> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n>>> index a6b8a78d42a..0ea85754c52 100644\n>>> --- a/builtin/pack-objects.c\n>>> +++ b/builtin/pack-objects.c\n>>> @@ -4652,6 +4652,9 @@ int cmd_pack_objects(int argc,\n>>>           if (use_bitmap_index > 0 ||\n>>>               !use_internal_rev_list)\n>>>               path_walk = 0;\n>>> +        else if (the_repository->gitdir &&\n>>> +             the_repository->settings.pack_use_path_walk)\n>>> +            path_walk = 1;\n>>>           else\n>>>               path_walk = git_env_bool(\"GIT_TEST_PACK_PATH_WALK\", 0);\n>>>       }\n>>\n>> The limited diff context makes it hard for me to tell for sure, but this\n>> takes place after git_config(), right? If so, I think we can avoid using\n>> the repository settings machinery here and just use the config API\n>> directly.\n>>\n>> (FWIW, I typically think of repository settings as a way to expose\n>> config information to some part of the codebase that doesn't otherwise\n>> have easy access to, e.g., a static field that was set by a git_config()\n>> callback).\n\nRereading these comments, I didn't adequately reply to \"why use the\nrepo settings?\" and the reason comes due to the use of assigning the\nvalue in feature.experimental=true. This provides a common place for\nthe logic around both path.usePathWalk and feature.experimental.\n\nA similar behavior is already present for pack.useSparse, which was\nin feature.experimental for a while before it became enabled by\ndefault.\n\nThanks,\n-Stolee\n\n"},{"id":"518263","messageId":"7ee3884e-b629-4b59-8e9a-a9a56abce786@gmail.com","threadId":"63083","inReplyTo":"aBqz+VfSvecJdWHw@nand.local","subject":"Re: [PATCH v2 10/13] pack-objects: refactor path-walk delta phase","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-05-16T16:27:29Z","receivedAt":"2025-05-16T16:27:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/6/25 9:14 PM, Taylor Blau wrote:\n> On Mon, Mar 24, 2025 at 03:22:46PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> The current implementation is not integrated with threads, but could be\n>> done in a future update.\n> \n> I think that \"in a future update\" may be worth replacing with \"the\n> following commit\", as the former suggests (to me) that it may be perused\n> outside of this series.\n\nNoted and fixed.\n\n\n>> +static void find_deltas_for_region(struct object_entry *list,\n>> +\t\t\t\t   struct packing_region *region,\n>> +\t\t\t\t   unsigned int *processed)\n>> +{\n>> +\tstruct object_entry **delta_list;\n>> +\tuint32_t delta_list_nr = 0;\n>> +\n>> +\tALLOC_ARRAY(delta_list, region->nr);\n>> +\tfor (uint32_t i = 0; i < region->nr; i++) {\n> \n> I know that these values are all unsigned, and very unlikely to get near\n> the 32-bit maximum, but I do think we should use size_t here (and\n> likewise when dealing with values that deal with how many entries in a\n> list we've allocated and indexes into that list).\n\nI've updated things to size_t where I could across this patch. Thanks.\n\n>> +static void find_deltas_by_region(struct object_entry *list,\n>> +\t\t\t\t  struct packing_region *regions,\n>> +\t\t\t\t  uint32_t start, uint32_t nr)\n> \n> I imagine that \"start\" here is setup for the following commit which will\n> parallelize this task across multiple threads, with each thread starting\n> at a different position.\n> \n> I wonder if (as an alternative) we could get away with passing in a\n> (struct packing_region *, size_t) pair and drop \"start\". I think this\n> can work if you pass in the result of adding whatever the value of\n> \"start\" would be to to_pack.regions here.\n> \n> That somewhat hides the fact that this code is meant to be run across\n> multiple threads, but in a way that I think is worth doing. It lets the\n> function avoid having to do things like:\n> \n>    for (size_t i = start; i < start + nr; i++)\n> \n> and instead do the simpler:\n> \n>    for (size_t i = 0; i < nr; i++)\n> \n> since the caller already adjusted the regions pointer for us. As a\n> side-effect, it also means that the call below in prepare_pack() can\n> avoid passing a literal zero.\n\nI generally avoid this kind of pattern, because it makes it more\ndifficult to understand what is going on when debugging. The cost\nof adding 'start' to the index seems low relative to having a\nconsistent indexing scheme when accessing from the \"global\" list\nof items.\n\nThis only gets more complicated when the threaded version iterates\non a \"consecutive sublist of regions\".\n\n>> +/**\n> \n> This is an extreme nitpick, but I think that \"/**\" (as opposed to \"/*\")\n> is preferred, as the former is used for Doxygen-style comments. I feel\n> like I have seen Junio comment on this before, but searching 'f:Junio\n> \"/**\"' yields a measly 83,000+ results, so I am unlikely to find it ;-).\n> \n> (Not worth rerolling on its own, but if you are rerolling anyway, which\n> is what I gathered from some of your earlier replies, it may be worth\n> picking up along the way.)\n\nI'm confused. Are you saying I should change from \"/**\" to \"/*\" (which\nis the opposite of your preferred statement) or did you misread that I\nhad used \"/**\"? I can't determine which statement is the intended one.\n  Thanks,\n-Stolee\n\n"},{"id":"518276","messageId":"pull.1819.v3.git.1747419124.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v2.git.1742829769.gitgitgadget@gmail.com","subject":"[PATCH v3 00/13] PATH WALK II: Add --path-walk option to 'git pack-objects'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-05-16T18:11:50Z","receivedAt":"2025-05-16T18:12:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Here is a full submission of the --path-walk feature for 'git pack-objects'\nand 'git repack'. It's been discussed in an RFC [1], as a future application\nfor the path walk API [2], and is updated now that --name-hash-version=2\nexists (as a replacement for the --full-name-hash option from the RFC) [3].\n\n[1]\nhttps://lore.kernel.org/git/pull.1813.v2.git.1729431810.gitgitgadget@gmail.com/\n\n[2]\nhttps://lore.kernel.org/git/pull.1818.git.1730356023.gitgitgadget@gmail.com\n\n[3]\nhttps://lore.kernel.org/git/pull.1813.git.1728396723.gitgitgadget@gmail.com\n\nThis patch series does the following:\n\n 1. Add a new '--path-walk' option to 'git pack-objects' that uses the\n    path-walk API instead of the revision API to collect objects for delta\n    compression.\n\n 2. Add a new '--path-walk' option to 'git repack' to pass this option along\n    to 'git pack-objects'.\n\n 3. Add a new 'pack.usePathWalk' config option to opt into this option\n    implicitly, such as in 'git push'.\n\n 4. Optimize the '--path-walk' option using threading so it better competes\n    with the existing multi-threaded delta compression mechanism.\n\n 5. Update the path-walk API with a new 'edge_aggressive' option that pairs\n    close to the --edge-aggressive option in the revision API. This is\n    useful when creating thin packs inside shallow clones.\n\nThis feature works by using the path-walk API to emit groups of objects that\nappear at the same path. These groups are tracked so they can be tested for\ndelta compression with each other, and then after those groups are tested a\nsecond pass using the name-hash attempts to find better (or first time)\ndeltas across path boundaries. This second pass is much faster than a fresh\npass since the existing deltas are used as a limit for the size of\npotentially new deltas, short-circuiting the checks when the delta size\nexceeds the current-best.\n\nThe benefits of the --path-walk feature first come into play when the name\nhash functions have many collisions, so sorting by name hash value leads to\nunhelpful groupings of objects. Many of these benefits are improved by\n--name-hash-version=2, but collisions still exist with any hash-based\napproach. There are also performance benefits in some cases due to the\nisolation of delta compression testing within path groups.\n\nAll of the benefits of the --path-walk feature are less dramatic when\ncompared to --name-hash-version=2, but they can still exist in many cases. I\nhave also seen some cases where --name-hash-version=2 compresses better than\n--path-walk with --name-hash-version=1, but these options can be combined to\nget the best of both worlds.\n\nDetailed statistics are provided within patch messages, but a few are\nhighlighted here:\n\nThe microsoft/fluentui is a public Javascript repo that suffers from many of\nthe name hash collisions as internal repositories I've worked with. Here is\na comparison of the compressed size and end-to-end time of the repack:\n\nRepack Method    Pack Size       Time\n---------------------------------------\nHash v1             439.4M      87.24s\nHash v2             161.7M      21.51s\nPath Walk           142.5M      28.16s\n\n\nLess dramatic, but perhaps more standardly structured is the nodejs/node\nrepository, with these stats:\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                739.9M      71.18s\nHash v2                764.6M      67.82s\nPath Walk              698.0M      75.10s\n\n\nEven the Linux kernel repository gains some benefits, even though the number\nof hash collisions is relatively low due to a preference for short\nfilenames:\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                  2.5G     554.41s\nHash v2                  2.5G     549.62s\nPath Walk                2.2G     559.00s\n\n\nThe drawbacks of the --path-walk feature is that it will be harder to\nintegrate it with bitmap features, specifically delta islands. This is not\ninsurmountable, but would require more work, such as a revision walk to\npaint objects with reachability information before using that during delta\ncomputations.\n\nHowever, there should still be significant benefits to Git clients trying to\nsave space and improve local performance.\n\nThis feature was shipped with similar features in microsoft/git as of\nv2.47.0.vfs.0.3 [4]. This was used in CI machines for an internal monorepo\nthat had significant repository growth due to constructing a batch of\nbeachball [5] CHANGELOG.[md|json] files and pushing them to a release\nbranch. These pushes were frequently 70-200 MB due to poor delta\ncompression. Using the 'pack.usePathWalk=true' config, these pushes dropped\nin size by 100x while improving performance. Since these CI machines were\nworking with a shallow clone, the 'edge_aggressive' changes were required to\nenable the path-walk option.\n\n[4] https://github.com/microsoft/git/releases/tag/v2.47.0.vfs.0.3\n\n[5] https://github.com/microsoft/beachball\n\n\nUpdates in v2\n=============\n\n * Re-added a dropped comment when moving code in patch 1.\n * Updated documentation to include interaction with --use-bitmap-index.\n * An UNUSED parameter is now used, reducing the use of global variables\n   slightly.\n\n\nUpdates in v3\n=============\n\nThanks for the review, Taylor. Sorry for my delay in getting back to your\nfeedback.\n\n * Documentation has been edited slightly for simplicity.\n * is_oid_interesting() was swapped to is_oid_uninteresting()\n * sub_list_size renamed to sub_list_nr\n * Several uint32_t and uint64_t variables were converted to size_t.\n * Several 'unsigned int' variables were required to stay as-is, for now,\n   until a refactor can be done.\n * An unnecessary update of tag_objects was removed.\n * The logic and error message around incompatible options is simpler.\n * Tests are expanded, especially around config options.\n * Fixed commit message typos.\n * Extra care around ALLOC_ARRAY() to avoid a zero- or negative-length\n   array.\n\nThanks, -Stolee\n\nDerrick Stolee (13):\n  pack-objects: extract should_attempt_deltas()\n  pack-objects: add --path-walk option\n  pack-objects: update usage to match docs\n  p5313: add performance tests for --path-walk\n  pack-objects: introduce GIT_TEST_PACK_PATH_WALK\n  t5538: add tests to confirm deltas in shallow pushes\n  repack: add --path-walk option\n  pack-objects: enable --path-walk via config\n  scalar: enable path-walk during push via config\n  pack-objects: refactor path-walk delta phase\n  pack-objects: thread the path-based compression\n  path-walk: add new 'edge_aggressive' option\n  pack-objects: allow --shallow and --path-walk\n\n Documentation/config/feature.adoc          |   4 +\n Documentation/config/pack.adoc             |   4 +\n Documentation/git-pack-objects.adoc        |  25 +-\n Documentation/git-repack.adoc              |   5 +-\n Documentation/technical/api-path-walk.adoc |   9 +\n builtin/pack-objects.c                     | 425 +++++++++++++++++++--\n builtin/repack.c                           |   7 +-\n pack-objects.h                             |  12 +\n path-walk.c                                |   6 +-\n path-walk.h                                |   7 +\n repo-settings.c                            |   3 +\n repo-settings.h                            |   1 +\n scalar.c                                   |   1 +\n t/README                                   |   4 +\n t/helper/test-path-walk.c                  |   2 +\n t/perf/p5313-pack-objects.sh               |  37 +-\n t/t0411-clone-from-partial.sh              |   6 +\n t/t0450/adoc-help-mismatches               |   1 -\n t/t5300-pack-object.sh                     |  19 +\n t/t5306-pack-nobase.sh                     |   5 +\n t/t5310-pack-bitmaps.sh                    |  13 +-\n t/t5316-pack-delta-depth.sh                |   9 +-\n t/t5332-multi-pack-reuse.sh                |   7 +\n t/t5516-fetch-push.sh                      |  10 +\n t/t5538-push-shallow.sh                    |  41 ++\n t/t6601-path-walk.sh                       |  20 +\n t/t7406-submodule-update.sh                |   3 +\n 27 files changed, 620 insertions(+), 66 deletions(-)\n\n\nbase-commit: a36e024e989f4d35f35987a60e3af8022cac3420\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1819%2Fderrickstolee%2Fpath-walk-upstream-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1819/derrickstolee/path-walk-upstream-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1819\n\nRange-diff vs v2:\n\n  1:  57c1cc20de0 =  1:  57c1cc20de0 pack-objects: extract should_attempt_deltas()\n  2:  a271d6245c2 !  2:  e3ea17e326e pack-objects: add --path-walk option\n     @@ Documentation/git-pack-objects.adoc: many different directories. At the moment,\n       will be automatically changed to version `1`.\n       \n      +--path-walk::\n     -+\tBy default, `git pack-objects` walks objects in an order that\n     -+\tpresents trees and blobs in an order unrelated to the path they\n     -+\tappear relative to a commit's root tree. The `--path-walk` option\n     -+\tenables a different walking algorithm that organizes trees and\n     -+\tblobs by path. This has the potential to improve delta compression\n     -+\tespecially in the presence of filenames that cause collisions in\n     -+\tGit's default name-hash algorithm. Due to changing how the objects\n     -+\tare walked, this option is not compatible with `--delta-islands`,\n     -+\t`--shallow`, or `--filter`. The `--use-bitmap-index` option will\n     -+\tbe ignored in the presence of `--path-walk.`\n     ++\tPerform compression by first organizing objects by path, then a\n     ++\tsecond pass that compresses across paths as normal. This has the\n     ++\tpotential to improve delta compression especially in the presence\n     ++\tof filenames that cause collisions in Git's default name-hash\n     ++\talgorithm.\n     +++\n     ++Incompatible with `--delta-islands`, `--shallow`, or `--filter`. The\n     ++`--use-bitmap-index` option will be ignored in the presence of\n     ++`--path-walk.`\n      +\n       \n       DELTA ISLANDS\n     @@ builtin/pack-objects.c: static void mark_bitmap_preferred_tips(void)\n       \t}\n       }\n       \n     -+static inline int is_oid_interesting(struct repository *repo,\n     -+\t\t\t\t     struct object_id *oid)\n     ++static inline int is_oid_uninteresting(struct repository *repo,\n     ++\t\t\t\t       struct object_id *oid)\n      +{\n      +\tstruct object *o = lookup_object(repo, oid);\n     -+\treturn o && !(o->flags & UNINTERESTING);\n     ++\treturn !o || (o->flags & UNINTERESTING);\n      +}\n      +\n      +static int add_objects_by_path(const char *path,\n     @@ builtin/pack-objects.c: static void mark_bitmap_preferred_tips(void)\n      +\t\t\t       enum object_type type,\n      +\t\t\t       void *data)\n      +{\n     -+\tstruct object_entry **delta_list;\n     ++\tstruct object_entry **delta_list = NULL;\n      +\tsize_t oe_start = to_pack.nr_objects;\n      +\tsize_t oe_end;\n     -+\tunsigned int sub_list_size;\n     ++\tunsigned int sub_list_nr;\n      +\tunsigned int *processed = data;\n      +\n      +\t/*\n     @@ builtin/pack-objects.c: static void mark_bitmap_preferred_tips(void)\n      +\t\t\t\t\t     OBJECT_INFO_FOR_PREFETCH) < 0)\n      +\t\t\tcontinue;\n      +\n     -+\t\texclude = !is_oid_interesting(the_repository, oid);\n     ++\t\texclude = is_oid_uninteresting(the_repository, oid);\n      +\n      +\t\tif (exclude && !thin)\n      +\t\t\tcontinue;\n     @@ builtin/pack-objects.c: static void mark_bitmap_preferred_tips(void)\n      +\tif (oe_end == oe_start || !window)\n      +\t\treturn 0;\n      +\n     -+\tsub_list_size = 0;\n     -+\tALLOC_ARRAY(delta_list, oe_end - oe_start);\n     ++\tsub_list_nr = 0;\n     ++\tif (oe_end > oe_start)\n     ++\t\tALLOC_ARRAY(delta_list, oe_end - oe_start);\n      +\n      +\tfor (size_t i = 0; i < oe_end - oe_start; i++) {\n      +\t\tstruct object_entry *entry = to_pack.objects + oe_start + i;\n     @@ builtin/pack-objects.c: static void mark_bitmap_preferred_tips(void)\n      +\t\tif (!should_attempt_deltas(entry))\n      +\t\t\tcontinue;\n      +\n     -+\t\tdelta_list[sub_list_size++] = entry;\n     ++\t\tdelta_list[sub_list_nr++] = entry;\n      +\t}\n      +\n      +\t/*\n     @@ builtin/pack-objects.c: static void mark_bitmap_preferred_tips(void)\n      +\t * consider creating a list of regions in the full to_pack.objects array\n      +\t * that could be picked up by the threaded delta computation.\n      +\t */\n     -+\tif (sub_list_size && window) {\n     -+\t\tQSORT(delta_list, sub_list_size, type_size_sort);\n     -+\t\tfind_deltas(delta_list, &sub_list_size, window, depth, processed);\n     ++\tif (sub_list_nr && window) {\n     ++\t\tQSORT(delta_list, sub_list_nr, type_size_sort);\n     ++\t\tfind_deltas(delta_list, &sub_list_nr, window, depth, processed);\n      +\t}\n      +\n      +\tfree(delta_list);\n     @@ builtin/pack-objects.c: static void mark_bitmap_preferred_tips(void)\n      +\tinfo.revs = revs;\n      +\tinfo.path_fn = add_objects_by_path;\n      +\tinfo.path_fn_data = &processed;\n     -+\trevs->tag_objects = 1;\n      +\n      +\t/*\n      +\t * Allow the --[no-]sparse option to be interesting here, if only\n     @@ builtin/pack-objects.c: static void mark_bitmap_preferred_tips(void)\n       {\n       \tstruct setup_revision_opt s_r_opt = {\n      @@ builtin/pack-objects.c: static void get_object_list(struct rev_info *revs, int ac, const char **av)\n     - \n     - \twarn_on_object_refname_ambiguity = save_warning;\n     - \n     --\tif (use_bitmap_index && !get_object_list_from_bitmap(revs))\n     -+\tif (use_bitmap_index && !path_walk && !get_object_list_from_bitmap(revs))\n     - \t\treturn;\n     - \n     - \tif (use_delta_islands)\n     -@@ builtin/pack-objects.c: static void get_object_list(struct rev_info *revs, int ac, const char **av)\n       \tif (write_bitmap_index)\n       \t\tmark_bitmap_preferred_tips();\n       \n     @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n       \tstrvec_push(&rp, \"pack-objects\");\n      -\tif (thin) {\n      +\n     -+\tif (path_walk && filter_options.choice) {\n     -+\t\twarning(_(\"cannot use --filter with --path-walk\"));\n     -+\t\tpath_walk = 0;\n     -+\t}\n     -+\tif (path_walk && use_delta_islands) {\n     -+\t\twarning(_(\"cannot use delta islands with --path-walk\"));\n     -+\t\tpath_walk = 0;\n     -+\t}\n     -+\tif (path_walk && shallow) {\n     -+\t\twarning(_(\"cannot use --shallow with --path-walk\"));\n     -+\t\tpath_walk = 0;\n     ++\tif (path_walk) {\n     ++\t\tconst char *option = NULL;\n     ++\t\tif (filter_options.choice)\n     ++\t\t\toption = \"--filter\";\n     ++\t\telse if (use_delta_islands)\n     ++\t\t\toption = \"--delta-islands\";\n     ++\t\telse if (shallow)\n     ++\t\t\toption = \"--shallow\";\n     ++\n     ++\t\tif (option) {\n     ++\t\t\twarning(_(\"cannot use %s with %s\"),\n     ++\t\t\t\toption, \"--path-walk\");\n     ++\t\t\tpath_walk = 0;\n     ++\t\t}\n      +\t}\n      +\tif (path_walk) {\n      +\t\tstrvec_push(&rp, \"--boundary\");\n  3:  dcff01392ff =  3:  6d65fce5cc3 pack-objects: update usage to match docs\n  4:  97a0b52ccee !  4:  ad65148558e p5313: add performance tests for --path-walk\n     @@ Commit message\n          additions are made to the path walk API.\n      \n          The cases where the --path-walk option really shines is when the default\n     -    name-hash is overwhelmed with collisions. An open source example can be\n     -    found in the microsoft/fluentui repo [1] at a certain commit [2].\n     +    name-hash is overwhelmed with unhelpful collisions. An open source\n     +    example can be found in the microsoft/fluentui repo [1] at a certain\n     +    commit [2].\n      \n          [1] https://github.com/microsoft/fluentui\n          [2] e70848ebac1cd720875bccaa3026f4a9ed700e08\n  5:  0d49bb3d30a !  5:  6ed6d4e3382 pack-objects: introduce GIT_TEST_PACK_PATH_WALK\n     @@ Commit message\n          when specified.\n      \n          This was useful in testing the implementation of the --path-walk\n     -    implementation, especially in conjunction with test such as:\n     +    implementation, helping to find tests that are overly specific to the\n     +    default object walk. These include:\n      \n           - t0411-clone-from-partial.sh : One test fetches from a repo that does\n             not have the boundary objects. This causes the path-based walk to\n  6:  ddf804e606a !  6:  19c6651183a t5538: add tests to confirm deltas in shallow pushes\n     @@ t/t5538-push-shallow.sh: EOF\n      +\tgit -C deltas checkout -b deltas &&\n      +\ttest_seq 1 1000 >deltas/a &&\n      +\tgit -C deltas commit -a -m \"bigger a\" &&\n     -+\tGIT_TRACE2_PERF=\"$(pwd)/trace.txt\" \\\n      +\tGIT_PROGRESS_DELAY=0 git -C deltas push --progress origin deltas 2>err &&\n      +\n      +\ttest_grep \"Enumerating objects: 5, done\" err &&\n  7:  11767e7653e !  7:  7f9e6a077bc repack: add --path-walk option\n     @@ Documentation/git-repack.adoc: linkgit:git-multi-pack-index[1]).\n       \tSee linkgit:git-pack-objects[1] for full details.\n       \n      +--path-walk::\n     -+\tThis option passes the `--path-walk` option to the underlying\n     -+\t`git pack-options` process (see linkgit:git-pack-objects[1]).\n     -+\tBy default, `git pack-objects` walks objects in an order that\n     -+\tpresents trees and blobs in an order unrelated to the path they\n     -+\tappear relative to a commit's root tree. The `--path-walk` option\n     -+\tenables a different walking algorithm that organizes trees and\n     -+\tblobs by path. This has the potential to improve delta compression\n     -+\tespecially in the presence of filenames that cause collisions in\n     -+\tGit's default name-hash algorithm. Due to changing how the objects\n     -+\tare walked, this option is not compatible with `--delta-islands`\n     -+\tor `--filter`.\n     ++\tPass the `--path-walk` option to the underlying `git pack-objects`\n     ++\tprocess. See linkgit:git-pack-objects[1] for full details.\n       \n       CONFIGURATION\n       -------------\n  8:  dd66a5b46f2 !  8:  b98e71d6d54 pack-objects: enable --path-walk via config\n     @@ Commit message\n          settings infrastructure to make the new \"pack.usePathWalk\" config enabled by\n          \"feature.experimental\" and \"feature.manyFiles\".\n      \n     +    In order to test that this config works, add a new trace2 region around\n     +    the path walk code that can be checked by a 'git push' command.\n     +\n          Signed-off-by: Derrick Stolee <stolee@gmail.com>\n      \n       ## Documentation/config/feature.adoc ##\n     @@ Documentation/config/pack.adoc: pack.useSparse::\n       \t`true`.\n       \n      +pack.usePathWalk::\n     -+\tWhen true, git will default to using the '--path-walk' option in\n     -+\t'git pack-objects' when the '--revs' option is present. This\n     -+\talgorithm groups objects by path to maximize the ability to\n     -+\tcompute delta chains across historical versions of the same\n     -+\tobject. This may disable other options, such as using bitmaps to\n     -+\tenumerate objects.\n     ++\tEnable the `--path-walk` option by default for `git pack-objects`\n     ++\tprocesses. See linkgit:git-pack-objects[1] for full details.\n      +\n       pack.preferBitmapTips::\n       \tWhen selecting which commits will receive bitmaps, prefer a\n       \tcommit at the tip of any reference that is a suffix of any value\n      \n       ## builtin/pack-objects.c ##\n     +@@\n     + #include \"blob.h\"\n     + #include \"tree.h\"\n     + #include \"path-walk.h\"\n     ++#include \"trace2.h\"\n     + \n     + /*\n     +  * Objects we are going to pack are collected in the `to_pack` structure.\n     +@@ builtin/pack-objects.c: static void get_object_list_path_walk(struct rev_info *revs)\n     + {\n     + \tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n     + \tunsigned int processed = 0;\n     ++\tint result;\n     + \n     + \tinfo.revs = revs;\n     + \tinfo.path_fn = add_objects_by_path;\n     +@@ builtin/pack-objects.c: static void get_object_list_path_walk(struct rev_info *revs)\n     + \t */\n     + \tinfo.prune_all_uninteresting = sparse;\n     + \n     +-\tif (walk_objects_by_path(&info))\n     ++\ttrace2_region_enter(\"pack-objects\", \"path-walk\", revs->repo);\n     ++\tresult = walk_objects_by_path(&info);\n     ++\ttrace2_region_leave(\"pack-objects\", \"path-walk\", revs->repo);\n     ++\n     ++\tif (result)\n     + \t\tdie(_(\"failed to pack objects via path-walk\"));\n     + }\n     + \n      @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n       \t\tif (use_bitmap_index > 0 ||\n       \t\t    !use_internal_rev_list)\n     @@ repo-settings.h: struct repo_settings {\n       \tenum fetch_negotiation_setting fetch_negotiation_algorithm;\n       \n       \tint core_multi_pack_index;\n     +\n     + ## t/t5516-fetch-push.sh ##\n     +@@ t/t5516-fetch-push.sh: test_expect_success 'push with config push.useBitmaps' '\n     + \t\t--thin --delta-base-offset -q --no-use-bitmap-index <false\n     + '\n     + \n     ++test_expect_success 'push with config pack.usePathWalk=true' '\n     ++\tmk_test testrepo heads/main &&\n     ++\tgit checkout main &&\n     ++\ttest_config pack.usePathWalk true &&\n     ++\tGIT_TRACE2_EVENT=\"$(pwd)/path-walk.txt\" \\\n     ++\tgit push --quiet testrepo main:test &&\n     ++\n     ++\ttest_region pack-objects path-walk path-walk.txt\n     ++'\n     ++\n     + test_done\n  9:  e5624c379d5 =  9:  8e1d141a1f8 scalar: enable path-walk during push via config\n 10:  622439d7855 ! 10:  2c1d8479872 pack-objects: refactor path-walk delta phase\n     @@ Commit message\n          This presents a new progress indicator that can be used in tests to\n          verify that this stage is happening.\n      \n     -    The current implementation is not integrated with threads, but could be\n     -    done in a future update.\n     +    The current implementation is not integrated with threads, but we are\n     +    setting it up to arrive in the next change.\n      \n          Since we do not attempt to sort objects by size until after exploring\n          all trees, we can remove the previous change to t5530 due to a different\n     @@ builtin/pack-objects.c: static int should_attempt_deltas(struct object_entry *en\n      +\t\t\t\t   unsigned int *processed)\n      +{\n      +\tstruct object_entry **delta_list;\n     -+\tuint32_t delta_list_nr = 0;\n     ++\tunsigned int delta_list_nr = 0;\n      +\n      +\tALLOC_ARRAY(delta_list, region->nr);\n     -+\tfor (uint32_t i = 0; i < region->nr; i++) {\n     ++\tfor (size_t i = 0; i < region->nr; i++) {\n      +\t\tstruct object_entry *entry = list + region->start + i;\n      +\t\tif (should_attempt_deltas(entry))\n      +\t\t\tdelta_list[delta_list_nr++] = entry;\n     @@ builtin/pack-objects.c: static int should_attempt_deltas(struct object_entry *en\n      +\n      +static void find_deltas_by_region(struct object_entry *list,\n      +\t\t\t\t  struct packing_region *regions,\n     -+\t\t\t\t  uint32_t start, uint32_t nr)\n     ++\t\t\t\t  size_t start, size_t nr)\n      +{\n      +\tunsigned int processed = 0;\n     -+\tuint32_t progress_nr;\n     ++\tsize_t progress_nr;\n      +\n      +\tif (!nr)\n      +\t\treturn;\n     @@ builtin/pack-objects.c: static int add_objects_by_path(const char *path,\n       \t\t\t       enum object_type type,\n       \t\t\t       void *data)\n       {\n     --\tstruct object_entry **delta_list;\n     +-\tstruct object_entry **delta_list = NULL;\n       \tsize_t oe_start = to_pack.nr_objects;\n       \tsize_t oe_end;\n     --\tunsigned int sub_list_size;\n     +-\tunsigned int sub_list_nr;\n       \tunsigned int *processed = data;\n       \n       \t/*\n     @@ builtin/pack-objects.c: static int add_objects_by_path(const char *path,\n       \tif (oe_end == oe_start || !window)\n       \t\treturn 0;\n       \n     --\tsub_list_size = 0;\n     --\tALLOC_ARRAY(delta_list, oe_end - oe_start);\n     --\n     --\tfor (size_t i = 0; i < oe_end - oe_start; i++) {\n     --\t\tstruct object_entry *entry = to_pack.objects + oe_start + i;\n     +-\tsub_list_nr = 0;\n     +-\tif (oe_end > oe_start)\n     +-\t\tALLOC_ARRAY(delta_list, oe_end - oe_start);\n      +\tALLOC_GROW(to_pack.regions,\n      +\t\t   to_pack.nr_regions + 1,\n      +\t\t   to_pack.nr_regions_alloc);\n       \n     --\t\tif (!should_attempt_deltas(entry))\n     --\t\t\tcontinue;\n     --\n     --\t\tdelta_list[sub_list_size++] = entry;\n     --\t}\n     +-\tfor (size_t i = 0; i < oe_end - oe_start; i++) {\n     +-\t\tstruct object_entry *entry = to_pack.objects + oe_start + i;\n      +\tto_pack.regions[to_pack.nr_regions].start = oe_start;\n      +\tto_pack.regions[to_pack.nr_regions].nr = oe_end - oe_start;\n      +\tto_pack.nr_regions++;\n       \n     +-\t\tif (!should_attempt_deltas(entry))\n     +-\t\t\tcontinue;\n     ++\t*processed += oids->nr;\n     ++\tdisplay_progress(progress_state, *processed);\n     + \n     +-\t\tdelta_list[sub_list_nr++] = entry;\n     +-\t}\n     +-\n      -\t/*\n      -\t * Find delta bases among this list of objects that all match the same\n      -\t * path. This causes the delta compression to be interleaved in the\n     @@ builtin/pack-objects.c: static int add_objects_by_path(const char *path,\n      -\t * consider creating a list of regions in the full to_pack.objects array\n      -\t * that could be picked up by the threaded delta computation.\n      -\t */\n     --\tif (sub_list_size && window) {\n     --\t\tQSORT(delta_list, sub_list_size, type_size_sort);\n     --\t\tfind_deltas(delta_list, &sub_list_size, window, depth, processed);\n     +-\tif (sub_list_nr && window) {\n     +-\t\tQSORT(delta_list, sub_list_nr, type_size_sort);\n     +-\t\tfind_deltas(delta_list, &sub_list_nr, window, depth, processed);\n      -\t}\n     -+\t*processed += oids->nr;\n     -+\tdisplay_progress(progress_state, *processed);\n     - \n     +-\n      -\tfree(delta_list);\n       \treturn 0;\n       }\n     @@ pack-objects.h: struct object_entry {\n      + * as given by a starting index and a number of elements.\n      + */\n      +struct packing_region {\n     -+\tuint32_t start;\n     -+\tuint32_t nr;\n     ++\tsize_t start;\n     ++\tsize_t nr;\n      +};\n      +\n       struct packing_data {\n     @@ pack-objects.h: struct object_entry {\n       \tuint32_t nr_objects, nr_alloc;\n       \n      +\tstruct packing_region *regions;\n     -+\tuint64_t nr_regions, nr_regions_alloc;\n     ++\tsize_t nr_regions, nr_regions_alloc;\n      +\n       \tint32_t *index;\n       \tuint32_t index_size;\n 11:  ae73d26319a ! 11:  623913f284e pack-objects: thread the path-based compression\n     @@ Commit message\n          overall time improves enough to outperform the --name-hash-version=1\n          case. These time improvements correlate with repositories with data\n          shapes that significantly improve their data size as well. The\n     -    --path-walk feature frequently takes longer than --name-hash-verison=2,\n     -    trading some extrac computation for some additional compression. The\n     +    --path-walk feature frequently takes longer than --name-hash-version=2,\n     +    trading some extra computation for some additional compression. The\n          natural place where this additional computation comes from is the two\n          compression passes that --path-walk takes, though the first pass is\n          naturally faster due to the path boundaries avoiding a number of delta\n     @@ builtin/pack-objects.c: static void find_deltas_by_region(struct object_entry *l\n      +\t}\n      +\n      +\tif (progress > pack_to_stdout)\n     -+\t\tfprintf_ln(stderr, _(\"Path-based delta compression using up to %d threads\"),\n     ++\t\tfprintf_ln(stderr,\n     ++\t\t\t   Q_(\"Path-based delta compression using up to %d thread\",\n     ++\t\t\t      \"Path-based delta compression using up to %d threads\",\n     ++\t\t\t      delta_search_threads),\n      +\t\t\t   delta_search_threads);\n      +\tCALLOC_ARRAY(p, delta_search_threads);\n      +\n 12:  c45f35c204e = 12:  a14d447d35e path-walk: add new 'edge_aggressive' option\n 13:  d5484ebd942 ! 13:  18241d38207 pack-objects: allow --shallow and --path-walk\n     @@ builtin/pack-objects.c: static void get_object_list_path_walk(struct rev_info *r\n       \tinfo.prune_all_uninteresting = sparse;\n      +\tinfo.edge_aggressive = shallow;\n       \n     - \tif (walk_objects_by_path(&info))\n     - \t\tdie(_(\"failed to pack objects via path-walk\"));\n     + \ttrace2_region_enter(\"pack-objects\", \"path-walk\", revs->repo);\n     + \tresult = walk_objects_by_path(&info);\n      @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n       \t\t     struct repository *repo UNUSED)\n       {\n     @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n       \tstruct strvec rp = STRVEC_INIT;\n       \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n      @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n     - \t\twarning(_(\"cannot use delta islands with --path-walk\"));\n     - \t\tpath_walk = 0;\n     - \t}\n     --\tif (path_walk && shallow) {\n     --\t\twarning(_(\"cannot use --shallow with --path-walk\"));\n     --\t\tpath_walk = 0;\n     --\t}\n     - \tif (path_walk) {\n     - \t\tstrvec_push(&rp, \"--boundary\");\n     - \t\t /*\n     + \t\t\toption = \"--filter\";\n     + \t\telse if (use_delta_islands)\n     + \t\t\toption = \"--delta-islands\";\n     +-\t\telse if (shallow)\n     +-\t\t\toption = \"--shallow\";\n     + \n     + \t\tif (option) {\n     + \t\t\twarning(_(\"cannot use %s with %s\"),\n     +\n     + ## t/t5538-push-shallow.sh ##\n     +@@ t/t5538-push-shallow.sh: test_expect_success 'push new commit from shallow clone has good deltas' '\n     + \n     + \t# If the delta base is found, then this message uses \"bytes\".\n     + \t# If the delta base is not found, then this message uses \"KiB\".\n     +-\ttest_grep \"Writing objects: .* bytes\" err\n     ++\ttest_grep \"Writing objects: .* bytes\" err &&\n     ++\n     ++\tgit -C deltas commit --amend -m \"changed message\" &&\n     ++\tGIT_TRACE2_EVENT=\"$(pwd)/config-push.txt\" \\\n     ++\tGIT_PROGRESS_DELAY=0 git -C deltas -c pack.usePathWalk=true \\\n     ++\t\tpush --progress -f origin deltas 2>err &&\n     ++\n     ++\ttest_grep \"Enumerating objects: 1, done\" err &&\n     ++\ttest_region pack-objects path-walk config-push.txt\n     + '\n     + \n     + test_done\n\n-- \ngitgitgadget\n"},{"id":"518277","messageId":"57c1cc20de0c80b84ad11a6546763826fe4b1a09.1747419124.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v3.git.1747419124.gitgitgadget@gmail.com","subject":"[PATCH v3 01/13] pack-objects: extract should_attempt_deltas()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-05-16T18:11:51Z","receivedAt":"2025-05-16T18:12:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThis will be helpful in a future change, which will reuse this logic.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c | 56 ++++++++++++++++++++++++------------------\n 1 file changed, 32 insertions(+), 24 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 58a9b1612626..7805429f5d1c 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3196,6 +3196,36 @@ static int add_ref_tag(const char *tag UNUSED, const char *referent UNUSED, cons\n \treturn 0;\n }\n \n+static int should_attempt_deltas(struct object_entry *entry)\n+{\n+\tif (DELTA(entry))\n+\t\t/* This happens if we decided to reuse existing\n+\t\t * delta from a pack. \"reuse_delta &&\" is implied.\n+\t\t */\n+\t\treturn 0;\n+\n+\tif (!entry->type_valid ||\n+\t    oe_size_less_than(&to_pack, entry, 50))\n+\t\treturn 0;\n+\n+\tif (entry->no_try_delta)\n+\t\treturn 0;\n+\n+\tif (!entry->preferred_base) {\n+\t\tif (oe_type(entry) < 0)\n+\t\t\tdie(_(\"unable to get type of object %s\"),\n+\t\t\t\toid_to_hex(&entry->idx.oid));\n+\t} else if (oe_type(entry) < 0) {\n+\t\t/*\n+\t\t * This object is not found, but we\n+\t\t * don't have to include it anyway.\n+\t\t */\n+\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+}\n+\n static void prepare_pack(int window, int depth)\n {\n \tstruct object_entry **delta_list;\n@@ -3226,33 +3256,11 @@ static void prepare_pack(int window, int depth)\n \tfor (i = 0; i < to_pack.nr_objects; i++) {\n \t\tstruct object_entry *entry = to_pack.objects + i;\n \n-\t\tif (DELTA(entry))\n-\t\t\t/* This happens if we decided to reuse existing\n-\t\t\t * delta from a pack.  \"reuse_delta &&\" is implied.\n-\t\t\t */\n-\t\t\tcontinue;\n-\n-\t\tif (!entry->type_valid ||\n-\t\t    oe_size_less_than(&to_pack, entry, 50))\n+\t\tif (!should_attempt_deltas(entry))\n \t\t\tcontinue;\n \n-\t\tif (entry->no_try_delta)\n-\t\t\tcontinue;\n-\n-\t\tif (!entry->preferred_base) {\n+\t\tif (!entry->preferred_base)\n \t\t\tnr_deltas++;\n-\t\t\tif (oe_type(entry) < 0)\n-\t\t\t\tdie(_(\"unable to get type of object %s\"),\n-\t\t\t\t    oid_to_hex(&entry->idx.oid));\n-\t\t} else {\n-\t\t\tif (oe_type(entry) < 0) {\n-\t\t\t\t/*\n-\t\t\t\t * This object is not found, but we\n-\t\t\t\t * don't have to include it anyway.\n-\t\t\t\t */\n-\t\t\t\tcontinue;\n-\t\t\t}\n-\t\t}\n \n \t\tdelta_list[n++] = entry;\n \t}\n-- \ngitgitgadget\n\n"},{"id":"518278","messageId":"e3ea17e326e5997ffbd3fdae8c5e011c816c88ca.1747419124.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v3.git.1747419124.gitgitgadget@gmail.com","subject":"[PATCH v3 02/13] pack-objects: add --path-walk option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-05-16T18:11:52Z","receivedAt":"2025-05-16T18:12:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nIn order to more easily compute delta bases among objects that appear at\nthe exact same path, add a --path-walk option to 'git pack-objects'.\n\nThis option will use the path-walk API instead of the object walk given\nby the revision machinery. Since objects will be provided in batches\nrepresenting a common path, those objects can be tested for delta bases\nimmediately instead of waiting for a sort of the full object list by\nname-hash. This has multiple benefits, including avoiding collisions by\nname-hash.\n\nThe objects marked as UNINTERESTING are included in these batches, so we\nare guaranteeing some locality to find good delta bases.\n\nAfter the individual passes are done on a per-path basis, the default\nname-hash is used to find other opportunistic delta bases that did not\nmatch exactly by the full path name.\n\nThe current implementation performs delta calculations while walking\nobjects, which is not ideal for a few reasons. First, this will cause\nthe \"Enumerating objects\" phase to be much longer than usual. Second, it\ndoes not take advantage of threading during the path-scoped delta\ncalculations. Even with this lack of threading, the path-walk option is\nsometimes faster than the usual approach. Future changes will refactor\nthis code to allow for threading, but that complexity is deferred until\nlater to keep this patch as simple as possible.\n\nThis new walk is incompatible with some features and is ignored by\nothers:\n\n * Object filters are not currently integrated with the path-walk API,\n   such as sparse-checkout or tree depth. A blobless packfile could be\n   integrated easily, but that is deferred for later.\n\n * Server-focused features such as delta islands, shallow packs, and\n   using a bitmap index are incompatible with the path-walk API.\n\n * The path walk API is only compatible with the --revs option, not\n   taking object lists or pack lists over stdin. These alternative ways\n   to specify the objects currently ignores the --path-walk option\n   without even a warning.\n\nFuture changes will create performance tests that demonstrate the power\nof this approach.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-pack-objects.adoc        |  13 +-\n Documentation/technical/api-path-walk.adoc |   1 +\n builtin/pack-objects.c                     | 148 +++++++++++++++++++--\n t/t5300-pack-object.sh                     |  15 +++\n 4 files changed, 168 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc\nindex 7f69ae4855f6..3b803d3a7830 100644\n--- a/Documentation/git-pack-objects.adoc\n+++ b/Documentation/git-pack-objects.adoc\n@@ -16,7 +16,7 @@ SYNOPSIS\n \t[--cruft] [--cruft-expiration=<time>]\n \t[--stdout [--filter=<filter-spec>] | <base-name>]\n \t[--shallow] [--keep-true-parents] [--[no-]sparse]\n-\t[--name-hash-version=<n>] < <object-list>\n+\t[--name-hash-version=<n>] [--path-walk] < <object-list>\n \n \n DESCRIPTION\n@@ -375,6 +375,17 @@ many different directories. At the moment, this version is not allowed\n when writing reachability bitmap files with `--write-bitmap-index` and it\n will be automatically changed to version `1`.\n \n+--path-walk::\n+\tPerform compression by first organizing objects by path, then a\n+\tsecond pass that compresses across paths as normal. This has the\n+\tpotential to improve delta compression especially in the presence\n+\tof filenames that cause collisions in Git's default name-hash\n+\talgorithm.\n++\n+Incompatible with `--delta-islands`, `--shallow`, or `--filter`. The\n+`--use-bitmap-index` option will be ignored in the presence of\n+`--path-walk.`\n+\n \n DELTA ISLANDS\n -------------\ndiff --git a/Documentation/technical/api-path-walk.adoc b/Documentation/technical/api-path-walk.adoc\nindex 3e089211fb4d..e522695dd9fa 100644\n--- a/Documentation/technical/api-path-walk.adoc\n+++ b/Documentation/technical/api-path-walk.adoc\n@@ -69,4 +69,5 @@ Examples\n \n See example usages in:\n \t`t/helper/test-path-walk.c`,\n+\t`builtin/pack-objects.c`,\n \t`builtin/backfill.c`\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7805429f5d1c..bd0883890376 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -41,6 +41,9 @@\n #include \"promisor-remote.h\"\n #include \"pack-mtimes.h\"\n #include \"parse-options.h\"\n+#include \"blob.h\"\n+#include \"tree.h\"\n+#include \"path-walk.h\"\n \n /*\n  * Objects we are going to pack are collected in the `to_pack` structure.\n@@ -217,6 +220,7 @@ static int delta_search_threads;\n static int pack_to_stdout;\n static int sparse;\n static int thin;\n+static int path_walk;\n static int num_preferred_base;\n static struct progress *progress_state;\n \n@@ -4191,6 +4195,105 @@ static void mark_bitmap_preferred_tips(void)\n \t}\n }\n \n+static inline int is_oid_uninteresting(struct repository *repo,\n+\t\t\t\t       struct object_id *oid)\n+{\n+\tstruct object *o = lookup_object(repo, oid);\n+\treturn !o || (o->flags & UNINTERESTING);\n+}\n+\n+static int add_objects_by_path(const char *path,\n+\t\t\t       struct oid_array *oids,\n+\t\t\t       enum object_type type,\n+\t\t\t       void *data)\n+{\n+\tstruct object_entry **delta_list = NULL;\n+\tsize_t oe_start = to_pack.nr_objects;\n+\tsize_t oe_end;\n+\tunsigned int sub_list_nr;\n+\tunsigned int *processed = data;\n+\n+\t/*\n+\t * First, add all objects to the packing data, including the ones\n+\t * marked UNINTERESTING (translated to 'exclude') as they can be\n+\t * used as delta bases.\n+\t */\n+\tfor (size_t i = 0; i < oids->nr; i++) {\n+\t\tint exclude;\n+\t\tstruct object_info oi = OBJECT_INFO_INIT;\n+\t\tstruct object_id *oid = &oids->oid[i];\n+\n+\t\t/* Skip objects that do not exist locally. */\n+\t\tif (exclude_promisor_objects &&\n+\t\t    oid_object_info_extended(the_repository, oid, &oi,\n+\t\t\t\t\t     OBJECT_INFO_FOR_PREFETCH) < 0)\n+\t\t\tcontinue;\n+\n+\t\texclude = is_oid_uninteresting(the_repository, oid);\n+\n+\t\tif (exclude && !thin)\n+\t\t\tcontinue;\n+\n+\t\tadd_object_entry(oid, type, path, exclude);\n+\t}\n+\n+\toe_end = to_pack.nr_objects;\n+\n+\t/* We can skip delta calculations if it is a no-op. */\n+\tif (oe_end == oe_start || !window)\n+\t\treturn 0;\n+\n+\tsub_list_nr = 0;\n+\tif (oe_end > oe_start)\n+\t\tALLOC_ARRAY(delta_list, oe_end - oe_start);\n+\n+\tfor (size_t i = 0; i < oe_end - oe_start; i++) {\n+\t\tstruct object_entry *entry = to_pack.objects + oe_start + i;\n+\n+\t\tif (!should_attempt_deltas(entry))\n+\t\t\tcontinue;\n+\n+\t\tdelta_list[sub_list_nr++] = entry;\n+\t}\n+\n+\t/*\n+\t * Find delta bases among this list of objects that all match the same\n+\t * path. This causes the delta compression to be interleaved in the\n+\t * object walk, which can lead to confusing progress indicators. This is\n+\t * also incompatible with threaded delta calculations. In the future,\n+\t * consider creating a list of regions in the full to_pack.objects array\n+\t * that could be picked up by the threaded delta computation.\n+\t */\n+\tif (sub_list_nr && window) {\n+\t\tQSORT(delta_list, sub_list_nr, type_size_sort);\n+\t\tfind_deltas(delta_list, &sub_list_nr, window, depth, processed);\n+\t}\n+\n+\tfree(delta_list);\n+\treturn 0;\n+}\n+\n+static void get_object_list_path_walk(struct rev_info *revs)\n+{\n+\tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n+\tunsigned int processed = 0;\n+\n+\tinfo.revs = revs;\n+\tinfo.path_fn = add_objects_by_path;\n+\tinfo.path_fn_data = &processed;\n+\n+\t/*\n+\t * Allow the --[no-]sparse option to be interesting here, if only\n+\t * for testing purposes. Paths with no interesting objects will not\n+\t * contribute to the resulting pack, but only create noisy preferred\n+\t * base objects.\n+\t */\n+\tinfo.prune_all_uninteresting = sparse;\n+\n+\tif (walk_objects_by_path(&info))\n+\t\tdie(_(\"failed to pack objects via path-walk\"));\n+}\n+\n static void get_object_list(struct rev_info *revs, int ac, const char **av)\n {\n \tstruct setup_revision_opt s_r_opt = {\n@@ -4246,15 +4349,19 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n \tif (write_bitmap_index)\n \t\tmark_bitmap_preferred_tips();\n \n-\tif (prepare_revision_walk(revs))\n-\t\tdie(_(\"revision walk setup failed\"));\n-\tmark_edges_uninteresting(revs, show_edge, sparse);\n-\n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n-\ttraverse_commit_list(revs,\n-\t\t\t     show_commit, fn_show_object,\n-\t\t\t     NULL);\n+\n+\tif (path_walk) {\n+\t\tget_object_list_path_walk(revs);\n+\t} else {\n+\t\tif (prepare_revision_walk(revs))\n+\t\t\tdie(_(\"revision walk setup failed\"));\n+\t\tmark_edges_uninteresting(revs, show_edge, sparse);\n+\t\ttraverse_commit_list(revs,\n+\t\t\t\tshow_commit, fn_show_object,\n+\t\t\t\tNULL);\n+\t}\n \n \tif (unpack_unreachable_expiration) {\n \t\trevs->ignore_missing_links = 1;\n@@ -4464,6 +4571,8 @@ int cmd_pack_objects(int argc,\n \t\t\t N_(\"use the sparse reachability algorithm\")),\n \t\tOPT_BOOL(0, \"thin\", &thin,\n \t\t\t N_(\"create thin packs\")),\n+\t\tOPT_BOOL(0, \"path-walk\", &path_walk,\n+\t\t\t N_(\"use the path-walk API to walk objects when possible\")),\n \t\tOPT_BOOL(0, \"shallow\", &shallow,\n \t\t\t N_(\"create packs suitable for shallow fetches\")),\n \t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep_on_disk,\n@@ -4549,7 +4658,30 @@ int cmd_pack_objects(int argc,\n \t\twindow = 0;\n \n \tstrvec_push(&rp, \"pack-objects\");\n-\tif (thin) {\n+\n+\tif (path_walk) {\n+\t\tconst char *option = NULL;\n+\t\tif (filter_options.choice)\n+\t\t\toption = \"--filter\";\n+\t\telse if (use_delta_islands)\n+\t\t\toption = \"--delta-islands\";\n+\t\telse if (shallow)\n+\t\t\toption = \"--shallow\";\n+\n+\t\tif (option) {\n+\t\t\twarning(_(\"cannot use %s with %s\"),\n+\t\t\t\toption, \"--path-walk\");\n+\t\t\tpath_walk = 0;\n+\t\t}\n+\t}\n+\tif (path_walk) {\n+\t\tstrvec_push(&rp, \"--boundary\");\n+\t\t /*\n+\t\t  * We must disable the bitmaps because we are removing\n+\t\t  * the --objects / --objects-edge[-aggressive] options.\n+\t\t  */\n+\t\tuse_bitmap_index = 0;\n+\t} else if (thin) {\n \t\tuse_internal_rev_list = 1;\n \t\tstrvec_push(&rp, shallow\n \t\t\t\t? \"--objects-edge-aggressive\"\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 5ac8d39094b6..16420d128639 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -723,4 +723,19 @@ test_expect_success '--name-hash-version=2 and --write-bitmap-index are incompat\n \t! test_grep \"currently, --write-bitmap-index requires --name-hash-version=1\" err\n '\n \n+test_expect_success '--path-walk pack everything' '\n+\tgit -C server rev-parse HEAD >in &&\n+\tgit -C server pack-objects --stdout --revs --path-walk <in >out.pack &&\n+\tgit -C server index-pack --stdin <out.pack\n+'\n+\n+test_expect_success '--path-walk thin pack' '\n+\tcat >in <<-EOF &&\n+\t$(git -C server rev-parse HEAD)\n+\t^$(git -C server rev-parse HEAD~2)\n+\tEOF\n+\tgit -C server pack-objects --thin --stdout --revs --path-walk <in >out.pack &&\n+\tgit -C server index-pack --fix-thin --stdin <out.pack\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"518279","messageId":"6d65fce5cc31621b7b4452ce57cc7c002ab0caef.1747419124.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v3.git.1747419124.gitgitgadget@gmail.com","subject":"[PATCH v3 03/13] pack-objects: update usage to match docs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-05-16T18:11:53Z","receivedAt":"2025-05-16T18:12:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThe t0450 test script verifies that builtin usage matches the synopsis\nin the documentation. Adjust the builtin to match and then remove 'git\npack-objects' from the exception list.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-pack-objects.adoc | 14 +++++++-------\n builtin/pack-objects.c              | 10 ++++++++--\n t/t0450/adoc-help-mismatches        |  1 -\n 3 files changed, 15 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc\nindex 3b803d3a7830..b1c5aa27da4d 100644\n--- a/Documentation/git-pack-objects.adoc\n+++ b/Documentation/git-pack-objects.adoc\n@@ -10,13 +10,13 @@ SYNOPSIS\n --------\n [verse]\n 'git pack-objects' [-q | --progress | --all-progress] [--all-progress-implied]\n-\t[--no-reuse-delta] [--delta-base-offset] [--non-empty]\n-\t[--local] [--incremental] [--window=<n>] [--depth=<n>]\n-\t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n-\t[--cruft] [--cruft-expiration=<time>]\n-\t[--stdout [--filter=<filter-spec>] | <base-name>]\n-\t[--shallow] [--keep-true-parents] [--[no-]sparse]\n-\t[--name-hash-version=<n>] [--path-walk] < <object-list>\n+\t\t   [--no-reuse-delta] [--delta-base-offset] [--non-empty]\n+\t\t   [--local] [--incremental] [--window=<n>] [--depth=<n>]\n+\t\t   [--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n+\t\t   [--cruft] [--cruft-expiration=<time>]\n+\t\t   [--stdout [--filter=<filter-spec>] | <base-name>]\n+\t\t   [--shallow] [--keep-true-parents] [--[no-]sparse]\n+\t\t   [--name-hash-version=<n>] [--path-walk] < <object-list>\n \n \n DESCRIPTION\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex bd0883890376..59d640d9f255 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -187,8 +187,14 @@ static inline void oe_set_delta_size(struct packing_data *pack,\n #define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val)\n \n static const char *pack_usage[] = {\n-\tN_(\"git pack-objects --stdout [<options>] [< <ref-list> | < <object-list>]\"),\n-\tN_(\"git pack-objects [<options>] <base-name> [< <ref-list> | < <object-list>]\"),\n+\tN_(\"git pack-objects [-q | --progress | --all-progress] [--all-progress-implied]\\n\"\n+\t   \"                 [--no-reuse-delta] [--delta-base-offset] [--non-empty]\\n\"\n+\t   \"                 [--local] [--incremental] [--window=<n>] [--depth=<n>]\\n\"\n+\t   \"                 [--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\\n\"\n+\t   \"                 [--cruft] [--cruft-expiration=<time>]\\n\"\n+\t   \"                 [--stdout [--filter=<filter-spec>] | <base-name>]\\n\"\n+\t   \"                 [--shallow] [--keep-true-parents] [--[no-]sparse]\\n\"\n+\t   \"                 [--name-hash-version=<n>] [--path-walk] < <object-list>\"),\n \tNULL\n };\n \ndiff --git a/t/t0450/adoc-help-mismatches b/t/t0450/adoc-help-mismatches\nindex c4a15fd0cb88..06b469bdee20 100644\n--- a/t/t0450/adoc-help-mismatches\n+++ b/t/t0450/adoc-help-mismatches\n@@ -38,7 +38,6 @@ merge-one-file\n multi-pack-index\n name-rev\n notes\n-pack-objects\n push\n range-diff\n rebase\n-- \ngitgitgadget\n\n"},{"id":"518280","messageId":"ad65148558eb698c6e7a8bbc1cfb6e6e6004e973.1747419124.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v3.git.1747419124.gitgitgadget@gmail.com","subject":"[PATCH v3 04/13] p5313: add performance tests for --path-walk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-05-16T18:11:54Z","receivedAt":"2025-05-16T18:12:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThe previous change added a --path-walk option to 'git pack-objects'.\nCreate a performance test that demonstrates the time and space benefits\nof the feature.\n\nIn order to get an appropriate comparison, we need to avoid reusing\ndeltas and recompute them from scratch.\n\nCompare the creation of a thin pack representing a small push and the\ncreation of a relatively large non-thin pack.\n\nRunning on my copy of the Git repository results in this data (removing\nthe repack tests for --name-hash-version):\n\nTest                                                     this tree\n------------------------------------------------------------------------\n5313.2: thin pack with --name-hash-version=1             0.02(0.01+0.01)\n5313.3: thin pack size with --name-hash-version=1                   1.6K\n5313.4: big pack with --name-hash-version=1              2.55(4.20+0.26)\n5313.5: big pack size with --name-hash-version=1                   16.4M\n5313.6: shallow fetch pack with --name-hash-version=1    1.24(2.03+0.08)\n5313.7: shallow pack size with --name-hash-version=1               12.2M\n5313.10: thin pack with --name-hash-version=2            0.03(0.01+0.01)\n5313.11: thin pack size with --name-hash-version=2                  1.6K\n5313.12: big pack with --name-hash-version=2             1.91(3.23+0.20)\n5313.13: big pack size with --name-hash-version=2                  16.4M\n5313.14: shallow fetch pack with --name-hash-version=2   1.06(1.57+0.10)\n5313.15: shallow pack size with --name-hash-version=2              12.5M\n5313.18: thin pack with --path-walk                      0.03(0.01+0.01)\n5313.19: thin pack size with --path-walk                            1.6K\n5313.20: big pack with --path-walk                       2.05(3.24+0.27)\n5313.21: big pack size with --path-walk                            16.3M\n5313.22: shallow fetch pack with --path-walk             1.08(1.66+0.07)\n5313.23: shallow pack size with --path-walk                        12.4M\n\nThis can be reformatted as follows:\n\nPack Type            Hash v1   Hash v2     Path Walk\n---------------------------------------------------\nthin pack    (time)    0.02s      0.03s      0.03s\n             (size)    1.6K       1.6K       1.6K\nbig pack     (time)    2.55s      1.91s      2.05s\n             (size)   16.4M      16.4M      16.3M\nshallow pack (time)    1.24s      1.06s      1.08s\n             (size)   12.2M      12.5M      12.4M\n\nNote that the timing is slower because there is no threading in the\n--path-walk case (yet). Also, the shallow pack cases are really not\nusing the --path-walk logic right now because it is disabled until some\nadditions are made to the path walk API.\n\nThe cases where the --path-walk option really shines is when the default\nname-hash is overwhelmed with unhelpful collisions. An open source\nexample can be found in the microsoft/fluentui repo [1] at a certain\ncommit [2].\n\n[1] https://github.com/microsoft/fluentui\n[2] e70848ebac1cd720875bccaa3026f4a9ed700e08\n\nRunning the tests on this repo results in the following comparison table:\n\nPack Type            Hash v1    Hash v2    Path Walk\n---------------------------------------------------\nthin pack    (time)    0.36s      0.12s      0.08s\n             (size)    1.2M      22.0K      18.4K\nbig pack     (time)    2.00s      2.90s      2.21s\n             (size)   20.4M      25.9M      19.5M\nshallow pack (time)    1.41s      1.80s      1.65s\n             (size)   34.4M      33.7M      33.6M\n\nNotice in particular that in the small thin pack, the time performance\nhas improved from 0.36s for --name-hash-version=1 to 0.08s and this is\nlikely due to the improved size of the resulting pack: 18.4K instead of\n1.2M.  The relatively new --name-hash-version=2 is competitive with\n--path-walk (0.12s and 22.0K) but not quite as successful.\n\nFinally, running this on a copy of the Linux kernel repository results\nin these data points:\n\nPack Type            Hash v1    Hash v2    Path Walk\n---------------------------------------------------\nthin pack    (time)    0.03s      0.13s      0.03s\n             (size)    4.6K       4.6K       4.6K\nbig pack     (time)   15.29s     12.32s     13.92s\n             (size)  201.1M     159.1M     158.5M\nshallow pack (time)   10.88s     22.93s     22.74s\n             (size)  269.2M     273.8M     267.7M\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n t/perf/p5313-pack-objects.sh | 37 ++++++++++++++++++++++--------------\n 1 file changed, 23 insertions(+), 14 deletions(-)\n\ndiff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh\nindex be5229a0ecdc..cd6dd3abb710 100755\n--- a/t/perf/p5313-pack-objects.sh\n+++ b/t/perf/p5313-pack-objects.sh\n@@ -25,46 +25,55 @@ test_expect_success 'create rev input' '\n \tEOF\n '\n \n-for version in 1 2\n-do\n-\texport version\n+test_all_with_args () {\n+\tparameter=$1\n+\texport parameter\n \n-\ttest_perf \"thin pack with version $version\" '\n+\ttest_perf \"thin pack with $parameter\" '\n \t\tgit pack-objects --thin --stdout --revs --sparse \\\n-\t\t\t--name-hash-version=$version <in-thin >out\n+\t\t\t$parameter <in-thin >out\n \t'\n \n-\ttest_size \"thin pack size with version $version\" '\n+\ttest_size \"thin pack size with $parameter\" '\n \t\ttest_file_size out\n \t'\n \n-\ttest_perf \"big pack with version $version\" '\n+\ttest_perf \"big pack with $parameter\" '\n \t\tgit pack-objects --stdout --revs --sparse \\\n-\t\t\t--name-hash-version=$version <in-big >out\n+\t\t\t$parameter <in-big >out\n \t'\n \n-\ttest_size \"big pack size with version $version\" '\n+\ttest_size \"big pack size with $parameter\" '\n \t\ttest_file_size out\n \t'\n \n-\ttest_perf \"shallow fetch pack with version $version\" '\n+\ttest_perf \"shallow fetch pack with $parameter\" '\n \t\tgit pack-objects --stdout --revs --sparse --shallow \\\n-\t\t\t--name-hash-version=$version <in-shallow >out\n+\t\t\t$parameter <in-shallow >out\n \t'\n \n-\ttest_size \"shallow pack size with version $version\" '\n+\ttest_size \"shallow pack size with $parameter\" '\n \t\ttest_file_size out\n \t'\n+}\n \n-\ttest_perf \"repack with version $version\" '\n+for version in 1 2\n+do\n+\texport version\n+\n+\ttest_all_with_args --name-hash-version=$version\n+\n+\ttest_perf \"repack with --name-hash-version=$version\" '\n \t\tgit repack -adf --name-hash-version=$version\n \t'\n \n-\ttest_size \"repack size with version $version\" '\n+\ttest_size \"repack size with --name-hash-version=$version\" '\n \t\tgitdir=$(git rev-parse --git-dir) &&\n \t\tpack=$(ls $gitdir/objects/pack/pack-*.pack) &&\n \t\ttest_file_size \"$pack\"\n \t'\n done\n \n+test_all_with_args --path-walk\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"518281","messageId":"6ed6d4e33824ec652aeb995c292da2f3819015c4.1747419124.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v3.git.1747419124.gitgitgadget@gmail.com","subject":"[PATCH v3 05/13] pack-objects: introduce GIT_TEST_PACK_PATH_WALK","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-05-16T18:11:55Z","receivedAt":"2025-05-16T18:12:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThere are many tests that validate whether 'git pack-objects' works as\nexpected. Instead of duplicating these tests, add a new test environment\nvariable, GIT_TEST_PACK_PATH_WALK, that implies --path-walk by default\nwhen specified.\n\nThis was useful in testing the implementation of the --path-walk\nimplementation, helping to find tests that are overly specific to the\ndefault object walk. These include:\n\n - t0411-clone-from-partial.sh : One test fetches from a repo that does\n   not have the boundary objects. This causes the path-based walk to\n   fail. Disable the variable for this test.\n\n - t5306-pack-nobase.sh : Similar to t0411, one test fetches from a repo\n   without a boundary object.\n\n - t5310-pack-bitmaps.sh : One test compares the case when packing with\n   bitmaps to the case when packing without them. Since we disable the\n   test variable when writing bitmaps, this causes a difference in the\n   object list (the --path-walk option adds an extra object). Specify\n   --no-path-walk in both processes for the comparison. Another test\n   checks for a specific delta base, but when computing dynamically\n   without using bitmaps, the base object it too small to be considered\n   in the delta calculations so no base is used.\n\n - t5316-pack-delta-depth.sh : This script cares about certain delta\n   choices and their chain lengths. The --path-walk option changes how\n   these chains are selected, and thus changes the results of this test.\n\n - t5322-pack-objects-sparse.sh : This demonstrates the effectiveness of\n   the --sparse option and how it combines with --path-walk.\n\n - t5332-multi-pack-reuse.sh : This test verifies that the preferred\n   pack is used for delta reuse when possible. The --path-walk option is\n   not currently aware of the preferred pack at all, so finds a\n   different delta base.\n\n - t7406-submodule-update.sh : When using the variable, the --depth\n   option collides with the --path-walk feature, resulting in a warning\n   message. Disable the variable so this warning does not appear.\n\nI want to call out one specific test change that is only temporary:\n\n - t5530-upload-pack-error.sh : One test cares specifically about an\n   \"unable to read\" error message. Since the current implementation\n   performs delta calculations within the path-walk API callback, a\n   different \"unable to get size\" error message appears. When this\n   is changed in a future refactoring, this test change can be reverted.\n\nSimilar to GIT_TEST_NAME_HASH_VERSION, we do not add this option to the\nlinux-TEST-vars CI build as that's already an overloaded build.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c        | 12 ++++++++++--\n t/README                      |  4 ++++\n t/t0411-clone-from-partial.sh |  6 ++++++\n t/t5306-pack-nobase.sh        |  5 +++++\n t/t5310-pack-bitmaps.sh       | 13 +++++++++++--\n t/t5316-pack-delta-depth.sh   |  9 ++++++---\n t/t5332-multi-pack-reuse.sh   |  7 +++++++\n t/t5530-upload-pack-error.sh  |  6 ++++++\n t/t7406-submodule-update.sh   |  3 +++\n 9 files changed, 58 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 59d640d9f255..4fd88476dd29 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -226,7 +226,7 @@ static int delta_search_threads;\n static int pack_to_stdout;\n static int sparse;\n static int thin;\n-static int path_walk;\n+static int path_walk = -1;\n static int num_preferred_base;\n static struct progress *progress_state;\n \n@@ -4230,7 +4230,7 @@ static int add_objects_by_path(const char *path,\n \t\tstruct object_id *oid = &oids->oid[i];\n \n \t\t/* Skip objects that do not exist locally. */\n-\t\tif (exclude_promisor_objects &&\n+\t\tif ((exclude_promisor_objects || arg_missing_action != MA_ERROR) &&\n \t\t    oid_object_info_extended(the_repository, oid, &oi,\n \t\t\t\t\t     OBJECT_INFO_FOR_PREFETCH) < 0)\n \t\t\tcontinue;\n@@ -4648,6 +4648,14 @@ int cmd_pack_objects(int argc,\n \tif (pack_to_stdout != !base_name || argc)\n \t\tusage_with_options(pack_usage, pack_objects_options);\n \n+\tif (path_walk < 0) {\n+\t\tif (use_bitmap_index > 0 ||\n+\t\t    !use_internal_rev_list)\n+\t\t\tpath_walk = 0;\n+\t\telse\n+\t\t\tpath_walk = git_env_bool(\"GIT_TEST_PACK_PATH_WALK\", 0);\n+\t}\n+\n \tif (depth < 0)\n \t\tdepth = 0;\n \tif (depth >= (1 << OE_DEPTH_BITS)) {\ndiff --git a/t/README b/t/README\nindex 53e5b4a71074..ae06e6288150 100644\n--- a/t/README\n+++ b/t/README\n@@ -415,6 +415,10 @@ GIT_TEST_PACK_SPARSE=<boolean> if disabled will default the pack-objects\n builtin to use the non-sparse object walk. This can still be overridden by\n the --sparse command-line argument.\n \n+GIT_TEST_PACK_PATH_WALK=<boolean> if enabled will default the pack-objects\n+builtin to use the path-walk API for the object walk. This can still be\n+overridden by the --no-path-walk command-line argument.\n+\n GIT_TEST_PRELOAD_INDEX=<boolean> exercises the preload-index code path\n by overriding the minimum number of cache entries required per thread.\n \ndiff --git a/t/t0411-clone-from-partial.sh b/t/t0411-clone-from-partial.sh\nindex 196fc617843c..9e6bca56255b 100755\n--- a/t/t0411-clone-from-partial.sh\n+++ b/t/t0411-clone-from-partial.sh\n@@ -59,6 +59,12 @@ test_expect_success 'pack-objects should fetch from promisor remote and execute\n \n test_expect_success 'clone from promisor remote does not lazy-fetch by default' '\n \trm -f script-executed &&\n+\n+\t# The --path-walk feature of \"git pack-objects\" is not\n+\t# compatible with this kind of fetch from an incomplete repo.\n+\tGIT_TEST_PACK_PATH_WALK=0 &&\n+\texport GIT_TEST_PACK_PATH_WALK &&\n+\n \ttest_must_fail git clone evil no-lazy 2>err &&\n \ttest_grep \"lazy fetching disabled\" err &&\n \ttest_path_is_missing script-executed\ndiff --git a/t/t5306-pack-nobase.sh b/t/t5306-pack-nobase.sh\nindex 805d60ff3179..609399d54fbb 100755\n--- a/t/t5306-pack-nobase.sh\n+++ b/t/t5306-pack-nobase.sh\n@@ -59,6 +59,11 @@ test_expect_success 'indirectly clone patch_clone' '\n \t git pull ../.git &&\n \t test $(git rev-parse HEAD) = $B &&\n \n+\t# The --path-walk feature of \"git pack-objects\" is not\n+\t# compatible with this kind of fetch from an incomplete repo.\n+\tGIT_TEST_PACK_PATH_WALK=0 &&\n+\texport GIT_TEST_PACK_PATH_WALK &&\n+\n \t git pull ../patch_clone/.git &&\n \t test $(git rev-parse HEAD) = $C\n \t)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 621bbbdd26ed..e01df807a62d 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -158,8 +158,9 @@ test_bitmap_cases () {\n \t\tls .git/objects/pack/ | grep bitmap >output &&\n \t\ttest_line_count = 1 output &&\n \t\t# verify equivalent packs are generated with/without using bitmap index\n-\t\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa </dev/null) &&\n-\t\tpackbsha1=$(git pack-objects --use-bitmap-index --all packb </dev/null) &&\n+\t\t# Be careful to not use the path-walk option in either case.\n+\t\tpackasha1=$(git pack-objects --no-use-bitmap-index --no-path-walk --all packa </dev/null) &&\n+\t\tpackbsha1=$(git pack-objects --use-bitmap-index --no-path-walk --all packb </dev/null) &&\n \t\tlist_packed_objects packa-$packasha1.idx >packa.objects &&\n \t\tlist_packed_objects packb-$packbsha1.idx >packb.objects &&\n \t\ttest_cmp packa.objects packb.objects\n@@ -388,6 +389,14 @@ test_bitmap_cases () {\n \t\tgit init --bare client.git &&\n \t\t(\n \t\t\tcd client.git &&\n+\n+\t\t\t# This test relies on reusing a delta, but if the\n+\t\t\t# path-walk machinery is engaged, the base object\n+\t\t\t# is considered too small to use during the\n+\t\t\t# dynamic computation, so is not used.\n+\t\t\tGIT_TEST_PACK_PATH_WALK=0 &&\n+\t\t\texport GIT_TEST_PACK_PATH_WALK &&\n+\n \t\t\tgit config transfer.unpackLimit 1 &&\n \t\t\tgit fetch .. delta-reuse-old:delta-reuse-old &&\n \t\t\tgit fetch .. delta-reuse-new:delta-reuse-new &&\ndiff --git a/t/t5316-pack-delta-depth.sh b/t/t5316-pack-delta-depth.sh\nindex 32cf4227451f..167c3a352345 100755\n--- a/t/t5316-pack-delta-depth.sh\n+++ b/t/t5316-pack-delta-depth.sh\n@@ -89,15 +89,18 @@ max_chain() {\n # adjusted (or scrapped if the heuristics have become too unreliable)\n test_expect_success 'packing produces a long delta' '\n \t# Use --window=0 to make sure we are seeing reused deltas,\n-\t# not computing a new long chain.\n-\tpack=$(git pack-objects --all --window=0 </dev/null pack) &&\n+\t# not computing a new long chain. (Also avoid the --path-walk\n+\t# option as it may break delta chains.)\n+\tpack=$(git pack-objects --all --window=0 --no-path-walk </dev/null pack) &&\n \techo 9 >expect &&\n \tmax_chain pack-$pack.pack >actual &&\n \ttest_cmp expect actual\n '\n \n test_expect_success '--depth limits depth' '\n-\tpack=$(git pack-objects --all --depth=5 </dev/null pack) &&\n+\t# Avoid --path-walk to avoid breaking delta chains across path\n+\t# boundaries.\n+\tpack=$(git pack-objects --all --depth=5 --no-path-walk </dev/null pack) &&\n \techo 5 >expect &&\n \tmax_chain pack-$pack.pack >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t5332-multi-pack-reuse.sh b/t/t5332-multi-pack-reuse.sh\nindex 57cad7708f80..395d09444ced 100755\n--- a/t/t5332-multi-pack-reuse.sh\n+++ b/t/t5332-multi-pack-reuse.sh\n@@ -7,6 +7,13 @@ test_description='pack-objects multi-pack reuse'\n \n GIT_TEST_MULTI_PACK_INDEX=0\n GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n+\n+# The --path-walk option does not consider the preferred pack\n+# at all for reusing deltas, so this variable changes the\n+# behavior of this test, if enabled.\n+GIT_TEST_PACK_PATH_WALK=0\n+export GIT_TEST_PACK_PATH_WALK\n+\n objdir=.git/objects\n packdir=$objdir/pack\n \ndiff --git a/t/t5530-upload-pack-error.sh b/t/t5530-upload-pack-error.sh\nindex 558eedf25a4c..8eb6fea839a6 100755\n--- a/t/t5530-upload-pack-error.sh\n+++ b/t/t5530-upload-pack-error.sh\n@@ -34,6 +34,12 @@ test_expect_success 'upload-pack fails due to error in pack-objects packing' '\n \thexsz=$(test_oid hexsz) &&\n \tprintf \"%04xwant %s\\n00000009done\\n0000\" \\\n \t\t$(($hexsz + 10)) $head >input &&\n+\n+\t# The current implementation of path-walk causes a different\n+\t# error message. This will be changed by a future refactoring.\n+\tGIT_TEST_PACK_PATH_WALK=0 &&\n+\texport GIT_TEST_PACK_PATH_WALK &&\n+\n \ttest_must_fail git upload-pack . <input >/dev/null 2>output.err &&\n \ttest_grep \"unable to read\" output.err &&\n \ttest_grep \"pack-objects died\" output.err\ndiff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh\nindex c562bad042ab..ab76d4b6dc41 100755\n--- a/t/t7406-submodule-update.sh\n+++ b/t/t7406-submodule-update.sh\n@@ -1095,12 +1095,15 @@ test_expect_success 'submodule update --quiet passes quietness to fetch with a s\n \t(cd super5 &&\n \t # This test var can mess with the stderr output checked in this test.\n \t GIT_TEST_NAME_HASH_VERSION=1 \\\n+\t GIT_TEST_PACK_PATH_WALK=0 \\\n \t\tgit submodule update --quiet --init --depth=1 submodule3 >out 2>err &&\n \t test_must_be_empty out &&\n \t test_must_be_empty err\n \t) &&\n \tgit clone super4 super6 &&\n \t(cd super6 &&\n+\t # This test variable will create a \"warning\" message to stderr\n+\t GIT_TEST_PACK_PATH_WALK=0 \\\n \t git submodule update --init --depth=1 submodule3 >out 2>err &&\n \t test_file_not_empty out &&\n \t test_file_not_empty err\n-- \ngitgitgadget\n\n"},{"id":"518282","messageId":"19c6651183a7daab4af7994f669aedc22f0a6746.1747419124.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v3.git.1747419124.gitgitgadget@gmail.com","subject":"[PATCH v3 06/13] t5538: add tests to confirm deltas in shallow pushes","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-05-16T18:11:56Z","receivedAt":"2025-05-16T18:12:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nIt can be notoriously difficult to detect if delta bases are being\ncomputed properly during 'git push'. Construct an example where it will\nmake a kilobyte worth of difference when a delta base is not found. We\ncan then use the progress indicators to distinguish between bytes and\nKiB depending on whether the delta base is found and used.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n t/t5538-push-shallow.sh | 33 +++++++++++++++++++++++++++++++++\n 1 file changed, 33 insertions(+)\n\ndiff --git a/t/t5538-push-shallow.sh b/t/t5538-push-shallow.sh\nindex e91fcc173e81..c406f9d7ed5f 100755\n--- a/t/t5538-push-shallow.sh\n+++ b/t/t5538-push-shallow.sh\n@@ -123,4 +123,37 @@ EOF\n \tgit cat-file blob $(echo 1|git hash-object --stdin) >/dev/null\n \t)\n '\n+\n+test_expect_success 'push new commit from shallow clone has correct object count' '\n+\tgit init origin &&\n+\ttest_commit -C origin a &&\n+\ttest_commit -C origin b &&\n+\n+\tgit clone --depth=1 \"file://$(pwd)/origin\" client &&\n+\tgit -C client checkout -b topic &&\n+\tgit -C client commit --allow-empty -m \"empty\" &&\n+\tGIT_PROGRESS_DELAY=0 git -C client push --progress origin topic 2>err &&\n+\ttest_grep \"Enumerating objects: 1, done.\" err\n+'\n+\n+test_expect_success 'push new commit from shallow clone has good deltas' '\n+\tgit init base &&\n+\ttest_seq 1 999 >base/a &&\n+\ttest_commit -C base initial &&\n+\tgit -C base add a &&\n+\tgit -C base commit -m \"big a\" &&\n+\n+\tgit clone --depth=1 \"file://$(pwd)/base\" deltas &&\n+\tgit -C deltas checkout -b deltas &&\n+\ttest_seq 1 1000 >deltas/a &&\n+\tgit -C deltas commit -a -m \"bigger a\" &&\n+\tGIT_PROGRESS_DELAY=0 git -C deltas push --progress origin deltas 2>err &&\n+\n+\ttest_grep \"Enumerating objects: 5, done\" err &&\n+\n+\t# If the delta base is found, then this message uses \"bytes\".\n+\t# If the delta base is not found, then this message uses \"KiB\".\n+\ttest_grep \"Writing objects: .* bytes\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"518283","messageId":"7f9e6a077bc43aae55b2eb9ae5b4b10385cef11f.1747419124.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v3.git.1747419124.gitgitgadget@gmail.com","subject":"[PATCH v3 07/13] repack: add --path-walk option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-05-16T18:11:57Z","receivedAt":"2025-05-16T18:12:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nSince 'git pack-objects' supports a --path-walk option, allow passing it\nthrough in 'git repack'. This presents interesting testing opportunities for\ncomparing the different repacking strategies against each other.\n\nAdd the --path-walk option to the performance tests in p5313.\n\nFor the microsoft/fluentui repo [1] checked out at a specific commit [2],\nthe --path-walk tests in p5313 look like this:\n\nTest                                                     this tree\n-------------------------------------------------------------------------\n5313.18: thin pack with --path-walk                      0.08(0.06+0.02)\n5313.19: thin pack size with --path-walk                           18.4K\n5313.20: big pack with --path-walk                       2.10(7.80+0.26)\n5313.21: big pack size with --path-walk                            19.8M\n5313.22: shallow fetch pack with --path-walk             1.62(3.38+0.17)\n5313.23: shallow pack size with --path-walk                        33.6M\n5313.24: repack with --path-walk                         81.29(96.08+0.71)\n5313.25: repack size with --path-walk                             142.5M\n\n[1] https://github.com/microsoft/fluentui\n[2] e70848ebac1cd720875bccaa3026f4a9ed700e08\n\nAlong with the earlier tests in p5313, I'll instead reformat the\ncomparison as follows:\n\nRepack Method    Pack Size       Time\n---------------------------------------\nHash v1             439.4M      87.24s\nHash v2             161.7M      21.51s\nPath Walk           142.5M      81.29s\n\nThere are a few things to notice here:\n\n 1. The benefits of --name-hash-version=2 over --name-hash-version=1 are\n    significant, but --path-walk still compresses better than that\n    option.\n\n 2. The --path-walk command is still using --name-hash-version=1 for the\n    second pass of delta computation, using the increased name hash\n    collisions as a potential method for opportunistic compression on\n    top of the path-focused compression.\n\n 3. The --path-walk algorithm is currently sequential and does not use\n    multiple threads for delta compression. Threading will be\n    implemented in a future change so the computation time will improve\n    to better compete in this metric.\n\nThere are small benefits in size for my copy of the Git repository:\n\nRepack Method    Pack Size       Time\n---------------------------------------\nHash v1             248.8M      30.44s\nHash v2             249.0M      30.15s\nPath Walk           213.2M     142.50s\n\nAs well as in the nodejs/node repository [3]:\n\nRepack Method    Pack Size       Time\n---------------------------------------\nHash v1             739.9M      71.18s\nHash v2             764.6M      67.82s\nPath Walk           698.1M     208.10s\n\n[3] https://github.com/nodejs/node\n\nThis benefit also repeats in my copy of the Linux kernel repository:\n\nRepack Method    Pack Size       Time\n---------------------------------------\nHash v1               2.5G     554.41s\nHash v2               2.5G     549.62s\nPath Walk             2.2G    1562.36s\n\nIt is important to see that even when the repository shape does not have\nmany name-hash collisions, there is a slight space boost to be found\nusing this method.\n\nAs this repacking strategy was released in Git for Windows 2.47.0, some\nusers have reported cases where the --path-walk compression is slightly\nworse than the --name-hash-version=2 option. In those cases, it may be\nbeneficial to combine the two options. However, there has not been a\nreleased version of Git that has both options and I don't have access to\nthese repos for testing.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-repack.adoc |  5 ++++-\n builtin/repack.c              |  7 ++++++-\n t/perf/p5313-pack-objects.sh  | 18 ++++++++----------\n 3 files changed, 18 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/git-repack.adoc b/Documentation/git-repack.adoc\nindex 5852a5c97368..aa1bc081e50a 100644\n--- a/Documentation/git-repack.adoc\n+++ b/Documentation/git-repack.adoc\n@@ -11,7 +11,7 @@ SYNOPSIS\n [verse]\n 'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n \t[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n-\t[--write-midx] [--name-hash-version=<n>]\n+\t[--write-midx] [--name-hash-version=<n>] [--path-walk]\n \n DESCRIPTION\n -----------\n@@ -255,6 +255,9 @@ linkgit:git-multi-pack-index[1]).\n \tProvide this argument to the underlying `git pack-objects` process.\n \tSee linkgit:git-pack-objects[1] for full details.\n \n+--path-walk::\n+\tPass the `--path-walk` option to the underlying `git pack-objects`\n+\tprocess. See linkgit:git-pack-objects[1] for full details.\n \n CONFIGURATION\n -------------\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 75e3752353a2..d7f798280c0e 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -43,7 +43,7 @@ static char *packdir, *packtmp_name, *packtmp;\n static const char *const git_repack_usage[] = {\n \tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n \t   \"[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\\n\"\n-\t   \"[--write-midx] [--name-hash-version=<n>]\"),\n+\t   \"[--write-midx] [--name-hash-version=<n>] [--path-walk]\"),\n \tNULL\n };\n \n@@ -63,6 +63,7 @@ struct pack_objects_args {\n \tint quiet;\n \tint local;\n \tint name_hash_version;\n+\tint path_walk;\n \tstruct list_objects_filter_options filter_options;\n };\n \n@@ -313,6 +314,8 @@ static void prepare_pack_objects(struct child_process *cmd,\n \t\tstrvec_pushf(&cmd->args, \"--no-reuse-object\");\n \tif (args->name_hash_version)\n \t\tstrvec_pushf(&cmd->args, \"--name-hash-version=%d\", args->name_hash_version);\n+\tif (args->path_walk)\n+\t\tstrvec_pushf(&cmd->args, \"--path-walk\");\n \tif (args->local)\n \t\tstrvec_push(&cmd->args,  \"--local\");\n \tif (args->quiet)\n@@ -1212,6 +1215,8 @@ int cmd_repack(int argc,\n \t\t\t\tN_(\"pass --no-reuse-object to git-pack-objects\")),\n \t\tOPT_INTEGER(0, \"name-hash-version\", &po_args.name_hash_version,\n \t\t\t\tN_(\"specify the name hash version to use for grouping similar objects by path\")),\n+\t\tOPT_BOOL(0, \"path-walk\", &po_args.path_walk,\n+\t\t\t\tN_(\"pass --path-walk to git-pack-objects\")),\n \t\tOPT_NEGBIT('n', NULL, &run_update_server_info,\n \t\t\t\tN_(\"do not run git-update-server-info\"), 1),\n \t\tOPT__QUIET(&po_args.quiet, N_(\"be quiet\")),\ndiff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh\nindex cd6dd3abb710..98748b0e203a 100755\n--- a/t/perf/p5313-pack-objects.sh\n+++ b/t/perf/p5313-pack-objects.sh\n@@ -55,23 +55,21 @@ test_all_with_args () {\n \ttest_size \"shallow pack size with $parameter\" '\n \t\ttest_file_size out\n \t'\n-}\n-\n-for version in 1 2\n-do\n-\texport version\n-\n-\ttest_all_with_args --name-hash-version=$version\n \n-\ttest_perf \"repack with --name-hash-version=$version\" '\n-\t\tgit repack -adf --name-hash-version=$version\n+\ttest_perf \"repack with $parameter\" '\n+\t\tgit repack -adf $parameter\n \t'\n \n-\ttest_size \"repack size with --name-hash-version=$version\" '\n+\ttest_size \"repack size with $parameter\" '\n \t\tgitdir=$(git rev-parse --git-dir) &&\n \t\tpack=$(ls $gitdir/objects/pack/pack-*.pack) &&\n \t\ttest_file_size \"$pack\"\n \t'\n+}\n+\n+for version in 1 2\n+do\n+\ttest_all_with_args --name-hash-version=$version\n done\n \n test_all_with_args --path-walk\n-- \ngitgitgadget\n\n"},{"id":"518284","messageId":"b98e71d6d549d916f3d827e1eb3da6ac7391392b.1747419124.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v3.git.1747419124.gitgitgadget@gmail.com","subject":"[PATCH v3 08/13] pack-objects: enable --path-walk via config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-05-16T18:11:58Z","receivedAt":"2025-05-16T18:12:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nUsers may want to enable the --path-walk option for 'git pack-objects' by\ndefault, especially underneath commands like 'git push' or 'git repack'.\n\nThis should be limited to client repositories, since the --path-walk option\ndisables bitmap walks, so would be bad to include in Git servers when\nserving fetches and clones. There is potential that it may be helpful to\nconsider when repacking the repository, to take advantage of improved deltas\nacross historical versions of the same files.\n\nMuch like how \"pack.useSparse\" was introduced and included in\n\"feature.experimental\" before being enabled by default, use the repository\nsettings infrastructure to make the new \"pack.usePathWalk\" config enabled by\n\"feature.experimental\" and \"feature.manyFiles\".\n\nIn order to test that this config works, add a new trace2 region around\nthe path walk code that can be checked by a 'git push' command.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/config/feature.adoc |  4 ++++\n Documentation/config/pack.adoc    |  4 ++++\n builtin/pack-objects.c            | 11 ++++++++++-\n repo-settings.c                   |  3 +++\n repo-settings.h                   |  1 +\n t/t5516-fetch-push.sh             | 10 ++++++++++\n 6 files changed, 32 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/feature.adoc b/Documentation/config/feature.adoc\nindex f061b64b7484..cb49ff2604a6 100644\n--- a/Documentation/config/feature.adoc\n+++ b/Documentation/config/feature.adoc\n@@ -20,6 +20,10 @@ walking fewer objects.\n +\n * `pack.allowPackReuse=multi` may improve the time it takes to create a pack by\n reusing objects from multiple packs instead of just one.\n++\n+* `pack.usePathWalk` may speed up packfile creation and make the packfiles be\n+significantly smaller in the presence of certain filename collisions with Git's\n+default name-hash.\n \n feature.manyFiles::\n \tEnable config options that optimize for repos with many files in the\ndiff --git a/Documentation/config/pack.adoc b/Documentation/config/pack.adoc\nindex da527377fafc..75402d5579d4 100644\n--- a/Documentation/config/pack.adoc\n+++ b/Documentation/config/pack.adoc\n@@ -155,6 +155,10 @@ pack.useSparse::\n \tcommits contain certain types of direct renames. Default is\n \t`true`.\n \n+pack.usePathWalk::\n+\tEnable the `--path-walk` option by default for `git pack-objects`\n+\tprocesses. See linkgit:git-pack-objects[1] for full details.\n+\n pack.preferBitmapTips::\n \tWhen selecting which commits will receive bitmaps, prefer a\n \tcommit at the tip of any reference that is a suffix of any value\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 4fd88476dd29..bdd20c074a9b 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -44,6 +44,7 @@\n #include \"blob.h\"\n #include \"tree.h\"\n #include \"path-walk.h\"\n+#include \"trace2.h\"\n \n /*\n  * Objects we are going to pack are collected in the `to_pack` structure.\n@@ -4283,6 +4284,7 @@ static void get_object_list_path_walk(struct rev_info *revs)\n {\n \tstruct path_walk_info info = PATH_WALK_INFO_INIT;\n \tunsigned int processed = 0;\n+\tint result;\n \n \tinfo.revs = revs;\n \tinfo.path_fn = add_objects_by_path;\n@@ -4296,7 +4298,11 @@ static void get_object_list_path_walk(struct rev_info *revs)\n \t */\n \tinfo.prune_all_uninteresting = sparse;\n \n-\tif (walk_objects_by_path(&info))\n+\ttrace2_region_enter(\"pack-objects\", \"path-walk\", revs->repo);\n+\tresult = walk_objects_by_path(&info);\n+\ttrace2_region_leave(\"pack-objects\", \"path-walk\", revs->repo);\n+\n+\tif (result)\n \t\tdie(_(\"failed to pack objects via path-walk\"));\n }\n \n@@ -4652,6 +4658,9 @@ int cmd_pack_objects(int argc,\n \t\tif (use_bitmap_index > 0 ||\n \t\t    !use_internal_rev_list)\n \t\t\tpath_walk = 0;\n+\t\telse if (the_repository->gitdir &&\n+\t\t\t the_repository->settings.pack_use_path_walk)\n+\t\t\tpath_walk = 1;\n \t\telse\n \t\t\tpath_walk = git_env_bool(\"GIT_TEST_PACK_PATH_WALK\", 0);\n \t}\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 67e9cfd2e63d..9b5595c708e6 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -47,11 +47,13 @@ void prepare_repo_settings(struct repository *r)\n \t\tr->settings.fetch_negotiation_algorithm = FETCH_NEGOTIATION_SKIPPING;\n \t\tr->settings.pack_use_bitmap_boundary_traversal = 1;\n \t\tr->settings.pack_use_multi_pack_reuse = 1;\n+\t\tr->settings.pack_use_path_walk = 1;\n \t}\n \tif (manyfiles) {\n \t\tr->settings.index_version = 4;\n \t\tr->settings.index_skip_hash = 1;\n \t\tr->settings.core_untracked_cache = UNTRACKED_CACHE_WRITE;\n+\t\tr->settings.pack_use_path_walk = 1;\n \t}\n \n \t/* Commit graph config or default, does not cascade (simple) */\n@@ -66,6 +68,7 @@ void prepare_repo_settings(struct repository *r)\n \n \t/* Boolean config or default, does not cascade (simple)  */\n \trepo_cfg_bool(r, \"pack.usesparse\", &r->settings.pack_use_sparse, 1);\n+\trepo_cfg_bool(r, \"pack.usepathwalk\", &r->settings.pack_use_path_walk, 0);\n \trepo_cfg_bool(r, \"core.multipackindex\", &r->settings.core_multi_pack_index, 1);\n \trepo_cfg_bool(r, \"index.sparse\", &r->settings.sparse_index, 0);\n \trepo_cfg_bool(r, \"index.skiphash\", &r->settings.index_skip_hash, r->settings.index_skip_hash);\ndiff --git a/repo-settings.h b/repo-settings.h\nindex ddc11967e015..a31decad2211 100644\n--- a/repo-settings.h\n+++ b/repo-settings.h\n@@ -56,6 +56,7 @@ struct repo_settings {\n \tenum untracked_cache_setting core_untracked_cache;\n \n \tint pack_use_sparse;\n+\tint pack_use_path_walk;\n \tenum fetch_negotiation_setting fetch_negotiation_algorithm;\n \n \tint core_multi_pack_index;\ndiff --git a/t/t5516-fetch-push.sh b/t/t5516-fetch-push.sh\nindex 85ed049627d2..33cd186cd7ca 100755\n--- a/t/t5516-fetch-push.sh\n+++ b/t/t5516-fetch-push.sh\n@@ -1907,4 +1907,14 @@ test_expect_success 'push with config push.useBitmaps' '\n \t\t--thin --delta-base-offset -q --no-use-bitmap-index <false\n '\n \n+test_expect_success 'push with config pack.usePathWalk=true' '\n+\tmk_test testrepo heads/main &&\n+\tgit checkout main &&\n+\ttest_config pack.usePathWalk true &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/path-walk.txt\" \\\n+\tgit push --quiet testrepo main:test &&\n+\n+\ttest_region pack-objects path-walk path-walk.txt\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"518285","messageId":"8e1d141a1f87dcdfd967c239a68a864380f39d02.1747419124.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v3.git.1747419124.gitgitgadget@gmail.com","subject":"[PATCH v3 09/13] scalar: enable path-walk during push via config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-05-16T18:11:59Z","receivedAt":"2025-05-16T18:12:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nRepositories registered with Scalar are expected to be client-only\nrepositories that are rather large. This means that they are more likely to\nbe good candidates for using the --path-walk option when running 'git\npack-objects', especially under the hood of 'git push'. Enable this config\nin Scalar repositories.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n scalar.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/scalar.c b/scalar.c\nindex da42b4be0cc9..bf638fa34b80 100644\n--- a/scalar.c\n+++ b/scalar.c\n@@ -170,6 +170,7 @@ static int set_recommended_config(int reconfigure)\n \t\t{ \"core.autoCRLF\", \"false\" },\n \t\t{ \"core.safeCRLF\", \"false\" },\n \t\t{ \"fetch.showForcedUpdates\", \"false\" },\n+\t\t{ \"pack.usePathWalk\", \"true\" },\n \t\t{ NULL, NULL },\n \t};\n \tint i;\n-- \ngitgitgadget\n\n"},{"id":"518286","messageId":"2c1d847987285ab98abb6a21d7dcd0bbed750d75.1747419124.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v3.git.1747419124.gitgitgadget@gmail.com","subject":"[PATCH v3 10/13] pack-objects: refactor path-walk delta phase","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-05-16T18:12:00Z","receivedAt":"2025-05-16T18:12:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nPreviously, the --path-walk option to 'git pack-objects' would compute\ndeltas inline with the path-walk logic. This would make the progress\nindicator look like it is taking a long time to enumerate objects, and\nthen very quickly computed deltas.\n\nInstead of computing deltas on each region of objects organized by tree,\nstore a list of regions corresponding to these groups. These can later\nbe pulled from the list for delta compression before doing the \"global\"\ndelta search.\n\nThis presents a new progress indicator that can be used in tests to\nverify that this stage is happening.\n\nThe current implementation is not integrated with threads, but we are\nsetting it up to arrive in the next change.\n\nSince we do not attempt to sort objects by size until after exploring\nall trees, we can remove the previous change to t5530 due to a different\nerror message appearing first.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c       | 83 +++++++++++++++++++++++++-----------\n pack-objects.h               | 12 ++++++\n t/t5300-pack-object.sh       |  8 +++-\n t/t5530-upload-pack-error.sh |  6 ---\n 4 files changed, 75 insertions(+), 34 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex bdd20c074a9b..c7bf3fbc0267 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3237,6 +3237,51 @@ static int should_attempt_deltas(struct object_entry *entry)\n \treturn 1;\n }\n \n+static void find_deltas_for_region(struct object_entry *list,\n+\t\t\t\t   struct packing_region *region,\n+\t\t\t\t   unsigned int *processed)\n+{\n+\tstruct object_entry **delta_list;\n+\tunsigned int delta_list_nr = 0;\n+\n+\tALLOC_ARRAY(delta_list, region->nr);\n+\tfor (size_t i = 0; i < region->nr; i++) {\n+\t\tstruct object_entry *entry = list + region->start + i;\n+\t\tif (should_attempt_deltas(entry))\n+\t\t\tdelta_list[delta_list_nr++] = entry;\n+\t}\n+\n+\tQSORT(delta_list, delta_list_nr, type_size_sort);\n+\tfind_deltas(delta_list, &delta_list_nr, window, depth, processed);\n+\tfree(delta_list);\n+}\n+\n+static void find_deltas_by_region(struct object_entry *list,\n+\t\t\t\t  struct packing_region *regions,\n+\t\t\t\t  size_t start, size_t nr)\n+{\n+\tunsigned int processed = 0;\n+\tsize_t progress_nr;\n+\n+\tif (!nr)\n+\t\treturn;\n+\n+\tprogress_nr = regions[nr - 1].start + regions[nr - 1].nr;\n+\n+\tif (progress)\n+\t\tprogress_state = start_progress(the_repository,\n+\t\t\t\t\t\t_(\"Compressing objects by path\"),\n+\t\t\t\t\t\tprogress_nr);\n+\n+\twhile (nr--)\n+\t\tfind_deltas_for_region(list,\n+\t\t\t\t       &regions[start++],\n+\t\t\t\t       &processed);\n+\n+\tdisplay_progress(progress_state, progress_nr);\n+\tstop_progress(&progress_state);\n+}\n+\n static void prepare_pack(int window, int depth)\n {\n \tstruct object_entry **delta_list;\n@@ -3261,6 +3306,10 @@ static void prepare_pack(int window, int depth)\n \tif (!to_pack.nr_objects || !window || !depth)\n \t\treturn;\n \n+\tif (path_walk)\n+\t\tfind_deltas_by_region(to_pack.objects, to_pack.regions,\n+\t\t\t\t      0, to_pack.nr_regions);\n+\n \tALLOC_ARRAY(delta_list, to_pack.nr_objects);\n \tnr_deltas = n = 0;\n \n@@ -4214,10 +4263,8 @@ static int add_objects_by_path(const char *path,\n \t\t\t       enum object_type type,\n \t\t\t       void *data)\n {\n-\tstruct object_entry **delta_list = NULL;\n \tsize_t oe_start = to_pack.nr_objects;\n \tsize_t oe_end;\n-\tunsigned int sub_list_nr;\n \tunsigned int *processed = data;\n \n \t/*\n@@ -4250,33 +4297,17 @@ static int add_objects_by_path(const char *path,\n \tif (oe_end == oe_start || !window)\n \t\treturn 0;\n \n-\tsub_list_nr = 0;\n-\tif (oe_end > oe_start)\n-\t\tALLOC_ARRAY(delta_list, oe_end - oe_start);\n+\tALLOC_GROW(to_pack.regions,\n+\t\t   to_pack.nr_regions + 1,\n+\t\t   to_pack.nr_regions_alloc);\n \n-\tfor (size_t i = 0; i < oe_end - oe_start; i++) {\n-\t\tstruct object_entry *entry = to_pack.objects + oe_start + i;\n+\tto_pack.regions[to_pack.nr_regions].start = oe_start;\n+\tto_pack.regions[to_pack.nr_regions].nr = oe_end - oe_start;\n+\tto_pack.nr_regions++;\n \n-\t\tif (!should_attempt_deltas(entry))\n-\t\t\tcontinue;\n+\t*processed += oids->nr;\n+\tdisplay_progress(progress_state, *processed);\n \n-\t\tdelta_list[sub_list_nr++] = entry;\n-\t}\n-\n-\t/*\n-\t * Find delta bases among this list of objects that all match the same\n-\t * path. This causes the delta compression to be interleaved in the\n-\t * object walk, which can lead to confusing progress indicators. This is\n-\t * also incompatible with threaded delta calculations. In the future,\n-\t * consider creating a list of regions in the full to_pack.objects array\n-\t * that could be picked up by the threaded delta computation.\n-\t */\n-\tif (sub_list_nr && window) {\n-\t\tQSORT(delta_list, sub_list_nr, type_size_sort);\n-\t\tfind_deltas(delta_list, &sub_list_nr, window, depth, processed);\n-\t}\n-\n-\tfree(delta_list);\n \treturn 0;\n }\n \ndiff --git a/pack-objects.h b/pack-objects.h\nindex d73e3843c92e..51e1ff6b95bf 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -119,11 +119,23 @@ struct object_entry {\n \tunsigned ext_base:1; /* delta_idx points outside packlist */\n };\n \n+/**\n+ * A packing region is a section of the packing_data.objects array\n+ * as given by a starting index and a number of elements.\n+ */\n+struct packing_region {\n+\tsize_t start;\n+\tsize_t nr;\n+};\n+\n struct packing_data {\n \tstruct repository *repo;\n \tstruct object_entry *objects;\n \tuint32_t nr_objects, nr_alloc;\n \n+\tstruct packing_region *regions;\n+\tsize_t nr_regions, nr_regions_alloc;\n+\n \tint32_t *index;\n \tuint32_t index_size;\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 16420d128639..c8df6afd7844 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -725,7 +725,9 @@ test_expect_success '--name-hash-version=2 and --write-bitmap-index are incompat\n \n test_expect_success '--path-walk pack everything' '\n \tgit -C server rev-parse HEAD >in &&\n-\tgit -C server pack-objects --stdout --revs --path-walk <in >out.pack &&\n+\tGIT_PROGRESS_DELAY=0 git -C server pack-objects \\\n+\t\t--stdout --revs --path-walk --progress <in >out.pack 2>err &&\n+\tgrep \"Compressing objects by path\" err &&\n \tgit -C server index-pack --stdin <out.pack\n '\n \n@@ -734,7 +736,9 @@ test_expect_success '--path-walk thin pack' '\n \t$(git -C server rev-parse HEAD)\n \t^$(git -C server rev-parse HEAD~2)\n \tEOF\n-\tgit -C server pack-objects --thin --stdout --revs --path-walk <in >out.pack &&\n+\tGIT_PROGRESS_DELAY=0 git -C server pack-objects \\\n+\t\t--thin --stdout --revs --path-walk --progress <in >out.pack 2>err &&\n+\tgrep \"Compressing objects by path\" err &&\n \tgit -C server index-pack --fix-thin --stdin <out.pack\n '\n \ndiff --git a/t/t5530-upload-pack-error.sh b/t/t5530-upload-pack-error.sh\nindex 8eb6fea839a6..558eedf25a4c 100755\n--- a/t/t5530-upload-pack-error.sh\n+++ b/t/t5530-upload-pack-error.sh\n@@ -34,12 +34,6 @@ test_expect_success 'upload-pack fails due to error in pack-objects packing' '\n \thexsz=$(test_oid hexsz) &&\n \tprintf \"%04xwant %s\\n00000009done\\n0000\" \\\n \t\t$(($hexsz + 10)) $head >input &&\n-\n-\t# The current implementation of path-walk causes a different\n-\t# error message. This will be changed by a future refactoring.\n-\tGIT_TEST_PACK_PATH_WALK=0 &&\n-\texport GIT_TEST_PACK_PATH_WALK &&\n-\n \ttest_must_fail git upload-pack . <input >/dev/null 2>output.err &&\n \ttest_grep \"unable to read\" output.err &&\n \ttest_grep \"pack-objects died\" output.err\n-- \ngitgitgadget\n\n"},{"id":"518287","messageId":"623913f284eafae7dfb8b87f1183ef141cead62a.1747419124.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v3.git.1747419124.gitgitgadget@gmail.com","subject":"[PATCH v3 11/13] pack-objects: thread the path-based compression","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-05-16T18:12:01Z","receivedAt":"2025-05-16T18:12:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nAdapting the implementation of ll_find_deltas(), create a threaded\nversion of the --path-walk compression step in 'git pack-objects'.\n\nThis involves adding a 'regions' member to the thread_params struct,\nallowing each thread to own a section of paths. We can simplify the way\njobs are split because there is no value in extending the batch based on\nname-hash the way sections of the object entry array are attempted to be\ngrouped. We re-use the 'list_size' and 'remaining' items for the purpose\nof borrowing work in progress from other \"victim\" threads when a thread\nhas finished its batch of work more quickly.\n\nUsing the Git repository as a test repo, the p5313 performance test\nshows that the resulting size of the repo is the same, but the threaded\nimplementation gives gains of varying degrees depending on the number of\nobjects being packed. (This was tested on a 16-core machine.)\n\nTest                        HEAD~1      HEAD\n---------------------------------------------------\n5313.20: big pack             2.38      1.99 -16.4%\n5313.21: big pack size       16.1M     16.0M  -0.2%\n5313.24: repack             107.32     45.41 -57.7%\n5313.25: repack size        213.3M    213.2M  -0.0%\n\n(Test output is formatted to better fit in message.)\n\nThis ~60% reduction in 'git repack --path-walk' time is typical across\nall repos I used for testing. What is interesting is to compare when the\noverall time improves enough to outperform the --name-hash-version=1\ncase. These time improvements correlate with repositories with data\nshapes that significantly improve their data size as well. The\n--path-walk feature frequently takes longer than --name-hash-version=2,\ntrading some extra computation for some additional compression. The\nnatural place where this additional computation comes from is the two\ncompression passes that --path-walk takes, though the first pass is\nnaturally faster due to the path boundaries avoiding a number of delta\ncompression attempts.\n\nFor example, the microsoft/fluentui repo has significant size reduction\nfrom --name-hash-version=1 to --name-hash-version=2 followed by further\nimprovements with --path-walk. The threaded computation makes\n--path-walk more competitive in time compared to --name-hash-version=2,\nthough still ~31% more expensive in that metric.\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                439.4M      87.24s\nHash v2                161.7M      21.51s\nPath Walk (Before)     142.5M      81.29s\nPath Walk (After)      142.5M      28.16s\n\nSimilar results hold for the Git repository:\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                248.8M      30.44s\nHash v2                249.0M      30.15s\nPath Walk (Before)     213.2M     142.50s\nPath Walk (After)      213.3M      45.41s\n\n...as well as the nodejs/node repository:\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                739.9M      71.18s\nHash v2                764.6M      67.82s\nPath Walk (Before)     698.1M     208.10s\nPath Walk (After)      698.0M      75.10s\n\nFinally, the Linux kernel repository is a good test for this repacking\ntime change, even though the space savings is more subtle:\n\nRepack Method       Pack Size       Time\n------------------------------------------\nHash v1                  2.5G     554.41s\nHash v2                  2.5G     549.62s\nPath Walk (before)       2.2G    1562.36s\nPath Walk (before)       2.2G     559.00s\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c | 166 ++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 164 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex c7bf3fbc0267..df3cca0a668a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2965,6 +2965,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,\n struct thread_params {\n \tpthread_t thread;\n \tstruct object_entry **list;\n+\tstruct packing_region *regions;\n \tunsigned list_size;\n \tunsigned remaining;\n \tint window;\n@@ -3282,6 +3283,167 @@ static void find_deltas_by_region(struct object_entry *list,\n \tstop_progress(&progress_state);\n }\n \n+static void *threaded_find_deltas_by_path(void *arg)\n+{\n+\tstruct thread_params *me = arg;\n+\n+\tprogress_lock();\n+\twhile (me->remaining) {\n+\t\twhile (me->remaining) {\n+\t\t\tprogress_unlock();\n+\t\t\tfind_deltas_for_region(to_pack.objects,\n+\t\t\t\t\t       me->regions,\n+\t\t\t\t\t       me->processed);\n+\t\t\tprogress_lock();\n+\t\t\tme->remaining--;\n+\t\t\tme->regions++;\n+\t\t}\n+\n+\t\tme->working = 0;\n+\t\tpthread_cond_signal(&progress_cond);\n+\t\tprogress_unlock();\n+\n+\t\t/*\n+\t\t * We must not set ->data_ready before we wait on the\n+\t\t * condition because the main thread may have set it to 1\n+\t\t * before we get here. In order to be sure that new\n+\t\t * work is available if we see 1 in ->data_ready, it\n+\t\t * was initialized to 0 before this thread was spawned\n+\t\t * and we reset it to 0 right away.\n+\t\t */\n+\t\tpthread_mutex_lock(&me->mutex);\n+\t\twhile (!me->data_ready)\n+\t\t\tpthread_cond_wait(&me->cond, &me->mutex);\n+\t\tme->data_ready = 0;\n+\t\tpthread_mutex_unlock(&me->mutex);\n+\n+\t\tprogress_lock();\n+\t}\n+\tprogress_unlock();\n+\t/* leave ->working 1 so that this doesn't get more work assigned */\n+\treturn NULL;\n+}\n+\n+static void ll_find_deltas_by_region(struct object_entry *list,\n+\t\t\t\t     struct packing_region *regions,\n+\t\t\t\t     uint32_t start, uint32_t nr)\n+{\n+\tstruct thread_params *p;\n+\tint i, ret, active_threads = 0;\n+\tunsigned int processed = 0;\n+\tuint32_t progress_nr;\n+\tinit_threaded_search();\n+\n+\tif (!nr)\n+\t\treturn;\n+\n+\tprogress_nr =  regions[nr - 1].start + regions[nr - 1].nr;\n+\tif (delta_search_threads <= 1) {\n+\t\tfind_deltas_by_region(list, regions, start, nr);\n+\t\tcleanup_threaded_search();\n+\t\treturn;\n+\t}\n+\n+\tif (progress > pack_to_stdout)\n+\t\tfprintf_ln(stderr,\n+\t\t\t   Q_(\"Path-based delta compression using up to %d thread\",\n+\t\t\t      \"Path-based delta compression using up to %d threads\",\n+\t\t\t      delta_search_threads),\n+\t\t\t   delta_search_threads);\n+\tCALLOC_ARRAY(p, delta_search_threads);\n+\n+\tif (progress)\n+\t\tprogress_state = start_progress(the_repository,\n+\t\t\t\t\t\t_(\"Compressing objects by path\"),\n+\t\t\t\t\t\tprogress_nr);\n+\t/* Partition the work amongst work threads. */\n+\tfor (i = 0; i < delta_search_threads; i++) {\n+\t\tunsigned sub_size = nr / (delta_search_threads - i);\n+\n+\t\tp[i].window = window;\n+\t\tp[i].depth = depth;\n+\t\tp[i].processed = &processed;\n+\t\tp[i].working = 1;\n+\t\tp[i].data_ready = 0;\n+\n+\t\tp[i].regions = regions;\n+\t\tp[i].list_size = sub_size;\n+\t\tp[i].remaining = sub_size;\n+\n+\t\tregions += sub_size;\n+\t\tnr -= sub_size;\n+\t}\n+\n+\t/* Start work threads. */\n+\tfor (i = 0; i < delta_search_threads; i++) {\n+\t\tif (!p[i].list_size)\n+\t\t\tcontinue;\n+\t\tpthread_mutex_init(&p[i].mutex, NULL);\n+\t\tpthread_cond_init(&p[i].cond, NULL);\n+\t\tret = pthread_create(&p[i].thread, NULL,\n+\t\t\t\t     threaded_find_deltas_by_path, &p[i]);\n+\t\tif (ret)\n+\t\t\tdie(_(\"unable to create thread: %s\"), strerror(ret));\n+\t\tactive_threads++;\n+\t}\n+\n+\t/*\n+\t * Now let's wait for work completion.  Each time a thread is done\n+\t * with its work, we steal half of the remaining work from the\n+\t * thread with the largest number of unprocessed objects and give\n+\t * it to that newly idle thread.  This ensure good load balancing\n+\t * until the remaining object list segments are simply too short\n+\t * to be worth splitting anymore.\n+\t */\n+\twhile (active_threads) {\n+\t\tstruct thread_params *target = NULL;\n+\t\tstruct thread_params *victim = NULL;\n+\t\tunsigned sub_size = 0;\n+\n+\t\tprogress_lock();\n+\t\tfor (;;) {\n+\t\t\tfor (i = 0; !target && i < delta_search_threads; i++)\n+\t\t\t\tif (!p[i].working)\n+\t\t\t\t\ttarget = &p[i];\n+\t\t\tif (target)\n+\t\t\t\tbreak;\n+\t\t\tpthread_cond_wait(&progress_cond, &progress_mutex);\n+\t\t}\n+\n+\t\tfor (i = 0; i < delta_search_threads; i++)\n+\t\t\tif (p[i].remaining > 2*window &&\n+\t\t\t    (!victim || victim->remaining < p[i].remaining))\n+\t\t\t\tvictim = &p[i];\n+\t\tif (victim) {\n+\t\t\tsub_size = victim->remaining / 2;\n+\t\t\ttarget->regions = victim->regions + victim->remaining - sub_size;\n+\t\t\tvictim->list_size -= sub_size;\n+\t\t\tvictim->remaining -= sub_size;\n+\t\t}\n+\t\ttarget->list_size = sub_size;\n+\t\ttarget->remaining = sub_size;\n+\t\ttarget->working = 1;\n+\t\tprogress_unlock();\n+\n+\t\tpthread_mutex_lock(&target->mutex);\n+\t\ttarget->data_ready = 1;\n+\t\tpthread_cond_signal(&target->cond);\n+\t\tpthread_mutex_unlock(&target->mutex);\n+\n+\t\tif (!sub_size) {\n+\t\t\tpthread_join(target->thread, NULL);\n+\t\t\tpthread_cond_destroy(&target->cond);\n+\t\t\tpthread_mutex_destroy(&target->mutex);\n+\t\t\tactive_threads--;\n+\t\t}\n+\t}\n+\tcleanup_threaded_search();\n+\tfree(p);\n+\n+\tdisplay_progress(progress_state, progress_nr);\n+\tstop_progress(&progress_state);\n+}\n+\n static void prepare_pack(int window, int depth)\n {\n \tstruct object_entry **delta_list;\n@@ -3307,8 +3469,8 @@ static void prepare_pack(int window, int depth)\n \t\treturn;\n \n \tif (path_walk)\n-\t\tfind_deltas_by_region(to_pack.objects, to_pack.regions,\n-\t\t\t\t      0, to_pack.nr_regions);\n+\t\tll_find_deltas_by_region(to_pack.objects, to_pack.regions,\n+\t\t\t\t\t 0, to_pack.nr_regions);\n \n \tALLOC_ARRAY(delta_list, to_pack.nr_objects);\n \tnr_deltas = n = 0;\n-- \ngitgitgadget\n\n"},{"id":"518288","messageId":"a14d447d35eb32effadf58d4f93fecdcccaa12cd.1747419124.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v3.git.1747419124.gitgitgadget@gmail.com","subject":"[PATCH v3 12/13] path-walk: add new 'edge_aggressive' option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-05-16T18:12:02Z","receivedAt":"2025-05-16T18:12:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nIn preparation for allowing both the --shallow and --path-walk options\nin the 'git pack-objects' builtin, create a new 'edge_aggressive' option\nin the path-walk API. This option will help walk the boundary more\nthoroughly and help avoid sending extra objects during fetches and\npushes.\n\nThe only use of the 'edge_hint_aggressive' option in the revision API is\nwithin mark_edges_uninteresting(), which is usually called before\nbetween prepare_revision_walk() and before visiting commits with\nget_revision(). In prepare_revision_walk(), the UNINTERESTING commits\nare walked until a boundary is found.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/technical/api-path-walk.adoc |  8 ++++++++\n path-walk.c                                |  6 +++++-\n path-walk.h                                |  7 +++++++\n t/helper/test-path-walk.c                  |  2 ++\n t/t6601-path-walk.sh                       | 20 ++++++++++++++++++++\n 5 files changed, 42 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/technical/api-path-walk.adoc b/Documentation/technical/api-path-walk.adoc\nindex e522695dd9fa..34c905eb9c31 100644\n--- a/Documentation/technical/api-path-walk.adoc\n+++ b/Documentation/technical/api-path-walk.adoc\n@@ -56,6 +56,14 @@ better off using the revision walk API instead.\n \tthe revision walk so that the walk emits commits marked with the\n \t`UNINTERESTING` flag.\n \n+`edge_aggressive`::\n+\tFor performance reasons, usually only the boundary commits are\n+\texplored to find UNINTERESTING objects. However, in the case of\n+\tshallow clones it can be helpful to mark all trees and blobs\n+\treachable from UNINTERESTING tip commits as UNINTERESTING. This\n+\tmatches the behavior of `--objects-edge-aggressive` in the\n+\trevision API.\n+\n `pl`::\n \tThis pattern list pointer allows focusing the path-walk search to\n \ta set of patterns, only emitting paths that match the given\ndiff --git a/path-walk.c b/path-walk.c\nindex 341bdd2ba4ef..2d4ddbadd50f 100644\n--- a/path-walk.c\n+++ b/path-walk.c\n@@ -503,7 +503,11 @@ int walk_objects_by_path(struct path_walk_info *info)\n \tif (prepare_revision_walk(info->revs))\n \t\tdie(_(\"failed to setup revision walk\"));\n \n-\t/* Walk trees to mark them as UNINTERESTING. */\n+\t/*\n+\t * Walk trees to mark them as UNINTERESTING.\n+\t * This is particularly important when 'edge_aggressive' is set.\n+\t */\n+\tinfo->revs->edge_hint_aggressive = info->edge_aggressive;\n \tedge_repo = info->revs->repo;\n \tedge_tree_list = root_tree_list;\n \tmark_edges_uninteresting(info->revs, show_edge,\ndiff --git a/path-walk.h b/path-walk.h\nindex 473ee9d361c8..5ef5a8440e6b 100644\n--- a/path-walk.h\n+++ b/path-walk.h\n@@ -50,6 +50,13 @@ struct path_walk_info {\n \t */\n \tint prune_all_uninteresting;\n \n+\t/**\n+\t * When 'edge_aggressive' is set, then the revision walk will use\n+\t * the '--object-edge-aggressive' option to mark even more objects\n+\t * as uninteresting.\n+\t */\n+\tint edge_aggressive;\n+\n \t/**\n \t * Specify a sparse-checkout definition to match our paths to. Do not\n \t * walk outside of this sparse definition. If the patterns are in\ndiff --git a/t/helper/test-path-walk.c b/t/helper/test-path-walk.c\nindex 61e845e5ec25..fe63002c2be2 100644\n--- a/t/helper/test-path-walk.c\n+++ b/t/helper/test-path-walk.c\n@@ -82,6 +82,8 @@ int cmd__path_walk(int argc, const char **argv)\n \t\t\t N_(\"toggle inclusion of tree objects\")),\n \t\tOPT_BOOL(0, \"prune\", &info.prune_all_uninteresting,\n \t\t\t N_(\"toggle pruning of uninteresting paths\")),\n+\t\tOPT_BOOL(0, \"edge-aggressive\", &info.edge_aggressive,\n+\t\t\t N_(\"toggle aggressive edge walk\")),\n \t\tOPT_BOOL(0, \"stdin-pl\", &stdin_pl,\n \t\t\t N_(\"read a pattern list over stdin\")),\n \t\tOPT_END(),\ndiff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh\nindex c89b0f1e19d9..785c2f22373d 100755\n--- a/t/t6601-path-walk.sh\n+++ b/t/t6601-path-walk.sh\n@@ -378,6 +378,26 @@ test_expect_success 'topic, not base, boundary with pruning' '\n \ttest_cmp_sorted expect out\n '\n \n+test_expect_success 'topic, not base, --edge-aggressive with pruning' '\n+\ttest-tool path-walk --prune --edge-aggressive -- topic --not base >out &&\n+\n+\tcat >expect <<-EOF &&\n+\t0:commit::$(git rev-parse topic)\n+\t1:tree::$(git rev-parse topic^{tree})\n+\t1:tree::$(git rev-parse base^{tree}):UNINTERESTING\n+\t2:tree:right/:$(git rev-parse topic:right)\n+\t2:tree:right/:$(git rev-parse base:right):UNINTERESTING\n+\t3:blob:right/c:$(git rev-parse base:right/c):UNINTERESTING\n+\t3:blob:right/c:$(git rev-parse topic:right/c)\n+\tblobs:2\n+\tcommits:1\n+\ttags:0\n+\ttrees:4\n+\tEOF\n+\n+\ttest_cmp_sorted expect out\n+'\n+\n test_expect_success 'trees are reported exactly once' '\n \ttest_when_finished \"rm -rf unique-trees\" &&\n \ttest_create_repo unique-trees &&\n-- \ngitgitgadget\n\n"},{"id":"518289","messageId":"18241d382072a95463189d558d5873b867fe9bd4.1747419124.git.gitgitgadget@gmail.com","threadId":"63083","inReplyTo":"pull.1819.v3.git.1747419124.gitgitgadget@gmail.com","subject":"[PATCH v3 13/13] pack-objects: allow --shallow and --path-walk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-05-16T18:12:03Z","receivedAt":"2025-05-16T18:12:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThere does not appear to be anything particularly incompatible about the\n--shallow and --path-walk options of 'git pack-objects'. If shallow\ncommits are to be handled differently, then it is by the revision walk\nthat defines the commit set and which are interesting or uninteresting.\n\nHowever, before the previous change, a trivial removal of the warning\nwould cause a failure in t5500-fetch-pack.sh when\nGIT_TEST_PACK_PATH_WALK is enabled. The shallow fetch would provide more\nobjects than we desired, due to some incorrect behavior of the path-walk\nAPI, especially around walking uninteresting objects.\n\nThe recently-added tests in t5538-push-shallow.sh help to confirm this\nbehavior is working with the --path-walk option if\nGIT_TEST_PACK_PATH_WALK is enabled. These tests passed previously due to\nthe --path-walk feature being disabled in the presence of a shallow\nclone.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c  |  5 ++---\n t/t5538-push-shallow.sh | 10 +++++++++-\n 2 files changed, 11 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex df3cca0a668a..a87546530632 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -210,6 +210,7 @@ static int keep_unreachable, unpack_unreachable, include_tag;\n static timestamp_t unpack_unreachable_expiration;\n static int pack_loose_unreachable;\n static int cruft;\n+static int shallow = 0;\n static timestamp_t cruft_expiration;\n static int local;\n static int have_non_local_packs;\n@@ -4490,6 +4491,7 @@ static void get_object_list_path_walk(struct rev_info *revs)\n \t * base objects.\n \t */\n \tinfo.prune_all_uninteresting = sparse;\n+\tinfo.edge_aggressive = shallow;\n \n \ttrace2_region_enter(\"pack-objects\", \"path-walk\", revs->repo);\n \tresult = walk_objects_by_path(&info);\n@@ -4695,7 +4697,6 @@ int cmd_pack_objects(int argc,\n \t\t     struct repository *repo UNUSED)\n {\n \tint use_internal_rev_list = 0;\n-\tint shallow = 0;\n \tint all_progress_implied = 0;\n \tstruct strvec rp = STRVEC_INIT;\n \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n@@ -4881,8 +4882,6 @@ int cmd_pack_objects(int argc,\n \t\t\toption = \"--filter\";\n \t\telse if (use_delta_islands)\n \t\t\toption = \"--delta-islands\";\n-\t\telse if (shallow)\n-\t\t\toption = \"--shallow\";\n \n \t\tif (option) {\n \t\t\twarning(_(\"cannot use %s with %s\"),\ndiff --git a/t/t5538-push-shallow.sh b/t/t5538-push-shallow.sh\nindex c406f9d7ed5f..dc0e972943a6 100755\n--- a/t/t5538-push-shallow.sh\n+++ b/t/t5538-push-shallow.sh\n@@ -153,7 +153,15 @@ test_expect_success 'push new commit from shallow clone has good deltas' '\n \n \t# If the delta base is found, then this message uses \"bytes\".\n \t# If the delta base is not found, then this message uses \"KiB\".\n-\ttest_grep \"Writing objects: .* bytes\" err\n+\ttest_grep \"Writing objects: .* bytes\" err &&\n+\n+\tgit -C deltas commit --amend -m \"changed message\" &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/config-push.txt\" \\\n+\tGIT_PROGRESS_DELAY=0 git -C deltas -c pack.usePathWalk=true \\\n+\t\tpush --progress -f origin deltas 2>err &&\n+\n+\ttest_grep \"Enumerating objects: 1, done\" err &&\n+\ttest_region pack-objects path-walk config-push.txt\n '\n \n test_done\n-- \ngitgitgadget\n"},{"id":"519129","messageId":"aDent08iBbQrFWIB@nand.local","threadId":"63083","inReplyTo":"7ee3884e-b629-4b59-8e9a-a9a56abce786@gmail.com","subject":"Re: [PATCH v2 10/13] pack-objects: refactor path-walk delta phase","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-29T00:17:59Z","receivedAt":"2025-05-29T00:18:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, May 16, 2025 at 12:27:29PM -0400, Derrick Stolee wrote:\n> > I imagine that \"start\" here is setup for the following commit which will\n> > parallelize this task across multiple threads, with each thread starting\n> > at a different position.\n> >\n> > I wonder if (as an alternative) we could get away with passing in a\n> > (struct packing_region *, size_t) pair and drop \"start\". I think this\n> > can work if you pass in the result of adding whatever the value of\n> > \"start\" would be to to_pack.regions here.\n> >\n> > That somewhat hides the fact that this code is meant to be run across\n> > multiple threads, but in a way that I think is worth doing. It lets the\n> > function avoid having to do things like:\n> >\n> >    for (size_t i = start; i < start + nr; i++)\n> >\n> > and instead do the simpler:\n> >\n> >    for (size_t i = 0; i < nr; i++)\n> >\n> > since the caller already adjusted the regions pointer for us. As a\n> > side-effect, it also means that the call below in prepare_pack() can\n> > avoid passing a literal zero.\n>\n> I generally avoid this kind of pattern, because it makes it more\n> difficult to understand what is going on when debugging. The cost\n> of adding 'start' to the index seems low relative to having a\n> consistent indexing scheme when accessing from the \"global\" list\n> of items.\n>\n> This only gets more complicated when the threaded version iterates\n> on a \"consecutive sublist of regions\".\n\nThat's fair. After thinking about it some more, I agree that enumerating\n[start, start + nr) is clearer than enumerating [0, nr) and adjusting\nthe offset at each invocation of the loop body.\n\nFunny enough, I applied that suggestion in a couple of spots in one of\nthe incremental MIDX patches I sent recently.\n\n> > This is an extreme nitpick, but I think that \"/**\" (as opposed to \"/*\")\n> > is preferred, as the former is used for Doxygen-style comments. I feel\n> > like I have seen Junio comment on this before, but searching 'f:Junio\n> > \"/**\"' yields a measly 83,000+ results, so I am unlikely to find it ;-).\n> >\n> > (Not worth rerolling on its own, but if you are rerolling anyway, which\n> > is what I gathered from some of your earlier replies, it may be worth\n> > picking up along the way.)\n>\n> I'm confused. Are you saying I should change from \"/**\" to \"/*\" (which\n> is the opposite of your preferred statement) or did you misread that I\n> had used \"/**\"? I can't determine which statement is the intended one.\n\nOops, this is a typo on my part. I was saying that we should prefer \"/*\"\nto \"/**\".\n\nThe \"/**\"-style lives on in v4, but this is *extremely* minor and I\ndon't think is worth sending another round when everything (of what I\nhave read so far from v4) looks good to me.\n\nThanks,\nTaylor\n"},{"id":"519130","messageId":"aDeoSW0pZ7r7rCog@nand.local","threadId":"63083","inReplyTo":"pull.1819.v3.git.1747419124.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/13] PATH WALK II: Add --path-walk option to 'git pack-objects'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-29T00:20:25Z","receivedAt":"2025-05-29T00:20:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, May 16, 2025 at 06:11:50PM +0000, Derrick Stolee via GitGitGadget wrote:\n> Updates in v3\n> =============\n>\n> Thanks for the review, Taylor. Sorry for my delay in getting back to your\n> feedback.\n>\n>  * Documentation has been edited slightly for simplicity.\n>  * is_oid_interesting() was swapped to is_oid_uninteresting()\n>  * sub_list_size renamed to sub_list_nr\n>  * Several uint32_t and uint64_t variables were converted to size_t.\n>  * Several 'unsigned int' variables were required to stay as-is, for now,\n>    until a refactor can be done.\n>  * An unnecessary update of tag_objects was removed.\n>  * The logic and error message around incompatible options is simpler.\n>  * Tests are expanded, especially around config options.\n>  * Fixed commit message typos.\n>  * Extra care around ALLOC_ARRAY() to avoid a zero- or negative-length\n>    array.\n\nThanks for addressing my feedback :-). This list matches the range-diff\nbelow, which looks good. I think that this one is ready to go from my\nperspective.\n\nThanks,\nTaylor\n"}]}