{"thread":{"id":"60565","subject":"[PATCH 00/24] pack-objects: multi-pack verbatim reuse","startedAt":"2023-11-28T19:07:56Z","lastAt":"2024-01-04T22:24:48Z","messageCount":107,"participants":["Taylor Blau","Patrick Steinhardt","Junio C Hamano","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":24},"messages":[{"id":"485198","messageId":"cover.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":null,"subject":"[PATCH 00/24] pack-objects: multi-pack verbatim reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:07:54Z","receivedAt":"2023-11-28T19:07:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Back in fff42755ef (pack-bitmap: add support for bitmap indexes,\n2013-12-21), we added support for reachability bitmaps, and taught\npack-objects how to reuse verbatim chunks from the bitmapped pack. When\nmulti-pack bitmaps were introduced, this pack-reuse mechanism evolved to\nuse the MIDX's \"preferred\" pack as the source for verbatim reuse.\n\nThis allows repositories to incrementally repack themselves (e.g., using\na `--geometric` repack), storing the result in a MIDX, and generating a\ncorresponding bitmap. This keeps our bitmap coverage up-to-date, while\nmaintaining a relatively small number of packs.\n\nHowever, it is recommended (and matches what we do in production at\nGitHub) that repositories repack themselves all-into-one, and\ngenerate a corresponding single-pack reachability bitmap. This is done\nfor a couple of reasons, but the most relevant one to this series is\nthat it enables us to perform verbatim pack-reuse over a complete copy\nof the repository, since the entire repository resides in a single pack\n(and thus is eligible for verbatim pack-reuse).\n\nAs repositories grow larger, packing their contents into a single pack\nbecomes less feasible. This series extends the pack-reuse mechanism to\noperate over multiple packs which are known ahead of time to be disjoint\nwith respect to one another's set of objects.\n\nThe implementation has a few components:\n\n - A new MIDX chunk called \"Disjoint packfiles\" or DISP is introduced\n to keep track of the bitmap position, number of objects, and\n disjointed-ness for each pack contained in the MIDX.\n\n - A new mode for `git multi-pack-index write --stdin-packs` that\n allows specifying disjoint packs, as well as a new option\n `--retain-disjoint` which preserves the set of existing disjoint\n packs in the new MIDX.\n\n - A new pack-objects mode `--ignore-disjoint`, which produces packs\n which are disjoint with respect to the current set of disjoint packs\n (i.e. it discards any objects from the packing list which appear in\n any of the known-disjoint packs).\n\n - A new repack mode, `--extend-disjoint` which causes any new pack(s)\n which are generated to be disjoint with respect to the set of packs\n currently marked as disjoint, minus any pack(s) which are about to\n be deleted.\n\nWith all of that in place, the patch series then rewrites all of the\npack-reuse functions in terms of the new `bitmapped_pack` structure.\nOnce we have dropped all of the assumptions stemming from only\nperforming pack-reuse over a single candidate pack, we can then enable\nreuse over all of the disjoint packs.\n\nIn addition to the many new tests in t5332 added by that series, I tried\nto simulate a \"real world\" test on git.git by breaking the repository\ninto chunks of 1,000 commits (plus their set of reachable objects not\nreachable from earlier chunk(s)) and packing those chunks. This produces\na large number of packs with the objects from git.git which are known to\nbe disjoint with respect to one another.\n\n $ git clone git@github.com:git/git.git base\n\n $ cd base\n $ mv .git/objects/pack/pack-*.idx{,.bak}\n $ git unpack-objects <.git/objects/pack/pack-*.pack\n\n # pack the objects from each successive block of 1k commits\n $ for rev in $(git rev-list --all | awk '(NR) % 1000 == 0' | tac)\n do\n echo $rev |\n git.compile pack-objects --revs --unpacked .git/objects/pack/pack || return 1\n done\n # and grab any stragglers, pruning the unpacked objects\n $ git repack -d\n I then constructed a MIDX and corresponding bitmap\n\n $ find_pack () {\n for idx in .git/objects/pack/pack-*.idx\n do\n git show-index <$idx | grep -q \"$1\" && basename $idx\n done\n }\n $ preferred=\"$(find_pack $(git rev-parse HEAD))\"\n\n $ ( cd .git/objects/pack && ls -1 *.idx ) | sed -e 's/^/+/g' |\n git.compile multi-pack-index write --bitmap --stdin-packs \\\n --preferred-pack=$preferred\n $ git for-each-ref --format='%(objectname)' refs/heads refs/tags >in\n\nWith all of that in place, I was able to produce a significant speed-up\nby reusing objects from multiple packs:\n\n $ hyperfine -L v single,multi -n '{v}-pack reuse' 'git.compile -c pack.allowPackReuse={v} pack-objects --revs --stdout --use-bitmap-index --delta-base-offset /dev/null'\n Benchmark 1: single-pack reuse\n Time (mean ± σ): 6.094 s ± 0.023 s [User: 43.723 s, System: 0.358 s]\n Range (min … max): 6.063 s … 6.126 s 10 runs\n\n Benchmark 2: multi-pack reuse\n Time (mean ± σ): 906.5 ms ± 3.2 ms [User: 1081.5 ms, System: 30.9 ms]\n Range (min … max): 903.5 ms … 912.7 ms 10 runs\n\n Summary\n multi-pack reuse ran\n 6.72 ± 0.03 times faster than single-pack reuse\n\n(There are corresponding tests in p5332 that test different sized chunks\nand measure the runtime performance as well as resulting pack size).\n\nPerforming verbatim pack reuse naturally trades off between CPU time and\nthe resulting pack size. In the above example, the single-pack reuse\ncase produces a clone size of ~194 MB on my machine, while the\nmulti-pack reuse case produces a clone size closer to ~266 MB, which is\na ~37% increase in clone size.\n\nI think there is still some opportunity to close this gap, since the\n\"packing\" strategy here is extremely naive. In a production setting, I'm\nsure that there are more well thought out repacking strategies that\nwould produce more similar clone sizes.\n\nI considered breaking this series up into smaller chunks, but was\nunsatisfied with the result. Since this series is rather large, if you\nhave alternate suggestions on better ways to structure this, please let\nme know.\n\nThanks in advance for your review!\n\nTaylor Blau (24):\n pack-objects: free packing_data in more places\n pack-bitmap-write: deep-clear the `bb_commit` slab\n pack-bitmap: plug leak in find_objects()\n midx: factor out `fill_pack_info()`\n midx: implement `DISP` chunk\n midx: implement `midx_locate_pack()`\n midx: implement `--retain-disjoint` mode\n pack-objects: implement `--ignore-disjoint` mode\n repack: implement `--extend-disjoint` mode\n pack-bitmap: pass `bitmapped_pack` struct to pack-reuse functions\n pack-bitmap: simplify `reuse_partial_packfile_from_bitmap()` signature\n pack-bitmap: return multiple packs via\n `reuse_partial_packfile_from_bitmap()`\n pack-objects: parameterize pack-reuse routines over a single pack\n pack-objects: keep track of `pack_start` for each reuse pack\n pack-objects: pass `bitmapped_pack`'s to pack-reuse functions\n pack-objects: prepare `write_reused_pack()` for multi-pack reuse\n pack-objects: prepare `write_reused_pack_verbatim()` for multi-pack\n reuse\n pack-objects: include number of packs reused in output\n pack-bitmap: prepare to mark objects from multiple packs for reuse\n pack-objects: add tracing for various packfile metrics\n t/test-lib-functions.sh: implement `test_trace2_data` helper\n pack-objects: allow setting `pack.allowPackReuse` to \"single\"\n pack-bitmap: reuse objects from all disjoint packs\n t/perf: add performance tests for multi-pack reuse\n\n Documentation/config/pack.txt | 8 +-\n Documentation/git-multi-pack-index.txt | 12 ++\n Documentation/git-pack-objects.txt | 8 +\n Documentation/git-repack.txt | 12 ++\n Documentation/gitformat-pack.txt | 109 ++++++++++\n builtin/multi-pack-index.c | 13 +-\n builtin/pack-objects.c | 200 +++++++++++++++----\n builtin/repack.c | 57 +++++-\n midx.c | 218 +++++++++++++++++---\n midx.h | 11 +-\n pack-bitmap-write.c | 9 +-\n pack-bitmap.c | 265 ++++++++++++++++++++-----\n pack-bitmap.h | 18 +-\n pack-objects.c | 15 ++\n pack-objects.h | 1 +\n t/helper/test-read-midx.c | 31 ++-\n t/lib-disjoint.sh | 49 +++++\n t/perf/p5332-multi-pack-reuse.sh | 81 ++++++++\n t/t5319-multi-pack-index.sh | 140 +++++++++++++\n t/t5331-pack-objects-stdin.sh | 156 +++++++++++++++\n t/t5332-multi-pack-reuse.sh | 219 ++++++++++++++++++++\n t/t6113-rev-list-bitmap-filters.sh | 2 +\n t/t7700-repack.sh | 4 +-\n t/t7705-repack-extend-disjoint.sh | 142 +++++++++++++\n t/test-lib-functions.sh | 14 ++\n 25 files changed, 1650 insertions(+), 144 deletions(-)\n create mode 100644 t/lib-disjoint.sh\n create mode 100755 t/perf/p5332-multi-pack-reuse.sh\n create mode 100755 t/t5332-multi-pack-reuse.sh\n create mode 100755 t/t7705-repack-extend-disjoint.sh\n\n\nbase-commit: 564d0252ca632e0264ed670534a51d18a689ef5d\n-- \n2.43.0.24.g980b318f98\n"},{"id":"485199","messageId":"101d6a2841a909aa9c99225ebf1b65695cf116cb.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 01/24] pack-objects: free packing_data in more places","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:07:57Z","receivedAt":"2023-11-28T19:07:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pack-objects internals use a packing_data struct to track what\nobjects are part of the pack(s) being formed.\n\nSince these structures contain allocated fields, failing to\nappropriately free() them results in a leak. Plug that leak by\nintroducing a free_packing_data() function, and call it in the\nappropriate spots.\n\nThis is a fairly straightforward leak to plug, since none of the callers\nexpect to read any values or have any references to parts of the address\nspace being freed.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 1 +\n midx.c | 5 +++++\n pack-objects.c | 15 +++++++++++++++\n pack-objects.h | 1 +\n 4 files changed, 22 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 89a8b5a976..bfa60359d4 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4522,6 +4522,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t reuse_packfile_objects);\n \n cleanup:\n+\tfree_packing_data(&to_pack);\n \tlist_objects_filter_release(&filter_options);\n \tstrvec_clear(&rp);\n \ndiff --git a/midx.c b/midx.c\nindex 2f3863c936..3b727dc633 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1592,8 +1592,13 @@ static int write_midx_internal(const char *object_dir,\n \t\t\t\t flags) < 0) {\n \t\t\terror(_(\"could not write multi-pack bitmap\"));\n \t\t\tresult = 1;\n+\t\t\tfree_packing_data(&pdata);\n+\t\t\tfree(commits);\n \t\t\tgoto cleanup;\n \t\t}\n+\n+\t\tfree_packing_data(&pdata);\n+\t\tfree(commits);\n \t}\n \t/*\n \t * NOTE: Do not use ctx.entries beyond this point, since it might\ndiff --git a/pack-objects.c b/pack-objects.c\nindex f403ca6986..1c7bedcc94 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -151,6 +151,21 @@ void prepare_packing_data(struct repository *r, struct packing_data *pdata)\n \tinit_recursive_mutex(&pdata->odb_lock);\n }\n \n+void free_packing_data(struct packing_data *pdata)\n+{\n+\tif (!pdata)\n+\t\treturn;\n+\n+\tfree(pdata->cruft_mtime);\n+\tfree(pdata->in_pack);\n+\tfree(pdata->in_pack_by_idx);\n+\tfree(pdata->in_pack_pos);\n+\tfree(pdata->index);\n+\tfree(pdata->layer);\n+\tfree(pdata->objects);\n+\tfree(pdata->tree_depth);\n+}\n+\n struct object_entry *packlist_alloc(struct packing_data *pdata,\n \t\t\t\t const struct object_id *oid)\n {\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 0d78db40cb..336217e8cd 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -169,6 +169,7 @@ struct packing_data {\n };\n \n void prepare_packing_data(struct repository *r, struct packing_data *pdata);\n+void free_packing_data(struct packing_data *pdata);\n \n /* Protect access to object database */\n static inline void packing_data_lock(struct packing_data *pdata)\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485200","messageId":"6f5ff96998946f3f49da56fd05c096b949521339.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 02/24] pack-bitmap-write: deep-clear the `bb_commit` slab","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:07:59Z","receivedAt":"2023-11-28T19:08:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `bb_commit` commit slab is used by the pack-bitmap-write machinery\nto track various pieces of bookkeeping used to generate reachability\nbitmaps.\n\nEven though we clear the slab when freeing the bitmap_builder struct\n(with `bitmap_builder_clear()`), there are still pointers which point to\nlocations in memory that have not yet been freed, resulting in a leak.\n\nPlug the leak by introducing a suitable `free_fn` for the `struct\nbb_commit` type, and make sure it is called on each member of the slab\nvia the `deep_clear_bb_data()` function.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex f4ecdf8b0e..dd3a415b9d 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -198,6 +198,13 @@ struct bb_commit {\n \tunsigned idx; /* within selected array */\n };\n \n+static void clear_bb_commit(struct bb_commit *commit)\n+{\n+\tfree(commit->reverse_edges);\n+\tbitmap_free(commit->commit_mask);\n+\tbitmap_free(commit->bitmap);\n+}\n+\n define_commit_slab(bb_data, struct bb_commit);\n \n struct bitmap_builder {\n@@ -339,7 +346,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \n static void bitmap_builder_clear(struct bitmap_builder *bb)\n {\n-\tclear_bb_data(&bb->data);\n+\tdeep_clear_bb_data(&bb->data, clear_bb_commit);\n \tfree(bb->commits);\n \tbb->commits_nr = bb->commits_alloc = 0;\n }\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485201","messageId":"bc38fba65547f5716a2ff9904dd615e755ea84bb.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 03/24] pack-bitmap: plug leak in find_objects()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:02Z","receivedAt":"2023-11-28T19:08:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `find_objects()` function creates an object_list for any tips of the\nreachability query which do not have corresponding bitmaps.\n\nThe object_list is not used outside of `find_objects()`, but we never\nfree it with `object_list_free()`, resulting in a leak. Let's plug that\nleak by calling `object_list_free()`, which results in t6113 becoming\nleak-free.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 2 ++\n t/t6113-rev-list-bitmap-filters.sh | 2 ++\n 2 files changed, 4 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 0260890341..d2f1306960 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1280,6 +1280,8 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\tbase = fill_in_bitmap(bitmap_git, revs, base, seen);\n \t}\n \n+\tobject_list_free(¬_mapped);\n+\n \treturn base;\n }\n \ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex 86c70521f1..459f0d7412 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -4,6 +4,8 @@ test_description='rev-list combining bitmaps and filters'\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-bitmap.sh\n \n+TEST_PASSES_SANITIZE_LEAK=true\n+\n test_expect_success 'set up bitmapped repo' '\n \t# one commit will have bitmaps, the other will not\n \ttest_commit one &&\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485202","messageId":"ccf1337305db60f1c8174e9b309e2a9e04ce1487.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 04/24] midx: factor out `fill_pack_info()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:05Z","receivedAt":"2023-11-28T19:08:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When selecting which packfiles will be written while generating a MIDX,\nthe MIDX internals fill out a 'struct pack_info' with various pieces of\nbook-keeping.\n\nInstead of filling out each field of the `pack_info` structure\nindividually in each of the two spots that modify the array of such\nstructures (`ctx->info`), extract a common routine that does this for\nus.\n\nThis reduces the code duplication by a modest amount. But more\nimportantly, it zero-initializes the structure before assigning values\ninto it. This hardens us for a future change which will add additional\nfields to this structure which (until this patch) was not\nzero-initialized.\n\nAs a result, any new fields added to the `pack_info` structure need only\nbe updated in a single location, instead of at each spot within midx.c.\n\nThere are no functional changes in this patch.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 35 +++++++++++++++++++----------------\n 1 file changed, 19 insertions(+), 16 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3b727dc633..591b3c636e 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -464,6 +464,17 @@ struct pack_info {\n \tunsigned expired : 1;\n };\n \n+static void fill_pack_info(struct pack_info *info,\n+\t\t\t struct packed_git *p, char *pack_name,\n+\t\t\t uint32_t orig_pack_int_id)\n+{\n+\tmemset(info, 0, sizeof(struct pack_info));\n+\n+\tinfo->orig_pack_int_id = orig_pack_int_id;\n+\tinfo->pack_name = pack_name;\n+\tinfo->p = p;\n+}\n+\n static int pack_info_compare(const void *_a, const void *_b)\n {\n \tstruct pack_info *a = (struct pack_info *)_a;\n@@ -504,6 +515,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t\t const char *file_name, void *data)\n {\n \tstruct write_midx_context *ctx = data;\n+\tstruct packed_git *p;\n \n \tif (ends_with(file_name, \".idx\")) {\n \t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n@@ -530,17 +542,14 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \n \t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n \n-\t\tctx->info[ctx->nr].p = add_packed_git(full_path,\n-\t\t\t\t\t\t full_path_len,\n-\t\t\t\t\t\t 0);\n-\n-\t\tif (!ctx->info[ctx->nr].p) {\n+\t\tp = add_packed_git(full_path, full_path_len, 0);\n+\t\tif (!p) {\n \t\t\twarning(_(\"failed to add packfile '%s'\"),\n \t\t\t\tfull_path);\n \t\t\treturn;\n \t\t}\n \n-\t\tif (open_pack_index(ctx->info[ctx->nr].p)) {\n+\t\tif (open_pack_index(p)) {\n \t\t\twarning(_(\"failed to open pack-index '%s'\"),\n \t\t\t\tfull_path);\n \t\t\tclose_pack(ctx->info[ctx->nr].p);\n@@ -548,9 +557,8 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t\treturn;\n \t\t}\n \n-\t\tctx->info[ctx->nr].pack_name = xstrdup(file_name);\n-\t\tctx->info[ctx->nr].orig_pack_int_id = ctx->nr;\n-\t\tctx->info[ctx->nr].expired = 0;\n+\t\tfill_pack_info(&ctx->info[ctx->nr], p, xstrdup(file_name),\n+\t\t\t ctx->nr);\n \t\tctx->nr++;\n \t}\n }\n@@ -1310,11 +1318,6 @@ static int write_midx_internal(const char *object_dir,\n \t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n \t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n \n-\t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n-\t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n-\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n-\t\t\tctx.info[ctx.nr].expired = 0;\n-\n \t\t\tif (flags & MIDX_WRITE_REV_INDEX) {\n \t\t\t\t/*\n \t\t\t\t * If generating a reverse index, need to have\n@@ -1330,10 +1333,10 @@ static int write_midx_internal(const char *object_dir,\n \t\t\t\tif (open_pack_index(ctx.m->packs[i]))\n \t\t\t\t\tdie(_(\"could not open index for %s\"),\n \t\t\t\t\t ctx.m->packs[i]->pack_name);\n-\t\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n \t\t\t}\n \n-\t\t\tctx.nr++;\n+\t\t\tfill_pack_info(&ctx.info[ctx.nr++], ctx.m->packs[i],\n+\t\t\t\t xstrdup(ctx.m->pack_names[i]), i);\n \t\t}\n \t}\n \n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485204","messageId":"c52d7e7b27a9add4f58b8334db4fe4498af1c90f.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 05/24] midx: implement `DISP` chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:08Z","receivedAt":"2023-11-28T19:08:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When a multi-pack bitmap is used to implement verbatim pack reuse (that\nis, when verbatim chunks from an on-disk packfile are copied\ndirectly[^1]), it does so by using its \"preferred pack\" as the source\nfor pack-reuse.\n\nThis allows repositories to pack the majority of their objects into a\nsingle (often large) pack, and then use it as the single source for\nverbatim pack reuse. This increases the amount of objects that are\nreused verbatim (and consequently, decrease the amount of time it takes\nto generate many packs). But this performance comes at a cost, which is\nthat the preferred packfile must pace its growth with that of the entire\nrepository in order to maintain the utility of verbatim pack reuse.\n\nAs repositories grow beyond what we can reasonably store in a single\npackfile, the utility of verbatim pack reuse diminishes. Or, at the very\nleast, it becomes increasingly more expensive to maintain as the pack\ngrows larger and larger.\n\nIt would be beneficial to be able to perform this same optimization over\nmultiple packs, provided some modest constraints (most importantly, that\nthe set of packs eligible for verbatim reuse are disjoint with respect\nto the objects that they contain).\n\nIf we assume that the packs which we treat as candidates for verbatim\nreuse are disjoint with respect to their objects, we need to make only\nmodest modifications to the verbatim pack-reuse code itself. Most\nnotably, we need to remove the assumption that the bits in the\nreachability bitmap corresponding to objects from the single reuse pack\nbegin at the first bit position.\n\nFuture patches will unwind these assumptions and reimplement their\nexisting functionality as special cases of the more general assumptions\n(e.g. that reuse bits can start anywhere within the bitset, but happen\nto start at 0 for all existing cases).\n\nThis patch does not yet relax any of those assumptions. Instead, it\nimplements a foundational data-structure, the \"Disjoint Packs\" (`DISP`)\nchunk of the multi-pack index. The `DISP` chunk's contents are described\nin detail here. Importantly, the `DISP` chunk contains information to\nmap regions of a multi-pack index's reachability bitmap to the packs\nwhose objects they represent.\n\nFor now, this chunk is only written, not read (outside of the test-tool\nused in this patch to test the new chunk's behavior). Future patches\nwill begin to make use of this new chunk.\n\nThis patch implements reading (though no callers outside of the above\none perform any reading) and writing this new chunk. It also extends the\n`--stdin-packs` format used by the `git multi-pack-index write` builtin\nto be able to designate that a given pack is to be marked as \"disjoint\"\nby prefixing it with a '+' character.\n\n[^1]: Modulo patching any `OFS_DELTA`'s that cross over a region of the\n pack that wasn't used verbatim.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 4 +\n Documentation/gitformat-pack.txt | 109 +++++++++++++++++++++++\n builtin/multi-pack-index.c | 10 ++-\n midx.c | 116 ++++++++++++++++++++++---\n midx.h | 5 ++\n pack-bitmap.h | 9 ++\n t/helper/test-read-midx.c | 31 ++++++-\n t/t5319-multi-pack-index.sh | 58 +++++++++++++\n 8 files changed, 325 insertions(+), 17 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 3696506eb3..d130e65b28 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -49,6 +49,10 @@ write::\n \t--stdin-packs::\n \t\tWrite a multi-pack index containing only the set of\n \t\tline-delimited pack index basenames provided over stdin.\n+\t\tLines beginning with a '+' character (followed by the\n+\t\tpack index basename as before) have their pack marked as\n+\t\t\"disjoint\". See the \"`DISP` chunk and disjoint packs\"\n+\t\tsection in linkgit:gitformat-pack[5] for more.\n \n \t--refs-snapshot=::\n \t\tWith `--bitmap`, optionally specify a file which\ndiff --git a/Documentation/gitformat-pack.txt b/Documentation/gitformat-pack.txt\nindex 9fcb29a9c8..658682ddd5 100644\n--- a/Documentation/gitformat-pack.txt\n+++ b/Documentation/gitformat-pack.txt\n@@ -396,6 +396,22 @@ CHUNK DATA:\n \t is padded at the end with between 0 and 3 NUL bytes to make the\n \t chunk size a multiple of 4 bytes.\n \n+\tDisjoint Packfiles (ID: {'D', 'I', 'S', 'P'})\n+\t Stores a table of three 4-byte unsigned integers in network order.\n+\t Each table entry corresponds to a single pack (in the order that\n+\t they appear above in the `PNAM` chunk). The values for each table\n+\t entry are as follows:\n+\t - The first bit position (in psuedo-pack order, see below) to\n+\t contain an object from that pack.\n+\t - The number of bits whose objects are selected from that pack.\n+\t - A \"meta\" value, whose least-significant bit indicates whether or\n+\t not the pack is disjoint with respect to other packs. The\n+\t remaining bits are unused.\n+\t Two packs are \"disjoint\" with respect to one another when they have\n+\t disjoint sets of objects. In other words, any object found in a pack\n+\t contained in the set of disjoint packfiles is guaranteed to be\n+\t uniquely located among those packs.\n+\n \tOID Fanout (ID: {'O', 'I', 'D', 'F'})\n \t The ith entry, F[i], stores the number of OIDs with first\n \t byte at most i. Thus F[255] stores the total\n@@ -509,6 +525,99 @@ packs arranged in MIDX order (with the preferred pack coming first).\n The MIDX's reverse index is stored in the optional 'RIDX' chunk within\n the MIDX itself.\n \n+=== `DISP` chunk and disjoint packs\n+\n+The Disjoint Packfiles (`DISP`) chunk encodes additional information\n+about the objects in the multi-pack index's reachability bitmap. Recall\n+that objects from the MIDX are arranged in \"pseudo-pack\" order (see:\n+above) for reachability bitmaps.\n+\n+From the example above, suppose we have packs \"a\", \"b\", and \"c\", with\n+10, 15, and 20 objects, respectively. In pseudo-pack order, those would\n+be arranged as follows:\n+\n+ |a,0|a,1|...|a,9|b,0|b,1|...|b,14|c,0|c,1|...|c,19|\n+\n+When working with single-pack bitmaps (or, equivalently, multi-pack\n+reachability bitmaps without any packs marked as disjoint),\n+linkgit:git-pack-objects[1] performs ``verbatim'' reuse, attempting to\n+reuse chunks of the existing packfile instead of adding objects to the\n+packing list.\n+\n+When a chunk of bytes are reused from an existing pack, any objects\n+contained therein do not need to be added to the packing list, saving\n+memory and CPU time. But a chunk from an existing packfile can only be\n+reused when the following conditions are met:\n+\n+ - The chunk contains only objects which were requested by the caller\n+ (i.e. does not contain any objects which the caller didn't ask for\n+ explicitly or implicitly).\n+\n+ - All objects stored as offset- or reference-deltas also include their\n+ base object in the resulting pack.\n+\n+Additionally, packfiles many not contain more than one copy of any given\n+object. This introduces an additional constraint over the set of packs\n+we may want to reuse. The most straightforward approach is to mandate\n+that the set of packs is disjoint with respect to the set of objects\n+contained in each pack. In other words, for each object `o` in the union\n+of all objects stored by the disjoint set of packs, `o` is contained in\n+exactly one pack from the disjoint set.\n+\n+One alternative design choice for multi-pack reuse might instead involve\n+imposing a chunk-level constraint that allows packs in the reusable set\n+to contain multiple copies across different packs, but restricts each\n+chunk against including more than one copy of such an object. This is in\n+theory possible to implement, but significantly more complicated than\n+forcing packs themselves to be disjoint. Most notably, we would have to\n+keep track of which objects have already been sent during verbatim\n+pack-reuse, defeating the main purpose of verbatim pack reuse (that we\n+don't have to keep track of individual objects).\n+\n+The `DISP` chunk encodes the necessary information in order to implement\n+multi-pack reuse over a disjoint set of packs as described above.\n+Specifically, the `DISP` chunk encodes three pieces of information (all\n+32-bit unsigned integers in network byte-order) for each packfile `p`\n+that is stored in the MIDX, as follows:\n+\n+`bitmap_pos`:: The first bit position (in pseudo-pack order) in the\n+ multi-pack index's reachability bitmap occupied by an object from `p`.\n+\n+`bitmap_nr`:: The number of bit positions (including the one at\n+ `bitmap_pos`) that encode objects from that pack `p`.\n+\n+`disjoint`:: Metadata, including whether or not the pack `p` is\n+ ``disjoint''. The least significant bit stores whether or not the pack\n+ is disjoint. The remaining bits are reserved for future use.\n+\n+For example, the `DISP` chunk corresponding to the above example (with\n+packs ``a'', ``b'', and ``c'') would look like:\n+\n+[cols=\"1,2,2,2\"]\n+|===\n+| |`bitmap_pos` |`bitmap_nr` |`disjoint`\n+\n+|packfile ``a''\n+|`0`\n+|`10`\n+|`0x1`\n+\n+|packfile ``b''\n+|`10`\n+|`15`\n+|`0x1`\n+\n+|packfile ``c''\n+|`25`\n+|`20`\n+|`0x1`\n+|===\n+\n+With these constraints and information in place, we can treat each\n+packfile marked as disjoint as individually reusable in the same fashion\n+as verbatim pack reuse is performed on individual packs prior to the\n+implementation of the `DISP` chunk.\n+\n == cruft packs\n \n The cruft packs feature offer an alternative to Git's traditional mechanism of\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex a72aebecaa..0f1dd4651d 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -106,11 +106,17 @@ static int git_multi_pack_index_write_config(const char *var, const char *value,\n \treturn 0;\n }\n \n+#define DISJOINT ((void*)(uintptr_t)1)\n+\n static void read_packs_from_stdin(struct string_list *to)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n-\twhile (strbuf_getline(&buf, stdin) != EOF)\n-\t\tstring_list_append(to, buf.buf);\n+\twhile (strbuf_getline(&buf, stdin) != EOF) {\n+\t\tif (*buf.buf == '+')\n+\t\t\tstring_list_append(to, buf.buf + 1)->util = DISJOINT;\n+\t\telse\n+\t\t\tstring_list_append(to, buf.buf);\n+\t}\n \tstring_list_sort(to);\n \n \tstrbuf_release(&buf);\ndiff --git a/midx.c b/midx.c\nindex 591b3c636e..f55020072f 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -33,6 +33,7 @@\n \n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n+#define MIDX_CHUNKID_DISJOINTPACKS 0x44495350 /* \"DISP\" */\n #define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n@@ -182,6 +183,9 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \n \tpair_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS, &m->chunk_large_offsets,\n \t\t &m->chunk_large_offsets_len);\n+\tpair_chunk(cf, MIDX_CHUNKID_DISJOINTPACKS,\n+\t\t (const unsigned char **)&m->chunk_disjoint_packs,\n+\t\t &m->chunk_disjoint_packs_len);\n \n \tif (git_env_bool(\"GIT_TEST_MIDX_READ_RIDX\", 1))\n \t\tpair_chunk(cf, MIDX_CHUNKID_REVINDEX, &m->chunk_revindex,\n@@ -275,6 +279,23 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t\n \treturn 0;\n }\n \n+int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n+\t\t struct bitmapped_pack *bp, uint32_t pack_int_id)\n+{\n+\tif (!m->chunk_disjoint_packs)\n+\t\treturn error(_(\"MIDX does not contain the DISP chunk\"));\n+\n+\tif (prepare_midx_pack(r, m, pack_int_id))\n+\t\treturn error(_(\"could not load disjoint pack %\"PRIu32), pack_int_id);\n+\n+\tbp->p = m->packs[pack_int_id];\n+\tbp->bitmap_pos = get_be32(m->chunk_disjoint_packs + 3 * pack_int_id);\n+\tbp->bitmap_nr = get_be32(m->chunk_disjoint_packs + 3 * pack_int_id + 1);\n+\tbp->disjoint = !!get_be32(m->chunk_disjoint_packs + 3 * pack_int_id + 2);\n+\n+\treturn 0;\n+}\n+\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result)\n {\n \treturn bsearch_hash(oid->hash, m->chunk_oid_fanout, m->chunk_oid_lookup,\n@@ -457,11 +478,18 @@ static size_t write_midx_header(struct hashfile *f,\n \treturn MIDX_HEADER_SIZE;\n }\n \n+#define BITMAP_POS_UNKNOWN (~((uint32_t)0))\n+\n struct pack_info {\n \tuint32_t orig_pack_int_id;\n \tchar *pack_name;\n \tstruct packed_git *p;\n-\tunsigned expired : 1;\n+\n+\tuint32_t bitmap_pos;\n+\tuint32_t bitmap_nr;\n+\n+\tunsigned expired : 1,\n+\t\t disjoint : 1;\n };\n \n static void fill_pack_info(struct pack_info *info,\n@@ -473,6 +501,7 @@ static void fill_pack_info(struct pack_info *info,\n \tinfo->orig_pack_int_id = orig_pack_int_id;\n \tinfo->pack_name = pack_name;\n \tinfo->p = p;\n+\tinfo->bitmap_pos = BITMAP_POS_UNKNOWN;\n }\n \n static int pack_info_compare(const void *_a, const void *_b)\n@@ -516,6 +545,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n {\n \tstruct write_midx_context *ctx = data;\n \tstruct packed_git *p;\n+\tstruct string_list_item *item = NULL;\n \n \tif (ends_with(file_name, \".idx\")) {\n \t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n@@ -534,11 +564,13 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t * should be performed independently (likely checking\n \t\t * to_include before the existing MIDX).\n \t\t */\n-\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n-\t\t\treturn;\n-\t\telse if (ctx->to_include &&\n-\t\t\t !string_list_has_string(ctx->to_include, file_name))\n+\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name)) {\n \t\t\treturn;\n+\t\t} else if (ctx->to_include) {\n+\t\t\titem = string_list_lookup(ctx->to_include, file_name);\n+\t\t\tif (!item)\n+\t\t\t\treturn;\n+\t\t}\n \n \t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n \n@@ -559,6 +591,8 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \n \t\tfill_pack_info(&ctx->info[ctx->nr], p, xstrdup(file_name),\n \t\t\t ctx->nr);\n+\t\tif (item)\n+\t\t\tctx->info[ctx->nr].disjoint = !!item->util;\n \t\tctx->nr++;\n \t}\n }\n@@ -568,7 +602,8 @@ struct pack_midx_entry {\n \tuint32_t pack_int_id;\n \ttime_t pack_mtime;\n \tuint64_t offset;\n-\tunsigned preferred : 1;\n+\tunsigned preferred : 1,\n+\t\t disjoint : 1;\n };\n \n static int midx_oid_compare(const void *_a, const void *_b)\n@@ -586,6 +621,12 @@ static int midx_oid_compare(const void *_a, const void *_b)\n \tif (a->preferred < b->preferred)\n \t\treturn 1;\n \n+\t/* Sort objects in a disjoint pack last when multiple copies exist. */\n+\tif (a->disjoint < b->disjoint)\n+\t\treturn -1;\n+\tif (a->disjoint > b->disjoint)\n+\t\treturn 1;\n+\n \tif (a->pack_mtime > b->pack_mtime)\n \t\treturn -1;\n \telse if (a->pack_mtime < b->pack_mtime)\n@@ -671,6 +712,7 @@ static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n \t\t\t\t\t &fanout->entries[fanout->nr],\n \t\t\t\t\t cur_object);\n \t\tfanout->entries[fanout->nr].preferred = 0;\n+\t\tfanout->entries[fanout->nr].disjoint = 0;\n \t\tfanout->nr++;\n \t}\n }\n@@ -696,6 +738,7 @@ static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout,\n \t\t\t\tcur_object,\n \t\t\t\t&fanout->entries[fanout->nr],\n \t\t\t\tpreferred);\n+\t\tfanout->entries[fanout->nr].disjoint = info->disjoint;\n \t\tfanout->nr++;\n \t}\n }\n@@ -764,14 +807,22 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \t\t * Take only the first duplicate.\n \t\t */\n \t\tfor (cur_object = 0; cur_object < fanout.nr; cur_object++) {\n-\t\t\tif (cur_object && oideq(&fanout.entries[cur_object - 1].oid,\n-\t\t\t\t\t\t&fanout.entries[cur_object].oid))\n-\t\t\t\tcontinue;\n+\t\t\tstruct pack_midx_entry *ours = &fanout.entries[cur_object];\n+\t\t\tif (cur_object) {\n+\t\t\t\tstruct pack_midx_entry *prev = &fanout.entries[cur_object - 1];\n+\t\t\t\tif (oideq(&prev->oid, &ours->oid)) {\n+\t\t\t\t\tif (prev->disjoint && ours->disjoint)\n+\t\t\t\t\t\tdie(_(\"duplicate object '%s' among disjoint packs '%s', '%s'\"),\n+\t\t\t\t\t\t oid_to_hex(&prev->oid),\n+\t\t\t\t\t\t info[prev->pack_int_id].pack_name,\n+\t\t\t\t\t\t info[ours->pack_int_id].pack_name);\n+\t\t\t\t\tcontinue;\n+\t\t\t\t}\n+\t\t\t}\n \n \t\t\tALLOC_GROW(deduplicated_entries, st_add(*nr_objects, 1),\n \t\t\t\t alloc_objects);\n-\t\t\tmemcpy(&deduplicated_entries[*nr_objects],\n-\t\t\t &fanout.entries[cur_object],\n+\t\t\tmemcpy(&deduplicated_entries[*nr_objects], ours,\n \t\t\t sizeof(struct pack_midx_entry));\n \t\t\t(*nr_objects)++;\n \t\t}\n@@ -814,6 +865,27 @@ static int write_midx_pack_names(struct hashfile *f, void *data)\n \treturn 0;\n }\n \n+static int write_midx_disjoint_packs(struct hashfile *f, void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tsize_t i;\n+\n+\tfor (i = 0; i < ctx->nr; i++) {\n+\t\tstruct pack_info *pack = &ctx->info[i];\n+\t\tif (pack->expired)\n+\t\t\tcontinue;\n+\n+\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN && pack->bitmap_nr)\n+\t\t\tBUG(\"pack '%s' has no bitmap position, but has %d bitmapped object(s)\",\n+\t\t\t pack->pack_name, pack->bitmap_nr);\n+\n+\t\thashwrite_be32(f, pack->bitmap_pos);\n+\t\thashwrite_be32(f, pack->bitmap_nr);\n+\t\thashwrite_be32(f, !!pack->disjoint);\n+\t}\n+\treturn 0;\n+}\n+\n static int write_midx_oid_fanout(struct hashfile *f,\n \t\t\t\t void *data)\n {\n@@ -981,8 +1053,19 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \tQSORT(data, ctx->entries_nr, midx_pack_order_cmp);\n \n \tALLOC_ARRAY(pack_order, ctx->entries_nr);\n-\tfor (i = 0; i < ctx->entries_nr; i++)\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *e = &ctx->entries[data[i].nr];\n+\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]];\n+\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n+\t\t\tpack->bitmap_pos = i;\n+\t\tpack->bitmap_nr++;\n \t\tpack_order[i] = data[i].nr;\n+\t}\n+\tfor (i = 0; i < ctx->nr; i++) {\n+\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[i]];\n+\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n+\t\t\tpack->bitmap_pos = 0;\n+\t}\n \tfree(data);\n \n \ttrace2_region_leave(\"midx\", \"midx_pack_order\", the_repository);\n@@ -1283,6 +1366,7 @@ static int write_midx_internal(const char *object_dir,\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n+\tint pack_disjoint_concat_len = 0;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n@@ -1495,8 +1579,10 @@ static int write_midx_internal(const char *object_dir,\n \t}\n \n \tfor (i = 0; i < ctx.nr; i++) {\n-\t\tif (!ctx.info[i].expired)\n-\t\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n+\t\tif (ctx.info[i].expired)\n+\t\t\tcontinue;\n+\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n+\t\tpack_disjoint_concat_len += 3 * sizeof(uint32_t);\n \t}\n \n \t/* Check that the preferred pack wasn't expired (if given). */\n@@ -1556,6 +1642,8 @@ static int write_midx_internal(const char *object_dir,\n \t\tadd_chunk(cf, MIDX_CHUNKID_REVINDEX,\n \t\t\t st_mult(ctx.entries_nr, sizeof(uint32_t)),\n \t\t\t write_midx_revindex);\n+\t\tadd_chunk(cf, MIDX_CHUNKID_DISJOINTPACKS,\n+\t\t\t pack_disjoint_concat_len, write_midx_disjoint_packs);\n \t}\n \n \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\ndiff --git a/midx.h b/midx.h\nindex a5d98919c8..cdd16a8378 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -7,6 +7,7 @@\n struct object_id;\n struct pack_entry;\n struct repository;\n+struct bitmapped_pack;\n \n #define GIT_TEST_MULTI_PACK_INDEX \"GIT_TEST_MULTI_PACK_INDEX\"\n #define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \\\n@@ -33,6 +34,8 @@ struct multi_pack_index {\n \n \tconst unsigned char *chunk_pack_names;\n \tsize_t chunk_pack_names_len;\n+\tconst uint32_t *chunk_disjoint_packs;\n+\tsize_t chunk_disjoint_packs_len;\n \tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_object_offsets;\n@@ -58,6 +61,8 @@ void get_midx_rev_filename(struct strbuf *out, struct multi_pack_index *m);\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n+int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n+\t\t struct bitmapped_pack *bp, uint32_t pack_int_id);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);\n uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 5273a6a019..b7fa1a42a9 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -52,6 +52,15 @@ typedef int (*show_reachable_fn)(\n \n struct bitmap_index;\n \n+struct bitmapped_pack {\n+\tstruct packed_git *p;\n+\n+\tuint32_t bitmap_pos;\n+\tuint32_t bitmap_nr;\n+\n+\tunsigned disjoint : 1;\n+};\n+\n struct bitmap_index *prepare_bitmap_git(struct repository *r);\n struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx);\n void count_bitmap_commit_list(struct bitmap_index *, uint32_t *commits,\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex e9a444ddba..4b44995dca 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -100,10 +100,37 @@ static int read_midx_preferred_pack(const char *object_dir)\n \treturn 0;\n }\n \n+static int read_midx_bitmapped_packs(const char *object_dir)\n+{\n+\tstruct multi_pack_index *midx = NULL;\n+\tstruct bitmapped_pack pack;\n+\tuint32_t i;\n+\n+\tsetup_git_directory();\n+\n+\tmidx = load_multi_pack_index(object_dir, 1);\n+\tif (!midx)\n+\t\treturn 1;\n+\n+\tfor (i = 0; i < midx->num_packs; i++) {\n+\t\tif (nth_bitmapped_pack(the_repository, midx, &pack, i) < 0)\n+\t\t\treturn 1;\n+\n+\t\tprintf(\"%s\\n\", pack_basename(pack.p));\n+\t\tprintf(\" bitmap_pos: %\"PRIuMAX\"\\n\", (uintmax_t)pack.bitmap_pos);\n+\t\tprintf(\" bitmap_nr: %\"PRIuMAX\"\\n\", (uintmax_t)pack.bitmap_nr);\n+\t\tprintf(\" disjoint: %s\\n\", pack.disjoint & 0x1 ? \"yes\" : \"no\");\n+\t}\n+\n+\tclose_midx(midx);\n+\n+\treturn 0;\n+}\n+\n int cmd__read_midx(int argc, const char **argv)\n {\n \tif (!(argc == 2 || argc == 3))\n-\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack] \");\n+\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] \");\n \n \tif (!strcmp(argv[1], \"--show-objects\"))\n \t\treturn read_midx_file(argv[2], 1);\n@@ -111,5 +138,7 @@ int cmd__read_midx(int argc, const char **argv)\n \t\treturn read_midx_checksum(argv[2]);\n \telse if (!strcmp(argv[1], \"--preferred-pack\"))\n \t\treturn read_midx_preferred_pack(argv[2]);\n+\telse if (!strcmp(argv[1], \"--bitmap\"))\n+\t\treturn read_midx_bitmapped_packs(argv[2]);\n \treturn read_midx_file(argv[1], 0);\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex c4c6060cee..fd24e0c952 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -1157,4 +1157,62 @@ test_expect_success 'reader notices too-small revindex chunk' '\n \ttest_cmp expect.err err\n '\n \n+test_expect_success 'disjoint packs are stored via the DISP chunk' '\n+\ttest_when_finished \"rm -fr repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\tfor i in 1 2 3 4 5\n+\t\tdo\n+\t\t\ttest_commit \"$i\" &&\n+\t\t\tgit repack -d || return 1\n+\t\tdone &&\n+\n+\t\tfind $objdir/pack -type f -name \"*.idx\" | xargs -n 1 basename | sort >packs &&\n+\n+\t\tgit multi-pack-index write --stdin-packs err &&\n+\t\tcat >expect <<-\\EOF &&\n+\t\terror: MIDX does not contain the DISP chunk\n+\t\tEOF\n+\t\ttest_cmp expect err &&\n+\n+\t\tsed -e \"s/^/+/g\" packs >in &&\n+\t\tgit multi-pack-index write --stdin-packs --bitmap \\\n+\t\t\t--preferred-pack=\"$(head -n1 actual &&\n+\t\tfor i in $(test_seq $(wc -l expect &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'non-disjoint packs are detected' '\n+\ttest_when_finished \"rm -fr repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit base &&\n+\t\tgit repack -d &&\n+\t\ttest_commit other &&\n+\t\tgit repack -a &&\n+\n+\t\tls -la .git/objects/pack/ &&\n+\n+\t\tfind $objdir/pack -type f -name \"*.idx\" |\n+\t\t\tsed -e \"s/.*\\/\\(.*\\)$/+\\1/g\" >in &&\n+\n+\t\ttest_must_fail git multi-pack-index write --stdin-packs \\\n+\t\t\t--bitmap err &&\n+\t\tgrep \"duplicate object.* among disjoint packs\" err\n+\t)\n+'\n+\n test_done\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485203","messageId":"541fbb442b78942dd6319821a340b2298c836138.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 06/24] midx: implement `midx_locate_pack()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:10Z","receivedAt":"2023-11-28T19:08:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The multi-pack index API exposes a `midx_contains_pack()` function that\ntakes in a string ending in either \".idx\" or \".pack\" and returns whether\nor not the MIDX contains a given pack corresponding to that string.\n\nThere is no corresponding function to locate the position of a pack\nwithin the MIDX's pack order (sorted lexically by pack filename).\n\nWe could add an optional out parameter to `midx_contains_pack()` that is\nfilled out with the pack's position when the parameter is non-NULL. To\nminimize the amount of fallout from this change, instead introduce a new\nfunction by renaming `midx_contains_pack()` to `midx_locate_pack()`,\nadding that output parameter, and then reimplementing\n`midx_contains_pack()` in terms of it.\n\nFuture patches will make use of this new function.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 13 +++++++++++--\n midx.h | 5 ++++-\n 2 files changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex f55020072f..65ba0c70fe 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -413,7 +413,8 @@ static int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \treturn strcmp(idx_or_pack_name, idx_name);\n }\n \n-int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n+int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n+\t\t uint32_t *pos)\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n@@ -424,8 +425,11 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n \n \t\tcurrent = m->pack_names[mid];\n \t\tcmp = cmp_idx_or_pack_name(idx_or_pack_name, current);\n-\t\tif (!cmp)\n+\t\tif (!cmp) {\n+\t\t\tif (pos)\n+\t\t\t\t*pos = mid;\n \t\t\treturn 1;\n+\t\t}\n \t\tif (cmp > 0) {\n \t\t\tfirst = mid + 1;\n \t\t\tcontinue;\n@@ -436,6 +440,11 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n \treturn 0;\n }\n \n+int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n+{\n+\treturn midx_locate_pack(m, idx_or_pack_name, NULL);\n+}\n+\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local)\n {\n \tstruct multi_pack_index *m;\ndiff --git a/midx.h b/midx.h\nindex cdd16a8378..a6e969c2ea 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -70,7 +70,10 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct multi_pack_index *m,\n \t\t\t\t\tuint32_t n);\n int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m);\n-int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name);\n+int midx_contains_pack(struct multi_pack_index *m,\n+\t\t const char *idx_or_pack_name);\n+int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n+\t\t uint32_t *pos);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n \n /*\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485205","messageId":"3019738b52ba8cd78ea696a3b800fa91e722eb66.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 07/24] midx: implement `--retain-disjoint` mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:13Z","receivedAt":"2023-11-28T19:08:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Once multi-pack reachability bitmaps learn how to perform pack reuse\nover the set of disjoint packs, we will want to teach `git repack` to\nevolve the set of disjoint packs over time.\n\nTo evolve the set of disjoint packs means any new packs made by `repack`\nshould be disjoint with respect to the existing set of disjoint packs so\nas to be able to join that set when updating the multi-pack index.\n\nThe details of generating such packs will be left to future commits. But\nany new pack(s) created by repack as disjoint will be marked as such by\npassing them over `--stdin-packs` with the special '+' marker when\ngenerating a new MIDX.\n\nThis patch, however, addresses the question of how we retain the\nexisting set of disjoint packs when updating the multi-pack index. One\noption would be for `repack` to keep track of the set of disjoint packs\nitself by querying the MIDX, and then adding the special '+' marker\nappropriately when generating the input for `--stdin-packs`.\n\nBut this is verbose and error-prone, since two different parts of Git\nwould need to maintain the same notion of the set of disjoint packs.\nWhen one disagrees with the other, the set of so-called disjoint packs\nmay actually contain two or more packs which have one or more object(s)\nin common, making the set non-disjoint.\n\nInstead, introduce a `--retain-disjoint` mode for the `git\nmulti-pack-index write` sub-command which keeps any packs which are:\n\n - marked as disjoint in the existing MIDX, and\n\n - not deleted (e.g., they are not excluded from the input for\n `--stdin-packs`).\n\nThis will allow the `repack` command to not have to keep track of the\nset of currently-disjoint packs itself, reducing the number of lines of\ncode necessary to implement this feature, and making the resulting\nimplementation less error-prone.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 8 +++\n builtin/multi-pack-index.c | 3 +\n midx.c | 49 +++++++++++++++\n midx.h | 1 +\n t/lib-disjoint.sh | 38 ++++++++++++\n t/t5319-multi-pack-index.sh | 82 ++++++++++++++++++++++++++\n 6 files changed, 181 insertions(+)\n create mode 100644 t/lib-disjoint.sh\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex d130e65b28..ac0c7b124b 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -54,6 +54,14 @@ write::\n \t\t\"disjoint\". See the \"`DISP` chunk and disjoint packs\"\n \t\tsection in linkgit:gitformat-pack[5] for more.\n \n+\t--retain-disjoint::\n+\t\tWhen writing a multi-pack index with a reachability\n+\t\tbitmap, keep any packs marked as disjoint in the\n+\t\texisting MIDX (if any) as such in the new MIDX. Existing\n+\t\tdisjoint packs which are removed (e.g., not listed via\n+\t\t`--stdin-packs`) are ignored. This option works in\n+\t\taddition to the '+' marker for `--stdin-packs`.\n+\n \t--refs-snapshot=::\n \t\tWith `--bitmap`, optionally specify a file which\n \t\tcontains a \"refs snapshot\" taken prior to repacking.\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 0f1dd4651d..dcfabf2626 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -138,6 +138,9 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n \t\t\t N_(\"write multi-pack index containing only given indexes\")),\n \t\tOPT_FILENAME(0, \"refs-snapshot\", &opts.refs_snapshot,\n \t\t\t N_(\"refs snapshot for selecting bitmap commits\")),\n+\t\tOPT_BIT(0, \"retain-disjoint\", &opts.flags,\n+\t\t\tN_(\"retain non-deleted disjoint packs\"),\n+\t\t\tMIDX_WRITE_RETAIN_DISJOINT),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/midx.c b/midx.c\nindex 65ba0c70fe..ce67da9f85 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -721,6 +721,12 @@ static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n \t\t\t\t\t &fanout->entries[fanout->nr],\n \t\t\t\t\t cur_object);\n \t\tfanout->entries[fanout->nr].preferred = 0;\n+\t\t/*\n+\t\t * It's OK to set disjoint to 0 here, even with\n+\t\t * `--retain-disjoint`, since we will always see the disjoint\n+\t\t * copy of some object below in get_sorted_entries(), causing us\n+\t\t * to die().\n+\t\t */\n \t\tfanout->entries[fanout->nr].disjoint = 0;\n \t\tfanout->nr++;\n \t}\n@@ -1362,6 +1368,37 @@ static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n \treturn result;\n }\n \n+static int midx_retain_existing_disjoint(struct repository *r,\n+\t\t\t\t\t struct multi_pack_index *from,\n+\t\t\t\t\t struct write_midx_context *ctx)\n+{\n+\tstruct bitmapped_pack bp;\n+\tuint32_t i, midx_pos;\n+\n+\tfor (i = 0; i < ctx->nr; i++) {\n+\t\tstruct pack_info *info = &ctx->info[i];\n+\t\t/*\n+\t\t * Having to call `midx_locate_pack()` in a loop is\n+\t\t * sub-optimal, since it is O(n*log(n)) in the number\n+\t\t * of packs.\n+\t\t *\n+\t\t * When reusing an existing MIDX, we know that the first\n+\t\t * 'n' packs appear in the same order, so we could avoid\n+\t\t * this when reusing an existing MIDX. But we may be\n+\t\t * instead relying on the order given to us by\n+\t\t * for_each_file_in_pack_dir(), in which case we can't\n+\t\t * make any such guarantees.\n+\t\t */\n+\t\tif (!midx_locate_pack(from, info->pack_name, &midx_pos))\n+\t\t\tcontinue;\n+\n+\t\tif (nth_bitmapped_pack(r, from, &bp, midx_pos) < 0)\n+\t\t\treturn -1;\n+\t\tinfo->disjoint = bp.disjoint;\n+\t}\n+\treturn 0;\n+}\n+\n static int write_midx_internal(const char *object_dir,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t struct string_list *packs_to_drop,\n@@ -1444,6 +1481,18 @@ static int write_midx_internal(const char *object_dir,\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n+\tif (flags & MIDX_WRITE_RETAIN_DISJOINT) {\n+\t\tstruct multi_pack_index *m = ctx.m;\n+\t\tif (!m)\n+\t\t\tm = lookup_multi_pack_index(the_repository, object_dir);\n+\n+\t\tif (m) {\n+\t\t\tresult = midx_retain_existing_disjoint(the_repository, m, &ctx);\n+\t\t\tif (result)\n+\t\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n \tif ((ctx.m && ctx.nr == ctx.m->num_packs) &&\n \t !(packs_to_include || packs_to_drop)) {\n \t\tstruct bitmap_index *bitmap_git;\ndiff --git a/midx.h b/midx.h\nindex a6e969c2ea..d7ce52ff7b 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -54,6 +54,7 @@ struct multi_pack_index {\n #define MIDX_WRITE_BITMAP (1 << 2)\n #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)\n #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n+#define MIDX_WRITE_RETAIN_DISJOINT (1 << 5)\n \n const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n void get_midx_filename(struct strbuf *out, const char *object_dir);\ndiff --git a/t/lib-disjoint.sh b/t/lib-disjoint.sh\nnew file mode 100644\nindex 0000000000..c6c6e74aba\n--- /dev/null\n+++ b/t/lib-disjoint.sh\n@@ -0,0 +1,38 @@\n+# Helpers for scripts testing disjoint packs; see t5319 for example usage.\n+\n+objdir=.git/objects\n+\n+test_disjoint_1 () {\n+\tlocal pack=\"$1\"\n+\tlocal want=\"$2\"\n+\n+\ttest-tool read-midx --bitmap $objdir >out &&\n+\tgrep -A 3 \"$pack\" out >found &&\n+\n+\tif ! test -s found\n+\tthen\n+\t\techo >&2 \"could not find '$pack' in MIDX\"\n+\t\treturn 1\n+\tfi\n+\n+\tif ! grep -q \"disjoint: $want\" found\n+\tthen\n+\t\techo >&2 \"incorrect disjoint state for pack '$pack'\"\n+\t\treturn 1\n+\tfi\n+\treturn 0\n+}\n+\n+# test_must_be_disjoint \n+#\n+# Ensures that the given pack is marked as disjoint.\n+test_must_be_disjoint () {\n+\ttest_disjoint_1 \"$1\" \"yes\"\n+}\n+\n+# test_must_not_be_disjoint \n+#\n+# Ensures that the given pack is not marked as disjoint.\n+test_must_not_be_disjoint () {\n+\ttest_disjoint_1 \"$1\" \"no\"\n+}\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex fd24e0c952..02cfddf151 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -3,6 +3,7 @@\n test_description='multi-pack-indexes'\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-chunk.sh\n+. \"$TEST_DIRECTORY\"/lib-disjoint.sh\n \n GIT_TEST_MULTI_PACK_INDEX=0\n objdir=.git/objects\n@@ -1215,4 +1216,85 @@ test_expect_success 'non-disjoint packs are detected' '\n \t)\n '\n \n+test_expect_success 'retain disjoint packs while writing' '\n+\ttest_when_finished \"rm -fr repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\tfor i in 1 2\n+\t\tdo\n+\t\t\ttest_commit \"$i\" && git repack -d || return 1\n+\t\tdone &&\n+\n+\t\tfind $objdir/pack -type f -name \"pack-*.idx\" |\n+\t\tsed -e \"s/^.*\\/\\(.*\\)/\\1/g\" | sort >packs.old &&\n+\n+\t\ttest_line_count = 2 packs.old &&\n+\t\tdisjoint=\"$(head -n 1 packs.old)\" &&\n+\t\tnon_disjoint=\"$(tail -n 1 packs.old)\" &&\n+\n+\t\tcat >in <<-EOF &&\n+\t\t+$disjoint\n+\t\t$non_disjoint\n+\t\tEOF\n+\t\tgit multi-pack-index write --stdin-packs --bitmap packs.new &&\n+\n+\t\tnew_disjoint=\"$(comm -13 packs.old packs.new)\" &&\n+\t\tcat >in <<-EOF &&\n+\t\t$disjoint\n+\t\t$non_disjoint\n+\t\t+$new_disjoint\n+\t\tEOF\n+\t\tgit multi-pack-index write --stdin-packs --bitmap \\\n+\t\t\t--retain-disjoint in <<-EOF &&\n+\t\t+pack-$base.idx\n+\t\tEOF\n+\t\tgit multi-pack-index write --stdin-packs --bitmap in <<-EOF &&\n+\t\tpack-$base.idx\n+\t\t+pack-$other.idx\n+\t\tEOF\n+\t\ttest_must_fail git multi-pack-index write --stdin-packs --retain-disjoint --bitmap err &&\n+\t\tgrep \"duplicate object.* among disjoint packs\" err &&\n+\n+\t\ttest_must_fail git multi-pack-index write --retain-disjoint --bitmap 2>err &&\n+\t\tgrep \"duplicate object.* among disjoint packs\" err\n+\t)\n+'\n+\n test_done\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485206","messageId":"0368f7ab37669163b50b82185725935bde5bc946.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 08/24] pack-objects: implement `--ignore-disjoint` mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:16Z","receivedAt":"2023-11-28T19:08:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Before multi-pack reachability bitmaps learn how to perform pack reuse\nover the set of disjoint packs, we will need a way to generate packs\nthat are known to be disjoint with respect to the currently marked set\nof disjoint packs.\n\nIn other words, we want a way to make a pack which does not have any\nobjects contained in the union of the set of packs which are currently\nmarked as disjoint.\n\nThere are a various ways that we could go about this, for example:\n\n - passing `--unpacked`, which would exclude all packed objects (and\n thus would not contain any objects from the disjoint pack)\n\n - passing `--stdin-packs` with the set of packs currently marked as\n disjoint as \"excluded\", indicating that `pack-objects` should\n discard any objects present in any of the excluded packs (thus\n producing a disjoint pack)\n\n - marking each of the disjoint packs as kept in-core with the\n `--keep-pack` flag, and then passing `--honor-pack-keep` to\n similarly ignore any object(s) from kept packs (thus also producing\n a pack which is disjoint with respect to the current set)\n\n`git repack` is the main entry-point to generating a new pack, by\ninvoking `pack-objects` and then adding the new pack to the set of\ndisjoint packs if generating a new MIDX. However, `repack` has a number\nof ways to invoke `pack-objects` (e.g., all-into-one repacks, geometric\nrepacks, incremental repacks, etc.), all of which would require careful\nreasoning in order to prove that the resulting set of packs is disjoint.\n\nThe most appealing option of the above would be to pass the set of\ndisjoint packs as kept (via `--keep-pack`) and then ignore their\ncontents (with `--honor-pack-keep`), doing so for all kinds of\n`pack-objects` invocations. But there may be more disjoint packs than we\ncan easily fit into the command-line arguments.\n\nInstead, teach `pack-objects` a special `--ignore-disjoint` which is the\nmoral equivalent of marking the set of disjoint packs as kept, and\nignoring their contents, even if it would have otherwise been packed. In\nfact, this similarity extends down to the implementation, where each\ndisjoint pack is first loaded, then has its `pack_keep_in_core` bit set.\n\nWith this in place, we can use the kept-pack cache from 20b031fede\n(packfile: add kept-pack cache for find_kept_pack_entry(), 2021-02-22),\nwhich looks up objects first in a cache containing just the set of kept\n(in this case, disjoint) packs. Assuming that the set of disjoint packs\nis a relatively small portion of the entire repository (which should be\na safe assumption to make), each object lookup will be very inexpensive.\n\nThe only place we want to avoid using `--ignore-disjoint` is in\nconjunction with `--cruft`, since doing so may cause us to omit an\nobject which would have been included in a new cruft pack in order to\nfreshen it. In other words, failing to do so might cause that object to\nbe pruned from the repository earlier than expected.\n\nOtherwise, `--ignore-disjoint` is compatible with most other modes of\n`pack-objects`. These various combinations are tested below. As a\nresult, `repack` will be able to unconditionally (except for the cruft\npack) pass `--ignore-disjoint` when trying to add a new pack to the\ndisjoint set, and the result will be usable, without having to carefully\nconsider and reason about each individual case.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-pack-objects.txt | 8 ++\n builtin/pack-objects.c | 31 +++++-\n t/lib-disjoint.sh | 11 ++\n t/t5331-pack-objects-stdin.sh | 156 +++++++++++++++++++++++++++++\n 4 files changed, 203 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex e32404c6aa..592c4ce742 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -96,6 +96,14 @@ base-name::\n Incompatible with `--revs`, or options that imply `--revs` (such as\n `--all`), with the exception of `--unpacked`, which is compatible.\n \n+--ignore-disjoint::\n+\tThis flag causes an object that appears in any pack marked as\n+\t\"disjoint\" by the multi-pack index to be ignored, even if it\n+\twould have otherwise been packed. When used with\n+\t`--stdin-packs`, objects from disjoint packs may be included if\n+\tand only if a disjoint pack is explicitly given as an input pack\n+\tto `--stdin-packs`. Incompatible with `--cruft`.\n+\n --cruft::\n \tPacks unreachable objects into a separate \"cruft\" pack, denoted\n \tby the existence of a `.mtimes` file. Typically used by `git\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex bfa60359d4..107154db34 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -207,6 +207,7 @@ static int have_non_local_packs;\n static int incremental;\n static int ignore_packed_keep_on_disk;\n static int ignore_packed_keep_in_core;\n+static int ignore_midx_disjoint_packs;\n static int allow_ofs_delta;\n static struct pack_idx_option pack_idx_opts;\n static const char *base_name;\n@@ -1403,7 +1404,8 @@ static int want_found_object(const struct object_id *oid, int exclude,\n \t/*\n \t * Then handle .keep first, as we have a fast(er) path there.\n \t */\n-\tif (ignore_packed_keep_on_disk || ignore_packed_keep_in_core) {\n+\tif (ignore_packed_keep_on_disk || ignore_packed_keep_in_core ||\n+\t ignore_midx_disjoint_packs) {\n \t\t/*\n \t\t * Set the flags for the kept-pack cache to be the ones we want\n \t\t * to ignore.\n@@ -1415,7 +1417,7 @@ static int want_found_object(const struct object_id *oid, int exclude,\n \t\tunsigned flags = 0;\n \t\tif (ignore_packed_keep_on_disk)\n \t\t\tflags |= ON_DISK_KEEP_PACKS;\n-\t\tif (ignore_packed_keep_in_core)\n+\t\tif (ignore_packed_keep_in_core || ignore_midx_disjoint_packs)\n \t\t\tflags |= IN_CORE_KEEP_PACKS;\n \n \t\tif (ignore_packed_keep_on_disk && p->pack_keep)\n@@ -3389,6 +3391,7 @@ static void read_packs_list_from_stdin(void)\n \t\t\tdie(_(\"could not find pack '%s'\"), item->string);\n \t\tif (!is_pack_valid(p))\n \t\t\tdie(_(\"packfile %s cannot be accessed\"), p->pack_name);\n+\t\tp->pack_keep_in_core = 0;\n \t}\n \n \t/*\n@@ -4266,6 +4269,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"create packs suitable for shallow fetches\")),\n \t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep_on_disk,\n \t\t\t N_(\"ignore packs that have companion .keep file\")),\n+\t\tOPT_BOOL(0, \"ignore-disjoint\", &ignore_midx_disjoint_packs,\n+\t\t\t N_(\"ignore packs that are marked disjoint in the MIDX\")),\n \t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n \t\t\t\tN_(\"ignore this pack\")),\n \t\tOPT_INTEGER(0, \"compression\", &pack_compression_level,\n@@ -4412,7 +4417,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tif (use_internal_rev_list)\n \t\t\tdie(_(\"cannot use internal rev list with --cruft\"));\n \t\tif (stdin_packs)\n-\t\t\tdie(_(\"cannot use --stdin-packs with --cruft\"));\n+\t\t\tdie(_(\"cannot use %s with %s\"), \"--stdin-packs\", \"--cruft\");\n+\t\tif (ignore_midx_disjoint_packs)\n+\t\t\tdie(_(\"cannot use %s with %s\"), \"--ignore-disjoint\", \"--cruft\");\n \t}\n \n \t/*\n@@ -4452,6 +4459,24 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tif (!p) /* no keep-able packs found */\n \t\t\tignore_packed_keep_on_disk = 0;\n \t}\n+\tif (ignore_midx_disjoint_packs) {\n+\t\tstruct multi_pack_index *m = get_multi_pack_index(the_repository);\n+\t\tstruct bitmapped_pack pack;\n+\t\tunsigned any_disjoint = 0;\n+\t\tuint32_t i;\n+\n+\t\tfor (i = 0; m && m->chunk_disjoint_packs && i < m->num_packs; i++) {\n+\t\t\tif (nth_bitmapped_pack(the_repository, m, &pack, i) < 0)\n+\t\t\t\tdie(_(\"could not load bitmapped pack %i\"), i);\n+\t\t\tif (pack.disjoint) {\n+\t\t\t\tpack.p->pack_keep_in_core = 1;\n+\t\t\t\tany_disjoint = 1;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!any_disjoint) /* no disjoint packs to ignore */\n+\t\t\tignore_midx_disjoint_packs = 0;\n+\t}\n \tif (local) {\n \t\t/*\n \t\t * unlike ignore_packed_keep_on_disk above, we do not\ndiff --git a/t/lib-disjoint.sh b/t/lib-disjoint.sh\nindex c6c6e74aba..c802ca6940 100644\n--- a/t/lib-disjoint.sh\n+++ b/t/lib-disjoint.sh\n@@ -36,3 +36,14 @@ test_must_be_disjoint () {\n test_must_not_be_disjoint () {\n \ttest_disjoint_1 \"$1\" \"no\"\n }\n+\n+# packed_contents \n+#\n+# Prints the set of objects packed in the given pack indexes.\n+packed_contents () {\n+\tfor idx in \"$@\"\n+\tdo\n+\t\tgit show-index <$idx || return 1\n+\tdone >tmp &&\n+\tcut -d\" \" -f2 tmp-object-list &&\n@@ -237,4 +238,159 @@ test_expect_success 'pack-objects --stdin with packfiles from main and alternate\n \ttest_cmp expected-objects actual-objects\n '\n \n+objdir=.git/objects\n+packdir=$objdir/pack\n+\n+test_expect_success 'loose objects also in disjoint packs are ignored' '\n+\ttest_when_finished \"rm -fr repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\t# create a pack containing the objects in each commit below, but\n+\t\t# do not delete their loose copies\n+\t\ttest_commit base &&\n+\t\tbase_pack=\"$(echo base | git pack-objects --revs $packdir/pack)\" &&\n+\n+\t\ttest_commit other &&\n+\t\tother_pack=\"$(echo base..other | git pack-objects --revs $packdir/pack)\" &&\n+\n+\t\tcat >in <<-EOF &&\n+\t\tpack-$base_pack.idx\n+\t\t+pack-$other_pack.idx\n+\t\tEOF\n+\t\tgit multi-pack-index write --stdin-packs --bitmap all &&\n+\t\tpacked_contents \"$packdir/pack-$other_pack.idx\" >disjoint &&\n+\n+\t\t# make sure that the set of objects we just generated matches\n+\t\t# \"all \\ disjoint\"\n+\t\tpacked_contents \"$packdir/pack-$out.idx\" >got &&\n+\t\tcomm -23 all disjoint >want &&\n+\t\ttest_cmp want got\n+\t)\n+'\n+\n+test_expect_success 'objects in disjoint packs are ignored (--unpacked)' '\n+\ttest_when_finished \"rm -fr repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\tfor c in A B\n+\t\tdo\n+\t\t\ttest_commit \"$c\" || return 1\n+\t\tdone &&\n+\n+\t\tA=\"$(echo \"A\" | git pack-objects --revs $packdir/pack)\" &&\n+\t\tB=\"$(echo \"A..B\" | git pack-objects --revs $packdir/pack)\" &&\n+\n+\t\tcat >in <<-EOF &&\n+\t\tpack-$A.idx\n+\t\t+pack-$B.idx\n+\t\tEOF\n+\t\tgit multi-pack-index write --stdin-packs --bitmap actual &&\n+\n+\t\tgit rev-list --objects --no-object-names B..C >expect.raw &&\n+\t\tsort expect &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'objects in disjoint packs are ignored (--stdin-packs)' '\n+\t# Create objects in three separate packs:\n+\t#\n+\t# - pack A (midx, non disjoint)\n+\t# - pack B (midx, disjoint)\n+\t# - pack C (non-midx)\n+\t#\n+\t# Then create a new pack with `--stdin-packs` and `--ignore-disjoint`\n+\t# including packs A, B, and C. The resulting pack should contain\n+\t# only the objects from packs A, and C, excluding those from\n+\t# pack B as it is marked as disjoint.\n+\ttest_when_finished \"rm -fr repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\tfor c in A B C\n+\t\tdo\n+\t\t\ttest_commit \"$c\" || return 1\n+\t\tdone &&\n+\n+\t\tA=\"$(echo \"A\" | git pack-objects --revs $packdir/pack)\" &&\n+\t\tB=\"$(echo \"A..B\" | git pack-objects --revs $packdir/pack)\" &&\n+\t\tC=\"$(echo \"B..C\" | git pack-objects --revs $packdir/pack)\" &&\n+\n+\t\tcat >in <<-EOF &&\n+\t\tpack-$A.idx\n+\t\t+pack-$B.idx\n+\t\tEOF\n+\t\tgit multi-pack-index write --stdin-packs --bitmap in <<-EOF &&\n+\t\tpack-$A.pack\n+\t\t^pack-$B.pack\n+\t\tpack-$C.pack\n+\t\tEOF\n+\t\tgot=\"$(git pack-objects --stdin-packs --ignore-disjoint $packdir/pack actual &&\n+\t\tpacked_contents \"$packdir/pack-$A.idx\" \\\n+\t\t\t\t\"$packdir/pack-$C.idx\" >expect &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\t# Generate another pack with `--stdin-packs`, this time\n+\t\t# using packs \"B\" and \"C\". The objects from pack \"B\" are\n+\t\t# expected to be in the final pack, despite it being a\n+\t\t# disjoint pack, because \"B\" was mentioned explicitly\n+\t\t# via `stdin-packs`.\n+\t\tcat >in <<-EOF &&\n+\t\tpack-$B.pack\n+\t\tpack-$C.pack\n+\t\tEOF\n+\t\tgot=\"$(git pack-objects --stdin-packs --ignore-disjoint $packdir/pack actual &&\n+\t\tpacked_contents \"$packdir/pack-$B.idx\" \\\n+\t\t\t\t\"$packdir/pack-$C.idx\" >expect &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success '--cruft is incompatible with --ignore-disjoint' '\n+\ttest_must_fail git pack-objects --cruft --ignore-disjoint --stdout \\\n+\t\t/dev/null 2>actual &&\n+\tcat >expect <<-\\EOF &&\n+\tfatal: cannot use --ignore-disjoint with --cruft\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485207","messageId":"b75869befba26899d88d6c6d413cc756aeadbd80.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 09/24] repack: implement `--extend-disjoint` mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:18Z","receivedAt":"2023-11-28T19:08:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that we can generate packs which are disjoint with respect to the\nset of currently-disjoint packs, implement a mode of `git repack` which\nextends the set of disjoint packs with any new (non-cruft) pack(s)\ngenerated during the repack.\n\nThe idea is mostly straightforward, with a couple of gotcha's. The\nstraightforward part is to make sure that any new packs are disjoint\nwith respect to the set of currently disjoint packs which are _not_\nbeing removed from the repository as a result of the repack.\n\nIf a pack which is currently marked as disjoint is, on the other hand,\nabout to be removed from the repository, it is OK (and expected) that\nnew pack(s) will contain some or all of its objects. Since the pack\noriginally marked as disjoint will be removed, it will necessarily leave\nthe disjoint set, making room for new packs with its same objects to\ntake its place. In other words, the resulting set of disjoint packs will\nbe disjoint with respect to one another.\n\nThe gotchas mostly have to do with making sure that we do not generate a\ndisjoint pack in the following scenarios:\n\n - promisor packs\n - cruft packs (which may necessarily need to include an object from a\n disjoint pack in order to freshen it in certain circumstances)\n - all-into-one repacks without '-d'\n - `--filter-to`, which conceptually could work with the new\n `--extend-disjoint` option, but only in limited circumstances\n\nOtherwise, we mark which packs were created as disjoint by using a new\nbit in the `generated_pack_data` struct, and then marking those pack(s)\nas disjoint accordingly when generating the MIDX. Non-deleted packs\nwhich are marked as disjoint are retained as such by passing the\nequivalent of `--retain-disjoint` when calling the MIDX API to update\nthe MIDX.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-repack.txt | 12 +++\n builtin/repack.c | 57 +++++++++---\n t/t7700-repack.sh | 4 +-\n t/t7705-repack-extend-disjoint.sh | 142 ++++++++++++++++++++++++++++++\n 4 files changed, 203 insertions(+), 12 deletions(-)\n create mode 100755 t/t7705-repack-extend-disjoint.sh\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex c902512a9e..50ba5e7f9c 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -249,6 +249,18 @@ linkgit:git-multi-pack-index[1]).\n \tWrite a multi-pack index (see linkgit:git-multi-pack-index[1])\n \tcontaining the non-redundant packs.\n \n+--extend-disjoint::\n+\tExtends the set of disjoint packs. All new non-cruft pack(s)\n+\tgenerated are constructed to be disjoint with respect to the set\n+\tof currently disjoint packs, excluding any packs that will be\n+\tremoved as a result of the repack operation. For more on\n+\tdisjoint packs, see the details in linkgit:gitformat-pack[5],\n+\tunder the section \"`DISP` chunk and disjoint packs\".\n++\n+Useful only with the combination of `--write-midx` and\n+`--write-bitmap-index`. Incompatible with `--filter-to`. Incompatible\n+with `-A`, `-a`, or `--cruft` unless `-d` is given.\n+\n CONFIGURATION\n -------------\n \ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex edaee4dbec..0601bd16c4 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -58,6 +58,7 @@ struct pack_objects_args {\n \tint no_reuse_object;\n \tint quiet;\n \tint local;\n+\tint ignore_disjoint;\n \tstruct list_objects_filter_options filter_options;\n };\n \n@@ -293,6 +294,8 @@ static void prepare_pack_objects(struct child_process *cmd,\n \t\tstrvec_push(&cmd->args, \"--local\");\n \tif (args->quiet)\n \t\tstrvec_push(&cmd->args, \"--quiet\");\n+\tif (args->ignore_disjoint)\n+\t\tstrvec_push(&cmd->args, \"--ignore-disjoint\");\n \tif (delta_base_offset)\n \t\tstrvec_push(&cmd->args, \"--delta-base-offset\");\n \tstrvec_push(&cmd->args, out);\n@@ -334,9 +337,11 @@ static struct {\n \n struct generated_pack_data {\n \tstruct tempfile *tempfiles[ARRAY_SIZE(exts)];\n+\tunsigned disjoint : 1;\n };\n \n-static struct generated_pack_data *populate_pack_exts(const char *name)\n+static struct generated_pack_data *populate_pack_exts(const char *name,\n+\t\t\t\t\t\t unsigned disjoint)\n {\n \tstruct stat statbuf;\n \tstruct strbuf path = STRBUF_INIT;\n@@ -353,6 +358,8 @@ static struct generated_pack_data *populate_pack_exts(const char *name)\n \t\tdata->tempfiles[i] = register_tempfile(path.buf);\n \t}\n \n+\tdata->disjoint = disjoint;\n+\n \tstrbuf_release(&path);\n \treturn data;\n }\n@@ -379,6 +386,8 @@ static void repack_promisor_objects(const struct pack_objects_args *args,\n \tprepare_pack_objects(&cmd, args, packtmp);\n \tcmd.in = -1;\n \n+\tstrvec_pushf(&cmd.args, \"--no-ignore-disjoint\");\n+\n \t/*\n \t * NEEDSWORK: Giving pack-objects only the OIDs without any ordering\n \t * hints may result in suboptimal deltas in the resulting pack. See if\n@@ -421,7 +430,7 @@ static void repack_promisor_objects(const struct pack_objects_args *args,\n \t\t\t\t\t line.buf);\n \t\twrite_promisor_file(promisor_name, NULL, 0);\n \n-\t\titem->util = populate_pack_exts(item->string);\n+\t\titem->util = populate_pack_exts(item->string, 0);\n \n \t\tfree(promisor_name);\n \t}\n@@ -731,8 +740,13 @@ static void midx_included_packs(struct string_list *include,\n \n \tfor_each_string_list_item(item, &existing->kept_packs)\n \t\tstring_list_insert(include, xstrfmt(\"%s.idx\", item->string));\n-\tfor_each_string_list_item(item, names)\n-\t\tstring_list_insert(include, xstrfmt(\"pack-%s.idx\", item->string));\n+\tfor_each_string_list_item(item, names) {\n+\t\tconst char *marker = \"\";\n+\t\tstruct generated_pack_data *data = item->util;\n+\t\tif (data->disjoint)\n+\t\t\tmarker = \"+\";\n+\t\tstring_list_insert(include, xstrfmt(\"%spack-%s.idx\", marker, item->string));\n+\t}\n \tif (geometry->split_factor) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n \t\tuint32_t i;\n@@ -788,7 +802,8 @@ static int write_midx_included_packs(struct string_list *include,\n \t\t\t\t struct pack_geometry *geometry,\n \t\t\t\t struct string_list *names,\n \t\t\t\t const char *refs_snapshot,\n-\t\t\t\t int show_progress, int write_bitmaps)\n+\t\t\t\t int show_progress, int write_bitmaps,\n+\t\t\t\t int exclude_disjoint)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tstruct string_list_item *item;\n@@ -852,6 +867,9 @@ static int write_midx_included_packs(struct string_list *include,\n \tif (refs_snapshot)\n \t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\", refs_snapshot);\n \n+\tif (exclude_disjoint)\n+\t\tstrvec_push(&cmd.args, \"--retain-disjoint\");\n+\n \tret = start_command(&cmd);\n \tif (ret)\n \t\treturn ret;\n@@ -895,7 +913,7 @@ static void remove_redundant_bitmaps(struct string_list *include,\n \n static int finish_pack_objects_cmd(struct child_process *cmd,\n \t\t\t\t struct string_list *names,\n-\t\t\t\t int local)\n+\t\t\t\t int local, int disjoint)\n {\n \tFILE *out;\n \tstruct strbuf line = STRBUF_INIT;\n@@ -913,7 +931,7 @@ static int finish_pack_objects_cmd(struct child_process *cmd,\n \t\t */\n \t\tif (local) {\n \t\t\titem = string_list_append(names, line.buf);\n-\t\t\titem->util = populate_pack_exts(line.buf);\n+\t\t\titem->util = populate_pack_exts(line.buf, disjoint);\n \t\t}\n \t}\n \tfclose(out);\n@@ -970,7 +988,7 @@ static int write_filtered_pack(const struct pack_objects_args *args,\n \t\tfprintf(in, \"%s%s.pack\\n\", caret, item->string);\n \tfclose(in);\n \n-\treturn finish_pack_objects_cmd(&cmd, names, local);\n+\treturn finish_pack_objects_cmd(&cmd, names, local, 0);\n }\n \n static int existing_cruft_pack_cmp(const void *va, const void *vb)\n@@ -1098,7 +1116,7 @@ static int write_cruft_pack(const struct pack_objects_args *args,\n \t\tfprintf(in, \"%s.pack\\n\", item->string);\n \tfclose(in);\n \n-\treturn finish_pack_objects_cmd(&cmd, names, local);\n+\treturn finish_pack_objects_cmd(&cmd, names, local, 0);\n }\n \n static const char *find_pack_prefix(const char *packdir, const char *packtmp)\n@@ -1190,6 +1208,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"pack prefix to store a pack containing pruned objects\")),\n \t\tOPT_STRING(0, \"filter-to\", &filter_to, N_(\"dir\"),\n \t\t\t N_(\"pack prefix to store a pack containing filtered out objects\")),\n+\t\tOPT_BOOL(0, \"extend-disjoint\", &po_args.ignore_disjoint,\n+\t\t\t N_(\"add new packs to the set of disjoint ones\")),\n \t\tOPT_END()\n \t};\n \n@@ -1255,6 +1275,16 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tstrbuf_release(&path);\n \t}\n \n+\tif (po_args.ignore_disjoint) {\n+\t\tif (filter_to)\n+\t\t\tdie(_(\"options '%s' and '%s' cannot be used together\"),\n+\t\t\t \"--filter-to\", \"--extend-disjoint\");\n+\t\tif (pack_everything && !delete_redundant)\n+\t\t\tdie(_(\"cannot use '--extend-disjoint' with '%s' but not '-d'\"),\n+\t\t\t pack_everything & LOOSEN_UNREACHABLE ? \"-A\" :\n+\t\t\t pack_everything & PACK_CRUFT ? \"--cruft\" : \"-a\");\n+\t}\n+\n \tpackdir = mkpathdup(\"%s/pack\", get_object_directory());\n \tpacktmp_name = xstrfmt(\".tmp-%d-pack\", (int)getpid());\n \tpacktmp = mkpathdup(\"%s/%s\", packdir, packtmp_name);\n@@ -1308,6 +1338,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (pack_everything & ALL_INTO_ONE) {\n \t\trepack_promisor_objects(&po_args, &names);\n \n+\t\tif (delete_redundant)\n+\t\t\tstrvec_pushf(&cmd.args, \"--no-ignore-disjoint\");\n+\n \t\tif (has_existing_non_kept_packs(&existing) &&\n \t\t delete_redundant &&\n \t\t !(pack_everything & PACK_CRUFT)) {\n@@ -1364,7 +1397,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tfclose(in);\n \t}\n \n-\tret = finish_pack_objects_cmd(&cmd, &names, 1);\n+\tret = finish_pack_objects_cmd(&cmd, &names, 1, po_args.ignore_disjoint);\n \tif (ret)\n \t\tgoto cleanup;\n \n@@ -1387,6 +1420,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \n \t\tcruft_po_args.local = po_args.local;\n \t\tcruft_po_args.quiet = po_args.quiet;\n+\t\tcruft_po_args.ignore_disjoint = 0;\n \n \t\tret = write_cruft_pack(&cruft_po_args, packtmp, pack_prefix,\n \t\t\t\t cruft_expiration, &names,\n@@ -1487,7 +1521,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \n \t\tret = write_midx_included_packs(&include, &geometry, &names,\n \t\t\t\t\t\trefs_snapshot ? get_tempfile_path(refs_snapshot) : NULL,\n-\t\t\t\t\t\tshow_progress, write_bitmaps > 0);\n+\t\t\t\t\t\tshow_progress, write_bitmaps > 0,\n+\t\t\t\t\t\tpo_args.ignore_disjoint);\n \n \t\tif (!ret && write_bitmaps)\n \t\t\tremove_redundant_bitmaps(&include, packdir);\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex d2975e6c93..277f1ff1d7 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -6,6 +6,7 @@ test_description='git repack works correctly'\n . \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n . \"${TEST_DIRECTORY}/lib-midx.sh\"\n . \"${TEST_DIRECTORY}/lib-terminal.sh\"\n+. \"${TEST_DIRECTORY}/lib-disjoint.sh\"\n \n commit_and_pack () {\n \ttest_commit \"$@\" 1>&2 &&\n@@ -525,7 +526,8 @@ test_expect_success '--filter works with --max-pack-size' '\n '\n \n objdir=.git/objects\n-midx=$objdir/pack/multi-pack-index\n+packdir=$objdir/pack\n+midx=$packdir/multi-pack-index\n \n test_expect_success 'setup for --write-midx tests' '\n \tgit init midx &&\ndiff --git a/t/t7705-repack-extend-disjoint.sh b/t/t7705-repack-extend-disjoint.sh\nnew file mode 100755\nindex 0000000000..0c8be1cb3f\n--- /dev/null\n+++ b/t/t7705-repack-extend-disjoint.sh\n@@ -0,0 +1,142 @@\n+#!/bin/sh\n+\n+test_description='git repack --extend-disjoint works correctly'\n+\n+. ./test-lib.sh\n+. \"${TEST_DIRECTORY}/lib-disjoint.sh\"\n+\n+packdir=.git/objects/pack\n+\n+GIT_TEST_MULTI=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n+test_expect_success 'repack --extend-disjoint creates new disjoint packs' '\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\n+\t\tA=\"$(echo A | git pack-objects --revs $packdir/pack)\" &&\n+\t\tB=\"$(echo A..B | git pack-objects --revs $packdir/pack)\" &&\n+\n+\t\tgit prune-packed &&\n+\n+\t\tcat >in <<-EOF &&\n+\t\tpack-$A.idx\n+\t\t+pack-$B.idx\n+\t\tEOF\n+\t\tgit multi-pack-index write --bitmap --stdin-packs packs.before &&\n+\t\tgit repack --write-midx --write-bitmap-index --extend-disjoint &&\n+\t\tfind $packdir -type f -name \"*.idx\" | sort >packs.after &&\n+\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\n+\t\ttest_line_count = 1 packs.new &&\n+\n+\t\ttest_must_not_be_disjoint \"pack-$A.pack\" &&\n+\t\ttest_must_be_disjoint \"pack-$B.pack\" &&\n+\t\ttest_must_be_disjoint \"$(basename $(cat packs.new) .idx).pack\"\n+\t)\n+'\n+\n+test_expect_success 'repack --extend-disjoint combines existing disjoint packs' '\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit D &&\n+\n+\t\tgit repack -a -d --write-midx --write-bitmap-index --extend-disjoint &&\n+\n+\t\tfind $packdir -type f -name \"*.pack\" >packs &&\n+\t\ttest_line_count = 1 packs &&\n+\n+\t\ttest_must_be_disjoint \"$(basename $(cat packs))\"\n+\n+\t)\n+'\n+\n+test_expect_success 'repack --extend-disjoint with --geometric' '\n+\tgit init disjoint-geometric &&\n+\t(\n+\t\tcd disjoint-geometric &&\n+\n+\t\ttest_commit_bulk 8 &&\n+\t\tbase=\"$(basename $(ls $packdir/pack-*.idx))\" &&\n+\t\techo \"+$base\" >>in &&\n+\n+\t\ttest_commit A &&\n+\t\tA=\"$(echo HEAD^.. | git pack-objects --revs $packdir/pack)\" &&\n+\t\ttest_commit B &&\n+\t\tB=\"$(echo HEAD^.. | git pack-objects --revs $packdir/pack)\" &&\n+\n+\t\tgit prune-packed &&\n+\n+\t\tcat >>in <<-EOF &&\n+\t\t+pack-$A.idx\n+\t\t+pack-$B.idx\n+\t\tEOF\n+\t\tgit multi-pack-index write --bitmap --stdin-packs packs.before &&\n+\t\tgit repack --geometric=2 -d --write-midx --write-bitmap-index --extend-disjoint &&\n+\t\tfind $packdir -type f -name \"*.pack\" | sort >packs.after &&\n+\n+\t\tcomm -12 packs.before packs.after >packs.unchanged &&\n+\t\tcomm -23 packs.before packs.after >packs.removed &&\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\t$packdir/${base%.idx}.pack\n+\t\tEOF\n+\t\ttest_cmp expect packs.unchanged &&\n+\n+\t\tsort >expect <<-EOF &&\n+\t\t$packdir/pack-$A.pack\n+\t\t$packdir/pack-$B.pack\n+\t\tEOF\n+\t\ttest_cmp expect packs.removed &&\n+\n+\t\ttest_line_count = 1 packs.new &&\n+\n+\t\ttest_must_be_disjoint \"$(basename $(cat packs.new))\" &&\n+\t\ttest_must_be_disjoint \"${base%.idx}.pack\"\n+\t)\n+'\n+\n+for flag in \"-A\" \"-a\" \"--cruft\"\n+do\n+\ttest_expect_success \"repack --extend-disjoint incompatible with $flag without -d\" '\n+\t\ttest_must_fail git repack $flag --extend-disjoint \\\n+\t\t\t--write-midx --write-bitmap-index 2>actual &&\n+\t\tcat >expect <<-EOF &&\n+\t\tfatal: cannot use $SQ--extend-disjoint$SQ with $SQ$flag$SQ but not $SQ-d$SQ\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t'\n+done\n+\n+test_expect_success 'repack --extend-disjoint is incompatible with --filter-to' '\n+\ttest_must_fail git repack --extend-disjoint --filter-to=dir 2>actual &&\n+\n+\tcat >expect <<-EOF &&\n+\tfatal: options $SQ--filter-to$SQ and $SQ--extend-disjoint$SQ cannot be used together\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485208","messageId":"970bd9eaeae038adb6e7d4c399c9b033668a8864.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 10/24] pack-bitmap: pass `bitmapped_pack` struct to pack-reuse functions","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:21Z","receivedAt":"2023-11-28T19:08:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When trying to assemble a pack with bitmaps using `--use-bitmap-index`,\n`pack-objects` asks the pack-bitmap machinery for a bitmap which\nindicates the set of objects we can \"reuse\" verbatim from on-disk.\n\nThis set is roughly comprised of: a prefix of objects in the bitmapped\npack (or preferred pack, in the case of a multi-pack reachability\nbitmap), plus any other objects not included in the prefix, excluding\nany deltas whose base we are not sending in the resulting pack.\n\nThe pack-bitmap machinery is responsible for computing this bitmap, and\ndoes so with the following functions:\n\n - reuse_partial_packfile_from_bitmap()\n - try_partial_reuse()\n\nIn the existing implementation, the first function is responsible for\n(a) marking the prefix of objects in the reusable pack, and then (b)\ncalling try_partial_reuse() on any remaining objects to ensure that they\nare also reusable (and removing them from the bitmapped set if they are\nnot).\n\nLikewise, the `try_partial_reuse()` function is responsible for checking\nwhether an isolated object (that is, an object from the bitmapped\npack/preferred pack not contained in the prefix from earlier) may be\nreused, i.e. that it isn't a delta of an object that we are not sending\nin the resulting pack.\n\nThese functions are based on two core assumptions, which we will unwind\nin this and the following commits:\n\n 1. There is only a single pack from the bitmap which is eligible for\n verbatim pack-reuse. For single-pack bitmaps, this is trivially the\n bitmapped pack. For multi-pack bitmaps, this is (currently) the\n MIDX's preferred pack.\n\n 2. The pack eligible for reuse has its first object in bit position 0,\n and all objects from that pack follow in pack-order from that first\n bit position.\n\nIn order to perform verbatim pack reuse over multiple packs, we must\nunwind these two assumptions. Most notably, in order to reuse bits from\na given packfile, we need to know the first bit position occupied by\nan object form that packfile. To propagate this information around, pass\na `struct bitmapped_pack *` anywhere we previously passed a `struct\npacked_git *`, since the former contains the bitmap position we're\ninterested in (as well as a pointer to the latter).\n\nAs an additional step, factor out a sub-routine from the main\n`reuse_partial_packfile_from_bitmap()` function, called\n`reuse_partial_packfile_from_bitmap_1()`. This new function will be\nresponsible for figuring out which objects may be reused from a single\npack, and the existing function will dispatch multiple calls to its new\nhelper function for each reusable pack.\n\nConsequently, `reuse_partial_packfile_from_bitmap()` will now maintain\nan array of reusable packs instead of a single such pack. We currently\nexpect that array to have only a single element, so this awkward state\nis short-lived. It will serve as useful scaffolding in subsequent\ncommits as we begin to work towards enabling multi-pack reuse.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 105 +++++++++++++++++++++++++++++++++++---------------\n 1 file changed, 74 insertions(+), 31 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d2f1306960..2ebe2c314e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1836,7 +1836,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n * -1 means \"stop trying further objects\"; 0 means we may or may not have\n * reused, but you can keep feeding bits.\n */\n-static int try_partial_reuse(struct packed_git *pack,\n+static int try_partial_reuse(struct bitmapped_pack *pack,\n \t\t\t size_t pos,\n \t\t\t struct bitmap *reuse,\n \t\t\t struct pack_window **w_curs)\n@@ -1868,11 +1868,11 @@ static int try_partial_reuse(struct packed_git *pack,\n \t * preferred pack precede all bits from other packs.\n \t */\n \n-\tif (pos >= pack->num_objects)\n+\tif (pos >= pack->p->num_objects)\n \t\treturn -1; /* not actually in the pack or MIDX preferred pack */\n \n-\toffset = delta_obj_offset = pack_pos_to_offset(pack, pos);\n-\ttype = unpack_object_header(pack, w_curs, &offset, &size);\n+\toffset = delta_obj_offset = pack_pos_to_offset(pack->p, pos);\n+\ttype = unpack_object_header(pack->p, w_curs, &offset, &size);\n \tif (type < 0)\n \t\treturn -1; /* broken packfile, punt */\n \n@@ -1888,11 +1888,11 @@ static int try_partial_reuse(struct packed_git *pack,\n \t\t * and the normal slow path will complain about it in\n \t\t * more detail.\n \t\t */\n-\t\tbase_offset = get_delta_base(pack, w_curs, &offset, type,\n+\t\tbase_offset = get_delta_base(pack->p, w_curs, &offset, type,\n \t\t\t\t\t delta_obj_offset);\n \t\tif (!base_offset)\n \t\t\treturn 0;\n-\t\tif (offset_to_pack_pos(pack, base_offset, &base_pos) < 0)\n+\t\tif (offset_to_pack_pos(pack->p, base_offset, &base_pos) < 0)\n \t\t\treturn 0;\n \n \t\t/*\n@@ -1915,14 +1915,14 @@ static int try_partial_reuse(struct packed_git *pack,\n \t\t * to REF_DELTA on the fly. Better to just let the normal\n \t\t * object_entry code path handle it.\n \t\t */\n-\t\tif (!bitmap_get(reuse, base_pos))\n+\t\tif (!bitmap_get(reuse, pack->bitmap_pos + base_pos))\n \t\t\treturn 0;\n \t}\n \n \t/*\n \t * If we got here, then the object is OK to reuse. Mark it.\n \t */\n-\tbitmap_set(reuse, pos);\n+\tbitmap_set(reuse, pack->bitmap_pos + pos);\n \treturn 0;\n }\n \n@@ -1934,29 +1934,13 @@ uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n \treturn nth_midxed_pack_int_id(m, pack_pos_to_midx(bitmap_git->midx, 0));\n }\n \n-int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n-\t\t\t\t struct packed_git **packfile_out,\n-\t\t\t\t uint32_t *entries,\n-\t\t\t\t struct bitmap **reuse_out)\n+static void reuse_partial_packfile_from_bitmap_1(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct bitmapped_pack *pack,\n+\t\t\t\t\t\t struct bitmap *reuse)\n {\n-\tstruct repository *r = the_repository;\n-\tstruct packed_git *pack;\n \tstruct bitmap *result = bitmap_git->result;\n-\tstruct bitmap *reuse;\n \tstruct pack_window *w_curs = NULL;\n \tsize_t i = 0;\n-\tuint32_t offset;\n-\tuint32_t objects_nr;\n-\n-\tassert(result);\n-\n-\tload_reverse_index(r, bitmap_git);\n-\n-\tif (bitmap_is_midx(bitmap_git))\n-\t\tpack = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n-\telse\n-\t\tpack = bitmap_git->pack;\n-\tobjects_nr = pack->num_objects;\n \n \twhile (i < result->word_alloc && result->words[i] == (eword_t)~0)\n \t\ti++;\n@@ -1969,15 +1953,15 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t * we use it instead of another pack. In single-pack bitmaps, the choice\n \t * is made for us.\n \t */\n-\tif (i > objects_nr / BITS_IN_EWORD)\n-\t\ti = objects_nr / BITS_IN_EWORD;\n+\tif (i > pack->p->num_objects / BITS_IN_EWORD)\n+\t\ti = pack->p->num_objects / BITS_IN_EWORD;\n \n-\treuse = bitmap_word_alloc(i);\n \tmemset(reuse->words, 0xFF, i * sizeof(eword_t));\n \n \tfor (; i < result->word_alloc; ++i) {\n \t\teword_t word = result->words[i];\n \t\tsize_t pos = (i * BITS_IN_EWORD);\n+\t\tsize_t offset;\n \n \t\tfor (offset = 0; offset < BITS_IN_EWORD; ++offset) {\n \t\t\tif ((word >> offset) == 0)\n@@ -2002,6 +1986,65 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \n done:\n \tunuse_pack(&w_curs);\n+}\n+\n+int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct packed_git **packfile_out,\n+\t\t\t\t uint32_t *entries,\n+\t\t\t\t struct bitmap **reuse_out)\n+{\n+\tstruct repository *r = the_repository;\n+\tstruct bitmapped_pack *packs = NULL;\n+\tstruct bitmap *result = bitmap_git->result;\n+\tstruct bitmap *reuse;\n+\tsize_t i;\n+\tsize_t packs_nr = 0, packs_alloc = 0;\n+\tsize_t word_alloc;\n+\tuint32_t objects_nr = 0;\n+\n+\tassert(result);\n+\n+\tload_reverse_index(r, bitmap_git);\n+\n+\tif (bitmap_is_midx(bitmap_git)) {\n+\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\t\t\tstruct bitmapped_pack pack;\n+\t\t\tif (nth_bitmapped_pack(r, bitmap_git->midx, &pack, i) < 0) {\n+\t\t\t\twarning(_(\"unable to load pack: '%s', disabling pack-reuse\"),\n+\t\t\t\t\tbitmap_git->midx->pack_names[i]);\n+\t\t\t\tfree(packs);\n+\t\t\t\treturn -1;\n+\t\t\t}\n+\t\t\tif (!pack.bitmap_nr)\n+\t\t\t\tcontinue; /* no objects from this pack */\n+\t\t\tif (pack.bitmap_pos)\n+\t\t\t\tcontinue; /* not preferred pack */\n+\n+\t\t\tALLOC_GROW(packs, packs_nr + 1, packs_alloc);\n+\t\t\tmemcpy(&packs[packs_nr++], &pack, sizeof(pack));\n+\n+\t\t\tobjects_nr += pack.p->num_objects;\n+\t\t}\n+\t} else {\n+\t\tALLOC_GROW(packs, packs_nr + 1, packs_alloc);\n+\n+\t\tpacks[packs_nr].p = bitmap_git->pack;\n+\t\tpacks[packs_nr].bitmap_pos = 0;\n+\t\tpacks[packs_nr].bitmap_nr = bitmap_git->pack->num_objects;\n+\t\tpacks[packs_nr].disjoint = 1;\n+\n+\t\tobjects_nr = packs[packs_nr++].p->num_objects;\n+\t}\n+\n+\tword_alloc = objects_nr / BITS_IN_EWORD;\n+\tif (objects_nr % BITS_IN_EWORD)\n+\t\tword_alloc++;\n+\treuse = bitmap_word_alloc(word_alloc);\n+\n+\tif (packs_nr != 1)\n+\t\tBUG(\"pack reuse not yet implemented for multiple packs\");\n+\n+\treuse_partial_packfile_from_bitmap_1(bitmap_git, packs, reuse);\n \n \t*entries = bitmap_popcount(reuse);\n \tif (!*entries) {\n@@ -2014,7 +2057,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t * need to be handled separately.\n \t */\n \tbitmap_and_not(result, reuse);\n-\t*packfile_out = pack;\n+\t*packfile_out = packs[0].p;\n \t*reuse_out = reuse;\n \treturn 0;\n }\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485209","messageId":"432854b27c6731bd6ab1fa739b3a086ec0a90be8.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 11/24] pack-bitmap: simplify `reuse_partial_packfile_from_bitmap()` signature","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:24Z","receivedAt":"2023-11-28T19:08:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The signature of `reuse_partial_packfile_from_bitmap()` currently takes\nin a bitmap, as well as three output parameters (filled through\npointers, and passed as arguments), and also returns an integer result.\n\nThe output parameters are filled out with: (a) the packfile used for\npack-reuse, (b) the number of objects from that pack that we can reuse,\nand (c) a bitmap indicating which objects we can reuse. The return value\nis either -1 (when there are no objects to reuse), or 0 (when there is\nat least one object to reuse).\n\nSome of these parameters are redundant. Notably, we can infer from the\nbitmap how many objects are reused by calling bitmap_popcount(). And we\ncan similar compute the return value based on that number as well.\n\nAs such, clean up the signature of this function to drop the \"*entries\"\nparameter, as well as the int return value, since the single caller of\nthis function can infer these values themself.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 16 +++++++++-------\n pack-bitmap.c | 16 +++++++---------\n pack-bitmap.h | 7 +++----\n 3 files changed, 19 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 107154db34..2bb1b64e8f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3946,13 +3946,15 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n \t\treturn -1;\n \n-\tif (pack_options_allow_reuse() &&\n-\t !reuse_partial_packfile_from_bitmap(\n-\t\t\tbitmap_git,\n-\t\t\t&reuse_packfile,\n-\t\t\t&reuse_packfile_objects,\n-\t\t\t&reuse_packfile_bitmap)) {\n-\t\tassert(reuse_packfile_objects);\n+\tif (pack_options_allow_reuse())\n+\t\treuse_partial_packfile_from_bitmap(bitmap_git, &reuse_packfile,\n+\t\t\t\t\t\t &reuse_packfile_bitmap);\n+\n+\tif (reuse_packfile) {\n+\t\treuse_packfile_objects = bitmap_popcount(reuse_packfile_bitmap);\n+\t\tif (!reuse_packfile_objects)\n+\t\t\tBUG(\"expected non-empty reuse bitmap\");\n+\n \t\tnr_result += reuse_packfile_objects;\n \t\tnr_seen += reuse_packfile_objects;\n \t\tdisplay_progress(progress_state, nr_seen);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 2ebe2c314e..614fc09a4e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1988,10 +1988,9 @@ static void reuse_partial_packfile_from_bitmap_1(struct bitmap_index *bitmap_git\n \tunuse_pack(&w_curs);\n }\n \n-int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n-\t\t\t\t struct packed_git **packfile_out,\n-\t\t\t\t uint32_t *entries,\n-\t\t\t\t struct bitmap **reuse_out)\n+void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n+\t\t\t\t\tstruct packed_git **packfile_out,\n+\t\t\t\t\tstruct bitmap **reuse_out)\n {\n \tstruct repository *r = the_repository;\n \tstruct bitmapped_pack *packs = NULL;\n@@ -2013,7 +2012,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\twarning(_(\"unable to load pack: '%s', disabling pack-reuse\"),\n \t\t\t\t\tbitmap_git->midx->pack_names[i]);\n \t\t\t\tfree(packs);\n-\t\t\t\treturn -1;\n+\t\t\t\treturn;\n \t\t\t}\n \t\t\tif (!pack.bitmap_nr)\n \t\t\t\tcontinue; /* no objects from this pack */\n@@ -2046,10 +2045,10 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \n \treuse_partial_packfile_from_bitmap_1(bitmap_git, packs, reuse);\n \n-\t*entries = bitmap_popcount(reuse);\n-\tif (!*entries) {\n+\tif (!bitmap_popcount(reuse)) {\n+\t\tfree(packs);\n \t\tbitmap_free(reuse);\n-\t\treturn -1;\n+\t\treturn;\n \t}\n \n \t/*\n@@ -2059,7 +2058,6 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \tbitmap_and_not(result, reuse);\n \t*packfile_out = packs[0].p;\n \t*reuse_out = reuse;\n-\treturn 0;\n }\n \n int bitmap_walk_contains(struct bitmap_index *bitmap_git,\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex b7fa1a42a9..5bc1ca5b65 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -78,10 +78,9 @@ int test_bitmap_hashes(struct repository *r);\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\t\t\t int filter_provided_objects);\n uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git);\n-int reuse_partial_packfile_from_bitmap(struct bitmap_index *,\n-\t\t\t\t struct packed_git **packfile,\n-\t\t\t\t uint32_t *entries,\n-\t\t\t\t struct bitmap **reuse_out);\n+void reuse_partial_packfile_from_bitmap(struct bitmap_index *,\n+\t\t\t\t\tstruct packed_git **packfile,\n+\t\t\t\t\tstruct bitmap **reuse_out);\n int rebuild_existing_bitmaps(struct bitmap_index *, struct packing_data *mapping,\n \t\t\t kh_oid_map_t *reused_bitmaps, int show_progress);\n void free_bitmap_index(struct bitmap_index *);\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485210","messageId":"36b794d9e1eaa413b3b87eb7e5a0d82098d05dca.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 12/24] pack-bitmap: return multiple packs via `reuse_partial_packfile_from_bitmap()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:26Z","receivedAt":"2023-11-28T19:08:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Further prepare for enabling verbatim pack-reuse over multiple packfiles\nby changing the signature of reuse_partial_packfile_from_bitmap() to\npopulate an array of `struct bitmapped_pack *`'s instead of a pointer to\na single packfile.\n\nSince the array we're filling out is sized dynamically[^1], add an\nadditional `size_t *` parameter which will hold the number of reusable\npacks (equal to the number of elements in the array).\n\nNote that since we still have not implemented true multi-pack reuse,\nthese changes aren't propagated out to the rest of the caller in\nbuiltin/pack-objects.c.\n\nIn the interim state, we expect that the array has a single element, and\nwe use that element to fill out the static `reuse_packfile` variable\n(which is a bog-standard `struct packed_git *`). Future commits will\ncontinue to push this change further out through the pack-objects code.\n\n[^1]: That is, even though we know the number of packs which are\n candidates for pack-reuse, we do not know how many of those\n candidates we can actually reuse.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 9 +++++++--\n pack-bitmap.c | 6 ++++--\n pack-bitmap.h | 5 +++--\n 3 files changed, 14 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 2bb1b64e8f..89de23f39a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3943,14 +3943,19 @@ static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n+\tstruct bitmapped_pack *packs = NULL;\n+\tsize_t packs_nr = 0;\n+\n \tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse())\n-\t\treuse_partial_packfile_from_bitmap(bitmap_git, &reuse_packfile,\n+\t\treuse_partial_packfile_from_bitmap(bitmap_git, &packs,\n+\t\t\t\t\t\t &packs_nr,\n \t\t\t\t\t\t &reuse_packfile_bitmap);\n \n-\tif (reuse_packfile) {\n+\tif (packs) {\n+\t\treuse_packfile = packs[0].p;\n \t\treuse_packfile_objects = bitmap_popcount(reuse_packfile_bitmap);\n \t\tif (!reuse_packfile_objects)\n \t\t\tBUG(\"expected non-empty reuse bitmap\");\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 614fc09a4e..670deec909 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1989,7 +1989,8 @@ static void reuse_partial_packfile_from_bitmap_1(struct bitmap_index *bitmap_git\n }\n \n void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n-\t\t\t\t\tstruct packed_git **packfile_out,\n+\t\t\t\t\tstruct bitmapped_pack **packs_out,\n+\t\t\t\t\tsize_t *packs_nr_out,\n \t\t\t\t\tstruct bitmap **reuse_out)\n {\n \tstruct repository *r = the_repository;\n@@ -2056,7 +2057,8 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t * need to be handled separately.\n \t */\n \tbitmap_and_not(result, reuse);\n-\t*packfile_out = packs[0].p;\n+\t*packs_out = packs;\n+\t*packs_nr_out = packs_nr;\n \t*reuse_out = reuse;\n }\n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 5bc1ca5b65..901a3b86ed 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -78,8 +78,9 @@ int test_bitmap_hashes(struct repository *r);\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\t\t\t int filter_provided_objects);\n uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git);\n-void reuse_partial_packfile_from_bitmap(struct bitmap_index *,\n-\t\t\t\t\tstruct packed_git **packfile,\n+void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n+\t\t\t\t\tstruct bitmapped_pack **packs_out,\n+\t\t\t\t\tsize_t *packs_nr_out,\n \t\t\t\t\tstruct bitmap **reuse_out);\n int rebuild_existing_bitmaps(struct bitmap_index *, struct packing_data *mapping,\n \t\t\t kh_oid_map_t *reused_bitmaps, int show_progress);\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485211","messageId":"dca1083d8ec20cc6b224abf87959af933cca59ca.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 13/24] pack-objects: parameterize pack-reuse routines over a single pack","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:29Z","receivedAt":"2023-11-28T19:08:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The routines pack-objects uses to perform verbatim pack-reuse are:\n\n - write_reused_pack_one()\n - write_reused_pack_verbatim()\n - write_reused_pack()\n\n, all of which assume that there is exactly one packfile being reused:\nthe global constant `reuse_packfile`.\n\nPrepare for reusing objects from multiple packs by making reuse packfile\na parameter of each of the above functions in preparation for calling\nthese functions in a loop with multiple packfiles.\n\nNote that we still have the global \"reuse_packfile\", but pass it through\neach of the above function's parameter lists, eliminating all but one\ndirect access (the top-level caller in `write_pack_file()`). Even after\nthis series, we will still have a global, but it will hold the array of\nreusable packfiles, and we'll pass them one at a time to these functions\nin a loop.\n\nNote also that we will eventually need to pass a `bitmapped_pack`\ninstead of a `packed_git` in order to hold onto additional information\nrequired for reuse (such as the bit position of the first object\nbelonging to that pack). But that change will be made in a future commit\nso as to minimize the noise below as much as possible.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 16 ++++++++++------\n 1 file changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 89de23f39a..7682bd65bb 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1014,7 +1014,8 @@ static off_t find_reused_offset(off_t where)\n \treturn reused_chunks[lo-1].difference;\n }\n \n-static void write_reused_pack_one(size_t pos, struct hashfile *out,\n+static void write_reused_pack_one(struct packed_git *reuse_packfile,\n+\t\t\t\t size_t pos, struct hashfile *out,\n \t\t\t\t struct pack_window **w_curs)\n {\n \toff_t offset, next, cur;\n@@ -1092,7 +1093,8 @@ static void write_reused_pack_one(size_t pos, struct hashfile *out,\n \tcopy_pack_data(out, reuse_packfile, w_curs, offset, next - offset);\n }\n \n-static size_t write_reused_pack_verbatim(struct hashfile *out,\n+static size_t write_reused_pack_verbatim(struct packed_git *reuse_packfile,\n+\t\t\t\t\t struct hashfile *out,\n \t\t\t\t\t struct pack_window **w_curs)\n {\n \tsize_t pos = 0;\n@@ -1119,14 +1121,15 @@ static size_t write_reused_pack_verbatim(struct hashfile *out,\n \treturn pos;\n }\n \n-static void write_reused_pack(struct hashfile *f)\n+static void write_reused_pack(struct packed_git *reuse_packfile,\n+\t\t\t struct hashfile *f)\n {\n \tsize_t i = 0;\n \tuint32_t offset;\n \tstruct pack_window *w_curs = NULL;\n \n \tif (allow_ofs_delta)\n-\t\ti = write_reused_pack_verbatim(f, &w_curs);\n+\t\ti = write_reused_pack_verbatim(reuse_packfile, f, &w_curs);\n \n \tfor (; i < reuse_packfile_bitmap->word_alloc; ++i) {\n \t\teword_t word = reuse_packfile_bitmap->words[i];\n@@ -1142,7 +1145,8 @@ static void write_reused_pack(struct hashfile *f)\n \t\t\t * bitmaps. See comment in try_partial_reuse()\n \t\t\t * for why.\n \t\t\t */\n-\t\t\twrite_reused_pack_one(pos + offset, f, &w_curs);\n+\t\t\twrite_reused_pack_one(reuse_packfile, pos + offset, f,\n+\t\t\t\t\t &w_curs);\n \t\t\tdisplay_progress(progress_state, ++written);\n \t\t}\n \t}\n@@ -1200,7 +1204,7 @@ static void write_pack_file(void)\n \n \t\tif (reuse_packfile) {\n \t\t\tassert(pack_to_stdout);\n-\t\t\twrite_reused_pack(f);\n+\t\t\twrite_reused_pack(reuse_packfile, f);\n \t\t\toffset = hashfile_total(f);\n \t\t}\n \n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485212","messageId":"6f4fba861b59f909148775ee64c3ba89afc872b5.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 14/24] pack-objects: keep track of `pack_start` for each reuse pack","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:32Z","receivedAt":"2023-11-28T19:08:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When reusing objects from a pack, we keep track of a set of one or more\n`reused_chunk`s, corresponding to sections of one or more object(s) from\na source pack that we are reusing. Each chunk contains two pieces of\ninformation:\n\n - the offset of the first object in the source pack (relative to the\n beginning of the source pack)\n - the difference between that offset, and the corresponding offset in\n the pack we're generating\n\nThe purpose of keeping track of these is so that we can patch an\nOFS_DELTAs that cross over a section of the reuse pack that we didn't\ntake.\n\nFor instance, consider a hypothetical pack as shown below:\n\n (chunk #2)\n __________...\n /\n /\n +--------+---------+-------------------+---------+\n ... | | | (unused) | | ...\n +--------+---------+-------------------+---------+\n \\ /\n \\______________/\n (chunk #1)\n\nSuppose that we are sending objects \"base\", \"other\", and \"delta\", and\nthat the \"delta\" object is stored as an OFS_DELTA, and that its base is\n\"base\". If we don't send any objects in the \"(unused)\" range, we can't\ncopy the delta'd object directly, since its delta offset includes a\nrange of the pack that we didn't copy, so we have to account for that\ndifference when patching and reassembling the delta.\n\nIn order to compute this value correctly, we need to know not only where\nwe are in the packfile we're assembling (with `hashfile_total(f)`) but\nalso the position of the first byte of the packfile that we are\ncurrently reusing.\n\nTogether, these two allow us to compute the reused chunk's offset\ndifference relative to the start of the reused pack, as desired.\n\nHelped-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7682bd65bb..eb8be514d1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1016,6 +1016,7 @@ static off_t find_reused_offset(off_t where)\n \n static void write_reused_pack_one(struct packed_git *reuse_packfile,\n \t\t\t\t size_t pos, struct hashfile *out,\n+\t\t\t\t off_t pack_start,\n \t\t\t\t struct pack_window **w_curs)\n {\n \toff_t offset, next, cur;\n@@ -1025,7 +1026,8 @@ static void write_reused_pack_one(struct packed_git *reuse_packfile,\n \toffset = pack_pos_to_offset(reuse_packfile, pos);\n \tnext = pack_pos_to_offset(reuse_packfile, pos + 1);\n \n-\trecord_reused_object(offset, offset - hashfile_total(out));\n+\trecord_reused_object(offset,\n+\t\t\t offset - (hashfile_total(out) - pack_start));\n \n \tcur = offset;\n \ttype = unpack_object_header(reuse_packfile, w_curs, &cur, &size);\n@@ -1095,6 +1097,7 @@ static void write_reused_pack_one(struct packed_git *reuse_packfile,\n \n static size_t write_reused_pack_verbatim(struct packed_git *reuse_packfile,\n \t\t\t\t\t struct hashfile *out,\n+\t\t\t\t\t off_t pack_start UNUSED,\n \t\t\t\t\t struct pack_window **w_curs)\n {\n \tsize_t pos = 0;\n@@ -1126,10 +1129,12 @@ static void write_reused_pack(struct packed_git *reuse_packfile,\n {\n \tsize_t i = 0;\n \tuint32_t offset;\n+\toff_t pack_start = hashfile_total(f) - sizeof(struct pack_header);\n \tstruct pack_window *w_curs = NULL;\n \n \tif (allow_ofs_delta)\n-\t\ti = write_reused_pack_verbatim(reuse_packfile, f, &w_curs);\n+\t\ti = write_reused_pack_verbatim(reuse_packfile, f, pack_start,\n+\t\t\t\t\t &w_curs);\n \n \tfor (; i < reuse_packfile_bitmap->word_alloc; ++i) {\n \t\teword_t word = reuse_packfile_bitmap->words[i];\n@@ -1146,7 +1151,7 @@ static void write_reused_pack(struct packed_git *reuse_packfile,\n \t\t\t * for why.\n \t\t\t */\n \t\t\twrite_reused_pack_one(reuse_packfile, pos + offset, f,\n-\t\t\t\t\t &w_curs);\n+\t\t\t\t\t pack_start, &w_curs);\n \t\t\tdisplay_progress(progress_state, ++written);\n \t\t}\n \t}\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485213","messageId":"2bb01e2370fd01d8468e69da474c24266ba851fb.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 15/24] pack-objects: pass `bitmapped_pack`'s to pack-reuse functions","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:34Z","receivedAt":"2023-11-28T19:08:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Further prepare pack-objects to perform verbatim pack-reuse over\nmultiple packfiles by converting functions that take in a pointer to a\n`struct packed_git` to instead take in a pointer to a `struct\nbitmapped_pack`.\n\nThe additional information found in the bitmapped_pack struct (such as\nthe bit position corresponding to the beginning of the pack) will be\nnecessary in order to perform verbatim pack-reuse.\n\nNote that we don't use any of the extra pieces of information contained\nin the bitmapped_pack struct, so this step is merely preparatory and\ndoes not introduce any functional changes.\n\nNote further that we do not change the argument type to\nwrite_reused_pack_one(). That function is responsible for copying\nsections of the packfile directly and optionally patching any OFS_DELTAs\nto account for not reusing sections of the packfile in between a delta\nand its base.\n\nAs such, that function is (and should remain) oblivious to multi-pack\nreuse, and does not require any of the extra pieces of information\nstored in the bitmapped_pack struct.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 33 +++++++++++++++++----------------\n 1 file changed, 17 insertions(+), 16 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex eb8be514d1..3b7704d062 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -222,7 +222,8 @@ static int thin;\n static int num_preferred_base;\n static struct progress *progress_state;\n \n-static struct packed_git *reuse_packfile;\n+static struct bitmapped_pack *reuse_packfiles;\n+static size_t reuse_packfiles_nr;\n static uint32_t reuse_packfile_objects;\n static struct bitmap *reuse_packfile_bitmap;\n \n@@ -1095,7 +1096,7 @@ static void write_reused_pack_one(struct packed_git *reuse_packfile,\n \tcopy_pack_data(out, reuse_packfile, w_curs, offset, next - offset);\n }\n \n-static size_t write_reused_pack_verbatim(struct packed_git *reuse_packfile,\n+static size_t write_reused_pack_verbatim(struct bitmapped_pack *reuse_packfile,\n \t\t\t\t\t struct hashfile *out,\n \t\t\t\t\t off_t pack_start UNUSED,\n \t\t\t\t\t struct pack_window **w_curs)\n@@ -1110,13 +1111,13 @@ static size_t write_reused_pack_verbatim(struct packed_git *reuse_packfile,\n \t\toff_t to_write;\n \n \t\twritten = (pos * BITS_IN_EWORD);\n-\t\tto_write = pack_pos_to_offset(reuse_packfile, written)\n+\t\tto_write = pack_pos_to_offset(reuse_packfile->p, written)\n \t\t\t- sizeof(struct pack_header);\n \n \t\t/* We're recording one chunk, not one object. */\n \t\trecord_reused_object(sizeof(struct pack_header), 0);\n \t\thashflush(out);\n-\t\tcopy_pack_data(out, reuse_packfile, w_curs,\n+\t\tcopy_pack_data(out, reuse_packfile->p, w_curs,\n \t\t\tsizeof(struct pack_header), to_write);\n \n \t\tdisplay_progress(progress_state, written);\n@@ -1124,7 +1125,7 @@ static size_t write_reused_pack_verbatim(struct packed_git *reuse_packfile,\n \treturn pos;\n }\n \n-static void write_reused_pack(struct packed_git *reuse_packfile,\n+static void write_reused_pack(struct bitmapped_pack *reuse_packfile,\n \t\t\t struct hashfile *f)\n {\n \tsize_t i = 0;\n@@ -1150,8 +1151,8 @@ static void write_reused_pack(struct packed_git *reuse_packfile,\n \t\t\t * bitmaps. See comment in try_partial_reuse()\n \t\t\t * for why.\n \t\t\t */\n-\t\t\twrite_reused_pack_one(reuse_packfile, pos + offset, f,\n-\t\t\t\t\t pack_start, &w_curs);\n+\t\t\twrite_reused_pack_one(reuse_packfile->p, pos + offset,\n+\t\t\t\t\t f, pack_start, &w_curs);\n \t\t\tdisplay_progress(progress_state, ++written);\n \t\t}\n \t}\n@@ -1207,9 +1208,12 @@ static void write_pack_file(void)\n \n \t\toffset = write_pack_header(f, nr_remaining);\n \n-\t\tif (reuse_packfile) {\n+\t\tif (reuse_packfiles_nr) {\n \t\t\tassert(pack_to_stdout);\n-\t\t\twrite_reused_pack(reuse_packfile, f);\n+\t\t\tfor (j = 0; j < reuse_packfiles_nr; j++) {\n+\t\t\t\treused_chunks_nr = 0;\n+\t\t\t\twrite_reused_pack(&reuse_packfiles[j], f);\n+\t\t\t}\n \t\t\toffset = hashfile_total(f);\n \t\t}\n \n@@ -3952,19 +3956,16 @@ static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n-\tstruct bitmapped_pack *packs = NULL;\n-\tsize_t packs_nr = 0;\n-\n \tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse())\n-\t\treuse_partial_packfile_from_bitmap(bitmap_git, &packs,\n-\t\t\t\t\t\t &packs_nr,\n+\t\treuse_partial_packfile_from_bitmap(bitmap_git,\n+\t\t\t\t\t\t &reuse_packfiles,\n+\t\t\t\t\t\t &reuse_packfiles_nr,\n \t\t\t\t\t\t &reuse_packfile_bitmap);\n \n-\tif (packs) {\n-\t\treuse_packfile = packs[0].p;\n+\tif (reuse_packfiles) {\n \t\treuse_packfile_objects = bitmap_popcount(reuse_packfile_bitmap);\n \t\tif (!reuse_packfile_objects)\n \t\t\tBUG(\"expected non-empty reuse bitmap\");\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485214","messageId":"67a8196978244b56d4f60861f140b78c59d15e30.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 16/24] pack-objects: prepare `write_reused_pack()` for multi-pack reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:37Z","receivedAt":"2023-11-28T19:08:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `write_reused_pack()` within `builtin/pack-objects.c` is\nresponsible for performing pack-reuse on a single pack, and has two main\nfunctions:\n\n - it dispatches a call to `write_reused_pack_verbatim()` to see if we\n can reuse portions of the packfile in whole-word chunks\n\n - for any remaining objects (that is, any objects that appear after\n the first \"gap\" in the bitmap), call write_reused_pack_one() on that\n object to record it for reuse.\n\nPrepare this function for multi-pack reuse by removing the assumption\nthat the bit position corresponding to the first object being reused\nfrom a given pack may not be at bit position zero.\n\nThe changes in this function are mostly straightforward. Initialize `i`\nto the position of the first word to contain bits corresponding to that\nreuse pack. In most situations, we throw the initialized value away,\nsince we end up replacing it with the return value from\nwrite_reused_pack_verbatim(), moving us past the section of whole words\nthat we reused.\n\nLikewise, modify the per-object loop to ignore any bits at the beginning\nof the first word that do not belong to the pack currently being reused,\nas well as skip to the \"done\" section once we have processed the last\nbit corresponding to this pack.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 3b7704d062..b5e6f6377a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1128,7 +1128,7 @@ static size_t write_reused_pack_verbatim(struct bitmapped_pack *reuse_packfile,\n static void write_reused_pack(struct bitmapped_pack *reuse_packfile,\n \t\t\t struct hashfile *f)\n {\n-\tsize_t i = 0;\n+\tsize_t i = reuse_packfile->bitmap_pos / BITS_IN_EWORD;\n \tuint32_t offset;\n \toff_t pack_start = hashfile_total(f) - sizeof(struct pack_header);\n \tstruct pack_window *w_curs = NULL;\n@@ -1146,17 +1146,23 @@ static void write_reused_pack(struct bitmapped_pack *reuse_packfile,\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n+\t\t\tif (pos + offset < reuse_packfile->bitmap_pos)\n+\t\t\t\tcontinue;\n+\t\t\tif (pos + offset >= reuse_packfile->bitmap_pos + reuse_packfile->bitmap_nr)\n+\t\t\t\tgoto done;\n \t\t\t/*\n \t\t\t * Can use bit positions directly, even for MIDX\n \t\t\t * bitmaps. See comment in try_partial_reuse()\n \t\t\t * for why.\n \t\t\t */\n-\t\t\twrite_reused_pack_one(reuse_packfile->p, pos + offset,\n+\t\t\twrite_reused_pack_one(reuse_packfile->p,\n+\t\t\t\t\t pos + offset - reuse_packfile->bitmap_pos,\n \t\t\t\t\t f, pack_start, &w_curs);\n \t\t\tdisplay_progress(progress_state, ++written);\n \t\t}\n \t}\n \n+done:\n \tunuse_pack(&w_curs);\n }\n \n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485215","messageId":"1f766648df5f80a178d6eae31cb25fc8484ef84e.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 17/24] pack-objects: prepare `write_reused_pack_verbatim()` for multi-pack reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:40Z","receivedAt":"2023-11-28T19:08:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `write_reused_pack_verbatim()` within\n`builtin/pack-objects.c` is responsible for writing out a continuous\nset of objects beginning at the start of the reuse packfile.\n\nIn the existing implementation, we did something like:\n\n while (pos < reuse_packfile_bitmap->word_alloc &&\n reuse_packfile_bitmap->words[pos] == (eword_t)~0)\n pos++;\n\n if (pos)\n /* write first `pos * BITS_IN_WORD` objects from pack */\n\nas an optimization to record a single chunk for the longest continuous\nprefix of objects wanted out of the reuse pack, instead of having a\nchunk for each individual object. For more details, see bb514de356\n(pack-objects: improve partial packfile reuse, 2019-12-18).\n\nIn order to retain this optimization in a multi-pack reuse world, we can\nno longer assume that the first object in a pack is on a word boundary\nin the bitmap storing the set of reusable objects.\n\nAssuming that all objects from the beginning of the reuse packfile up to\nthe object corresponding to the first bit on a word boundary are part of\nthe result, consume whole words at a time until the last whole word\nbelonging to the reuse packfile. Copy those objects to the resulting\npackfile, and track that we reused them by recording a single chunk.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 73 ++++++++++++++++++++++++++++++++++--------\n 1 file changed, 60 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b5e6f6377a..e37509568b 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1098,31 +1098,78 @@ static void write_reused_pack_one(struct packed_git *reuse_packfile,\n \n static size_t write_reused_pack_verbatim(struct bitmapped_pack *reuse_packfile,\n \t\t\t\t\t struct hashfile *out,\n-\t\t\t\t\t off_t pack_start UNUSED,\n+\t\t\t\t\t off_t pack_start,\n \t\t\t\t\t struct pack_window **w_curs)\n {\n-\tsize_t pos = 0;\n+\tsize_t pos = reuse_packfile->bitmap_pos;\n+\tsize_t end;\n \n-\twhile (pos < reuse_packfile_bitmap->word_alloc &&\n-\t\t\treuse_packfile_bitmap->words[pos] == (eword_t)~0)\n-\t\tpos++;\n+\tif (pos % BITS_IN_EWORD) {\n+\t\tsize_t word_pos = (pos / BITS_IN_EWORD);\n+\t\tsize_t offset = pos % BITS_IN_EWORD;\n+\t\tsize_t last;\n+\t\teword_t word = reuse_packfile_bitmap->words[word_pos];\n \n-\tif (pos) {\n-\t\toff_t to_write;\n+\t\tif (offset + reuse_packfile->bitmap_nr < BITS_IN_EWORD)\n+\t\t\tlast = offset + reuse_packfile->bitmap_nr;\n+\t\telse\n+\t\t\tlast = BITS_IN_EWORD;\n \n-\t\twritten = (pos * BITS_IN_EWORD);\n-\t\tto_write = pack_pos_to_offset(reuse_packfile->p, written)\n-\t\t\t- sizeof(struct pack_header);\n+\t\tfor (; offset < last; offset++) {\n+\t\t\tif (word >> offset == 0)\n+\t\t\t\treturn word_pos;\n+\t\t\tif (!bitmap_get(reuse_packfile_bitmap,\n+\t\t\t\t\tword_pos * BITS_IN_EWORD + offset))\n+\t\t\t\treturn word_pos;\n+\t\t}\n+\n+\t\tpos += BITS_IN_EWORD - (pos % BITS_IN_EWORD);\n+\t}\n+\n+\t/*\n+\t * Now we're going to copy as many whole eword_t's as possible.\n+\t * \"end\" is the index of the last whole eword_t we copy, but\n+\t * there may be additional bits to process. Those are handled\n+\t * individually by write_reused_pack().\n+\t *\n+\t * Begin by advancing to the first word boundary in range of the\n+\t * bit positions occupied by objects in \"reuse_packfile\". Then\n+\t * pick the last word boundary in the same range. If we have at\n+\t * least one word's worth of bits to process, continue on.\n+\t */\n+\tend = reuse_packfile->bitmap_pos + reuse_packfile->bitmap_nr;\n+\tif (end % BITS_IN_EWORD)\n+\t\tend -= end % BITS_IN_EWORD;\n+\tif (pos >= end)\n+\t\treturn reuse_packfile->bitmap_pos / BITS_IN_EWORD;\n+\n+\twhile (pos < end &&\n+\t reuse_packfile_bitmap->words[pos / BITS_IN_EWORD] == (eword_t)~0)\n+\t\tpos += BITS_IN_EWORD;\n+\n+\tif (pos > end)\n+\t\tpos = end;\n+\n+\tif (reuse_packfile->bitmap_pos < pos) {\n+\t\toff_t pack_start_off = pack_pos_to_offset(reuse_packfile->p, 0);\n+\t\toff_t pack_end_off = pack_pos_to_offset(reuse_packfile->p,\n+\t\t\t\t\t\t\tpos - reuse_packfile->bitmap_pos);\n+\n+\t\twritten += pos - reuse_packfile->bitmap_pos;\n \n \t\t/* We're recording one chunk, not one object. */\n-\t\trecord_reused_object(sizeof(struct pack_header), 0);\n+\t\trecord_reused_object(pack_start_off,\n+\t\t\t\t pack_start_off - (hashfile_total(out) - pack_start));\n \t\thashflush(out);\n \t\tcopy_pack_data(out, reuse_packfile->p, w_curs,\n-\t\t\tsizeof(struct pack_header), to_write);\n+\t\t\tpack_start_off, pack_end_off - pack_start_off);\n \n \t\tdisplay_progress(progress_state, written);\n \t}\n-\treturn pos;\n+\tif (pos % BITS_IN_EWORD)\n+\t\tBUG(\"attempted to jump past a word boundary to %\"PRIuMAX,\n+\t\t (uintmax_t)pos);\n+\treturn pos / BITS_IN_EWORD;\n }\n \n static void write_reused_pack(struct bitmapped_pack *reuse_packfile,\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485216","messageId":"4cd9f99bfdda0fa7db2372ca194572df1000304f.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 18/24] pack-objects: include number of packs reused in output","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:42Z","receivedAt":"2023-11-28T19:08:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In addition to including the number of objects reused verbatim from a\nreuse-pack, include the number of packs from which objects were reused.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 8 ++++++--\n 1 file changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e37509568b..902e70abc5 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -224,6 +224,7 @@ static struct progress *progress_state;\n \n static struct bitmapped_pack *reuse_packfiles;\n static size_t reuse_packfiles_nr;\n+static size_t reuse_packfiles_used_nr;\n static uint32_t reuse_packfile_objects;\n static struct bitmap *reuse_packfile_bitmap;\n \n@@ -1266,6 +1267,8 @@ static void write_pack_file(void)\n \t\t\tfor (j = 0; j < reuse_packfiles_nr; j++) {\n \t\t\t\treused_chunks_nr = 0;\n \t\t\t\twrite_reused_pack(&reuse_packfiles[j], f);\n+\t\t\t\tif (reused_chunks_nr)\n+\t\t\t\t\treuse_packfiles_used_nr++;\n \t\t\t}\n \t\t\toffset = hashfile_total(f);\n \t\t}\n@@ -4612,9 +4615,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tfprintf_ln(stderr,\n \t\t\t _(\"Total %\"PRIu32\" (delta %\"PRIu32\"),\"\n \t\t\t \" reused %\"PRIu32\" (delta %\"PRIu32\"),\"\n-\t\t\t \" pack-reused %\"PRIu32),\n+\t\t\t \" pack-reused %\"PRIu32\" (from %\"PRIuMAX\")\"),\n \t\t\t written, written_delta, reused, reused_delta,\n-\t\t\t reuse_packfile_objects);\n+\t\t\t reuse_packfile_objects,\n+\t\t\t (uintmax_t)reuse_packfiles_used_nr);\n \n cleanup:\n \tfree_packing_data(&to_pack);\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485217","messageId":"176c4c95ac90a37b9ccf684e39126836d2cadc97.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 19/24] pack-bitmap: prepare to mark objects from multiple packs for reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:45Z","receivedAt":"2023-11-28T19:08:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the pack-objects code is equipped to handle reusing objects\nfrom multiple packs, prepare the pack-bitmap code to mark objects from\nmultiple packs as reuse candidates.\n\nIn order to prepare the pack-bitmap code for this change, remove the\nsame set of assumptions we unwound in previous commits from the helper\nfunction `reuse_partial_packfile_from_bitmap_1()`, in preparation for it\nto be called in a loop over the set of disjoint packs in a following\ncommit.\n\nSpecifically, remove the assumption that the bit position corresponding\nto the first object in a given reuse pack candidate is at a word\nboundary. Like in previous commits, we have to walk up to the first word\nboundary before marking whole words at a time for reuse. Unlike in\nprevious commits, however, we have to keep track of whether all of the\nobjects in the run-up to the first word boundary are wanted in the\nresulting pack. This is because we cannot blindly reuse whole words at a\ntime unless we know for certain that we are sending all bases for any\nobjects stored as deltas within each word.\n\nOnce we're on a word boundary (provided that we want a complete prefix\nof objects from the pack), we can then reuse the same \"whole-words\"\noptimization from previous patches, marking all of the bits in a single\nword at a time.\n\nAny remaining objects (either from a partial word corresponding to\nobjects at the end of a pack, or starting from the middle of the pack if\nwe do not have a complete prefix) are dealt with individually via\ntry_partial_reuse(), which (among other things) ensures that we are\nsending the necessary bases for all objects packed as OFS_DELTA or\nREF_DELTA.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 113 +++++++++++++++++++++++++++++++++++++++++---------\n 1 file changed, 93 insertions(+), 20 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 670deec909..be53fc6da5 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1940,36 +1940,109 @@ static void reuse_partial_packfile_from_bitmap_1(struct bitmap_index *bitmap_git\n {\n \tstruct bitmap *result = bitmap_git->result;\n \tstruct pack_window *w_curs = NULL;\n-\tsize_t i = 0;\n+\tsize_t pos, offset;\n+\tunsigned complete_prefix = 1;\n \n-\twhile (i < result->word_alloc && result->words[i] == (eword_t)~0)\n-\t\ti++;\n+\tpos = pack->bitmap_pos / BITS_IN_EWORD;\n+\toffset = pack->bitmap_pos % BITS_IN_EWORD;\n \n \t/*\n-\t * Don't mark objects not in the packfile or preferred pack. This bitmap\n-\t * marks objects eligible for reuse, but the pack-reuse code only\n-\t * understands how to reuse a single pack. Since the preferred pack is\n-\t * guaranteed to have all bases for its deltas (in a multi-pack bitmap),\n-\t * we use it instead of another pack. In single-pack bitmaps, the choice\n-\t * is made for us.\n+\t * If the position of our first object is not on a word\n+\t * boundary, check all bits individually until we reach the\n+\t * first word boundary.\n+\t *\n+\t * If no bits are missing between pack->bitmap_pos and the next\n+\t * word boundary, then we can move by whole words instead of by\n+\t * individual objects. If one or more of the objects are missing\n+\t * in that range, we must evaluate each subsequent object\n+\t * individually in order to exclude deltas whose base we are not\n+\t * sending, etc.\n \t */\n-\tif (i > pack->p->num_objects / BITS_IN_EWORD)\n-\t\ti = pack->p->num_objects / BITS_IN_EWORD;\n+\tif (offset) {\n+\t\t/*\n+\t\t * Scan to the next word boundary, or through the last\n+\t\t * object in this bitmap, whichever occurs earlier.\n+\t\t */\n+\t\tsize_t last;\n+\t\teword_t word = result->words[pos];\n+\t\tif (pack->bitmap_nr < BITS_IN_EWORD - offset)\n+\t\t\tlast = offset + pack->bitmap_nr;\n+\t\telse\n+\t\t\tlast = BITS_IN_EWORD;\n \n-\tmemset(reuse->words, 0xFF, i * sizeof(eword_t));\n+\t\tfor (; offset < last; offset++) {\n+\t\t\tsize_t pack_pos;\n+\t\t\tif (word >> offset == 0) {\n+\t\t\t\tcomplete_prefix = 0;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \n-\tfor (; i < result->word_alloc; ++i) {\n-\t\teword_t word = result->words[i];\n-\t\tsize_t pos = (i * BITS_IN_EWORD);\n-\t\tsize_t offset;\n+\t\t\toffset += ewah_bit_ctz64(word >> offset);\n \n-\t\tfor (offset = 0; offset < BITS_IN_EWORD; ++offset) {\n-\t\t\tif ((word >> offset) == 0)\n+\t\t\tpack_pos = pos * BITS_IN_EWORD + offset;\n+\t\t\tpack_pos -= pack->bitmap_pos;\n+\n+\t\t\ttry_partial_reuse(pack, pack_pos, reuse, &w_curs);\n+\t\t\tif (!bitmap_get(reuse, pos + pack->bitmap_pos))\n+\t\t\t\tcomplete_prefix = 0;\n+\t\t}\n+\n+\t\tpos++;\n+\t}\n+\n+\tif (complete_prefix) {\n+\t\t/*\n+\t\t * If we are using all of the objects at the beginning\n+\t\t * of this pack, we can safely reuse objects in eword_t\n+\t\t * sized chunks, since we are guaranteed to send all\n+\t\t * potential delta bases.\n+\t\t *\n+\t\t * Scan the nearest word boundaries within range of this\n+\t\t * pack's bit positions. If the pack does not start on a\n+\t\t * word boundary, skip to the next boundary, since we\n+\t\t * have already checked above.\n+\t\t */\n+\t\tsize_t start = pos;\n+\t\tsize_t word_end = (pack->bitmap_pos + pack->bitmap_nr) / BITS_IN_EWORD;\n+\t\twhile (start <= pos && pos < word_end &&\n+\t\t pos < result->word_alloc &&\n+\t\t result->words[pos] == (eword_t)~0)\n+\t\t\tpos++;\n+\t\tmemset(reuse->words + start, 0xFF, (pos - start) * sizeof(eword_t));\n+\t}\n+\n+\t/*\n+\t * At this point, we know that we are at an eword boundary,\n+\t * either because:\n+\t *\n+\t * - we started at one and used zero or more whole words\n+\t * following pack->bitmap_pos\n+\t *\n+\t * - we started in between two word boundaries, advanced\n+\t * forward to the next word boundary, and then used zero or\n+\t * more (assuming a complete prefix) whole words following.\n+\t */\n+\tfor (; pos < result->word_alloc; pos++) {\n+\t\teword_t word = result->words[pos];\n+\n+\t\tfor (offset = 0; offset < BITS_IN_EWORD; offset++) {\n+\t\t\tsize_t bit_pos, pack_pos;\n+\t\t\tif (word >> offset == 0)\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n-\t\t\tif (try_partial_reuse(pack, pos + offset,\n-\t\t\t\t\t reuse, &w_curs) < 0) {\n+\n+\t\t\tbit_pos = pos * BITS_IN_EWORD + offset;\n+\t\t\tif (bit_pos >= pack->bitmap_pos + pack->bitmap_nr)\n+\t\t\t\tgoto done;\n+\n+\t\t\tpack_pos = bit_pos - pack->bitmap_pos;\n+\t\t\tif (pack_pos >= pack->p->num_objects)\n+\t\t\t\tBUG(\"advanced beyond the end of pack %s (%\"PRIuMAX\" > %\"PRIu32\")\",\n+\t\t\t\t pack_basename(pack->p), (uintmax_t)pack_pos,\n+\t\t\t\t pack->p->num_objects);\n+\n+\t\t\tif (try_partial_reuse(pack, pack_pos, reuse, &w_curs) < 0) {\n \t\t\t\t/*\n \t\t\t\t * try_partial_reuse indicated we couldn't reuse\n \t\t\t\t * any bits, so there is no point in trying more\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485218","messageId":"46f1a03b8b15f0c4432c26aca1e842670638c798.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 20/24] pack-objects: add tracing for various packfile metrics","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:47Z","receivedAt":"2023-11-28T19:08:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"As part of the multi-pack reuse effort, we will want to add some tests\nthat assert that we reused a certain number of objects from a certain\nnumber of packs.\n\nWe could do this by grepping through the stderr output of\n`pack-objects`, but doing so would be brittle in case the output format\nchanged.\n\nInstead, let's use the trace2 mechanism to log various pieces of\ninformation about the generated packfile, which we can then use to\ncompare against desired values.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 902e70abc5..fa71fe1ccf 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4620,6 +4620,13 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t reuse_packfile_objects,\n \t\t\t (uintmax_t)reuse_packfiles_used_nr);\n \n+\ttrace2_data_intmax(\"pack-objects\", the_repository, \"written\", written);\n+\ttrace2_data_intmax(\"pack-objects\", the_repository, \"written/delta\", written_delta);\n+\ttrace2_data_intmax(\"pack-objects\", the_repository, \"reused\", reused);\n+\ttrace2_data_intmax(\"pack-objects\", the_repository, \"reused/delta\", reused_delta);\n+\ttrace2_data_intmax(\"pack-objects\", the_repository, \"pack-reused\", reuse_packfile_objects);\n+\ttrace2_data_intmax(\"pack-objects\", the_repository, \"packs-reused\", reuse_packfiles_used_nr);\n+\n cleanup:\n \tfree_packing_data(&to_pack);\n \tlist_objects_filter_release(&filter_options);\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485219","messageId":"fb764fbacceb2796f2e9973ee10c3946978586c4.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 21/24] t/test-lib-functions.sh: implement `test_trace2_data` helper","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:50Z","receivedAt":"2023-11-28T19:08:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a helper function which looks for a specific (category, key,\nvalue) tuple in the output of a trace2 event stream.\n\nWe will use this function in a future patch to ensure that the expected\nnumber of objects are reused from an expected number of packs.\n\nSigned-off-by: Taylor Blau \n---\n t/test-lib-functions.sh | 14 ++++++++++++++\n 1 file changed, 14 insertions(+)\n\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 9c3cf12b26..93fe819b0a 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1874,6 +1874,20 @@ test_region () {\n \treturn 0\n }\n \n+# Check that the given data fragment was included as part of the\n+# trace2-format trace on stdin.\n+#\n+#\ttest_trace2_data \n+#\n+# For example, to look for trace2_data_intmax(\"pack-objects\", repo,\n+# \"reused\", N) in an invocation of \"git pack-objects\", run:\n+#\n+#\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" git pack-objects ... &&\n+#\ttest_trace2_data pack-objects reused N \n---\n Documentation/config/pack.txt | 2 +-\n builtin/pack-objects.c | 19 ++++++++++++++++---\n 2 files changed, 17 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\nindex f50df9dbce..fe100d0fb7 100644\n--- a/Documentation/config/pack.txt\n+++ b/Documentation/config/pack.txt\n@@ -28,7 +28,7 @@ all existing objects. You can force recompression by passing the -F option\n to linkgit:git-repack[1].\n \n pack.allowPackReuse::\n-\tWhen true, and when reachability bitmaps are enabled,\n+\tWhen true or \"single\", and when reachability bitmaps are enabled,\n \tpack-objects will try to send parts of the bitmapped packfile\n \tverbatim. This can reduce memory and CPU usage to serve fetches,\n \tbut might result in sending a slightly larger pack. Defaults to\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex fa71fe1ccf..4853e91251 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -230,7 +230,10 @@ static struct bitmap *reuse_packfile_bitmap;\n \n static int use_bitmap_index_default = 1;\n static int use_bitmap_index = -1;\n-static int allow_pack_reuse = 1;\n+static enum {\n+\tNO_PACK_REUSE = 0,\n+\tSINGLE_PACK_REUSE,\n+} allow_pack_reuse = SINGLE_PACK_REUSE;\n static enum {\n \tWRITE_BITMAP_FALSE = 0,\n \tWRITE_BITMAP_QUIET,\n@@ -3246,7 +3249,17 @@ static int git_pack_config(const char *k, const char *v,\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.allowpackreuse\")) {\n-\t\tallow_pack_reuse = git_config_bool(k, v);\n+\t\tint res = git_parse_maybe_bool_text(v);\n+\t\tif (res < 0) {\n+\t\t\tif (!strcasecmp(v, \"single\"))\n+\t\t\t\tallow_pack_reuse = SINGLE_PACK_REUSE;\n+\t\t\telse\n+\t\t\t\tdie(_(\"invalid pack.allowPackReuse value: '%s'\"), v);\n+\t\t} else if (res) {\n+\t\t\tallow_pack_reuse = SINGLE_PACK_REUSE;\n+\t\t} else {\n+\t\t\tallow_pack_reuse = NO_PACK_REUSE;\n+\t\t}\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.threads\")) {\n@@ -4002,7 +4015,7 @@ static void loosen_unused_packed_objects(void)\n */\n static int pack_options_allow_reuse(void)\n {\n-\treturn allow_pack_reuse &&\n+\treturn allow_pack_reuse != NO_PACK_REUSE &&\n \t pack_to_stdout &&\n \t !ignore_packed_keep_on_disk &&\n \t !ignore_packed_keep_in_core &&\n-- \n2.43.0.24.g980b318f98\n\n"},{"id":"485221","messageId":"7345e3946743cd098a57c384436a3f44592e5fc6.1701198172.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH 23/24] pack-bitmap: reuse objects from all disjoint packs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-28T19:08:56Z","receivedAt":"2023-11-28T19:08:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that both the pack-bitmap and pack-objects code are prepared to\nhandle marking and using objects from multiple disjoint packs for\nverbatim reuse, allow marking objects from all disjoint packs as\neligible for reuse.\n\nWithin the `reuse_partial_packfile_from_bitmap()` function, we no longer\nonly mark the pack whose first object is at bit position zero for reuse,\nand instead mark any pack which is flagged as disjoint by the MIDX as a\nreuse candidate. If no such packs exist (i.e because we are reading a\nMIDX written before the \"DISP\" chunk was introduced), then treat the\npreferred pack as disjoint for the purposes of reuse. This is a safe\nassumption to make since all duplicate objects are resolved in favor of\nthe preferred pack.\n\nProvide a handful of test cases in a new script (t5332) exercising\ninteresting behavior for multi-pack reuse to ensure that we performed\nall of the previous steps correctly.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/pack.txt | 6 +-\n builtin/pack-objects.c | 6 +-\n pack-bitmap.c | 73 +++++++++---\n pack-bitmap.h | 3 +-\n t/t5332-multi-pack-reuse.sh | 219 ++++++++++++++++++++++++++++++++++\n 5 files changed, 290 insertions(+), 17 deletions(-)\n create mode 100755 t/t5332-multi-pack-reuse.sh\n\ndiff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\nindex fe100d0fb7..9fe48d41c9 100644\n--- a/Documentation/config/pack.txt\n+++ b/Documentation/config/pack.txt\n@@ -30,7 +30,11 @@ to linkgit:git-repack[1].\n pack.allowPackReuse::\n \tWhen true or \"single\", and when reachability bitmaps are enabled,\n \tpack-objects will try to send parts of the bitmapped packfile\n-\tverbatim. This can reduce memory and CPU usage to serve fetches,\n+\tverbatim. When \"multi\", and when a multi-pack reachability bitmap is\n+\tavailable, pack-objects will try to send parts of all packs marked as\n+\tdisjoint by the MIDX. If only a single pack bitmap is available, and\n+\t`pack.allowPackReuse` is set to \"multi\", reuse parts of just the\n+\tbitmapped packfile. This can reduce memory and CPU usage to serve fetches,\n \tbut might result in sending a slightly larger pack. Defaults to\n \ttrue.\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 4853e91251..43b77bff7c 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -233,6 +233,7 @@ static int use_bitmap_index = -1;\n static enum {\n \tNO_PACK_REUSE = 0,\n \tSINGLE_PACK_REUSE,\n+\tMULTI_PACK_REUSE,\n } allow_pack_reuse = SINGLE_PACK_REUSE;\n static enum {\n \tWRITE_BITMAP_FALSE = 0,\n@@ -3253,6 +3254,8 @@ static int git_pack_config(const char *k, const char *v,\n \t\tif (res < 0) {\n \t\t\tif (!strcasecmp(v, \"single\"))\n \t\t\t\tallow_pack_reuse = SINGLE_PACK_REUSE;\n+\t\t\telse if (!strcasecmp(v, \"multi\"))\n+\t\t\t\tallow_pack_reuse = MULTI_PACK_REUSE;\n \t\t\telse\n \t\t\t\tdie(_(\"invalid pack.allowPackReuse value: '%s'\"), v);\n \t\t} else if (res) {\n@@ -4032,7 +4035,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \t\treuse_partial_packfile_from_bitmap(bitmap_git,\n \t\t\t\t\t\t &reuse_packfiles,\n \t\t\t\t\t\t &reuse_packfiles_nr,\n-\t\t\t\t\t\t &reuse_packfile_bitmap);\n+\t\t\t\t\t\t &reuse_packfile_bitmap,\n+\t\t\t\t\t\t allow_pack_reuse == MULTI_PACK_REUSE);\n \n \tif (reuse_packfiles) {\n \t\treuse_packfile_objects = bitmap_popcount(reuse_packfile_bitmap);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex be53fc6da5..561690c679 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -2061,10 +2061,19 @@ static void reuse_partial_packfile_from_bitmap_1(struct bitmap_index *bitmap_git\n \tunuse_pack(&w_curs);\n }\n \n+static void make_disjoint_pack(struct bitmapped_pack *out, struct packed_git *p)\n+{\n+\tout->p = p;\n+\tout->bitmap_pos = 0;\n+\tout->bitmap_nr = p->num_objects;\n+\tout->disjoint = 1;\n+}\n+\n void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\t\tstruct bitmapped_pack **packs_out,\n \t\t\t\t\tsize_t *packs_nr_out,\n-\t\t\t\t\tstruct bitmap **reuse_out)\n+\t\t\t\t\tstruct bitmap **reuse_out,\n+\t\t\t\t\tint multi_pack_reuse)\n {\n \tstruct repository *r = the_repository;\n \tstruct bitmapped_pack *packs = NULL;\n@@ -2088,24 +2097,62 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\tfree(packs);\n \t\t\t\treturn;\n \t\t\t}\n-\t\t\tif (!pack.bitmap_nr)\n-\t\t\t\tcontinue; /* no objects from this pack */\n-\t\t\tif (pack.bitmap_pos)\n-\t\t\t\tcontinue; /* not preferred pack */\n+\n+\t\t\tif (!pack.disjoint)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (!multi_pack_reuse && pack.bitmap_pos) {\n+\t\t\t\t/*\n+\t\t\t\t * If we're only reusing a single pack, skip\n+\t\t\t\t * over any packs which are not positioned at\n+\t\t\t\t * the beginning of the MIDX bitmap.\n+\t\t\t\t *\n+\t\t\t\t * This is consistent with the existing\n+\t\t\t\t * single-pack reuse behavior, which only reuses\n+\t\t\t\t * parts of the MIDX's preferred pack.\n+\t\t\t\t */\n+\t\t\t\tcontinue;\n+\t\t\t}\n \n \t\t\tALLOC_GROW(packs, packs_nr + 1, packs_alloc);\n \t\t\tmemcpy(&packs[packs_nr++], &pack, sizeof(pack));\n \n \t\t\tobjects_nr += pack.p->num_objects;\n+\n+\t\t\tif (!multi_pack_reuse)\n+\t\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (!packs_nr) {\n+\t\t\t/*\n+\t\t\t * Old MIDXs (i.e. those written before the \"DISP\" chunk\n+\t\t\t * existed) will not have any packs marked as disjoint.\n+\t\t\t *\n+\t\t\t * But we still want to perform pack reuse with the\n+\t\t\t * special \"preferred pack\" as before. To do this, form\n+\t\t\t * the singleton set containing just the preferred pack,\n+\t\t\t * which is trivially disjoint with itself.\n+\t\t\t *\n+\t\t\t * Moreover, the MIDX is guaranteed to resolve duplicate\n+\t\t\t * objects in favor of the copy in the preferred pack\n+\t\t\t * (if one exists). Thus, we can safely perform pack\n+\t\t\t * reuse on this pack.\n+\t\t\t */\n+\t\t\tuint32_t preferred_pack_pos;\n+\t\t\tstruct packed_git *preferred_pack;\n+\n+\t\t\tpreferred_pack_pos = midx_preferred_pack(bitmap_git);\n+\t\t\tpreferred_pack = bitmap_git->midx->packs[preferred_pack_pos];\n+\n+\t\t\tALLOC_GROW(packs, packs_nr + 1, packs_alloc);\n+\n+\t\t\tmake_disjoint_pack(&packs[packs_nr], preferred_pack);\n+\t\t\tobjects_nr = packs[packs_nr++].p->num_objects;\n \t\t}\n \t} else {\n \t\tALLOC_GROW(packs, packs_nr + 1, packs_alloc);\n \n-\t\tpacks[packs_nr].p = bitmap_git->pack;\n-\t\tpacks[packs_nr].bitmap_pos = 0;\n-\t\tpacks[packs_nr].bitmap_nr = bitmap_git->pack->num_objects;\n-\t\tpacks[packs_nr].disjoint = 1;\n-\n+\t\tmake_disjoint_pack(&packs[packs_nr], bitmap_git->pack);\n \t\tobjects_nr = packs[packs_nr++].p->num_objects;\n \t}\n \n@@ -2114,10 +2161,8 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\tword_alloc++;\n \treuse = bitmap_word_alloc(word_alloc);\n \n-\tif (packs_nr != 1)\n-\t\tBUG(\"pack reuse not yet implemented for multiple packs\");\n-\n-\treuse_partial_packfile_from_bitmap_1(bitmap_git, packs, reuse);\n+\tfor (i = 0; i < packs_nr; i++)\n+\t\treuse_partial_packfile_from_bitmap_1(bitmap_git, &packs[i], reuse);\n \n \tif (!bitmap_popcount(reuse)) {\n \t\tfree(packs);\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 901a3b86ed..8bb316ce52 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -81,7 +81,8 @@ uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git);\n void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\t\tstruct bitmapped_pack **packs_out,\n \t\t\t\t\tsize_t *packs_nr_out,\n-\t\t\t\t\tstruct bitmap **reuse_out);\n+\t\t\t\t\tstruct bitmap **reuse_out,\n+\t\t\t\t\tint multi_pack_reuse);\n int rebuild_existing_bitmaps(struct bitmap_index *, struct packing_data *mapping,\n \t\t\t kh_oid_map_t *reused_bitmaps, int show_progress);\n void free_bitmap_index(struct bitmap_index *);\ndiff --git a/t/t5332-multi-pack-reuse.sh b/t/t5332-multi-pack-reuse.sh\nnew file mode 100755\nindex 0000000000..a9bd3870e6\n--- /dev/null\n+++ b/t/t5332-multi-pack-reuse.sh\n@@ -0,0 +1,219 @@\n+#!/bin/sh\n+\n+test_description='pack-objects multi-pack reuse'\n+\n+. ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-bitmap.sh\n+. \"$TEST_DIRECTORY\"/lib-disjoint.sh\n+\n+objdir=.git/objects\n+packdir=$objdir/pack\n+\n+all_packs () {\n+\tfind $packdir -type f -name \"*.idx\" | sed -e 's/^.*\\/\\([^\\/]\\)/\\1/g'\n+}\n+\n+all_disjoint () {\n+\tall_packs | sed -e 's/^/+/g'\n+}\n+\n+test_pack_reused () {\n+\ttest_trace2_data pack-objects pack-reused \"$1\"\n+}\n+\n+test_packs_reused () {\n+\ttest_trace2_data pack-objects packs-reused \"$1\"\n+}\n+\n+\n+# pack_position objects &&\n+\tgrep \"$1\" objects | cut -d\" \" -f1\n+}\n+\n+test_expect_success 'setup' '\n+\tgit config pack.allowPackReuse multi\n+'\n+\n+test_expect_success 'preferred pack is reused without packs marked disjoint' '\n+\ttest_commit A &&\n+\ttest_commit B &&\n+\n+\tA=\"$(echo A | git pack-objects --unpacked --delta-base-offset $packdir/pack)\" &&\n+\tB=\"$(echo B | git pack-objects --unpacked --delta-base-offset $packdir/pack)\" &&\n+\n+\tgit prune-packed &&\n+\n+\tgit multi-pack-index write --bitmap &&\n+\n+\ttest_must_not_be_disjoint \"pack-$A.pack\" &&\n+\ttest_must_not_be_disjoint \"pack-$B.pack\" &&\n+\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n+\t\tgit pack-objects --stdout --revs --all >/dev/null &&\n+\n+\ttest_pack_reused 3 in <<-EOF &&\n+\tpack-$A.idx\n+\t+pack-$B.idx\n+\t+pack-$C.idx\n+\tEOF\n+\tgit multi-pack-index write --bitmap --stdin-packs trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n+\t\tgit pack-objects --stdout --revs --all >/dev/null &&\n+\n+\ttest_pack_reused 6 in &&\n+\tgit multi-pack-index write --bitmap --stdin-packs trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n+\t\tgit pack-objects --stdout --revs --all >/dev/null &&\n+\n+\ttest_pack_reused 9 in &&\n+\tgit multi-pack-index write --bitmap --preferred-pack=\"pack-$D.idx\" \\\n+\t\t--stdin-packs in <<-EOF &&\n+\t$(git rev-parse E)\n+\t^$(git rev-parse D)\n+\tEOF\n+\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n+\t\tgit pack-objects --stdout --delta-base-offset --revs /dev/null &&\n+\n+\ttest_pack_reused 3 in &&\n+\tgit multi-pack-index write --bitmap --preferred-pack=\"pack-$A.idx\" \\\n+\t\t--stdin-packs in <<-EOF &&\n+\t$(git rev-parse E)\n+\t^$(git rev-parse D)\n+\tEOF\n+\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n+\t\tgit pack-objects --stdout --delta-base-offset --revs /dev/null &&\n+\n+\ttest_pack_reused 3 f &&\n+\tgit add f &&\n+\ttest_tick &&\n+\tgit commit -m \"delta\" &&\n+\tdelta=\"$(git rev-parse HEAD)\" &&\n+\n+\ttest_seq 64 >f &&\n+\ttest_tick &&\n+\tgit commit -a -m \"base\" &&\n+\tbase=\"$(git rev-parse HEAD)\" &&\n+\n+\ttest_commit other &&\n+\n+\tgit repack -d &&\n+\n+\thave_delta \"$(git rev-parse $delta:f)\" \"$(git rev-parse $base:f)\" &&\n+\n+\tall_disjoint >in &&\n+\tgit multi-pack-index write --bitmap --stdin-packs in <<-EOF &&\n+\t$(git rev-parse other)\n+\t^$base\n+\tEOF\n+\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n+\t\tgit pack-objects --stdout --delta-base-offset --revs /dev/null &&\n+\n+\t# Even though all packs are marked disjoint, we can only reuse\n+\t# the 3 objects corresponding to \"other\" from the latest pack.\n+\t#\n+\t# This is because even though we want \"delta\", we do not want\n+\t# \"base\", meaning that we have to inflate the delta/base-pair\n+\t# corresponding to the blob in commit \"delta\", which bypasses\n+\t# the pack-reuse mechanism.\n+\t#\n+\t# The remaining objects from the other pack are similarly not\n+\t# reused because their objects are on the uninteresting side of\n+\t# the query.\n+\ttest_pack_reused 3 \n---\n t/perf/p5332-multi-pack-reuse.sh | 81 ++++++++++++++++++++++++++++++++\n 1 file changed, 81 insertions(+)\n create mode 100755 t/perf/p5332-multi-pack-reuse.sh\n\ndiff --git a/t/perf/p5332-multi-pack-reuse.sh b/t/perf/p5332-multi-pack-reuse.sh\nnew file mode 100755\nindex 0000000000..5c6c575d62\n--- /dev/null\n+++ b/t/perf/p5332-multi-pack-reuse.sh\n@@ -0,0 +1,81 @@\n+#!/bin/sh\n+\n+test_description='tests pack performance with multi-pack reuse'\n+\n+. ./perf-lib.sh\n+. \"${TEST_DIRECTORY}/perf/lib-pack.sh\"\n+\n+packdir=.git/objects/pack\n+\n+test_perf_large_repo\n+\n+find_pack () {\n+\tfor idx in $packdir/pack-*.idx\n+\tdo\n+\t\tif git show-index <$idx | grep -q \"$1\"\n+\t\tthen\n+\t\t\tbasename $idx\n+\t\tfi || return 1\n+\tdone\n+}\n+\n+repack_into_n_chunks () {\n+\tgit repack -adk &&\n+\n+\ttest \"$1\" -eq 1 && return ||\n+\n+\tfind $packdir -type f | sort >packs.before &&\n+\n+\t# partition the repository into $1 chunks of consecutive commits, and\n+\t# then create $1 packs with the objects reachable from each chunk\n+\t# (excluding any objects reachable from the previous chunks)\n+\tsz=\"$(($(git rev-list --count --all) / $1))\"\n+\tfor rev in $(git rev-list --all | awk \"NR % $sz == 0\" | tac)\n+\tdo\n+\t\tpack=\"$(echo \"$rev\" | git pack-objects --revs \\\n+\t\t\t--honor-pack-keep --delta-base-offset $packdir/pack)\" &&\n+\t\ttouch $packdir/pack-$pack.keep || return 1\n+\tdone\n+\n+\t# grab any remaining objects not packed by the previous step(s)\n+\tgit pack-objects --revs --all --honor-pack-keep --delta-base-offset \\\n+\t\t$packdir/pack &&\n+\n+\tfind $packdir -type f | sort >packs.after &&\n+\n+\t# and install the whole thing\n+\tfor f in $(comm -12 packs.before packs.after)\n+\tdo\n+\t\trm -f \"$f\" || return 1\n+\tdone\n+\trm -fr $packdir/*.keep\n+}\n+\n+for nr_packs in 1 10 100\n+do\n+\ttest_expect_success \"create $nr_packs-pack scenario\" '\n+\t\trepack_into_n_chunks $nr_packs\n+\t'\n+\n+\ttest_expect_success \"setup bitmaps for $nr_packs-pack scenario\" '\n+\t\tfind $packdir -type f -name \"*.idx\" | sed -e \"s/.*\\/\\(.*\\)$/+\\1/g\" |\n+\t\tgit multi-pack-index write --stdin-packs --bitmap \\\n+\t\t\t--preferred-pack=\"$(find_pack $(git rev-parse HEAD))\"\n+\t'\n+\n+\tfor reuse in single multi\n+\tdo\n+\t\ttest_perf \"clone for $nr_packs-pack scenario ($reuse-pack reuse)\" \"\n+\t\t\tgit for-each-ref --format='%(objectname)' refs/heads refs/tags >in &&\n+\t\t\tgit -c pack.allowPackReuse=$reuse pack-objects \\\n+\t\t\t\t--revs --delta-base-offset --use-bitmap-index \\\n+\t\t\t\t--stdout result\n+\t\t\"\n+\n+\t\ttest_size \"clone size for $nr_packs-pack scenario ($reuse-pack reuse)\" '\n+\t\t\twc -c The pack-objects internals use a packing_data struct to track what\n> objects are part of the pack(s) being formed.\n> \n> Since these structures contain allocated fields, failing to\n> appropriately free() them results in a leak. Plug that leak by\n> introducing a free_packing_data() function, and call it in the\n> appropriate spots.\n> \n> This is a fairly straightforward leak to plug, since none of the callers\n> expect to read any values or have any references to parts of the address\n> space being freed.\n> \n> Signed-off-by: Taylor Blau \n> ---\n> builtin/pack-objects.c | 1 +\n> midx.c | 5 +++++\n> pack-objects.c | 15 +++++++++++++++\n> pack-objects.h | 1 +\n> 4 files changed, 22 insertions(+)\n> \n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 89a8b5a976..bfa60359d4 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -4522,6 +4522,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> \t\t\t reuse_packfile_objects);\n> \n> cleanup:\n> +\tfree_packing_data(&to_pack);\n> \tlist_objects_filter_release(&filter_options);\n> \tstrvec_clear(&rp);\n> \n> diff --git a/midx.c b/midx.c\n> index 2f3863c936..3b727dc633 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -1592,8 +1592,13 @@ static int write_midx_internal(const char *object_dir,\n> \t\t\t\t flags) < 0) {\n> \t\t\terror(_(\"could not write multi-pack bitmap\"));\n> \t\t\tresult = 1;\n> +\t\t\tfree_packing_data(&pdata);\n> +\t\t\tfree(commits);\n> \t\t\tgoto cleanup;\n> \t\t}\n> +\n> +\t\tfree_packing_data(&pdata);\n> +\t\tfree(commits);\n> \t}\n> \t/*\n> \t * NOTE: Do not use ctx.entries beyond this point, since it might\n> diff --git a/pack-objects.c b/pack-objects.c\n> index f403ca6986..1c7bedcc94 100644\n> --- a/pack-objects.c\n> +++ b/pack-objects.c\n> @@ -151,6 +151,21 @@ void prepare_packing_data(struct repository *r, struct packing_data *pdata)\n> \tinit_recursive_mutex(&pdata->odb_lock);\n> }\n> \n> +void free_packing_data(struct packing_data *pdata)\n\nNit: shouldn't this rather be called `clear_packing_data`? `free` to me\nindicates that the data structure itself will be free'd, as well, which\nis not the case.\n\nPatrick\n\n> +{\n> +\tif (!pdata)\n> +\t\treturn;\n> +\n> +\tfree(pdata->cruft_mtime);\n> +\tfree(pdata->in_pack);\n> +\tfree(pdata->in_pack_by_idx);\n> +\tfree(pdata->in_pack_pos);\n> +\tfree(pdata->index);\n> +\tfree(pdata->layer);\n> +\tfree(pdata->objects);\n> +\tfree(pdata->tree_depth);\n> +}\n> +\n> struct object_entry *packlist_alloc(struct packing_data *pdata,\n> \t\t\t\t const struct object_id *oid)\n> {\n> diff --git a/pack-objects.h b/pack-objects.h\n> index 0d78db40cb..336217e8cd 100644\n> --- a/pack-objects.h\n> +++ b/pack-objects.h\n> @@ -169,6 +169,7 @@ struct packing_data {\n> };\n> \n> void prepare_packing_data(struct repository *r, struct packing_data *pdata);\n> +void free_packing_data(struct packing_data *pdata);\n> \n> /* Protect access to object database */\n> static inline void packing_data_lock(struct packing_data *pdata)\n> -- \n> 2.43.0.24.g980b318f98\n> \n"},{"id":"485261","messageId":"ZWhha2h2zzZWCXrw@tanuki","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"Re: [PATCH 00/24] pack-objects: multi-pack verbatim reuse","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-30T10:18:19Z","receivedAt":"2023-11-30T10:18:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 28, 2023 at 02:07:54PM -0500, Taylor Blau wrote:\n> Back in fff42755ef (pack-bitmap: add support for bitmap indexes,\n> 2013-12-21), we added support for reachability bitmaps, and taught\n> pack-objects how to reuse verbatim chunks from the bitmapped pack. When\n> multi-pack bitmaps were introduced, this pack-reuse mechanism evolved to\n> use the MIDX's \"preferred\" pack as the source for verbatim reuse.\n> \n> This allows repositories to incrementally repack themselves (e.g., using\n> a `--geometric` repack), storing the result in a MIDX, and generating a\n> corresponding bitmap. This keeps our bitmap coverage up-to-date, while\n> maintaining a relatively small number of packs.\n> \n> However, it is recommended (and matches what we do in production at\n> GitHub) that repositories repack themselves all-into-one, and\n> generate a corresponding single-pack reachability bitmap. This is done\n> for a couple of reasons, but the most relevant one to this series is\n> that it enables us to perform verbatim pack-reuse over a complete copy\n> of the repository, since the entire repository resides in a single pack\n> (and thus is eligible for verbatim pack-reuse).\n> \n> As repositories grow larger, packing their contents into a single pack\n> becomes less feasible. This series extends the pack-reuse mechanism to\n> operate over multiple packs which are known ahead of time to be disjoint\n> with respect to one another's set of objects.\n> \n> The implementation has a few components:\n> \n> - A new MIDX chunk called \"Disjoint packfiles\" or DISP is introduced\n> to keep track of the bitmap position, number of objects, and\n> disjointed-ness for each pack contained in the MIDX.\n> \n> - A new mode for `git multi-pack-index write --stdin-packs` that\n> allows specifying disjoint packs, as well as a new option\n> `--retain-disjoint` which preserves the set of existing disjoint\n> packs in the new MIDX.\n> \n> - A new pack-objects mode `--ignore-disjoint`, which produces packs\n> which are disjoint with respect to the current set of disjoint packs\n> (i.e. it discards any objects from the packing list which appear in\n> any of the known-disjoint packs).\n> \n> - A new repack mode, `--extend-disjoint` which causes any new pack(s)\n> which are generated to be disjoint with respect to the set of packs\n> currently marked as disjoint, minus any pack(s) which are about to\n> be deleted.\n> \n> With all of that in place, the patch series then rewrites all of the\n> pack-reuse functions in terms of the new `bitmapped_pack` structure.\n> Once we have dropped all of the assumptions stemming from only\n> performing pack-reuse over a single candidate pack, we can then enable\n> reuse over all of the disjoint packs.\n> \n> In addition to the many new tests in t5332 added by that series, I tried\n> to simulate a \"real world\" test on git.git by breaking the repository\n> into chunks of 1,000 commits (plus their set of reachable objects not\n> reachable from earlier chunk(s)) and packing those chunks. This produces\n> a large number of packs with the objects from git.git which are known to\n> be disjoint with respect to one another.\n> \n> $ git clone git@github.com:git/git.git base\n> \n> $ cd base\n> $ mv .git/objects/pack/pack-*.idx{,.bak}\n> $ git unpack-objects <.git/objects/pack/pack-*.pack\n> \n> # pack the objects from each successive block of 1k commits\n> $ for rev in $(git rev-list --all | awk '(NR) % 1000 == 0' | tac)\n> do\n> echo $rev |\n> git.compile pack-objects --revs --unpacked .git/objects/pack/pack || return 1\n> done\n> # and grab any stragglers, pruning the unpacked objects\n> $ git repack -d\n> I then constructed a MIDX and corresponding bitmap\n> \n> $ find_pack () {\n> for idx in .git/objects/pack/pack-*.idx\n> do\n> git show-index <$idx | grep -q \"$1\" && basename $idx\n> done\n> }\n> $ preferred=\"$(find_pack $(git rev-parse HEAD))\"\n> \n> $ ( cd .git/objects/pack && ls -1 *.idx ) | sed -e 's/^/+/g' |\n> git.compile multi-pack-index write --bitmap --stdin-packs \\\n> --preferred-pack=$preferred\n> $ git for-each-ref --format='%(objectname)' refs/heads refs/tags >in\n> \n> With all of that in place, I was able to produce a significant speed-up\n> by reusing objects from multiple packs:\n> \n> $ hyperfine -L v single,multi -n '{v}-pack reuse' 'git.compile -c pack.allowPackReuse={v} pack-objects --revs --stdout --use-bitmap-index --delta-base-offset /dev/null'\n> Benchmark 1: single-pack reuse\n> Time (mean ± σ): 6.094 s ± 0.023 s [User: 43.723 s, System: 0.358 s]\n> Range (min … max): 6.063 s … 6.126 s 10 runs\n> \n> Benchmark 2: multi-pack reuse\n> Time (mean ± σ): 906.5 ms ± 3.2 ms [User: 1081.5 ms, System: 30.9 ms]\n> Range (min … max): 903.5 ms … 912.7 ms 10 runs\n> \n> Summary\n> multi-pack reuse ran\n> 6.72 ± 0.03 times faster than single-pack reuse\n> \n> (There are corresponding tests in p5332 that test different sized chunks\n> and measure the runtime performance as well as resulting pack size).\n> \n> Performing verbatim pack reuse naturally trades off between CPU time and\n> the resulting pack size. In the above example, the single-pack reuse\n> case produces a clone size of ~194 MB on my machine, while the\n> multi-pack reuse case produces a clone size closer to ~266 MB, which is\n> a ~37% increase in clone size.\n\nQuite exciting, and a tradeoff that may be worth it for Git hosters. I\nexpect that this is going to be an extreme example of the benefits\nprovided by your patch series -- do you by any chance also have \"real\"\nnumbers that make it possible to quantify the effect a bit better?\n\nNo worry if you don't, I'm just curious.\n\n> I think there is still some opportunity to close this gap, since the\n> \"packing\" strategy here is extremely naive. In a production setting, I'm\n> sure that there are more well thought out repacking strategies that\n> would produce more similar clone sizes.\n> \n> I considered breaking this series up into smaller chunks, but was\n> unsatisfied with the result. Since this series is rather large, if you\n> have alternate suggestions on better ways to structure this, please let\n> me know.\n\nThe series is indeed very involved to review. I only made it up to patch\n8/24 and already spent quite some time on it. So I'd certainly welcome\nit if this was split up into smaller parts, but don't have a suggestion\nas to how this should be done (also because I didn't yet read the other\n16 patches).\n\nI'll review the remaining patches at a later point in time.\n\nPatrick\n\n> Thanks in advance for your review!\n> \n> Taylor Blau (24):\n> pack-objects: free packing_data in more places\n> pack-bitmap-write: deep-clear the `bb_commit` slab\n> pack-bitmap: plug leak in find_objects()\n> midx: factor out `fill_pack_info()`\n> midx: implement `DISP` chunk\n> midx: implement `midx_locate_pack()`\n> midx: implement `--retain-disjoint` mode\n> pack-objects: implement `--ignore-disjoint` mode\n> repack: implement `--extend-disjoint` mode\n> pack-bitmap: pass `bitmapped_pack` struct to pack-reuse functions\n> pack-bitmap: simplify `reuse_partial_packfile_from_bitmap()` signature\n> pack-bitmap: return multiple packs via\n> `reuse_partial_packfile_from_bitmap()`\n> pack-objects: parameterize pack-reuse routines over a single pack\n> pack-objects: keep track of `pack_start` for each reuse pack\n> pack-objects: pass `bitmapped_pack`'s to pack-reuse functions\n> pack-objects: prepare `write_reused_pack()` for multi-pack reuse\n> pack-objects: prepare `write_reused_pack_verbatim()` for multi-pack\n> reuse\n> pack-objects: include number of packs reused in output\n> pack-bitmap: prepare to mark objects from multiple packs for reuse\n> pack-objects: add tracing for various packfile metrics\n> t/test-lib-functions.sh: implement `test_trace2_data` helper\n> pack-objects: allow setting `pack.allowPackReuse` to \"single\"\n> pack-bitmap: reuse objects from all disjoint packs\n> t/perf: add performance tests for multi-pack reuse\n> \n> Documentation/config/pack.txt | 8 +-\n> Documentation/git-multi-pack-index.txt | 12 ++\n> Documentation/git-pack-objects.txt | 8 +\n> Documentation/git-repack.txt | 12 ++\n> Documentation/gitformat-pack.txt | 109 ++++++++++\n> builtin/multi-pack-index.c | 13 +-\n> builtin/pack-objects.c | 200 +++++++++++++++----\n> builtin/repack.c | 57 +++++-\n> midx.c | 218 +++++++++++++++++---\n> midx.h | 11 +-\n> pack-bitmap-write.c | 9 +-\n> pack-bitmap.c | 265 ++++++++++++++++++++-----\n> pack-bitmap.h | 18 +-\n> pack-objects.c | 15 ++\n> pack-objects.h | 1 +\n> t/helper/test-read-midx.c | 31 ++-\n> t/lib-disjoint.sh | 49 +++++\n> t/perf/p5332-multi-pack-reuse.sh | 81 ++++++++\n> t/t5319-multi-pack-index.sh | 140 +++++++++++++\n> t/t5331-pack-objects-stdin.sh | 156 +++++++++++++++\n> t/t5332-multi-pack-reuse.sh | 219 ++++++++++++++++++++\n> t/t6113-rev-list-bitmap-filters.sh | 2 +\n> t/t7700-repack.sh | 4 +-\n> t/t7705-repack-extend-disjoint.sh | 142 +++++++++++++\n> t/test-lib-functions.sh | 14 ++\n> 25 files changed, 1650 insertions(+), 144 deletions(-)\n> create mode 100644 t/lib-disjoint.sh\n> create mode 100755 t/perf/p5332-multi-pack-reuse.sh\n> create mode 100755 t/t5332-multi-pack-reuse.sh\n> create mode 100755 t/t7705-repack-extend-disjoint.sh\n> \n> \n> base-commit: 564d0252ca632e0264ed670534a51d18a689ef5d\n> -- \n> 2.43.0.24.g980b318f98\n"},{"id":"485262","messageId":"ZWhhd5gLzYwPEgBl@tanuki","threadId":"60565","inReplyTo":"6f5ff96998946f3f49da56fd05c096b949521339.1701198172.git.me@ttaylorr.com","subject":"Re: [PATCH 02/24] pack-bitmap-write: deep-clear the `bb_commit` slab","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-30T10:18:31Z","receivedAt":"2023-11-30T10:18:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 28, 2023 at 02:07:59PM -0500, Taylor Blau wrote:\n> The `bb_commit` commit slab is used by the pack-bitmap-write machinery\n> to track various pieces of bookkeeping used to generate reachability\n> bitmaps.\n> \n> Even though we clear the slab when freeing the bitmap_builder struct\n> (with `bitmap_builder_clear()`), there are still pointers which point to\n> locations in memory that have not yet been freed, resulting in a leak.\n> \n> Plug the leak by introducing a suitable `free_fn` for the `struct\n> bb_commit` type, and make sure it is called on each member of the slab\n> via the `deep_clear_bb_data()` function.\n> \n> Signed-off-by: Taylor Blau \n> ---\n> pack-bitmap-write.c | 9 ++++++++-\n> 1 file changed, 8 insertions(+), 1 deletion(-)\n> \n> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index f4ecdf8b0e..dd3a415b9d 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -198,6 +198,13 @@ struct bb_commit {\n> \tunsigned idx; /* within selected array */\n> };\n> \n> +static void clear_bb_commit(struct bb_commit *commit)\n> +{\n> +\tfree(commit->reverse_edges);\n\nI'd have expected to see `free_commit_list()` here instead of a simple\nfree. Is there any reason why we don't use it?\n\nPatrick\n\n> +\tbitmap_free(commit->commit_mask);\n> +\tbitmap_free(commit->bitmap);\n> +}\n> +\n> define_commit_slab(bb_data, struct bb_commit);\n> \n> struct bitmap_builder {\n> @@ -339,7 +346,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n> \n> static void bitmap_builder_clear(struct bitmap_builder *bb)\n> {\n> -\tclear_bb_data(&bb->data);\n> +\tdeep_clear_bb_data(&bb->data, clear_bb_commit);\n> \tfree(bb->commits);\n> \tbb->commits_nr = bb->commits_alloc = 0;\n> }\n> -- \n> 2.43.0.24.g980b318f98\n> \n"},{"id":"485263","messageId":"ZWhhfXoovgYzIYE0@tanuki","threadId":"60565","inReplyTo":"ccf1337305db60f1c8174e9b309e2a9e04ce1487.1701198172.git.me@ttaylorr.com","subject":"Re: [PATCH 04/24] midx: factor out `fill_pack_info()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-30T10:18:37Z","receivedAt":"2023-11-30T10:18:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 28, 2023 at 02:08:05PM -0500, Taylor Blau wrote:\n> When selecting which packfiles will be written while generating a MIDX,\n> the MIDX internals fill out a 'struct pack_info' with various pieces of\n> book-keeping.\n> \n> Instead of filling out each field of the `pack_info` structure\n> individually in each of the two spots that modify the array of such\n> structures (`ctx->info`), extract a common routine that does this for\n> us.\n> \n> This reduces the code duplication by a modest amount. But more\n> importantly, it zero-initializes the structure before assigning values\n> into it. This hardens us for a future change which will add additional\n> fields to this structure which (until this patch) was not\n> zero-initialized.\n> \n> As a result, any new fields added to the `pack_info` structure need only\n> be updated in a single location, instead of at each spot within midx.c.\n> \n> There are no functional changes in this patch.\n> \n> Signed-off-by: Taylor Blau \n> ---\n> midx.c | 35 +++++++++++++++++++----------------\n> 1 file changed, 19 insertions(+), 16 deletions(-)\n> \n> diff --git a/midx.c b/midx.c\n> index 3b727dc633..591b3c636e 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -464,6 +464,17 @@ struct pack_info {\n> \tunsigned expired : 1;\n> };\n> \n> +static void fill_pack_info(struct pack_info *info,\n> +\t\t\t struct packed_git *p, char *pack_name,\n> +\t\t\t uint32_t orig_pack_int_id)\n> +{\n> +\tmemset(info, 0, sizeof(struct pack_info));\n> +\n> +\tinfo->orig_pack_int_id = orig_pack_int_id;\n> +\tinfo->pack_name = pack_name;\n> +\tinfo->p = p;\n> +}\n\nNit: all callers manually call `xstrdup(pack_name)` and pass that to\n`fill_pack_info()`. We could consider doing this in here instead so that\nownership of the string becomes a tad clearer.\n\n> static int pack_info_compare(const void *_a, const void *_b)\n> {\n> \tstruct pack_info *a = (struct pack_info *)_a;\n> @@ -504,6 +515,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n> \t\t\t const char *file_name, void *data)\n> {\n> \tstruct write_midx_context *ctx = data;\n> +\tstruct packed_git *p;\n> \n> \tif (ends_with(file_name, \".idx\")) {\n> \t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n> @@ -530,17 +542,14 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n> \n> \t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n> \n> -\t\tctx->info[ctx->nr].p = add_packed_git(full_path,\n> -\t\t\t\t\t\t full_path_len,\n> -\t\t\t\t\t\t 0);\n> -\n> -\t\tif (!ctx->info[ctx->nr].p) {\n> +\t\tp = add_packed_git(full_path, full_path_len, 0);\n> +\t\tif (!p) {\n> \t\t\twarning(_(\"failed to add packfile '%s'\"),\n> \t\t\t\tfull_path);\n> \t\t\treturn;\n> \t\t}\n> \n> -\t\tif (open_pack_index(ctx->info[ctx->nr].p)) {\n> +\t\tif (open_pack_index(p)) {\n> \t\t\twarning(_(\"failed to open pack-index '%s'\"),\n> \t\t\t\tfull_path);\n> \t\t\tclose_pack(ctx->info[ctx->nr].p);\n\nIsn't `ctx->info[ctx->nr].p` still uninitialized at this point?\n\n> @@ -548,9 +557,8 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n> \t\t\treturn;\n> \t\t}\n> \n> -\t\tctx->info[ctx->nr].pack_name = xstrdup(file_name);\n> -\t\tctx->info[ctx->nr].orig_pack_int_id = ctx->nr;\n> -\t\tctx->info[ctx->nr].expired = 0;\n> +\t\tfill_pack_info(&ctx->info[ctx->nr], p, xstrdup(file_name),\n> +\t\t\t ctx->nr);\n> \t\tctx->nr++;\n> \t}\n> }\n> @@ -1310,11 +1318,6 @@ static int write_midx_internal(const char *object_dir,\n> \t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n> \t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n> \n> -\t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n> -\t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n> -\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n> -\t\t\tctx.info[ctx.nr].expired = 0;\n> -\n> \t\t\tif (flags & MIDX_WRITE_REV_INDEX) {\n> \t\t\t\t/*\n> \t\t\t\t * If generating a reverse index, need to have\n> @@ -1330,10 +1333,10 @@ static int write_midx_internal(const char *object_dir,\n> \t\t\t\tif (open_pack_index(ctx.m->packs[i]))\n> \t\t\t\t\tdie(_(\"could not open index for %s\"),\n> \t\t\t\t\t ctx.m->packs[i]->pack_name);\n> -\t\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n\nJust to make sure I'm not missing anything, but this assignment here was\nbasically redundant before this patch already, right?\n\nPatrick\n\n> \t\t\t}\n> \n> -\t\t\tctx.nr++;\n> +\t\t\tfill_pack_info(&ctx.info[ctx.nr++], ctx.m->packs[i],\n> +\t\t\t\t xstrdup(ctx.m->pack_names[i]), i);\n> \t\t}\n> \t}\n> \n> -- \n> 2.43.0.24.g980b318f98\n> \n"},{"id":"485264","messageId":"ZWhhhaXiF_wC3er7@tanuki","threadId":"60565","inReplyTo":"c52d7e7b27a9add4f58b8334db4fe4498af1c90f.1701198172.git.me@ttaylorr.com","subject":"Re: [PATCH 05/24] midx: implement `DISP` chunk","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-30T10:18:45Z","receivedAt":"2023-11-30T10:18:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 28, 2023 at 02:08:08PM -0500, Taylor Blau wrote:\n> When a multi-pack bitmap is used to implement verbatim pack reuse (that\n> is, when verbatim chunks from an on-disk packfile are copied\n> directly[^1]), it does so by using its \"preferred pack\" as the source\n> for pack-reuse.\n> \n> This allows repositories to pack the majority of their objects into a\n> single (often large) pack, and then use it as the single source for\n> verbatim pack reuse. This increases the amount of objects that are\n> reused verbatim (and consequently, decrease the amount of time it takes\n> to generate many packs). But this performance comes at a cost, which is\n> that the preferred packfile must pace its growth with that of the entire\n> repository in order to maintain the utility of verbatim pack reuse.\n> \n> As repositories grow beyond what we can reasonably store in a single\n> packfile, the utility of verbatim pack reuse diminishes. Or, at the very\n> least, it becomes increasingly more expensive to maintain as the pack\n> grows larger and larger.\n> \n> It would be beneficial to be able to perform this same optimization over\n> multiple packs, provided some modest constraints (most importantly, that\n> the set of packs eligible for verbatim reuse are disjoint with respect\n> to the objects that they contain).\n> \n> If we assume that the packs which we treat as candidates for verbatim\n> reuse are disjoint with respect to their objects, we need to make only\n> modest modifications to the verbatim pack-reuse code itself. Most\n> notably, we need to remove the assumption that the bits in the\n> reachability bitmap corresponding to objects from the single reuse pack\n> begin at the first bit position.\n> \n> Future patches will unwind these assumptions and reimplement their\n> existing functionality as special cases of the more general assumptions\n> (e.g. that reuse bits can start anywhere within the bitset, but happen\n> to start at 0 for all existing cases).\n> \n> This patch does not yet relax any of those assumptions. Instead, it\n> implements a foundational data-structure, the \"Disjoint Packs\" (`DISP`)\n> chunk of the multi-pack index. The `DISP` chunk's contents are described\n> in detail here. Importantly, the `DISP` chunk contains information to\n> map regions of a multi-pack index's reachability bitmap to the packs\n> whose objects they represent.\n> \n> For now, this chunk is only written, not read (outside of the test-tool\n> used in this patch to test the new chunk's behavior). Future patches\n> will begin to make use of this new chunk.\n> \n> This patch implements reading (though no callers outside of the above\n> one perform any reading) and writing this new chunk. It also extends the\n> `--stdin-packs` format used by the `git multi-pack-index write` builtin\n> to be able to designate that a given pack is to be marked as \"disjoint\"\n> by prefixing it with a '+' character.\n> \n> [^1]: Modulo patching any `OFS_DELTA`'s that cross over a region of the\n> pack that wasn't used verbatim.\n> \n> Signed-off-by: Taylor Blau \n> ---\n> Documentation/git-multi-pack-index.txt | 4 +\n> Documentation/gitformat-pack.txt | 109 +++++++++++++++++++++++\n> builtin/multi-pack-index.c | 10 ++-\n> midx.c | 116 ++++++++++++++++++++++---\n> midx.h | 5 ++\n> pack-bitmap.h | 9 ++\n> t/helper/test-read-midx.c | 31 ++++++-\n> t/t5319-multi-pack-index.sh | 58 +++++++++++++\n> 8 files changed, 325 insertions(+), 17 deletions(-)\n> \n> diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> index 3696506eb3..d130e65b28 100644\n> --- a/Documentation/git-multi-pack-index.txt\n> +++ b/Documentation/git-multi-pack-index.txt\n> @@ -49,6 +49,10 @@ write::\n> \t--stdin-packs::\n> \t\tWrite a multi-pack index containing only the set of\n> \t\tline-delimited pack index basenames provided over stdin.\n> +\t\tLines beginning with a '+' character (followed by the\n> +\t\tpack index basename as before) have their pack marked as\n> +\t\t\"disjoint\". See the \"`DISP` chunk and disjoint packs\"\n> +\t\tsection in linkgit:gitformat-pack[5] for more.\n> \n> \t--refs-snapshot=::\n> \t\tWith `--bitmap`, optionally specify a file which\n> diff --git a/Documentation/gitformat-pack.txt b/Documentation/gitformat-pack.txt\n> index 9fcb29a9c8..658682ddd5 100644\n> --- a/Documentation/gitformat-pack.txt\n> +++ b/Documentation/gitformat-pack.txt\n> @@ -396,6 +396,22 @@ CHUNK DATA:\n> \t is padded at the end with between 0 and 3 NUL bytes to make the\n> \t chunk size a multiple of 4 bytes.\n> \n> +\tDisjoint Packfiles (ID: {'D', 'I', 'S', 'P'})\n> +\t Stores a table of three 4-byte unsigned integers in network order.\n> +\t Each table entry corresponds to a single pack (in the order that\n> +\t they appear above in the `PNAM` chunk). The values for each table\n> +\t entry are as follows:\n> +\t - The first bit position (in psuedo-pack order, see below) to\n\ns/psuedo/pseudo/\n\n> +\t contain an object from that pack.\n> +\t - The number of bits whose objects are selected from that pack.\n> +\t - A \"meta\" value, whose least-significant bit indicates whether or\n> +\t not the pack is disjoint with respect to other packs. The\n> +\t remaining bits are unused.\n> +\t Two packs are \"disjoint\" with respect to one another when they have\n> +\t disjoint sets of objects. In other words, any object found in a pack\n> +\t contained in the set of disjoint packfiles is guaranteed to be\n> +\t uniquely located among those packs.\n> +\n> \tOID Fanout (ID: {'O', 'I', 'D', 'F'})\n> \t The ith entry, F[i], stores the number of OIDs with first\n> \t byte at most i. Thus F[255] stores the total\n> @@ -509,6 +525,99 @@ packs arranged in MIDX order (with the preferred pack coming first).\n> The MIDX's reverse index is stored in the optional 'RIDX' chunk within\n> the MIDX itself.\n> \n> +=== `DISP` chunk and disjoint packs\n> +\n> +The Disjoint Packfiles (`DISP`) chunk encodes additional information\n> +about the objects in the multi-pack index's reachability bitmap. Recall\n> +that objects from the MIDX are arranged in \"pseudo-pack\" order (see:\n\nThe colon feels a bit out-of-place here, so: s/see:/see/\n\n> +above) for reachability bitmaps.\n> +\n> +From the example above, suppose we have packs \"a\", \"b\", and \"c\", with\n> +10, 15, and 20 objects, respectively. In pseudo-pack order, those would\n> +be arranged as follows:\n> +\n> + |a,0|a,1|...|a,9|b,0|b,1|...|b,14|c,0|c,1|...|c,19|\n> +\n> +When working with single-pack bitmaps (or, equivalently, multi-pack\n> +reachability bitmaps without any packs marked as disjoint),\n> +linkgit:git-pack-objects[1] performs ``verbatim'' reuse, attempting to\n> +reuse chunks of the existing packfile instead of adding objects to the\n> +packing list.\n\nI'm not sure I full understand this paragraph. In the context of a\nsingle pack bitmap it's clear enough. But I stumbled over the MIDX case,\nbecause here we potentially have multiple packfiles, so it's not exactly\nclear to me what you refer to with \"the existing packfile\" in that case.\nI'd think that we perform verbatim reuse of the preferred packfile,\nright? If so, we might want to make that a bit more explicit.\n\n> +When a chunk of bytes are reused from an existing pack, any objects\n\ns/are/is/, as it refers to the single chunk and not the plural bytes.\n\n> +contained therein do not need to be added to the packing list, saving\n> +memory and CPU time. But a chunk from an existing packfile can only be\n> +reused when the following conditions are met:\n> +\n> + - The chunk contains only objects which were requested by the caller\n> + (i.e. does not contain any objects which the caller didn't ask for\n> + explicitly or implicitly).\n> +\n> + - All objects stored as offset- or reference-deltas also include their\n> + base object in the resulting pack.\n> +\n> +Additionally, packfiles many not contain more than one copy of any given\n\ns/many/may\n\n> +object. This introduces an additional constraint over the set of packs\n> +we may want to reuse. The most straightforward approach is to mandate\n> +that the set of packs is disjoint with respect to the set of objects\n> +contained in each pack. In other words, for each object `o` in the union\n> +of all objects stored by the disjoint set of packs, `o` is contained in\n> +exactly one pack from the disjoint set.\n\nIs this a property that usually holds for our normal housekeeping, or\ndoes it require careful managing by the user/admin? How about geometric\nrepacking?\n\n> +One alternative design choice for multi-pack reuse might instead involve\n> +imposing a chunk-level constraint that allows packs in the reusable set\n> +to contain multiple copies across different packs, but restricts each\n> +chunk against including more than one copy of such an object. This is in\n> +theory possible to implement, but significantly more complicated than\n> +forcing packs themselves to be disjoint. Most notably, we would have to\n> +keep track of which objects have already been sent during verbatim\n> +pack-reuse, defeating the main purpose of verbatim pack reuse (that we\n> +don't have to keep track of individual objects).\n> +\n> +The `DISP` chunk encodes the necessary information in order to implement\n> +multi-pack reuse over a disjoint set of packs as described above.\n> +Specifically, the `DISP` chunk encodes three pieces of information (all\n> +32-bit unsigned integers in network byte-order) for each packfile `p`\n> +that is stored in the MIDX, as follows:\n> +\n> +`bitmap_pos`:: The first bit position (in pseudo-pack order) in the\n> + multi-pack index's reachability bitmap occupied by an object from `p`.\n> +\n> +`bitmap_nr`:: The number of bit positions (including the one at\n> + `bitmap_pos`) that encode objects from that pack `p`.\n> +\n> +`disjoint`:: Metadata, including whether or not the pack `p` is\n> + ``disjoint''. The least significant bit stores whether or not the pack\n> + is disjoint. The remaining bits are reserved for future use.\n> +\n> +For example, the `DISP` chunk corresponding to the above example (with\n> +packs ``a'', ``b'', and ``c'') would look like:\n> +\n> +[cols=\"1,2,2,2\"]\n> +|===\n> +| |`bitmap_pos` |`bitmap_nr` |`disjoint`\n> +\n> +|packfile ``a''\n> +|`0`\n> +|`10`\n> +|`0x1`\n> +\n> +|packfile ``b''\n> +|`10`\n> +|`15`\n> +|`0x1`\n> +\n> +|packfile ``c''\n> +|`25`\n> +|`20`\n> +|`0x1`\n> +|===\n> +\n> +With these constraints and information in place, we can treat each\n> +packfile marked as disjoint as individually reusable in the same fashion\n> +as verbatim pack reuse is performed on individual packs prior to the\n> +implementation of the `DISP` chunk.\n> +\n> == cruft packs\n> \n> The cruft packs feature offer an alternative to Git's traditional mechanism of\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> index a72aebecaa..0f1dd4651d 100644\n> --- a/builtin/multi-pack-index.c\n> +++ b/builtin/multi-pack-index.c\n> @@ -106,11 +106,17 @@ static int git_multi_pack_index_write_config(const char *var, const char *value,\n> \treturn 0;\n> }\n> \n> +#define DISJOINT ((void*)(uintptr_t)1)\n> +\n> static void read_packs_from_stdin(struct string_list *to)\n> {\n> \tstruct strbuf buf = STRBUF_INIT;\n> -\twhile (strbuf_getline(&buf, stdin) != EOF)\n> -\t\tstring_list_append(to, buf.buf);\n> +\twhile (strbuf_getline(&buf, stdin) != EOF) {\n> +\t\tif (*buf.buf == '+')\n> +\t\t\tstring_list_append(to, buf.buf + 1)->util = DISJOINT;\n> +\t\telse\n> +\t\t\tstring_list_append(to, buf.buf);\n> +\t}\n> \tstring_list_sort(to);\n> \n> \tstrbuf_release(&buf);\n> diff --git a/midx.c b/midx.c\n> index 591b3c636e..f55020072f 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -33,6 +33,7 @@\n> \n> #define MIDX_CHUNK_ALIGNMENT 4\n> #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n> +#define MIDX_CHUNKID_DISJOINTPACKS 0x44495350 /* \"DISP\" */\n> #define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n> #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n> #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n> @@ -182,6 +183,9 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n> \n> \tpair_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS, &m->chunk_large_offsets,\n> \t\t &m->chunk_large_offsets_len);\n> +\tpair_chunk(cf, MIDX_CHUNKID_DISJOINTPACKS,\n> +\t\t (const unsigned char **)&m->chunk_disjoint_packs,\n> +\t\t &m->chunk_disjoint_packs_len);\n> \n> \tif (git_env_bool(\"GIT_TEST_MIDX_READ_RIDX\", 1))\n> \t\tpair_chunk(cf, MIDX_CHUNKID_REVINDEX, &m->chunk_revindex,\n> @@ -275,6 +279,23 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t\n> \treturn 0;\n> }\n> \n> +int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n> +\t\t struct bitmapped_pack *bp, uint32_t pack_int_id)\n> +{\n> +\tif (!m->chunk_disjoint_packs)\n> +\t\treturn error(_(\"MIDX does not contain the DISP chunk\"));\n> +\n> +\tif (prepare_midx_pack(r, m, pack_int_id))\n> +\t\treturn error(_(\"could not load disjoint pack %\"PRIu32), pack_int_id);\n> +\n> +\tbp->p = m->packs[pack_int_id];\n> +\tbp->bitmap_pos = get_be32(m->chunk_disjoint_packs + 3 * pack_int_id);\n> +\tbp->bitmap_nr = get_be32(m->chunk_disjoint_packs + 3 * pack_int_id + 1);\n> +\tbp->disjoint = !!get_be32(m->chunk_disjoint_packs + 3 * pack_int_id + 2);\n> +\n> +\treturn 0;\n> +}\n> +\n> int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result)\n> {\n> \treturn bsearch_hash(oid->hash, m->chunk_oid_fanout, m->chunk_oid_lookup,\n> @@ -457,11 +478,18 @@ static size_t write_midx_header(struct hashfile *f,\n> \treturn MIDX_HEADER_SIZE;\n> }\n> \n> +#define BITMAP_POS_UNKNOWN (~((uint32_t)0))\n> +\n> struct pack_info {\n> \tuint32_t orig_pack_int_id;\n> \tchar *pack_name;\n> \tstruct packed_git *p;\n> -\tunsigned expired : 1;\n> +\n> +\tuint32_t bitmap_pos;\n> +\tuint32_t bitmap_nr;\n> +\n> +\tunsigned expired : 1,\n> +\t\t disjoint : 1;\n> };\n> \n> static void fill_pack_info(struct pack_info *info,\n> @@ -473,6 +501,7 @@ static void fill_pack_info(struct pack_info *info,\n> \tinfo->orig_pack_int_id = orig_pack_int_id;\n> \tinfo->pack_name = pack_name;\n> \tinfo->p = p;\n> +\tinfo->bitmap_pos = BITMAP_POS_UNKNOWN;\n> }\n> \n> static int pack_info_compare(const void *_a, const void *_b)\n> @@ -516,6 +545,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n> {\n> \tstruct write_midx_context *ctx = data;\n> \tstruct packed_git *p;\n> +\tstruct string_list_item *item = NULL;\n> \n> \tif (ends_with(file_name, \".idx\")) {\n> \t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n> @@ -534,11 +564,13 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n> \t\t * should be performed independently (likely checking\n> \t\t * to_include before the existing MIDX).\n> \t\t */\n> -\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n> -\t\t\treturn;\n> -\t\telse if (ctx->to_include &&\n> -\t\t\t !string_list_has_string(ctx->to_include, file_name))\n> +\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name)) {\n> \t\t\treturn;\n> +\t\t} else if (ctx->to_include) {\n> +\t\t\titem = string_list_lookup(ctx->to_include, file_name);\n> +\t\t\tif (!item)\n> +\t\t\t\treturn;\n> +\t\t}\n> \n> \t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n> \n> @@ -559,6 +591,8 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n> \n> \t\tfill_pack_info(&ctx->info[ctx->nr], p, xstrdup(file_name),\n> \t\t\t ctx->nr);\n> +\t\tif (item)\n> +\t\t\tctx->info[ctx->nr].disjoint = !!item->util;\n> \t\tctx->nr++;\n> \t}\n> }\n> @@ -568,7 +602,8 @@ struct pack_midx_entry {\n> \tuint32_t pack_int_id;\n> \ttime_t pack_mtime;\n> \tuint64_t offset;\n> -\tunsigned preferred : 1;\n> +\tunsigned preferred : 1,\n> +\t\t disjoint : 1;\n> };\n> \n> static int midx_oid_compare(const void *_a, const void *_b)\n> @@ -586,6 +621,12 @@ static int midx_oid_compare(const void *_a, const void *_b)\n> \tif (a->preferred < b->preferred)\n> \t\treturn 1;\n> \n> +\t/* Sort objects in a disjoint pack last when multiple copies exist. */\n> +\tif (a->disjoint < b->disjoint)\n> +\t\treturn -1;\n> +\tif (a->disjoint > b->disjoint)\n> +\t\treturn 1;\n> +\n> \tif (a->pack_mtime > b->pack_mtime)\n> \t\treturn -1;\n> \telse if (a->pack_mtime < b->pack_mtime)\n> @@ -671,6 +712,7 @@ static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n> \t\t\t\t\t &fanout->entries[fanout->nr],\n> \t\t\t\t\t cur_object);\n> \t\tfanout->entries[fanout->nr].preferred = 0;\n> +\t\tfanout->entries[fanout->nr].disjoint = 0;\n> \t\tfanout->nr++;\n> \t}\n> }\n> @@ -696,6 +738,7 @@ static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout,\n> \t\t\t\tcur_object,\n> \t\t\t\t&fanout->entries[fanout->nr],\n> \t\t\t\tpreferred);\n> +\t\tfanout->entries[fanout->nr].disjoint = info->disjoint;\n> \t\tfanout->nr++;\n> \t}\n> }\n> @@ -764,14 +807,22 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n> \t\t * Take only the first duplicate.\n> \t\t */\n> \t\tfor (cur_object = 0; cur_object < fanout.nr; cur_object++) {\n> -\t\t\tif (cur_object && oideq(&fanout.entries[cur_object - 1].oid,\n> -\t\t\t\t\t\t&fanout.entries[cur_object].oid))\n> -\t\t\t\tcontinue;\n> +\t\t\tstruct pack_midx_entry *ours = &fanout.entries[cur_object];\n> +\t\t\tif (cur_object) {\n> +\t\t\t\tstruct pack_midx_entry *prev = &fanout.entries[cur_object - 1];\n> +\t\t\t\tif (oideq(&prev->oid, &ours->oid)) {\n> +\t\t\t\t\tif (prev->disjoint && ours->disjoint)\n> +\t\t\t\t\t\tdie(_(\"duplicate object '%s' among disjoint packs '%s', '%s'\"),\n> +\t\t\t\t\t\t oid_to_hex(&prev->oid),\n> +\t\t\t\t\t\t info[prev->pack_int_id].pack_name,\n> +\t\t\t\t\t\t info[ours->pack_int_id].pack_name);\n\nShouldn't we die if `prev->disjoint || ours->disjoint` instead of `&&`?\nEven if one of the packs isn't marked as disjoint, it's still wrong if\nthe other one is and one of its objects exists in multiple packs.\n\nOr am I misunderstanding, and we only guarantee the disjoint property\nacross packfiles that are actually marked as such?\n\nPatrick\n\n> +\t\t\t\t\tcontinue;\n> +\t\t\t\t}\n> +\t\t\t}\n> \n> \t\t\tALLOC_GROW(deduplicated_entries, st_add(*nr_objects, 1),\n> \t\t\t\t alloc_objects);\n> -\t\t\tmemcpy(&deduplicated_entries[*nr_objects],\n> -\t\t\t &fanout.entries[cur_object],\n> +\t\t\tmemcpy(&deduplicated_entries[*nr_objects], ours,\n> \t\t\t sizeof(struct pack_midx_entry));\n> \t\t\t(*nr_objects)++;\n> \t\t}\n> @@ -814,6 +865,27 @@ static int write_midx_pack_names(struct hashfile *f, void *data)\n> \treturn 0;\n> }\n> \n> +static int write_midx_disjoint_packs(struct hashfile *f, void *data)\n> +{\n> +\tstruct write_midx_context *ctx = data;\n> +\tsize_t i;\n> +\n> +\tfor (i = 0; i < ctx->nr; i++) {\n> +\t\tstruct pack_info *pack = &ctx->info[i];\n> +\t\tif (pack->expired)\n> +\t\t\tcontinue;\n> +\n> +\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN && pack->bitmap_nr)\n> +\t\t\tBUG(\"pack '%s' has no bitmap position, but has %d bitmapped object(s)\",\n> +\t\t\t pack->pack_name, pack->bitmap_nr);\n> +\n> +\t\thashwrite_be32(f, pack->bitmap_pos);\n> +\t\thashwrite_be32(f, pack->bitmap_nr);\n> +\t\thashwrite_be32(f, !!pack->disjoint);\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> static int write_midx_oid_fanout(struct hashfile *f,\n> \t\t\t\t void *data)\n> {\n> @@ -981,8 +1053,19 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n> \tQSORT(data, ctx->entries_nr, midx_pack_order_cmp);\n> \n> \tALLOC_ARRAY(pack_order, ctx->entries_nr);\n> -\tfor (i = 0; i < ctx->entries_nr; i++)\n> +\tfor (i = 0; i < ctx->entries_nr; i++) {\n> +\t\tstruct pack_midx_entry *e = &ctx->entries[data[i].nr];\n> +\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]];\n> +\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n> +\t\t\tpack->bitmap_pos = i;\n> +\t\tpack->bitmap_nr++;\n> \t\tpack_order[i] = data[i].nr;\n> +\t}\n> +\tfor (i = 0; i < ctx->nr; i++) {\n> +\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[i]];\n> +\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n> +\t\t\tpack->bitmap_pos = 0;\n> +\t}\n> \tfree(data);\n> \n> \ttrace2_region_leave(\"midx\", \"midx_pack_order\", the_repository);\n> @@ -1283,6 +1366,7 @@ static int write_midx_internal(const char *object_dir,\n> \tstruct hashfile *f = NULL;\n> \tstruct lock_file lk;\n> \tstruct write_midx_context ctx = { 0 };\n> +\tint pack_disjoint_concat_len = 0;\n> \tint pack_name_concat_len = 0;\n> \tint dropped_packs = 0;\n> \tint result = 0;\n> @@ -1495,8 +1579,10 @@ static int write_midx_internal(const char *object_dir,\n> \t}\n> \n> \tfor (i = 0; i < ctx.nr; i++) {\n> -\t\tif (!ctx.info[i].expired)\n> -\t\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n> +\t\tif (ctx.info[i].expired)\n> +\t\t\tcontinue;\n> +\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n> +\t\tpack_disjoint_concat_len += 3 * sizeof(uint32_t);\n> \t}\n> \n> \t/* Check that the preferred pack wasn't expired (if given). */\n> @@ -1556,6 +1642,8 @@ static int write_midx_internal(const char *object_dir,\n> \t\tadd_chunk(cf, MIDX_CHUNKID_REVINDEX,\n> \t\t\t st_mult(ctx.entries_nr, sizeof(uint32_t)),\n> \t\t\t write_midx_revindex);\n> +\t\tadd_chunk(cf, MIDX_CHUNKID_DISJOINTPACKS,\n> +\t\t\t pack_disjoint_concat_len, write_midx_disjoint_packs);\n> \t}\n> \n> \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n> diff --git a/midx.h b/midx.h\n> index a5d98919c8..cdd16a8378 100644\n> --- a/midx.h\n> +++ b/midx.h\n> @@ -7,6 +7,7 @@\n> struct object_id;\n> struct pack_entry;\n> struct repository;\n> +struct bitmapped_pack;\n> \n> #define GIT_TEST_MULTI_PACK_INDEX \"GIT_TEST_MULTI_PACK_INDEX\"\n> #define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \\\n> @@ -33,6 +34,8 @@ struct multi_pack_index {\n> \n> \tconst unsigned char *chunk_pack_names;\n> \tsize_t chunk_pack_names_len;\n> +\tconst uint32_t *chunk_disjoint_packs;\n> +\tsize_t chunk_disjoint_packs_len;\n> \tconst uint32_t *chunk_oid_fanout;\n> \tconst unsigned char *chunk_oid_lookup;\n> \tconst unsigned char *chunk_object_offsets;\n> @@ -58,6 +61,8 @@ void get_midx_rev_filename(struct strbuf *out, struct multi_pack_index *m);\n> \n> struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n> int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n> +int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n> +\t\t struct bitmapped_pack *bp, uint32_t pack_int_id);\n> int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n> off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);\n> uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);\n> diff --git a/pack-bitmap.h b/pack-bitmap.h\n> index 5273a6a019..b7fa1a42a9 100644\n> --- a/pack-bitmap.h\n> +++ b/pack-bitmap.h\n> @@ -52,6 +52,15 @@ typedef int (*show_reachable_fn)(\n> \n> struct bitmap_index;\n> \n> +struct bitmapped_pack {\n> +\tstruct packed_git *p;\n> +\n> +\tuint32_t bitmap_pos;\n> +\tuint32_t bitmap_nr;\n> +\n> +\tunsigned disjoint : 1;\n> +};\n> +\n> struct bitmap_index *prepare_bitmap_git(struct repository *r);\n> struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx);\n> void count_bitmap_commit_list(struct bitmap_index *, uint32_t *commits,\n> diff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\n> index e9a444ddba..4b44995dca 100644\n> --- a/t/helper/test-read-midx.c\n> +++ b/t/helper/test-read-midx.c\n> @@ -100,10 +100,37 @@ static int read_midx_preferred_pack(const char *object_dir)\n> \treturn 0;\n> }\n> \n> +static int read_midx_bitmapped_packs(const char *object_dir)\n> +{\n> +\tstruct multi_pack_index *midx = NULL;\n> +\tstruct bitmapped_pack pack;\n> +\tuint32_t i;\n> +\n> +\tsetup_git_directory();\n> +\n> +\tmidx = load_multi_pack_index(object_dir, 1);\n> +\tif (!midx)\n> +\t\treturn 1;\n> +\n> +\tfor (i = 0; i < midx->num_packs; i++) {\n> +\t\tif (nth_bitmapped_pack(the_repository, midx, &pack, i) < 0)\n> +\t\t\treturn 1;\n> +\n> +\t\tprintf(\"%s\\n\", pack_basename(pack.p));\n> +\t\tprintf(\" bitmap_pos: %\"PRIuMAX\"\\n\", (uintmax_t)pack.bitmap_pos);\n> +\t\tprintf(\" bitmap_nr: %\"PRIuMAX\"\\n\", (uintmax_t)pack.bitmap_nr);\n> +\t\tprintf(\" disjoint: %s\\n\", pack.disjoint & 0x1 ? \"yes\" : \"no\");\n> +\t}\n> +\n> +\tclose_midx(midx);\n> +\n> +\treturn 0;\n> +}\n> +\n> int cmd__read_midx(int argc, const char **argv)\n> {\n> \tif (!(argc == 2 || argc == 3))\n> -\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack] \");\n> +\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] \");\n> \n> \tif (!strcmp(argv[1], \"--show-objects\"))\n> \t\treturn read_midx_file(argv[2], 1);\n> @@ -111,5 +138,7 @@ int cmd__read_midx(int argc, const char **argv)\n> \t\treturn read_midx_checksum(argv[2]);\n> \telse if (!strcmp(argv[1], \"--preferred-pack\"))\n> \t\treturn read_midx_preferred_pack(argv[2]);\n> +\telse if (!strcmp(argv[1], \"--bitmap\"))\n> +\t\treturn read_midx_bitmapped_packs(argv[2]);\n> \treturn read_midx_file(argv[1], 0);\n> }\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> index c4c6060cee..fd24e0c952 100755\n> --- a/t/t5319-multi-pack-index.sh\n> +++ b/t/t5319-multi-pack-index.sh\n> @@ -1157,4 +1157,62 @@ test_expect_success 'reader notices too-small revindex chunk' '\n> \ttest_cmp expect.err err\n> '\n> \n> +test_expect_success 'disjoint packs are stored via the DISP chunk' '\n> +\ttest_when_finished \"rm -fr repo\" &&\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\n> +\t\tfor i in 1 2 3 4 5\n> +\t\tdo\n> +\t\t\ttest_commit \"$i\" &&\n> +\t\t\tgit repack -d || return 1\n> +\t\tdone &&\n> +\n> +\t\tfind $objdir/pack -type f -name \"*.idx\" | xargs -n 1 basename | sort >packs &&\n> +\n> +\t\tgit multi-pack-index write --stdin-packs +\t\ttest_must_fail test-tool read-midx --bitmap $objdir 2>err &&\n> +\t\tcat >expect <<-\\EOF &&\n> +\t\terror: MIDX does not contain the DISP chunk\n> +\t\tEOF\n> +\t\ttest_cmp expect err &&\n> +\n> +\t\tsed -e \"s/^/+/g\" packs >in &&\n> +\t\tgit multi-pack-index write --stdin-packs --bitmap \\\n> +\t\t\t--preferred-pack=\"$(head -n1 +\t\ttest-tool read-midx --bitmap $objdir >actual &&\n> +\t\tfor i in $(test_seq $(wc -l +\t\tdo\n> +\t\t\tsed -ne \"${i}s/\\.idx$/\\.pack/p\" packs &&\n> +\t\t\techo \" bitmap_pos: $(( $(( $i - 1 )) * 3 ))\" &&\n> +\t\t\techo \" bitmap_nr: 3\" &&\n> +\t\t\techo \" disjoint: yes\" || return 1\n> +\t\tdone >expect &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> +test_expect_success 'non-disjoint packs are detected' '\n> +\ttest_when_finished \"rm -fr repo\" &&\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\n> +\t\ttest_commit base &&\n> +\t\tgit repack -d &&\n> +\t\ttest_commit other &&\n> +\t\tgit repack -a &&\n> +\n> +\t\tls -la .git/objects/pack/ &&\n> +\n> +\t\tfind $objdir/pack -type f -name \"*.idx\" |\n> +\t\t\tsed -e \"s/.*\\/\\(.*\\)$/+\\1/g\" >in &&\n> +\n> +\t\ttest_must_fail git multi-pack-index write --stdin-packs \\\n> +\t\t\t--bitmap err &&\n> +\t\tgrep \"duplicate object.* among disjoint packs\" err\n> +\t)\n> +'\n> +\n> test_done\n> -- \n> 2.43.0.24.g980b318f98\n> \n"},{"id":"485265","messageId":"ZWhhi15VpeCRflDB@tanuki","threadId":"60565","inReplyTo":"3019738b52ba8cd78ea696a3b800fa91e722eb66.1701198172.git.me@ttaylorr.com","subject":"Re: [PATCH 07/24] midx: implement `--retain-disjoint` mode","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-30T10:18:51Z","receivedAt":"2023-11-30T10:18:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 28, 2023 at 02:08:13PM -0500, Taylor Blau wrote:\n> Once multi-pack reachability bitmaps learn how to perform pack reuse\n> over the set of disjoint packs, we will want to teach `git repack` to\n> evolve the set of disjoint packs over time.\n> \n> To evolve the set of disjoint packs means any new packs made by `repack`\n> should be disjoint with respect to the existing set of disjoint packs so\n> as to be able to join that set when updating the multi-pack index.\n> \n> The details of generating such packs will be left to future commits. But\n> any new pack(s) created by repack as disjoint will be marked as such by\n> passing them over `--stdin-packs` with the special '+' marker when\n> generating a new MIDX.\n> \n> This patch, however, addresses the question of how we retain the\n> existing set of disjoint packs when updating the multi-pack index. One\n> option would be for `repack` to keep track of the set of disjoint packs\n> itself by querying the MIDX, and then adding the special '+' marker\n> appropriately when generating the input for `--stdin-packs`.\n> \n> But this is verbose and error-prone, since two different parts of Git\n> would need to maintain the same notion of the set of disjoint packs.\n> When one disagrees with the other, the set of so-called disjoint packs\n> may actually contain two or more packs which have one or more object(s)\n> in common, making the set non-disjoint.\n> \n> Instead, introduce a `--retain-disjoint` mode for the `git\n> multi-pack-index write` sub-command which keeps any packs which are:\n> \n> - marked as disjoint in the existing MIDX, and\n> \n> - not deleted (e.g., they are not excluded from the input for\n> `--stdin-packs`).\n> \n> This will allow the `repack` command to not have to keep track of the\n> set of currently-disjoint packs itself, reducing the number of lines of\n> code necessary to implement this feature, and making the resulting\n> implementation less error-prone.\n> \n> Signed-off-by: Taylor Blau \n> ---\n> Documentation/git-multi-pack-index.txt | 8 +++\n> builtin/multi-pack-index.c | 3 +\n> midx.c | 49 +++++++++++++++\n> midx.h | 1 +\n> t/lib-disjoint.sh | 38 ++++++++++++\n> t/t5319-multi-pack-index.sh | 82 ++++++++++++++++++++++++++\n> 6 files changed, 181 insertions(+)\n> create mode 100644 t/lib-disjoint.sh\n> \n> diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> index d130e65b28..ac0c7b124b 100644\n> --- a/Documentation/git-multi-pack-index.txt\n> +++ b/Documentation/git-multi-pack-index.txt\n> @@ -54,6 +54,14 @@ write::\n> \t\t\"disjoint\". See the \"`DISP` chunk and disjoint packs\"\n> \t\tsection in linkgit:gitformat-pack[5] for more.\n> \n> +\t--retain-disjoint::\n> +\t\tWhen writing a multi-pack index with a reachability\n> +\t\tbitmap, keep any packs marked as disjoint in the\n> +\t\texisting MIDX (if any) as such in the new MIDX. Existing\n> +\t\tdisjoint packs which are removed (e.g., not listed via\n> +\t\t`--stdin-packs`) are ignored. This option works in\n> +\t\taddition to the '+' marker for `--stdin-packs`.\n\nI'm trying to understand when you're expected to pass this flag and when\nyou're expected not to pass it. This documentation could also help in\nthe documentation here so that the user can make a more informed\ndecision.\n\nPatrick\n\n> \t--refs-snapshot=::\n> \t\tWith `--bitmap`, optionally specify a file which\n> \t\tcontains a \"refs snapshot\" taken prior to repacking.\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> index 0f1dd4651d..dcfabf2626 100644\n> --- a/builtin/multi-pack-index.c\n> +++ b/builtin/multi-pack-index.c\n> @@ -138,6 +138,9 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,\n> \t\t\t N_(\"write multi-pack index containing only given indexes\")),\n> \t\tOPT_FILENAME(0, \"refs-snapshot\", &opts.refs_snapshot,\n> \t\t\t N_(\"refs snapshot for selecting bitmap commits\")),\n> +\t\tOPT_BIT(0, \"retain-disjoint\", &opts.flags,\n> +\t\t\tN_(\"retain non-deleted disjoint packs\"),\n> +\t\t\tMIDX_WRITE_RETAIN_DISJOINT),\n> \t\tOPT_END(),\n> \t};\n> \n> diff --git a/midx.c b/midx.c\n> index 65ba0c70fe..ce67da9f85 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -721,6 +721,12 @@ static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n> \t\t\t\t\t &fanout->entries[fanout->nr],\n> \t\t\t\t\t cur_object);\n> \t\tfanout->entries[fanout->nr].preferred = 0;\n> +\t\t/*\n> +\t\t * It's OK to set disjoint to 0 here, even with\n> +\t\t * `--retain-disjoint`, since we will always see the disjoint\n> +\t\t * copy of some object below in get_sorted_entries(), causing us\n> +\t\t * to die().\n> +\t\t */\n> \t\tfanout->entries[fanout->nr].disjoint = 0;\n> \t\tfanout->nr++;\n> \t}\n> @@ -1362,6 +1368,37 @@ static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n> \treturn result;\n> }\n> \n> +static int midx_retain_existing_disjoint(struct repository *r,\n> +\t\t\t\t\t struct multi_pack_index *from,\n> +\t\t\t\t\t struct write_midx_context *ctx)\n> +{\n> +\tstruct bitmapped_pack bp;\n> +\tuint32_t i, midx_pos;\n> +\n> +\tfor (i = 0; i < ctx->nr; i++) {\n> +\t\tstruct pack_info *info = &ctx->info[i];\n> +\t\t/*\n> +\t\t * Having to call `midx_locate_pack()` in a loop is\n> +\t\t * sub-optimal, since it is O(n*log(n)) in the number\n> +\t\t * of packs.\n> +\t\t *\n> +\t\t * When reusing an existing MIDX, we know that the first\n> +\t\t * 'n' packs appear in the same order, so we could avoid\n> +\t\t * this when reusing an existing MIDX. But we may be\n> +\t\t * instead relying on the order given to us by\n> +\t\t * for_each_file_in_pack_dir(), in which case we can't\n> +\t\t * make any such guarantees.\n> +\t\t */\n> +\t\tif (!midx_locate_pack(from, info->pack_name, &midx_pos))\n> +\t\t\tcontinue;\n> +\n> +\t\tif (nth_bitmapped_pack(r, from, &bp, midx_pos) < 0)\n> +\t\t\treturn -1;\n> +\t\tinfo->disjoint = bp.disjoint;\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> static int write_midx_internal(const char *object_dir,\n> \t\t\t struct string_list *packs_to_include,\n> \t\t\t struct string_list *packs_to_drop,\n> @@ -1444,6 +1481,18 @@ static int write_midx_internal(const char *object_dir,\n> \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n> \tstop_progress(&ctx.progress);\n> \n> +\tif (flags & MIDX_WRITE_RETAIN_DISJOINT) {\n> +\t\tstruct multi_pack_index *m = ctx.m;\n> +\t\tif (!m)\n> +\t\t\tm = lookup_multi_pack_index(the_repository, object_dir);\n> +\n> +\t\tif (m) {\n> +\t\t\tresult = midx_retain_existing_disjoint(the_repository, m, &ctx);\n> +\t\t\tif (result)\n> +\t\t\t\tgoto cleanup;\n> +\t\t}\n> +\t}\n> +\n> \tif ((ctx.m && ctx.nr == ctx.m->num_packs) &&\n> \t !(packs_to_include || packs_to_drop)) {\n> \t\tstruct bitmap_index *bitmap_git;\n> diff --git a/midx.h b/midx.h\n> index a6e969c2ea..d7ce52ff7b 100644\n> --- a/midx.h\n> +++ b/midx.h\n> @@ -54,6 +54,7 @@ struct multi_pack_index {\n> #define MIDX_WRITE_BITMAP (1 << 2)\n> #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)\n> #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)\n> +#define MIDX_WRITE_RETAIN_DISJOINT (1 << 5)\n> \n> const unsigned char *get_midx_checksum(struct multi_pack_index *m);\n> void get_midx_filename(struct strbuf *out, const char *object_dir);\n> diff --git a/t/lib-disjoint.sh b/t/lib-disjoint.sh\n> new file mode 100644\n> index 0000000000..c6c6e74aba\n> --- /dev/null\n> +++ b/t/lib-disjoint.sh\n> @@ -0,0 +1,38 @@\n> +# Helpers for scripts testing disjoint packs; see t5319 for example usage.\n> +\n> +objdir=.git/objects\n> +\n> +test_disjoint_1 () {\n> +\tlocal pack=\"$1\"\n> +\tlocal want=\"$2\"\n> +\n> +\ttest-tool read-midx --bitmap $objdir >out &&\n> +\tgrep -A 3 \"$pack\" out >found &&\n> +\n> +\tif ! test -s found\n> +\tthen\n> +\t\techo >&2 \"could not find '$pack' in MIDX\"\n> +\t\treturn 1\n> +\tfi\n> +\n> +\tif ! grep -q \"disjoint: $want\" found\n> +\tthen\n> +\t\techo >&2 \"incorrect disjoint state for pack '$pack'\"\n> +\t\treturn 1\n> +\tfi\n> +\treturn 0\n> +}\n> +\n> +# test_must_be_disjoint \n> +#\n> +# Ensures that the given pack is marked as disjoint.\n> +test_must_be_disjoint () {\n> +\ttest_disjoint_1 \"$1\" \"yes\"\n> +}\n> +\n> +# test_must_not_be_disjoint \n> +#\n> +# Ensures that the given pack is not marked as disjoint.\n> +test_must_not_be_disjoint () {\n> +\ttest_disjoint_1 \"$1\" \"no\"\n> +}\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> index fd24e0c952..02cfddf151 100755\n> --- a/t/t5319-multi-pack-index.sh\n> +++ b/t/t5319-multi-pack-index.sh\n> @@ -3,6 +3,7 @@\n> test_description='multi-pack-indexes'\n> . ./test-lib.sh\n> . \"$TEST_DIRECTORY\"/lib-chunk.sh\n> +. \"$TEST_DIRECTORY\"/lib-disjoint.sh\n> \n> GIT_TEST_MULTI_PACK_INDEX=0\n> objdir=.git/objects\n> @@ -1215,4 +1216,85 @@ test_expect_success 'non-disjoint packs are detected' '\n> \t)\n> '\n> \n> +test_expect_success 'retain disjoint packs while writing' '\n> +\ttest_when_finished \"rm -fr repo\" &&\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\n> +\t\tfor i in 1 2\n> +\t\tdo\n> +\t\t\ttest_commit \"$i\" && git repack -d || return 1\n> +\t\tdone &&\n> +\n> +\t\tfind $objdir/pack -type f -name \"pack-*.idx\" |\n> +\t\tsed -e \"s/^.*\\/\\(.*\\)/\\1/g\" | sort >packs.old &&\n> +\n> +\t\ttest_line_count = 2 packs.old &&\n> +\t\tdisjoint=\"$(head -n 1 packs.old)\" &&\n> +\t\tnon_disjoint=\"$(tail -n 1 packs.old)\" &&\n> +\n> +\t\tcat >in <<-EOF &&\n> +\t\t+$disjoint\n> +\t\t$non_disjoint\n> +\t\tEOF\n> +\t\tgit multi-pack-index write --stdin-packs --bitmap +\n> +\t\ttest_must_be_disjoint \"${disjoint%.idx}.pack\" &&\n> +\t\ttest_must_not_be_disjoint \"${non_disjoint%.idx}.pack\" &&\n> +\n> +\t\ttest_commit 3 &&\n> +\t\tgit repack -d &&\n> +\n> +\t\tfind $objdir/pack -type f -name \"pack-*.idx\" |\n> +\t\tsed -e \"s/^.*\\/\\(.*\\)/\\1/g\" | sort >packs.new &&\n> +\n> +\t\tnew_disjoint=\"$(comm -13 packs.old packs.new)\" &&\n> +\t\tcat >in <<-EOF &&\n> +\t\t$disjoint\n> +\t\t$non_disjoint\n> +\t\t+$new_disjoint\n> +\t\tEOF\n> +\t\tgit multi-pack-index write --stdin-packs --bitmap \\\n> +\t\t\t--retain-disjoint +\n> +\t\ttest_must_be_disjoint \"${disjoint%.idx}.pack\" &&\n> +\t\ttest_must_be_disjoint \"${new_disjoint%.idx}.pack\" &&\n> +\t\ttest_must_not_be_disjoint \"${non_disjoint%.idx}.pack\"\n> +\n> +\t)\n> +'\n> +\n> +test_expect_success 'non-disjoint packs are detected via --retain-disjoint' '\n> +\ttest_when_finished \"rm -fr repo\" &&\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\tpackdir=.git/objects/pack &&\n> +\n> +\t\ttest_commit base &&\n> +\t\tbase=\"$(echo base | git pack-objects --revs $packdir/pack)\" &&\n> +\n> +\t\tcat >in <<-EOF &&\n> +\t\t+pack-$base.idx\n> +\t\tEOF\n> +\t\tgit multi-pack-index write --stdin-packs --bitmap +\n> +\t\ttest_must_be_disjoint \"pack-$base.pack\" &&\n> +\n> +\t\ttest_commit other &&\n> +\t\tother=\"$(echo other | git pack-objects --revs $packdir/pack)\" &&\n> +\n> +\t\tcat >in <<-EOF &&\n> +\t\tpack-$base.idx\n> +\t\t+pack-$other.idx\n> +\t\tEOF\n> +\t\ttest_must_fail git multi-pack-index write --stdin-packs --retain-disjoint --bitmap err &&\n> +\t\tgrep \"duplicate object.* among disjoint packs\" err &&\n> +\n> +\t\ttest_must_fail git multi-pack-index write --retain-disjoint --bitmap 2>err &&\n> +\t\tgrep \"duplicate object.* among disjoint packs\" err\n> +\t)\n> +'\n> +\n> test_done\n> -- \n> 2.43.0.24.g980b318f98\n> \n"},{"id":"485266","messageId":"ZWhhkdnVZ9w7tDBv@tanuki","threadId":"60565","inReplyTo":"0368f7ab37669163b50b82185725935bde5bc946.1701198172.git.me@ttaylorr.com","subject":"Re: [PATCH 08/24] pack-objects: implement `--ignore-disjoint` mode","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-30T10:18:57Z","receivedAt":"2023-11-30T10:19:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 28, 2023 at 02:08:16PM -0500, Taylor Blau wrote:\n> Before multi-pack reachability bitmaps learn how to perform pack reuse\n> over the set of disjoint packs, we will need a way to generate packs\n> that are known to be disjoint with respect to the currently marked set\n> of disjoint packs.\n> \n> In other words, we want a way to make a pack which does not have any\n> objects contained in the union of the set of packs which are currently\n> marked as disjoint.\n> \n> There are a various ways that we could go about this, for example:\n> \n> - passing `--unpacked`, which would exclude all packed objects (and\n> thus would not contain any objects from the disjoint pack)\n> \n> - passing `--stdin-packs` with the set of packs currently marked as\n> disjoint as \"excluded\", indicating that `pack-objects` should\n> discard any objects present in any of the excluded packs (thus\n> producing a disjoint pack)\n> \n> - marking each of the disjoint packs as kept in-core with the\n> `--keep-pack` flag, and then passing `--honor-pack-keep` to\n> similarly ignore any object(s) from kept packs (thus also producing\n> a pack which is disjoint with respect to the current set)\n> \n> `git repack` is the main entry-point to generating a new pack, by\n> invoking `pack-objects` and then adding the new pack to the set of\n> disjoint packs if generating a new MIDX. However, `repack` has a number\n> of ways to invoke `pack-objects` (e.g., all-into-one repacks, geometric\n> repacks, incremental repacks, etc.), all of which would require careful\n> reasoning in order to prove that the resulting set of packs is disjoint.\n> \n> The most appealing option of the above would be to pass the set of\n> disjoint packs as kept (via `--keep-pack`) and then ignore their\n> contents (with `--honor-pack-keep`), doing so for all kinds of\n> `pack-objects` invocations. But there may be more disjoint packs than we\n> can easily fit into the command-line arguments.\n> \n> Instead, teach `pack-objects` a special `--ignore-disjoint` which is the\n> moral equivalent of marking the set of disjoint packs as kept, and\n> ignoring their contents, even if it would have otherwise been packed. In\n> fact, this similarity extends down to the implementation, where each\n> disjoint pack is first loaded, then has its `pack_keep_in_core` bit set.\n> \n> With this in place, we can use the kept-pack cache from 20b031fede\n> (packfile: add kept-pack cache for find_kept_pack_entry(), 2021-02-22),\n> which looks up objects first in a cache containing just the set of kept\n> (in this case, disjoint) packs. Assuming that the set of disjoint packs\n> is a relatively small portion of the entire repository (which should be\n> a safe assumption to make), each object lookup will be very inexpensive.\n\nThis cought me by surprise a bit. I'd have expected that in the end,\nmost of the packfiles in a repository would be disjoint. Using for\nexample geometric repacks, my expectation was that all of the packs that\nget written via geometric repacking would eventually become disjoint\nwhereas new packs added to the repository would initially not be.\n\nPatrick\n\n> The only place we want to avoid using `--ignore-disjoint` is in\n> conjunction with `--cruft`, since doing so may cause us to omit an\n> object which would have been included in a new cruft pack in order to\n> freshen it. In other words, failing to do so might cause that object to\n> be pruned from the repository earlier than expected.\n> \n> Otherwise, `--ignore-disjoint` is compatible with most other modes of\n> `pack-objects`. These various combinations are tested below. As a\n> result, `repack` will be able to unconditionally (except for the cruft\n> pack) pass `--ignore-disjoint` when trying to add a new pack to the\n> disjoint set, and the result will be usable, without having to carefully\n> consider and reason about each individual case.\n> \n> Signed-off-by: Taylor Blau \n> ---\n> Documentation/git-pack-objects.txt | 8 ++\n> builtin/pack-objects.c | 31 +++++-\n> t/lib-disjoint.sh | 11 ++\n> t/t5331-pack-objects-stdin.sh | 156 +++++++++++++++++++++++++++++\n> 4 files changed, 203 insertions(+), 3 deletions(-)\n> \n> diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\n> index e32404c6aa..592c4ce742 100644\n> --- a/Documentation/git-pack-objects.txt\n> +++ b/Documentation/git-pack-objects.txt\n> @@ -96,6 +96,14 @@ base-name::\n> Incompatible with `--revs`, or options that imply `--revs` (such as\n> `--all`), with the exception of `--unpacked`, which is compatible.\n> \n> +--ignore-disjoint::\n> +\tThis flag causes an object that appears in any pack marked as\n> +\t\"disjoint\" by the multi-pack index to be ignored, even if it\n> +\twould have otherwise been packed. When used with\n> +\t`--stdin-packs`, objects from disjoint packs may be included if\n> +\tand only if a disjoint pack is explicitly given as an input pack\n> +\tto `--stdin-packs`. Incompatible with `--cruft`.\n> +\n> --cruft::\n> \tPacks unreachable objects into a separate \"cruft\" pack, denoted\n> \tby the existence of a `.mtimes` file. Typically used by `git\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index bfa60359d4..107154db34 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -207,6 +207,7 @@ static int have_non_local_packs;\n> static int incremental;\n> static int ignore_packed_keep_on_disk;\n> static int ignore_packed_keep_in_core;\n> +static int ignore_midx_disjoint_packs;\n> static int allow_ofs_delta;\n> static struct pack_idx_option pack_idx_opts;\n> static const char *base_name;\n> @@ -1403,7 +1404,8 @@ static int want_found_object(const struct object_id *oid, int exclude,\n> \t/*\n> \t * Then handle .keep first, as we have a fast(er) path there.\n> \t */\n> -\tif (ignore_packed_keep_on_disk || ignore_packed_keep_in_core) {\n> +\tif (ignore_packed_keep_on_disk || ignore_packed_keep_in_core ||\n> +\t ignore_midx_disjoint_packs) {\n> \t\t/*\n> \t\t * Set the flags for the kept-pack cache to be the ones we want\n> \t\t * to ignore.\n> @@ -1415,7 +1417,7 @@ static int want_found_object(const struct object_id *oid, int exclude,\n> \t\tunsigned flags = 0;\n> \t\tif (ignore_packed_keep_on_disk)\n> \t\t\tflags |= ON_DISK_KEEP_PACKS;\n> -\t\tif (ignore_packed_keep_in_core)\n> +\t\tif (ignore_packed_keep_in_core || ignore_midx_disjoint_packs)\n> \t\t\tflags |= IN_CORE_KEEP_PACKS;\n> \n> \t\tif (ignore_packed_keep_on_disk && p->pack_keep)\n> @@ -3389,6 +3391,7 @@ static void read_packs_list_from_stdin(void)\n> \t\t\tdie(_(\"could not find pack '%s'\"), item->string);\n> \t\tif (!is_pack_valid(p))\n> \t\t\tdie(_(\"packfile %s cannot be accessed\"), p->pack_name);\n> +\t\tp->pack_keep_in_core = 0;\n> \t}\n> \n> \t/*\n> @@ -4266,6 +4269,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> \t\t\t N_(\"create packs suitable for shallow fetches\")),\n> \t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep_on_disk,\n> \t\t\t N_(\"ignore packs that have companion .keep file\")),\n> +\t\tOPT_BOOL(0, \"ignore-disjoint\", &ignore_midx_disjoint_packs,\n> +\t\t\t N_(\"ignore packs that are marked disjoint in the MIDX\")),\n> \t\tOPT_STRING_LIST(0, \"keep-pack\", &keep_pack_list, N_(\"name\"),\n> \t\t\t\tN_(\"ignore this pack\")),\n> \t\tOPT_INTEGER(0, \"compression\", &pack_compression_level,\n> @@ -4412,7 +4417,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> \t\tif (use_internal_rev_list)\n> \t\t\tdie(_(\"cannot use internal rev list with --cruft\"));\n> \t\tif (stdin_packs)\n> -\t\t\tdie(_(\"cannot use --stdin-packs with --cruft\"));\n> +\t\t\tdie(_(\"cannot use %s with %s\"), \"--stdin-packs\", \"--cruft\");\n> +\t\tif (ignore_midx_disjoint_packs)\n> +\t\t\tdie(_(\"cannot use %s with %s\"), \"--ignore-disjoint\", \"--cruft\");\n> \t}\n> \n> \t/*\n> @@ -4452,6 +4459,24 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> \t\tif (!p) /* no keep-able packs found */\n> \t\t\tignore_packed_keep_on_disk = 0;\n> \t}\n> +\tif (ignore_midx_disjoint_packs) {\n> +\t\tstruct multi_pack_index *m = get_multi_pack_index(the_repository);\n> +\t\tstruct bitmapped_pack pack;\n> +\t\tunsigned any_disjoint = 0;\n> +\t\tuint32_t i;\n> +\n> +\t\tfor (i = 0; m && m->chunk_disjoint_packs && i < m->num_packs; i++) {\n> +\t\t\tif (nth_bitmapped_pack(the_repository, m, &pack, i) < 0)\n> +\t\t\t\tdie(_(\"could not load bitmapped pack %i\"), i);\n> +\t\t\tif (pack.disjoint) {\n> +\t\t\t\tpack.p->pack_keep_in_core = 1;\n> +\t\t\t\tany_disjoint = 1;\n> +\t\t\t}\n> +\t\t}\n> +\n> +\t\tif (!any_disjoint) /* no disjoint packs to ignore */\n> +\t\t\tignore_midx_disjoint_packs = 0;\n> +\t}\n> \tif (local) {\n> \t\t/*\n> \t\t * unlike ignore_packed_keep_on_disk above, we do not\n> diff --git a/t/lib-disjoint.sh b/t/lib-disjoint.sh\n> index c6c6e74aba..c802ca6940 100644\n> --- a/t/lib-disjoint.sh\n> +++ b/t/lib-disjoint.sh\n> @@ -36,3 +36,14 @@ test_must_be_disjoint () {\n> test_must_not_be_disjoint () {\n> \ttest_disjoint_1 \"$1\" \"no\"\n> }\n> +\n> +# packed_contents \n> +#\n> +# Prints the set of objects packed in the given pack indexes.\n> +packed_contents () {\n> +\tfor idx in \"$@\"\n> +\tdo\n> +\t\tgit show-index <$idx || return 1\n> +\tdone >tmp &&\n> +\tcut -d\" \" -f2 +}\n> diff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh\n> index 2dcf1eecee..e522aa3f7d 100755\n> --- a/t/t5331-pack-objects-stdin.sh\n> +++ b/t/t5331-pack-objects-stdin.sh\n> @@ -6,6 +6,7 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n> \n> TEST_PASSES_SANITIZE_LEAK=true\n> . ./test-lib.sh\n> +. \"$TEST_DIRECTORY\"/lib-disjoint.sh\n> \n> packed_objects () {\n> \tgit show-index <\"$1\" >tmp-object-list &&\n> @@ -237,4 +238,159 @@ test_expect_success 'pack-objects --stdin with packfiles from main and alternate\n> \ttest_cmp expected-objects actual-objects\n> '\n> \n> +objdir=.git/objects\n> +packdir=$objdir/pack\n> +\n> +test_expect_success 'loose objects also in disjoint packs are ignored' '\n> +\ttest_when_finished \"rm -fr repo\" &&\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\n> +\t\t# create a pack containing the objects in each commit below, but\n> +\t\t# do not delete their loose copies\n> +\t\ttest_commit base &&\n> +\t\tbase_pack=\"$(echo base | git pack-objects --revs $packdir/pack)\" &&\n> +\n> +\t\ttest_commit other &&\n> +\t\tother_pack=\"$(echo base..other | git pack-objects --revs $packdir/pack)\" &&\n> +\n> +\t\tcat >in <<-EOF &&\n> +\t\tpack-$base_pack.idx\n> +\t\t+pack-$other_pack.idx\n> +\t\tEOF\n> +\t\tgit multi-pack-index write --stdin-packs --bitmap +\n> +\t\ttest_commit more &&\n> +\t\tout=\"$(git pack-objects --all --ignore-disjoint $packdir/pack)\" &&\n> +\n> +\t\t# gather all objects in \"all\", and objects from the disjoint\n> +\t\t# pack in \"disjoint\"\n> +\t\tgit cat-file --batch-all-objects --batch-check=\"%(objectname)\" >all &&\n> +\t\tpacked_contents \"$packdir/pack-$other_pack.idx\" >disjoint &&\n> +\n> +\t\t# make sure that the set of objects we just generated matches\n> +\t\t# \"all \\ disjoint\"\n> +\t\tpacked_contents \"$packdir/pack-$out.idx\" >got &&\n> +\t\tcomm -23 all disjoint >want &&\n> +\t\ttest_cmp want got\n> +\t)\n> +'\n> +\n> +test_expect_success 'objects in disjoint packs are ignored (--unpacked)' '\n> +\ttest_when_finished \"rm -fr repo\" &&\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\n> +\t\tfor c in A B\n> +\t\tdo\n> +\t\t\ttest_commit \"$c\" || return 1\n> +\t\tdone &&\n> +\n> +\t\tA=\"$(echo \"A\" | git pack-objects --revs $packdir/pack)\" &&\n> +\t\tB=\"$(echo \"A..B\" | git pack-objects --revs $packdir/pack)\" &&\n> +\n> +\t\tcat >in <<-EOF &&\n> +\t\tpack-$A.idx\n> +\t\t+pack-$B.idx\n> +\t\tEOF\n> +\t\tgit multi-pack-index write --stdin-packs --bitmap +\n> +\t\ttest_must_not_be_disjoint \"pack-$A.pack\" &&\n> +\t\ttest_must_be_disjoint \"pack-$B.pack\" &&\n> +\n> +\t\ttest_commit C &&\n> +\n> +\t\tgot=\"$(git pack-objects --all --unpacked --ignore-disjoint $packdir/pack)\" &&\n> +\t\tpacked_contents \"$packdir/pack-$got.idx\" >actual &&\n> +\n> +\t\tgit rev-list --objects --no-object-names B..C >expect.raw &&\n> +\t\tsort expect &&\n> +\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> +test_expect_success 'objects in disjoint packs are ignored (--stdin-packs)' '\n> +\t# Create objects in three separate packs:\n> +\t#\n> +\t# - pack A (midx, non disjoint)\n> +\t# - pack B (midx, disjoint)\n> +\t# - pack C (non-midx)\n> +\t#\n> +\t# Then create a new pack with `--stdin-packs` and `--ignore-disjoint`\n> +\t# including packs A, B, and C. The resulting pack should contain\n> +\t# only the objects from packs A, and C, excluding those from\n> +\t# pack B as it is marked as disjoint.\n> +\ttest_when_finished \"rm -fr repo\" &&\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\n> +\t\tfor c in A B C\n> +\t\tdo\n> +\t\t\ttest_commit \"$c\" || return 1\n> +\t\tdone &&\n> +\n> +\t\tA=\"$(echo \"A\" | git pack-objects --revs $packdir/pack)\" &&\n> +\t\tB=\"$(echo \"A..B\" | git pack-objects --revs $packdir/pack)\" &&\n> +\t\tC=\"$(echo \"B..C\" | git pack-objects --revs $packdir/pack)\" &&\n> +\n> +\t\tcat >in <<-EOF &&\n> +\t\tpack-$A.idx\n> +\t\t+pack-$B.idx\n> +\t\tEOF\n> +\t\tgit multi-pack-index write --stdin-packs --bitmap +\n> +\t\ttest_must_not_be_disjoint \"pack-$A.pack\" &&\n> +\t\ttest_must_be_disjoint \"pack-$B.pack\" &&\n> +\n> +\t\t# Generate a pack with `--stdin-packs` using packs \"A\" and \"C\",\n> +\t\t# but excluding objects from \"B\". The objects from pack \"B\" are\n> +\t\t# expected to be omitted from the generated pack for two\n> +\t\t# reasons:\n> +\t\t#\n> +\t\t# - because it was specified as a negated tip via\n> +\t\t# `--stdin-packs`\n> +\t\t# - because it is a disjoint pack.\n> +\t\tcat >in <<-EOF &&\n> +\t\tpack-$A.pack\n> +\t\t^pack-$B.pack\n> +\t\tpack-$C.pack\n> +\t\tEOF\n> +\t\tgot=\"$(git pack-objects --stdin-packs --ignore-disjoint $packdir/pack +\n> +\t\tpacked_contents \"$packdir/pack-$got.idx\" >actual &&\n> +\t\tpacked_contents \"$packdir/pack-$A.idx\" \\\n> +\t\t\t\t\"$packdir/pack-$C.idx\" >expect &&\n> +\t\ttest_cmp expect actual &&\n> +\n> +\t\t# Generate another pack with `--stdin-packs`, this time\n> +\t\t# using packs \"B\" and \"C\". The objects from pack \"B\" are\n> +\t\t# expected to be in the final pack, despite it being a\n> +\t\t# disjoint pack, because \"B\" was mentioned explicitly\n> +\t\t# via `stdin-packs`.\n> +\t\tcat >in <<-EOF &&\n> +\t\tpack-$B.pack\n> +\t\tpack-$C.pack\n> +\t\tEOF\n> +\t\tgot=\"$(git pack-objects --stdin-packs --ignore-disjoint $packdir/pack +\n> +\t\tpacked_contents \"$packdir/pack-$got.idx\" >actual &&\n> +\t\tpacked_contents \"$packdir/pack-$B.idx\" \\\n> +\t\t\t\t\"$packdir/pack-$C.idx\" >expect &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> +test_expect_success '--cruft is incompatible with --ignore-disjoint' '\n> +\ttest_must_fail git pack-objects --cruft --ignore-disjoint --stdout \\\n> +\t\t/dev/null 2>actual &&\n> +\tcat >expect <<-\\EOF &&\n> +\tfatal: cannot use --ignore-disjoint with --cruft\n> +\tEOF\n> +\ttest_cmp expect actual\n> +'\n> +\n> test_done\n> -- \n> 2.43.0.24.g980b318f98\n> \n"},{"id":"485279","messageId":"ZWjdu4h96CSKU/Tr@nand.local","threadId":"60565","inReplyTo":"ZWhhcv25B5nUNoyu@tanuki","subject":"Re: [PATCH 01/24] pack-objects: free packing_data in more places","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-30T19:08:43Z","receivedAt":"2023-11-30T19:08:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Nov 30, 2023 at 11:18:26AM +0100, Patrick Steinhardt wrote:\n> > diff --git a/pack-objects.c b/pack-objects.c\n> > index f403ca6986..1c7bedcc94 100644\n> > --- a/pack-objects.c\n> > +++ b/pack-objects.c\n> > @@ -151,6 +151,21 @@ void prepare_packing_data(struct repository *r, struct packing_data *pdata)\n> > \tinit_recursive_mutex(&pdata->odb_lock);\n> > }\n> >\n> > +void free_packing_data(struct packing_data *pdata)\n>\n> Nit: shouldn't this rather be called `clear_packing_data`? `free` to me\n> indicates that the data structure itself will be free'd, as well, which\n> is not the case.\n\nThanks, that's a good suggestion. I've made the changes locally and will\ninclude it in the subsequent round(s) ;-).\n\nThanks,\nTaylor\n"},{"id":"485280","messageId":"ZWjeYP2vGTyAuQo+@nand.local","threadId":"60565","inReplyTo":"ZWhhd5gLzYwPEgBl@tanuki","subject":"Re: [PATCH 02/24] pack-bitmap-write: deep-clear the `bb_commit` slab","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-30T19:11:28Z","receivedAt":"2023-11-30T19:11:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Nov 30, 2023 at 11:18:31AM +0100, Patrick Steinhardt wrote:\n> > diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> > index f4ecdf8b0e..dd3a415b9d 100644\n> > --- a/pack-bitmap-write.c\n> > +++ b/pack-bitmap-write.c\n> > @@ -198,6 +198,13 @@ struct bb_commit {\n> > \tunsigned idx; /* within selected array */\n> > };\n> >\n> > +static void clear_bb_commit(struct bb_commit *commit)\n> > +{\n> > +\tfree(commit->reverse_edges);\n>\n> I'd have expected to see `free_commit_list()` here instead of a simple\n> free. Is there any reason why we don't use it?\n\nThanks for spotting an oversight on my part. We should definitely be\nusing free_commit_list() here instead of a bare free() to avoid leaking\nthe tail.\n\nThanks,\nTaylor\n"},{"id":"485281","messageId":"ZWjgTfz0ufVoPgbw@nand.local","threadId":"60565","inReplyTo":"ZWhhfXoovgYzIYE0@tanuki","subject":"Re: [PATCH 04/24] midx: factor out `fill_pack_info()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-30T19:19:41Z","receivedAt":"2023-11-30T19:19:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Nov 30, 2023 at 11:18:37AM +0100, Patrick Steinhardt wrote:\n> On Tue, Nov 28, 2023 at 02:08:05PM -0500, Taylor Blau wrote:\n> > When selecting which packfiles will be written while generating a MIDX,\n> > the MIDX internals fill out a 'struct pack_info' with various pieces of\n> > book-keeping.\n> >\n> > Instead of filling out each field of the `pack_info` structure\n> > individually in each of the two spots that modify the array of such\n> > structures (`ctx->info`), extract a common routine that does this for\n> > us.\n> >\n> > This reduces the code duplication by a modest amount. But more\n> > importantly, it zero-initializes the structure before assigning values\n> > into it. This hardens us for a future change which will add additional\n> > fields to this structure which (until this patch) was not\n> > zero-initialized.\n> >\n> > As a result, any new fields added to the `pack_info` structure need only\n> > be updated in a single location, instead of at each spot within midx.c.\n> >\n> > There are no functional changes in this patch.\n> >\n> > Signed-off-by: Taylor Blau \n> > ---\n> > midx.c | 35 +++++++++++++++++++----------------\n> > 1 file changed, 19 insertions(+), 16 deletions(-)\n> >\n> > diff --git a/midx.c b/midx.c\n> > index 3b727dc633..591b3c636e 100644\n> > --- a/midx.c\n> > +++ b/midx.c\n> > @@ -464,6 +464,17 @@ struct pack_info {\n> > \tunsigned expired : 1;\n> > };\n> >\n> > +static void fill_pack_info(struct pack_info *info,\n> > +\t\t\t struct packed_git *p, char *pack_name,\n> > +\t\t\t uint32_t orig_pack_int_id)\n> > +{\n> > +\tmemset(info, 0, sizeof(struct pack_info));\n> > +\n> > +\tinfo->orig_pack_int_id = orig_pack_int_id;\n> > +\tinfo->pack_name = pack_name;\n> > +\tinfo->p = p;\n> > +}\n>\n> Nit: all callers manually call `xstrdup(pack_name)` and pass that to\n> `fill_pack_info()`. We could consider doing this in here instead so that\n> ownership of the string becomes a tad clearer.\n\nThat's a great idea. I think we'd also want to mark the pack_name\nargument as const, not just because xstrdup() requires it, but also\nbecause it communicates the ownership more clearly.\n\nI'll squash something like this in:\n\n--- >8 ---\ndiff --git a/midx.c b/midx.c\nindex b8b3f41024..6fb5e237b7 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -465,13 +465,13 @@ struct pack_info {\n };\n\n static void fill_pack_info(struct pack_info *info,\n-\t\t\t struct packed_git *p, char *pack_name,\n+\t\t\t struct packed_git *p, const char *pack_name,\n \t\t\t uint32_t orig_pack_int_id)\n {\n \tmemset(info, 0, sizeof(struct pack_info));\n\n \tinfo->orig_pack_int_id = orig_pack_int_id;\n-\tinfo->pack_name = pack_name;\n+\tinfo->pack_name = xstrdup(pack_name);\n \tinfo->p = p;\n }\n\n@@ -557,8 +557,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t\treturn;\n \t\t}\n\n-\t\tfill_pack_info(&ctx->info[ctx->nr], p, xstrdup(file_name),\n-\t\t\t ctx->nr);\n+\t\tfill_pack_info(&ctx->info[ctx->nr], p, file_name, ctx->nr);\n \t\tctx->nr++;\n \t}\n }\n@@ -1336,7 +1335,7 @@ static int write_midx_internal(const char *object_dir,\n \t\t\t}\n\n \t\t\tfill_pack_info(&ctx.info[ctx.nr++], ctx.m->packs[i],\n-\t\t\t\t xstrdup(ctx.m->pack_names[i]), i);\n+\t\t\t\t ctx.m->pack_names[i], i);\n \t\t}\n \t}\n--- 8< ---\n\n> > -\t\tif (open_pack_index(ctx->info[ctx->nr].p)) {\n> > +\t\tif (open_pack_index(p)) {\n> > \t\t\twarning(_(\"failed to open pack-index '%s'\"),\n> > \t\t\t\tfull_path);\n> > \t\t\tclose_pack(ctx->info[ctx->nr].p);\n>\n> Isn't `ctx->info[ctx->nr].p` still uninitialized at this point?\n\nGreat catch, thank you!\n\n> > @@ -1330,10 +1333,10 @@ static int write_midx_internal(const char *object_dir,\n> > \t\t\t\tif (open_pack_index(ctx.m->packs[i]))\n> > \t\t\t\t\tdie(_(\"could not open index for %s\"),\n> > \t\t\t\t\t ctx.m->packs[i]->pack_name);\n> > -\t\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n>\n> Just to make sure I'm not missing anything, but this assignment here was\n> basically redundant before this patch already, right?\n\nI think that's right, but in either case we're assigning the pack once\nat the end of each loop iteration via a single call to fill_pack_info().\nSince we're using ctx.m->packs[i] in both places (after a call to\nprepare_midx_pack()), we should be OK here.\n\nThanks,\nTaylor\n"},{"id":"485282","messageId":"ZWjiHe/roQCqpjsu@nand.local","threadId":"60565","inReplyTo":"ZWhhhaXiF_wC3er7@tanuki","subject":"Re: [PATCH 05/24] midx: implement `DISP` chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-30T19:27:25Z","receivedAt":"2023-11-30T19:27:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Nov 30, 2023 at 11:18:45AM +0100, Patrick Steinhardt wrote:\n> > diff --git a/Documentation/gitformat-pack.txt b/Documentation/gitformat-pack.txt\n> > index 9fcb29a9c8..658682ddd5 100644\n> > --- a/Documentation/gitformat-pack.txt\n> > +++ b/Documentation/gitformat-pack.txt\n> > @@ -396,6 +396,22 @@ CHUNK DATA:\n> > \t is padded at the end with between 0 and 3 NUL bytes to make the\n> > \t chunk size a multiple of 4 bytes.\n> >\n> > +\tDisjoint Packfiles (ID: {'D', 'I', 'S', 'P'})\n> > +\t Stores a table of three 4-byte unsigned integers in network order.\n> > +\t Each table entry corresponds to a single pack (in the order that\n> > +\t they appear above in the `PNAM` chunk). The values for each table\n> > +\t entry are as follows:\n> > +\t - The first bit position (in psuedo-pack order, see below) to\n>\n> s/psuedo/pseudo/\n\nGood catch, thanks. Not sure how that escaped my spell-checker...\n\n> > +=== `DISP` chunk and disjoint packs\n> > +\n> > +The Disjoint Packfiles (`DISP`) chunk encodes additional information\n> > +about the objects in the multi-pack index's reachability bitmap. Recall\n> > +that objects from the MIDX are arranged in \"pseudo-pack\" order (see:\n>\n> The colon feels a bit out-of-place here, so: s/see:/see/\n\nThanks, I'll fix that up.\n\n> > +above) for reachability bitmaps.\n> > +\n> > +From the example above, suppose we have packs \"a\", \"b\", and \"c\", with\n> > +10, 15, and 20 objects, respectively. In pseudo-pack order, those would\n> > +be arranged as follows:\n> > +\n> > + |a,0|a,1|...|a,9|b,0|b,1|...|b,14|c,0|c,1|...|c,19|\n> > +\n> > +When working with single-pack bitmaps (or, equivalently, multi-pack\n> > +reachability bitmaps without any packs marked as disjoint),\n> > +linkgit:git-pack-objects[1] performs ``verbatim'' reuse, attempting to\n> > +reuse chunks of the existing packfile instead of adding objects to the\n> > +packing list.\n>\n> I'm not sure I full understand this paragraph. In the context of a\n> single pack bitmap it's clear enough. But I stumbled over the MIDX case,\n> because here we potentially have multiple packfiles, so it's not exactly\n> clear to me what you refer to with \"the existing packfile\" in that case.\n> I'd think that we perform verbatim reuse of the preferred packfile,\n> right? If so, we might want to make that a bit more explicit.\n\nYep, sorry, I can see how that would be confusing. Since we're talking\nabout the existing behavior at this point in the series (before\nmulti-pack reuse is implemented), I changed this to:\n\n \"reuse chunks of the bitmapped or preferred packfile [...]\"\n\nThanks for carefully reading and spotting my errors ;-).\n\n> > +object. This introduces an additional constraint over the set of packs\n> > +we may want to reuse. The most straightforward approach is to mandate\n> > +that the set of packs is disjoint with respect to the set of objects\n> > +contained in each pack. In other words, for each object `o` in the union\n> > +of all objects stored by the disjoint set of packs, `o` is contained in\n> > +exactly one pack from the disjoint set.\n>\n> Is this a property that usually holds for our normal housekeeping, or\n> does it require careful managing by the user/admin? How about geometric\n> repacking?\n\nAt this point in the series, it would require careful managing to ensure\nthat this is the case. In practice MIDX'd packs generated with a\ngeometric repack are mostly disjoint, but definitely not guaranteed to\nbe.\n\nFurther down in this series we'll introduce new options to generate\npacks which are guaranteed to be disjoint with respect to the\ncurrently-marked set of packs in the DISP chunk.\n\n> > @@ -764,14 +807,22 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n> > \t\t * Take only the first duplicate.\n> > \t\t */\n> > \t\tfor (cur_object = 0; cur_object < fanout.nr; cur_object++) {\n> > -\t\t\tif (cur_object && oideq(&fanout.entries[cur_object - 1].oid,\n> > -\t\t\t\t\t\t&fanout.entries[cur_object].oid))\n> > -\t\t\t\tcontinue;\n> > +\t\t\tstruct pack_midx_entry *ours = &fanout.entries[cur_object];\n> > +\t\t\tif (cur_object) {\n> > +\t\t\t\tstruct pack_midx_entry *prev = &fanout.entries[cur_object - 1];\n> > +\t\t\t\tif (oideq(&prev->oid, &ours->oid)) {\n> > +\t\t\t\t\tif (prev->disjoint && ours->disjoint)\n> > +\t\t\t\t\t\tdie(_(\"duplicate object '%s' among disjoint packs '%s', '%s'\"),\n> > +\t\t\t\t\t\t oid_to_hex(&prev->oid),\n> > +\t\t\t\t\t\t info[prev->pack_int_id].pack_name,\n> > +\t\t\t\t\t\t info[ours->pack_int_id].pack_name);\n>\n> Shouldn't we die if `prev->disjoint || ours->disjoint` instead of `&&`?\n> Even if one of the packs isn't marked as disjoint, it's still wrong if\n> the other one is and one of its objects exists in multiple packs.\n>\n> Or am I misunderstanding, and we only guarantee the disjoint property\n> across packfiles that are actually marked as such?\n\nRight, we only guarantee disjointed-ness among the set of packs that are\nmarked disjoint. It's fine for the same object to appear in a disjoint\nand non-disjoint pack, and for both of those packs to end up in the\nMIDX. But that is only because we'll use the disjoint copy in our\nbitmap.\n\nIf there were two packs that are marked as supposedly disjoint, but\ncontain at least one duplicate of an object, then we will reject those\npacks as non-disjoint.\n\nThanks,\nTaylor\n"},{"id":"485284","messageId":"ZWjisIgiuziMvBph@nand.local","threadId":"60565","inReplyTo":"ZWhhi15VpeCRflDB@tanuki","subject":"Re: [PATCH 07/24] midx: implement `--retain-disjoint` mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-30T19:29:52Z","receivedAt":"2023-11-30T19:29:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Nov 30, 2023 at 11:18:51AM +0100, Patrick Steinhardt wrote:\n> > diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> > index d130e65b28..ac0c7b124b 100644\n> > --- a/Documentation/git-multi-pack-index.txt\n> > +++ b/Documentation/git-multi-pack-index.txt\n> > @@ -54,6 +54,14 @@ write::\n> > \t\t\"disjoint\". See the \"`DISP` chunk and disjoint packs\"\n> > \t\tsection in linkgit:gitformat-pack[5] for more.\n> >\n> > +\t--retain-disjoint::\n> > +\t\tWhen writing a multi-pack index with a reachability\n> > +\t\tbitmap, keep any packs marked as disjoint in the\n> > +\t\texisting MIDX (if any) as such in the new MIDX. Existing\n> > +\t\tdisjoint packs which are removed (e.g., not listed via\n> > +\t\t`--stdin-packs`) are ignored. This option works in\n> > +\t\taddition to the '+' marker for `--stdin-packs`.\n>\n> I'm trying to understand when you're expected to pass this flag and when\n> you're expected not to pass it. This documentation could also help in\n> the documentation here so that the user can make a more informed\n> decision.\n\nI think there are multiple reasons that you may or may not want to pass\nthat flag. Certainly if you're not using disjoint packs (and instead\nonly care about single-pack verbatim reuse over the MIDX's preferred\npackfile), then you don't need to pass it.\n\nBut if you are using disjoint packs, you may want to pass it if you are\nadding packs to the MIDX which are disjoint, _and_ you want to hold onto\nthe existing set of disjoint packs.\n\nBut if you want to change the set of disjoint packs entirely, you would\nwant to omit this flag (unless you knew a-priori that you were going to\ndrop all of the currently marked disjoint packs from the new MIDX you\nare writing, e.g. with --stdin-packs).\n\nIf you think it would be useful, I could try and distill some of this\ndown, but I think that there is likely too much detail here for it to be\nuseful in user-facing documentation.\n\nThanks,\nTaylor\n"},{"id":"485285","messageId":"ZWjjSOJHw6Q1qQ+y@nand.local","threadId":"60565","inReplyTo":"ZWhhkdnVZ9w7tDBv@tanuki","subject":"Re: [PATCH 08/24] pack-objects: implement `--ignore-disjoint` mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-30T19:32:24Z","receivedAt":"2023-11-30T19:32:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Nov 30, 2023 at 11:18:57AM +0100, Patrick Steinhardt wrote:\n> > Instead, teach `pack-objects` a special `--ignore-disjoint` which is the\n> > moral equivalent of marking the set of disjoint packs as kept, and\n> > ignoring their contents, even if it would have otherwise been packed. In\n> > fact, this similarity extends down to the implementation, where each\n> > disjoint pack is first loaded, then has its `pack_keep_in_core` bit set.\n> >\n> > With this in place, we can use the kept-pack cache from 20b031fede\n> > (packfile: add kept-pack cache for find_kept_pack_entry(), 2021-02-22),\n> > which looks up objects first in a cache containing just the set of kept\n> > (in this case, disjoint) packs. Assuming that the set of disjoint packs\n> > is a relatively small portion of the entire repository (which should be\n> > a safe assumption to make), each object lookup will be very inexpensive.\n>\n> This cought me by surprise a bit. I'd have expected that in the end,\n> most of the packfiles in a repository would be disjoint. Using for\n> example geometric repacks, my expectation was that all of the packs that\n> get written via geometric repacking would eventually become disjoint\n> whereas new packs added to the repository would initially not be.\n\nWhich part are you referring to here? If you're referring to the part\nwhere I say that the set of disjoint packs is relatively small in\nproposition to the rest of the packs, I think I know where the confusion\nis.\n\nI'm not saying that the set of disjoint packs is small in comparison to\nthe rest of the repository by object count, but rather by count of packs\noverall. You're right that packs from pushes will not be guaranteed to\nbe disjoint upon entering the repository, but will become disjoint when\ngeometrically repacked (assuming that the caller uses --ignore-disjoint\nwhen repacking).\n\nThanks,\nTaylor\n"},{"id":"485287","messageId":"ZWjk/TYzsrdHefgy@nand.local","threadId":"60565","inReplyTo":"ZWhha2h2zzZWCXrw@tanuki","subject":"Re: [PATCH 00/24] pack-objects: multi-pack verbatim reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-11-30T19:39:41Z","receivedAt":"2023-11-30T19:39:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Nov 30, 2023 at 11:18:19AM +0100, Patrick Steinhardt wrote:\n> > Performing verbatim pack reuse naturally trades off between CPU time and\n> > the resulting pack size. In the above example, the single-pack reuse\n> > case produces a clone size of ~194 MB on my machine, while the\n> > multi-pack reuse case produces a clone size closer to ~266 MB, which is\n> > a ~37% increase in clone size.\n>\n> Quite exciting, and a tradeoff that may be worth it for Git hosters. I\n> expect that this is going to be an extreme example of the benefits\n> provided by your patch series -- do you by any chance also have \"real\"\n> numbers that make it possible to quantify the effect a bit better?\n>\n> No worry if you don't, I'm just curious.\n\nI don't have a great sense, no. I haven't run these patches yet in\nproduction, although would like to do so soon for internal repositories\nto get a better sense here.\n\nThere are some performance tests at the end which try and give you a\nsense of at least the relative speed-up depending on how many disjoint\npacks you have (IIRC, we test for 1, 10, and 100 disjoint packs).\n\n> > I think there is still some opportunity to close this gap, since the\n> > \"packing\" strategy here is extremely naive. In a production setting, I'm\n> > sure that there are more well thought out repacking strategies that\n> > would produce more similar clone sizes.\n> >\n> > I considered breaking this series up into smaller chunks, but was\n> > unsatisfied with the result. Since this series is rather large, if you\n> > have alternate suggestions on better ways to structure this, please let\n> > me know.\n>\n> The series is indeed very involved to review. I only made it up to patch\n> 8/24 and already spent quite some time on it. So I'd certainly welcome\n> it if this was split up into smaller parts, but don't have a suggestion\n> as to how this should be done (also because I didn't yet read the other\n> 16 patches).\n\nI suppose that one way to break it up might be:\n\n pack-objects: free packing_data in more places\n pack-bitmap-write: deep-clear the `bb_commit` slab\n pack-bitmap: plug leak in find_objects()\n\n midx: factor out `fill_pack_info()`\n midx: implement `DISP` chunk\n midx: implement `midx_locate_pack()`\n midx: implement `--retain-disjoint` mode\n\n pack-objects: implement `--ignore-disjoint` mode\n repack: implement `--extend-disjoint` mode\n\n pack-bitmap: pass `bitmapped_pack` struct to pack-reuse functions\n pack-bitmap: simplify `reuse_partial_packfile_from_bitmap()` signature\n pack-bitmap: return multiple packs via `reuse_partial_packfile_from_bitmap()`\n pack-objects: parameterize pack-reuse routines over a single pack\n pack-objects: keep track of `pack_start` for each reuse pack\n pack-objects: pass `bitmapped_pack`'s to pack-reuse functions\n pack-objects: prepare `write_reused_pack()` for multi-pack reuse\n pack-objects: prepare `write_reused_pack_verbatim()` for multi-pack reuse\n pack-objects: include number of packs reused in output\n pack-bitmap: prepare to mark objects from multiple packs for reuse\n\n pack-objects: add tracing for various packfile metrics\n t/test-lib-functions.sh: implement `test_trace2_data` helper\n pack-objects: allow setting `pack.allowPackReuse` to \"single\"\n pack-bitmap: reuse objects from all disjoint packs\n t/perf: add performance tests for multi-pack reuse\n\nThen you'd have five patch series, where each series does roughly the\nfollowing:\n\n 1. Preparatory clean-up.\n 2. Implementing the DISP chunk, as well as --retain-disjoint, without\n a way to generate such packs.\n 3. Implement a way to generate such packs, but without actually being\n able to reuse more than one of them.\n 4. Implement multi-pack reuse, but without actually reusing any packs.\n 5. Enable multi-pack reuse (and implement the required scaffolding to\n do so), and test it.\n\nThat's the most sensible split that I could come up with, at least. But\nI still find it relatively unsatisfying for a couple of reasons:\n\n - With the exception of the last group of patches, none of the earlier\n series enable any new, useful behavior on their own. IOW, if we just\n merged the first three series and then forgot about this topic, we\n wouldn't have done anything useful ;-).\n\n - The fourth series (which actually implements multi-pack reuse) still\n remains the most complicated, and would likely be the most difficult\n to review. So I think you'd still have one difficult series to\n review, plus four other series which are slightly less difficult to\n review ;-).\n\nIt's entirely possible that I'm just too close to these patches to see a\nbetter split, so if you (or others) have any suggestions on how to break\nthis up, please don't hesitate to share them.\n\n> I'll review the remaining patches at a later point in time.\n\nThanks, I'll look forward to more of your review as usual :-).\n\nThanks,\nTaylor\n"},{"id":"485302","messageId":"ZWmTAO_OfR_g3-3p@tanuki","threadId":"60565","inReplyTo":"ZWjisIgiuziMvBph@nand.local","subject":"Re: [PATCH 07/24] midx: implement `--retain-disjoint` mode","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-12-01T08:02:08Z","receivedAt":"2023-12-01T08:02:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Nov 30, 2023 at 02:29:52PM -0500, Taylor Blau wrote:\n> On Thu, Nov 30, 2023 at 11:18:51AM +0100, Patrick Steinhardt wrote:\n> > > diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> > > index d130e65b28..ac0c7b124b 100644\n> > > --- a/Documentation/git-multi-pack-index.txt\n> > > +++ b/Documentation/git-multi-pack-index.txt\n> > > @@ -54,6 +54,14 @@ write::\n> > > \t\t\"disjoint\". See the \"`DISP` chunk and disjoint packs\"\n> > > \t\tsection in linkgit:gitformat-pack[5] for more.\n> > >\n> > > +\t--retain-disjoint::\n> > > +\t\tWhen writing a multi-pack index with a reachability\n> > > +\t\tbitmap, keep any packs marked as disjoint in the\n> > > +\t\texisting MIDX (if any) as such in the new MIDX. Existing\n> > > +\t\tdisjoint packs which are removed (e.g., not listed via\n> > > +\t\t`--stdin-packs`) are ignored. This option works in\n> > > +\t\taddition to the '+' marker for `--stdin-packs`.\n> >\n> > I'm trying to understand when you're expected to pass this flag and when\n> > you're expected not to pass it. This documentation could also help in\n> > the documentation here so that the user can make a more informed\n> > decision.\n> \n> I think there are multiple reasons that you may or may not want to pass\n> that flag. Certainly if you're not using disjoint packs (and instead\n> only care about single-pack verbatim reuse over the MIDX's preferred\n> packfile), then you don't need to pass it.\n> \n> But if you are using disjoint packs, you may want to pass it if you are\n> adding packs to the MIDX which are disjoint, _and_ you want to hold onto\n> the existing set of disjoint packs.\n> \n> But if you want to change the set of disjoint packs entirely, you would\n> want to omit this flag (unless you knew a-priori that you were going to\n> drop all of the currently marked disjoint packs from the new MIDX you\n> are writing, e.g. with --stdin-packs).\n> \n> If you think it would be useful, I could try and distill some of this\n> down, but I think that there is likely too much detail here for it to be\n> useful in user-facing documentation.\n\nYeah, this indeed feels too detailed to be added here. I was hoping for\na simple \"Never do this if\"-style rule that points out why it is unwise\nunder some circumstances, but seems like it's not as simple as that.\n\nWell, so be it. Thanks!\n\nPatrick\n"},{"id":"485303","messageId":"ZWmWojF6BlOTzkcc@tanuki","threadId":"60565","inReplyTo":"ZWjjSOJHw6Q1qQ+y@nand.local","subject":"Re: [PATCH 08/24] pack-objects: implement `--ignore-disjoint` mode","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-12-01T08:17:38Z","receivedAt":"2023-12-01T08:17:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Nov 30, 2023 at 02:32:24PM -0500, Taylor Blau wrote:\n> On Thu, Nov 30, 2023 at 11:18:57AM +0100, Patrick Steinhardt wrote:\n> > > Instead, teach `pack-objects` a special `--ignore-disjoint` which is the\n> > > moral equivalent of marking the set of disjoint packs as kept, and\n> > > ignoring their contents, even if it would have otherwise been packed. In\n> > > fact, this similarity extends down to the implementation, where each\n> > > disjoint pack is first loaded, then has its `pack_keep_in_core` bit set.\n> > >\n> > > With this in place, we can use the kept-pack cache from 20b031fede\n> > > (packfile: add kept-pack cache for find_kept_pack_entry(), 2021-02-22),\n> > > which looks up objects first in a cache containing just the set of kept\n> > > (in this case, disjoint) packs. Assuming that the set of disjoint packs\n> > > is a relatively small portion of the entire repository (which should be\n> > > a safe assumption to make), each object lookup will be very inexpensive.\n> >\n> > This cought me by surprise a bit. I'd have expected that in the end,\n> > most of the packfiles in a repository would be disjoint. Using for\n> > example geometric repacks, my expectation was that all of the packs that\n> > get written via geometric repacking would eventually become disjoint\n> > whereas new packs added to the repository would initially not be.\n> \n> Which part are you referring to here? If you're referring to the part\n> where I say that the set of disjoint packs is relatively small in\n> proposition to the rest of the packs, I think I know where the confusion\n> is.\n\nYeah, that's what I was referring to.\n\n> I'm not saying that the set of disjoint packs is small in comparison to\n> the rest of the repository by object count, but rather by count of packs\n> overall. You're right that packs from pushes will not be guaranteed to\n> be disjoint upon entering the repository, but will become disjoint when\n> geometrically repacked (assuming that the caller uses --ignore-disjoint\n> when repacking).\n\nI was actually thinking about it in the number of packfiles, not number\nof objects. I'm mostly coming from the angle of geometric repacking\nhere, where it is totally expected that you have a comparatively large\nnumber of packfiles when your repository is big. With a geometric factor\nof 2, you'll have up to `log2($numobjects)` many packfiles in your repo\nwhile keeping the geometric sequence intact.\n\nIn something like linux.git with almost 10M objects that boils down to\n23 packfiles, and I'd assume that all of these would be disjoint in the\nbest case. So if you gain new packfiles by pushing into the repository\nthen I'd think that it's quite likely that the number of non-disjoint\npackfiles is smaller than the number of disjoint ones.\n\nI do realize though that in absolute numbers, this isn't all that many.\nI was also thinking ahead though to a future where we have something\nlike geometric repacking with maximum packfile sizes working well\ntogether so that we'll be able to merge packfiles together until they\nreach a certain maximum size, and afterwards they are just left alone.\nThis would help to avoid those \"surprise\" repack cases where everything\nis again packed into a single packfile for the biggest repositories out\nthere. But it would of course also lead to an increase in packfiles in\nthose huge repositories.\n\nAnyway, I feel like I'm rambling. In the end it's probably going to be\nfine, I was simply surprised by your assumption that the number of\ndisjoint packfiles should usually be much smaller than the number of\nnon-disjoint ones.\n\nPatrick\n"},{"id":"485304","messageId":"ZWmZ0pk6fr-8dmF1@tanuki","threadId":"60565","inReplyTo":"ZWjk/TYzsrdHefgy@nand.local","subject":"Re: [PATCH 00/24] pack-objects: multi-pack verbatim reuse","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-12-01T08:31:14Z","receivedAt":"2023-12-01T08:31:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Nov 30, 2023 at 02:39:41PM -0500, Taylor Blau wrote:\n> On Thu, Nov 30, 2023 at 11:18:19AM +0100, Patrick Steinhardt wrote:\n[snip]\n> Then you'd have five patch series, where each series does roughly the\n> following:\n> \n> 1. Preparatory clean-up.\n> 2. Implementing the DISP chunk, as well as --retain-disjoint, without\n> a way to generate such packs.\n> 3. Implement a way to generate such packs, but without actually being\n> able to reuse more than one of them.\n> 4. Implement multi-pack reuse, but without actually reusing any packs.\n> 5. Enable multi-pack reuse (and implement the required scaffolding to\n> do so), and test it.\n> \n> That's the most sensible split that I could come up with, at least.\n\nLooks sensible to me.\n\n> But\n> I still find it relatively unsatisfying for a couple of reasons:\n> \n> - With the exception of the last group of patches, none of the earlier\n> series enable any new, useful behavior on their own. IOW, if we just\n> merged the first three series and then forgot about this topic, we\n> wouldn't have done anything useful ;-).\n\nWell, sometimes I wish we'd buy more into the iterative style of working\nin the Git project, where it's fine to land patch series that only work\ninto the direction of a specific topic but don't yet do anything\ninteresting by themselves. The benefits would be both that individual\npieces land faster while also ensuring that the review load is kept at\nbay.\n\nBut there's of course also downsides to this, especially in an open\nsource project like Git:\n\n - Contributors may go away in the middle of their endeavour, leaving\n behind dangling pieces that might have complicated some of our\n architecture without actually reaping the intended benefits.\n\n - Overall, it may take significantly longer to get all pieces into\n Git.\n\n - Contributors need to do a much better job to explain where they are\n headed when the series by itself doesn't do anything interesting\n yet.\n\nSo I understand why we typically don't work this way.\n\nI wonder whether a compromise would be to continue sending complete\npatch series, but explicitly point out \"break points\" in your patch\nseries. These break points could be an indicator to the maintainer that\nit's fine to merge everything up to it while still keeping out the\nremainder of the patch series.\n\n> - The fourth series (which actually implements multi-pack reuse) still\n> remains the most complicated, and would likely be the most difficult\n> to review. So I think you'd still have one difficult series to\n> review, plus four other series which are slightly less difficult to\n> review ;-).\n\nThat's fine in my opinion, there's no surprise here that a complicated\ntopic is demanding for both the author and the reviewer.\n\nPatrick\n"},{"id":"485307","messageId":"ZWo69V0sNbQCBEeD@nand.local","threadId":"60565","inReplyTo":"ZWmWojF6BlOTzkcc@tanuki","subject":"Re: [PATCH 08/24] pack-objects: implement `--ignore-disjoint` mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-01T19:58:45Z","receivedAt":"2023-12-01T19:58:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Dec 01, 2023 at 09:17:38AM +0100, Patrick Steinhardt wrote:\n> In something like linux.git with almost 10M objects that boils down to\n> 23 packfiles, and I'd assume that all of these would be disjoint in the\n> best case. So if you gain new packfiles by pushing into the repository\n> then I'd think that it's quite likely that the number of non-disjoint\n> packfiles is smaller than the number of disjoint ones.\n\nRight, although if you have 10M objects over 23 packs with a geometric\nrepacking factor of two, the last pack should have just around a single\nobject in it. In other words, as soon as you receive a push, your\ngeometric progression will collapse into a single pack.\n\nSo having a repository with 10M objects split across 23 packs is a\nrelatively short-lived state. And in general we should only be in that\nstate every time a repository doubles (again, assuming a factor of two).\n\nIn that sense, I'd expect relatively few packs to be disjoint, and for\neach of those packs to have a relatively large number of objects,\naccounting for most of the non-recent parts of the repository.\n\nThanks,\nTaylor\n"},{"id":"485308","messageId":"ZWo7zjs+q8SZ5o1o@nand.local","threadId":"60565","inReplyTo":"ZWmZ0pk6fr-8dmF1@tanuki","subject":"Re: [PATCH 00/24] pack-objects: multi-pack verbatim reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-01T20:02:22Z","receivedAt":"2023-12-01T20:02:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Dec 01, 2023 at 09:31:14AM +0100, Patrick Steinhardt wrote:\n> > But\n> > I still find it relatively unsatisfying for a couple of reasons:\n> >\n> > - With the exception of the last group of patches, none of the earlier\n> > series enable any new, useful behavior on their own. IOW, if we just\n> > merged the first three series and then forgot about this topic, we\n> > wouldn't have done anything useful ;-).\n>\n> Well, sometimes I wish we'd buy more into the iterative style of working\n> in the Git project, where it's fine to land patch series that only work\n> into the direction of a specific topic but don't yet do anything\n> interesting by themselves. The benefits would be both that individual\n> pieces land faster while also ensuring that the review load is kept at\n> bay.\n>\n> But there's of course also downsides to this, especially in an open\n> source project like Git:\n\nI tend to agree with the downsides you list. My biggest concern with\nthis series in particular is that we're trying to break down an\nall-or-nothing change into smaller components. So if we landed four out\nof the five of those series, it would be better to have landed none of\nthem, since the first four aren't really all that useful on their own.\n\nI suppose if we're relatively confident that the last series will be\nmerged eventually, then that seems like less of a concern. But I'm not\nsure that we're at that point yet.\n\n> I wonder whether a compromise would be to continue sending complete\n> patch series, but explicitly point out \"break points\" in your patch\n> series. These break points could be an indicator to the maintainer that\n> it's fine to merge everything up to it while still keeping out the\n> remainder of the patch series.\n\nI think that's a reasonable alternative. It does create a minor headache\nfor the maintainer[^1], though, so I'd like to avoid it if possible.\n\n> > - The fourth series (which actually implements multi-pack reuse) still\n> > remains the most complicated, and would likely be the most difficult\n> > to review. So I think you'd still have one difficult series to\n> > review, plus four other series which are slightly less difficult to\n> > review ;-).\n>\n> That's fine in my opinion, there's no surprise here that a complicated\n> topic is demanding for both the author and the reviewer.\n\nMy preference is to avoid splitting the series if we can help it. But if\nyou feel strongly, or others feel similarly, I'm happy to take another\ncrack at breaking it up. Thanks for all of your review so far!\n\nThanks,\nTaylor\n"},{"id":"485335","messageId":"xmqqjzpv4ecg.fsf@gitster.g","threadId":"60565","inReplyTo":"c52d7e7b27a9add4f58b8334db4fe4498af1c90f.1701198172.git.me@ttaylorr.com","subject":"Re: [PATCH 05/24] midx: implement `DISP` chunk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-12-03T13:15:11Z","receivedAt":"2023-12-03T13:15:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> index 3696506eb3..d130e65b28 100644\n> --- a/Documentation/git-multi-pack-index.txt\n> +++ b/Documentation/git-multi-pack-index.txt\n> @@ -49,6 +49,10 @@ write::\n> \t--stdin-packs::\n> \t\tWrite a multi-pack index containing only the set of\n> \t\tline-delimited pack index basenames provided over stdin.\n> +\t\tLines beginning with a '+' character (followed by the\n> +\t\tpack index basename as before) have their pack marked as\n> +\t\t\"disjoint\". See the \"`DISP` chunk and disjoint packs\"\n> +\t\tsection in linkgit:gitformat-pack[5] for more.\n\nMakes one wonder who computes the set of packfiles, decides to\nprefix '+' to which ones, and how it does so, none of which appear\nin this step (which is understandable). As the flow of information\nis from the producer of individual \"disjoint\" packs (not in this\nstep) to this new logic in \"--stdin-packs\" to the new \"DISP\" chunk\nwriter (the primary focus of this step) to the final consumer of\n\"DISP\" chunk (not in this step), we are digging from the middle\n(hopefully to both directions in other steps). It is probably the\neasiest way to explain the idea to start from the primary data\nstructures and \"DISP\" seems to be a good place to start.\n\n> +\t Two packs are \"disjoint\" with respect to one another when they have\n> +\t disjoint sets of objects.\n> + In other words, any object found in a pack\n> +\t contained in the set of disjoint packfiles is guaranteed to be\n> +\t uniquely located among those packs.\n\nI often advise people to rethink what they wrote _before_ \"In other\nwords\", because the use of that phrase is a sign that the author\nconsiders the statement is hard to grok and needs rephrasing, in\nwhich case, the rephrased version may be a better way to explain the\nconcept being presented without the harder-to-grok version.\n\nBut I do not think this one is a good example to apply the advice.\nIt is because \"In other words,\" is somewhat misused in the sentence.\nTwo \"disjoint\" packs do not store any common object (which is how\nyou defined the adjective \"disjoint\" in the first sentence). \"As a\nconsequence\"/\"Hence\", an object found in one pack among many\n\"disjoint\" packs will not appear in others.\n\nBy the way, how strict does this disjointness have to be?\n\nLet's examine an extreme case. When you have two packs that are\n\"mostly\" disjoint, but have one single object in common, how would\nthat object interfere with the bulk streaming of existing packdata\nout of these two packs? Would we be able to, say, safely pretend\nthat the problematic single object lives only in one but not in the\nother (in other words, can we safely \"ignore\" the presence of the\ncopy in the other pack)? I think it would break down if that\nignored copy is used as a delta base of another object in the same\npack, and the base object for the delta is recorded as OFS_DELTA\n(which most likely every delta is these days), because we no longer\ncan stream out such deltified object without re-pointing its base to\nthe other copy, which will in turn screw up the relative offset of\nother objects in the same stream.\n\nOK, so it seems they really need to be strictly disjoint in order to\nparticipate in the reuse of the existing packdata. \n\n> +When a chunk of bytes are reused from an existing pack, any objects\n> +contained therein do not need to be added to the packing list, saving\n> +memory and CPU time. But a chunk from an existing packfile can only be\n> +reused when the following conditions are met:\n> +\n> + - The chunk contains only objects which were requested by the caller\n> + (i.e. does not contain any objects which the caller didn't ask for\n> + explicitly or implicitly).\n\nOK.\n\n> + - All objects stored as offset- or reference-deltas also include their\n> + base object in the resulting pack.\n\nAre thin packs obsolete?\n\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> index c4c6060cee..fd24e0c952 100755\n> --- a/t/t5319-multi-pack-index.sh\n> +++ b/t/t5319-multi-pack-index.sh\n> @@ -1157,4 +1157,62 @@ test_expect_success 'reader notices too-small revindex chunk' '\n> \ttest_cmp expect.err err\n> '\n> \n> +test_expect_success 'disjoint packs are stored via the DISP chunk' '\n> +\ttest_when_finished \"rm -fr repo\" &&\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\n> +\t\tfor i in 1 2 3 4 5\n> +\t\tdo\n> +\t\t\ttest_commit \"$i\" &&\n> +\t\t\tgit repack -d || return 1\n> +\t\tdone &&\n> +\n> +\t\tfind $objdir/pack -type f -name \"*.idx\" | xargs -n 1 basename | sort >packs &&\n\nThat is an overly-long line.\n\n> +test_expect_success 'non-disjoint packs are detected' '\n> +\ttest_when_finished \"rm -fr repo\" &&\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\n> +\t\ttest_commit base &&\n> +\t\tgit repack -d &&\n> +\t\ttest_commit other &&\n> +\t\tgit repack -a &&\n> +\n> +\t\tls -la .git/objects/pack/ &&\n\nIs this line a leftover debugging aid?\n\n> +\t\tfind $objdir/pack -type f -name \"*.idx\" |\n> +\t\t\tsed -e \"s/.*\\/\\(.*\\)$/+\\1/g\" >in &&\n\nLose \"g\"; it adds unnecessary cognitive burden to the readers if the\npatterh is expected to match multiple times, and you know that is\nnot possible (your pattern is right anchored at the end). This may\napply equally to other uses of \"sed\" in this patch.\n\n> +\t\ttest_must_fail git multi-pack-index write --stdin-packs \\\n> +\t\t\t--bitmap err &&\n> +\t\tgrep \"duplicate object.* among disjoint packs\" err\n> +\t)\n> +'\n> +\n> test_done\n"},{"id":"485346","messageId":"ZW2SpFbBIq6p1Myu@tanuki","threadId":"60565","inReplyTo":"ZWo7zjs+q8SZ5o1o@nand.local","subject":"Re: [PATCH 00/24] pack-objects: multi-pack verbatim reuse","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-12-04T08:49:40Z","receivedAt":"2023-12-04T08:49:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Dec 01, 2023 at 03:02:22PM -0500, Taylor Blau wrote:\n> On Fri, Dec 01, 2023 at 09:31:14AM +0100, Patrick Steinhardt wrote:\n[snip]\n> I suppose if we're relatively confident that the last series will be\n> merged eventually, then that seems like less of a concern. But I'm not\n> sure that we're at that point yet.\n\nThat's an additional valid concern indeed.\n\n[snip]\n> > > - The fourth series (which actually implements multi-pack reuse) still\n> > > remains the most complicated, and would likely be the most difficult\n> > > to review. So I think you'd still have one difficult series to\n> > > review, plus four other series which are slightly less difficult to\n> > > review ;-).\n> >\n> > That's fine in my opinion, there's no surprise here that a complicated\n> > topic is demanding for both the author and the reviewer.\n> \n> My preference is to avoid splitting the series if we can help it. But if\n> you feel strongly, or others feel similarly, I'm happy to take another\n> crack at breaking it up. Thanks for all of your review so far!\n\nI don't feel strongly about this at all, I've only tried to spell out my\nown thoughts in this context as I thought they were kind of relevant\nhere. I've thought quite a lot about this topic recently due to my work\non the reftable backend, where I'm trying to get as many pieces as\npossible landed individually before landing the actual backend itself.\nIt's working well for most of the part, but in other contexts it's a bit\nweird that we try to cater towards something that doesn't exist yet. But\nnaturally, the reftable work is of different nature than the topic you\nwork on here and thus my own takeaways may likely not apply heer.\n\nTo summarize, I think there is merit in splitting up patches into chunks\nthat make it in individually and thus gradually work toward a topic, but\nI also totally understand why you (or Junio as the maintainer) might\nthink that this is not a good idea. The ultimate decision for how to\nhandle topics should be with the patch series' author and maintainer.\n\nPatrick\n"},{"id":"485357","messageId":"ZW95WSErCXvkfrAG@nand.local","threadId":"60565","inReplyTo":"xmqqjzpv4ecg.fsf@gitster.g","subject":"Re: [PATCH 05/24] midx: implement `DISP` chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-05T19:26:17Z","receivedAt":"2023-12-05T19:26:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Dec 03, 2023 at 10:15:11PM +0900, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> > index 3696506eb3..d130e65b28 100644\n> > --- a/Documentation/git-multi-pack-index.txt\n> > +++ b/Documentation/git-multi-pack-index.txt\n> > @@ -49,6 +49,10 @@ write::\n> > \t--stdin-packs::\n> > \t\tWrite a multi-pack index containing only the set of\n> > \t\tline-delimited pack index basenames provided over stdin.\n> > +\t\tLines beginning with a '+' character (followed by the\n> > +\t\tpack index basename as before) have their pack marked as\n> > +\t\t\"disjoint\". See the \"`DISP` chunk and disjoint packs\"\n> > +\t\tsection in linkgit:gitformat-pack[5] for more.\n>\n> Makes one wonder who computes the set of packfiles, decides to\n> prefix '+' to which ones, and how it does so, none of which appear\n> in this step (which is understandable). As the flow of information\n> is from the producer of individual \"disjoint\" packs (not in this\n> step) to this new logic in \"--stdin-packs\" to the new \"DISP\" chunk\n> writer (the primary focus of this step) to the final consumer of\n> \"DISP\" chunk (not in this step), we are digging from the middle\n> (hopefully to both directions in other steps). It is probably the\n> easiest way to explain the idea to start from the primary data\n> structures and \"DISP\" seems to be a good place to start.\n\nThanks. I found that laying out this series was rather tricky, since all\nof the individual pieces really depend on the end state in order to make\nany sense.\n\nHopefully you're satisfied with the way things are split up and\norganized currently, but if you have suggestions on other ways I could\nslice or dice this, please let me know.\n\n> > +\t Two packs are \"disjoint\" with respect to one another when they have\n> > +\t disjoint sets of objects.\n> > + In other words, any object found in a pack\n> > +\t contained in the set of disjoint packfiles is guaranteed to be\n> > +\t uniquely located among those packs.\n>\n> I often advise people to rethink what they wrote _before_ \"In other\n> words\", because the use of that phrase is a sign that the author\n> considers the statement is hard to grok and needs rephrasing, in\n> which case, the rephrased version may be a better way to explain the\n> concept being presented without the harder-to-grok version.\n>\n> But I do not think this one is a good example to apply the advice.\n> It is because \"In other words,\" is somewhat misused in the sentence.\n> Two \"disjoint\" packs do not store any common object (which is how\n> you defined the adjective \"disjoint\" in the first sentence). \"As a\n> consequence\"/\"Hence\", an object found in one pack among many\n> \"disjoint\" packs will not appear in others.\n\nThanks, I'll replace this with \"As a consequence\", and try to follow\nthat general advice more often in the future ;-).\n\n> OK, so it seems they really need to be strictly disjoint in order to\n> participate in the reuse of the existing packdata.\n\nI think that's generally true, though there are some exceptions.\n\nI think the real condition here is that the *reused sections* must be\ndisjoint with respect to one another, not necessarily the packs\nthemselves. So having the packs be disjoint is a sufficient condition,\nsince we know that no matter which section(s) we reuse, they are\nguaranteed to be disjoint.\n\nI think that there is opportunity to be more clever here, e.g., by\nallowing for different disjoint \"groups\" of packs, or mandating that you\ncan only reuse certain sections from different combinations of packs in\norder to satisfy this property.\n\nThat's part of the reason why I left more space than is needed for the\n\"disjoint\" state in the DISP chunk (it is 32 bits, of which we're only\nusing one of them). I'm not sure that we would want more relaxed\nconstraints here, since they'd be harder to satisfy. But my hope is that\nwe would be able to learn from running this in production to figure out\nwhether or not such a thing would be useful.\n\n> > + - All objects stored as offset- or reference-deltas also include their\n> > + base object in the resulting pack.\n>\n> Are thin packs obsolete?\n\nNo, I think I should clarify this to make it more obvious that this only\napplies to non-thin packs.\n\n> > +\t\tfind $objdir/pack -type f -name \"*.idx\" | xargs -n 1 basename | sort >packs &&\n>\n> That is an overly-long line.\n\nThanks for spotting.\n\n> > +test_expect_success 'non-disjoint packs are detected' '\n> > +\ttest_when_finished \"rm -fr repo\" &&\n> > +\tgit init repo &&\n> > +\t(\n> > +\t\tcd repo &&\n> > +\n> > +\t\ttest_commit base &&\n> > +\t\tgit repack -d &&\n> > +\t\ttest_commit other &&\n> > +\t\tgit repack -a &&\n> > +\n> > +\t\tls -la .git/objects/pack/ &&\n>\n> Is this line a leftover debugging aid?\n\nIndeed, thanks.\n\n> > +\t\tfind $objdir/pack -type f -name \"*.idx\" |\n> > +\t\t\tsed -e \"s/.*\\/\\(.*\\)$/+\\1/g\" >in &&\n>\n> Lose \"g\"; it adds unnecessary cognitive burden to the readers if the\n> patterh is expected to match multiple times, and you know that is\n> not possible (your pattern is right anchored at the end). This may\n> apply equally to other uses of \"sed\" in this patch.\n\nThanks, I dropped the 'g' in both instances.\n\nThanks,\nTaylor\n"},{"id":"485447","messageId":"ZXHE5Lce_6CAWKFT@tanuki","threadId":"60565","inReplyTo":"b75869befba26899d88d6c6d413cc756aeadbd80.1701198172.git.me@ttaylorr.com","subject":"Re: [PATCH 09/24] repack: implement `--extend-disjoint` mode","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-12-07T13:13:08Z","receivedAt":"2023-12-07T13:13:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 28, 2023 at 02:08:18PM -0500, Taylor Blau wrote:\n> Now that we can generate packs which are disjoint with respect to the\n> set of currently-disjoint packs, implement a mode of `git repack` which\n> extends the set of disjoint packs with any new (non-cruft) pack(s)\n> generated during the repack.\n> \n> The idea is mostly straightforward, with a couple of gotcha's. The\n> straightforward part is to make sure that any new packs are disjoint\n> with respect to the set of currently disjoint packs which are _not_\n> being removed from the repository as a result of the repack.\n> \n> If a pack which is currently marked as disjoint is, on the other hand,\n> about to be removed from the repository, it is OK (and expected) that\n> new pack(s) will contain some or all of its objects. Since the pack\n> originally marked as disjoint will be removed, it will necessarily leave\n> the disjoint set, making room for new packs with its same objects to\n> take its place. In other words, the resulting set of disjoint packs will\n> be disjoint with respect to one another.\n> \n> The gotchas mostly have to do with making sure that we do not generate a\n> disjoint pack in the following scenarios:\n\nOkay, let me verify whether I understand the reasons:\n\n> - promisor packs\n\nWhich is because promisor packs actually don't contain any objects?\n\n> - cruft packs (which may necessarily need to include an object from a\n> disjoint pack in order to freshen it in certain circumstances)\n\nThis one took me a while to figure out. If we'd mark crufts as disjoint,\nthen it would mean that new packfiles cannot be marked as disjoint if\nobjects which were previously unreachable do become reachable again.\nSo we'd be pessimizing packfiles for live objects in favor of others\nwhich aren't.\n\n> - all-into-one repacks without '-d'\n\nBecause here the old packfiles that this would make redundant aren't\ndeleted and thus the objects are duplicate now.\n\n> - `--filter-to`, which conceptually could work with the new\n> `--extend-disjoint` option, but only in limited circumstances\n\nWe're probably also not properly set up to handle the new alternate\nobject directory and exclude objects that are part of a potentially\ndisjoint packfile that exists already. Also, the current MIDX may not\neven cover the alternate.\n\n> Otherwise, we mark which packs were created as disjoint by using a new\n> bit in the `generated_pack_data` struct, and then marking those pack(s)\n> as disjoint accordingly when generating the MIDX. Non-deleted packs\n> which are marked as disjoint are retained as such by passing the\n> equivalent of `--retain-disjoint` when calling the MIDX API to update\n> the MIDX.\n\nOkay. I had a bit of trouble to sift through the various different\nflags like \"--retain-disjoint\", \"--extend-disjoint\", \"--ignore-disjoint\"\nand so on. But well, they do different things and it's been a few days\nsince I've reviewed the preceding patches, so this is probably fine.\n\nOne thing I wondered: do we need to consider the `-l` flag? When using\nan alternate object directory it is totally feasible that the alternate\nmay be creating new disjoint packages without us knowing, and thus we\nmay not be able to guarantee the disjoint property anymore.\n\nPatrick\n"},{"id":"485446","messageId":"ZXHE6Ym3CICtNxFd@tanuki","threadId":"60565","inReplyTo":"970bd9eaeae038adb6e7d4c399c9b033668a8864.1701198172.git.me@ttaylorr.com","subject":"Re: [PATCH 10/24] pack-bitmap: pass `bitmapped_pack` struct to pack-reuse functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-12-07T13:13:13Z","receivedAt":"2023-12-07T13:13:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 28, 2023 at 02:08:21PM -0500, Taylor Blau wrote:\n[snip]\n> @@ -2002,6 +1986,65 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n> \n> done:\n> \tunuse_pack(&w_curs);\n> +}\n> +\n> +int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n> +\t\t\t\t struct packed_git **packfile_out,\n> +\t\t\t\t uint32_t *entries,\n> +\t\t\t\t struct bitmap **reuse_out)\n> +{\n> +\tstruct repository *r = the_repository;\n> +\tstruct bitmapped_pack *packs = NULL;\n> +\tstruct bitmap *result = bitmap_git->result;\n> +\tstruct bitmap *reuse;\n> +\tsize_t i;\n> +\tsize_t packs_nr = 0, packs_alloc = 0;\n> +\tsize_t word_alloc;\n> +\tuint32_t objects_nr = 0;\n> +\n> +\tassert(result);\n> +\n> +\tload_reverse_index(r, bitmap_git);\n> +\n> +\tif (bitmap_is_midx(bitmap_git)) {\n> +\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n> +\t\t\tstruct bitmapped_pack pack;\n> +\t\t\tif (nth_bitmapped_pack(r, bitmap_git->midx, &pack, i) < 0) {\n> +\t\t\t\twarning(_(\"unable to load pack: '%s', disabling pack-reuse\"),\n> +\t\t\t\t\tbitmap_git->midx->pack_names[i]);\n> +\t\t\t\tfree(packs);\n> +\t\t\t\treturn -1;\n> +\t\t\t}\n> +\t\t\tif (!pack.bitmap_nr)\n> +\t\t\t\tcontinue; /* no objects from this pack */\n> +\t\t\tif (pack.bitmap_pos)\n> +\t\t\t\tcontinue; /* not preferred pack */\n> +\n> +\t\t\tALLOC_GROW(packs, packs_nr + 1, packs_alloc);\n> +\t\t\tmemcpy(&packs[packs_nr++], &pack, sizeof(pack));\n> +\n> +\t\t\tobjects_nr += pack.p->num_objects;\n> +\t\t}\n> +\t} else {\n> +\t\tALLOC_GROW(packs, packs_nr + 1, packs_alloc);\n> +\n> +\t\tpacks[packs_nr].p = bitmap_git->pack;\n> +\t\tpacks[packs_nr].bitmap_pos = 0;\n> +\t\tpacks[packs_nr].bitmap_nr = bitmap_git->pack->num_objects;\n> +\t\tpacks[packs_nr].disjoint = 1;\n> +\n> +\t\tobjects_nr = packs[packs_nr++].p->num_objects;\n> +\t}\n> +\n> +\tword_alloc = objects_nr / BITS_IN_EWORD;\n> +\tif (objects_nr % BITS_IN_EWORD)\n> +\t\tword_alloc++;\n> +\treuse = bitmap_word_alloc(word_alloc);\n> +\n> +\tif (packs_nr != 1)\n> +\t\tBUG(\"pack reuse not yet implemented for multiple packs\");\n\nCan't it happen that we have no pack here? In the MIDX-case we skip all\npacks that either do not have a bitmap or are not preferred. So does it\nmean that in reverse, every preferred packfile must have a a bitmap? I'd\nthink that to not be true in case bitmaps are turned off.\n\nPatrick\n"},{"id":"485448","messageId":"ZXHE7_KwukSRBET1@tanuki","threadId":"60565","inReplyTo":"432854b27c6731bd6ab1fa739b3a086ec0a90be8.1701198172.git.me@ttaylorr.com","subject":"Re: [PATCH 11/24] pack-bitmap: simplify `reuse_partial_packfile_from_bitmap()` signature","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-12-07T13:13:19Z","receivedAt":"2023-12-07T13:13:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 28, 2023 at 02:08:24PM -0500, Taylor Blau wrote:\n> The signature of `reuse_partial_packfile_from_bitmap()` currently takes\n> in a bitmap, as well as three output parameters (filled through\n> pointers, and passed as arguments), and also returns an integer result.\n> \n> The output parameters are filled out with: (a) the packfile used for\n> pack-reuse, (b) the number of objects from that pack that we can reuse,\n> and (c) a bitmap indicating which objects we can reuse. The return value\n> is either -1 (when there are no objects to reuse), or 0 (when there is\n> at least one object to reuse).\n> \n> Some of these parameters are redundant. Notably, we can infer from the\n> bitmap how many objects are reused by calling bitmap_popcount(). And we\n> can similar compute the return value based on that number as well.\n> \n> As such, clean up the signature of this function to drop the \"*entries\"\n> parameter, as well as the int return value, since the single caller of\n> this function can infer these values themself.\n> \n> Signed-off-by: Taylor Blau \n> ---\n> builtin/pack-objects.c | 16 +++++++++-------\n> pack-bitmap.c | 16 +++++++---------\n> pack-bitmap.h | 7 +++----\n> 3 files changed, 19 insertions(+), 20 deletions(-)\n> \n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 107154db34..2bb1b64e8f 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3946,13 +3946,15 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n> \tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n> \t\treturn -1;\n> \n> -\tif (pack_options_allow_reuse() &&\n> -\t !reuse_partial_packfile_from_bitmap(\n> -\t\t\tbitmap_git,\n> -\t\t\t&reuse_packfile,\n> -\t\t\t&reuse_packfile_objects,\n> -\t\t\t&reuse_packfile_bitmap)) {\n> -\t\tassert(reuse_packfile_objects);\n> +\tif (pack_options_allow_reuse())\n> +\t\treuse_partial_packfile_from_bitmap(bitmap_git, &reuse_packfile,\n> +\t\t\t\t\t\t &reuse_packfile_bitmap);\n> +\n> +\tif (reuse_packfile) {\n> +\t\treuse_packfile_objects = bitmap_popcount(reuse_packfile_bitmap);\n> +\t\tif (!reuse_packfile_objects)\n> +\t\t\tBUG(\"expected non-empty reuse bitmap\");\n\nWe're now re-computing `bitmap_popcount()` for the bitmap a second time.\nBut I really don't think this is ever going to be a problem in practice\ngiven that it only does a bunch of math. Any performance regression\nwould thus ultimately be drowned out by everything else.\n\nIn other words: this is probably fine.\n\nPatrick\n"},{"id":"485449","messageId":"ZXHE9L7iqXQAit_1@tanuki","threadId":"60565","inReplyTo":"6f4fba861b59f909148775ee64c3ba89afc872b5.1701198172.git.me@ttaylorr.com","subject":"Re: [PATCH 14/24] pack-objects: keep track of `pack_start` for each reuse pack","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-12-07T13:13:24Z","receivedAt":"2023-12-07T13:13:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 28, 2023 at 02:08:32PM -0500, Taylor Blau wrote:\n> When reusing objects from a pack, we keep track of a set of one or more\n> `reused_chunk`s, corresponding to sections of one or more object(s) from\n> a source pack that we are reusing. Each chunk contains two pieces of\n> information:\n> \n> - the offset of the first object in the source pack (relative to the\n> beginning of the source pack)\n> - the difference between that offset, and the corresponding offset in\n> the pack we're generating\n> \n> The purpose of keeping track of these is so that we can patch an\n> OFS_DELTAs that cross over a section of the reuse pack that we didn't\n> take.\n> \n> For instance, consider a hypothetical pack as shown below:\n> \n> (chunk #2)\n> __________...\n> /\n> /\n> +--------+---------+-------------------+---------+\n> ... | | | (unused) | | ...\n> +--------+---------+-------------------+---------+\n> \\ /\n> \\______________/\n> (chunk #1)\n> \n> Suppose that we are sending objects \"base\", \"other\", and \"delta\", and\n> that the \"delta\" object is stored as an OFS_DELTA, and that its base is\n> \"base\". If we don't send any objects in the \"(unused)\" range, we can't\n> copy the delta'd object directly, since its delta offset includes a\n> range of the pack that we didn't copy, so we have to account for that\n> difference when patching and reassembling the delta.\n> \n> In order to compute this value correctly, we need to know not only where\n> we are in the packfile we're assembling (with `hashfile_total(f)`) but\n> also the position of the first byte of the packfile that we are\n> currently reusing.\n> \n> Together, these two allow us to compute the reused chunk's offset\n> difference relative to the start of the reused pack, as desired.\n\nHm. I'm not quite sure I fully understand the motivation here. Is this\nsomething that was broken all along? Why does it become a problem now?\nSorry if I'm missing the obvious here.\n\n> Helped-by: Jeff King \n> Signed-off-by: Taylor Blau \n> ---\n> builtin/pack-objects.c | 11 ++++++++---\n> 1 file changed, 8 insertions(+), 3 deletions(-)\n> \n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 7682bd65bb..eb8be514d1 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -1016,6 +1016,7 @@ static off_t find_reused_offset(off_t where)\n> \n> static void write_reused_pack_one(struct packed_git *reuse_packfile,\n> \t\t\t\t size_t pos, struct hashfile *out,\n> +\t\t\t\t off_t pack_start,\n> \t\t\t\t struct pack_window **w_curs)\n> {\n> \toff_t offset, next, cur;\n> @@ -1025,7 +1026,8 @@ static void write_reused_pack_one(struct packed_git *reuse_packfile,\n> \toffset = pack_pos_to_offset(reuse_packfile, pos);\n> \tnext = pack_pos_to_offset(reuse_packfile, pos + 1);\n> \n> -\trecord_reused_object(offset, offset - hashfile_total(out));\n> +\trecord_reused_object(offset,\n> +\t\t\t offset - (hashfile_total(out) - pack_start));\n> \n> \tcur = offset;\n> \ttype = unpack_object_header(reuse_packfile, w_curs, &cur, &size);\n> @@ -1095,6 +1097,7 @@ static void write_reused_pack_one(struct packed_git *reuse_packfile,\n> \n> static size_t write_reused_pack_verbatim(struct packed_git *reuse_packfile,\n> \t\t\t\t\t struct hashfile *out,\n> +\t\t\t\t\t off_t pack_start UNUSED,\n> \t\t\t\t\t struct pack_window **w_curs)\n> {\n> \tsize_t pos = 0;\n> @@ -1126,10 +1129,12 @@ static void write_reused_pack(struct packed_git *reuse_packfile,\n> {\n> \tsize_t i = 0;\n> \tuint32_t offset;\n> +\toff_t pack_start = hashfile_total(f) - sizeof(struct pack_header);\n\nGiven that this patch in its current state doesn't seem to do anything\nyet, am I right in assuming that `hashfile_total(f) - sizeof(struct\npack_header)` is always expected to be zero for now?\n\nPatrick\n"},{"id":"485450","messageId":"ZXHE-cLrP7iRHWHY@tanuki","threadId":"60565","inReplyTo":"67a8196978244b56d4f60861f140b78c59d15e30.1701198172.git.me@ttaylorr.com","subject":"Re: [PATCH 16/24] pack-objects: prepare `write_reused_pack()` for multi-pack reuse","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-12-07T13:13:29Z","receivedAt":"2023-12-07T13:13:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 28, 2023 at 02:08:37PM -0500, Taylor Blau wrote:\n> The function `write_reused_pack()` within `builtin/pack-objects.c` is\n> responsible for performing pack-reuse on a single pack, and has two main\n> functions:\n> \n> - it dispatches a call to `write_reused_pack_verbatim()` to see if we\n> can reuse portions of the packfile in whole-word chunks\n> \n> - for any remaining objects (that is, any objects that appear after\n> the first \"gap\" in the bitmap), call write_reused_pack_one() on that\n> object to record it for reuse.\n> \n> Prepare this function for multi-pack reuse by removing the assumption\n> that the bit position corresponding to the first object being reused\n> from a given pack may not be at bit position zero.\n\nIs this double-negation intended? We remove the assumption that we start\nreading at position zero, but the paragraph here states that we remove\nthe assumption that we do _not_ start at bit zero.\n\nI'll stop reviewing here and will come back to this series somewhen next\nweek.\n\nPatrick\n"},{"id":"485451","messageId":"ZXHYetGa/e1VjwLn@nand.local","threadId":"60565","inReplyTo":"ZXHE7_KwukSRBET1@tanuki","subject":"Re: [PATCH 11/24] pack-bitmap: simplify `reuse_partial_packfile_from_bitmap()` signature","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-07T14:36:42Z","receivedAt":"2023-12-07T14:36:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Dec 07, 2023 at 02:13:19PM +0100, Patrick Steinhardt wrote:\n> > +\tif (reuse_packfile) {\n> > +\t\treuse_packfile_objects = bitmap_popcount(reuse_packfile_bitmap);\n> > +\t\tif (!reuse_packfile_objects)\n> > +\t\t\tBUG(\"expected non-empty reuse bitmap\");\n>\n> We're now re-computing `bitmap_popcount()` for the bitmap a second time.\n> But I really don't think this is ever going to be a problem in practice\n> given that it only does a bunch of math. Any performance regression\n> would thus ultimately be drowned out by everything else.\n>\n> In other words: this is probably fine.\n\nI definitely agree that any performance regression from calling\nbitmap_popcount() twice would be drowned out by the rest of what\npack-objects is doing.\n\nFor what it's worth:\n\n- The bitmap_popcount() call is a loop over ewah_bit_popcount64() for\n each of the allocated words. And the latter is more or less three\n copies of:\n\n b7:\t55 55 55\n ba:\t48 23 45 f8 \tand -0x8(%rbp),%rax\n be:\t48 8b 55 f8 \tmov -0x8(%rbp),%rdx\n c2:\t48 89 d1 \tmov %rdx,%rcx\n c5:\t48 d1 e9 \tshr %rcx\n c8:\t48 ba 55 55 55 55 55 \tmovabs $0x5555555555555555,%rdx\n cf:\t55 55 55\n d2:\t48 21 ca \tand %rcx,%rdx\n d5:\t48 01 d0 \tadd %rdx,%rax\n d8:\t48 89 45 f8 \tmov %rax,-0x8(%rbp)\n dc:\t48 b8 33 33 33 33 33 \tmovabs $0x3333333333333333,%rax\n\n Followed by:\n\n 144:\t48 0f af c2 \timul %rdx,%rax\n 148:\t48 c1 e8 38 \tshr $0x38,%rax\n 14c:\t5d \tpop %rbp\n 14d:\tc3 \tret\n\n With the usual x86 ABI preamble and postamble. So this should be an\n extremely cheap function to compute.\n\n- But, the earlier bitmap_popcount() call in\n reuse_partial_packfile_from_bitmap() is not necessary, since we only\n care whether or not there are _any_ bits set in the bitmap, not how\n many of them there are.\n\n So we could write something like `bitmap_empty(reuse)` instead, which\n would be much cheaper (again, not that I think we'll notice this\n either way, but throwing away the result of bitmap_popcount() and\n calling it twice does leave me a little unsatisfied).\n\nSo I think we could reasonably do something like:\n\n--- 8< ---\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex 7b525b1ecd..ac7e0af622 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -169,6 +169,15 @@ size_t bitmap_popcount(struct bitmap *self)\n \treturn count;\n }\n\n+int bitmap_is_empty(struct bitmap *self)\n+{\n+\tsize_t i;\n+\tfor (i = 0; i < self->word_alloc; i++)\n+\t\tif (self->words[i])\n+\t\t\treturn 0;\n+\treturn 1;\n+}\n+\n int bitmap_equals(struct bitmap *self, struct bitmap *other)\n {\n \tstruct bitmap *big, *small;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 7eb8b9b630..c11d76c6f3 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -189,5 +189,6 @@ void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other);\n void bitmap_or(struct bitmap *self, const struct bitmap *other);\n\n size_t bitmap_popcount(struct bitmap *self);\n+int bitmap_is_empty(struct bitmap *self);\n\n #endif\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 614fc09a4e..e50b322779 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -2045,7 +2045,7 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n\n \treuse_partial_packfile_from_bitmap_1(bitmap_git, packs, reuse);\n\n-\tif (!bitmap_popcount(reuse)) {\n+\tif (bitmap_is_empty(reuse)) {\n \t\tfree(packs);\n \t\tbitmap_free(reuse);\n \t\treturn;\n--- >8 ---\n\nThanks,\nTaylor\n"},{"id":"485452","messageId":"ZXIq4mjDUoqlGvgW@nand.local","threadId":"60565","inReplyTo":"ZXHE5Lce_6CAWKFT@tanuki","subject":"Re: [PATCH 09/24] repack: implement `--extend-disjoint` mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-07T20:28:18Z","receivedAt":"2023-12-07T20:28:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Dec 07, 2023 at 02:13:08PM +0100, Patrick Steinhardt wrote:\n> > The gotchas mostly have to do with making sure that we do not generate a\n> > disjoint pack in the following scenarios:\n>\n> Okay, let me verify whether I understand the reasons:\n>\n> > - promisor packs\n>\n> Which is because promisor packs actually don't contain any objects?\n\nRight.\n\n> > - cruft packs (which may necessarily need to include an object from a\n> > disjoint pack in order to freshen it in certain circumstances)\n>\n> This one took me a while to figure out. If we'd mark crufts as disjoint,\n> then it would mean that new packfiles cannot be marked as disjoint if\n> objects which were previously unreachable do become reachable again.\n> So we'd be pessimizing packfiles for live objects in favor of others\n> which aren't.\n\nYeah, that's right, too. There are a couple of cases where more than one\ncruft pack may contain the same object, one of them being the\nflip-flopping between reachable and unreachable as you suggest above.\nAnother is that you have a non-prunable unreachable object which is\nalready in a cruft pack. If the object's mtime gets updated (and still\ncannot be pruned), we'll end up freshening the object loose, and then\npacking it again (with the more recent mtime) into a new cruft pack.\n\nThat aside, I actually think that there are ways to mark cruft packs\ndisjoint. But they're complicated, and moreover, I don't think you'd\never *want* to mark a cruft pack as disjoint. Cruft packs usually\ncontain garbage, which is unlikely to be useful to any fetches/clones.\n\nIf we did mark them as disjoint, it would mean that we could reuse\nverbatim sections of the cruft pack in our output, but we would likely\nend up with very few such sections.\n\n> > - all-into-one repacks without '-d'\n>\n> Because here the old packfiles that this would make redundant aren't\n> deleted and thus the objects are duplicate now.\n\nYep.\n\n> > Otherwise, we mark which packs were created as disjoint by using a new\n> > bit in the `generated_pack_data` struct, and then marking those pack(s)\n> > as disjoint accordingly when generating the MIDX. Non-deleted packs\n> > which are marked as disjoint are retained as such by passing the\n> > equivalent of `--retain-disjoint` when calling the MIDX API to update\n> > the MIDX.\n>\n> Okay. I had a bit of trouble to sift through the various different\n> flags like \"--retain-disjoint\", \"--extend-disjoint\", \"--ignore-disjoint\"\n> and so on. But well, they do different things and it's been a few days\n> since I've reviewed the preceding patches, so this is probably fine.\n\nYeah, I am definitely open to better naming conventions here? I figured\nthat:\n\n - --retain-disjoint was a good name for the MIDX option, since it is\n retaining existing disjoint packs in the new MIDX\n - --extend-disjoint was a good name for the repack option, since it is\n extending the set of disjoint packs\n - --ignore-disjoint was a good name for the pack-objects option, since\n it is ignoring objects in disjoint packs\n\nWriting this out, I think that you could make an argument that\n`--exclude-disjoint` is a better name for the last option. So I'm\ndefinitely open to suggestions here, but I don't want to get too bogged\ndown on command-line option naming (so long as we're all reasonably\nhappy with the result).\n\n> One thing I wondered: do we need to consider the `-l` flag? When using\n> an alternate object directory it is totally feasible that the alternate\n> may be creating new disjoint packages without us knowing, and thus we\n> may not be able to guarantee the disjoint property anymore.\n\nI don't think so. We'd only care about one direction of this (that\nalternates do not create disjoint packs which overlap with ours, instead\nof the other way around), but since we don't put non-local packs in the\nMIDX, I think we're OK.\n\nI suppose that you might run into trouble if you use the chained MIDX\nthing (via its `->next` pointer). I haven't used that feature myself, so\nI'd have to play around with it.\n\nThanks,\nTaylor\n"},{"id":"485453","messageId":"ZXIsbO++u9n/yDYi@nand.local","threadId":"60565","inReplyTo":"ZXHE6Ym3CICtNxFd@tanuki","subject":"Re: [PATCH 10/24] pack-bitmap: pass `bitmapped_pack` struct to pack-reuse functions","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-07T20:34:52Z","receivedAt":"2023-12-07T20:34:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Dec 07, 2023 at 02:13:13PM +0100, Patrick Steinhardt wrote:\n> > +\tif (bitmap_is_midx(bitmap_git)) {\n> > +\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n> > +\t\t\tstruct bitmapped_pack pack;\n> > +\t\t\tif (nth_bitmapped_pack(r, bitmap_git->midx, &pack, i) < 0) {\n> > +\t\t\t\twarning(_(\"unable to load pack: '%s', disabling pack-reuse\"),\n> > +\t\t\t\t\tbitmap_git->midx->pack_names[i]);\n> > +\t\t\t\tfree(packs);\n> > +\t\t\t\treturn -1;\n> > +\t\t\t}\n> > +\t\t\tif (!pack.bitmap_nr)\n> > +\t\t\t\tcontinue; /* no objects from this pack */\n> > +\t\t\tif (pack.bitmap_pos)\n> > +\t\t\t\tcontinue; /* not preferred pack */\n> > +\n> > +\t\t\tALLOC_GROW(packs, packs_nr + 1, packs_alloc);\n> > +\t\t\tmemcpy(&packs[packs_nr++], &pack, sizeof(pack));\n> > +\n> > +\t\t\tobjects_nr += pack.p->num_objects;\n> > +\t\t}\n> > +\t} else {\n> > +\t\tALLOC_GROW(packs, packs_nr + 1, packs_alloc);\n> > +\n> > +\t\tpacks[packs_nr].p = bitmap_git->pack;\n> > +\t\tpacks[packs_nr].bitmap_pos = 0;\n> > +\t\tpacks[packs_nr].bitmap_nr = bitmap_git->pack->num_objects;\n> > +\t\tpacks[packs_nr].disjoint = 1;\n> > +\n> > +\t\tobjects_nr = packs[packs_nr++].p->num_objects;\n> > +\t}\n> > +\n> > +\tword_alloc = objects_nr / BITS_IN_EWORD;\n> > +\tif (objects_nr % BITS_IN_EWORD)\n> > +\t\tword_alloc++;\n> > +\treuse = bitmap_word_alloc(word_alloc);\n> > +\n> > +\tif (packs_nr != 1)\n> > +\t\tBUG(\"pack reuse not yet implemented for multiple packs\");\n>\n> Can't it happen that we have no pack here? In the MIDX-case we skip all\n> packs that either do not have a bitmap or are not preferred. So does it\n> mean that in reverse, every preferred packfile must have a a bitmap? I'd\n> think that to not be true in case bitmaps are turned off.\n\nIt's subtle, but this state is indeed not possible. If we have a MIDX\nand it has a bitmap, we know that there is at least one object at least\none pack.\n\nOn the \"at least one object front\", that check was added in eb57277ba3\n(midx: prevent writing a .bitmap without any objects, 2022-02-09). And\nwe know that our preferred pack (either explicitly given or the one we\ninfer automatically) is non-empty, via the check added in 5d3cd09a80\n(midx: reject empty `--preferred-pack`'s, 2021-08-31).\n\n(As a fun/non-fun aside, looking these up gave me some serious deja-vu\nand reminded me of how painful discovering and fixing those bugs was!)\n\nSo we're OK here. We could add a comment which captures what I wrote\nabove here, but since this is a temporary state (and we're going to\nchange how we select which packs are reuse candidates in a later patch),\nI think it's OK to avoid (but please let me know if you feel differently).\n\nThanks,\nTaylor\n"},{"id":"485454","messageId":"ZXIugoH4Py3Ie3WT@nand.local","threadId":"60565","inReplyTo":"ZXHE9L7iqXQAit_1@tanuki","subject":"Re: [PATCH 14/24] pack-objects: keep track of `pack_start` for each reuse pack","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-07T20:43:46Z","receivedAt":"2023-12-07T20:43:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Dec 07, 2023 at 02:13:24PM +0100, Patrick Steinhardt wrote:\n> > In order to compute this value correctly, we need to know not only where\n> > we are in the packfile we're assembling (with `hashfile_total(f)`) but\n> > also the position of the first byte of the packfile that we are\n> > currently reusing.\n> >\n> > Together, these two allow us to compute the reused chunk's offset\n> > difference relative to the start of the reused pack, as desired.\n>\n> Hm. I'm not quite sure I fully understand the motivation here. Is this\n> something that was broken all along? Why does it become a problem now?\n> Sorry if I'm missing the obvious here.\n\nNo worries, I should have explained this better. Indeed we do have to\nworry about patching deltas today when reusing objects from a pack. But\nwe have to extend the implementation in order to perform reuse over\nmultiple packs when any of them (excluding the first, which would work\nwith the existing logic) have delta/base pairs on either side of a gap.\n\nI'll try to make it a little clearer, thanks for pointing that out.\n\n> > @@ -1126,10 +1129,12 @@ static void write_reused_pack(struct packed_git *reuse_packfile,\n> > {\n> > \tsize_t i = 0;\n> > \tuint32_t offset;\n> > +\toff_t pack_start = hashfile_total(f) - sizeof(struct pack_header);\n>\n> Given that this patch in its current state doesn't seem to do anything\n> yet, am I right in assuming that `hashfile_total(f) - sizeof(struct\n> pack_header)` is always expected to be zero for now?\n\nYep, that's right.\n\nThanks,\nTaylor\n"},{"id":"485455","messageId":"ZXIvUfRaXUxB+m7w@nand.local","threadId":"60565","inReplyTo":"ZXHE-cLrP7iRHWHY@tanuki","subject":"Re: [PATCH 16/24] pack-objects: prepare `write_reused_pack()` for multi-pack reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-07T20:47:13Z","receivedAt":"2023-12-07T20:47:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Dec 07, 2023 at 02:13:29PM +0100, Patrick Steinhardt wrote:\n> On Tue, Nov 28, 2023 at 02:08:37PM -0500, Taylor Blau wrote:\n> > The function `write_reused_pack()` within `builtin/pack-objects.c` is\n> > responsible for performing pack-reuse on a single pack, and has two main\n> > functions:\n> >\n> > - it dispatches a call to `write_reused_pack_verbatim()` to see if we\n> > can reuse portions of the packfile in whole-word chunks\n> >\n> > - for any remaining objects (that is, any objects that appear after\n> > the first \"gap\" in the bitmap), call write_reused_pack_one() on that\n> > object to record it for reuse.\n> >\n> > Prepare this function for multi-pack reuse by removing the assumption\n> > that the bit position corresponding to the first object being reused\n> > from a given pack may not be at bit position zero.\n>\n> Is this double-negation intended? We remove the assumption that we start\n> reading at position zero, but the paragraph here states that we remove\n> the assumption that we do _not_ start at bit zero.\n\nOops, great catch. I'll s/may not/must in the last paragraph to clarify.\n\n> I'll stop reviewing here and will come back to this series somewhen next\n> week.\n\nThanks as usual for your review -- I appreciate you digging through this\nrather dense series.\n\nThanks,\nTaylor\n"},{"id":"485457","messageId":"ZXLRjeu66qE6J1K1@tanuki","threadId":"60565","inReplyTo":"ZXIq4mjDUoqlGvgW@nand.local","subject":"Re: [PATCH 09/24] repack: implement `--extend-disjoint` mode","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-12-08T08:19:25Z","receivedAt":"2023-12-08T08:19:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Dec 07, 2023 at 03:28:18PM -0500, Taylor Blau wrote:\n> On Thu, Dec 07, 2023 at 02:13:08PM +0100, Patrick Steinhardt wrote:\n> > > - cruft packs (which may necessarily need to include an object from a\n> > > disjoint pack in order to freshen it in certain circumstances)\n> >\n> > This one took me a while to figure out. If we'd mark crufts as disjoint,\n> > then it would mean that new packfiles cannot be marked as disjoint if\n> > objects which were previously unreachable do become reachable again.\n> > So we'd be pessimizing packfiles for live objects in favor of others\n> > which aren't.\n> \n> Yeah, that's right, too. There are a couple of cases where more than one\n> cruft pack may contain the same object, one of them being the\n> flip-flopping between reachable and unreachable as you suggest above.\n> Another is that you have a non-prunable unreachable object which is\n> already in a cruft pack. If the object's mtime gets updated (and still\n> cannot be pruned), we'll end up freshening the object loose, and then\n> packing it again (with the more recent mtime) into a new cruft pack.\n> \n> That aside, I actually think that there are ways to mark cruft packs\n> disjoint. But they're complicated, and moreover, I don't think you'd\n> ever *want* to mark a cruft pack as disjoint. Cruft packs usually\n> contain garbage, which is unlikely to be useful to any fetches/clones.\n> \n> If we did mark them as disjoint, it would mean that we could reuse\n> verbatim sections of the cruft pack in our output, but we would likely\n> end up with very few such sections.\n\nMakes sense. It also doesn't feel worth it to introduce additional\ncomplexity for objects that for most of the part wouldn't ever be served\non a fetch anyway.\n\n[snip]\n> > Okay. I had a bit of trouble to sift through the various different\n> > flags like \"--retain-disjoint\", \"--extend-disjoint\", \"--ignore-disjoint\"\n> > and so on. But well, they do different things and it's been a few days\n> > since I've reviewed the preceding patches, so this is probably fine.\n> \n> Yeah, I am definitely open to better naming conventions here? I figured\n> that:\n> \n> - --retain-disjoint was a good name for the MIDX option, since it is\n> retaining existing disjoint packs in the new MIDX\n> - --extend-disjoint was a good name for the repack option, since it is\n> extending the set of disjoint packs\n> - --ignore-disjoint was a good name for the pack-objects option, since\n> it is ignoring objects in disjoint packs\n> \n> Writing this out, I think that you could make an argument that\n> `--exclude-disjoint` is a better name for the last option. So I'm\n> definitely open to suggestions here, but I don't want to get too bogged\n> down on command-line option naming (so long as we're all reasonably\n> happy with the result).\n\nYeah, as said, I don't mind it too much. It's a complex area and the\nflags all do different things, so it's expected that you may have to do\nsome research on what exactly they do. That being said, I do like your\nproposed `--exclude-disjoint` a lot more than `--ignore-disjoint`.\n\n> > One thing I wondered: do we need to consider the `-l` flag? When using\n> > an alternate object directory it is totally feasible that the alternate\n> > may be creating new disjoint packages without us knowing, and thus we\n> > may not be able to guarantee the disjoint property anymore.\n> \n> I don't think so. We'd only care about one direction of this (that\n> alternates do not create disjoint packs which overlap with ours, instead\n> of the other way around), but since we don't put non-local packs in the\n> MIDX, I think we're OK.\n> \n> I suppose that you might run into trouble if you use the chained MIDX\n> thing (via its `->next` pointer). I haven't used that feature myself, so\n> I'd have to play around with it.\n\nWe do use this feature at GitLab for forks, where forks connect to a\ncommon alternate object directory to deduplicate objects. As both the\nfork repository and the alternate object directory use an MIDX I think\nthey would be set up exactly like that.\n\nI guess the only really viable solution here is to ignore disjoint packs\nin the main repo that connects to the alternate in the case where the\nalternate has any disjoint packs itself.\n\nPatrick\n"},{"id":"485458","messageId":"ZXLRkry2-UFu4RQp@tanuki","threadId":"60565","inReplyTo":"ZXIsbO++u9n/yDYi@nand.local","subject":"Re: [PATCH 10/24] pack-bitmap: pass `bitmapped_pack` struct to pack-reuse functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-12-08T08:19:30Z","receivedAt":"2023-12-08T08:19:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Dec 07, 2023 at 03:34:52PM -0500, Taylor Blau wrote:\n> On Thu, Dec 07, 2023 at 02:13:13PM +0100, Patrick Steinhardt wrote:\n[snip]\n> > Can't it happen that we have no pack here? In the MIDX-case we skip all\n> > packs that either do not have a bitmap or are not preferred. So does it\n> > mean that in reverse, every preferred packfile must have a a bitmap? I'd\n> > think that to not be true in case bitmaps are turned off.\n> \n> It's subtle, but this state is indeed not possible. If we have a MIDX\n> and it has a bitmap, we know that there is at least one object at least\n> one pack.\n> \n> On the \"at least one object front\", that check was added in eb57277ba3\n> (midx: prevent writing a .bitmap without any objects, 2022-02-09). And\n> we know that our preferred pack (either explicitly given or the one we\n> infer automatically) is non-empty, via the check added in 5d3cd09a80\n> (midx: reject empty `--preferred-pack`'s, 2021-08-31).\n> \n> (As a fun/non-fun aside, looking these up gave me some serious deja-vu\n> and reminded me of how painful discovering and fixing those bugs was!)\n> \n> So we're OK here. We could add a comment which captures what I wrote\n> above here, but since this is a temporary state (and we're going to\n> change how we select which packs are reuse candidates in a later patch),\n> I think it's OK to avoid (but please let me know if you feel differently).\n\nMakes sense, thanks for the explanation!\n\nPatrick\n"},{"id":"485491","messageId":"ZXOdPLotcS5daNke@nand.local","threadId":"60565","inReplyTo":"ZXLRjeu66qE6J1K1@tanuki","subject":"Re: [PATCH 09/24] repack: implement `--extend-disjoint` mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-08T22:48:28Z","receivedAt":"2023-12-08T22:48:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Dec 08, 2023 at 09:19:25AM +0100, Patrick Steinhardt wrote:\n> > Writing this out, I think that you could make an argument that\n> > `--exclude-disjoint` is a better name for the last option. So I'm\n> > definitely open to suggestions here, but I don't want to get too bogged\n> > down on command-line option naming (so long as we're all reasonably\n> > happy with the result).\n>\n> Yeah, as said, I don't mind it too much. It's a complex area and the\n> flags all do different things, so it's expected that you may have to do\n> some research on what exactly they do. That being said, I do like your\n> proposed `--exclude-disjoint` a lot more than `--ignore-disjoint`.\n\nI think that's fair, I renamed the option to be \"--exclude-disjoint\"\ninstead of \"--ignore-disjoint\" for any subsequent round(s) of this\nseries.\n\n> > > One thing I wondered: do we need to consider the `-l` flag? When using\n> > > an alternate object directory it is totally feasible that the alternate\n> > > may be creating new disjoint packages without us knowing, and thus we\n> > > may not be able to guarantee the disjoint property anymore.\n> >\n> > I don't think so. We'd only care about one direction of this (that\n> > alternates do not create disjoint packs which overlap with ours, instead\n> > of the other way around), but since we don't put non-local packs in the\n> > MIDX, I think we're OK.\n> >\n> > I suppose that you might run into trouble if you use the chained MIDX\n> > thing (via its `->next` pointer). I haven't used that feature myself, so\n> > I'd have to play around with it.\n>\n> We do use this feature at GitLab for forks, where forks connect to a\n> common alternate object directory to deduplicate objects. As both the\n> fork repository and the alternate object directory use an MIDX I think\n> they would be set up exactly like that.\n\nYep, that's right. I wasn't sure whether or not this feature had been\nused extensively in production or not (we don't use it at GitHub, since\nobjects only live in their fork repositories for a short while before\nmoving to the fork network repository).\n\n> I guess the only really viable solution here is to ignore disjoint packs\n> in the main repo that connects to the alternate in the case where the\n> alternate has any disjoint packs itself.\n\nI think the behavior you'd get here is that we'd only look for disjoint\npacks in the first MIDX in the chain (which is always the local one),\nand we'd only recognizes packs from that MIDX as being potentially\ndisjoint.\n\nIf you have the bulk of your repositories in the alternate, then I think\nyou might want to consider how we combine the two. My sense is that\nyou'd want to be disjoint with respect to anything downstream of you.\n\nWhether or not this is a feature that you/others need, I definitely\nthink we should leave it out of this series, since I am (a) fairly\ncertain that this is possible to do, and (b) already feel like this\nseries on its own is complicated enough.\n\nThanks,\nTaylor\n"},{"id":"485499","messageId":"xmqqlea4nofm.fsf@gitster.g","threadId":"60565","inReplyTo":"ZW95WSErCXvkfrAG@nand.local","subject":"Re: [PATCH 05/24] midx: implement `DISP` chunk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-12-09T01:40:29Z","receivedAt":"2023-12-09T01:40:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Hopefully you're satisfied with the way things are split up and\n> organized currently, but if you have suggestions on other ways I could\n> slice or dice this, please let me know.\n\nI did wonder how expensive to recompute and validate the \"distinct\"\ninformation (in other words, is it too expensive for the consumers\nof an existing midx file to see which packs are distinct on demand\nbefore they stream contents out of the underlying packs?), as the\nway the packs are marked as distinct looked rather error prone (you\ncan very easily list packfiles with overlaps with \"+\" prefix and the\nDISK chunk writer does not even notice that you lied to it). As long\nas \"git fsck\" catches when two packs that are marked as distinct share\nan object, that is OK, but the arrangement did look rather brittle\nto me.\n"},{"id":"485501","messageId":"ZXPRL5yCTQKr0k7e@nand.local","threadId":"60565","inReplyTo":"xmqqlea4nofm.fsf@gitster.g","subject":"Re: [PATCH 05/24] midx: implement `DISP` chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-09T02:30:07Z","receivedAt":"2023-12-09T02:30:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Dec 08, 2023 at 05:40:29PM -0800, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > Hopefully you're satisfied with the way things are split up and\n> > organized currently, but if you have suggestions on other ways I could\n> > slice or dice this, please let me know.\n>\n> I did wonder how expensive to recompute and validate the \"distinct\"\n> information (in other words, is it too expensive for the consumers\n> of an existing midx file to see which packs are distinct on demand\n> before they stream contents out of the underlying packs?), as the\n> way the packs are marked as distinct looked rather error prone (you\n> can very easily list packfiles with overlaps with \"+\" prefix and the\n> DISK chunk writer does not even notice that you lied to it). As long\n> as \"git fsck\" catches when two packs that are marked as distinct share\n> an object, that is OK, but the arrangement did look rather brittle\n> to me.\n\nIt's likely too expensive to do on the reading side for every\npack-objects operation or MIDX load. But we do check this property when\nwe write the MIDX, see these lines from midx.c::get_sorted_entries():\n\n for (cur_object = 0; cur_object < fanout.nr; cur_object++) {\n struct pack_midx_entry *ours = &fanout.entries[cur_object];\n if (cur_object) {\n struct pack_midx_entry *prev = &fanout.entries[cur_object - 1];\n if (oideq(&prev->oid, &ours->oid)) {\n if (prev->disjoint && ours->disjoint)\n die(_(\"duplicate object '%s' among disjoint packs '%s', '%s'\"),\n oid_to_hex(&prev->oid),\n info[prev->pack_int_id].pack_name,\n info[ours->pack_int_id].pack_name);\n continue;\n }\n }\n\nThis series doesn't yet have a corresponding step in the fsck builtin,\nbut I will investigate adding one.\n\nI'm happy to include it in a subsequent round here, but I worry that\nthis series is already on the verge of being too complex as-is, so it\nmay be nice as a follow-up, too.\n\nThanks,\nTaylor\n"},{"id":"485514","messageId":"ZXbF2O4qjIr2L7b8@tanuki","threadId":"60565","inReplyTo":"ZXOdPLotcS5daNke@nand.local","subject":"Re: [PATCH 09/24] repack: implement `--extend-disjoint` mode","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-12-11T08:18:32Z","receivedAt":"2023-12-11T08:18:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Dec 08, 2023 at 05:48:28PM -0500, Taylor Blau wrote:\n> On Fri, Dec 08, 2023 at 09:19:25AM +0100, Patrick Steinhardt wrote:\n> > > > One thing I wondered: do we need to consider the `-l` flag? When using\n> > > > an alternate object directory it is totally feasible that the alternate\n> > > > may be creating new disjoint packages without us knowing, and thus we\n> > > > may not be able to guarantee the disjoint property anymore.\n> > >\n> > > I don't think so. We'd only care about one direction of this (that\n> > > alternates do not create disjoint packs which overlap with ours, instead\n> > > of the other way around), but since we don't put non-local packs in the\n> > > MIDX, I think we're OK.\n> > >\n> > > I suppose that you might run into trouble if you use the chained MIDX\n> > > thing (via its `->next` pointer). I haven't used that feature myself, so\n> > > I'd have to play around with it.\n> >\n> > We do use this feature at GitLab for forks, where forks connect to a\n> > common alternate object directory to deduplicate objects. As both the\n> > fork repository and the alternate object directory use an MIDX I think\n> > they would be set up exactly like that.\n> \n> Yep, that's right. I wasn't sure whether or not this feature had been\n> used extensively in production or not (we don't use it at GitHub, since\n> objects only live in their fork repositories for a short while before\n> moving to the fork network repository).\n> \n> > I guess the only really viable solution here is to ignore disjoint packs\n> > in the main repo that connects to the alternate in the case where the\n> > alternate has any disjoint packs itself.\n> \n> I think the behavior you'd get here is that we'd only look for disjoint\n> packs in the first MIDX in the chain (which is always the local one),\n> and we'd only recognizes packs from that MIDX as being potentially\n> disjoint.\n> \n> If you have the bulk of your repositories in the alternate, then I think\n> you might want to consider how we combine the two. \n\nYes, in the general case the bulk of objects is indeed contained in the\nalternate.\n\n> My sense is that you'd want to be disjoint with respect to anything\n> downstream of you.\n\nIdeally yes, but this is unfortunately not easily achievable in the\ngeneral case. It's one of the many painpoints that alternates bring with\nthem.\n\nSuppose two forks A and B are connected to the same alternate. Both A\nand B now gain the same set of objects via whatever means. At this point\nthese objects can be stored in disjoint packs in each of the repos as\nthey are not yet deduplicated via the alternate. But if you were to pull\nobjects from either A or B into the alternate then you cannot ensure\ndisjointedness at all anymore because you would first have to repack\nobjects in both A and B.\n\nFor two forks this might still seem manageable. But as soon as your fork\nnetwork grows larger it's clear that this becomes almost impossible to\ndo. So ultimately, I don't see an alternative to ignoring disjointedness\nbits in either of the two linked-together repos.\n\n> Whether or not this is a feature that you/others need, I definitely\n> think we should leave it out of this series, since I am (a) fairly\n> certain that this is possible to do, and (b) already feel like this\n> series on its own is complicated enough.\n\nI'm perfectly fine if we say that the benefits of your patch series\ncannot yet be applied to repositories with alternates. But from my point\nof view it's a requirement that this patch series does not silently\nbreak this usecase due to Git starting to generate disjointed packs\nwhere it cannot ensure that the disjointedness property holds.\n\nAs I haven't yet read through this whole patch series, the question is\nboils down to whether the end result is opt-in or opt-out. If it was\nopt-out then I could see the above usecase breaking silently. If it was\nopt-in then things should be fine and we can address this ommission in a\nfollow up patch series. We at GitLab would definitely be interested in\nhelping out with this given that it directly affects us and that the\ndemonstrated savings seem very promising.\n\nPatrick\n"},{"id":"485542","messageId":"ZXdqLhkvqpJ9Jx9t@nand.local","threadId":"60565","inReplyTo":"ZXbF2O4qjIr2L7b8@tanuki","subject":"Re: [PATCH 09/24] repack: implement `--extend-disjoint` mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-11T19:59:42Z","receivedAt":"2023-12-11T19:59:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Dec 11, 2023 at 09:18:32AM +0100, Patrick Steinhardt wrote:\n> > If you have the bulk of your repositories in the alternate, then I think\n> > you might want to consider how we combine the two.\n>\n> Yes, in the general case the bulk of objects is indeed contained in the\n> alternate.\n\nI thought about this for a while this morning, and think that while it\nis possible, I'm not sure I can think of a compelling use-case where\nyou'd want to reuse objects from packs across an alternate boundary.\n\nOn the \"I think it's possible front\":\n\nThe challenge is making sure that the set of disjoint packs among each\nobject store is globally disjoint in one direction along the alternate\npath. I think the rule would require you to honor the disjointed-ness of\nany packs in alternate(s) you might have when constructing new disjoint\npacks.\n\nSo if repository fork.git is an alternate of network.git (and both had\nlong-lived MIDXs), network.git is free to make any set of disjoint packs\nit chooses, and fork.git can only create disjoint packs which are\ndisjoint with respect to (a) the other disjoint packs in fork.git (if\nany), and (b) the disjoint packs in network.git (and recursively for any\nrepositories that network.git is an alternate of in the general case).\n\nNow on the \"I can't think of a compelling use-case front\":\n\nI think the only reason you'd want to be able to reuse objects from\nMIDXs across the alternates boundary is if you have MIDX bitmaps in both\nthe repository and its alternate. Indeed, the only time that we kick in\npack-reuse in general is when we have a bitmap, so in order to reuse\nobjects from both the repo and its alternate, you'd have to have a\nbitmap in both repositories.\n\nBut having a MIDX bitmap means that any packs in the MIDX for which\nyou're generating a bitmap have to be closed over object reachability.\nSo unless the repository and its alternate have totally distinct lines\nof history (in which case, I'm not sure you would want to share objects\nbetween the two in the first place), any pack you bitmap in the child\nrepository fundamentally couldn't be disjoint with respect to its\nparent.\n\nThis is because if it were to be disjoint, it would have to be repacked\nwith '-l' (or some equivalent '-l'-like flag that only ignores non-local\npacks which are marked as disjoint). But if you exclude those objects\nand any one (or more) of them is reachable from some object(s) you\ndidn't exclude, you wouldn't be able to generate a bitmap in the first\nplace.\n\nIt's very possible that there's something about your setup that I'm not\nfully grokking, but I don't think in general this is something that we'd\nwant to do (even if it is theoretically possible).\n\n> > Whether or not this is a feature that you/others need, I definitely\n> > think we should leave it out of this series, since I am (a) fairly\n> > certain that this is possible to do, and (b) already feel like this\n> > series on its own is complicated enough.\n>\n> I'm perfectly fine if we say that the benefits of your patch series\n> cannot yet be applied to repositories with alternates. But from my point\n> of view it's a requirement that this patch series does not silently\n> break this usecase due to Git starting to generate disjointed packs\n> where it cannot ensure that the disjointedness property holds.\n\nI think one thing you could reasonably do is use *only* the non-local\nMIDX bitmaps when doing pack reuse.\n\nCurrently we'll use the first MIDX we find, which is guaranteed to be\nthe local one, if it exists. This was the case before this series, and\nthis series does not change that behavior. Unless you had a pack bitmap\nin the alternated repository (which I think is unlikely, since it would\nrequire a full reachability closure, thus eliminating any de-duplication\nbenefits you'd otherwise get when using alternates), you'd be find\nbefore and after this series.\n\n> As I haven't yet read through this whole patch series, the question is\n> boils down to whether the end result is opt-in or opt-out. If it was\n> opt-out then I could see the above usecase breaking silently. If it was\n> opt-in then things should be fine and we can address this ommission in a\n> follow up patch series. We at GitLab would definitely be interested in\n> helping out with this given that it directly affects us and that the\n> demonstrated savings seem very promising.\n\nThe end result is opt-in, you have to change the `pack.allowPackReuse`\nconfiguration from its default value of \"true\" (or the alternate\nspelling taught in this series, \"single\") to \"multi\" in order to enable\nthe new behavior.\n\nThanks,\nTaylor\n"},{"id":"485567","messageId":"20231212070406.GA1117953@coredump.intra.peff.net","threadId":"60565","inReplyTo":"6f5ff96998946f3f49da56fd05c096b949521339.1701198172.git.me@ttaylorr.com","subject":"Re: [PATCH 02/24] pack-bitmap-write: deep-clear the `bb_commit` slab","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-12-12T07:04:06Z","receivedAt":"2023-12-12T07:04:10Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 28, 2023 at 02:07:59PM -0500, Taylor Blau wrote:\n\n> +static void clear_bb_commit(struct bb_commit *commit)\n> +{\n> +\tfree(commit->reverse_edges);\n> +\tbitmap_free(commit->commit_mask);\n> +\tbitmap_free(commit->bitmap);\n> +}\n\nI think these bitmaps may sometimes be NULL. But double-checking\nbitmap_free(), it sensibly is noop when passed NULL. So this look good\nto me.\n\n-Peff\n"},{"id":"485568","messageId":"20231212070458.GB1117953@coredump.intra.peff.net","threadId":"60565","inReplyTo":"bc38fba65547f5716a2ff9904dd615e755ea84bb.1701198172.git.me@ttaylorr.com","subject":"Re: [PATCH 03/24] pack-bitmap: plug leak in find_objects()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-12-12T07:04:58Z","receivedAt":"2023-12-12T07:05:00Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 28, 2023 at 02:08:02PM -0500, Taylor Blau wrote:\n\n> The `find_objects()` function creates an object_list for any tips of the\n> reachability query which do not have corresponding bitmaps.\n> \n> The object_list is not used outside of `find_objects()`, but we never\n> free it with `object_list_free()`, resulting in a leak. Let's plug that\n> leak by calling `object_list_free()`, which results in t6113 becoming\n> leak-free.\n\nMakes sense.\n\n> @@ -1280,6 +1280,8 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n> \t\tbase = fill_in_bitmap(bitmap_git, revs, base, seen);\n> \t}\n> \n> +\tobject_list_free(¬_mapped);\n> +\n> \treturn base;\n> }\n\nThere's an extra return earlier in the function, but it triggers only\nwhen not_mapped is NULL. So this covers all cases. Good.\n\n> +++ b/t/t6113-rev-list-bitmap-filters.sh\n> [..]\n> +TEST_PASSES_SANITIZE_LEAK=true\n\nYay. :)\n\n-Peff\n"},{"id":"485574","messageId":"20231212080332.GC1117953@coredump.intra.peff.net","threadId":"60565","inReplyTo":"ZXPRL5yCTQKr0k7e@nand.local","subject":"Re: [PATCH 05/24] midx: implement `DISP` chunk","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-12-12T08:03:32Z","receivedAt":"2023-12-12T08:03:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 08, 2023 at 09:30:07PM -0500, Taylor Blau wrote:\n\n> > I did wonder how expensive to recompute and validate the \"distinct\"\n> > information (in other words, is it too expensive for the consumers\n> > of an existing midx file to see which packs are distinct on demand\n> > before they stream contents out of the underlying packs?), as the\n> > way the packs are marked as distinct looked rather error prone (you\n> > can very easily list packfiles with overlaps with \"+\" prefix and the\n> > DISK chunk writer does not even notice that you lied to it). As long\n> > as \"git fsck\" catches when two packs that are marked as distinct share\n> > an object, that is OK, but the arrangement did look rather brittle\n> > to me.\n> \n> It's likely too expensive to do on the reading side for every\n> pack-objects operation or MIDX load.\n\nThis made me think a bit. Obviously we can check for disjointedness in\nO(n log k), where n is the total number of objects and k is the number\nof packs, by doing a k-merge of the sorted lists. But that's a\nnon-starter, because we may be serving a request that is much smaller\nthan all \"n\" objects (e.g., any small fetch, but also even clones when\nthe pack contains a bunch of irrelevant objects).\n\nBut we can relax our condition a bit. The packs only need to be disjoint\nwith respect to the set of objects that we are going to output (we'll\ncall that \"m\"). So at the very least, you can do O(mk) lookups (each one\nitself \"log n\", of course). We know that the work is already going to\nbe linear in \"m\". In theory we want to generally keep \"k\" small, but\npart of the point of using midx in this way is to let \"k\" grow a bit.\nSo that might turn out pretty bad in practice.\n\nSo let's take one more step back. One thing I didn't feel like I saw\neither in this patch or the cover letter is exactly why we care about\ndisjointedness. IIRC, the main issue is making sure that for any object\nX we send verbatim, it is either a non-delta or its delta base is\nviable. And the two reasons I can think of for the base to be non-viable\nare:\n\n 1. We are not sending the base at all.\n\n 2. The base is in another pack, and we are worried about creating a\n cycle (i.e., in pack A we have X as a delta against Y, and in pack\n B we have Y as a delta against X, and we send both deltas).\n\nWe already deal with (1) for the single-pack case by finding the base\nobject offset, converting it to a pack position, and then making sure\nthat position is also marked for verbatim reuse.\n\nThe interesting one is (2), which is new for midx (since a single pack\ncannot have two copies of an object). But I'm not sure if it's possible.\nThe verbatim reuse code depends on using bitmaps in the first place. And\nthere is already a preference-order to the packs in the midx bitmaps.\n\nThat is, we'll choose all of the objects for pack A over any duplicates\nin B, and duplicates from B over C, and so on. If we likewise try\nverbatim reuse in that order, then we know that an object in pack A can\nnever have a base that is selected from pack B or C (because if they do\nhave duplicates, we'd have selected A's copy to put in the midx bitmap).\nAnd likewise, a copy of an object in pack B will always have its base\neither in A or B, but never in C.\n\nSo it kind of seems to me that this would \"just work\" if\ntry_partial_reuse() did something like for the midx case:\n\n - convert offset of base object into an object id hash using the pack\n revindex (similar to offset_to_pack_pos)\n\n - look up the object id in the midx to get a pack/offset combo\n\n - use the midx revindex to convert that into a bit position\n\n - check if that bit is marked for verbatim reuse\n\nThe assumption there is that in the second step, the midx has resolved\nduplicates by putting all of pack A before pack B, and so on, as above.\nIt also assumes that we are trying verbatim reuse in the same order\n(though a different order would not produce wrong results, it would only\nresult in less verbatim reuse).\n\nAll of which makes me think I'm missing some other case that is a\nproblem. While I wait for you to explain it, though, let's continue our\nthought experiment for a moment.\n\nIf we assume that any cross-pack deltas are a problem, we could always\njust skip them for verbatim reuse. That is, once we look up the object\nid in the midx, we can see if it's in the same pack we're currently\nprocessing. If not, we could punt and let the non-verbatim code paths\nhandle it as usual.\n\nThat still leaves the problem of realizing we skipped over a chunk of\nthe packfile (say we are pulling object X from pack B, and it is a delta\nof Y, but we already decided to reuse Y from pack A). But the reuse code\nalready has to accommodate gaps. I think it happens naturally in\nwrite_reused_pack_one(), where we feed the actual byte offsets into\nrecord_reused_object(). You'd have to take some care not to go past gaps\nin the blind copy that happens write_reused_pack_verbatim(). So you\nmight need to mark the first such gap somehow (if it's hard, I'd suggest\njust skipping write_reused_pack_verbatim() entirely; it is a fairly\nminor optimization compared to the rest of it).\n\nAnd of course there's a bunch of hard work teaching all of those\nfunctions about midx's and multiple packs in the first place, but you\nalready had to do all that in the latter part of your series, and it\nwould still be applicable.\n\nOK, this is the part where you tell me what I'm missing. ;)\n\n-Peff\n"},{"id":"485575","messageId":"20231212081238.GD1117953@coredump.intra.peff.net","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"Re: [PATCH 00/24] pack-objects: multi-pack verbatim reuse","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-12-12T08:12:38Z","receivedAt":"2023-12-12T08:12:39Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 28, 2023 at 02:07:54PM -0500, Taylor Blau wrote:\n\n> Performing verbatim pack reuse naturally trades off between CPU time and\n> the resulting pack size. In the above example, the single-pack reuse\n> case produces a clone size of ~194 MB on my machine, while the\n> multi-pack reuse case produces a clone size closer to ~266 MB, which is\n> a ~37% increase in clone size.\n\nRight, it's definitely a tradeoff. So taking a really big step back,\nthere are a few optimizations all tied up in the verbatim reuse code:\n\n 1. in some cases we get to dump whole swaths of the on-disk packfile\n to the output, covering many objects with a few memcpy() calls.\n (This is still O(n), of course, but it's fewer instructions per\n object).\n\n 2. any other reused objects have only a small-ish amount of work to\n fix up ofs deltas, handle gaps, and so on. We get to skip adding\n them to the packing_list struct (this saves some CPU, but also a\n lot of memory)\n\n 3. we skip the delta search for these reused objects. This is where\n your big CPU / output size tradeoff comes into play, I'd think.\n\nSo my question is: how much of what you're seeing is from (1) and (2),\nand how much is from (3)? Because there are other ways to trigger (3),\nsuch as lowering the window size. For example, if you try your same\npacking example with --window=0, how do the CPU and output size compare\nto the results of your series? (I'd also check peak memory usage).\n\n-Peff\n"},{"id":"485582","messageId":"ZXiOzN4T06yy/NvV@nand.local","threadId":"60565","inReplyTo":"20231212070406.GA1117953@coredump.intra.peff.net","subject":"Re: [PATCH 02/24] pack-bitmap-write: deep-clear the `bb_commit` slab","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-12T16:48:12Z","receivedAt":"2023-12-12T16:48:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Dec 12, 2023 at 02:04:06AM -0500, Jeff King wrote:\n> On Tue, Nov 28, 2023 at 02:07:59PM -0500, Taylor Blau wrote:\n>\n> > +static void clear_bb_commit(struct bb_commit *commit)\n> > +{\n> > +\tfree(commit->reverse_edges);\n> > +\tbitmap_free(commit->commit_mask);\n> > +\tbitmap_free(commit->bitmap);\n> > +}\n>\n> I think these bitmaps may sometimes be NULL. But double-checking\n> bitmap_free(), it sensibly is noop when passed NULL. So this look good\n> to me.\n\nYeah, bitamp_free() handles a NULL input correctly, so we can pass a\npossibly-NULL `commit->commit_mask` or `commit->bitmap` argument and be\nOK.\n\nThanks,\nTaylor\n"},{"id":"485624","messageId":"ZXn33HVBFqidAThn@nand.local","threadId":"60565","inReplyTo":"20231212080332.GC1117953@coredump.intra.peff.net","subject":"Re: [PATCH 05/24] midx: implement `DISP` chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-13T18:28:44Z","receivedAt":"2023-12-13T18:28:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Peff,\n\nOn Tue, Dec 12, 2023 at 03:03:32AM -0500, Jeff King wrote:\n> [...]\n\nWell this took me longer to respond to than I thought it would ;-).\n\n> So it kind of seems to me that this would \"just work\" if\n> try_partial_reuse() did something like for the midx case:\n>\n> - convert offset of base object into an object id hash using the pack\n> revindex (similar to offset_to_pack_pos)\n>\n> - look up the object id in the midx to get a pack/offset combo\n>\n> - use the midx revindex to convert that into a bit position\n>\n> - check if that bit is marked for verbatim reuse\n>\n> The assumption there is that in the second step, the midx has resolved\n> duplicates by putting all of pack A before pack B, and so on, as above.\n> It also assumes that we are trying verbatim reuse in the same order\n> (though a different order would not produce wrong results, it would only\n> result in less verbatim reuse).\n\nAfter much thinking, I agree with your conclusion here. Which is great\nnews indeed, since it allows us to implement multi-pack reuse without\nrequiring that the candidate packs be disjoint with respect to their\nobjects.\n\nEven though we have some protections in place for ensuring these packs'\ndisjointed-ness, I agree with Junio upthread that this is likely the\nmost fragile part of this series. That is, even though we check in\nmidx.c::get_sorted_entries() that the marked packs are disjoint, if we\nmiss something, the results would be rather bad. At best it would result\nin sending a corrupt packfile to the client, and at worst it could\nresult in permanent data corruption if repacking a repository with\nmulti-pack reuse enabled.\n\n> If we assume that any cross-pack deltas are a problem, we could always\n> just skip them for verbatim reuse. That is, once we look up the object\n> id in the midx, we can see if it's in the same pack we're currently\n> processing. If not, we could punt and let the non-verbatim code paths\n> handle it as usual.\n\nLet me think aloud for a second, since it took me quite a lot of\nthinking to fully wrap my head around this. Suppose we have two packs,\nP1 and P2 where P1 has object A delta'd against B, and P2 has its own\ncopy of object B. If the MIDX chose the copy of B from P2, but also\ndecided to send P1 (which it chose from A), then if P1 is stored as an\nOFS delta, we would be sending a corrupt packfile to the client.\n\nI think there are a few things that we could reasonably do here:\n\n - Reject cross-pack deltas entirely (as you suggest). This is the\n easiest implementation choice in this already-complex series, and it\n doesn't paint us into a corner in the sense that we could relax\n these requirements in the future.\n\n - Turn any cross-pack deltas (which are stored as OFS_DELTAs) into\n REF_DELTAs. We already do this today when reusing an OFS_DELTA\n without `--delta-base-offset` enabled, so it's not a huge stretch to\n do the same for cross-pack deltas even when `--delta-base-offset` is\n enabled.\n\n This would work, but would obviously result in larger-than-necessary\n packs, as we in theory *could* represent these cross-pack deltas by\n patching an existing OFS_DELTA. But it's not clear how much that\n would matter in practice. I suspect it would have a lot to do with\n how you pack your repository in the first place.\n\n - Finally, we could patch OFS_DELTAs across packs in a similar fashion\n as we do today for OFS_DELTAs within a single pack on either side of\n a gap. This would result in the smallest packs of the three options\n here, but implementing this would be more involved.\n\n At minimum, you'd have to keep the reusable chunks list for all\n reused packs, not just the one we're currently processing. And you'd\n have to ensure that any bases which are a part of cross-pack deltas\n appear before the delta. I think this is possible to do, but would\n require assembling the reusable chunks list potentially in a\n different order than they appear in the source packs.\n\n> And of course there's a bunch of hard work teaching all of those\n> functions about midx's and multiple packs in the first place, but you\n> already had to do all that in the latter part of your series, and it\n> would still be applicable.\n\nYep, all of that is still a requirement here, and lives on in the\nrevised version of this series.\n\nThe naive implementation where we call try_partial_reuse() on every\nobject which is a candidate for reuse and check for cross-pack deltas\nwould work, but have poor performance. The reason is that we would be\ndoing a significant amount of cache-inefficient work to determine\nwhether or not the base for some delta/base-pair resides in the same\npack:\n\n - If you see a delta in some pack while processing a MIDX bitmap for\n reuse, you first have to figure out the location of its base in that\n same pack. (Note: this may or may not be the copy of the base object\n chosen by the MIDX).\n\n - To figure out whether or not the MIDX chose that copy, you would\n naively have to do something like:\n\n - Convert the base object's offset into a packfile position using\n the pack revindex.\n\n - Convert the base object's packfile position into an index\n position, which would then be used to obtain its OID.\n\n - Then binary search through the MIDX for that OID found in the\n previous step, filling out the MIDX entry for that object.\n\n - Toss out the cross-pack delta/base pair if the MIDX entry in the\n previous step indicates that the MIDX chose a copy of the base\n from a different pack than the one we're currently processing\n (i.e. where the delta resides).\n\nThat's rather inefficient. But, we can do better. Instead of going back\nand forth through both the pack and MIDX's reverse index, we can simply\nbinary search in the MIDX's reverse index for the (pack_id, offset) pair\ncorresponding to the base.\n\nIf we find a match, then we know that the MIDX chose its copy of the\nbase object from the same pack, and we can reuse the delta/base-pair. If\nwe don't, then we know that the MIDX chose its copy of the base object\nfrom a different pack, and we have to throw out the delta/base-pair.\n\nThis is a bit more involved than the naive implementation, but it's (a)\nefficient, and (b) most of the code for it already exists in the form of\nmidx_to_pack_pos(), which implements a binary search over the MIDX\nbitmap's pseudo-pack order.\n\nWith some light refactoring, we can repurpose this code to perform a\nbinary search for a given (pack, offset) pair instead of starting with a\nMIDX lex position and converting it into the (pack, offset) pair. So\nthat works, and is what I've done in the revised version of this series.\n\nThere is one other snag, which is that we can no longer blindly reuse\nwhole-words from the reuse bitmap if we have non-disjoint packs. That\nis, we can't do something like:\n\n while (pos < result->word_alloc && result->words[pos] == (eword_t)~0)\n pos++;\n\nwhen processing anything but the first pack.\n\nThe reason is that we know the first pack has all duplicate object ties\nbroken in its favor, but we don't have the same guarantee for subsequent\npacks. So we have to be more careful about which bits we reuse from\nthose subsequent packs, since we may inadvertently pick up a cross-pack\ndelta/base pair without inspecting it more closely.\n\nAs I mentioned, you can still perform this optimization over the first\npack, and I think that will be sufficient for most repositories. It's\nnot clear to me exactly how much this optimization is helping us in\ncontrast to all of the other work that pack-objects is doing, but that\nis probably something worth measuring.\n\nThanks for the terrific suggestion. I'll clean up the results of trying\nto implement it, and share it with the list soon (ideally before the end\nof this week, after which I'm on vacation until the new year).\n\nThanks,\nTaylor\n"},{"id":"485625","messageId":"xmqqwmthnc31.fsf@gitster.g","threadId":"60565","inReplyTo":"ZXn33HVBFqidAThn@nand.local","subject":"Re: [PATCH 05/24] midx: implement `DISP` chunk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-12-13T19:20:50Z","receivedAt":"2023-12-13T19:20:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> - Turn any cross-pack deltas (which are stored as OFS_DELTAs) into\n> REF_DELTAs. We already do this today when reusing an OFS_DELTA\n> without `--delta-base-offset` enabled, so it's not a huge stretch to\n> do the same for cross-pack deltas even when `--delta-base-offset` is\n> enabled.\n>\n> This would work, but would obviously result in larger-than-necessary\n> packs, as we in theory *could* represent these cross-pack deltas by\n> patching an existing OFS_DELTA. But it's not clear how much that\n> would matter in practice. I suspect it would have a lot to do with\n> how you pack your repository in the first place.\n\nI have to wonder if the cost of additional computation to see when\nis safe to allow cross-pack delta can be kept reasonably low, as\nyou'd need to prove that you are not introducing a cycle by doing\nso. Compared to that, spending a dozen or so bytes for the offset\nfor rare cases of having to express such a cross-pack delta pair\nwould be much less worrisome to me.\n\n> Thanks for the terrific suggestion. I'll clean up the results of trying\n> to implement it, and share it with the list soon (ideally before the end\n> of this week, after which I'm on vacation until the new year).\n\nGood to hear that you'll be recharging soon. Enjoy.\n"},{"id":"485675","messageId":"cover.1702592603.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1701198172.git.me@ttaylorr.com","subject":"[PATCH v2 00/26] pack-objects: multi-pack verbatim reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:23:33Z","receivedAt":"2023-12-14T22:23:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This is a reroll of my series to implement multi-pack verbatim reuse.\n\nSince last time, I implemented a suggestion from Peff in [1] that\nrequires reusable packs be disjoint with respect to the objects they are\nsending, not their entire set of objects. This allows for us to\nimplement multi-pack reuse without the DISP chunk or the\n`--extend-disjoint`, `--exclude-disjoint`, `--retain-disjoint`, etc\noptions from the previous round.\n\nBesides that, much is unchanged from last time, and the bulk of the\nremaining changes are from Patrick Steinhardt's review of the first 2/3\nof this series.\n\nPerformance remains mostly unchanged since last time, and I was able to\nachieve the following results in hyperfine on my worst-case scenario\ntest repository (broken into ~100 packs):\n\n $ hyperfine -L v single,multi -n '{v}-pack reuse' \\\n 'git.compile -c pack.allowPackReuse={v} pack-objects --revs --stdout --use-bitmap-index --delta-base-offset /dev/null'\n Benchmark 1: single-pack reuse\n Time (mean ± σ): 6.234 s ± 0.026 s [User: 43.733 s, System: 0.349 s]\n Range (min … max): 6.197 s … 6.293 s 10 runs\n\n Benchmark 2: multi-pack reuse\n Time (mean ± σ): 959.5 ms ± 2.4 ms [User: 1133.8 ms, System: 36.3 ms]\n Range (min … max): 957.2 ms … 964.8 ms 10 runs\n\n Summary\n multi-pack reuse ran\n 6.50 ± 0.03 times faster than single-pack reuse\n\nAs before, this series is still quite large, despite losing a fair bit\nof code related to disjoint packs. After thinking on it some more, I\nstill couldn't come up with a satisfying way to break up this series, so\nhere it is presented in one big chunk. If others have ideas on how\nbetter to present this series, please let me know.\n\nIn the meantime, thanks in advance for your review!\n\n[1]: https://lore.kernel.org/git/20231212080332.GC1117953@coredump.intra.peff.net/\n\nTaylor Blau (26):\n pack-objects: free packing_data in more places\n pack-bitmap-write: deep-clear the `bb_commit` slab\n pack-bitmap: plug leak in find_objects()\n midx: factor out `fill_pack_info()`\n midx: implement `BTMP` chunk\n midx: implement `midx_locate_pack()`\n pack-bitmap: pass `bitmapped_pack` struct to pack-reuse functions\n ewah: implement `bitmap_is_empty()`\n pack-bitmap: simplify `reuse_partial_packfile_from_bitmap()` signature\n pack-bitmap: return multiple packs via\n `reuse_partial_packfile_from_bitmap()`\n pack-objects: parameterize pack-reuse routines over a single pack\n pack-objects: keep track of `pack_start` for each reuse pack\n pack-objects: pass `bitmapped_pack`'s to pack-reuse functions\n pack-objects: prepare `write_reused_pack()` for multi-pack reuse\n pack-objects: prepare `write_reused_pack_verbatim()` for multi-pack\n reuse\n pack-objects: include number of packs reused in output\n git-compat-util.h: implement checked size_t to uint32_t conversion\n midx: implement `midx_preferred_pack()`\n pack-revindex: factor out `midx_key_to_pack_pos()` helper\n pack-revindex: implement `midx_pair_to_pack_pos()`\n pack-bitmap: prepare to mark objects from multiple packs for reuse\n pack-objects: add tracing for various packfile metrics\n t/test-lib-functions.sh: implement `test_trace2_data` helper\n pack-objects: allow setting `pack.allowPackReuse` to \"single\"\n pack-bitmap: enable reuse from all bitmapped packs\n t/perf: add performance tests for multi-pack reuse\n\n Documentation/config/pack.txt | 16 +-\n Documentation/gitformat-pack.txt | 76 +++++++\n builtin/pack-objects.c | 169 +++++++++++----\n ewah/bitmap.c | 9 +\n ewah/ewok.h | 1 +\n git-compat-util.h | 9 +\n midx.c | 151 +++++++++++---\n midx.h | 12 +-\n pack-bitmap-write.c | 9 +-\n pack-bitmap.c | 321 +++++++++++++++++++----------\n pack-bitmap.h | 19 +-\n pack-objects.c | 15 ++\n pack-objects.h | 1 +\n pack-revindex.c | 50 +++--\n pack-revindex.h | 3 +\n t/helper/test-read-midx.c | 41 +++-\n t/perf/p5332-multi-pack-reuse.sh | 81 ++++++++\n t/t5319-multi-pack-index.sh | 35 ++++\n t/t5332-multi-pack-reuse.sh | 203 ++++++++++++++++++\n t/t6113-rev-list-bitmap-filters.sh | 2 +\n t/test-lib-functions.sh | 14 ++\n 21 files changed, 1039 insertions(+), 198 deletions(-)\n create mode 100755 t/perf/p5332-multi-pack-reuse.sh\n create mode 100755 t/t5332-multi-pack-reuse.sh\n\nRange-diff against v1:\n 1: 101d6a2841 ! 1: 7d65abfa1d pack-objects: free packing_data in more places\n @@ Commit message\n \n Since these structures contain allocated fields, failing to\n appropriately free() them results in a leak. Plug that leak by\n - introducing a free_packing_data() function, and call it in the\n + introducing a clear_packing_data() function, and call it in the\n appropriate spots.\n \n This is a fairly straightforward leak to plug, since none of the callers\n @@ builtin/pack-objects.c: int cmd_pack_objects(int argc, const char **argv, const\n \t\t\t reuse_packfile_objects);\n \n cleanup:\n -+\tfree_packing_data(&to_pack);\n ++\tclear_packing_data(&to_pack);\n \tlist_objects_filter_release(&filter_options);\n \tstrvec_clear(&rp);\n \n @@ midx.c: static int write_midx_internal(const char *object_dir,\n \t\t\t\t flags) < 0) {\n \t\t\terror(_(\"could not write multi-pack bitmap\"));\n \t\t\tresult = 1;\n -+\t\t\tfree_packing_data(&pdata);\n ++\t\t\tclear_packing_data(&pdata);\n +\t\t\tfree(commits);\n \t\t\tgoto cleanup;\n \t\t}\n +\n -+\t\tfree_packing_data(&pdata);\n ++\t\tclear_packing_data(&pdata);\n +\t\tfree(commits);\n \t}\n \t/*\n @@ pack-objects.c: void prepare_packing_data(struct repository *r, struct packing_d\n \tinit_recursive_mutex(&pdata->odb_lock);\n }\n \n -+void free_packing_data(struct packing_data *pdata)\n ++void clear_packing_data(struct packing_data *pdata)\n +{\n +\tif (!pdata)\n +\t\treturn;\n @@ pack-objects.h: struct packing_data {\n };\n \n void prepare_packing_data(struct repository *r, struct packing_data *pdata);\n -+void free_packing_data(struct packing_data *pdata);\n ++void clear_packing_data(struct packing_data *pdata);\n \n /* Protect access to object database */\n static inline void packing_data_lock(struct packing_data *pdata)\n 2: 6f5ff96998 ! 2: 19cdaf59c5 pack-bitmap-write: deep-clear the `bb_commit` slab\n @@ Commit message\n bb_commit` type, and make sure it is called on each member of the slab\n via the `deep_clear_bb_data()` function.\n \n + Note that it is possible for both of the arguments to `bitmap_free()` to\n + be NULL, but `bitmap_free()` is a noop for NULL arguments, so it is OK\n + to pass them unconditionally.\n +\n Signed-off-by: Taylor Blau \n \n ## pack-bitmap-write.c ##\n @@ pack-bitmap-write.c: struct bb_commit {\n \n +static void clear_bb_commit(struct bb_commit *commit)\n +{\n -+\tfree(commit->reverse_edges);\n ++\tfree_commit_list(commit->reverse_edges);\n +\tbitmap_free(commit->commit_mask);\n +\tbitmap_free(commit->bitmap);\n +}\n 3: bc38fba655 = 3: 477df6c974 pack-bitmap: plug leak in find_objects()\n 4: ccf1337305 ! 4: a06ed75af1 midx: factor out `fill_pack_info()`\n @@ midx.c: struct pack_info {\n };\n \n +static void fill_pack_info(struct pack_info *info,\n -+\t\t\t struct packed_git *p, char *pack_name,\n ++\t\t\t struct packed_git *p, const char *pack_name,\n +\t\t\t uint32_t orig_pack_int_id)\n +{\n +\tmemset(info, 0, sizeof(struct pack_info));\n +\n +\tinfo->orig_pack_int_id = orig_pack_int_id;\n -+\tinfo->pack_name = pack_name;\n ++\tinfo->pack_name = xstrdup(pack_name);\n +\tinfo->p = p;\n +}\n +\n @@ midx.c: static void add_pack_to_midx(const char *full_path, size_t full_path_len\n +\t\tif (open_pack_index(p)) {\n \t\t\twarning(_(\"failed to open pack-index '%s'\"),\n \t\t\t\tfull_path);\n - \t\t\tclose_pack(ctx->info[ctx->nr].p);\n -@@ midx.c: static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n +-\t\t\tclose_pack(ctx->info[ctx->nr].p);\n +-\t\t\tFREE_AND_NULL(ctx->info[ctx->nr].p);\n ++\t\t\tclose_pack(p);\n ++\t\t\tfree(p);\n \t\t\treturn;\n \t\t}\n \n -\t\tctx->info[ctx->nr].pack_name = xstrdup(file_name);\n -\t\tctx->info[ctx->nr].orig_pack_int_id = ctx->nr;\n -\t\tctx->info[ctx->nr].expired = 0;\n -+\t\tfill_pack_info(&ctx->info[ctx->nr], p, xstrdup(file_name),\n -+\t\t\t ctx->nr);\n ++\t\tfill_pack_info(&ctx->info[ctx->nr], p, file_name, ctx->nr);\n \t\tctx->nr++;\n \t}\n }\n @@ midx.c: static int write_midx_internal(const char *object_dir,\n \n -\t\t\tctx.nr++;\n +\t\t\tfill_pack_info(&ctx.info[ctx.nr++], ctx.m->packs[i],\n -+\t\t\t\t xstrdup(ctx.m->pack_names[i]), i);\n ++\t\t\t\t ctx.m->pack_names[i], i);\n \t\t}\n \t}\n \n 5: c52d7e7b27 ! 5: 6fdc68418f midx: implement `DISP` chunk\n @@ Metadata\n Author: Taylor Blau \n \n ## Commit message ##\n - midx: implement `DISP` chunk\n + midx: implement `BTMP` chunk\n \n When a multi-pack bitmap is used to implement verbatim pack reuse (that\n is, when verbatim chunks from an on-disk packfile are copied\n @@ Commit message\n It would be beneficial to be able to perform this same optimization over\n multiple packs, provided some modest constraints (most importantly, that\n the set of packs eligible for verbatim reuse are disjoint with respect\n - to the objects that they contain).\n + to the subset of their objects being sent).\n \n If we assume that the packs which we treat as candidates for verbatim\n - reuse are disjoint with respect to their objects, we need to make only\n - modest modifications to the verbatim pack-reuse code itself. Most\n - notably, we need to remove the assumption that the bits in the\n - reachability bitmap corresponding to objects from the single reuse pack\n - begin at the first bit position.\n + reuse are disjoint with respect to any of their objects we may output,\n + we need to make only modest modifications to the verbatim pack-reuse\n + code itself. Most notably, we need to remove the assumption that the\n + bits in the reachability bitmap corresponding to objects from the single\n + reuse pack begin at the first bit position.\n \n Future patches will unwind these assumptions and reimplement their\n existing functionality as special cases of the more general assumptions\n @@ Commit message\n to start at 0 for all existing cases).\n \n This patch does not yet relax any of those assumptions. Instead, it\n - implements a foundational data-structure, the \"Disjoint Packs\" (`DISP`)\n - chunk of the multi-pack index. The `DISP` chunk's contents are described\n - in detail here. Importantly, the `DISP` chunk contains information to\n + implements a foundational data-structure, the \"Bitampped Packs\" (`BTMP`)\n + chunk of the multi-pack index. The `BTMP` chunk's contents are described\n + in detail here. Importantly, the `BTMP` chunk contains information to\n map regions of a multi-pack index's reachability bitmap to the packs\n whose objects they represent.\n \n @@ Commit message\n used in this patch to test the new chunk's behavior). Future patches\n will begin to make use of this new chunk.\n \n - This patch implements reading (though no callers outside of the above\n - one perform any reading) and writing this new chunk. It also extends the\n - `--stdin-packs` format used by the `git multi-pack-index write` builtin\n - to be able to designate that a given pack is to be marked as \"disjoint\"\n - by prefixing it with a '+' character.\n -\n [^1]: Modulo patching any `OFS_DELTA`'s that cross over a region of the\n pack that wasn't used verbatim.\n \n Signed-off-by: Taylor Blau \n \n - ## Documentation/git-multi-pack-index.txt ##\n -@@ Documentation/git-multi-pack-index.txt: write::\n - \t--stdin-packs::\n - \t\tWrite a multi-pack index containing only the set of\n - \t\tline-delimited pack index basenames provided over stdin.\n -+\t\tLines beginning with a '+' character (followed by the\n -+\t\tpack index basename as before) have their pack marked as\n -+\t\t\"disjoint\". See the \"`DISP` chunk and disjoint packs\"\n -+\t\tsection in linkgit:gitformat-pack[5] for more.\n - \n - \t--refs-snapshot=::\n - \t\tWith `--bitmap`, optionally specify a file which\n -\n ## Documentation/gitformat-pack.txt ##\n @@ Documentation/gitformat-pack.txt: CHUNK DATA:\n \t is padded at the end with between 0 and 3 NUL bytes to make the\n \t chunk size a multiple of 4 bytes.\n \n -+\tDisjoint Packfiles (ID: {'D', 'I', 'S', 'P'})\n -+\t Stores a table of three 4-byte unsigned integers in network order.\n ++\tBitmapped Packfiles (ID: {'B', 'T', 'M', 'P'})\n ++\t Stores a table of two 4-byte unsigned integers in network order.\n +\t Each table entry corresponds to a single pack (in the order that\n +\t they appear above in the `PNAM` chunk). The values for each table\n +\t entry are as follows:\n -+\t - The first bit position (in psuedo-pack order, see below) to\n ++\t - The first bit position (in pseudo-pack order, see below) to\n +\t contain an object from that pack.\n +\t - The number of bits whose objects are selected from that pack.\n -+\t - A \"meta\" value, whose least-significant bit indicates whether or\n -+\t not the pack is disjoint with respect to other packs. The\n -+\t remaining bits are unused.\n -+\t Two packs are \"disjoint\" with respect to one another when they have\n -+\t disjoint sets of objects. In other words, any object found in a pack\n -+\t contained in the set of disjoint packfiles is guaranteed to be\n -+\t uniquely located among those packs.\n +\n \tOID Fanout (ID: {'O', 'I', 'D', 'F'})\n \t The ith entry, F[i], stores the number of OIDs with first\n @@ Documentation/gitformat-pack.txt: packs arranged in MIDX order (with the preferr\n The MIDX's reverse index is stored in the optional 'RIDX' chunk within\n the MIDX itself.\n \n -+=== `DISP` chunk and disjoint packs\n ++=== `BTMP` chunk\n +\n -+The Disjoint Packfiles (`DISP`) chunk encodes additional information\n ++The Bitmapped Packfiles (`BTMP`) chunk encodes additional information\n +about the objects in the multi-pack index's reachability bitmap. Recall\n -+that objects from the MIDX are arranged in \"pseudo-pack\" order (see:\n ++that objects from the MIDX are arranged in \"pseudo-pack\" order (see\n +above) for reachability bitmaps.\n +\n +From the example above, suppose we have packs \"a\", \"b\", and \"c\", with\n @@ Documentation/gitformat-pack.txt: packs arranged in MIDX order (with the preferr\n + |a,0|a,1|...|a,9|b,0|b,1|...|b,14|c,0|c,1|...|c,19|\n +\n +When working with single-pack bitmaps (or, equivalently, multi-pack\n -+reachability bitmaps without any packs marked as disjoint),\n -+linkgit:git-pack-objects[1] performs ``verbatim'' reuse, attempting to\n -+reuse chunks of the existing packfile instead of adding objects to the\n -+packing list.\n ++reachability bitmaps with a preferred pack), linkgit:git-pack-objects[1]\n ++performs ``verbatim'' reuse, attempting to reuse chunks of the bitmapped\n ++or preferred packfile instead of adding objects to the packing list.\n +\n -+When a chunk of bytes are reused from an existing pack, any objects\n ++When a chunk of bytes is reused from an existing pack, any objects\n +contained therein do not need to be added to the packing list, saving\n +memory and CPU time. But a chunk from an existing packfile can only be\n +reused when the following conditions are met:\n @@ Documentation/gitformat-pack.txt: packs arranged in MIDX order (with the preferr\n + (i.e. does not contain any objects which the caller didn't ask for\n + explicitly or implicitly).\n +\n -+ - All objects stored as offset- or reference-deltas also include their\n -+ base object in the resulting pack.\n ++ - All objects stored in non-thin packs as offset- or reference-deltas\n ++ also include their base object in the resulting pack.\n +\n -+Additionally, packfiles many not contain more than one copy of any given\n -+object. This introduces an additional constraint over the set of packs\n -+we may want to reuse. The most straightforward approach is to mandate\n -+that the set of packs is disjoint with respect to the set of objects\n -+contained in each pack. In other words, for each object `o` in the union\n -+of all objects stored by the disjoint set of packs, `o` is contained in\n -+exactly one pack from the disjoint set.\n -+\n -+One alternative design choice for multi-pack reuse might instead involve\n -+imposing a chunk-level constraint that allows packs in the reusable set\n -+to contain multiple copies across different packs, but restricts each\n -+chunk against including more than one copy of such an object. This is in\n -+theory possible to implement, but significantly more complicated than\n -+forcing packs themselves to be disjoint. Most notably, we would have to\n -+keep track of which objects have already been sent during verbatim\n -+pack-reuse, defeating the main purpose of verbatim pack reuse (that we\n -+don't have to keep track of individual objects).\n -+\n -+The `DISP` chunk encodes the necessary information in order to implement\n -+multi-pack reuse over a disjoint set of packs as described above.\n -+Specifically, the `DISP` chunk encodes three pieces of information (all\n ++The `BTMP` chunk encodes the necessary information in order to implement\n ++multi-pack reuse over a set of packfiles as described above.\n ++Specifically, the `BTMP` chunk encodes three pieces of information (all\n +32-bit unsigned integers in network byte-order) for each packfile `p`\n +that is stored in the MIDX, as follows:\n +\n @@ Documentation/gitformat-pack.txt: packs arranged in MIDX order (with the preferr\n +`bitmap_nr`:: The number of bit positions (including the one at\n + `bitmap_pos`) that encode objects from that pack `p`.\n +\n -+`disjoint`:: Metadata, including whether or not the pack `p` is\n -+ ``disjoint''. The least significant bit stores whether or not the pack\n -+ is disjoint. The remaining bits are reserved for future use.\n -+\n -+For example, the `DISP` chunk corresponding to the above example (with\n ++For example, the `BTMP` chunk corresponding to the above example (with\n +packs ``a'', ``b'', and ``c'') would look like:\n +\n -+[cols=\"1,2,2,2\"]\n ++[cols=\"1,2,2\"]\n +|===\n -+| |`bitmap_pos` |`bitmap_nr` |`disjoint`\n ++| |`bitmap_pos` |`bitmap_nr`\n +\n +|packfile ``a''\n +|`0`\n +|`10`\n -+|`0x1`\n +\n +|packfile ``b''\n +|`10`\n +|`15`\n -+|`0x1`\n +\n +|packfile ``c''\n +|`25`\n +|`20`\n -+|`0x1`\n +|===\n +\n -+With these constraints and information in place, we can treat each\n -+packfile marked as disjoint as individually reusable in the same fashion\n -+as verbatim pack reuse is performed on individual packs prior to the\n -+implementation of the `DISP` chunk.\n ++With this information in place, we can treat each packfile as\n ++individually reusable in the same fashion as verbatim pack reuse is\n ++performed on individual packs prior to the implementation of the `BTMP`\n ++chunk.\n +\n == cruft packs\n \n The cruft packs feature offer an alternative to Git's traditional mechanism of\n \n - ## builtin/multi-pack-index.c ##\n -@@ builtin/multi-pack-index.c: static int git_multi_pack_index_write_config(const char *var, const char *value,\n - \treturn 0;\n - }\n - \n -+#define DISJOINT ((void*)(uintptr_t)1)\n -+\n - static void read_packs_from_stdin(struct string_list *to)\n - {\n - \tstruct strbuf buf = STRBUF_INIT;\n --\twhile (strbuf_getline(&buf, stdin) != EOF)\n --\t\tstring_list_append(to, buf.buf);\n -+\twhile (strbuf_getline(&buf, stdin) != EOF) {\n -+\t\tif (*buf.buf == '+')\n -+\t\t\tstring_list_append(to, buf.buf + 1)->util = DISJOINT;\n -+\t\telse\n -+\t\t\tstring_list_append(to, buf.buf);\n -+\t}\n - \tstring_list_sort(to);\n - \n - \tstrbuf_release(&buf);\n -\n ## midx.c ##\n @@\n \n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n -+#define MIDX_CHUNKID_DISJOINTPACKS 0x44495350 /* \"DISP\" */\n ++#define MIDX_CHUNKID_BITMAPPEDPACKS 0x42544d50 /* \"BTMP\" */\n #define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n +@@\n + #define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n + #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n + #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n ++#define MIDX_CHUNK_BITMAPPED_PACKS_WIDTH (2 * sizeof(uint32_t))\n + #define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n + \n + #define PACK_EXPIRED UINT_MAX\n @@ midx.c: struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \n \tpair_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS, &m->chunk_large_offsets,\n \t\t &m->chunk_large_offsets_len);\n -+\tpair_chunk(cf, MIDX_CHUNKID_DISJOINTPACKS,\n -+\t\t (const unsigned char **)&m->chunk_disjoint_packs,\n -+\t\t &m->chunk_disjoint_packs_len);\n ++\tpair_chunk(cf, MIDX_CHUNKID_BITMAPPEDPACKS,\n ++\t\t (const unsigned char **)&m->chunk_bitmapped_packs,\n ++\t\t &m->chunk_bitmapped_packs_len);\n \n \tif (git_env_bool(\"GIT_TEST_MIDX_READ_RIDX\", 1))\n \t\tpair_chunk(cf, MIDX_CHUNKID_REVINDEX, &m->chunk_revindex,\n @@ midx.c: int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,\n +int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n +\t\t struct bitmapped_pack *bp, uint32_t pack_int_id)\n +{\n -+\tif (!m->chunk_disjoint_packs)\n -+\t\treturn error(_(\"MIDX does not contain the DISP chunk\"));\n ++\tif (!m->chunk_bitmapped_packs)\n ++\t\treturn error(_(\"MIDX does not contain the BTMP chunk\"));\n +\n +\tif (prepare_midx_pack(r, m, pack_int_id))\n -+\t\treturn error(_(\"could not load disjoint pack %\"PRIu32), pack_int_id);\n ++\t\treturn error(_(\"could not load bitmapped pack %\"PRIu32), pack_int_id);\n +\n +\tbp->p = m->packs[pack_int_id];\n -+\tbp->bitmap_pos = get_be32(m->chunk_disjoint_packs + 3 * pack_int_id);\n -+\tbp->bitmap_nr = get_be32(m->chunk_disjoint_packs + 3 * pack_int_id + 1);\n -+\tbp->disjoint = !!get_be32(m->chunk_disjoint_packs + 3 * pack_int_id + 2);\n ++\tbp->bitmap_pos = get_be32((char *)m->chunk_bitmapped_packs +\n ++\t\t\t\t MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id);\n ++\tbp->bitmap_nr = get_be32((char *)m->chunk_bitmapped_packs +\n ++\t\t\t\t MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id +\n ++\t\t\t\t sizeof(uint32_t));\n ++\tbp->pack_int_id = pack_int_id;\n +\n +\treturn 0;\n +}\n @@ midx.c: static size_t write_midx_header(struct hashfile *f,\n \tuint32_t orig_pack_int_id;\n \tchar *pack_name;\n \tstruct packed_git *p;\n --\tunsigned expired : 1;\n +\n +\tuint32_t bitmap_pos;\n +\tuint32_t bitmap_nr;\n +\n -+\tunsigned expired : 1,\n -+\t\t disjoint : 1;\n + \tunsigned expired : 1;\n };\n \n - static void fill_pack_info(struct pack_info *info,\n @@ midx.c: static void fill_pack_info(struct pack_info *info,\n \tinfo->orig_pack_int_id = orig_pack_int_id;\n - \tinfo->pack_name = pack_name;\n + \tinfo->pack_name = xstrdup(pack_name);\n \tinfo->p = p;\n +\tinfo->bitmap_pos = BITMAP_POS_UNKNOWN;\n }\n \n static int pack_info_compare(const void *_a, const void *_b)\n -@@ midx.c: static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n - {\n - \tstruct write_midx_context *ctx = data;\n - \tstruct packed_git *p;\n -+\tstruct string_list_item *item = NULL;\n - \n - \tif (ends_with(file_name, \".idx\")) {\n - \t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n -@@ midx.c: static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n - \t\t * should be performed independently (likely checking\n - \t\t * to_include before the existing MIDX).\n - \t\t */\n --\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n --\t\t\treturn;\n --\t\telse if (ctx->to_include &&\n --\t\t\t !string_list_has_string(ctx->to_include, file_name))\n -+\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name)) {\n - \t\t\treturn;\n -+\t\t} else if (ctx->to_include) {\n -+\t\t\titem = string_list_lookup(ctx->to_include, file_name);\n -+\t\t\tif (!item)\n -+\t\t\t\treturn;\n -+\t\t}\n - \n - \t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n - \n -@@ midx.c: static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n - \n - \t\tfill_pack_info(&ctx->info[ctx->nr], p, xstrdup(file_name),\n - \t\t\t ctx->nr);\n -+\t\tif (item)\n -+\t\t\tctx->info[ctx->nr].disjoint = !!item->util;\n - \t\tctx->nr++;\n - \t}\n - }\n -@@ midx.c: struct pack_midx_entry {\n - \tuint32_t pack_int_id;\n - \ttime_t pack_mtime;\n - \tuint64_t offset;\n --\tunsigned preferred : 1;\n -+\tunsigned preferred : 1,\n -+\t\t disjoint : 1;\n - };\n - \n - static int midx_oid_compare(const void *_a, const void *_b)\n -@@ midx.c: static int midx_oid_compare(const void *_a, const void *_b)\n - \tif (a->preferred < b->preferred)\n - \t\treturn 1;\n - \n -+\t/* Sort objects in a disjoint pack last when multiple copies exist. */\n -+\tif (a->disjoint < b->disjoint)\n -+\t\treturn -1;\n -+\tif (a->disjoint > b->disjoint)\n -+\t\treturn 1;\n -+\n - \tif (a->pack_mtime > b->pack_mtime)\n - \t\treturn -1;\n - \telse if (a->pack_mtime < b->pack_mtime)\n -@@ midx.c: static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n - \t\t\t\t\t &fanout->entries[fanout->nr],\n - \t\t\t\t\t cur_object);\n - \t\tfanout->entries[fanout->nr].preferred = 0;\n -+\t\tfanout->entries[fanout->nr].disjoint = 0;\n - \t\tfanout->nr++;\n - \t}\n - }\n -@@ midx.c: static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout,\n - \t\t\t\tcur_object,\n - \t\t\t\t&fanout->entries[fanout->nr],\n - \t\t\t\tpreferred);\n -+\t\tfanout->entries[fanout->nr].disjoint = info->disjoint;\n - \t\tfanout->nr++;\n - \t}\n - }\n -@@ midx.c: static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n - \t\t * Take only the first duplicate.\n - \t\t */\n - \t\tfor (cur_object = 0; cur_object < fanout.nr; cur_object++) {\n --\t\t\tif (cur_object && oideq(&fanout.entries[cur_object - 1].oid,\n --\t\t\t\t\t\t&fanout.entries[cur_object].oid))\n --\t\t\t\tcontinue;\n -+\t\t\tstruct pack_midx_entry *ours = &fanout.entries[cur_object];\n -+\t\t\tif (cur_object) {\n -+\t\t\t\tstruct pack_midx_entry *prev = &fanout.entries[cur_object - 1];\n -+\t\t\t\tif (oideq(&prev->oid, &ours->oid)) {\n -+\t\t\t\t\tif (prev->disjoint && ours->disjoint)\n -+\t\t\t\t\t\tdie(_(\"duplicate object '%s' among disjoint packs '%s', '%s'\"),\n -+\t\t\t\t\t\t oid_to_hex(&prev->oid),\n -+\t\t\t\t\t\t info[prev->pack_int_id].pack_name,\n -+\t\t\t\t\t\t info[ours->pack_int_id].pack_name);\n -+\t\t\t\t\tcontinue;\n -+\t\t\t\t}\n -+\t\t\t}\n - \n - \t\t\tALLOC_GROW(deduplicated_entries, st_add(*nr_objects, 1),\n - \t\t\t\t alloc_objects);\n --\t\t\tmemcpy(&deduplicated_entries[*nr_objects],\n --\t\t\t &fanout.entries[cur_object],\n -+\t\t\tmemcpy(&deduplicated_entries[*nr_objects], ours,\n - \t\t\t sizeof(struct pack_midx_entry));\n - \t\t\t(*nr_objects)++;\n - \t\t}\n @@ midx.c: static int write_midx_pack_names(struct hashfile *f, void *data)\n \treturn 0;\n }\n \n -+static int write_midx_disjoint_packs(struct hashfile *f, void *data)\n ++static int write_midx_bitmapped_packs(struct hashfile *f, void *data)\n +{\n +\tstruct write_midx_context *ctx = data;\n +\tsize_t i;\n @@ midx.c: static int write_midx_pack_names(struct hashfile *f, void *data)\n +\n +\t\thashwrite_be32(f, pack->bitmap_pos);\n +\t\thashwrite_be32(f, pack->bitmap_nr);\n -+\t\thashwrite_be32(f, !!pack->disjoint);\n +\t}\n +\treturn 0;\n +}\n @@ midx.c: static int write_midx_internal(const char *object_dir,\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n -+\tint pack_disjoint_concat_len = 0;\n ++\tint bitmapped_packs_concat_len = 0;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n @@ midx.c: static int write_midx_internal(const char *object_dir,\n +\t\tif (ctx.info[i].expired)\n +\t\t\tcontinue;\n +\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n -+\t\tpack_disjoint_concat_len += 3 * sizeof(uint32_t);\n ++\t\tbitmapped_packs_concat_len += 2 * sizeof(uint32_t);\n \t}\n \n \t/* Check that the preferred pack wasn't expired (if given). */\n @@ midx.c: static int write_midx_internal(const char *object_dir,\n \t\tadd_chunk(cf, MIDX_CHUNKID_REVINDEX,\n \t\t\t st_mult(ctx.entries_nr, sizeof(uint32_t)),\n \t\t\t write_midx_revindex);\n -+\t\tadd_chunk(cf, MIDX_CHUNKID_DISJOINTPACKS,\n -+\t\t\t pack_disjoint_concat_len, write_midx_disjoint_packs);\n ++\t\tadd_chunk(cf, MIDX_CHUNKID_BITMAPPEDPACKS,\n ++\t\t\t bitmapped_packs_concat_len,\n ++\t\t\t write_midx_bitmapped_packs);\n \t}\n \n \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n @@ midx.h: struct multi_pack_index {\n \n \tconst unsigned char *chunk_pack_names;\n \tsize_t chunk_pack_names_len;\n -+\tconst uint32_t *chunk_disjoint_packs;\n -+\tsize_t chunk_disjoint_packs_len;\n ++\tconst uint32_t *chunk_bitmapped_packs;\n ++\tsize_t chunk_bitmapped_packs_len;\n \tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_object_offsets;\n @@ pack-bitmap.h: typedef int (*show_reachable_fn)(\n +\tuint32_t bitmap_pos;\n +\tuint32_t bitmap_nr;\n +\n -+\tunsigned disjoint : 1;\n ++\tuint32_t pack_int_id; /* MIDX only */\n +};\n +\n struct bitmap_index *prepare_bitmap_git(struct repository *r);\n @@ t/helper/test-read-midx.c: static int read_midx_preferred_pack(const char *objec\n +\t\tprintf(\"%s\\n\", pack_basename(pack.p));\n +\t\tprintf(\" bitmap_pos: %\"PRIuMAX\"\\n\", (uintmax_t)pack.bitmap_pos);\n +\t\tprintf(\" bitmap_nr: %\"PRIuMAX\"\\n\", (uintmax_t)pack.bitmap_nr);\n -+\t\tprintf(\" disjoint: %s\\n\", pack.disjoint & 0x1 ? \"yes\" : \"no\");\n +\t}\n +\n +\tclose_midx(midx);\n @@ t/helper/test-read-midx.c: int cmd__read_midx(int argc, const char **argv)\n }\n \n ## t/t5319-multi-pack-index.sh ##\n -@@ t/t5319-multi-pack-index.sh: test_expect_success 'reader notices too-small revindex chunk' '\n - \ttest_cmp expect.err err\n +@@ t/t5319-multi-pack-index.sh: test_expect_success 'reader notices out-of-bounds fanout' '\n + \ttest_cmp expect err\n '\n \n -+test_expect_success 'disjoint packs are stored via the DISP chunk' '\n ++test_expect_success 'bitmapped packs are stored via the BTMP chunk' '\n +\ttest_when_finished \"rm -fr repo\" &&\n +\tgit init repo &&\n +\t(\n @@ t/t5319-multi-pack-index.sh: test_expect_success 'reader notices too-small revin\n +\t\t\tgit repack -d || return 1\n +\t\tdone &&\n +\n -+\t\tfind $objdir/pack -type f -name \"*.idx\" | xargs -n 1 basename | sort >packs &&\n ++\t\tfind $objdir/pack -type f -name \"*.idx\" | xargs -n 1 basename |\n ++\t\tsort >packs &&\n +\n +\t\tgit multi-pack-index write --stdin-packs err &&\n +\t\tcat >expect <<-\\EOF &&\n -+\t\terror: MIDX does not contain the DISP chunk\n ++\t\terror: MIDX does not contain the BTMP chunk\n +\t\tEOF\n +\t\ttest_cmp expect err &&\n +\n -+\t\tsed -e \"s/^/+/g\" packs >in &&\n +\t\tgit multi-pack-index write --stdin-packs --bitmap \\\n -+\t\t\t--preferred-pack=\"$(head -n1 actual &&\n +\t\tfor i in $(test_seq $(wc -l expect &&\n +\t\ttest_cmp expect actual\n +\t)\n +'\n -+\n -+test_expect_success 'non-disjoint packs are detected' '\n -+\ttest_when_finished \"rm -fr repo\" &&\n -+\tgit init repo &&\n -+\t(\n -+\t\tcd repo &&\n -+\n -+\t\ttest_commit base &&\n -+\t\tgit repack -d &&\n -+\t\ttest_commit other &&\n -+\t\tgit repack -a &&\n -+\n -+\t\tls -la .git/objects/pack/ &&\n -+\n -+\t\tfind $objdir/pack -type f -name \"*.idx\" |\n -+\t\t\tsed -e \"s/.*\\/\\(.*\\)$/+\\1/g\" >in &&\n -+\n -+\t\ttest_must_fail git multi-pack-index write --stdin-packs \\\n -+\t\t\t--bitmap err &&\n -+\t\tgrep \"duplicate object.* among disjoint packs\" err\n -+\t)\n -+'\n +\n test_done\n 6: 541fbb442b = 6: 96f397a2b2 midx: implement `midx_locate_pack()`\n 7: 3019738b52 < -: ---------- midx: implement `--retain-disjoint` mode\n 8: 0368f7ab37 < -: ---------- pack-objects: implement `--ignore-disjoint` mode\n 9: b75869befb < -: ---------- repack: implement `--extend-disjoint` mode\n10: 970bd9eaea ! 7: df9233cf0f pack-bitmap: pass `bitmapped_pack` struct to pack-reuse functions\n @@ pack-bitmap.c: int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitma\n \tunuse_pack(&w_curs);\n +}\n +\n ++static int bitmapped_pack_cmp(const void *va, const void *vb)\n ++{\n ++\tconst struct bitmapped_pack *a = va;\n ++\tconst struct bitmapped_pack *b = vb;\n ++\n ++\tif (a->bitmap_pos < b->bitmap_pos)\n ++\t\treturn -1;\n ++\tif (a->bitmap_pos > b->bitmap_pos)\n ++\t\treturn 1;\n ++\treturn 0;\n ++}\n ++\n +int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n +\t\t\t\t struct packed_git **packfile_out,\n +\t\t\t\t uint32_t *entries,\n @@ pack-bitmap.c: int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitma\n +\n +\t\t\tobjects_nr += pack.p->num_objects;\n +\t\t}\n ++\n ++\t\tQSORT(packs, packs_nr, bitmapped_pack_cmp);\n +\t} else {\n +\t\tALLOC_GROW(packs, packs_nr + 1, packs_alloc);\n +\n +\t\tpacks[packs_nr].p = bitmap_git->pack;\n +\t\tpacks[packs_nr].bitmap_pos = 0;\n +\t\tpacks[packs_nr].bitmap_nr = bitmap_git->pack->num_objects;\n -+\t\tpacks[packs_nr].disjoint = 1;\n +\n +\t\tobjects_nr = packs[packs_nr++].p->num_objects;\n +\t}\n -: ---------- > 8: 595b3b6986 ewah: implement `bitmap_is_empty()`\n11: 432854b27c ! 9: d851f821fc pack-bitmap: simplify `reuse_partial_packfile_from_bitmap()` signature\n @@ builtin/pack-objects.c: static int get_object_list_from_bitmap(struct rev_info *\n \t\tdisplay_progress(progress_state, nr_seen);\n \n ## pack-bitmap.c ##\n -@@ pack-bitmap.c: static void reuse_partial_packfile_from_bitmap_1(struct bitmap_index *bitmap_git\n - \tunuse_pack(&w_curs);\n +@@ pack-bitmap.c: static int bitmapped_pack_cmp(const void *va, const void *vb)\n + \treturn 0;\n }\n \n -int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n @@ pack-bitmap.c: int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitma\n \n -\t*entries = bitmap_popcount(reuse);\n -\tif (!*entries) {\n -+\tif (!bitmap_popcount(reuse)) {\n ++\tif (bitmap_is_empty(reuse)) {\n +\t\tfree(packs);\n \t\tbitmap_free(reuse);\n -\t\treturn -1;\n12: 36b794d9e1 ! 10: f551892bab pack-bitmap: return multiple packs via `reuse_partial_packfile_from_bitmap()`\n @@ builtin/pack-objects.c: static int pack_options_allow_reuse(void)\n \t\t\tBUG(\"expected non-empty reuse bitmap\");\n \n ## pack-bitmap.c ##\n -@@ pack-bitmap.c: static void reuse_partial_packfile_from_bitmap_1(struct bitmap_index *bitmap_git\n +@@ pack-bitmap.c: static int bitmapped_pack_cmp(const void *va, const void *vb)\n }\n \n void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n13: dca1083d8e = 11: 67e4fd8a06 pack-objects: parameterize pack-reuse routines over a single pack\n14: 6f4fba861b ! 12: 9a5c38514b pack-objects: keep track of `pack_start` for each reuse pack\n @@ Commit message\n In order to compute this value correctly, we need to know not only where\n we are in the packfile we're assembling (with `hashfile_total(f)`) but\n also the position of the first byte of the packfile that we are\n - currently reusing.\n + currently reusing. Currently, this works just fine, since when reusing\n + only a single pack those two values are always identical (because\n + verbatim reuse is the first thing pack-objects does when enabled after\n + writing the pack header).\n +\n + But when reusing multiple packs which have one or more gaps, we'll need\n + to account for these two values diverging.\n \n Together, these two allow us to compute the reused chunk's offset\n difference relative to the start of the reused pack, as desired.\n15: 2bb01e2370 = 13: 5492d11f25 pack-objects: pass `bitmapped_pack`'s to pack-reuse functions\n16: 67a8196978 ! 14: b32742ebcb pack-objects: prepare `write_reused_pack()` for multi-pack reuse\n @@ Commit message\n \n Prepare this function for multi-pack reuse by removing the assumption\n that the bit position corresponding to the first object being reused\n - from a given pack may not be at bit position zero.\n + from a given pack must be at bit position zero.\n \n The changes in this function are mostly straightforward. Initialize `i`\n to the position of the first word to contain bits corresponding to that\n17: 1f766648df = 15: 805c42185a pack-objects: prepare `write_reused_pack_verbatim()` for multi-pack reuse\n18: 4cd9f99bfd ! 16: 55696bc1c9 pack-objects: include number of packs reused in output\n @@ builtin/pack-objects.c: int cmd_pack_objects(int argc, const char **argv, const\n +\t\t\t (uintmax_t)reuse_packfiles_used_nr);\n \n cleanup:\n - \tfree_packing_data(&to_pack);\n + \tclear_packing_data(&to_pack);\n19: 176c4c95ac < -: ---------- pack-bitmap: prepare to mark objects from multiple packs for reuse\n -: ---------- > 17: 6ede9e0603 git-compat-util.h: implement checked size_t to uint32_t conversion\n -: ---------- > 18: ab0456a71e midx: implement `midx_preferred_pack()`\n -: ---------- > 19: 14b054d272 pack-revindex: factor out `midx_key_to_pack_pos()` helper\n -: ---------- > 20: e99481014e pack-revindex: implement `midx_pair_to_pack_pos()`\n -: ---------- > 21: 3e3625aebe pack-bitmap: prepare to mark objects from multiple packs for reuse\n20: 46f1a03b8b ! 22: 1723cd0384 pack-objects: add tracing for various packfile metrics\n @@ builtin/pack-objects.c: int cmd_pack_objects(int argc, const char **argv, const\n +\ttrace2_data_intmax(\"pack-objects\", the_repository, \"packs-reused\", reuse_packfiles_used_nr);\n +\n cleanup:\n - \tfree_packing_data(&to_pack);\n + \tclear_packing_data(&to_pack);\n \tlist_objects_filter_release(&filter_options);\n21: fb764fbacc = 23: 79c830e37a t/test-lib-functions.sh: implement `test_trace2_data` helper\n22: 3140a1703a ! 24: d06b0961b5 pack-objects: allow setting `pack.allowPackReuse` to \"single\"\n @@ Commit message\n \"single\" implies the same behavior as \"true\", \"1\", \"yes\", and so on. But\n it will complement a new \"multi\" value (to be introduced in a future\n commit). When set to \"single\", we will only perform pack reuse on a\n - single pack, regardless of whether or not there are multiple disjoint\n + single pack, regardless of whether or not there are multiple MIDX'd\n packs.\n \n This requires no code changes (yet), since we only support single pack\n23: 7345e39467 ! 25: 7002cf08fe pack-bitmap: reuse objects from all disjoint packs\n @@ Metadata\n Author: Taylor Blau \n \n ## Commit message ##\n - pack-bitmap: reuse objects from all disjoint packs\n + pack-bitmap: enable reuse from all bitmapped packs\n \n Now that both the pack-bitmap and pack-objects code are prepared to\n - handle marking and using objects from multiple disjoint packs for\n - verbatim reuse, allow marking objects from all disjoint packs as\n + handle marking and using objects from multiple bitmapped packs for\n + verbatim reuse, allow marking objects from all bitmapped packs as\n eligible for reuse.\n \n Within the `reuse_partial_packfile_from_bitmap()` function, we no longer\n only mark the pack whose first object is at bit position zero for reuse,\n - and instead mark any pack which is flagged as disjoint by the MIDX as a\n - reuse candidate. If no such packs exist (i.e because we are reading a\n - MIDX written before the \"DISP\" chunk was introduced), then treat the\n - preferred pack as disjoint for the purposes of reuse. This is a safe\n - assumption to make since all duplicate objects are resolved in favor of\n - the preferred pack.\n + and instead mark any pack contained in the MIDX as a reuse candidate.\n \n Provide a handful of test cases in a new script (t5332) exercising\n interesting behavior for multi-pack reuse to ensure that we performed\n @@ Commit message\n Signed-off-by: Taylor Blau \n \n ## Documentation/config/pack.txt ##\n -@@ Documentation/config/pack.txt: to linkgit:git-repack[1].\n +@@ Documentation/config/pack.txt: all existing objects. You can force recompression by passing the -F option\n + to linkgit:git-repack[1].\n + \n pack.allowPackReuse::\n - \tWhen true or \"single\", and when reachability bitmaps are enabled,\n - \tpack-objects will try to send parts of the bitmapped packfile\n +-\tWhen true or \"single\", and when reachability bitmaps are enabled,\n +-\tpack-objects will try to send parts of the bitmapped packfile\n -\tverbatim. This can reduce memory and CPU usage to serve fetches,\n -+\tverbatim. When \"multi\", and when a multi-pack reachability bitmap is\n -+\tavailable, pack-objects will try to send parts of all packs marked as\n -+\tdisjoint by the MIDX. If only a single pack bitmap is available, and\n +-\tbut might result in sending a slightly larger pack. Defaults to\n +-\ttrue.\n ++\tWhen true or \"single\", and when reachability bitmaps are\n ++\tenabled, pack-objects will try to send parts of the bitmapped\n ++\tpackfile verbatim. When \"multi\", and when a multi-pack\n ++\treachability bitmap is available, pack-objects will try to send\n ++\tparts of all packs in the MIDX.\n +++\n ++\tIf only a single pack bitmap is available, and\n +\t`pack.allowPackReuse` is set to \"multi\", reuse parts of just the\n -+\tbitmapped packfile. This can reduce memory and CPU usage to serve fetches,\n - \tbut might result in sending a slightly larger pack. Defaults to\n - \ttrue.\n ++\tbitmapped packfile. This can reduce memory and CPU usage to\n ++\tserve fetches, but might result in sending a slightly larger\n ++\tpack. Defaults to true.\n \n + pack.island::\n + \tAn extended regular expression configuring a set of delta\n \n ## builtin/pack-objects.c ##\n @@ builtin/pack-objects.c: static int use_bitmap_index = -1;\n @@ builtin/pack-objects.c: static int get_object_list_from_bitmap(struct rev_info *\n \t\treuse_packfile_objects = bitmap_popcount(reuse_packfile_bitmap);\n \n ## pack-bitmap.c ##\n -@@ pack-bitmap.c: static void reuse_partial_packfile_from_bitmap_1(struct bitmap_index *bitmap_git\n - \tunuse_pack(&w_curs);\n - }\n - \n -+static void make_disjoint_pack(struct bitmapped_pack *out, struct packed_git *p)\n -+{\n -+\tout->p = p;\n -+\tout->bitmap_pos = 0;\n -+\tout->bitmap_nr = p->num_objects;\n -+\tout->disjoint = 1;\n -+}\n -+\n +@@ pack-bitmap.c: static int bitmapped_pack_cmp(const void *va, const void *vb)\n void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\t\tstruct bitmapped_pack **packs_out,\n \t\t\t\t\tsize_t *packs_nr_out,\n @@ pack-bitmap.c: void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitm\n \t\t\t\tfree(packs);\n \t\t\t\treturn;\n \t\t\t}\n --\t\t\tif (!pack.bitmap_nr)\n ++\n + \t\t\tif (!pack.bitmap_nr)\n -\t\t\t\tcontinue; /* no objects from this pack */\n -\t\t\tif (pack.bitmap_pos)\n -\t\t\t\tcontinue; /* not preferred pack */\n -+\n -+\t\t\tif (!pack.disjoint)\n +\t\t\t\tcontinue;\n +\n +\t\t\tif (!multi_pack_reuse && pack.bitmap_pos) {\n @@ pack-bitmap.c: void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitm\n +\n +\t\t\tif (!multi_pack_reuse)\n +\t\t\t\tbreak;\n -+\t\t}\n -+\n -+\t\tif (!packs_nr) {\n -+\t\t\t/*\n -+\t\t\t * Old MIDXs (i.e. those written before the \"DISP\" chunk\n -+\t\t\t * existed) will not have any packs marked as disjoint.\n -+\t\t\t *\n -+\t\t\t * But we still want to perform pack reuse with the\n -+\t\t\t * special \"preferred pack\" as before. To do this, form\n -+\t\t\t * the singleton set containing just the preferred pack,\n -+\t\t\t * which is trivially disjoint with itself.\n -+\t\t\t *\n -+\t\t\t * Moreover, the MIDX is guaranteed to resolve duplicate\n -+\t\t\t * objects in favor of the copy in the preferred pack\n -+\t\t\t * (if one exists). Thus, we can safely perform pack\n -+\t\t\t * reuse on this pack.\n -+\t\t\t */\n -+\t\t\tuint32_t preferred_pack_pos;\n -+\t\t\tstruct packed_git *preferred_pack;\n -+\n -+\t\t\tpreferred_pack_pos = midx_preferred_pack(bitmap_git);\n -+\t\t\tpreferred_pack = bitmap_git->midx->packs[preferred_pack_pos];\n -+\n -+\t\t\tALLOC_GROW(packs, packs_nr + 1, packs_alloc);\n -+\n -+\t\t\tmake_disjoint_pack(&packs[packs_nr], preferred_pack);\n -+\t\t\tobjects_nr = packs[packs_nr++].p->num_objects;\n \t\t}\n - \t} else {\n + \n + \t\tQSORT(packs, packs_nr, bitmapped_pack_cmp);\n +@@ pack-bitmap.c: void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\tALLOC_GROW(packs, packs_nr + 1, packs_alloc);\n \n --\t\tpacks[packs_nr].p = bitmap_git->pack;\n + \t\tpacks[packs_nr].p = bitmap_git->pack;\n -\t\tpacks[packs_nr].bitmap_pos = 0;\n --\t\tpacks[packs_nr].bitmap_nr = bitmap_git->pack->num_objects;\n --\t\tpacks[packs_nr].disjoint = 1;\n --\n -+\t\tmake_disjoint_pack(&packs[packs_nr], bitmap_git->pack);\n - \t\tobjects_nr = packs[packs_nr++].p->num_objects;\n + \t\tpacks[packs_nr].bitmap_nr = bitmap_git->pack->num_objects;\n ++\t\tpacks[packs_nr].bitmap_pos = 0;\n + \n +-\t\tobjects_nr = packs[packs_nr++].p->num_objects;\n ++\t\tobjects_nr = packs[packs_nr++].bitmap_nr;\n \t}\n \n + \tword_alloc = objects_nr / BITS_IN_EWORD;\n @@ pack-bitmap.c: void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\tword_alloc++;\n \treuse = bitmap_word_alloc(word_alloc);\n @@ pack-bitmap.c: void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitm\n +\tfor (i = 0; i < packs_nr; i++)\n +\t\treuse_partial_packfile_from_bitmap_1(bitmap_git, &packs[i], reuse);\n \n - \tif (!bitmap_popcount(reuse)) {\n + \tif (bitmap_is_empty(reuse)) {\n \t\tfree(packs);\n \n ## pack-bitmap.h ##\n -@@ pack-bitmap.h: uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git);\n +@@ pack-bitmap.h: struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\t\tstruct bitmapped_pack **packs_out,\n \t\t\t\t\tsize_t *packs_nr_out,\n @@ t/t5332-multi-pack-reuse.sh (new)\n +\n +. ./test-lib.sh\n +. \"$TEST_DIRECTORY\"/lib-bitmap.sh\n -+. \"$TEST_DIRECTORY\"/lib-disjoint.sh\n +\n +objdir=.git/objects\n +packdir=$objdir/pack\n +\n -+all_packs () {\n -+\tfind $packdir -type f -name \"*.idx\" | sed -e 's/^.*\\/\\([^\\/]\\)/\\1/g'\n -+}\n -+\n -+all_disjoint () {\n -+\tall_packs | sed -e 's/^/+/g'\n -+}\n -+\n +test_pack_reused () {\n +\ttest_trace2_data pack-objects pack-reused \"$1\"\n +}\n @@ t/t5332-multi-pack-reuse.sh (new)\n +\tgrep \"$1\" objects | cut -d\" \" -f1\n +}\n +\n -+test_expect_success 'setup' '\n ++test_expect_success 'preferred pack is reused for single-pack reuse' '\n ++\ttest_config pack.allowPackReuse single &&\n ++\n ++\tfor i in A B\n ++\tdo\n ++\t\ttest_commit \"$i\" &&\n ++\t\tgit repack -d || return 1\n ++\tdone &&\n ++\n ++\tgit multi-pack-index write --bitmap &&\n ++\n ++\t: >trace2.txt &&\n ++\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n ++\t\tgit pack-objects --stdout --revs --all >/dev/null &&\n ++\n ++\ttest_pack_reused 3 trace2.txt &&\n -+\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n -+\t\tgit pack-objects --stdout --revs --all >/dev/null &&\n -+\n -+\ttest_pack_reused 3 in <<-EOF &&\n -+\tpack-$A.idx\n -+\t+pack-$B.idx\n -+\t+pack-$C.idx\n ++\t$(git rev-parse C)\n ++\t^$(git rev-parse A)\n +\tEOF\n -+\tgit multi-pack-index write --bitmap --stdin-packs trace2.txt &&\n +\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n -+\t\tgit pack-objects --stdout --revs --all >/dev/null &&\n ++\t\tgit pack-objects --stdout --revs /dev/null &&\n +\n +\ttest_pack_reused 6 in &&\n -+\tgit multi-pack-index write --bitmap --stdin-packs trace2.txt &&\n +\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n +\t\tgit pack-objects --stdout --revs --all >/dev/null &&\n @@ t/t5332-multi-pack-reuse.sh (new)\n +\ttest_packs_reused 3 in &&\n -+\tgit multi-pack-index write --bitmap --preferred-pack=\"pack-$D.idx\" \\\n -+\t\t--stdin-packs in <<-EOF &&\n +\t$(git rev-parse E)\n @@ t/t5332-multi-pack-reuse.sh (new)\n +\ttest_packs_reused 1 in &&\n -+\tgit multi-pack-index write --bitmap --preferred-pack=\"pack-$A.idx\" \\\n -+\t\t--stdin-packs in <<-EOF &&\n +\t$(git rev-parse E)\n @@ t/t5332-multi-pack-reuse.sh (new)\n +\ttest_packs_reused 1 f &&\n @@ t/t5332-multi-pack-reuse.sh (new)\n +\n +\thave_delta \"$(git rev-parse $delta:f)\" \"$(git rev-parse $base:f)\" &&\n +\n -+\tall_disjoint >in &&\n -+\tgit multi-pack-index write --bitmap --stdin-packs in <<-EOF &&\n +\t$(git rev-parse other)\n @@ t/t5332-multi-pack-reuse.sh (new)\n +\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n +\t\tgit pack-objects --stdout --delta-base-offset --revs /dev/null &&\n +\n -+\t# Even though all packs are marked disjoint, we can only reuse\n -+\t# the 3 objects corresponding to \"other\" from the latest pack.\n ++\t# We can only reuse the 3 objects corresponding to \"other\" from\n ++\t# the latest pack.\n +\t#\n +\t# This is because even though we want \"delta\", we do not want\n +\t# \"base\", meaning that we have to inflate the delta/base-pair\n @@ t/t5332-multi-pack-reuse.sh (new)\n +\ttest_packs_reused 1 in <<-EOF &&\n ++\t$(git rev-parse $base)\n ++\t^$(git rev-parse $delta)\n ++\tEOF\n ++\n ++\tP=\"$(git pack-objects --revs $packdir/pack trace2.txt &&\n ++\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n ++\t\tgit pack-objects --stdout --delta-base-offset --all >/dev/null &&\n ++\n ++\tpacks_nr=\"$(find $packdir -type f -name \"pack-*.pack\" | wc -l)\" &&\n ++\tobjects_nr=\"$(git rev-list --count --all --objects)\" &&\n ++\n ++\ttest_pack_reused $(($objects_nr - 1)) \n---\n builtin/pack-objects.c | 1 +\n midx.c | 5 +++++\n pack-objects.c | 15 +++++++++++++++\n pack-objects.h | 1 +\n 4 files changed, 22 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 89a8b5a976..321d7effb0 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4522,6 +4522,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t reuse_packfile_objects);\n \n cleanup:\n+\tclear_packing_data(&to_pack);\n \tlist_objects_filter_release(&filter_options);\n \tstrvec_clear(&rp);\n \ndiff --git a/midx.c b/midx.c\nindex 1d14661dad..778dd536c8 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1603,8 +1603,13 @@ static int write_midx_internal(const char *object_dir,\n \t\t\t\t flags) < 0) {\n \t\t\terror(_(\"could not write multi-pack bitmap\"));\n \t\t\tresult = 1;\n+\t\t\tclear_packing_data(&pdata);\n+\t\t\tfree(commits);\n \t\t\tgoto cleanup;\n \t\t}\n+\n+\t\tclear_packing_data(&pdata);\n+\t\tfree(commits);\n \t}\n \t/*\n \t * NOTE: Do not use ctx.entries beyond this point, since it might\ndiff --git a/pack-objects.c b/pack-objects.c\nindex f403ca6986..a9d9855063 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -151,6 +151,21 @@ void prepare_packing_data(struct repository *r, struct packing_data *pdata)\n \tinit_recursive_mutex(&pdata->odb_lock);\n }\n \n+void clear_packing_data(struct packing_data *pdata)\n+{\n+\tif (!pdata)\n+\t\treturn;\n+\n+\tfree(pdata->cruft_mtime);\n+\tfree(pdata->in_pack);\n+\tfree(pdata->in_pack_by_idx);\n+\tfree(pdata->in_pack_pos);\n+\tfree(pdata->index);\n+\tfree(pdata->layer);\n+\tfree(pdata->objects);\n+\tfree(pdata->tree_depth);\n+}\n+\n struct object_entry *packlist_alloc(struct packing_data *pdata,\n \t\t\t\t const struct object_id *oid)\n {\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 0d78db40cb..b9898a4e64 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -169,6 +169,7 @@ struct packing_data {\n };\n \n void prepare_packing_data(struct repository *r, struct packing_data *pdata);\n+void clear_packing_data(struct packing_data *pdata);\n \n /* Protect access to object database */\n static inline void packing_data_lock(struct packing_data *pdata)\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485672","messageId":"19cdaf59c5e8c2aa58b757f7013ccb4ba1cc7f98.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 02/26] pack-bitmap-write: deep-clear the `bb_commit` slab","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:23:42Z","receivedAt":"2023-12-14T22:23:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `bb_commit` commit slab is used by the pack-bitmap-write machinery\nto track various pieces of bookkeeping used to generate reachability\nbitmaps.\n\nEven though we clear the slab when freeing the bitmap_builder struct\n(with `bitmap_builder_clear()`), there are still pointers which point to\nlocations in memory that have not yet been freed, resulting in a leak.\n\nPlug the leak by introducing a suitable `free_fn` for the `struct\nbb_commit` type, and make sure it is called on each member of the slab\nvia the `deep_clear_bb_data()` function.\n\nNote that it is possible for both of the arguments to `bitmap_free()` to\nbe NULL, but `bitmap_free()` is a noop for NULL arguments, so it is OK\nto pass them unconditionally.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex f4ecdf8b0e..ae37fb6976 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -198,6 +198,13 @@ struct bb_commit {\n \tunsigned idx; /* within selected array */\n };\n \n+static void clear_bb_commit(struct bb_commit *commit)\n+{\n+\tfree_commit_list(commit->reverse_edges);\n+\tbitmap_free(commit->commit_mask);\n+\tbitmap_free(commit->bitmap);\n+}\n+\n define_commit_slab(bb_data, struct bb_commit);\n \n struct bitmap_builder {\n@@ -339,7 +346,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \n static void bitmap_builder_clear(struct bitmap_builder *bb)\n {\n-\tclear_bb_data(&bb->data);\n+\tdeep_clear_bb_data(&bb->data, clear_bb_commit);\n \tfree(bb->commits);\n \tbb->commits_nr = bb->commits_alloc = 0;\n }\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485673","messageId":"477df6c974bf5ba7bf91d2f720e9de4f0e91f246.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 03/26] pack-bitmap: plug leak in find_objects()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:23:45Z","receivedAt":"2023-12-14T22:23:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `find_objects()` function creates an object_list for any tips of the\nreachability query which do not have corresponding bitmaps.\n\nThe object_list is not used outside of `find_objects()`, but we never\nfree it with `object_list_free()`, resulting in a leak. Let's plug that\nleak by calling `object_list_free()`, which results in t6113 becoming\nleak-free.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 2 ++\n t/t6113-rev-list-bitmap-filters.sh | 2 ++\n 2 files changed, 4 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 0260890341..d2f1306960 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1280,6 +1280,8 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\tbase = fill_in_bitmap(bitmap_git, revs, base, seen);\n \t}\n \n+\tobject_list_free(¬_mapped);\n+\n \treturn base;\n }\n \ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex 86c70521f1..459f0d7412 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -4,6 +4,8 @@ test_description='rev-list combining bitmaps and filters'\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-bitmap.sh\n \n+TEST_PASSES_SANITIZE_LEAK=true\n+\n test_expect_success 'set up bitmapped repo' '\n \t# one commit will have bitmaps, the other will not\n \ttest_commit one &&\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485696","messageId":"a06ed75af1276d9467ca2e5a414407c353412ddf.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 04/26] midx: factor out `fill_pack_info()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:23:48Z","receivedAt":"2023-12-14T22:23:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When selecting which packfiles will be written while generating a MIDX,\nthe MIDX internals fill out a 'struct pack_info' with various pieces of\nbook-keeping.\n\nInstead of filling out each field of the `pack_info` structure\nindividually in each of the two spots that modify the array of such\nstructures (`ctx->info`), extract a common routine that does this for\nus.\n\nThis reduces the code duplication by a modest amount. But more\nimportantly, it zero-initializes the structure before assigning values\ninto it. This hardens us for a future change which will add additional\nfields to this structure which (until this patch) was not\nzero-initialized.\n\nAs a result, any new fields added to the `pack_info` structure need only\nbe updated in a single location, instead of at each spot within midx.c.\n\nThere are no functional changes in this patch.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 38 ++++++++++++++++++++------------------\n 1 file changed, 20 insertions(+), 18 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 778dd536c8..8dba67ddbe 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -475,6 +475,17 @@ struct pack_info {\n \tunsigned expired : 1;\n };\n \n+static void fill_pack_info(struct pack_info *info,\n+\t\t\t struct packed_git *p, const char *pack_name,\n+\t\t\t uint32_t orig_pack_int_id)\n+{\n+\tmemset(info, 0, sizeof(struct pack_info));\n+\n+\tinfo->orig_pack_int_id = orig_pack_int_id;\n+\tinfo->pack_name = xstrdup(pack_name);\n+\tinfo->p = p;\n+}\n+\n static int pack_info_compare(const void *_a, const void *_b)\n {\n \tstruct pack_info *a = (struct pack_info *)_a;\n@@ -515,6 +526,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\t\t const char *file_name, void *data)\n {\n \tstruct write_midx_context *ctx = data;\n+\tstruct packed_git *p;\n \n \tif (ends_with(file_name, \".idx\")) {\n \t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n@@ -541,27 +553,22 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \n \t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n \n-\t\tctx->info[ctx->nr].p = add_packed_git(full_path,\n-\t\t\t\t\t\t full_path_len,\n-\t\t\t\t\t\t 0);\n-\n-\t\tif (!ctx->info[ctx->nr].p) {\n+\t\tp = add_packed_git(full_path, full_path_len, 0);\n+\t\tif (!p) {\n \t\t\twarning(_(\"failed to add packfile '%s'\"),\n \t\t\t\tfull_path);\n \t\t\treturn;\n \t\t}\n \n-\t\tif (open_pack_index(ctx->info[ctx->nr].p)) {\n+\t\tif (open_pack_index(p)) {\n \t\t\twarning(_(\"failed to open pack-index '%s'\"),\n \t\t\t\tfull_path);\n-\t\t\tclose_pack(ctx->info[ctx->nr].p);\n-\t\t\tFREE_AND_NULL(ctx->info[ctx->nr].p);\n+\t\t\tclose_pack(p);\n+\t\t\tfree(p);\n \t\t\treturn;\n \t\t}\n \n-\t\tctx->info[ctx->nr].pack_name = xstrdup(file_name);\n-\t\tctx->info[ctx->nr].orig_pack_int_id = ctx->nr;\n-\t\tctx->info[ctx->nr].expired = 0;\n+\t\tfill_pack_info(&ctx->info[ctx->nr], p, file_name, ctx->nr);\n \t\tctx->nr++;\n \t}\n }\n@@ -1321,11 +1328,6 @@ static int write_midx_internal(const char *object_dir,\n \t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n \t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n \n-\t\t\tctx.info[ctx.nr].orig_pack_int_id = i;\n-\t\t\tctx.info[ctx.nr].pack_name = xstrdup(ctx.m->pack_names[i]);\n-\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n-\t\t\tctx.info[ctx.nr].expired = 0;\n-\n \t\t\tif (flags & MIDX_WRITE_REV_INDEX) {\n \t\t\t\t/*\n \t\t\t\t * If generating a reverse index, need to have\n@@ -1341,10 +1343,10 @@ static int write_midx_internal(const char *object_dir,\n \t\t\t\tif (open_pack_index(ctx.m->packs[i]))\n \t\t\t\t\tdie(_(\"could not open index for %s\"),\n \t\t\t\t\t ctx.m->packs[i]->pack_name);\n-\t\t\t\tctx.info[ctx.nr].p = ctx.m->packs[i];\n \t\t\t}\n \n-\t\t\tctx.nr++;\n+\t\t\tfill_pack_info(&ctx.info[ctx.nr++], ctx.m->packs[i],\n+\t\t\t\t ctx.m->pack_names[i], i);\n \t\t}\n \t}\n \n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485677","messageId":"6fdc68418f196ad4d35866321760c3e4629c7ff7.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 05/26] midx: implement `BTMP` chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:23:51Z","receivedAt":"2023-12-14T22:23:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When a multi-pack bitmap is used to implement verbatim pack reuse (that\nis, when verbatim chunks from an on-disk packfile are copied\ndirectly[^1]), it does so by using its \"preferred pack\" as the source\nfor pack-reuse.\n\nThis allows repositories to pack the majority of their objects into a\nsingle (often large) pack, and then use it as the single source for\nverbatim pack reuse. This increases the amount of objects that are\nreused verbatim (and consequently, decrease the amount of time it takes\nto generate many packs). But this performance comes at a cost, which is\nthat the preferred packfile must pace its growth with that of the entire\nrepository in order to maintain the utility of verbatim pack reuse.\n\nAs repositories grow beyond what we can reasonably store in a single\npackfile, the utility of verbatim pack reuse diminishes. Or, at the very\nleast, it becomes increasingly more expensive to maintain as the pack\ngrows larger and larger.\n\nIt would be beneficial to be able to perform this same optimization over\nmultiple packs, provided some modest constraints (most importantly, that\nthe set of packs eligible for verbatim reuse are disjoint with respect\nto the subset of their objects being sent).\n\nIf we assume that the packs which we treat as candidates for verbatim\nreuse are disjoint with respect to any of their objects we may output,\nwe need to make only modest modifications to the verbatim pack-reuse\ncode itself. Most notably, we need to remove the assumption that the\nbits in the reachability bitmap corresponding to objects from the single\nreuse pack begin at the first bit position.\n\nFuture patches will unwind these assumptions and reimplement their\nexisting functionality as special cases of the more general assumptions\n(e.g. that reuse bits can start anywhere within the bitset, but happen\nto start at 0 for all existing cases).\n\nThis patch does not yet relax any of those assumptions. Instead, it\nimplements a foundational data-structure, the \"Bitampped Packs\" (`BTMP`)\nchunk of the multi-pack index. The `BTMP` chunk's contents are described\nin detail here. Importantly, the `BTMP` chunk contains information to\nmap regions of a multi-pack index's reachability bitmap to the packs\nwhose objects they represent.\n\nFor now, this chunk is only written, not read (outside of the test-tool\nused in this patch to test the new chunk's behavior). Future patches\nwill begin to make use of this new chunk.\n\n[^1]: Modulo patching any `OFS_DELTA`'s that cross over a region of the\n pack that wasn't used verbatim.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/gitformat-pack.txt | 76 ++++++++++++++++++++++++++++++++\n midx.c | 75 +++++++++++++++++++++++++++++--\n midx.h | 5 +++\n pack-bitmap.h | 9 ++++\n t/helper/test-read-midx.c | 30 ++++++++++++-\n t/t5319-multi-pack-index.sh | 35 +++++++++++++++\n 6 files changed, 226 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/gitformat-pack.txt b/Documentation/gitformat-pack.txt\nindex 9fcb29a9c8..d6ae229be5 100644\n--- a/Documentation/gitformat-pack.txt\n+++ b/Documentation/gitformat-pack.txt\n@@ -396,6 +396,15 @@ CHUNK DATA:\n \t is padded at the end with between 0 and 3 NUL bytes to make the\n \t chunk size a multiple of 4 bytes.\n \n+\tBitmapped Packfiles (ID: {'B', 'T', 'M', 'P'})\n+\t Stores a table of two 4-byte unsigned integers in network order.\n+\t Each table entry corresponds to a single pack (in the order that\n+\t they appear above in the `PNAM` chunk). The values for each table\n+\t entry are as follows:\n+\t - The first bit position (in pseudo-pack order, see below) to\n+\t contain an object from that pack.\n+\t - The number of bits whose objects are selected from that pack.\n+\n \tOID Fanout (ID: {'O', 'I', 'D', 'F'})\n \t The ith entry, F[i], stores the number of OIDs with first\n \t byte at most i. Thus F[255] stores the total\n@@ -509,6 +518,73 @@ packs arranged in MIDX order (with the preferred pack coming first).\n The MIDX's reverse index is stored in the optional 'RIDX' chunk within\n the MIDX itself.\n \n+=== `BTMP` chunk\n+\n+The Bitmapped Packfiles (`BTMP`) chunk encodes additional information\n+about the objects in the multi-pack index's reachability bitmap. Recall\n+that objects from the MIDX are arranged in \"pseudo-pack\" order (see\n+above) for reachability bitmaps.\n+\n+From the example above, suppose we have packs \"a\", \"b\", and \"c\", with\n+10, 15, and 20 objects, respectively. In pseudo-pack order, those would\n+be arranged as follows:\n+\n+ |a,0|a,1|...|a,9|b,0|b,1|...|b,14|c,0|c,1|...|c,19|\n+\n+When working with single-pack bitmaps (or, equivalently, multi-pack\n+reachability bitmaps with a preferred pack), linkgit:git-pack-objects[1]\n+performs ``verbatim'' reuse, attempting to reuse chunks of the bitmapped\n+or preferred packfile instead of adding objects to the packing list.\n+\n+When a chunk of bytes is reused from an existing pack, any objects\n+contained therein do not need to be added to the packing list, saving\n+memory and CPU time. But a chunk from an existing packfile can only be\n+reused when the following conditions are met:\n+\n+ - The chunk contains only objects which were requested by the caller\n+ (i.e. does not contain any objects which the caller didn't ask for\n+ explicitly or implicitly).\n+\n+ - All objects stored in non-thin packs as offset- or reference-deltas\n+ also include their base object in the resulting pack.\n+\n+The `BTMP` chunk encodes the necessary information in order to implement\n+multi-pack reuse over a set of packfiles as described above.\n+Specifically, the `BTMP` chunk encodes three pieces of information (all\n+32-bit unsigned integers in network byte-order) for each packfile `p`\n+that is stored in the MIDX, as follows:\n+\n+`bitmap_pos`:: The first bit position (in pseudo-pack order) in the\n+ multi-pack index's reachability bitmap occupied by an object from `p`.\n+\n+`bitmap_nr`:: The number of bit positions (including the one at\n+ `bitmap_pos`) that encode objects from that pack `p`.\n+\n+For example, the `BTMP` chunk corresponding to the above example (with\n+packs ``a'', ``b'', and ``c'') would look like:\n+\n+[cols=\"1,2,2\"]\n+|===\n+| |`bitmap_pos` |`bitmap_nr`\n+\n+|packfile ``a''\n+|`0`\n+|`10`\n+\n+|packfile ``b''\n+|`10`\n+|`15`\n+\n+|packfile ``c''\n+|`25`\n+|`20`\n+|===\n+\n+With this information in place, we can treat each packfile as\n+individually reusable in the same fashion as verbatim pack reuse is\n+performed on individual packs prior to the implementation of the `BTMP`\n+chunk.\n+\n == cruft packs\n \n The cruft packs feature offer an alternative to Git's traditional mechanism of\ndiff --git a/midx.c b/midx.c\nindex 8dba67ddbe..de25612b0c 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -33,6 +33,7 @@\n \n #define MIDX_CHUNK_ALIGNMENT 4\n #define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n+#define MIDX_CHUNKID_BITMAPPEDPACKS 0x42544d50 /* \"BTMP\" */\n #define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n #define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n@@ -41,6 +42,7 @@\n #define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n+#define MIDX_CHUNK_BITMAPPED_PACKS_WIDTH (2 * sizeof(uint32_t))\n #define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n \n #define PACK_EXPIRED UINT_MAX\n@@ -193,6 +195,9 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \n \tpair_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS, &m->chunk_large_offsets,\n \t\t &m->chunk_large_offsets_len);\n+\tpair_chunk(cf, MIDX_CHUNKID_BITMAPPEDPACKS,\n+\t\t (const unsigned char **)&m->chunk_bitmapped_packs,\n+\t\t &m->chunk_bitmapped_packs_len);\n \n \tif (git_env_bool(\"GIT_TEST_MIDX_READ_RIDX\", 1))\n \t\tpair_chunk(cf, MIDX_CHUNKID_REVINDEX, &m->chunk_revindex,\n@@ -286,6 +291,26 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t\n \treturn 0;\n }\n \n+int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n+\t\t struct bitmapped_pack *bp, uint32_t pack_int_id)\n+{\n+\tif (!m->chunk_bitmapped_packs)\n+\t\treturn error(_(\"MIDX does not contain the BTMP chunk\"));\n+\n+\tif (prepare_midx_pack(r, m, pack_int_id))\n+\t\treturn error(_(\"could not load bitmapped pack %\"PRIu32), pack_int_id);\n+\n+\tbp->p = m->packs[pack_int_id];\n+\tbp->bitmap_pos = get_be32((char *)m->chunk_bitmapped_packs +\n+\t\t\t\t MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id);\n+\tbp->bitmap_nr = get_be32((char *)m->chunk_bitmapped_packs +\n+\t\t\t\t MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id +\n+\t\t\t\t sizeof(uint32_t));\n+\tbp->pack_int_id = pack_int_id;\n+\n+\treturn 0;\n+}\n+\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result)\n {\n \treturn bsearch_hash(oid->hash, m->chunk_oid_fanout, m->chunk_oid_lookup,\n@@ -468,10 +493,16 @@ static size_t write_midx_header(struct hashfile *f,\n \treturn MIDX_HEADER_SIZE;\n }\n \n+#define BITMAP_POS_UNKNOWN (~((uint32_t)0))\n+\n struct pack_info {\n \tuint32_t orig_pack_int_id;\n \tchar *pack_name;\n \tstruct packed_git *p;\n+\n+\tuint32_t bitmap_pos;\n+\tuint32_t bitmap_nr;\n+\n \tunsigned expired : 1;\n };\n \n@@ -484,6 +515,7 @@ static void fill_pack_info(struct pack_info *info,\n \tinfo->orig_pack_int_id = orig_pack_int_id;\n \tinfo->pack_name = xstrdup(pack_name);\n \tinfo->p = p;\n+\tinfo->bitmap_pos = BITMAP_POS_UNKNOWN;\n }\n \n static int pack_info_compare(const void *_a, const void *_b)\n@@ -824,6 +856,26 @@ static int write_midx_pack_names(struct hashfile *f, void *data)\n \treturn 0;\n }\n \n+static int write_midx_bitmapped_packs(struct hashfile *f, void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tsize_t i;\n+\n+\tfor (i = 0; i < ctx->nr; i++) {\n+\t\tstruct pack_info *pack = &ctx->info[i];\n+\t\tif (pack->expired)\n+\t\t\tcontinue;\n+\n+\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN && pack->bitmap_nr)\n+\t\t\tBUG(\"pack '%s' has no bitmap position, but has %d bitmapped object(s)\",\n+\t\t\t pack->pack_name, pack->bitmap_nr);\n+\n+\t\thashwrite_be32(f, pack->bitmap_pos);\n+\t\thashwrite_be32(f, pack->bitmap_nr);\n+\t}\n+\treturn 0;\n+}\n+\n static int write_midx_oid_fanout(struct hashfile *f,\n \t\t\t\t void *data)\n {\n@@ -991,8 +1043,19 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n \tQSORT(data, ctx->entries_nr, midx_pack_order_cmp);\n \n \tALLOC_ARRAY(pack_order, ctx->entries_nr);\n-\tfor (i = 0; i < ctx->entries_nr; i++)\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *e = &ctx->entries[data[i].nr];\n+\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]];\n+\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n+\t\t\tpack->bitmap_pos = i;\n+\t\tpack->bitmap_nr++;\n \t\tpack_order[i] = data[i].nr;\n+\t}\n+\tfor (i = 0; i < ctx->nr; i++) {\n+\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[i]];\n+\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n+\t\t\tpack->bitmap_pos = 0;\n+\t}\n \tfree(data);\n \n \ttrace2_region_leave(\"midx\", \"midx_pack_order\", the_repository);\n@@ -1293,6 +1356,7 @@ static int write_midx_internal(const char *object_dir,\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n \tstruct write_midx_context ctx = { 0 };\n+\tint bitmapped_packs_concat_len = 0;\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n@@ -1505,8 +1569,10 @@ static int write_midx_internal(const char *object_dir,\n \t}\n \n \tfor (i = 0; i < ctx.nr; i++) {\n-\t\tif (!ctx.info[i].expired)\n-\t\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n+\t\tif (ctx.info[i].expired)\n+\t\t\tcontinue;\n+\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n+\t\tbitmapped_packs_concat_len += 2 * sizeof(uint32_t);\n \t}\n \n \t/* Check that the preferred pack wasn't expired (if given). */\n@@ -1566,6 +1632,9 @@ static int write_midx_internal(const char *object_dir,\n \t\tadd_chunk(cf, MIDX_CHUNKID_REVINDEX,\n \t\t\t st_mult(ctx.entries_nr, sizeof(uint32_t)),\n \t\t\t write_midx_revindex);\n+\t\tadd_chunk(cf, MIDX_CHUNKID_BITMAPPEDPACKS,\n+\t\t\t bitmapped_packs_concat_len,\n+\t\t\t write_midx_bitmapped_packs);\n \t}\n \n \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\ndiff --git a/midx.h b/midx.h\nindex a5d98919c8..b404235db5 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -7,6 +7,7 @@\n struct object_id;\n struct pack_entry;\n struct repository;\n+struct bitmapped_pack;\n \n #define GIT_TEST_MULTI_PACK_INDEX \"GIT_TEST_MULTI_PACK_INDEX\"\n #define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \\\n@@ -33,6 +34,8 @@ struct multi_pack_index {\n \n \tconst unsigned char *chunk_pack_names;\n \tsize_t chunk_pack_names_len;\n+\tconst uint32_t *chunk_bitmapped_packs;\n+\tsize_t chunk_bitmapped_packs_len;\n \tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_object_offsets;\n@@ -58,6 +61,8 @@ void get_midx_rev_filename(struct strbuf *out, struct multi_pack_index *m);\n \n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n+int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n+\t\t struct bitmapped_pack *bp, uint32_t pack_int_id);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);\n uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 5273a6a019..b68b213388 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -52,6 +52,15 @@ typedef int (*show_reachable_fn)(\n \n struct bitmap_index;\n \n+struct bitmapped_pack {\n+\tstruct packed_git *p;\n+\n+\tuint32_t bitmap_pos;\n+\tuint32_t bitmap_nr;\n+\n+\tuint32_t pack_int_id; /* MIDX only */\n+};\n+\n struct bitmap_index *prepare_bitmap_git(struct repository *r);\n struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx);\n void count_bitmap_commit_list(struct bitmap_index *, uint32_t *commits,\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex e9a444ddba..e48557aba1 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -100,10 +100,36 @@ static int read_midx_preferred_pack(const char *object_dir)\n \treturn 0;\n }\n \n+static int read_midx_bitmapped_packs(const char *object_dir)\n+{\n+\tstruct multi_pack_index *midx = NULL;\n+\tstruct bitmapped_pack pack;\n+\tuint32_t i;\n+\n+\tsetup_git_directory();\n+\n+\tmidx = load_multi_pack_index(object_dir, 1);\n+\tif (!midx)\n+\t\treturn 1;\n+\n+\tfor (i = 0; i < midx->num_packs; i++) {\n+\t\tif (nth_bitmapped_pack(the_repository, midx, &pack, i) < 0)\n+\t\t\treturn 1;\n+\n+\t\tprintf(\"%s\\n\", pack_basename(pack.p));\n+\t\tprintf(\" bitmap_pos: %\"PRIuMAX\"\\n\", (uintmax_t)pack.bitmap_pos);\n+\t\tprintf(\" bitmap_nr: %\"PRIuMAX\"\\n\", (uintmax_t)pack.bitmap_nr);\n+\t}\n+\n+\tclose_midx(midx);\n+\n+\treturn 0;\n+}\n+\n int cmd__read_midx(int argc, const char **argv)\n {\n \tif (!(argc == 2 || argc == 3))\n-\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack] \");\n+\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] \");\n \n \tif (!strcmp(argv[1], \"--show-objects\"))\n \t\treturn read_midx_file(argv[2], 1);\n@@ -111,5 +137,7 @@ int cmd__read_midx(int argc, const char **argv)\n \t\treturn read_midx_checksum(argv[2]);\n \telse if (!strcmp(argv[1], \"--preferred-pack\"))\n \t\treturn read_midx_preferred_pack(argv[2]);\n+\telse if (!strcmp(argv[1], \"--bitmap\"))\n+\t\treturn read_midx_bitmapped_packs(argv[2]);\n \treturn read_midx_file(argv[1], 0);\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex c20aafe99a..dd09134db0 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -1171,4 +1171,39 @@ test_expect_success 'reader notices out-of-bounds fanout' '\n \ttest_cmp expect err\n '\n \n+test_expect_success 'bitmapped packs are stored via the BTMP chunk' '\n+\ttest_when_finished \"rm -fr repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\tfor i in 1 2 3 4 5\n+\t\tdo\n+\t\t\ttest_commit \"$i\" &&\n+\t\t\tgit repack -d || return 1\n+\t\tdone &&\n+\n+\t\tfind $objdir/pack -type f -name \"*.idx\" | xargs -n 1 basename |\n+\t\tsort >packs &&\n+\n+\t\tgit multi-pack-index write --stdin-packs err &&\n+\t\tcat >expect <<-\\EOF &&\n+\t\terror: MIDX does not contain the BTMP chunk\n+\t\tEOF\n+\t\ttest_cmp expect err &&\n+\n+\t\tgit multi-pack-index write --stdin-packs --bitmap \\\n+\t\t\t--preferred-pack=\"$(head -n1 actual &&\n+\t\tfor i in $(test_seq $(wc -l expect &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485674","messageId":"96f397a2b2a48db87975cee4789a91e45bd3bc39.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 06/26] midx: implement `midx_locate_pack()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:23:54Z","receivedAt":"2023-12-14T22:23:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The multi-pack index API exposes a `midx_contains_pack()` function that\ntakes in a string ending in either \".idx\" or \".pack\" and returns whether\nor not the MIDX contains a given pack corresponding to that string.\n\nThere is no corresponding function to locate the position of a pack\nwithin the MIDX's pack order (sorted lexically by pack filename).\n\nWe could add an optional out parameter to `midx_contains_pack()` that is\nfilled out with the pack's position when the parameter is non-NULL. To\nminimize the amount of fallout from this change, instead introduce a new\nfunction by renaming `midx_contains_pack()` to `midx_locate_pack()`,\nadding that output parameter, and then reimplementing\n`midx_contains_pack()` in terms of it.\n\nFuture patches will make use of this new function.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 13 +++++++++++--\n midx.h | 5 ++++-\n 2 files changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex de25612b0c..beaf0c0de4 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -428,7 +428,8 @@ static int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n \treturn strcmp(idx_or_pack_name, idx_name);\n }\n \n-int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n+int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n+\t\t uint32_t *pos)\n {\n \tuint32_t first = 0, last = m->num_packs;\n \n@@ -439,8 +440,11 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n \n \t\tcurrent = m->pack_names[mid];\n \t\tcmp = cmp_idx_or_pack_name(idx_or_pack_name, current);\n-\t\tif (!cmp)\n+\t\tif (!cmp) {\n+\t\t\tif (pos)\n+\t\t\t\t*pos = mid;\n \t\t\treturn 1;\n+\t\t}\n \t\tif (cmp > 0) {\n \t\t\tfirst = mid + 1;\n \t\t\tcontinue;\n@@ -451,6 +455,11 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n \treturn 0;\n }\n \n+int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n+{\n+\treturn midx_locate_pack(m, idx_or_pack_name, NULL);\n+}\n+\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local)\n {\n \tstruct multi_pack_index *m;\ndiff --git a/midx.h b/midx.h\nindex b404235db5..89c5aa637e 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -70,7 +70,10 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct multi_pack_index *m,\n \t\t\t\t\tuint32_t n);\n int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m);\n-int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name);\n+int midx_contains_pack(struct multi_pack_index *m,\n+\t\t const char *idx_or_pack_name);\n+int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n+\t\t uint32_t *pos);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n \n /*\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485676","messageId":"df9233cf0f2fa835e4e7b5764acaee68f7352ecb.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 07/26] pack-bitmap: pass `bitmapped_pack` struct to pack-reuse functions","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:23:56Z","receivedAt":"2023-12-14T22:23:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When trying to assemble a pack with bitmaps using `--use-bitmap-index`,\n`pack-objects` asks the pack-bitmap machinery for a bitmap which\nindicates the set of objects we can \"reuse\" verbatim from on-disk.\n\nThis set is roughly comprised of: a prefix of objects in the bitmapped\npack (or preferred pack, in the case of a multi-pack reachability\nbitmap), plus any other objects not included in the prefix, excluding\nany deltas whose base we are not sending in the resulting pack.\n\nThe pack-bitmap machinery is responsible for computing this bitmap, and\ndoes so with the following functions:\n\n - reuse_partial_packfile_from_bitmap()\n - try_partial_reuse()\n\nIn the existing implementation, the first function is responsible for\n(a) marking the prefix of objects in the reusable pack, and then (b)\ncalling try_partial_reuse() on any remaining objects to ensure that they\nare also reusable (and removing them from the bitmapped set if they are\nnot).\n\nLikewise, the `try_partial_reuse()` function is responsible for checking\nwhether an isolated object (that is, an object from the bitmapped\npack/preferred pack not contained in the prefix from earlier) may be\nreused, i.e. that it isn't a delta of an object that we are not sending\nin the resulting pack.\n\nThese functions are based on two core assumptions, which we will unwind\nin this and the following commits:\n\n 1. There is only a single pack from the bitmap which is eligible for\n verbatim pack-reuse. For single-pack bitmaps, this is trivially the\n bitmapped pack. For multi-pack bitmaps, this is (currently) the\n MIDX's preferred pack.\n\n 2. The pack eligible for reuse has its first object in bit position 0,\n and all objects from that pack follow in pack-order from that first\n bit position.\n\nIn order to perform verbatim pack reuse over multiple packs, we must\nunwind these two assumptions. Most notably, in order to reuse bits from\na given packfile, we need to know the first bit position occupied by\nan object form that packfile. To propagate this information around, pass\na `struct bitmapped_pack *` anywhere we previously passed a `struct\npacked_git *`, since the former contains the bitmap position we're\ninterested in (as well as a pointer to the latter).\n\nAs an additional step, factor out a sub-routine from the main\n`reuse_partial_packfile_from_bitmap()` function, called\n`reuse_partial_packfile_from_bitmap_1()`. This new function will be\nresponsible for figuring out which objects may be reused from a single\npack, and the existing function will dispatch multiple calls to its new\nhelper function for each reusable pack.\n\nConsequently, `reuse_partial_packfile_from_bitmap()` will now maintain\nan array of reusable packs instead of a single such pack. We currently\nexpect that array to have only a single element, so this awkward state\nis short-lived. It will serve as useful scaffolding in subsequent\ncommits as we begin to work towards enabling multi-pack reuse.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 118 +++++++++++++++++++++++++++++++++++++-------------\n 1 file changed, 87 insertions(+), 31 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d2f1306960..d64a80c30c 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1836,7 +1836,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n * -1 means \"stop trying further objects\"; 0 means we may or may not have\n * reused, but you can keep feeding bits.\n */\n-static int try_partial_reuse(struct packed_git *pack,\n+static int try_partial_reuse(struct bitmapped_pack *pack,\n \t\t\t size_t pos,\n \t\t\t struct bitmap *reuse,\n \t\t\t struct pack_window **w_curs)\n@@ -1868,11 +1868,11 @@ static int try_partial_reuse(struct packed_git *pack,\n \t * preferred pack precede all bits from other packs.\n \t */\n \n-\tif (pos >= pack->num_objects)\n+\tif (pos >= pack->p->num_objects)\n \t\treturn -1; /* not actually in the pack or MIDX preferred pack */\n \n-\toffset = delta_obj_offset = pack_pos_to_offset(pack, pos);\n-\ttype = unpack_object_header(pack, w_curs, &offset, &size);\n+\toffset = delta_obj_offset = pack_pos_to_offset(pack->p, pos);\n+\ttype = unpack_object_header(pack->p, w_curs, &offset, &size);\n \tif (type < 0)\n \t\treturn -1; /* broken packfile, punt */\n \n@@ -1888,11 +1888,11 @@ static int try_partial_reuse(struct packed_git *pack,\n \t\t * and the normal slow path will complain about it in\n \t\t * more detail.\n \t\t */\n-\t\tbase_offset = get_delta_base(pack, w_curs, &offset, type,\n+\t\tbase_offset = get_delta_base(pack->p, w_curs, &offset, type,\n \t\t\t\t\t delta_obj_offset);\n \t\tif (!base_offset)\n \t\t\treturn 0;\n-\t\tif (offset_to_pack_pos(pack, base_offset, &base_pos) < 0)\n+\t\tif (offset_to_pack_pos(pack->p, base_offset, &base_pos) < 0)\n \t\t\treturn 0;\n \n \t\t/*\n@@ -1915,14 +1915,14 @@ static int try_partial_reuse(struct packed_git *pack,\n \t\t * to REF_DELTA on the fly. Better to just let the normal\n \t\t * object_entry code path handle it.\n \t\t */\n-\t\tif (!bitmap_get(reuse, base_pos))\n+\t\tif (!bitmap_get(reuse, pack->bitmap_pos + base_pos))\n \t\t\treturn 0;\n \t}\n \n \t/*\n \t * If we got here, then the object is OK to reuse. Mark it.\n \t */\n-\tbitmap_set(reuse, pos);\n+\tbitmap_set(reuse, pack->bitmap_pos + pos);\n \treturn 0;\n }\n \n@@ -1934,29 +1934,13 @@ uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n \treturn nth_midxed_pack_int_id(m, pack_pos_to_midx(bitmap_git->midx, 0));\n }\n \n-int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n-\t\t\t\t struct packed_git **packfile_out,\n-\t\t\t\t uint32_t *entries,\n-\t\t\t\t struct bitmap **reuse_out)\n+static void reuse_partial_packfile_from_bitmap_1(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct bitmapped_pack *pack,\n+\t\t\t\t\t\t struct bitmap *reuse)\n {\n-\tstruct repository *r = the_repository;\n-\tstruct packed_git *pack;\n \tstruct bitmap *result = bitmap_git->result;\n-\tstruct bitmap *reuse;\n \tstruct pack_window *w_curs = NULL;\n \tsize_t i = 0;\n-\tuint32_t offset;\n-\tuint32_t objects_nr;\n-\n-\tassert(result);\n-\n-\tload_reverse_index(r, bitmap_git);\n-\n-\tif (bitmap_is_midx(bitmap_git))\n-\t\tpack = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n-\telse\n-\t\tpack = bitmap_git->pack;\n-\tobjects_nr = pack->num_objects;\n \n \twhile (i < result->word_alloc && result->words[i] == (eword_t)~0)\n \t\ti++;\n@@ -1969,15 +1953,15 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t * we use it instead of another pack. In single-pack bitmaps, the choice\n \t * is made for us.\n \t */\n-\tif (i > objects_nr / BITS_IN_EWORD)\n-\t\ti = objects_nr / BITS_IN_EWORD;\n+\tif (i > pack->p->num_objects / BITS_IN_EWORD)\n+\t\ti = pack->p->num_objects / BITS_IN_EWORD;\n \n-\treuse = bitmap_word_alloc(i);\n \tmemset(reuse->words, 0xFF, i * sizeof(eword_t));\n \n \tfor (; i < result->word_alloc; ++i) {\n \t\teword_t word = result->words[i];\n \t\tsize_t pos = (i * BITS_IN_EWORD);\n+\t\tsize_t offset;\n \n \t\tfor (offset = 0; offset < BITS_IN_EWORD; ++offset) {\n \t\t\tif ((word >> offset) == 0)\n@@ -2002,6 +1986,78 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \n done:\n \tunuse_pack(&w_curs);\n+}\n+\n+static int bitmapped_pack_cmp(const void *va, const void *vb)\n+{\n+\tconst struct bitmapped_pack *a = va;\n+\tconst struct bitmapped_pack *b = vb;\n+\n+\tif (a->bitmap_pos < b->bitmap_pos)\n+\t\treturn -1;\n+\tif (a->bitmap_pos > b->bitmap_pos)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct packed_git **packfile_out,\n+\t\t\t\t uint32_t *entries,\n+\t\t\t\t struct bitmap **reuse_out)\n+{\n+\tstruct repository *r = the_repository;\n+\tstruct bitmapped_pack *packs = NULL;\n+\tstruct bitmap *result = bitmap_git->result;\n+\tstruct bitmap *reuse;\n+\tsize_t i;\n+\tsize_t packs_nr = 0, packs_alloc = 0;\n+\tsize_t word_alloc;\n+\tuint32_t objects_nr = 0;\n+\n+\tassert(result);\n+\n+\tload_reverse_index(r, bitmap_git);\n+\n+\tif (bitmap_is_midx(bitmap_git)) {\n+\t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\t\t\tstruct bitmapped_pack pack;\n+\t\t\tif (nth_bitmapped_pack(r, bitmap_git->midx, &pack, i) < 0) {\n+\t\t\t\twarning(_(\"unable to load pack: '%s', disabling pack-reuse\"),\n+\t\t\t\t\tbitmap_git->midx->pack_names[i]);\n+\t\t\t\tfree(packs);\n+\t\t\t\treturn -1;\n+\t\t\t}\n+\t\t\tif (!pack.bitmap_nr)\n+\t\t\t\tcontinue; /* no objects from this pack */\n+\t\t\tif (pack.bitmap_pos)\n+\t\t\t\tcontinue; /* not preferred pack */\n+\n+\t\t\tALLOC_GROW(packs, packs_nr + 1, packs_alloc);\n+\t\t\tmemcpy(&packs[packs_nr++], &pack, sizeof(pack));\n+\n+\t\t\tobjects_nr += pack.p->num_objects;\n+\t\t}\n+\n+\t\tQSORT(packs, packs_nr, bitmapped_pack_cmp);\n+\t} else {\n+\t\tALLOC_GROW(packs, packs_nr + 1, packs_alloc);\n+\n+\t\tpacks[packs_nr].p = bitmap_git->pack;\n+\t\tpacks[packs_nr].bitmap_pos = 0;\n+\t\tpacks[packs_nr].bitmap_nr = bitmap_git->pack->num_objects;\n+\n+\t\tobjects_nr = packs[packs_nr++].p->num_objects;\n+\t}\n+\n+\tword_alloc = objects_nr / BITS_IN_EWORD;\n+\tif (objects_nr % BITS_IN_EWORD)\n+\t\tword_alloc++;\n+\treuse = bitmap_word_alloc(word_alloc);\n+\n+\tif (packs_nr != 1)\n+\t\tBUG(\"pack reuse not yet implemented for multiple packs\");\n+\n+\treuse_partial_packfile_from_bitmap_1(bitmap_git, packs, reuse);\n \n \t*entries = bitmap_popcount(reuse);\n \tif (!*entries) {\n@@ -2014,7 +2070,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t * need to be handled separately.\n \t */\n \tbitmap_and_not(result, reuse);\n-\t*packfile_out = pack;\n+\t*packfile_out = packs[0].p;\n \t*reuse_out = reuse;\n \treturn 0;\n }\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485682","messageId":"595b3b698615cd341cef49623523c0d3ea1b6802.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 08/26] ewah: implement `bitmap_is_empty()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:23:59Z","receivedAt":"2023-12-14T22:24:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a future commit, we will want to check whether or not a bitmap has\nany bits set in any of its words. The best way to do this (prior to the\nexistence of this patch) is to call `bitmap_popcount()` and check\nwhether the result is non-zero.\n\nBut this is semi-wasteful, since we do not need to know the exact number\nof bits set, only whether or not there is at least one of them.\n\nImplement a new helper function to check just that.\n\nSuggested-by: Patrick Steinhardt \nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 9 +++++++++\n ewah/ewok.h | 1 +\n 2 files changed, 10 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex 7b525b1ecd..ac7e0af622 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -169,6 +169,15 @@ size_t bitmap_popcount(struct bitmap *self)\n \treturn count;\n }\n \n+int bitmap_is_empty(struct bitmap *self)\n+{\n+\tsize_t i;\n+\tfor (i = 0; i < self->word_alloc; i++)\n+\t\tif (self->words[i])\n+\t\t\treturn 0;\n+\treturn 1;\n+}\n+\n int bitmap_equals(struct bitmap *self, struct bitmap *other)\n {\n \tstruct bitmap *big, *small;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 7eb8b9b630..c11d76c6f3 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -189,5 +189,6 @@ void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other);\n void bitmap_or(struct bitmap *self, const struct bitmap *other);\n \n size_t bitmap_popcount(struct bitmap *self);\n+int bitmap_is_empty(struct bitmap *self);\n \n #endif\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485678","messageId":"d851f821fc1c1fa6af8532b7780b2587899b16d8.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 09/26] pack-bitmap: simplify `reuse_partial_packfile_from_bitmap()` signature","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:24:01Z","receivedAt":"2023-12-14T22:24:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The signature of `reuse_partial_packfile_from_bitmap()` currently takes\nin a bitmap, as well as three output parameters (filled through\npointers, and passed as arguments), and also returns an integer result.\n\nThe output parameters are filled out with: (a) the packfile used for\npack-reuse, (b) the number of objects from that pack that we can reuse,\nand (c) a bitmap indicating which objects we can reuse. The return value\nis either -1 (when there are no objects to reuse), or 0 (when there is\nat least one object to reuse).\n\nSome of these parameters are redundant. Notably, we can infer from the\nbitmap how many objects are reused by calling bitmap_popcount(). And we\ncan similar compute the return value based on that number as well.\n\nAs such, clean up the signature of this function to drop the \"*entries\"\nparameter, as well as the int return value, since the single caller of\nthis function can infer these values themself.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 16 +++++++++-------\n pack-bitmap.c | 16 +++++++---------\n pack-bitmap.h | 7 +++----\n 3 files changed, 19 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 321d7effb0..c3df6d9657 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3943,13 +3943,15 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n \t\treturn -1;\n \n-\tif (pack_options_allow_reuse() &&\n-\t !reuse_partial_packfile_from_bitmap(\n-\t\t\tbitmap_git,\n-\t\t\t&reuse_packfile,\n-\t\t\t&reuse_packfile_objects,\n-\t\t\t&reuse_packfile_bitmap)) {\n-\t\tassert(reuse_packfile_objects);\n+\tif (pack_options_allow_reuse())\n+\t\treuse_partial_packfile_from_bitmap(bitmap_git, &reuse_packfile,\n+\t\t\t\t\t\t &reuse_packfile_bitmap);\n+\n+\tif (reuse_packfile) {\n+\t\treuse_packfile_objects = bitmap_popcount(reuse_packfile_bitmap);\n+\t\tif (!reuse_packfile_objects)\n+\t\t\tBUG(\"expected non-empty reuse bitmap\");\n+\n \t\tnr_result += reuse_packfile_objects;\n \t\tnr_seen += reuse_packfile_objects;\n \t\tdisplay_progress(progress_state, nr_seen);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d64a80c30c..c75a83e9cc 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -2000,10 +2000,9 @@ static int bitmapped_pack_cmp(const void *va, const void *vb)\n \treturn 0;\n }\n \n-int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n-\t\t\t\t struct packed_git **packfile_out,\n-\t\t\t\t uint32_t *entries,\n-\t\t\t\t struct bitmap **reuse_out)\n+void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n+\t\t\t\t\tstruct packed_git **packfile_out,\n+\t\t\t\t\tstruct bitmap **reuse_out)\n {\n \tstruct repository *r = the_repository;\n \tstruct bitmapped_pack *packs = NULL;\n@@ -2025,7 +2024,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\twarning(_(\"unable to load pack: '%s', disabling pack-reuse\"),\n \t\t\t\t\tbitmap_git->midx->pack_names[i]);\n \t\t\t\tfree(packs);\n-\t\t\t\treturn -1;\n+\t\t\t\treturn;\n \t\t\t}\n \t\t\tif (!pack.bitmap_nr)\n \t\t\t\tcontinue; /* no objects from this pack */\n@@ -2059,10 +2058,10 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \n \treuse_partial_packfile_from_bitmap_1(bitmap_git, packs, reuse);\n \n-\t*entries = bitmap_popcount(reuse);\n-\tif (!*entries) {\n+\tif (bitmap_is_empty(reuse)) {\n+\t\tfree(packs);\n \t\tbitmap_free(reuse);\n-\t\treturn -1;\n+\t\treturn;\n \t}\n \n \t/*\n@@ -2072,7 +2071,6 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \tbitmap_and_not(result, reuse);\n \t*packfile_out = packs[0].p;\n \t*reuse_out = reuse;\n-\treturn 0;\n }\n \n int bitmap_walk_contains(struct bitmap_index *bitmap_git,\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex b68b213388..ab3fdcde6b 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -78,10 +78,9 @@ int test_bitmap_hashes(struct repository *r);\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\t\t\t int filter_provided_objects);\n uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git);\n-int reuse_partial_packfile_from_bitmap(struct bitmap_index *,\n-\t\t\t\t struct packed_git **packfile,\n-\t\t\t\t uint32_t *entries,\n-\t\t\t\t struct bitmap **reuse_out);\n+void reuse_partial_packfile_from_bitmap(struct bitmap_index *,\n+\t\t\t\t\tstruct packed_git **packfile,\n+\t\t\t\t\tstruct bitmap **reuse_out);\n int rebuild_existing_bitmaps(struct bitmap_index *, struct packing_data *mapping,\n \t\t\t kh_oid_map_t *reused_bitmaps, int show_progress);\n void free_bitmap_index(struct bitmap_index *);\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485681","messageId":"f551892bab54f855556ca6a35168717285ce4e75.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 10/26] pack-bitmap: return multiple packs via `reuse_partial_packfile_from_bitmap()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:24:04Z","receivedAt":"2023-12-14T22:24:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Further prepare for enabling verbatim pack-reuse over multiple packfiles\nby changing the signature of reuse_partial_packfile_from_bitmap() to\npopulate an array of `struct bitmapped_pack *`'s instead of a pointer to\na single packfile.\n\nSince the array we're filling out is sized dynamically[^1], add an\nadditional `size_t *` parameter which will hold the number of reusable\npacks (equal to the number of elements in the array).\n\nNote that since we still have not implemented true multi-pack reuse,\nthese changes aren't propagated out to the rest of the caller in\nbuiltin/pack-objects.c.\n\nIn the interim state, we expect that the array has a single element, and\nwe use that element to fill out the static `reuse_packfile` variable\n(which is a bog-standard `struct packed_git *`). Future commits will\ncontinue to push this change further out through the pack-objects code.\n\n[^1]: That is, even though we know the number of packs which are\n candidates for pack-reuse, we do not know how many of those\n candidates we can actually reuse.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 9 +++++++--\n pack-bitmap.c | 6 ++++--\n pack-bitmap.h | 5 +++--\n 3 files changed, 14 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex c3df6d9657..87e16636a8 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3940,14 +3940,19 @@ static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n+\tstruct bitmapped_pack *packs = NULL;\n+\tsize_t packs_nr = 0;\n+\n \tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse())\n-\t\treuse_partial_packfile_from_bitmap(bitmap_git, &reuse_packfile,\n+\t\treuse_partial_packfile_from_bitmap(bitmap_git, &packs,\n+\t\t\t\t\t\t &packs_nr,\n \t\t\t\t\t\t &reuse_packfile_bitmap);\n \n-\tif (reuse_packfile) {\n+\tif (packs) {\n+\t\treuse_packfile = packs[0].p;\n \t\treuse_packfile_objects = bitmap_popcount(reuse_packfile_bitmap);\n \t\tif (!reuse_packfile_objects)\n \t\t\tBUG(\"expected non-empty reuse bitmap\");\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex c75a83e9cc..4d5a484678 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -2001,7 +2001,8 @@ static int bitmapped_pack_cmp(const void *va, const void *vb)\n }\n \n void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n-\t\t\t\t\tstruct packed_git **packfile_out,\n+\t\t\t\t\tstruct bitmapped_pack **packs_out,\n+\t\t\t\t\tsize_t *packs_nr_out,\n \t\t\t\t\tstruct bitmap **reuse_out)\n {\n \tstruct repository *r = the_repository;\n@@ -2069,7 +2070,8 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t * need to be handled separately.\n \t */\n \tbitmap_and_not(result, reuse);\n-\t*packfile_out = packs[0].p;\n+\t*packs_out = packs;\n+\t*packs_nr_out = packs_nr;\n \t*reuse_out = reuse;\n }\n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex ab3fdcde6b..7a12a2ce81 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -78,8 +78,9 @@ int test_bitmap_hashes(struct repository *r);\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\t\t\t int filter_provided_objects);\n uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git);\n-void reuse_partial_packfile_from_bitmap(struct bitmap_index *,\n-\t\t\t\t\tstruct packed_git **packfile,\n+void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n+\t\t\t\t\tstruct bitmapped_pack **packs_out,\n+\t\t\t\t\tsize_t *packs_nr_out,\n \t\t\t\t\tstruct bitmap **reuse_out);\n int rebuild_existing_bitmaps(struct bitmap_index *, struct packing_data *mapping,\n \t\t\t kh_oid_map_t *reused_bitmaps, int show_progress);\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485679","messageId":"67e4fd8a061fdc008031f6d3746fd65664965ff2.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 11/26] pack-objects: parameterize pack-reuse routines over a single pack","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:24:07Z","receivedAt":"2023-12-14T22:24:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The routines pack-objects uses to perform verbatim pack-reuse are:\n\n - write_reused_pack_one()\n - write_reused_pack_verbatim()\n - write_reused_pack()\n\n, all of which assume that there is exactly one packfile being reused:\nthe global constant `reuse_packfile`.\n\nPrepare for reusing objects from multiple packs by making reuse packfile\na parameter of each of the above functions in preparation for calling\nthese functions in a loop with multiple packfiles.\n\nNote that we still have the global \"reuse_packfile\", but pass it through\neach of the above function's parameter lists, eliminating all but one\ndirect access (the top-level caller in `write_pack_file()`). Even after\nthis series, we will still have a global, but it will hold the array of\nreusable packfiles, and we'll pass them one at a time to these functions\nin a loop.\n\nNote also that we will eventually need to pass a `bitmapped_pack`\ninstead of a `packed_git` in order to hold onto additional information\nrequired for reuse (such as the bit position of the first object\nbelonging to that pack). But that change will be made in a future commit\nso as to minimize the noise below as much as possible.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 16 ++++++++++------\n 1 file changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 87e16636a8..102fe9a4f8 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1013,7 +1013,8 @@ static off_t find_reused_offset(off_t where)\n \treturn reused_chunks[lo-1].difference;\n }\n \n-static void write_reused_pack_one(size_t pos, struct hashfile *out,\n+static void write_reused_pack_one(struct packed_git *reuse_packfile,\n+\t\t\t\t size_t pos, struct hashfile *out,\n \t\t\t\t struct pack_window **w_curs)\n {\n \toff_t offset, next, cur;\n@@ -1091,7 +1092,8 @@ static void write_reused_pack_one(size_t pos, struct hashfile *out,\n \tcopy_pack_data(out, reuse_packfile, w_curs, offset, next - offset);\n }\n \n-static size_t write_reused_pack_verbatim(struct hashfile *out,\n+static size_t write_reused_pack_verbatim(struct packed_git *reuse_packfile,\n+\t\t\t\t\t struct hashfile *out,\n \t\t\t\t\t struct pack_window **w_curs)\n {\n \tsize_t pos = 0;\n@@ -1118,14 +1120,15 @@ static size_t write_reused_pack_verbatim(struct hashfile *out,\n \treturn pos;\n }\n \n-static void write_reused_pack(struct hashfile *f)\n+static void write_reused_pack(struct packed_git *reuse_packfile,\n+\t\t\t struct hashfile *f)\n {\n \tsize_t i = 0;\n \tuint32_t offset;\n \tstruct pack_window *w_curs = NULL;\n \n \tif (allow_ofs_delta)\n-\t\ti = write_reused_pack_verbatim(f, &w_curs);\n+\t\ti = write_reused_pack_verbatim(reuse_packfile, f, &w_curs);\n \n \tfor (; i < reuse_packfile_bitmap->word_alloc; ++i) {\n \t\teword_t word = reuse_packfile_bitmap->words[i];\n@@ -1141,7 +1144,8 @@ static void write_reused_pack(struct hashfile *f)\n \t\t\t * bitmaps. See comment in try_partial_reuse()\n \t\t\t * for why.\n \t\t\t */\n-\t\t\twrite_reused_pack_one(pos + offset, f, &w_curs);\n+\t\t\twrite_reused_pack_one(reuse_packfile, pos + offset, f,\n+\t\t\t\t\t &w_curs);\n \t\t\tdisplay_progress(progress_state, ++written);\n \t\t}\n \t}\n@@ -1199,7 +1203,7 @@ static void write_pack_file(void)\n \n \t\tif (reuse_packfile) {\n \t\t\tassert(pack_to_stdout);\n-\t\t\twrite_reused_pack(f);\n+\t\t\twrite_reused_pack(reuse_packfile, f);\n \t\t\toffset = hashfile_total(f);\n \t\t}\n \n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485680","messageId":"9a5c38514bb6dedc44c28bab71a98fe1c0150ecb.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 12/26] pack-objects: keep track of `pack_start` for each reuse pack","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:24:09Z","receivedAt":"2023-12-14T22:24:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When reusing objects from a pack, we keep track of a set of one or more\n`reused_chunk`s, corresponding to sections of one or more object(s) from\na source pack that we are reusing. Each chunk contains two pieces of\ninformation:\n\n - the offset of the first object in the source pack (relative to the\n beginning of the source pack)\n - the difference between that offset, and the corresponding offset in\n the pack we're generating\n\nThe purpose of keeping track of these is so that we can patch an\nOFS_DELTAs that cross over a section of the reuse pack that we didn't\ntake.\n\nFor instance, consider a hypothetical pack as shown below:\n\n (chunk #2)\n __________...\n /\n /\n +--------+---------+-------------------+---------+\n ... | | | (unused) | | ...\n +--------+---------+-------------------+---------+\n \\ /\n \\______________/\n (chunk #1)\n\nSuppose that we are sending objects \"base\", \"other\", and \"delta\", and\nthat the \"delta\" object is stored as an OFS_DELTA, and that its base is\n\"base\". If we don't send any objects in the \"(unused)\" range, we can't\ncopy the delta'd object directly, since its delta offset includes a\nrange of the pack that we didn't copy, so we have to account for that\ndifference when patching and reassembling the delta.\n\nIn order to compute this value correctly, we need to know not only where\nwe are in the packfile we're assembling (with `hashfile_total(f)`) but\nalso the position of the first byte of the packfile that we are\ncurrently reusing. Currently, this works just fine, since when reusing\nonly a single pack those two values are always identical (because\nverbatim reuse is the first thing pack-objects does when enabled after\nwriting the pack header).\n\nBut when reusing multiple packs which have one or more gaps, we'll need\nto account for these two values diverging.\n\nTogether, these two allow us to compute the reused chunk's offset\ndifference relative to the start of the reused pack, as desired.\n\nHelped-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 102fe9a4f8..f51b86d99f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1015,6 +1015,7 @@ static off_t find_reused_offset(off_t where)\n \n static void write_reused_pack_one(struct packed_git *reuse_packfile,\n \t\t\t\t size_t pos, struct hashfile *out,\n+\t\t\t\t off_t pack_start,\n \t\t\t\t struct pack_window **w_curs)\n {\n \toff_t offset, next, cur;\n@@ -1024,7 +1025,8 @@ static void write_reused_pack_one(struct packed_git *reuse_packfile,\n \toffset = pack_pos_to_offset(reuse_packfile, pos);\n \tnext = pack_pos_to_offset(reuse_packfile, pos + 1);\n \n-\trecord_reused_object(offset, offset - hashfile_total(out));\n+\trecord_reused_object(offset,\n+\t\t\t offset - (hashfile_total(out) - pack_start));\n \n \tcur = offset;\n \ttype = unpack_object_header(reuse_packfile, w_curs, &cur, &size);\n@@ -1094,6 +1096,7 @@ static void write_reused_pack_one(struct packed_git *reuse_packfile,\n \n static size_t write_reused_pack_verbatim(struct packed_git *reuse_packfile,\n \t\t\t\t\t struct hashfile *out,\n+\t\t\t\t\t off_t pack_start UNUSED,\n \t\t\t\t\t struct pack_window **w_curs)\n {\n \tsize_t pos = 0;\n@@ -1125,10 +1128,12 @@ static void write_reused_pack(struct packed_git *reuse_packfile,\n {\n \tsize_t i = 0;\n \tuint32_t offset;\n+\toff_t pack_start = hashfile_total(f) - sizeof(struct pack_header);\n \tstruct pack_window *w_curs = NULL;\n \n \tif (allow_ofs_delta)\n-\t\ti = write_reused_pack_verbatim(reuse_packfile, f, &w_curs);\n+\t\ti = write_reused_pack_verbatim(reuse_packfile, f, pack_start,\n+\t\t\t\t\t &w_curs);\n \n \tfor (; i < reuse_packfile_bitmap->word_alloc; ++i) {\n \t\teword_t word = reuse_packfile_bitmap->words[i];\n@@ -1145,7 +1150,7 @@ static void write_reused_pack(struct packed_git *reuse_packfile,\n \t\t\t * for why.\n \t\t\t */\n \t\t\twrite_reused_pack_one(reuse_packfile, pos + offset, f,\n-\t\t\t\t\t &w_curs);\n+\t\t\t\t\t pack_start, &w_curs);\n \t\t\tdisplay_progress(progress_state, ++written);\n \t\t}\n \t}\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485683","messageId":"5492d11f25ec54c8eff59e3fa266abd766a1f40c.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 13/26] pack-objects: pass `bitmapped_pack`'s to pack-reuse functions","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:24:12Z","receivedAt":"2023-12-14T22:24:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Further prepare pack-objects to perform verbatim pack-reuse over\nmultiple packfiles by converting functions that take in a pointer to a\n`struct packed_git` to instead take in a pointer to a `struct\nbitmapped_pack`.\n\nThe additional information found in the bitmapped_pack struct (such as\nthe bit position corresponding to the beginning of the pack) will be\nnecessary in order to perform verbatim pack-reuse.\n\nNote that we don't use any of the extra pieces of information contained\nin the bitmapped_pack struct, so this step is merely preparatory and\ndoes not introduce any functional changes.\n\nNote further that we do not change the argument type to\nwrite_reused_pack_one(). That function is responsible for copying\nsections of the packfile directly and optionally patching any OFS_DELTAs\nto account for not reusing sections of the packfile in between a delta\nand its base.\n\nAs such, that function is (and should remain) oblivious to multi-pack\nreuse, and does not require any of the extra pieces of information\nstored in the bitmapped_pack struct.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 33 +++++++++++++++++----------------\n 1 file changed, 17 insertions(+), 16 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f51b86d99f..07c849b5d4 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -221,7 +221,8 @@ static int thin;\n static int num_preferred_base;\n static struct progress *progress_state;\n \n-static struct packed_git *reuse_packfile;\n+static struct bitmapped_pack *reuse_packfiles;\n+static size_t reuse_packfiles_nr;\n static uint32_t reuse_packfile_objects;\n static struct bitmap *reuse_packfile_bitmap;\n \n@@ -1094,7 +1095,7 @@ static void write_reused_pack_one(struct packed_git *reuse_packfile,\n \tcopy_pack_data(out, reuse_packfile, w_curs, offset, next - offset);\n }\n \n-static size_t write_reused_pack_verbatim(struct packed_git *reuse_packfile,\n+static size_t write_reused_pack_verbatim(struct bitmapped_pack *reuse_packfile,\n \t\t\t\t\t struct hashfile *out,\n \t\t\t\t\t off_t pack_start UNUSED,\n \t\t\t\t\t struct pack_window **w_curs)\n@@ -1109,13 +1110,13 @@ static size_t write_reused_pack_verbatim(struct packed_git *reuse_packfile,\n \t\toff_t to_write;\n \n \t\twritten = (pos * BITS_IN_EWORD);\n-\t\tto_write = pack_pos_to_offset(reuse_packfile, written)\n+\t\tto_write = pack_pos_to_offset(reuse_packfile->p, written)\n \t\t\t- sizeof(struct pack_header);\n \n \t\t/* We're recording one chunk, not one object. */\n \t\trecord_reused_object(sizeof(struct pack_header), 0);\n \t\thashflush(out);\n-\t\tcopy_pack_data(out, reuse_packfile, w_curs,\n+\t\tcopy_pack_data(out, reuse_packfile->p, w_curs,\n \t\t\tsizeof(struct pack_header), to_write);\n \n \t\tdisplay_progress(progress_state, written);\n@@ -1123,7 +1124,7 @@ static size_t write_reused_pack_verbatim(struct packed_git *reuse_packfile,\n \treturn pos;\n }\n \n-static void write_reused_pack(struct packed_git *reuse_packfile,\n+static void write_reused_pack(struct bitmapped_pack *reuse_packfile,\n \t\t\t struct hashfile *f)\n {\n \tsize_t i = 0;\n@@ -1149,8 +1150,8 @@ static void write_reused_pack(struct packed_git *reuse_packfile,\n \t\t\t * bitmaps. See comment in try_partial_reuse()\n \t\t\t * for why.\n \t\t\t */\n-\t\t\twrite_reused_pack_one(reuse_packfile, pos + offset, f,\n-\t\t\t\t\t pack_start, &w_curs);\n+\t\t\twrite_reused_pack_one(reuse_packfile->p, pos + offset,\n+\t\t\t\t\t f, pack_start, &w_curs);\n \t\t\tdisplay_progress(progress_state, ++written);\n \t\t}\n \t}\n@@ -1206,9 +1207,12 @@ static void write_pack_file(void)\n \n \t\toffset = write_pack_header(f, nr_remaining);\n \n-\t\tif (reuse_packfile) {\n+\t\tif (reuse_packfiles_nr) {\n \t\t\tassert(pack_to_stdout);\n-\t\t\twrite_reused_pack(reuse_packfile, f);\n+\t\t\tfor (j = 0; j < reuse_packfiles_nr; j++) {\n+\t\t\t\treused_chunks_nr = 0;\n+\t\t\t\twrite_reused_pack(&reuse_packfiles[j], f);\n+\t\t\t}\n \t\t\toffset = hashfile_total(f);\n \t\t}\n \n@@ -3949,19 +3953,16 @@ static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n-\tstruct bitmapped_pack *packs = NULL;\n-\tsize_t packs_nr = 0;\n-\n \tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse())\n-\t\treuse_partial_packfile_from_bitmap(bitmap_git, &packs,\n-\t\t\t\t\t\t &packs_nr,\n+\t\treuse_partial_packfile_from_bitmap(bitmap_git,\n+\t\t\t\t\t\t &reuse_packfiles,\n+\t\t\t\t\t\t &reuse_packfiles_nr,\n \t\t\t\t\t\t &reuse_packfile_bitmap);\n \n-\tif (packs) {\n-\t\treuse_packfile = packs[0].p;\n+\tif (reuse_packfiles) {\n \t\treuse_packfile_objects = bitmap_popcount(reuse_packfile_bitmap);\n \t\tif (!reuse_packfile_objects)\n \t\t\tBUG(\"expected non-empty reuse bitmap\");\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485684","messageId":"b32742ebcb0632aabcc82427cae7823e75cf6b9c.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 14/26] pack-objects: prepare `write_reused_pack()` for multi-pack reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:24:14Z","receivedAt":"2023-12-14T22:24:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `write_reused_pack()` within `builtin/pack-objects.c` is\nresponsible for performing pack-reuse on a single pack, and has two main\nfunctions:\n\n - it dispatches a call to `write_reused_pack_verbatim()` to see if we\n can reuse portions of the packfile in whole-word chunks\n\n - for any remaining objects (that is, any objects that appear after\n the first \"gap\" in the bitmap), call write_reused_pack_one() on that\n object to record it for reuse.\n\nPrepare this function for multi-pack reuse by removing the assumption\nthat the bit position corresponding to the first object being reused\nfrom a given pack must be at bit position zero.\n\nThe changes in this function are mostly straightforward. Initialize `i`\nto the position of the first word to contain bits corresponding to that\nreuse pack. In most situations, we throw the initialized value away,\nsince we end up replacing it with the return value from\nwrite_reused_pack_verbatim(), moving us past the section of whole words\nthat we reused.\n\nLikewise, modify the per-object loop to ignore any bits at the beginning\nof the first word that do not belong to the pack currently being reused,\nas well as skip to the \"done\" section once we have processed the last\nbit corresponding to this pack.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 07c849b5d4..6ce52d88a9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1127,7 +1127,7 @@ static size_t write_reused_pack_verbatim(struct bitmapped_pack *reuse_packfile,\n static void write_reused_pack(struct bitmapped_pack *reuse_packfile,\n \t\t\t struct hashfile *f)\n {\n-\tsize_t i = 0;\n+\tsize_t i = reuse_packfile->bitmap_pos / BITS_IN_EWORD;\n \tuint32_t offset;\n \toff_t pack_start = hashfile_total(f) - sizeof(struct pack_header);\n \tstruct pack_window *w_curs = NULL;\n@@ -1145,17 +1145,23 @@ static void write_reused_pack(struct bitmapped_pack *reuse_packfile,\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n+\t\t\tif (pos + offset < reuse_packfile->bitmap_pos)\n+\t\t\t\tcontinue;\n+\t\t\tif (pos + offset >= reuse_packfile->bitmap_pos + reuse_packfile->bitmap_nr)\n+\t\t\t\tgoto done;\n \t\t\t/*\n \t\t\t * Can use bit positions directly, even for MIDX\n \t\t\t * bitmaps. See comment in try_partial_reuse()\n \t\t\t * for why.\n \t\t\t */\n-\t\t\twrite_reused_pack_one(reuse_packfile->p, pos + offset,\n+\t\t\twrite_reused_pack_one(reuse_packfile->p,\n+\t\t\t\t\t pos + offset - reuse_packfile->bitmap_pos,\n \t\t\t\t\t f, pack_start, &w_curs);\n \t\t\tdisplay_progress(progress_state, ++written);\n \t\t}\n \t}\n \n+done:\n \tunuse_pack(&w_curs);\n }\n \n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485691","messageId":"805c42185ab92bb4303e91fb15e884940380fc87.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 15/26] pack-objects: prepare `write_reused_pack_verbatim()` for multi-pack reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:24:17Z","receivedAt":"2023-12-14T22:24:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The function `write_reused_pack_verbatim()` within\n`builtin/pack-objects.c` is responsible for writing out a continuous\nset of objects beginning at the start of the reuse packfile.\n\nIn the existing implementation, we did something like:\n\n while (pos < reuse_packfile_bitmap->word_alloc &&\n reuse_packfile_bitmap->words[pos] == (eword_t)~0)\n pos++;\n\n if (pos)\n /* write first `pos * BITS_IN_WORD` objects from pack */\n\nas an optimization to record a single chunk for the longest continuous\nprefix of objects wanted out of the reuse pack, instead of having a\nchunk for each individual object. For more details, see bb514de356\n(pack-objects: improve partial packfile reuse, 2019-12-18).\n\nIn order to retain this optimization in a multi-pack reuse world, we can\nno longer assume that the first object in a pack is on a word boundary\nin the bitmap storing the set of reusable objects.\n\nAssuming that all objects from the beginning of the reuse packfile up to\nthe object corresponding to the first bit on a word boundary are part of\nthe result, consume whole words at a time until the last whole word\nbelonging to the reuse packfile. Copy those objects to the resulting\npackfile, and track that we reused them by recording a single chunk.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 73 ++++++++++++++++++++++++++++++++++--------\n 1 file changed, 60 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6ce52d88a9..31053128fc 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1097,31 +1097,78 @@ static void write_reused_pack_one(struct packed_git *reuse_packfile,\n \n static size_t write_reused_pack_verbatim(struct bitmapped_pack *reuse_packfile,\n \t\t\t\t\t struct hashfile *out,\n-\t\t\t\t\t off_t pack_start UNUSED,\n+\t\t\t\t\t off_t pack_start,\n \t\t\t\t\t struct pack_window **w_curs)\n {\n-\tsize_t pos = 0;\n+\tsize_t pos = reuse_packfile->bitmap_pos;\n+\tsize_t end;\n \n-\twhile (pos < reuse_packfile_bitmap->word_alloc &&\n-\t\t\treuse_packfile_bitmap->words[pos] == (eword_t)~0)\n-\t\tpos++;\n+\tif (pos % BITS_IN_EWORD) {\n+\t\tsize_t word_pos = (pos / BITS_IN_EWORD);\n+\t\tsize_t offset = pos % BITS_IN_EWORD;\n+\t\tsize_t last;\n+\t\teword_t word = reuse_packfile_bitmap->words[word_pos];\n \n-\tif (pos) {\n-\t\toff_t to_write;\n+\t\tif (offset + reuse_packfile->bitmap_nr < BITS_IN_EWORD)\n+\t\t\tlast = offset + reuse_packfile->bitmap_nr;\n+\t\telse\n+\t\t\tlast = BITS_IN_EWORD;\n \n-\t\twritten = (pos * BITS_IN_EWORD);\n-\t\tto_write = pack_pos_to_offset(reuse_packfile->p, written)\n-\t\t\t- sizeof(struct pack_header);\n+\t\tfor (; offset < last; offset++) {\n+\t\t\tif (word >> offset == 0)\n+\t\t\t\treturn word_pos;\n+\t\t\tif (!bitmap_get(reuse_packfile_bitmap,\n+\t\t\t\t\tword_pos * BITS_IN_EWORD + offset))\n+\t\t\t\treturn word_pos;\n+\t\t}\n+\n+\t\tpos += BITS_IN_EWORD - (pos % BITS_IN_EWORD);\n+\t}\n+\n+\t/*\n+\t * Now we're going to copy as many whole eword_t's as possible.\n+\t * \"end\" is the index of the last whole eword_t we copy, but\n+\t * there may be additional bits to process. Those are handled\n+\t * individually by write_reused_pack().\n+\t *\n+\t * Begin by advancing to the first word boundary in range of the\n+\t * bit positions occupied by objects in \"reuse_packfile\". Then\n+\t * pick the last word boundary in the same range. If we have at\n+\t * least one word's worth of bits to process, continue on.\n+\t */\n+\tend = reuse_packfile->bitmap_pos + reuse_packfile->bitmap_nr;\n+\tif (end % BITS_IN_EWORD)\n+\t\tend -= end % BITS_IN_EWORD;\n+\tif (pos >= end)\n+\t\treturn reuse_packfile->bitmap_pos / BITS_IN_EWORD;\n+\n+\twhile (pos < end &&\n+\t reuse_packfile_bitmap->words[pos / BITS_IN_EWORD] == (eword_t)~0)\n+\t\tpos += BITS_IN_EWORD;\n+\n+\tif (pos > end)\n+\t\tpos = end;\n+\n+\tif (reuse_packfile->bitmap_pos < pos) {\n+\t\toff_t pack_start_off = pack_pos_to_offset(reuse_packfile->p, 0);\n+\t\toff_t pack_end_off = pack_pos_to_offset(reuse_packfile->p,\n+\t\t\t\t\t\t\tpos - reuse_packfile->bitmap_pos);\n+\n+\t\twritten += pos - reuse_packfile->bitmap_pos;\n \n \t\t/* We're recording one chunk, not one object. */\n-\t\trecord_reused_object(sizeof(struct pack_header), 0);\n+\t\trecord_reused_object(pack_start_off,\n+\t\t\t\t pack_start_off - (hashfile_total(out) - pack_start));\n \t\thashflush(out);\n \t\tcopy_pack_data(out, reuse_packfile->p, w_curs,\n-\t\t\tsizeof(struct pack_header), to_write);\n+\t\t\tpack_start_off, pack_end_off - pack_start_off);\n \n \t\tdisplay_progress(progress_state, written);\n \t}\n-\treturn pos;\n+\tif (pos % BITS_IN_EWORD)\n+\t\tBUG(\"attempted to jump past a word boundary to %\"PRIuMAX,\n+\t\t (uintmax_t)pos);\n+\treturn pos / BITS_IN_EWORD;\n }\n \n static void write_reused_pack(struct bitmapped_pack *reuse_packfile,\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485685","messageId":"55696bc1c9fc4d8d99fc870b9f9f5c97dbb181e9.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 16/26] pack-objects: include number of packs reused in output","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:24:20Z","receivedAt":"2023-12-14T22:24:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In addition to including the number of objects reused verbatim from a\nreuse-pack, include the number of packs from which objects were reused.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 8 ++++++--\n 1 file changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 31053128fc..7eb035eb7d 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -223,6 +223,7 @@ static struct progress *progress_state;\n \n static struct bitmapped_pack *reuse_packfiles;\n static size_t reuse_packfiles_nr;\n+static size_t reuse_packfiles_used_nr;\n static uint32_t reuse_packfile_objects;\n static struct bitmap *reuse_packfile_bitmap;\n \n@@ -1265,6 +1266,8 @@ static void write_pack_file(void)\n \t\t\tfor (j = 0; j < reuse_packfiles_nr; j++) {\n \t\t\t\treused_chunks_nr = 0;\n \t\t\t\twrite_reused_pack(&reuse_packfiles[j], f);\n+\t\t\t\tif (reused_chunks_nr)\n+\t\t\t\t\treuse_packfiles_used_nr++;\n \t\t\t}\n \t\t\toffset = hashfile_total(f);\n \t\t}\n@@ -4587,9 +4590,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tfprintf_ln(stderr,\n \t\t\t _(\"Total %\"PRIu32\" (delta %\"PRIu32\"),\"\n \t\t\t \" reused %\"PRIu32\" (delta %\"PRIu32\"),\"\n-\t\t\t \" pack-reused %\"PRIu32),\n+\t\t\t \" pack-reused %\"PRIu32\" (from %\"PRIuMAX\")\"),\n \t\t\t written, written_delta, reused, reused_delta,\n-\t\t\t reuse_packfile_objects);\n+\t\t\t reuse_packfile_objects,\n+\t\t\t (uintmax_t)reuse_packfiles_used_nr);\n \n cleanup:\n \tclear_packing_data(&to_pack);\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485686","messageId":"6ede9e060332a0850269fe69a7a3a15ac229344b.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 17/26] git-compat-util.h: implement checked size_t to uint32_t conversion","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:24:22Z","receivedAt":"2023-12-14T22:24:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar fashion as other checked cast functions in this header\n(such as `cast_size_t_to_ulong()` and `cast_size_t_to_int()`), implement\na checked cast function for going from a size_t to a uint32_t value.\n\nThis function will be utilized in a future commit which needs to make\nsuch a conversion.\n\nSigned-off-by: Taylor Blau \n---\n git-compat-util.h | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 3e7a59b5ff..c3b6c2c226 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -1013,6 +1013,15 @@ static inline unsigned long cast_size_t_to_ulong(size_t a)\n \treturn (unsigned long)a;\n }\n \n+static inline uint32_t cast_size_t_to_uint32_t(size_t a)\n+{\n+\tif (a != (uint32_t)a)\n+\t\tdie(\"object too large to read on this platform: %\"\n+\t\t PRIuMAX\" is cut off to %u\",\n+\t\t (uintmax_t)a, (uint32_t)a);\n+\treturn (uint32_t)a;\n+}\n+\n static inline int cast_size_t_to_int(size_t a)\n {\n \tif (a > INT_MAX)\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485689","messageId":"ab0456a71e5385c772c385d727e74e9fddf30669.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 18/26] midx: implement `midx_preferred_pack()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:24:25Z","receivedAt":"2023-12-14T22:24:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When performing a binary search over the objects in a MIDX's bitmap\n(i.e. in pseudo-pack order), the reader reconstructs the pseudo-pack\nordering using a combination of (a) the preferred pack, (b) the pack's\nlexical position in the MIDX based on pack names, and (c) the object\noffset within the pack.\n\nIn order to perform this binary search, the reader must know the\nidentity of the preferred pack. This could be stored in the MIDX, but\nisn't for historical reasons, mostly because it can easily be inferred\nat read-time by looking at the object in the first bit position and\nfinding out which pack it was selected from in the MIDX, like so:\n\n nth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));\n\nIn midx_to_pack_pos() which performs this binary search, we look up the\nidentity of the preferred pack before each search. This is relatively\nquick, since it involves two table-driven lookups (one in the MIDX's\nrevindex for `pack_pos_to_midx()`, and another in the MIDX's object\ntable for `nth_midxed_pack_int_id()`).\n\nBut since the preferred pack does not change after the MIDX is written,\nit is safe to cache this value on the MIDX itself.\n\nWrite a helper to do just that, and rewrite all of the existing\ncall-sites that care about the identity of the preferred pack in terms\nof this new helper.\n\nThis will prepare us for a subsequent patch where we will need to binary\nsearch through the MIDX's pseudo-pack order multiple times.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 20 ++++++++++++++++++++\n midx.h | 2 ++\n pack-bitmap.c | 17 +++++++----------\n pack-bitmap.h | 1 -\n pack-revindex.c | 4 +++-\n t/helper/test-read-midx.c | 13 +++++--------\n 6 files changed, 37 insertions(+), 20 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex beaf0c0de4..85e1c2cd12 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -21,6 +21,7 @@\n #include \"refs.h\"\n #include \"revision.h\"\n #include \"list-objects.h\"\n+#include \"pack-revindex.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -177,6 +178,8 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \n \tm->num_packs = get_be32(m->data + MIDX_BYTE_NUM_PACKS);\n \n+\tm->preferred_pack_idx = -1;\n+\n \tcf = init_chunkfile(NULL);\n \n \tif (read_table_of_contents(cf, m->data, midx_size,\n@@ -460,6 +463,23 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n \treturn midx_locate_pack(m, idx_or_pack_name, NULL);\n }\n \n+int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)\n+{\n+\tif (m->preferred_pack_idx == -1) {\n+\t\tif (load_midx_revindex(m) < 0) {\n+\t\t\tm->preferred_pack_idx = -2;\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tm->preferred_pack_idx =\n+\t\t\tnth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));\n+\t} else if (m->preferred_pack_idx == -2)\n+\t\treturn -1; /* no revindex */\n+\n+\t*pack_int_id = m->preferred_pack_idx;\n+\treturn 0;\n+}\n+\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local)\n {\n \tstruct multi_pack_index *m;\ndiff --git a/midx.h b/midx.h\nindex 89c5aa637e..f87a8fff26 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -29,6 +29,7 @@ struct multi_pack_index {\n \tunsigned char num_chunks;\n \tuint32_t num_packs;\n \tuint32_t num_objects;\n+\tint preferred_pack_idx;\n \n \tint local;\n \n@@ -74,6 +75,7 @@ int midx_contains_pack(struct multi_pack_index *m,\n \t\t const char *idx_or_pack_name);\n int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,\n \t\t uint32_t *pos);\n+int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n \n /*\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 4d5a484678..1682f99596 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -338,7 +338,7 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \tstruct stat st;\n \tchar *bitmap_name = midx_bitmap_filename(midx);\n \tint fd = git_open(bitmap_name);\n-\tuint32_t i;\n+\tuint32_t i, preferred_pack;\n \tstruct packed_git *preferred;\n \n \tif (fd < 0) {\n@@ -393,7 +393,12 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \t\t}\n \t}\n \n-\tpreferred = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n+\tif (midx_preferred_pack(bitmap_git->midx, &preferred_pack) < 0) {\n+\t\twarning(_(\"could not determine MIDX preferred pack\"));\n+\t\tgoto cleanup;\n+\t}\n+\n+\tpreferred = bitmap_git->midx->packs[preferred_pack];\n \tif (!is_pack_valid(preferred)) {\n \t\twarning(_(\"preferred pack (%s) is invalid\"),\n \t\t\tpreferred->pack_name);\n@@ -1926,14 +1931,6 @@ static int try_partial_reuse(struct bitmapped_pack *pack,\n \treturn 0;\n }\n \n-uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n-{\n-\tstruct multi_pack_index *m = bitmap_git->midx;\n-\tif (!m)\n-\t\tBUG(\"midx_preferred_pack: requires non-empty MIDX\");\n-\treturn nth_midxed_pack_int_id(m, pack_pos_to_midx(bitmap_git->midx, 0));\n-}\n-\n static void reuse_partial_packfile_from_bitmap_1(struct bitmap_index *bitmap_git,\n \t\t\t\t\t\t struct bitmapped_pack *pack,\n \t\t\t\t\t\t struct bitmap *reuse)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 7a12a2ce81..179b343912 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -77,7 +77,6 @@ int test_bitmap_hashes(struct repository *r);\n \n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\t\t\t int filter_provided_objects);\n-uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git);\n void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\t\tstruct bitmapped_pack **packs_out,\n \t\t\t\t\tsize_t *packs_nr_out,\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex acf1dd9786..7dc6c776d5 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -542,7 +542,9 @@ int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)\n \t * implicitly is preferred (and includes all its objects, since ties are\n \t * broken first by pack identifier).\n \t */\n-\tkey.preferred_pack = nth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));\n+\tif (midx_preferred_pack(key.midx, &key.preferred_pack) < 0)\n+\t\treturn error(_(\"could not determine preferred pack\"));\n+\n \n \tfound = bsearch(&key, m->revindex_data, m->num_objects,\n \t\t\tsizeof(*m->revindex_data), midx_pack_order_cmp);\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex e48557aba1..4acae41bb9 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -6,6 +6,7 @@\n #include \"pack-bitmap.h\"\n #include \"packfile.h\"\n #include \"setup.h\"\n+#include \"gettext.h\"\n \n static int read_midx_file(const char *object_dir, int show_objects)\n {\n@@ -79,7 +80,7 @@ static int read_midx_checksum(const char *object_dir)\n static int read_midx_preferred_pack(const char *object_dir)\n {\n \tstruct multi_pack_index *midx = NULL;\n-\tstruct bitmap_index *bitmap = NULL;\n+\tuint32_t preferred_pack;\n \n \tsetup_git_directory();\n \n@@ -87,16 +88,12 @@ static int read_midx_preferred_pack(const char *object_dir)\n \tif (!midx)\n \t\treturn 1;\n \n-\tbitmap = prepare_bitmap_git(the_repository);\n-\tif (!bitmap)\n-\t\treturn 1;\n-\tif (!bitmap_is_midx(bitmap)) {\n-\t\tfree_bitmap_index(bitmap);\n+\tif (midx_preferred_pack(midx, &preferred_pack) < 0) {\n+\t\twarning(_(\"could not determine MIDX preferred pack\"));\n \t\treturn 1;\n \t}\n \n-\tprintf(\"%s\\n\", midx->pack_names[midx_preferred_pack(bitmap)]);\n-\tfree_bitmap_index(bitmap);\n+\tprintf(\"%s\\n\", midx->pack_names[preferred_pack]);\n \treturn 0;\n }\n \n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485688","messageId":"14b054d27283ffdda1d3f2a4078a4dae54868bfa.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 19/26] pack-revindex: factor out `midx_key_to_pack_pos()` helper","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:24:28Z","receivedAt":"2023-12-14T22:24:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `midx_to_pack_pos()` function implements a binary search over\nobjects in the MIDX between lexical and pseudo-pack order. It does this\nby taking in an index into the lexical order (i.e. the same argument\nyou'd use for `nth_midxed_object_id()` and similar) and spits out a\nposition in the pseudo-pack order.\n\nThis works for all callers, since they currently all are translating\nfrom lexical order to pseudo-pack order. But future callers may want to\ntranslate a known (offset, pack_id) tuple into an index into the\npsuedo-pack order, without knowing where that (offset, pack_id) tuple\nappears in lexical order.\n\nPrepare for implementing a function that translates between a (offset,\npack_id) tuple into an index into the psuedo-pack order by extracting a\nhelper function which does just that, and then reimplementing\nmidx_to_pack_pos() in terms of it.\n\nSigned-off-by: Taylor Blau \n---\n pack-revindex.c | 39 ++++++++++++++++++++++++---------------\n 1 file changed, 24 insertions(+), 15 deletions(-)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 7dc6c776d5..baa4657ed3 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -520,19 +520,12 @@ static int midx_pack_order_cmp(const void *va, const void *vb)\n \treturn 0;\n }\n \n-int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)\n+static int midx_key_to_pack_pos(struct multi_pack_index *m,\n+\t\t\t\tstruct midx_pack_key *key,\n+\t\t\t\tuint32_t *pos)\n {\n-\tstruct midx_pack_key key;\n \tuint32_t *found;\n \n-\tif (!m->revindex_data)\n-\t\tBUG(\"midx_to_pack_pos: reverse index not yet loaded\");\n-\tif (m->num_objects <= at)\n-\t\tBUG(\"midx_to_pack_pos: out-of-bounds object at %\"PRIu32, at);\n-\n-\tkey.pack = nth_midxed_pack_int_id(m, at);\n-\tkey.offset = nth_midxed_offset(m, at);\n-\tkey.midx = m;\n \t/*\n \t * The preferred pack sorts first, so determine its identifier by\n \t * looking at the first object in pseudo-pack order.\n@@ -542,16 +535,32 @@ int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)\n \t * implicitly is preferred (and includes all its objects, since ties are\n \t * broken first by pack identifier).\n \t */\n-\tif (midx_preferred_pack(key.midx, &key.preferred_pack) < 0)\n+\tif (midx_preferred_pack(key->midx, &key->preferred_pack) < 0)\n \t\treturn error(_(\"could not determine preferred pack\"));\n \n-\n-\tfound = bsearch(&key, m->revindex_data, m->num_objects,\n-\t\t\tsizeof(*m->revindex_data), midx_pack_order_cmp);\n+\tfound = bsearch(key, m->revindex_data, m->num_objects,\n+\t\t\tsizeof(*m->revindex_data),\n+\t\t\tmidx_pack_order_cmp);\n \n \tif (!found)\n-\t\treturn error(\"bad offset for revindex\");\n+\t\treturn -1;\n \n \t*pos = found - m->revindex_data;\n \treturn 0;\n }\n+\n+int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)\n+{\n+\tstruct midx_pack_key key;\n+\n+\tif (!m->revindex_data)\n+\t\tBUG(\"midx_to_pack_pos: reverse index not yet loaded\");\n+\tif (m->num_objects <= at)\n+\t\tBUG(\"midx_to_pack_pos: out-of-bounds object at %\"PRIu32, at);\n+\n+\tkey.pack = nth_midxed_pack_int_id(m, at);\n+\tkey.offset = nth_midxed_offset(m, at);\n+\tkey.midx = m;\n+\n+\treturn midx_key_to_pack_pos(m, &key, pos);\n+}\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485690","messageId":"e99481014ed9dba161b30dea5e6af2b692f5ca9f.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 20/26] pack-revindex: implement `midx_pair_to_pack_pos()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:24:30Z","receivedAt":"2023-12-14T22:24:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that we have extracted the `midx_key_to_pack_pos()` function, we can\nimplement the `midx_pair_to_pack_pos()` function which accepts (pack_id,\noffset) tuples and returns an index into the psuedo-pack order.\n\nThis will be used in a following commit in order to figure out whether\nor not the MIDX chose a given delta's base object from the same pack as\nthe delta resides in. It will do so by locating the base object's offset\nin the pack, and then performing a binary search using the same pack ID\nwith the base object's offset.\n\nIf (and only if) it finds a match (at any position) we can guarantee\nthat the MIDX selected both halves of the delta/base pair from the same\npack.\n\nSigned-off-by: Taylor Blau \n---\n pack-revindex.c | 11 +++++++++++\n pack-revindex.h | 3 +++\n 2 files changed, 14 insertions(+)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex baa4657ed3..a7624d8be8 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -564,3 +564,14 @@ int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)\n \n \treturn midx_key_to_pack_pos(m, &key, pos);\n }\n+\n+int midx_pair_to_pack_pos(struct multi_pack_index *m, uint32_t pack_int_id,\n+\t\t\t off_t ofs, uint32_t *pos)\n+{\n+\tstruct midx_pack_key key = {\n+\t\t.pack = pack_int_id,\n+\t\t.offset = ofs,\n+\t\t.midx = m,\n+\t};\n+\treturn midx_key_to_pack_pos(m, &key, pos);\n+}\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex 6dd47efea1..422c2487ae 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -142,4 +142,7 @@ uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos);\n */\n int midx_to_pack_pos(struct multi_pack_index *midx, uint32_t at, uint32_t *pos);\n \n+int midx_pair_to_pack_pos(struct multi_pack_index *midx, uint32_t pack_id,\n+\t\t\t off_t ofs, uint32_t *pos);\n+\n #endif\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485693","messageId":"3e3625aebe58e2c8048f9decb80c28c84b6cb0b4.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 21/26] pack-bitmap: prepare to mark objects from multiple packs for reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:24:34Z","receivedAt":"2023-12-14T22:24:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the pack-objects code is equipped to handle reusing objects\nfrom multiple packs, prepare the pack-bitmap code to mark objects from\nmultiple packs as reuse candidates.\n\nIn order to prepare the pack-bitmap code for this change, remove the\nsame set of assumptions we unwound in previous commits from the helper\nfunction `reuse_partial_packfile_from_bitmap_1()`, in preparation for it\nto be called in a loop over the set of bitmapped packs in a following\ncommit.\n\nMost importantly, we can no longer assume that the bit position\ncorresponding to the first object in a given reuse pack candidate is at\nthe beginning of the bitmap itself.\n\nFor the single pack that this assumption is still true for (in MIDX\nbitmaps, this is the preferred pack, in single-pack bitmaps it is the\npack the bitmap is tied to), we can still use our whole-words\noptimization.\n\nBut for all subsequent packs, we can not make use of this optimization,\nsince it assumes that all delta bases are being sent from the same pack,\nwhich would break if we are sending OFS_DELTAs down to the client. To\nunderstand why, consider two packs, P1 and P2 where:\n\n - P1 has object A which is a delta on base B\n - P2 has its own copy of B, in addition to other objects\n\nSuppose that the MIDX which covers P1 and P2 selected its copy of A from\nP1, but selected its copy of B from P2. Since A is a delta of B, but the\nbase was selected from a different pack, sending the bytes corresponding\nto A as an OFS_DELTA verbatim from P1 would be incorrect, since we don't\nguarantee that B is in the same place relative to A in the generated\npack as in P1.\n\nFor now, we detect and reject these cross-pack deltas by searching for\nthe (pack_id, offset) pair for the delta's base object (using the same\npack_id as the pack containing the delta'd object) in the MIDX. If we\nfind a match, that means that the MIDX did indeed pick the base object\nfrom the same pack, and we are OK to reuse the delta.\n\nIf we don't find a match, however, that means that the base object was\nselected from a different pack in the MIDX, and we can let the slower\npath handle re-delta'ing our candidate object.\n\nIn the future, there are a couple of other things we could do, namely:\n\n - Turn any cross-pack deltas (which are stored as OFS_DELTAs) into\n REF_DELTAs. We already do this today when reusing an OFS_DELTA\n without `--delta-base-offset` enabled, so it's not a huge stretch to\n do the same for cross-pack deltas even when `--delta-base-offset` is\n enabled.\n\n This would work, but would obviously result in larger-than-necessary\n packs, as we in theory *could* represent these cross-pack deltas by\n patching an existing OFS_DELTA. But it's not clear how much that\n would matter in practice. I suspect it would have a lot to do with\n how you pack your repository in the first place.\n\n - Finally, we could patch OFS_DELTAs across packs in a similar fashion\n as we do today for OFS_DELTAs within a single pack on either side of\n a gap. This would result in the smallest packs of the three options\n here, but implementing this would be more involved.\n\n At minimum, you'd have to keep the reusable chunks list for all\n reused packs, not just the one we're currently processing. And you'd\n have to ensure that any bases which are a part of cross-pack deltas\n appear before the delta. I think this is possible to do, but would\n require assembling the reusable chunks list potentially in a\n different order than they appear in the source packs.\n\nFor now, let's pursue the simplest approach and reject any cross-pack\ndeltas.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 172 +++++++++++++++++++++++++++++++-------------------\n 1 file changed, 106 insertions(+), 66 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 1682f99596..242a5908f7 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1841,8 +1841,10 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n * -1 means \"stop trying further objects\"; 0 means we may or may not have\n * reused, but you can keep feeding bits.\n */\n-static int try_partial_reuse(struct bitmapped_pack *pack,\n-\t\t\t size_t pos,\n+static int try_partial_reuse(struct bitmap_index *bitmap_git,\n+\t\t\t struct bitmapped_pack *pack,\n+\t\t\t size_t bitmap_pos,\n+\t\t\t uint32_t pack_pos,\n \t\t\t struct bitmap *reuse,\n \t\t\t struct pack_window **w_curs)\n {\n@@ -1850,33 +1852,10 @@ static int try_partial_reuse(struct bitmapped_pack *pack,\n \tenum object_type type;\n \tunsigned long size;\n \n-\t/*\n-\t * try_partial_reuse() is called either on (a) objects in the\n-\t * bitmapped pack (in the case of a single-pack bitmap) or (b)\n-\t * objects in the preferred pack of a multi-pack bitmap.\n-\t * Importantly, the latter can pretend as if only a single pack\n-\t * exists because:\n-\t *\n-\t * - The first pack->num_objects bits of a MIDX bitmap are\n-\t * reserved for the preferred pack, and\n-\t *\n-\t * - Ties due to duplicate objects are always resolved in\n-\t * favor of the preferred pack.\n-\t *\n-\t * Therefore we do not need to ever ask the MIDX for its copy of\n-\t * an object by OID, since it will always select it from the\n-\t * preferred pack. Likewise, the selected copy of the base\n-\t * object for any deltas will reside in the same pack.\n-\t *\n-\t * This means that we can reuse pos when looking up the bit in\n-\t * the reuse bitmap, too, since bits corresponding to the\n-\t * preferred pack precede all bits from other packs.\n-\t */\n+\tif (pack_pos >= pack->p->num_objects)\n+\t\treturn -1; /* not actually in the pack */\n \n-\tif (pos >= pack->p->num_objects)\n-\t\treturn -1; /* not actually in the pack or MIDX preferred pack */\n-\n-\toffset = delta_obj_offset = pack_pos_to_offset(pack->p, pos);\n+\toffset = delta_obj_offset = pack_pos_to_offset(pack->p, pack_pos);\n \ttype = unpack_object_header(pack->p, w_curs, &offset, &size);\n \tif (type < 0)\n \t\treturn -1; /* broken packfile, punt */\n@@ -1884,6 +1863,7 @@ static int try_partial_reuse(struct bitmapped_pack *pack,\n \tif (type == OBJ_REF_DELTA || type == OBJ_OFS_DELTA) {\n \t\toff_t base_offset;\n \t\tuint32_t base_pos;\n+\t\tuint32_t base_bitmap_pos;\n \n \t\t/*\n \t\t * Find the position of the base object so we can look it up\n@@ -1897,20 +1877,44 @@ static int try_partial_reuse(struct bitmapped_pack *pack,\n \t\t\t\t\t delta_obj_offset);\n \t\tif (!base_offset)\n \t\t\treturn 0;\n-\t\tif (offset_to_pack_pos(pack->p, base_offset, &base_pos) < 0)\n-\t\t\treturn 0;\n \n-\t\t/*\n-\t\t * We assume delta dependencies always point backwards. This\n-\t\t * lets us do a single pass, and is basically always true\n-\t\t * due to the way OFS_DELTAs work. You would not typically\n-\t\t * find REF_DELTA in a bitmapped pack, since we only bitmap\n-\t\t * packs we write fresh, and OFS_DELTA is the default). But\n-\t\t * let's double check to make sure the pack wasn't written with\n-\t\t * odd parameters.\n-\t\t */\n-\t\tif (base_pos >= pos)\n-\t\t\treturn 0;\n+\t\toffset_to_pack_pos(pack->p, base_offset, &base_pos);\n+\n+\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\t/*\n+\t\t\t * Cross-pack deltas are rejected for now, but could\n+\t\t\t * theoretically be supported in the future.\n+\t\t\t *\n+\t\t\t * We would need to ensure that we're sending both\n+\t\t\t * halves of the delta/base pair, regardless of whether\n+\t\t\t * or not the two cross a pack boundary. If they do,\n+\t\t\t * then we must convert the delta to an REF_DELTA to\n+\t\t\t * refer back to the base in the other pack.\n+\t\t\t * */\n+\t\t\tif (midx_pair_to_pack_pos(bitmap_git->midx,\n+\t\t\t\t\t\t pack->pack_int_id,\n+\t\t\t\t\t\t base_offset,\n+\t\t\t\t\t\t &base_bitmap_pos) < 0) {\n+\t\t\t\treturn 0;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tif (offset_to_pack_pos(pack->p, base_offset,\n+\t\t\t\t\t &base_pos) < 0)\n+\t\t\t\treturn 0;\n+\t\t\t/*\n+\t\t\t * We assume delta dependencies always point backwards.\n+\t\t\t * This lets us do a single pass, and is basically\n+\t\t\t * always true due to the way OFS_DELTAs work. You would\n+\t\t\t * not typically find REF_DELTA in a bitmapped pack,\n+\t\t\t * since we only bitmap packs we write fresh, and\n+\t\t\t * OFS_DELTA is the default). But let's double check to\n+\t\t\t * make sure the pack wasn't written with odd\n+\t\t\t * parameters.\n+\t\t\t */\n+\t\t\tif (base_pos >= pack_pos)\n+\t\t\t\treturn 0;\n+\t\t\tbase_bitmap_pos = pack->bitmap_pos + base_pos;\n+\t\t}\n \n \t\t/*\n \t\t * And finally, if we're not sending the base as part of our\n@@ -1920,14 +1924,14 @@ static int try_partial_reuse(struct bitmapped_pack *pack,\n \t\t * to REF_DELTA on the fly. Better to just let the normal\n \t\t * object_entry code path handle it.\n \t\t */\n-\t\tif (!bitmap_get(reuse, pack->bitmap_pos + base_pos))\n+\t\tif (!bitmap_get(reuse, base_bitmap_pos))\n \t\t\treturn 0;\n \t}\n \n \t/*\n \t * If we got here, then the object is OK to reuse. Mark it.\n \t */\n-\tbitmap_set(reuse, pack->bitmap_pos + pos);\n+\tbitmap_set(reuse, bitmap_pos);\n \treturn 0;\n }\n \n@@ -1937,36 +1941,72 @@ static void reuse_partial_packfile_from_bitmap_1(struct bitmap_index *bitmap_git\n {\n \tstruct bitmap *result = bitmap_git->result;\n \tstruct pack_window *w_curs = NULL;\n-\tsize_t i = 0;\n+\tsize_t pos = pack->bitmap_pos / BITS_IN_EWORD;\n \n-\twhile (i < result->word_alloc && result->words[i] == (eword_t)~0)\n-\t\ti++;\n+\tif (!pack->bitmap_pos) {\n+\t\t/*\n+\t\t * If we're processing the first (in the case of a MIDX, the\n+\t\t * preferred pack) or the only (in the case of single-pack\n+\t\t * bitmaps) pack, then we can reuse whole words at a time.\n+\t\t *\n+\t\t * This is because we know that any deltas in this range *must*\n+\t\t * have their bases chosen from the same pack, since:\n+\t\t *\n+\t\t * - In the single pack case, there is no other pack to choose\n+\t\t * them from.\n+\t\t *\n+\t\t * - In the MIDX case, the first pack is the preferred pack, so\n+\t\t * all ties are broken in favor of that pack (i.e. the one\n+\t\t * we're currently processing). So any duplicate bases will be\n+\t\t * resolved in favor of the pack we're processing.\n+\t\t */\n+\t\twhile (pos < result->word_alloc &&\n+\t\t pos < pack->bitmap_nr / BITS_IN_EWORD &&\n+\t\t result->words[pos] == (eword_t)~0)\n+\t\t\tpos++;\n+\t\tmemset(reuse->words, 0xFF, pos * sizeof(eword_t));\n+\t}\n \n-\t/*\n-\t * Don't mark objects not in the packfile or preferred pack. This bitmap\n-\t * marks objects eligible for reuse, but the pack-reuse code only\n-\t * understands how to reuse a single pack. Since the preferred pack is\n-\t * guaranteed to have all bases for its deltas (in a multi-pack bitmap),\n-\t * we use it instead of another pack. In single-pack bitmaps, the choice\n-\t * is made for us.\n-\t */\n-\tif (i > pack->p->num_objects / BITS_IN_EWORD)\n-\t\ti = pack->p->num_objects / BITS_IN_EWORD;\n-\n-\tmemset(reuse->words, 0xFF, i * sizeof(eword_t));\n-\n-\tfor (; i < result->word_alloc; ++i) {\n-\t\teword_t word = result->words[i];\n-\t\tsize_t pos = (i * BITS_IN_EWORD);\n+\tfor (; pos < result->word_alloc; pos++) {\n+\t\teword_t word = result->words[pos];\n \t\tsize_t offset;\n \n-\t\tfor (offset = 0; offset < BITS_IN_EWORD; ++offset) {\n-\t\t\tif ((word >> offset) == 0)\n+\t\tfor (offset = 0; offset < BITS_IN_EWORD; offset++) {\n+\t\t\tsize_t bit_pos;\n+\t\t\tuint32_t pack_pos;\n+\n+\t\t\tif (word >> offset == 0)\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n-\t\t\tif (try_partial_reuse(pack, pos + offset,\n-\t\t\t\t\t reuse, &w_curs) < 0) {\n+\n+\t\t\tbit_pos = pos * BITS_IN_EWORD + offset;\n+\t\t\tif (bit_pos < pack->bitmap_pos)\n+\t\t\t\tcontinue;\n+\t\t\tif (bit_pos >= pack->bitmap_pos + pack->bitmap_nr)\n+\t\t\t\tgoto done;\n+\n+\t\t\tif (bitmap_is_midx(bitmap_git)) {\n+\t\t\t\tuint32_t midx_pos;\n+\t\t\t\toff_t ofs;\n+\n+\t\t\t\tmidx_pos = pack_pos_to_midx(bitmap_git->midx, bit_pos);\n+\t\t\t\tofs = nth_midxed_offset(bitmap_git->midx, midx_pos);\n+\n+\t\t\t\tif (offset_to_pack_pos(pack->p, ofs, &pack_pos) < 0)\n+\t\t\t\t\tBUG(\"could not find object in pack %s \"\n+\t\t\t\t\t \"at offset %\"PRIuMAX\" in MIDX\",\n+\t\t\t\t\t pack_basename(pack->p), (uintmax_t)ofs);\n+\t\t\t} else {\n+\t\t\t\tpack_pos = cast_size_t_to_uint32_t(st_sub(bit_pos, pack->bitmap_pos));\n+\t\t\t\tif (pack_pos >= pack->p->num_objects)\n+\t\t\t\t\tBUG(\"advanced beyond the end of pack %s (%\"PRIuMAX\" > %\"PRIu32\")\",\n+\t\t\t\t\t pack_basename(pack->p), (uintmax_t)pack_pos,\n+\t\t\t\t\t pack->p->num_objects);\n+\t\t\t}\n+\n+\t\t\tif (try_partial_reuse(bitmap_git, pack, bit_pos,\n+\t\t\t\t\t pack_pos, reuse, &w_curs) < 0) {\n \t\t\t\t/*\n \t\t\t\t * try_partial_reuse indicated we couldn't reuse\n \t\t\t\t * any bits, so there is no point in trying more\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485692","messageId":"1723cd0384b56bbf5ae77ed249327c16cb937634.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 22/26] pack-objects: add tracing for various packfile metrics","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:24:36Z","receivedAt":"2023-12-14T22:24:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"As part of the multi-pack reuse effort, we will want to add some tests\nthat assert that we reused a certain number of objects from a certain\nnumber of packs.\n\nWe could do this by grepping through the stderr output of\n`pack-objects`, but doing so would be brittle in case the output format\nchanged.\n\nInstead, let's use the trace2 mechanism to log various pieces of\ninformation about the generated packfile, which we can then use to\ncompare against desired values.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7eb035eb7d..7aae9f104b 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4595,6 +4595,13 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t reuse_packfile_objects,\n \t\t\t (uintmax_t)reuse_packfiles_used_nr);\n \n+\ttrace2_data_intmax(\"pack-objects\", the_repository, \"written\", written);\n+\ttrace2_data_intmax(\"pack-objects\", the_repository, \"written/delta\", written_delta);\n+\ttrace2_data_intmax(\"pack-objects\", the_repository, \"reused\", reused);\n+\ttrace2_data_intmax(\"pack-objects\", the_repository, \"reused/delta\", reused_delta);\n+\ttrace2_data_intmax(\"pack-objects\", the_repository, \"pack-reused\", reuse_packfile_objects);\n+\ttrace2_data_intmax(\"pack-objects\", the_repository, \"packs-reused\", reuse_packfiles_used_nr);\n+\n cleanup:\n \tclear_packing_data(&to_pack);\n \tlist_objects_filter_release(&filter_options);\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485695","messageId":"79c830e37ae7acec826bc41b8473309b38ed006f.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 23/26] t/test-lib-functions.sh: implement `test_trace2_data` helper","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:24:39Z","receivedAt":"2023-12-14T22:24:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a helper function which looks for a specific (category, key,\nvalue) tuple in the output of a trace2 event stream.\n\nWe will use this function in a future patch to ensure that the expected\nnumber of objects are reused from an expected number of packs.\n\nSigned-off-by: Taylor Blau \n---\n t/test-lib-functions.sh | 14 ++++++++++++++\n 1 file changed, 14 insertions(+)\n\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 9c3cf12b26..93fe819b0a 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1874,6 +1874,20 @@ test_region () {\n \treturn 0\n }\n \n+# Check that the given data fragment was included as part of the\n+# trace2-format trace on stdin.\n+#\n+#\ttest_trace2_data \n+#\n+# For example, to look for trace2_data_intmax(\"pack-objects\", repo,\n+# \"reused\", N) in an invocation of \"git pack-objects\", run:\n+#\n+#\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" git pack-objects ... &&\n+#\ttest_trace2_data pack-objects reused N \n---\n Documentation/config/pack.txt | 2 +-\n builtin/pack-objects.c | 19 ++++++++++++++++---\n 2 files changed, 17 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\nindex f50df9dbce..fe100d0fb7 100644\n--- a/Documentation/config/pack.txt\n+++ b/Documentation/config/pack.txt\n@@ -28,7 +28,7 @@ all existing objects. You can force recompression by passing the -F option\n to linkgit:git-repack[1].\n \n pack.allowPackReuse::\n-\tWhen true, and when reachability bitmaps are enabled,\n+\tWhen true or \"single\", and when reachability bitmaps are enabled,\n \tpack-objects will try to send parts of the bitmapped packfile\n \tverbatim. This can reduce memory and CPU usage to serve fetches,\n \tbut might result in sending a slightly larger pack. Defaults to\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7aae9f104b..684698f679 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -229,7 +229,10 @@ static struct bitmap *reuse_packfile_bitmap;\n \n static int use_bitmap_index_default = 1;\n static int use_bitmap_index = -1;\n-static int allow_pack_reuse = 1;\n+static enum {\n+\tNO_PACK_REUSE = 0,\n+\tSINGLE_PACK_REUSE,\n+} allow_pack_reuse = SINGLE_PACK_REUSE;\n static enum {\n \tWRITE_BITMAP_FALSE = 0,\n \tWRITE_BITMAP_QUIET,\n@@ -3244,7 +3247,17 @@ static int git_pack_config(const char *k, const char *v,\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.allowpackreuse\")) {\n-\t\tallow_pack_reuse = git_config_bool(k, v);\n+\t\tint res = git_parse_maybe_bool_text(v);\n+\t\tif (res < 0) {\n+\t\t\tif (!strcasecmp(v, \"single\"))\n+\t\t\t\tallow_pack_reuse = SINGLE_PACK_REUSE;\n+\t\t\telse\n+\t\t\t\tdie(_(\"invalid pack.allowPackReuse value: '%s'\"), v);\n+\t\t} else if (res) {\n+\t\t\tallow_pack_reuse = SINGLE_PACK_REUSE;\n+\t\t} else {\n+\t\t\tallow_pack_reuse = NO_PACK_REUSE;\n+\t\t}\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.threads\")) {\n@@ -3999,7 +4012,7 @@ static void loosen_unused_packed_objects(void)\n */\n static int pack_options_allow_reuse(void)\n {\n-\treturn allow_pack_reuse &&\n+\treturn allow_pack_reuse != NO_PACK_REUSE &&\n \t pack_to_stdout &&\n \t !ignore_packed_keep_on_disk &&\n \t !ignore_packed_keep_in_core &&\n-- \n2.43.0.102.ga31d690331.dirty\n\n"},{"id":"485697","messageId":"7002cf08fe301f1de28137b798fab3c8c32337fa.1702592604.git.me@ttaylorr.com","threadId":"60565","inReplyTo":"cover.1702592603.git.me@ttaylorr.com","subject":"[PATCH v2 25/26] pack-bitmap: enable reuse from all bitmapped packs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-14T22:24:44Z","receivedAt":"2023-12-14T22:24:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that both the pack-bitmap and pack-objects code are prepared to\nhandle marking and using objects from multiple bitmapped packs for\nverbatim reuse, allow marking objects from all bitmapped packs as\neligible for reuse.\n\nWithin the `reuse_partial_packfile_from_bitmap()` function, we no longer\nonly mark the pack whose first object is at bit position zero for reuse,\nand instead mark any pack contained in the MIDX as a reuse candidate.\n\nProvide a handful of test cases in a new script (t5332) exercising\ninteresting behavior for multi-pack reuse to ensure that we performed\nall of the previous steps correctly.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/pack.txt | 16 ++-\n builtin/pack-objects.c | 6 +-\n pack-bitmap.c | 34 ++++--\n pack-bitmap.h | 3 +-\n t/t5332-multi-pack-reuse.sh | 203 ++++++++++++++++++++++++++++++++++\n 5 files changed, 245 insertions(+), 17 deletions(-)\n create mode 100755 t/t5332-multi-pack-reuse.sh\n\ndiff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\nindex fe100d0fb7..9c630863e6 100644\n--- a/Documentation/config/pack.txt\n+++ b/Documentation/config/pack.txt\n@@ -28,11 +28,17 @@ all existing objects. You can force recompression by passing the -F option\n to linkgit:git-repack[1].\n \n pack.allowPackReuse::\n-\tWhen true or \"single\", and when reachability bitmaps are enabled,\n-\tpack-objects will try to send parts of the bitmapped packfile\n-\tverbatim. This can reduce memory and CPU usage to serve fetches,\n-\tbut might result in sending a slightly larger pack. Defaults to\n-\ttrue.\n+\tWhen true or \"single\", and when reachability bitmaps are\n+\tenabled, pack-objects will try to send parts of the bitmapped\n+\tpackfile verbatim. When \"multi\", and when a multi-pack\n+\treachability bitmap is available, pack-objects will try to send\n+\tparts of all packs in the MIDX.\n++\n+\tIf only a single pack bitmap is available, and\n+\t`pack.allowPackReuse` is set to \"multi\", reuse parts of just the\n+\tbitmapped packfile. This can reduce memory and CPU usage to\n+\tserve fetches, but might result in sending a slightly larger\n+\tpack. Defaults to true.\n \n pack.island::\n \tAn extended regular expression configuring a set of delta\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 684698f679..5d3c42035b 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -232,6 +232,7 @@ static int use_bitmap_index = -1;\n static enum {\n \tNO_PACK_REUSE = 0,\n \tSINGLE_PACK_REUSE,\n+\tMULTI_PACK_REUSE,\n } allow_pack_reuse = SINGLE_PACK_REUSE;\n static enum {\n \tWRITE_BITMAP_FALSE = 0,\n@@ -3251,6 +3252,8 @@ static int git_pack_config(const char *k, const char *v,\n \t\tif (res < 0) {\n \t\t\tif (!strcasecmp(v, \"single\"))\n \t\t\t\tallow_pack_reuse = SINGLE_PACK_REUSE;\n+\t\t\telse if (!strcasecmp(v, \"multi\"))\n+\t\t\t\tallow_pack_reuse = MULTI_PACK_REUSE;\n \t\t\telse\n \t\t\t\tdie(_(\"invalid pack.allowPackReuse value: '%s'\"), v);\n \t\t} else if (res) {\n@@ -4029,7 +4032,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \t\treuse_partial_packfile_from_bitmap(bitmap_git,\n \t\t\t\t\t\t &reuse_packfiles,\n \t\t\t\t\t\t &reuse_packfiles_nr,\n-\t\t\t\t\t\t &reuse_packfile_bitmap);\n+\t\t\t\t\t\t &reuse_packfile_bitmap,\n+\t\t\t\t\t\t allow_pack_reuse == MULTI_PACK_REUSE);\n \n \tif (reuse_packfiles) {\n \t\treuse_packfile_objects = bitmap_popcount(reuse_packfile_bitmap);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 242a5908f7..229a11fb00 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -2040,7 +2040,8 @@ static int bitmapped_pack_cmp(const void *va, const void *vb)\n void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\t\tstruct bitmapped_pack **packs_out,\n \t\t\t\t\tsize_t *packs_nr_out,\n-\t\t\t\t\tstruct bitmap **reuse_out)\n+\t\t\t\t\tstruct bitmap **reuse_out,\n+\t\t\t\t\tint multi_pack_reuse)\n {\n \tstruct repository *r = the_repository;\n \tstruct bitmapped_pack *packs = NULL;\n@@ -2064,15 +2065,30 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\tfree(packs);\n \t\t\t\treturn;\n \t\t\t}\n+\n \t\t\tif (!pack.bitmap_nr)\n-\t\t\t\tcontinue; /* no objects from this pack */\n-\t\t\tif (pack.bitmap_pos)\n-\t\t\t\tcontinue; /* not preferred pack */\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (!multi_pack_reuse && pack.bitmap_pos) {\n+\t\t\t\t/*\n+\t\t\t\t * If we're only reusing a single pack, skip\n+\t\t\t\t * over any packs which are not positioned at\n+\t\t\t\t * the beginning of the MIDX bitmap.\n+\t\t\t\t *\n+\t\t\t\t * This is consistent with the existing\n+\t\t\t\t * single-pack reuse behavior, which only reuses\n+\t\t\t\t * parts of the MIDX's preferred pack.\n+\t\t\t\t */\n+\t\t\t\tcontinue;\n+\t\t\t}\n \n \t\t\tALLOC_GROW(packs, packs_nr + 1, packs_alloc);\n \t\t\tmemcpy(&packs[packs_nr++], &pack, sizeof(pack));\n \n \t\t\tobjects_nr += pack.p->num_objects;\n+\n+\t\t\tif (!multi_pack_reuse)\n+\t\t\t\tbreak;\n \t\t}\n \n \t\tQSORT(packs, packs_nr, bitmapped_pack_cmp);\n@@ -2080,10 +2096,10 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\tALLOC_GROW(packs, packs_nr + 1, packs_alloc);\n \n \t\tpacks[packs_nr].p = bitmap_git->pack;\n-\t\tpacks[packs_nr].bitmap_pos = 0;\n \t\tpacks[packs_nr].bitmap_nr = bitmap_git->pack->num_objects;\n+\t\tpacks[packs_nr].bitmap_pos = 0;\n \n-\t\tobjects_nr = packs[packs_nr++].p->num_objects;\n+\t\tobjects_nr = packs[packs_nr++].bitmap_nr;\n \t}\n \n \tword_alloc = objects_nr / BITS_IN_EWORD;\n@@ -2091,10 +2107,8 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\tword_alloc++;\n \treuse = bitmap_word_alloc(word_alloc);\n \n-\tif (packs_nr != 1)\n-\t\tBUG(\"pack reuse not yet implemented for multiple packs\");\n-\n-\treuse_partial_packfile_from_bitmap_1(bitmap_git, packs, reuse);\n+\tfor (i = 0; i < packs_nr; i++)\n+\t\treuse_partial_packfile_from_bitmap_1(bitmap_git, &packs[i], reuse);\n \n \tif (bitmap_is_empty(reuse)) {\n \t\tfree(packs);\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 179b343912..c7dea13217 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -80,7 +80,8 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t\t\tstruct bitmapped_pack **packs_out,\n \t\t\t\t\tsize_t *packs_nr_out,\n-\t\t\t\t\tstruct bitmap **reuse_out);\n+\t\t\t\t\tstruct bitmap **reuse_out,\n+\t\t\t\t\tint multi_pack_reuse);\n int rebuild_existing_bitmaps(struct bitmap_index *, struct packing_data *mapping,\n \t\t\t kh_oid_map_t *reused_bitmaps, int show_progress);\n void free_bitmap_index(struct bitmap_index *);\ndiff --git a/t/t5332-multi-pack-reuse.sh b/t/t5332-multi-pack-reuse.sh\nnew file mode 100755\nindex 0000000000..2ba788b042\n--- /dev/null\n+++ b/t/t5332-multi-pack-reuse.sh\n@@ -0,0 +1,203 @@\n+#!/bin/sh\n+\n+test_description='pack-objects multi-pack reuse'\n+\n+. ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-bitmap.sh\n+\n+objdir=.git/objects\n+packdir=$objdir/pack\n+\n+test_pack_reused () {\n+\ttest_trace2_data pack-objects pack-reused \"$1\"\n+}\n+\n+test_packs_reused () {\n+\ttest_trace2_data pack-objects packs-reused \"$1\"\n+}\n+\n+\n+# pack_position objects &&\n+\tgrep \"$1\" objects | cut -d\" \" -f1\n+}\n+\n+test_expect_success 'preferred pack is reused for single-pack reuse' '\n+\ttest_config pack.allowPackReuse single &&\n+\n+\tfor i in A B\n+\tdo\n+\t\ttest_commit \"$i\" &&\n+\t\tgit repack -d || return 1\n+\tdone &&\n+\n+\tgit multi-pack-index write --bitmap &&\n+\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n+\t\tgit pack-objects --stdout --revs --all >/dev/null &&\n+\n+\ttest_pack_reused 3 in <<-EOF &&\n+\t$(git rev-parse C)\n+\t^$(git rev-parse A)\n+\tEOF\n+\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n+\t\tgit pack-objects --stdout --revs /dev/null &&\n+\n+\ttest_pack_reused 6 trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n+\t\tgit pack-objects --stdout --revs --all >/dev/null &&\n+\n+\ttest_pack_reused 9 in <<-EOF &&\n+\t$(git rev-parse E)\n+\t^$(git rev-parse D)\n+\tEOF\n+\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n+\t\tgit pack-objects --stdout --delta-base-offset --revs /dev/null &&\n+\n+\ttest_pack_reused 3 in <<-EOF &&\n+\t$(git rev-parse E)\n+\t^$(git rev-parse D)\n+\tEOF\n+\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n+\t\tgit pack-objects --stdout --delta-base-offset --revs /dev/null &&\n+\n+\ttest_pack_reused 3 f &&\n+\tgit add f &&\n+\ttest_tick &&\n+\tgit commit -m \"delta\" &&\n+\tdelta=\"$(git rev-parse HEAD)\" &&\n+\n+\ttest_seq 64 >f &&\n+\ttest_tick &&\n+\tgit commit -a -m \"base\" &&\n+\tbase=\"$(git rev-parse HEAD)\" &&\n+\n+\ttest_commit other &&\n+\n+\tgit repack -d &&\n+\n+\thave_delta \"$(git rev-parse $delta:f)\" \"$(git rev-parse $base:f)\" &&\n+\n+\tgit multi-pack-index write --bitmap &&\n+\n+\tcat >in <<-EOF &&\n+\t$(git rev-parse other)\n+\t^$base\n+\tEOF\n+\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n+\t\tgit pack-objects --stdout --delta-base-offset --revs /dev/null &&\n+\n+\t# We can only reuse the 3 objects corresponding to \"other\" from\n+\t# the latest pack.\n+\t#\n+\t# This is because even though we want \"delta\", we do not want\n+\t# \"base\", meaning that we have to inflate the delta/base-pair\n+\t# corresponding to the blob in commit \"delta\", which bypasses\n+\t# the pack-reuse mechanism.\n+\t#\n+\t# The remaining objects from the other pack are similarly not\n+\t# reused because their objects are on the uninteresting side of\n+\t# the query.\n+\ttest_pack_reused 3 in <<-EOF &&\n+\t$(git rev-parse $base)\n+\t^$(git rev-parse $delta)\n+\tEOF\n+\n+\tP=\"$(git pack-objects --revs $packdir/pack trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$PWD/trace2.txt\" \\\n+\t\tgit pack-objects --stdout --delta-base-offset --all >/dev/null &&\n+\n+\tpacks_nr=\"$(find $packdir -type f -name \"pack-*.pack\" | wc -l)\" &&\n+\tobjects_nr=\"$(git rev-list --count --all --objects)\" &&\n+\n+\ttest_pack_reused $(($objects_nr - 1)) \n---\n t/perf/p5332-multi-pack-reuse.sh | 81 ++++++++++++++++++++++++++++++++\n 1 file changed, 81 insertions(+)\n create mode 100755 t/perf/p5332-multi-pack-reuse.sh\n\ndiff --git a/t/perf/p5332-multi-pack-reuse.sh b/t/perf/p5332-multi-pack-reuse.sh\nnew file mode 100755\nindex 0000000000..5c6c575d62\n--- /dev/null\n+++ b/t/perf/p5332-multi-pack-reuse.sh\n@@ -0,0 +1,81 @@\n+#!/bin/sh\n+\n+test_description='tests pack performance with multi-pack reuse'\n+\n+. ./perf-lib.sh\n+. \"${TEST_DIRECTORY}/perf/lib-pack.sh\"\n+\n+packdir=.git/objects/pack\n+\n+test_perf_large_repo\n+\n+find_pack () {\n+\tfor idx in $packdir/pack-*.idx\n+\tdo\n+\t\tif git show-index <$idx | grep -q \"$1\"\n+\t\tthen\n+\t\t\tbasename $idx\n+\t\tfi || return 1\n+\tdone\n+}\n+\n+repack_into_n_chunks () {\n+\tgit repack -adk &&\n+\n+\ttest \"$1\" -eq 1 && return ||\n+\n+\tfind $packdir -type f | sort >packs.before &&\n+\n+\t# partition the repository into $1 chunks of consecutive commits, and\n+\t# then create $1 packs with the objects reachable from each chunk\n+\t# (excluding any objects reachable from the previous chunks)\n+\tsz=\"$(($(git rev-list --count --all) / $1))\"\n+\tfor rev in $(git rev-list --all | awk \"NR % $sz == 0\" | tac)\n+\tdo\n+\t\tpack=\"$(echo \"$rev\" | git pack-objects --revs \\\n+\t\t\t--honor-pack-keep --delta-base-offset $packdir/pack)\" &&\n+\t\ttouch $packdir/pack-$pack.keep || return 1\n+\tdone\n+\n+\t# grab any remaining objects not packed by the previous step(s)\n+\tgit pack-objects --revs --all --honor-pack-keep --delta-base-offset \\\n+\t\t$packdir/pack &&\n+\n+\tfind $packdir -type f | sort >packs.after &&\n+\n+\t# and install the whole thing\n+\tfor f in $(comm -12 packs.before packs.after)\n+\tdo\n+\t\trm -f \"$f\" || return 1\n+\tdone\n+\trm -fr $packdir/*.keep\n+}\n+\n+for nr_packs in 1 10 100\n+do\n+\ttest_expect_success \"create $nr_packs-pack scenario\" '\n+\t\trepack_into_n_chunks $nr_packs\n+\t'\n+\n+\ttest_expect_success \"setup bitmaps for $nr_packs-pack scenario\" '\n+\t\tfind $packdir -type f -name \"*.idx\" | sed -e \"s/.*\\/\\(.*\\)$/+\\1/g\" |\n+\t\tgit multi-pack-index write --stdin-packs --bitmap \\\n+\t\t\t--preferred-pack=\"$(find_pack $(git rev-parse HEAD))\"\n+\t'\n+\n+\tfor reuse in single multi\n+\tdo\n+\t\ttest_perf \"clone for $nr_packs-pack scenario ($reuse-pack reuse)\" \"\n+\t\t\tgit for-each-ref --format='%(objectname)' refs/heads refs/tags >in &&\n+\t\t\tgit -c pack.allowPackReuse=$reuse pack-objects \\\n+\t\t\t\t--revs --delta-base-offset --use-bitmap-index \\\n+\t\t\t\t--stdout result\n+\t\t\"\n+\n+\t\ttest_size \"clone size for $nr_packs-pack scenario ($reuse-pack reuse)\" '\n+\t\t\twc -c I haven't looked into the details yet, but it seems that\n> t5309-pack-delta-cycles.sh fails under\n>\n> $ SANITIZE=leak GIT_TEST_PASSING_SANITIZE_LEAK=true make -j16 test\n\nHrm. I tried to reproduce this, but I'm not seeing it. I have:\n\n$ make SANITIZE=leak GIT_TEST_PASSING_SANITIZE_LEAK=true test\n[ ... ]\nAll tests successful.\nFiles=1001, Tests=14558, 48 wallclock secs ( 8.34 usr 2.10 sys + 391.60 cusr 319.67 csys = 721.71 CPU)\nResult: PASS\n\nWith this series applied on top of 1a87c842ec (Start the 2.44 cycle,\n2023-12-09). The tree I get at the end is d148e16f5cfba405a9823cb68540a8c83004f98f.\n\nDid we apply onto different bases?\n\nThanks,\nTaylor\n"},{"id":"485701","messageId":"xmqq7clgnvqv.fsf@gitster.g","threadId":"60565","inReplyTo":"xmqqbkasnxba.fsf@gitster.g","subject":"Re: [PATCH v2 00/26] pack-objects: multi-pack verbatim reuse","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-12-15T00:40:40Z","receivedAt":"2023-12-15T00:43:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> I haven't looked into the details yet, but it seems that\n> t5309-pack-delta-cycles.sh fails under\n>\n> $ SANITIZE=leak GIT_TEST_PASSING_SANITIZE_LEAK=true make -j16 test\n\nHmph, this seems to be elusive. I tried it again and then it did\nnot fail this time.\n"},{"id":"485702","messageId":"ZXurD1NTZ4TAs7WZ@nand.local","threadId":"60565","inReplyTo":"xmqq7clgnvqv.fsf@gitster.g","subject":"Re: [PATCH v2 00/26] pack-objects: multi-pack verbatim reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-15T01:25:35Z","receivedAt":"2023-12-15T01:25:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Dec 14, 2023 at 04:40:40PM -0800, Junio C Hamano wrote:\n> Junio C Hamano writes:\n>\n> > I haven't looked into the details yet, but it seems that\n> > t5309-pack-delta-cycles.sh fails under\n> >\n> > $ SANITIZE=leak GIT_TEST_PASSING_SANITIZE_LEAK=true make -j16 test\n>\n> Hmph, this seems to be elusive. I tried it again and then it did\n> not fail this time.\n\nIndeed, but I was able to reproduce the failure both on my branch and on\n'master' under --stress, yielding the following failure in t5309.6:\n\n + git index-pack --fix-thin --stdin\n fatal: REF_DELTA at offset 46 already resolved (duplicate base 01d7713666f4de822776c7622c10f1b07de280dc?)\n\n =================================================================\n ==3904583==ERROR: LeakSanitizer: detected memory leaks\n\n Direct leak of 32 byte(s) in 1 object(s) allocated from:\n #0 0x7fa790d01986 in __interceptor_realloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:98\n #1 0x7fa790add769 in __pthread_getattr_np nptl/pthread_getattr_np.c:180\n #2 0x7fa790d117c5 in __sanitizer::GetThreadStackTopAndBottom(bool, unsigned long*, unsigned long*) ../../../../src/libsanitizer/sanitizer_common/sanitizer_linux_libcdep.cpp:150\n #3 0x7fa790d11957 in __sanitizer::GetThreadStackAndTls(bool, unsigned long*, unsigned long*, unsigned long*, unsigned long*) ../../../../src/libsanitizer/sanitizer_common/sanitizer_linux_libcdep.cpp:598\n #4 0x7fa790d03fe8 in __lsan::ThreadStart(unsigned int, unsigned long long, __sanitizer::ThreadType) ../../../../src/libsanitizer/lsan/lsan_posix.cpp:51\n #5 0x7fa790d013fd in __lsan_thread_start_func ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:440\n #6 0x7fa790adc3eb in start_thread nptl/pthread_create.c:444\n #7 0x7fa790b5ca5b in clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81\n\n SUMMARY: LeakSanitizer: 32 byte(s) leaked in 1 allocation(s).\n Aborted\n\nThe duplicate base thing is a red-herring, and is an expected result of\nthe test. But the leak is definitely not, and I'm not sure what's going\non here since the frames listed above are in the LSan runtime, not Git.\n\nI'll try to dig into this a bit more, but I'm not quite sure what's\ngoing on yet.\n\nThanks,\nTaylor\n"},{"id":"485718","messageId":"ZXxy1USjjjAbBi++@nand.local","threadId":"60565","inReplyTo":"20231212081238.GD1117953@coredump.intra.peff.net","subject":"Re: [PATCH 00/24] pack-objects: multi-pack verbatim reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-12-15T15:37:57Z","receivedAt":"2023-12-15T15:38:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Dec 12, 2023 at 03:12:38AM -0500, Jeff King wrote:\n> So my question is: how much of what you're seeing is from (1) and (2),\n> and how much is from (3)? Because there are other ways to trigger (3),\n> such as lowering the window size. For example, if you try your same\n> packing example with --window=0, how do the CPU and output size compare\n> to the results of your series? (I'd also check peak memory usage).\n\nInteresting question! Here are some preliminary numbers on my machine\n(which runs Debian unstable with a Intel Xenon W-2255 CPU @ 3.70GHz and\n64GB of RAM).\n\nI ran the following hyperfine command on my testing repository, which\nhas the Git repository broken up into ~75 packs or so:\n\n $ hyperfine -L v single,multi -L window 0,10 \\\n --show-output \\\n -n '{v}-pack reuse, pack.window={window}' \\\n 'git.compile \\\n -c pack.allowPackReuse={v} \\\n -c pack.window={window} \\\n pack-objects --revs --stdout --use-bitmap-index --delta-base-offset /dev/null | wc -c'\n\nWhich gave the following results for runtime:\n\n Benchmark 1: single-pack reuse, pack.window=0\n [...]\n Time (mean ± σ): 1.248 s ± 0.004 s [User: 1.160 s, System: 0.188 s]\n Range (min … max): 1.244 s … 1.259 s 10 runs\n\n Benchmark 2: multi-pack reuse, pack.window=0\n [...]\n Time (mean ± σ): 1.075 s ± 0.005 s [User: 0.990 s, System: 0.188 s]\n Range (min … max): 1.071 s … 1.088 s 10 runs\n\n Benchmark 3: single-pack reuse, pack.window=10\n [...]\n Time (mean ± σ): 6.281 s ± 0.024 s [User: 43.727 s, System: 0.492 s]\n Range (min … max): 6.252 s … 6.326 s 10 runs\n\n Benchmark 4: multi-pack reuse, pack.window=10\n [...]\n Time (mean ± σ): 1.028 s ± 0.002 s [User: 1.150 s, System: 0.184 s]\n Range (min … max): 1.026 s … 1.032 s 10 runs\n\nHere are the average numbers for the resulting \"clone\" size in each of\nthe above configurations:\n\n Benchmark 1: single-pack reuse, pack.window=0\n 264.443 MB\n Benchmark 2: multi-pack reuse, pack.window=0\n 268.670 MB\n Benchmark 3: single-pack reuse, pack.window=10\n 194.355 MB\n Benchmark 4: multi-pack reuse, pack.window=10\n 266.473 MB\n\nSo it looks like setting pack.window=0 (with both single and multi-pack\nreuse) yields a similarly sized pack output as multi-pack reuse with any\nwindow setting.\n\nRunning the same benchmark as above again, but this time sending the\npack output to /dev/null and instead capturing the maximum RSS value\nfrom `/usr/bin/time -v` gives us the following (averages, in MB):\n\n Benchmark 1: single-pack reuse, pack.window=0\n 354.224 MB (max RSS)\n Benchmark 2: multi-pack reuse, pack.window=0\n 315.730 MB (max RSS)\n Benchmark 3: single-pack reuse, pack.window=10\n 470.651 MB (max RSS)\n Benchmark 4: multi-pack reuse, pack.window=10\n 328.786 MB (max RSS)\n\nSo memory usage is similar between runs except for the single-pack reuse\ncase with a window size of 10.\n\nIt looks like the sweet spot is probably multi-pack reuse with a\nsmall-ish window size, which achieves the best of both worlds (small\npack size, relative to other configurations that reuse large portions of\nthe pack, and low memory usage).\n\nIt's pretty close between multi-pack reuse with a window size of 0 and\na window size of 10. If you want to optimize for pack size, you could\ntrade a ~4% reduction in pack size for a ~1% increase in peak memory\nusage.\n\nOf course, YMMV depending on the repository, packing strategy, and\npack.window configuration (among others) while packing. But this should\ngive you a general idea of what to expect.\n\nThanks,\nTaylor\n"},{"id":"485922","messageId":"20231221105124.GD570888@coredump.intra.peff.net","threadId":"60565","inReplyTo":"ZXurD1NTZ4TAs7WZ@nand.local","subject":"Re: [PATCH v2 00/26] pack-objects: multi-pack verbatim reuse","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-12-21T10:51:24Z","receivedAt":"2023-12-21T10:51:25Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 14, 2023 at 08:25:35PM -0500, Taylor Blau wrote:\n\n> On Thu, Dec 14, 2023 at 04:40:40PM -0800, Junio C Hamano wrote:\n> > Junio C Hamano writes:\n> >\n> > > I haven't looked into the details yet, but it seems that\n> > > t5309-pack-delta-cycles.sh fails under\n> > >\n> > > $ SANITIZE=leak GIT_TEST_PASSING_SANITIZE_LEAK=true make -j16 test\n> >\n> > Hmph, this seems to be elusive. I tried it again and then it did\n> > not fail this time.\n> \n> Indeed, but I was able to reproduce the failure both on my branch and on\n> 'master' under --stress, yielding the following failure in t5309.6:\n\nOK, so it's nothing new, and we can ignore it for your series (I haven't\nseen it in the wild yet, but it is something we may need to deal with in\nthe long run if it keeps popping up).\n\n> + git index-pack --fix-thin --stdin\n> fatal: REF_DELTA at offset 46 already resolved (duplicate base 01d7713666f4de822776c7622c10f1b07de280dc?)\n> \n> =================================================================\n> ==3904583==ERROR: LeakSanitizer: detected memory leaks\n> \n> Direct leak of 32 byte(s) in 1 object(s) allocated from:\n> #0 0x7fa790d01986 in __interceptor_realloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:98\n> #1 0x7fa790add769 in __pthread_getattr_np nptl/pthread_getattr_np.c:180\n> #2 0x7fa790d117c5 in __sanitizer::GetThreadStackTopAndBottom(bool, unsigned long*, unsigned long*) ../../../../src/libsanitizer/sanitizer_common/sanitizer_linux_libcdep.cpp:150\n> #3 0x7fa790d11957 in __sanitizer::GetThreadStackAndTls(bool, unsigned long*, unsigned long*, unsigned long*, unsigned long*) ../../../../src/libsanitizer/sanitizer_common/sanitizer_linux_libcdep.cpp:598\n> #4 0x7fa790d03fe8 in __lsan::ThreadStart(unsigned int, unsigned long long, __sanitizer::ThreadType) ../../../../src/libsanitizer/lsan/lsan_posix.cpp:51\n> #5 0x7fa790d013fd in __lsan_thread_start_func ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:440\n> #6 0x7fa790adc3eb in start_thread nptl/pthread_create.c:444\n> #7 0x7fa790b5ca5b in clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81\n> \n> SUMMARY: LeakSanitizer: 32 byte(s) leaked in 1 allocation(s).\n> Aborted\n> \n> The duplicate base thing is a red-herring, and is an expected result of\n> the test. But the leak is definitely not, and I'm not sure what's going\n> on here since the frames listed above are in the LSan runtime, not Git.\n\nI suspect this is a race in LSan caused by a thread calling exit() while\nother threads are spawning. Here's my theory.\n\nWhen a thread is spawned, LSan needs to know where its stack is (so it\ncan look for points to reachable memory). It calls pthread_getattr_np(),\nwhich gets an attributes object that must be cleaned up with\npthread_attr_destroy(). Presumably it does this shortly after. But\nthere's a race window where that attr object is allocated and we haven't\nyet set up the new thread's info. If another thread calls exit() then,\nLSan will run but its book-keeping will be in an inconsistent state.\n\nOne way to work around that would be to make the creation of the threads\natomic. That is, create each in a suspended state, and only let them run\nonce they are all created. There's no option in pthreads to do this, but\nwe can simulate it by having them block on a mutex before starting. And\nindeed, we already have such a lock: the work_lock() that they all use\nto get data to process.\n\nAfter applying this patch:\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex dda94a9f46..0e94819216 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1257,13 +1257,15 @@ static void resolve_deltas(void)\n \tbase_cache_limit = delta_base_cache_limit * nr_threads;\n \tif (nr_threads > 1 || getenv(\"GIT_FORCE_THREADS\")) {\n \t\tinit_thread();\n+\t\twork_lock();\n \t\tfor (i = 0; i < nr_threads; i++) {\n \t\t\tint ret = pthread_create(&thread_data[i].thread, NULL,\n \t\t\t\t\t\t threaded_second_pass, thread_data + i);\n \t\t\tif (ret)\n \t\t\t\tdie(_(\"unable to create thread: %s\"),\n \t\t\t\t strerror(ret));\n \t\t}\n+\t\twork_unlock();\n \t\tfor (i = 0; i < nr_threads; i++)\n \t\t\tpthread_join(thread_data[i].thread, NULL);\n \t\tcleanup_thread();\n\nI ran t5309 with \"--stress --run=6\" for about 5 minutes with no failures\n(whereas without the patch, I usually see a failure in 10 seconds or\nso).\n\nSo it's a pretty easy fix, though I don't love it in general. Every\nplace that spawns multiple threads that can die() would need the same\ntreatment. And this isn't a \"real\" leak in any reasonable sense; it only\nhappens because we're exiting the program directly, at which point all\nof the memory is returned to the OS anyway. So I hate changing\nproduction code to satisfy a leak-checking false positive.\n\nOTOH, dealing with false positives is annoying for humans, and the\nrun-time cost should be negligible. We can work around this one, and\navoid making the same change in other spots unless somebody sees a racy\nfailure in practice.\n\n-Peff\n"},{"id":"485925","messageId":"20231221111333.GE570888@coredump.intra.peff.net","threadId":"60565","inReplyTo":"ZXxy1USjjjAbBi++@nand.local","subject":"Re: [PATCH 00/24] pack-objects: multi-pack verbatim reuse","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-12-21T11:13:33Z","receivedAt":"2023-12-21T11:13:35Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 15, 2023 at 10:37:57AM -0500, Taylor Blau wrote:\n\n> On Tue, Dec 12, 2023 at 03:12:38AM -0500, Jeff King wrote:\n> > So my question is: how much of what you're seeing is from (1) and (2),\n> > and how much is from (3)? Because there are other ways to trigger (3),\n> > such as lowering the window size. For example, if you try your same\n> > packing example with --window=0, how do the CPU and output size compare\n> > to the results of your series? (I'd also check peak memory usage).\n> \n> Interesting question! Here are some preliminary numbers on my machine\n> (which runs Debian unstable with a Intel Xenon W-2255 CPU @ 3.70GHz and\n> 64GB of RAM).\n> \n> I ran the following hyperfine command on my testing repository, which\n> has the Git repository broken up into ~75 packs or so:\n\nThanks for running these tests. The results are similar to what\nexpected, which is: yes, most of your CPU savings are from skipping\ndeltas, but not all.\n\nHere's what I see (which I think is mostly redundant with what you've\nsaid, but I just want to lay out my line of thinking). I'll reorder your\nquoted sections a bit as I go:\n\n> Benchmark 2: multi-pack reuse, pack.window=0\n> [...]\n> Time (mean ± σ): 1.075 s ± 0.005 s [User: 0.990 s, System: 0.188 s]\n> Range (min … max): 1.071 s … 1.088 s 10 runs\n>\n> Benchmark 4: multi-pack reuse, pack.window=10\n> [...]\n> Time (mean ± σ): 1.028 s ± 0.002 s [User: 1.150 s, System: 0.184 s]\n> Range (min … max): 1.026 s … 1.032 s 10 runs\n\nOK, so when we're doing more full (\"multi\") reuse, the pack window\ndoesn't make a big difference either way. You didn't show the stderr\nfrom each, but presumably most of the objects are hitting the \"reuse\"\npath, and only a few are deltas (and that is backed up by the fact that\ndoing deltas only gives us a slight improvement in the output size:\n\n> Benchmark 2: multi-pack reuse, pack.window=0\n> 268.670 MB\n> Benchmark 4: multi-pack reuse, pack.window=10\n> 266.473 MB\n\nComparing the runs with less reuse:\n\n> Benchmark 1: single-pack reuse, pack.window=0\n> [...]\n> Time (mean ± σ): 1.248 s ± 0.004 s [User: 1.160 s, System: 0.188 s]\n> Range (min … max): 1.244 s … 1.259 s 10 runs\n>\n> Benchmark 3: single-pack reuse, pack.window=10\n> [...]\n> Time (mean ± σ): 6.281 s ± 0.024 s [User: 43.727 s, System: 0.492 s]\n> Range (min … max): 6.252 s … 6.326 s 10 runs\n\nthere obviously is a huge amount of time saved by not doing deltas, but\nwe pay for it with a much bigger pack:\n\n> Benchmark 1: single-pack reuse, pack.window=0\n> 264.443 MB\n> Benchmark 3: single-pack reuse, pack.window=10\n> 194.355 MB\n\nBut of course that \"much bigger\" pack is about the same size as the one\nwe get from doing multi-pack reuse. Which is not surprising, because\nboth are avoiding looking for new deltas (and the packs after the\npreferred one probably have mediocre deltas).\n\nSo I do actually think that disabling pack.window gives you a\nsimilar-ish tradeoff to expanding the pack-reuse code (~6s down to ~1s,\nand a 36% embiggening of the resulting pack size).\n\nWhich implies that one option is to scrap your entire series and just\nset pack.window. Basically comparing multi/10 (your patches) to single/0\n(a hypothetical config option), which have similar run-times and pack\nsizes.\n\nBut that's not quite the whole story. There is still a CPU improvement\nin your series (1.2s vs 1.0s, a 20% speedup). And as I'd expect, a\nmemory improvement from avoiding the extra book-keeping (almost 10%):\n\n> Benchmark 1: single-pack reuse, pack.window=0\n> 354.224 MB (max RSS)\n> Benchmark 4: multi-pack reuse, pack.window=10\n> 328.786 MB (max RSS)\n\nSo while it's a lot less code to just set the window size, I do think\nthose improvements are worth it. And really, it's the same tradeoff we\nmake for the single-pack case (i.e., one could argue that we\ncould/should rip out the verbatim-reuse code entirely in favor of just\ntweaking the window size).\n\n> It's pretty close between multi-pack reuse with a window size of 0 and\n> a window size of 10. If you want to optimize for pack size, you could\n> trade a ~4% reduction in pack size for a ~1% increase in peak memory\n> usage.\n\nI think if you want to optimize for pack size, you should consider\nrepacking all-into-one to get better on-disk deltas. ;) I know that's\neasier said than done when the I/O costs are significant. I do wonder if\nstoring thin packs on disk would let us more cheaply reach a state that\ncould serve optimal-ish packs without spending CPU computing bespoke\ndeltas for each client. But that's a much larger topic.\n\n-Peff\n"},{"id":"486320","messageId":"ZZcvnF8wcOYoA8Dd@nand.local","threadId":"60565","inReplyTo":"20231221111333.GE570888@coredump.intra.peff.net","subject":"Re: [PATCH 00/24] pack-objects: multi-pack verbatim reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-01-04T22:22:20Z","receivedAt":"2024-01-04T22:22:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Dec 21, 2023 at 06:13:33AM -0500, Jeff King wrote:\n> But that's not quite the whole story. There is still a CPU improvement\n> in your series (1.2s vs 1.0s, a 20% speedup). And as I'd expect, a\n> memory improvement from avoiding the extra book-keeping (almost 10%):\n>\n> > Benchmark 1: single-pack reuse, pack.window=0\n> > 354.224 MB (max RSS)\n> > Benchmark 4: multi-pack reuse, pack.window=10\n> > 328.786 MB (max RSS)\n\nI agree. And I expect that we'd see larger savings on larger, real-world\nrepositories (the numbers here are generated from a semi out-of-date\ncopy of git.git).\n\n> So while it's a lot less code to just set the window size, I do think\n> those improvements are worth it. And really, it's the same tradeoff we\n> make for the single-pack case (i.e., one could argue that we\n> could/should rip out the verbatim-reuse code entirely in favor of just\n> tweaking the window size).\n\nDefinitely an interesting direction. One question that comes to mind is\nwhether or not we'd want to keep the \"verbatim\" reuse code around to\navoid adding objects to the packing list directly. That's a\nnon-negligible memory savings when generating large packs where we can\nreuse large swaths of existing packs.\n\nThat seems like a topic for another day, though ;-). Not quite\nleft-over-bits, so maybe... #leftoverboulders?\n\nThanks,\nTaylor\n"},{"id":"486323","messageId":"ZZcwLu8TO7Cxs6TZ@nand.local","threadId":"60565","inReplyTo":"20231221105124.GD570888@coredump.intra.peff.net","subject":"Re: [PATCH v2 00/26] pack-objects: multi-pack verbatim reuse","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-01-04T22:24:46Z","receivedAt":"2024-01-04T22:24:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Dec 21, 2023 at 05:51:24AM -0500, Jeff King wrote:\n> I suspect this is a race in LSan caused by a thread calling exit() while\n> other threads are spawning. Here's my theory.\n>\n> When a thread is spawned, LSan needs to know where its stack is (so it\n> can look for points to reachable memory). It calls pthread_getattr_np(),\n> which gets an attributes object that must be cleaned up with\n> pthread_attr_destroy(). Presumably it does this shortly after. But\n> there's a race window where that attr object is allocated and we haven't\n> yet set up the new thread's info. If another thread calls exit() then,\n> LSan will run but its book-keeping will be in an inconsistent state.\n\nThanks for digging. I agree with your theory, and am annoyed with how\nclever it is ;-).\n\n> So it's a pretty easy fix, though I don't love it in general. Every\n> place that spawns multiple threads that can die() would need the same\n> treatment. And this isn't a \"real\" leak in any reasonable sense; it only\n> happens because we're exiting the program directly, at which point all\n> of the memory is returned to the OS anyway. So I hate changing\n> production code to satisfy a leak-checking false positive.\n>\n> OTOH, dealing with false positives is annoying for humans, and the\n> run-time cost should be negligible. We can work around this one, and\n> avoid making the same change in other spots unless somebody sees a racy\n> failure in practice.\n\nYeah... I share your thoughts here as well. It's kind of gross that we\nhave to touch production code purely to appease the leak checker, but I\nthink that the trade-off is worth it since:\n\n - the false positives are annoying to diagnose (as you said, and as\n evidenced by the time that you, Junio, and myself have sunk into\n discussing this ;-)).\n\n - the run-time cost is negligible.\n\nSo I think that this is a good change to make, and I'm happy to see it\ngo through. I don't think we should necessarily try too hard to find all\nspots that might benefit from a similar change, and instead just apply\nthe same treatment if/when we notice false positives in CI.\n\nThanks,\nTaylor\n"}]}