{"thread":{"id":"56482","subject":"[PATCH 0/8] repack: introduce `--write-midx`","startedAt":"2021-09-11T03:32:38Z","lastAt":"2021-10-01T22:40:54Z","messageCount":76,"participants":["Taylor Blau","Junio C Hamano","Ævar Arnfjörð Bjarmason","Eric Sunshine","Jeff King","Josh Steadmon","Jonathan Tan"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"435506","messageId":"cover.1631331139.git.me@ttaylorr.com","threadId":"56482","inReplyTo":null,"subject":"[PATCH 0/8] repack: introduce `--write-midx`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-11T03:32:25Z","receivedAt":"2021-09-11T03:32:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This series implements the final component of multi-pack reachability bitmaps:\nthe ability to write them from `git repack`.\n\nThe gist is that it is currently impossible to write a MIDX based on the result\nof a `git repack` without the possibility that an existing MIDX may go away.\nThat's because `git repack` may have deleted a pack that was included in the\nMIDX (either because it was rolled up when doing a `--geometric` repack, or the\nrepack was done all-into-one), and MIDXs cannot refer to deleted packs.\n\nTo solve this, `git repack` is taught a new option to write a MIDX between the\ncritical section: right after the new packs are written and moved into place,\nbut right before any old packs are deleted.\n\nUnfortunately, there is quite a bit of auxiliary tooling we need to implement\nbefore this is possible. The series is laid out like so:\n\n  - The first two patches lay the ground work for and expose `git\n    multi-pack-index --stdin-packs`, which allows callers to write a MIDX\n    containing only certain packs.\n\n  - The next patch implements preliminary support for `--refs-snapshot`, which\n    allows the MIDX bitmap selection code to reference the prior state of\n    reference tips, preventing a race.\n\n  - The next two patches prepare the `repack` builtin to write a MIDX, and the\n    following patch actually implements `--write-midx`.\n\n  - The final two patches make sure the MIDX generated has an appropriately\n    chosen preferred pack, and that the refs-snapshot is provided.\n\nIt is based on `tb/multi-pack-bitmaps`. Thanks in advance for your review!\n\nTaylor Blau (8):\n  midx: expose 'write_midx_file_only()' publicly\n  builtin/multi-pack-index.c: support --stdin-packs mode\n  midx: preliminary support for `--refs-snapshot`\n  builtin/repack.c: keep track of existing packs unconditionally\n  builtin/repack.c: extract showing progress to a variable\n  builtin/repack.c: support writing a MIDX while repacking\n  builtin/repack.c: make largest pack preferred\n  builtin/repack.c: pass `--refs-snapshot` when writing bitmaps\n\n Documentation/git-multi-pack-index.txt |  19 ++\n Documentation/git-repack.txt           |  18 +-\n builtin/multi-pack-index.c             |  35 +++-\n builtin/repack.c                       | 251 ++++++++++++++++++++++---\n midx.c                                 | 102 ++++++++--\n midx.h                                 |   4 +-\n pack-bitmap.c                          |   2 +-\n pack-bitmap.h                          |   1 +\n t/helper/test-read-midx.c              |  25 ++-\n t/lib-midx.sh                          |   8 +\n t/t5319-multi-pack-index.sh            |  15 ++\n t/t5326-multi-pack-bitmaps.sh          |  82 ++++++++\n t/t7700-repack.sh                      |  96 ++++++++++\n t/t7703-repack-geometric.sh            |  23 +++\n 14 files changed, 630 insertions(+), 51 deletions(-)\n create mode 100644 t/lib-midx.sh\n\n-- \n2.33.0.96.g73915697e6\n"},{"id":"435507","messageId":"4afa03b972a1885c60fbf3716f22a7ab58056383.1631331139.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631331139.git.me@ttaylorr.com","subject":"[PATCH 1/8] midx: expose 'write_midx_file_only()' publicly","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-11T03:32:29Z","receivedAt":"2021-09-11T03:32:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Expose a variant of the write_midx_file() function which ignores packs\nthat aren't included in an explicit \"allow\" list.\n\nThis will be used in an upcoming patch to power a new `--stdin-packs`\nmode of `git multi-pack-index write` for callers that only want to\ninclude certain packs in a MIDX (and ignore any packs which may have\nhappened to enter the repository independently, e.g., from pushes).\n\nThose patches will provide test coverage for this new function.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx.c | 50 +++++++++++++++++++++++++++++++++++++++-----------\n midx.h |  2 ++\n 2 files changed, 41 insertions(+), 11 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 864034a6ad..29d1d107b3 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -475,6 +475,8 @@ struct write_midx_context {\n \tuint32_t num_large_offsets;\n \n \tint preferred_pack_idx;\n+\n+\tstruct string_list *to_include;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -484,8 +486,13 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \n \tif (ends_with(file_name, \".idx\")) {\n \t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n-\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n-\t\t\treturn;\n+\t\tif (ctx->m) {\n+\t\t\tif (midx_contains_pack(ctx->m, file_name))\n+\t\t\t\treturn;\n+\t\t} else if (ctx->to_include) {\n+\t\t\tif (!string_list_has_string(ctx->to_include, file_name))\n+\t\t\t\treturn;\n+\t\t}\n \n \t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n \n@@ -1058,6 +1065,7 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n }\n \n static int write_midx_internal(const char *object_dir,\n+\t\t\t       struct string_list *packs_to_include,\n \t\t\t       struct string_list *packs_to_drop,\n \t\t\t       const char *preferred_pack_name,\n \t\t\t       unsigned flags)\n@@ -1082,10 +1090,17 @@ static int write_midx_internal(const char *object_dir,\n \t\tdie_errno(_(\"unable to create leading directories of %s\"),\n \t\t\t  midx_name);\n \n-\tfor (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {\n-\t\tif (!strcmp(object_dir, cur->object_dir)) {\n-\t\t\tctx.m = cur;\n-\t\t\tbreak;\n+\tif (!packs_to_include) {\n+\t\t/*\n+\t\t * Only reference an existing MIDX when not filtering which\n+\t\t * packs to include, since all packs and objects are copied\n+\t\t * blindly from an existing MIDX if one is present.\n+\t\t */\n+\t\tfor (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {\n+\t\t\tif (!strcmp(object_dir, cur->object_dir)) {\n+\t\t\t\tctx.m = cur;\n+\t\t\t\tbreak;\n+\t\t\t}\n \t\t}\n \t}\n \n@@ -1136,10 +1151,13 @@ static int write_midx_internal(const char *object_dir,\n \telse\n \t\tctx.progress = NULL;\n \n+\tctx.to_include = packs_to_include;\n+\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n-\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop) {\n+\tif ((ctx.m && ctx.nr == ctx.m->num_packs) &&\n+\t    !(packs_to_include || packs_to_drop)) {\n \t\tstruct bitmap_index *bitmap_git;\n \t\tint bitmap_exists;\n \t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n@@ -1237,7 +1255,7 @@ static int write_midx_internal(const char *object_dir,\n \n \tQSORT(ctx.info, ctx.nr, pack_info_compare);\n \n-\tif (packs_to_drop && packs_to_drop->nr) {\n+\tif (ctx.m && packs_to_drop && packs_to_drop->nr) {\n \t\tint drop_index = 0;\n \t\tint missing_drops = 0;\n \n@@ -1380,7 +1398,17 @@ int write_midx_file(const char *object_dir,\n \t\t    const char *preferred_pack_name,\n \t\t    unsigned flags)\n {\n-\treturn write_midx_internal(object_dir, NULL, preferred_pack_name, flags);\n+\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n+\t\t\t\t   flags);\n+}\n+\n+int write_midx_file_only(const char *object_dir,\n+\t\t\t struct string_list *packs_to_include,\n+\t\t\t const char *preferred_pack_name,\n+\t\t\t unsigned flags)\n+{\n+\treturn write_midx_internal(object_dir, packs_to_include, NULL,\n+\t\t\t\t   preferred_pack_name, flags);\n }\n \n struct clear_midx_data {\n@@ -1660,7 +1688,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tfree(count);\n \n \tif (packs_to_drop.nr) {\n-\t\tresult = write_midx_internal(object_dir, &packs_to_drop, NULL, flags);\n+\t\tresult = write_midx_internal(object_dir, NULL, &packs_to_drop, NULL, flags);\n \t\tm = NULL;\n \t}\n \n@@ -1851,7 +1879,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t\tgoto cleanup;\n \t}\n \n-\tresult = write_midx_internal(object_dir, NULL, NULL, flags);\n+\tresult = write_midx_internal(object_dir, NULL, NULL, NULL, flags);\n \tm = NULL;\n \n cleanup:\ndiff --git a/midx.h b/midx.h\nindex aa3da557bb..aefa371c90 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -2,6 +2,7 @@\n #define MIDX_H\n \n #include \"repository.h\"\n+#include \"string-list.h\"\n \n struct object_id;\n struct pack_entry;\n@@ -62,6 +63,7 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n \n int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n+int write_midx_file_only(const char *object_dir, struct string_list *packs_to_include, const char *preferred_pack_name, unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);\n int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435508","messageId":"2a16f11790b79ab452233b6f28acac607c0abd28.1631331139.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631331139.git.me@ttaylorr.com","subject":"[PATCH 2/8] builtin/multi-pack-index.c: support --stdin-packs mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-11T03:32:31Z","receivedAt":"2021-09-11T03:32:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To power a new `--write-midx` mode, `git repack` will want to write a\nmulti-pack index containing a certain set of packs in the repository.\n\nThis new option will be used by `git repack` to write a MIDX which\ncontains only the packs which will survive after the repack (that is, it\nwill exclude any packs which are about to be deleted).\n\nThis patch effectively exposes the function implemented in the previous\ncommit via the `git multi-pack-index` builtin. An alternative approach\nwould have been to call that function from the `git repack` builtin\ndirectly, but this introduces awkward problems around closing and\nreopening the object store, so the MIDX will be written out-of-process.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-multi-pack-index.txt |  4 ++++\n builtin/multi-pack-index.c             | 26 ++++++++++++++++++++++++++\n t/t5319-multi-pack-index.sh            | 15 +++++++++++++++\n 3 files changed, 45 insertions(+)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex a9df3dbd32..009c989ef8 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -45,6 +45,10 @@ write::\n \n \t--[no-]bitmap::\n \t\tControl whether or not a multi-pack bitmap is written.\n+\n+\t--stdin-packs::\n+\t\tWrite a multi-pack index containing only the set of\n+\t\tline-delimited pack index basenames provided over stdin.\n --\n \n verify::\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 73c0113b48..77488b6b7b 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -47,6 +47,7 @@ static struct opts_multi_pack_index {\n \tconst char *preferred_pack;\n \tunsigned long batch_size;\n \tunsigned flags;\n+\tint stdin_packs;\n } opts;\n \n static struct option common_opts[] = {\n@@ -61,6 +62,15 @@ static struct option *add_common_options(struct option *prev)\n \treturn parse_options_concat(common_opts, prev);\n }\n \n+static void read_packs_from_stdin(struct string_list *to)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\twhile (strbuf_getline(&buf, stdin) != EOF) {\n+\t\tstring_list_append(to, strbuf_detach(&buf, NULL));\n+\t}\n+\tstring_list_sort(to);\n+}\n+\n static int cmd_multi_pack_index_write(int argc, const char **argv)\n {\n \tstruct option *options;\n@@ -70,6 +80,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\t\t   N_(\"pack for reuse when computing a multi-pack bitmap\")),\n \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n+\t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n+\t\t\t N_(\"write multi-pack index containing only given indexes\")),\n \t\tOPT_END(),\n \t};\n \n@@ -86,6 +98,20 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \n \tFREE_AND_NULL(options);\n \n+\tif (opts.stdin_packs) {\n+\t\tstruct string_list packs = STRING_LIST_INIT_NODUP;\n+\t\tint ret;\n+\n+\t\tread_packs_from_stdin(&packs);\n+\n+\t\tret = write_midx_file_only(opts.object_dir, &packs,\n+\t\t\t\t\t   opts.preferred_pack, opts.flags);\n+\n+\t\tstring_list_clear(&packs, 0);\n+\n+\t\treturn ret;\n+\n+\t}\n \treturn write_midx_file(opts.object_dir, opts.preferred_pack,\n \t\t\t       opts.flags);\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex bb04f0f23b..385f0a3efd 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -168,6 +168,21 @@ test_expect_success 'write midx with two packs' '\n \n compare_results_with_midx \"two packs\"\n \n+test_expect_success 'write midx with --stdin-packs' '\n+\trm -fr $objdir/pack/multi-pack-index &&\n+\n+\tidx=\"$(find $objdir/pack -name \"test-2-*.idx\")\" &&\n+\tbasename \"$idx\" >in &&\n+\n+\tgit multi-pack-index write --stdin-packs <in &&\n+\n+\ttest-tool read-midx $objdir | grep \"\\.idx$\" >packs &&\n+\n+\ttest_cmp packs in\n+'\n+\n+compare_results_with_midx \"mixed mode (one pack + extra)\"\n+\n test_expect_success 'write progress off for redirected stderr' '\n \tgit multi-pack-index --object-dir=$objdir write 2>err &&\n \ttest_line_count = 0 err\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435509","messageId":"243cf5f82cabfb930395ff3cb97d1354f4b16064.1631331139.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631331139.git.me@ttaylorr.com","subject":"[PATCH 4/8] builtin/repack.c: keep track of existing packs unconditionally","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-11T03:32:36Z","receivedAt":"2021-09-11T03:32:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In order to be able to write a multi-pack index during repacking, `git\nrepack` must keep track of which packs it wants to write into the MIDX.\nThis set is the union of existing packs which will not be deleted,\nnew pack(s) generated as a result of the repack, and .keep packs.\n\nPrior to this patch, `git repack` populated the list of existing packs\nonly when repacking all-into-one (i.e., with `-A` or `-a`), but we will\nsoon need to know this list when repacking when writing a MIDX without\na-i-o.\n\nPopulate the list of existing packs unconditionally, and guard removing\npacks from that list only when repacking a-i-o.\n\nAdditionally, keep track of filenames of kept packs separately, since\nthis, too, will be used in an upcoming patch.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/repack.c | 49 ++++++++++++++++++++++++++----------------------\n 1 file changed, 27 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 27158a897b..e55a650de5 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -98,8 +98,9 @@ static void remove_pack_on_signal(int signo)\n  * have a corresponding .keep file. These packs are not to\n  * be kept if we are going to pack everything into one file.\n  */\n-static void get_non_kept_pack_filenames(struct string_list *fname_list,\n-\t\t\t\t\tconst struct string_list *extra_keep)\n+static void collect_pack_filenames(struct string_list *fname_list,\n+\t\t\t\t   struct string_list *fname_kept_list,\n+\t\t\t\t   const struct string_list *extra_keep)\n {\n \tDIR *dir;\n \tstruct dirent *e;\n@@ -112,21 +113,20 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list,\n \t\tsize_t len;\n \t\tint i;\n \n+\t\tif (!strip_suffix(e->d_name, \".pack\", &len))\n+\t\t\tcontinue;\n+\n \t\tfor (i = 0; i < extra_keep->nr; i++)\n \t\t\tif (!fspathcmp(e->d_name, extra_keep->items[i].string))\n \t\t\t\tbreak;\n-\t\tif (extra_keep->nr > 0 && i < extra_keep->nr)\n-\t\t\tcontinue;\n-\n-\t\tif (!strip_suffix(e->d_name, \".pack\", &len))\n-\t\t\tcontinue;\n \n \t\tfname = xmemdupz(e->d_name, len);\n \n-\t\tif (!file_exists(mkpath(\"%s/%s.keep\", packdir, fname)))\n-\t\t\tstring_list_append_nodup(fname_list, fname);\n+\t\tif ((extra_keep->nr > 0 && i < extra_keep->nr) ||\n+\t\t    (file_exists(mkpath(\"%s/%s.keep\", packdir, fname))))\n+\t\t\tstring_list_append_nodup(fname_kept_list, fname);\n \t\telse\n-\t\t\tfree(fname);\n+\t\t\tstring_list_append_nodup(fname_list, fname);\n \t}\n \tclosedir(dir);\n }\n@@ -440,6 +440,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list names = STRING_LIST_INIT_DUP;\n \tstruct string_list rollback = STRING_LIST_INIT_NODUP;\n \tstruct string_list existing_packs = STRING_LIST_INIT_DUP;\n+\tstruct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n \tstruct pack_geometry *geometry = NULL;\n \tstruct strbuf line = STRBUF_INIT;\n \tint i, ext, ret;\n@@ -572,9 +573,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (use_delta_islands)\n \t\tstrvec_push(&cmd.args, \"--delta-islands\");\n \n+\tcollect_pack_filenames(&existing_packs, &existing_kept_packs,\n+\t\t\t       &keep_pack_list);\n+\n \tif (pack_everything & ALL_INTO_ONE) {\n-\t\tget_non_kept_pack_filenames(&existing_packs, &keep_pack_list);\n-\n \t\trepack_promisor_objects(&po_args, &names);\n \n \t\tif (existing_packs.nr && delete_redundant) {\n@@ -683,17 +685,19 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \treprepare_packed_git(the_repository);\n \n \tif (delete_redundant) {\n-\t\tconst int hexsz = the_hash_algo->hexsz;\n \t\tint opts = 0;\n-\t\tstring_list_sort(&names);\n-\t\tfor_each_string_list_item(item, &existing_packs) {\n-\t\t\tchar *sha1;\n-\t\t\tsize_t len = strlen(item->string);\n-\t\t\tif (len < hexsz)\n-\t\t\t\tcontinue;\n-\t\t\tsha1 = item->string + len - hexsz;\n-\t\t\tif (!string_list_has_string(&names, sha1))\n-\t\t\t\tremove_redundant_pack(packdir, item->string);\n+\t\tif (pack_everything & ALL_INTO_ONE) {\n+\t\t\tconst int hexsz = the_hash_algo->hexsz;\n+\t\t\tstring_list_sort(&names);\n+\t\t\tfor_each_string_list_item(item, &existing_packs) {\n+\t\t\t\tchar *sha1;\n+\t\t\t\tsize_t len = strlen(item->string);\n+\t\t\t\tif (len < hexsz)\n+\t\t\t\t\tcontinue;\n+\t\t\t\tsha1 = item->string + len - hexsz;\n+\t\t\t\tif (!string_list_has_string(&names, sha1))\n+\t\t\t\t\tremove_redundant_pack(packdir, item->string);\n+\t\t\t}\n \t\t}\n \n \t\tif (geometry) {\n@@ -739,6 +743,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstring_list_clear(&names, 0);\n \tstring_list_clear(&rollback, 0);\n \tstring_list_clear(&existing_packs, 0);\n+\tstring_list_clear(&existing_kept_packs, 0);\n \tclear_pack_geometry(geometry);\n \tstrbuf_release(&line);\n \n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435510","messageId":"137759fe6c7c095b980d55dc4ba37f663044a6dd.1631331139.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631331139.git.me@ttaylorr.com","subject":"[PATCH 3/8] midx: preliminary support for `--refs-snapshot`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-11T03:32:34Z","receivedAt":"2021-09-11T03:32:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To figure out which commits we can write a bitmap for, the multi-pack\nindex/bitmap code does a reachability traversal, marking any commit\nwhich can be found in the MIDX as eligible to receive a bitmap.\n\nThis approach will cause a problem when multi-pack bitmaps are able to\nbe generated from `git repack`, since the reference tips can change\nduring the repack. Even though we ignore commits that don't exist in\nthe MIDX (when doing a scan of the ref tips), it's possible that a\ncommit in the MIDX reaches something that isn't.\n\nThis can happen when a multi-pack index contains some pack which refers\nto loose objects (which by definition aren't included in the multi-pack\nindex).\n\nBy taking a snapshot of the references before we start repacking, we can\nclose that race window. In the above scenario (where we have a packed\nobject pointing at a loose one), we'll either (a) take a snapshot of the\nreferences before seeing the packed one, or (b) take it after, at which\npoint we can guarantee that the loose object will be packed and included\nin the MIDX.\n\nThis patch does just that. It writes a temporary \"reference snapshot\",\nwhich is a list of OIDs that are at the ref tips before writing a\nmulti-pack bitmap. References that are \"preferred\" (i.e,. are a suffix\nof at least one value of the 'pack.preferBitmapTips' configuration) are\nmarked with a special '+'.\n\nThe format is simple: one line per commit at each tip, with an optional\n'+' at the beginning (for preferred references, as described above).\n\nWhen provided, the reference snapshot is used to drive bitmap selection\ninstead of the MIDX code doing its own traversal. When it isn't\nprovided, the usual traversal takes place instead.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-multi-pack-index.txt | 15 +++++\n builtin/multi-pack-index.c             | 11 +++-\n builtin/repack.c                       |  2 +-\n midx.c                                 | 60 ++++++++++++++++---\n midx.h                                 |  4 +-\n t/t5326-multi-pack-bitmaps.sh          | 82 ++++++++++++++++++++++++++\n 6 files changed, 160 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 009c989ef8..27f83932e4 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -49,6 +49,21 @@ write::\n \t--stdin-packs::\n \t\tWrite a multi-pack index containing only the set of\n \t\tline-delimited pack index basenames provided over stdin.\n+\n+\t--refs-snapshot=<path>::\n+\t\tWith `--bitmap`, optionally specify a file which\n+\t\tcontains a \"refs snapshot\" taken prior to repacking.\n++\n+A reference snapshot is composed of line-delimited OIDs corresponding to\n+the reference tips, usually taken by `git repack` prior to generating a\n+new pack. A line may optionally start with a `+` character to indicate\n+that the reference which corresponds to that OID is \"preferred\" (see\n+linkgit:git-config[1]'s `pack.preferBitmapTips`.)\n++\n+The file given at `<path>` is expected to be readable, and can contain\n+duplicates. (If a given OID is given more than once, it is marked as\n+preferred if at least one instance of it begins with the special `+`\n+marker).\n --\n \n verify::\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 77488b6b7b..65a242f5cf 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -7,7 +7,8 @@\n #include \"object-store.h\"\n \n #define BUILTIN_MIDX_WRITE_USAGE \\\n-\tN_(\"git multi-pack-index [<options>] write [--preferred-pack=<pack>]\")\n+\tN_(\"git multi-pack-index [<options>] write [--preferred-pack=<pack>]\" \\\n+\t   \"[--refs-snapshot=<path>]\")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [<options>] verify\")\n@@ -45,6 +46,7 @@ static char const * const builtin_multi_pack_index_usage[] = {\n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n \tconst char *preferred_pack;\n+\tconst char *refs_snapshot;\n \tunsigned long batch_size;\n \tunsigned flags;\n \tint stdin_packs;\n@@ -82,6 +84,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\t N_(\"write multi-pack index containing only given indexes\")),\n+\t\tOPT_FILENAME(0, \"refs-snapshot\", &opts.refs_snapshot,\n+\t\t\t     N_(\"refs snapshot for selecting bitmap commits\")),\n \t\tOPT_END(),\n \t};\n \n@@ -105,7 +109,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\tread_packs_from_stdin(&packs);\n \n \t\tret = write_midx_file_only(opts.object_dir, &packs,\n-\t\t\t\t\t   opts.preferred_pack, opts.flags);\n+\t\t\t\t\t   opts.preferred_pack,\n+\t\t\t\t\t   opts.refs_snapshot, opts.flags);\n \n \t\tstring_list_clear(&packs, 0);\n \n@@ -113,7 +118,7 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \n \t}\n \treturn write_midx_file(opts.object_dir, opts.preferred_pack,\n-\t\t\t       opts.flags);\n+\t\t\t       opts.refs_snapshot, opts.flags);\n }\n \n static int cmd_multi_pack_index_verify(int argc, const char **argv)\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 82ab668272..27158a897b 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -733,7 +733,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tunsigned flags = 0;\n \t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))\n \t\t\tflags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;\n-\t\twrite_midx_file(get_object_directory(), NULL, flags);\n+\t\twrite_midx_file(get_object_directory(), NULL, NULL, flags);\n \t}\n \n \tstring_list_clear(&names, 0);\ndiff --git a/midx.c b/midx.c\nindex 29d1d107b3..46ded7a4cf 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -970,7 +970,42 @@ static void bitmap_show_commit(struct commit *commit, void *_data)\n \tdata->commits[data->commits_nr++] = commit;\n }\n \n+static int read_refs_snapshot(const char *refs_snapshot,\n+\t\t\t      struct rev_info *revs)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct object_id oid;\n+\tFILE *f = xfopen(refs_snapshot, \"r\");\n+\n+\twhile (strbuf_getline(&buf, f) != EOF) {\n+\t\tstruct object *object;\n+\t\tint preferred = 0;\n+\t\tchar *hex = buf.buf;\n+\t\tconst char *end = NULL;\n+\n+\t\tif (buf.len && *buf.buf == '+') {\n+\t\t\tpreferred = 1;\n+\t\t\thex = &buf.buf[1];\n+\t\t}\n+\n+\t\tif (parse_oid_hex(hex, &oid, &end) < 0)\n+\t\t\tdie(_(\"could not parse line: %s\"), buf.buf);\n+\t\tif (*end)\n+\t\t\tdie(_(\"malformed line: %s\"), buf.buf);\n+\n+\t\tobject = parse_object_or_die(&oid, NULL);\n+\t\tif (preferred)\n+\t\t\tobject->flags |= NEEDS_BITMAP;\n+\n+\t\tadd_pending_object(revs, object, \"\");\n+\t}\n+\n+\tfclose(f);\n+\treturn 0;\n+}\n+\n static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p,\n+\t\t\t\t\t\t    const char *refs_snapshot,\n \t\t\t\t\t\t    struct write_midx_context *ctx)\n {\n \tstruct rev_info revs;\n@@ -979,8 +1014,12 @@ static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr\n \tcb.ctx = ctx;\n \n \trepo_init_revisions(the_repository, &revs, NULL);\n-\tsetup_revisions(0, NULL, &revs, NULL);\n-\tfor_each_ref(add_ref_to_pending, &revs);\n+\tif (refs_snapshot) {\n+\t\tread_refs_snapshot(refs_snapshot, &revs);\n+\t} else {\n+\t\tsetup_revisions(0, NULL, &revs, NULL);\n+\t\tfor_each_ref(add_ref_to_pending, &revs);\n+\t}\n \n \t/*\n \t * Skipping promisor objects here is intentional, since it only excludes\n@@ -1009,6 +1048,7 @@ static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr\n \n static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n \t\t\t     struct write_midx_context *ctx,\n+\t\t\t     const char *refs_snapshot,\n \t\t\t     unsigned flags)\n {\n \tstruct packing_data pdata;\n@@ -1020,7 +1060,7 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n \n \tprepare_midx_packing_data(&pdata, ctx);\n \n-\tcommits = find_commits_for_midx_bitmap(&commits_nr, ctx);\n+\tcommits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, ctx);\n \n \t/*\n \t * Build the MIDX-order index based on pdata.objects (which is already\n@@ -1068,6 +1108,7 @@ static int write_midx_internal(const char *object_dir,\n \t\t\t       struct string_list *packs_to_include,\n \t\t\t       struct string_list *packs_to_drop,\n \t\t\t       const char *preferred_pack_name,\n+\t\t\t       const char *refs_snapshot,\n \t\t\t       unsigned flags)\n {\n \tchar *midx_name;\n@@ -1361,7 +1402,8 @@ static int write_midx_internal(const char *object_dir,\n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n \tif (flags & MIDX_WRITE_BITMAP) {\n-\t\tif (write_midx_bitmap(midx_name, midx_hash, &ctx, flags) < 0) {\n+\t\tif (write_midx_bitmap(midx_name, midx_hash, &ctx,\n+\t\t\t\t      refs_snapshot, flags) < 0) {\n \t\t\terror(_(\"could not write multi-pack bitmap\"));\n \t\t\tresult = 1;\n \t\t\tgoto cleanup;\n@@ -1396,19 +1438,21 @@ static int write_midx_internal(const char *object_dir,\n \n int write_midx_file(const char *object_dir,\n \t\t    const char *preferred_pack_name,\n+\t\t    const char *refs_snapshot,\n \t\t    unsigned flags)\n {\n \treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n-\t\t\t\t   flags);\n+\t\t\t\t   refs_snapshot, flags);\n }\n \n int write_midx_file_only(const char *object_dir,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n+\t\t\t const char *refs_snapshot,\n \t\t\t unsigned flags)\n {\n \treturn write_midx_internal(object_dir, packs_to_include, NULL,\n-\t\t\t\t   preferred_pack_name, flags);\n+\t\t\t\t   preferred_pack_name, refs_snapshot, flags);\n }\n \n struct clear_midx_data {\n@@ -1688,7 +1732,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tfree(count);\n \n \tif (packs_to_drop.nr) {\n-\t\tresult = write_midx_internal(object_dir, NULL, &packs_to_drop, NULL, flags);\n+\t\tresult = write_midx_internal(object_dir, NULL, &packs_to_drop, NULL, NULL, flags);\n \t\tm = NULL;\n \t}\n \n@@ -1879,7 +1923,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t\tgoto cleanup;\n \t}\n \n-\tresult = write_midx_internal(object_dir, NULL, NULL, NULL, flags);\n+\tresult = write_midx_internal(object_dir, NULL, NULL, NULL, NULL, flags);\n \tm = NULL;\n \n cleanup:\ndiff --git a/midx.h b/midx.h\nindex aefa371c90..dc37b94ea1 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -62,8 +62,8 @@ int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pa\n int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n \n-int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n-int write_midx_file_only(const char *object_dir, struct string_list *packs_to_include, const char *preferred_pack_name, unsigned flags);\n+int write_midx_file(const char *object_dir, const char *preferred_pack_name, const char *refs_snapshot, unsigned flags);\n+int write_midx_file_only(const char *object_dir, struct string_list *packs_to_include, const char *preferred_pack_name, const char *refs_snapshot, unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);\n int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 4ad7c2c969..069dab3e17 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -283,4 +283,86 @@ test_expect_success 'pack.preferBitmapTips' '\n \t)\n '\n \n+test_expect_success 'writing a bitmap with --refs-snapshot' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit one &&\n+\t\ttest_commit two &&\n+\n+\t\tgit rev-parse one >snapshot &&\n+\n+\t\tgit repack -ad &&\n+\n+\t\t# First, write a MIDX which see both refs/tags/one and\n+\t\t# refs/tags/two (causing both of those commits to receive\n+\t\t# bitmaps).\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\n+\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n+\t\tgrep \"$(git rev-parse two)\" bitmaps &&\n+\n+\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n+\t\trm -fr $midx-$(midx_checksum $objdir).rev &&\n+\t\trm -fr $midx &&\n+\n+\t\t# Then again, but with a refs snapshot which only sees\n+\t\t# refs/tags/one.\n+\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\n+\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n+\t\t! grep \"$(git rev-parse two)\" bitmaps\n+\t)\n+'\n+\n+test_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\n+\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\tsort <commits.raw >commits &&\n+\n+\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\tgit update-ref --stdin <refs &&\n+\n+\t\tgit multi-pack-index write --bitmap &&\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\n+\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\tcomm -13 bitmaps commits >before &&\n+\t\ttest_line_count = 1 before &&\n+\n+\t\t(\n+\t\t\tgrep -vf before commits.raw &&\n+\t\t\t# mark missing commits as preferred\n+\t\t\tsed \"s/^/+/\" before\n+\t\t) >snapshot &&\n+\n+\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n+\t\trm -fr $midx-$(midx_checksum $objdir).rev &&\n+\t\trm -fr $midx &&\n+\n+\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n+\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t! test_cmp before after\n+\t)\n+'\n+\n test_done\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435511","messageId":"aceb443ea966162309d4dd40651e4d5c48959b46.1631331139.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631331139.git.me@ttaylorr.com","subject":"[PATCH 5/8] builtin/repack.c: extract showing progress to a variable","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-11T03:32:42Z","receivedAt":"2021-09-11T03:32:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"We only ask whether stderr is a tty before calling\n'prune_packed_objects()', but the subsequent patch will add another use.\n\nExtract this check into a variable so that both can use it without\nhaving to call 'isatty()' twice.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/repack.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex e55a650de5..be470546e6 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -445,6 +445,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct strbuf line = STRBUF_INIT;\n \tint i, ext, ret;\n \tFILE *out;\n+\tint show_progress = isatty(2);\n \n \t/* variables to be filled by option parsing */\n \tint pack_everything = 0;\n@@ -718,7 +719,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t}\n \t\t\tstrbuf_release(&buf);\n \t\t}\n-\t\tif (!po_args.quiet && isatty(2))\n+\t\tif (!po_args.quiet && show_progress)\n \t\t\topts |= PRUNE_PACKED_VERBOSE;\n \t\tprune_packed_objects(opts);\n \n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435512","messageId":"06b99a0ab5ef52165538f1fe028515097276ed67.1631331139.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631331139.git.me@ttaylorr.com","subject":"[PATCH 6/8] builtin/repack.c: support writing a MIDX while repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-11T03:32:44Z","receivedAt":"2021-09-11T03:32:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Teach `git repack` a new `--write-midx` option for callers that wish to\npersist a multi-pack index in their repository while repacking.\n\nThere are two existing alternatives to this new flag, but they don't\ncover our particular use-case. These alternatives are:\n\n  - Call 'git multi-pack-index write' after running 'git repack', or\n\n  - Set 'GIT_TEST_MULTI_PACK_INDEX=1' in your environment when running\n    'git repack'.\n\nThe former works, but introduces a gap in bitmap coverage between\nrepacking and writing a new MIDX (since the repack may have deleted a\npack included in the existing MIDX, invalidating it altogether).\n\nSetting the 'GIT_TEST_' environment variable is obviously unsupported.\nIn fact, even if it were supported officially, it still wouldn't work,\nbecause it generates the MIDX *after* redundant packs have been dropped,\nleading to the same issue as above.\n\nIntroduce a new option which eliminates this race by teaching `git\nrepack` to generate the MIDX at the critical point: after the new packs\nhave been written and moved into place, but before the redundant packs\nhave been removed.\n\nThis option is compatible with `git repack`'s '--bitmap' option (it\nchanges the interpretation to be: \"write a bitmap corresponding to the\nMIDX after one has been generated\").\n\nThere is a little bit of additional noise in the patch below to avoid\nrepeating ourselves when selecting which packs to delete. Instead of a\nsingle loop as before (where we iterate over 'existing_packs', decide if\na pack is worth deleting, and if so, delete it), we have two loops (the\nfirst where we decide which ones are worth deleting, and the second\nwhere we actually do the deleting). This makes it so we have a single\ncheck we can make consistently when (1) telling the MIDX which packs we\nwant to exclude, and (2) actually unlinking the redundant packs.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-repack.txt |  14 ++--\n builtin/repack.c             | 132 ++++++++++++++++++++++++++++++-----\n t/lib-midx.sh                |   8 +++\n t/t7700-repack.sh            |  96 +++++++++++++++++++++++++\n 4 files changed, 227 insertions(+), 23 deletions(-)\n create mode 100644 t/lib-midx.sh\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 24c00c9384..0f2d235ca5 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -9,7 +9,7 @@ git-repack - Pack unpacked objects in a repository\n SYNOPSIS\n --------\n [verse]\n-'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n+'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m] [--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>] [--write-midx]\n \n DESCRIPTION\n -----------\n@@ -128,10 +128,11 @@ depth is 4095.\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\n-\tonly makes sense when used with `-a` or `-A`, as the bitmaps\n+\tonly makes sense when used with `-a`, `-A` or `-m`, as the bitmaps\n \tmust be able to refer to all reachable objects. This option\n-\toverrides the setting of `repack.writeBitmaps`.  This option\n-\thas no effect if multiple packfiles are created.\n+\toverrides the setting of `repack.writeBitmaps`. This option\n+\thas no effect if multiple packfiles are created, unless writing a\n+\tMIDX (in which case a multi-pack bitmap is created).\n \n --pack-kept-objects::\n \tInclude objects in `.keep` files when repacking.  Note that we\n@@ -190,6 +191,11 @@ to change in the future. This option (implying a drastically different\n repack mode) is not guaranteed to work with all other combinations of\n option to `git repack`.\n \n+-m::\n+--write-midx::\n+\tWrite a multi-pack index (see linkgit:git-multi-pack-index[1])\n+\tcontaining the non-redundant packs.\n+\n CONFIGURATION\n -------------\n \ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex be470546e6..cb4292ab37 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -433,6 +433,76 @@ static void clear_pack_geometry(struct pack_geometry *geometry)\n \tgeometry->split = 0;\n }\n \n+static void midx_included_packs(struct string_list *include,\n+\t\t\t\tstruct string_list *existing_packs,\n+\t\t\t\tstruct string_list *existing_kept_packs,\n+\t\t\t\tstruct string_list *names,\n+\t\t\t\tstruct pack_geometry *geometry)\n+{\n+\tstruct string_list_item *item;\n+\n+\tfor_each_string_list_item(item, existing_kept_packs) {\n+\t\tstring_list_insert(include, xstrfmt(\"%s.idx\", item->string));\n+\t}\n+\tfor_each_string_list_item(item, names) {\n+\t\tstring_list_insert(include, xstrfmt(\"pack-%s.idx\", item->string));\n+\t}\n+\tif (geometry) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tuint32_t i;\n+\t\tfor (i = geometry->split; i < geometry->pack_nr; i++) {\n+\t\t\tstruct packed_git *p = geometry->pack[i];\n+\n+\t\t\tstrbuf_addstr(&buf, pack_basename(p));\n+\t\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\t\tstring_list_insert(include, strbuf_detach(&buf, NULL));\n+\t\t}\n+\t} else {\n+\t\tfor_each_string_list_item(item, existing_packs) {\n+\t\t\tif (item->util)\n+\t\t\t\tcontinue;\n+\t\t\tstring_list_insert(include, xstrfmt(\"%s.idx\", item->string));\n+\t\t}\n+\t}\n+}\n+\n+static int write_midx_included_packs(struct string_list *include,\n+\t\t\t\t     int show_progress, int write_bitmaps)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct string_list_item *item;\n+\tFILE *in;\n+\tint ret;\n+\n+\tcmd.in = -1;\n+\tcmd.git_cmd = 1;\n+\n+\tstrvec_push(&cmd.args, \"multi-pack-index\");\n+\tstrvec_pushl(&cmd.args, \"write\", \"--stdin-packs\", NULL);\n+\n+\tif (show_progress)\n+\t\tstrvec_push(&cmd.args, \"--progress\");\n+\telse\n+\t\tstrvec_push(&cmd.args, \"--no-progress\");\n+\n+\tif (write_bitmaps)\n+\t\tstrvec_push(&cmd.args, \"--bitmap\");\n+\n+\tret = start_command(&cmd);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tin = xfdopen(cmd.in, \"w\");\n+\tfor_each_string_list_item(item, include) {\n+\t\tfprintf(in, \"%s\\n\", item->string);\n+\t}\n+\tfclose(in);\n+\n+\treturn finish_command(&cmd);\n+}\n+\n int cmd_repack(int argc, const char **argv, const char *prefix)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -456,6 +526,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint no_update_server_info = 0;\n \tstruct pack_objects_args po_args = {NULL};\n \tint geometric_factor = 0;\n+\tint write_midx = 0;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -498,6 +569,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"do not repack this pack\")),\n \t\tOPT_INTEGER('g', \"geometric\", &geometric_factor,\n \t\t\t    N_(\"find a geometric progression with factor <N>\")),\n+\t\tOPT_BOOL('m', \"write-midx\", &write_midx,\n+\t\t\t   N_(\"write a multi-pack index of the resulting packs\")),\n \t\tOPT_END()\n \t};\n \n@@ -514,8 +587,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"--keep-unreachable and -A are incompatible\"));\n \n \tif (write_bitmaps < 0) {\n-\t\tif (!(pack_everything & ALL_INTO_ONE) ||\n-\t\t    !is_bare_repository())\n+\t\tif (!write_midx &&\n+\t\t    (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n \t\t\twrite_bitmaps = 0;\n \t} else if (write_bitmaps &&\n \t\t   git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&\n@@ -525,7 +598,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (pack_kept_objects < 0)\n \t\tpack_kept_objects = write_bitmaps > 0;\n \n-\tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE))\n+\tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)\n \t\tdie(_(incremental_bitmap_conflict_error));\n \n \tif (geometric_factor) {\n@@ -567,10 +640,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t}\n \tif (has_promisor_remote())\n \t\tstrvec_push(&cmd.args, \"--exclude-promisor-objects\");\n-\tif (write_bitmaps > 0)\n-\t\tstrvec_push(&cmd.args, \"--write-bitmap-index\");\n-\telse if (write_bitmaps < 0)\n-\t\tstrvec_push(&cmd.args, \"--write-bitmap-index-quiet\");\n+\tif (!write_midx) {\n+\t\tif (write_bitmaps > 0)\n+\t\t\tstrvec_push(&cmd.args, \"--write-bitmap-index\");\n+\t\telse if (write_bitmaps < 0)\n+\t\t\tstrvec_push(&cmd.args, \"--write-bitmap-index-quiet\");\n+\t}\n \tif (use_delta_islands)\n \t\tstrvec_push(&cmd.args, \"--delta-islands\");\n \n@@ -683,22 +758,41 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t}\n \t/* End of pack replacement. */\n \n+\tif (delete_redundant && pack_everything & ALL_INTO_ONE) {\n+\t\tconst int hexsz = the_hash_algo->hexsz;\n+\t\tstring_list_sort(&names);\n+\t\tfor_each_string_list_item(item, &existing_packs) {\n+\t\t\tchar *sha1;\n+\t\t\tsize_t len = strlen(item->string);\n+\t\t\tif (len < hexsz)\n+\t\t\t\tcontinue;\n+\t\t\tsha1 = item->string + len - hexsz;\n+\t\t\titem->util = (void*)(intptr_t)!string_list_has_string(&names, sha1);\n+\t\t}\n+\t}\n+\n+\tif (write_midx) {\n+\t\tstruct string_list include = STRING_LIST_INIT_NODUP;\n+\t\tmidx_included_packs(&include, &existing_packs,\n+\t\t\t\t    &existing_kept_packs, &names, geometry);\n+\n+\t\tret = write_midx_included_packs(&include,\n+\t\t\t\t\t\tshow_progress, write_bitmaps > 0);\n+\n+\t\tstring_list_clear(&include, 0);\n+\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n \treprepare_packed_git(the_repository);\n \n \tif (delete_redundant) {\n \t\tint opts = 0;\n-\t\tif (pack_everything & ALL_INTO_ONE) {\n-\t\t\tconst int hexsz = the_hash_algo->hexsz;\n-\t\t\tstring_list_sort(&names);\n-\t\t\tfor_each_string_list_item(item, &existing_packs) {\n-\t\t\t\tchar *sha1;\n-\t\t\t\tsize_t len = strlen(item->string);\n-\t\t\t\tif (len < hexsz)\n-\t\t\t\t\tcontinue;\n-\t\t\t\tsha1 = item->string + len - hexsz;\n-\t\t\t\tif (!string_list_has_string(&names, sha1))\n-\t\t\t\t\tremove_redundant_pack(packdir, item->string);\n-\t\t\t}\n+\t\tfor_each_string_list_item(item, &existing_packs) {\n+\t\t\tif (!item->util)\n+\t\t\t\tcontinue;\n+\t\t\tremove_redundant_pack(packdir, item->string);\n \t\t}\n \n \t\tif (geometry) {\ndiff --git a/t/lib-midx.sh b/t/lib-midx.sh\nnew file mode 100644\nindex 0000000000..1261994744\n--- /dev/null\n+++ b/t/lib-midx.sh\n@@ -0,0 +1,8 @@\n+# test_midx_consistent <objdir>\n+test_midx_consistent () {\n+\tls $1/pack/pack-*.idx | xargs -n 1 basename | sort >expect &&\n+\ttest-tool read-midx $1 | grep ^pack-.*\\.idx$ | sort >actual &&\n+\n+\ttest_cmp expect actual &&\n+\tgit multi-pack-index --object-dir=$1 verify\n+}\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 98eda3bfeb..6792531dfd 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -3,6 +3,8 @@\n test_description='git repack works correctly'\n \n . ./test-lib.sh\n+. \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n+. \"${TEST_DIRECTORY}/lib-midx.sh\"\n \n commit_and_pack () {\n \ttest_commit \"$@\" 1>&2 &&\n@@ -234,4 +236,98 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \ttest_must_be_empty actual\n '\n \n+objdir=.git/objects\n+midx=$objdir/pack/multi-pack-index\n+\n+test_expect_success 'setup for --write-midx tests' '\n+\tgit init midx &&\n+\t(\n+\t\tcd midx &&\n+\t\tgit config core.multiPackIndex true &&\n+\n+\t\ttest_commit base\n+\t)\n+'\n+\n+test_expect_success '--write-midx unchanged' '\n+\t(\n+\t\tcd midx &&\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack &&\n+\t\ttest_path_is_missing $midx &&\n+\t\ttest_path_is_missing $midx-*.bitmap &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_missing $midx-*.bitmap &&\n+\t\ttest_midx_consistent $objdir\n+\t)\n+'\n+\n+test_expect_success '--write-midx with a new pack' '\n+\t(\n+\t\tcd midx &&\n+\t\ttest_commit loose &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_missing $midx-*.bitmap &&\n+\t\ttest_midx_consistent $objdir\n+\t)\n+'\n+\n+test_expect_success '--write-midx with -b' '\n+\t(\n+\t\tcd midx &&\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -mb &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-*.bitmap &&\n+\t\ttest_midx_consistent $objdir\n+\t)\n+'\n+\n+test_expect_success '--write-midx with -d' '\n+\t(\n+\t\tcd midx &&\n+\t\ttest_commit repack &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad --write-midx &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_missing $midx-*.bitmap &&\n+\t\ttest_midx_consistent $objdir\n+\t)\n+'\n+\n+test_expect_success 'cleans up MIDX when appropriate' '\n+\t(\n+\t\tcd midx &&\n+\n+\t\ttest_commit repack-2 &&\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&\n+\n+\t\tchecksum=$(midx_checksum $objdir) &&\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$checksum.bitmap &&\n+\t\ttest_path_is_file $midx-$checksum.rev &&\n+\n+\t\ttest_commit repack-3 &&\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_missing $midx-$checksum.bitmap &&\n+\t\ttest_path_is_missing $midx-$checksum.rev &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n+\n+\t\ttest_commit repack-4 &&\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb &&\n+\n+\t\tfind $objdir/pack -type f -name \"multi-pack-index*\" >files &&\n+\t\ttest_must_be_empty files\n+\t)\n+'\n+\n test_done\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435513","messageId":"a790ee5ac6c03d6832599e77f84c352f577d6287.1631331139.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631331139.git.me@ttaylorr.com","subject":"[PATCH 7/8] builtin/repack.c: make largest pack preferred","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-11T03:32:47Z","receivedAt":"2021-09-11T03:32:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When repacking into a geometric series and writing a multi-pack bitmap,\nit is beneficial to have the largest resulting pack be the preferred\nobject source in the bitmap's MIDX, since selecting the large packs can\nlead to fewer broken delta chains and better compression.\n\nTeach 'git repack' to identify this pack and pass it to the MIDX write\nmachinery in order to mark it as preferred.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-repack.txt |  4 ++++\n builtin/repack.c             | 23 ++++++++++++++++++++++-\n pack-bitmap.c                |  2 +-\n pack-bitmap.h                |  1 +\n t/helper/test-read-midx.c    | 25 ++++++++++++++++++++++++-\n t/t7703-repack-geometric.sh  | 23 +++++++++++++++++++++++\n 6 files changed, 75 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 0f2d235ca5..7183fb498f 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -190,6 +190,10 @@ this \"roll-up\", without respect to their reachability. This is subject\n to change in the future. This option (implying a drastically different\n repack mode) is not guaranteed to work with all other combinations of\n option to `git repack`.\n++\n+When writing a multi-pack bitmap, `git repack` selects the largest resulting\n+pack as the preferred pack for object selection by the MIDX (see\n+linkgit:git-multi-pack-index[1]).\n \n -m::\n --write-midx::\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex cb4292ab37..e958caff8b 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -422,6 +422,13 @@ static void split_pack_geometry(struct pack_geometry *geometry, int factor)\n \tgeometry->split = split;\n }\n \n+static struct packed_git *get_largest_active_pack(struct pack_geometry *geometry)\n+{\n+\tif (geometry->split == geometry->pack_nr)\n+\t\treturn NULL;\n+\treturn geometry->pack[geometry->pack_nr - 1];\n+}\n+\n static void clear_pack_geometry(struct pack_geometry *geometry)\n {\n \tif (!geometry)\n@@ -469,6 +476,7 @@ static void midx_included_packs(struct string_list *include,\n }\n \n static int write_midx_included_packs(struct string_list *include,\n+\t\t\t\t     struct pack_geometry *geometry,\n \t\t\t\t     int show_progress, int write_bitmaps)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -490,6 +498,19 @@ static int write_midx_included_packs(struct string_list *include,\n \tif (write_bitmaps)\n \t\tstrvec_push(&cmd.args, \"--bitmap\");\n \n+\tif (geometry) {\n+\t\tstruct packed_git *largest = get_largest_active_pack(geometry);\n+\t\tif (largest)\n+\t\t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n+\t\t\t\t     pack_basename(largest));\n+\t\telse\n+\t\t\t/*\n+\t\t\t * The largest pack was repacked, meaning only one pack\n+\t\t\t * exists (and tautologically, it is the largest).\n+\t\t\t */\n+\t\t\t;\n+\t}\n+\n \tret = start_command(&cmd);\n \tif (ret)\n \t\treturn ret;\n@@ -776,7 +797,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tmidx_included_packs(&include, &existing_packs,\n \t\t\t\t    &existing_kept_packs, &names, geometry);\n \n-\t\tret = write_midx_included_packs(&include,\n+\t\tret = write_midx_included_packs(&include, geometry,\n \t\t\t\t\t\tshow_progress, write_bitmaps > 0);\n \n \t\tstring_list_clear(&include, 0);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 8504110a4d..67be9be9a6 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1418,7 +1418,7 @@ static int try_partial_reuse(struct packed_git *pack,\n \treturn 0;\n }\n \n-static uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n+uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n {\n \tstruct multi_pack_index *m = bitmap_git->midx;\n \tif (!m)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 469090bad2..7d407c5a4c 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -55,6 +55,7 @@ int test_bitmap_commits(struct repository *r);\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\t\t\t struct list_objects_filter_options *filter,\n \t\t\t\t\t int filter_provided_objects);\n+uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git);\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *,\n \t\t\t\t       struct packed_git **packfile,\n \t\t\t\t       uint32_t *entries,\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex cb0d27049a..0038559129 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -3,6 +3,7 @@\n #include \"midx.h\"\n #include \"repository.h\"\n #include \"object-store.h\"\n+#include \"pack-bitmap.h\"\n \n static int read_midx_file(const char *object_dir, int show_objects)\n {\n@@ -72,14 +73,36 @@ static int read_midx_checksum(const char *object_dir)\n \treturn 0;\n }\n \n+static int read_midx_preferred_pack(const char *object_dir)\n+{\n+\tstruct multi_pack_index *midx = NULL;\n+\tstruct bitmap_index *bitmap = NULL;\n+\n+\tsetup_git_directory();\n+\n+\tmidx = load_multi_pack_index(object_dir, 1);\n+\tif (!midx)\n+\t\treturn 1;\n+\n+\tbitmap = prepare_bitmap_git(the_repository);\n+\tif (!(bitmap && bitmap_is_midx(bitmap)))\n+\t\treturn 1;\n+\n+\n+\tprintf(\"%s\\n\", midx->pack_names[midx_preferred_pack(bitmap)]);\n+\treturn 0;\n+}\n+\n int cmd__read_midx(int argc, const char **argv)\n {\n \tif (!(argc == 2 || argc == 3))\n-\t\tusage(\"read-midx [--show-objects|--checksum] <object-dir>\");\n+\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack] <object-dir>\");\n \n \tif (!strcmp(argv[1], \"--show-objects\"))\n \t\treturn read_midx_file(argv[2], 1);\n \telse if (!strcmp(argv[1], \"--checksum\"))\n \t\treturn read_midx_checksum(argv[2]);\n+\telse if (!strcmp(argv[1], \"--preferred-pack\"))\n+\t\treturn read_midx_preferred_pack(argv[2]);\n \treturn read_midx_file(argv[1], 0);\n }\ndiff --git a/t/t7703-repack-geometric.sh b/t/t7703-repack-geometric.sh\nindex 5ccaa440e0..7bdeffa111 100755\n--- a/t/t7703-repack-geometric.sh\n+++ b/t/t7703-repack-geometric.sh\n@@ -180,4 +180,27 @@ test_expect_success '--geometric ignores kept packs' '\n \t)\n '\n \n+test_expect_success '--geometric chooses largest MIDX preferred pack' '\n+\tgit init geometric &&\n+\ttest_when_finished \"rm -fr geometric\" &&\n+\t(\n+\t\tcd geometric &&\n+\t\tgit config core.multiPackIndex true &&\n+\n+\t\t# These packs already form a geometric progression.\n+\t\ttest_commit_bulk --start=1 1 && # 3 objects\n+\t\ttest_commit_bulk --start=2 2 && # 6 objects\n+\t\tls $objdir/pack/pack-*.idx >before &&\n+\t\ttest_commit_bulk --start=4 4 && # 12 objects\n+\t\tls $objdir/pack/pack-*.idx >after &&\n+\n+\t\tgit repack --geometric 2 -dbm &&\n+\n+\t\tcomm -3 before after | xargs -n 1 basename >expect &&\n+\t\ttest-tool read-midx --preferred-pack $objdir >actual &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435514","messageId":"6a1c52181e8c8c9fe2f0e2d7fbeb1057f68c1f3d.1631331139.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631331139.git.me@ttaylorr.com","subject":"[PATCH 8/8] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-11T03:32:49Z","receivedAt":"2021-09-11T03:32:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To prevent the race described in an earlier patch, generate and pass a\nreference snapshot to the multi-pack bitmap code, if we are writing one\nfrom `git repack`.\n\nThis patch is mostly limited to creating a temporary file, and then\ncalling for_each_ref(). Except we try to minimize duplicates, since\ndoing so can drastically reduce the size in network-of-forks style\nrepositories. In the kernel's fork network (the repository containing\nall objects from the kernel and all its forks), deduplicating the\nreferences drops the snapshot size from 934 MB to just 12 MB.\n\nBut since we're handling duplicates in this way, we have to make sure\nthat we preferred references (those listed in pack.preferBitmapTips)\nbefore non-preferred ones (to avoid recording an object which is pointed\nat by a preferred tip as non-preferred).\n\nWe accomplish this by doing separate passes over the references: first\nvisiting each prefix in pack.preferBitmapTips, and then over the rest of\nthe references.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/repack.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 76 insertions(+)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex e958caff8b..3fda837cb5 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -15,6 +15,8 @@\n #include \"promisor-remote.h\"\n #include \"shallow.h\"\n #include \"pack.h\"\n+#include \"pack-bitmap.h\"\n+#include \"refs.h\"\n \n static int delta_base_offset = 1;\n static int pack_kept_objects = -1;\n@@ -440,6 +442,61 @@ static void clear_pack_geometry(struct pack_geometry *geometry)\n \tgeometry->split = 0;\n }\n \n+struct midx_snapshot_ref_data {\n+\tstruct tempfile *f;\n+\tstruct oidset seen;\n+\tint preferred;\n+};\n+\n+static int midx_snapshot_ref_one(const char *refname,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t int flag, void *_data)\n+{\n+\tstruct midx_snapshot_ref_data *data = _data;\n+\tstruct object_id peeled;\n+\n+\tif (!peel_iterated_oid(oid, &peeled))\n+\t\toid = &peeled;\n+\n+\tif (oidset_insert(&data->seen, oid))\n+\t\treturn 0; /* already seen */\n+\n+\tif (oid_object_info(the_repository, oid, NULL) != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tfprintf(data->f->fp, \"%s%s\\n\", data->preferred ? \"+\" : \"\",\n+\t\toid_to_hex(oid));\n+\n+\treturn 0;\n+}\n+\n+static int midx_snapshot_refs(struct tempfile *f)\n+{\n+\tstruct midx_snapshot_ref_data data;\n+\tconst struct string_list *preferred = bitmap_preferred_tips(the_repository);\n+\n+\tdata.f = f;\n+\toidset_init(&data.seen, 0);\n+\n+\tif (!fdopen_tempfile(f, \"w\"))\n+\t\t die(_(\"could not open tempfile %s for writing\"),\n+\t\t     get_tempfile_path(f));\n+\n+\tif (preferred) {\n+\t\tstruct string_list_item *item;\n+\n+\t\tdata.preferred = 1;\n+\t\tfor_each_string_list_item(item, preferred) {\n+\t\t\tfor_each_ref_in(item->string, midx_snapshot_ref_one, &data);\n+\t\t}\n+\t\tdata.preferred = 0;\n+\t}\n+\n+\tfor_each_ref(midx_snapshot_ref_one, &data);\n+\n+\treturn close_tempfile_gently(f);\n+}\n+\n static void midx_included_packs(struct string_list *include,\n \t\t\t\tstruct string_list *existing_packs,\n \t\t\t\tstruct string_list *existing_kept_packs,\n@@ -477,6 +534,7 @@ static void midx_included_packs(struct string_list *include,\n \n static int write_midx_included_packs(struct string_list *include,\n \t\t\t\t     struct pack_geometry *geometry,\n+\t\t\t\t     const char *refs_snapshot,\n \t\t\t\t     int show_progress, int write_bitmaps)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -511,6 +569,9 @@ static int write_midx_included_packs(struct string_list *include,\n \t\t\t;\n \t}\n \n+\tif (refs_snapshot)\n+\t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\", refs_snapshot);\n+\n \tret = start_command(&cmd);\n \tif (ret)\n \t\treturn ret;\n@@ -534,6 +595,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n \tstruct pack_geometry *geometry = NULL;\n \tstruct strbuf line = STRBUF_INIT;\n+\tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n \tFILE *out;\n \tint show_progress = isatty(2);\n@@ -622,6 +684,19 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)\n \t\tdie(_(incremental_bitmap_conflict_error));\n \n+\tif (write_midx && write_bitmaps) {\n+\t\tstruct strbuf path = STRBUF_INIT;\n+\n+\t\tstrbuf_addf(&path, \"%s/%s_XXXXXX\", get_object_directory(),\n+\t\t\t    \"bitmap-ref-tips\");\n+\n+\t\trefs_snapshot = xmks_tempfile(path.buf);\n+\t\tif (midx_snapshot_refs(refs_snapshot) < 0)\n+\t\t\tdie(_(\"could not take a snapshot of references\"));\n+\n+\t\tstrbuf_release(&path);\n+\t}\n+\n \tif (geometric_factor) {\n \t\tif (pack_everything)\n \t\t\tdie(_(\"--geometric is incompatible with -A, -a\"));\n@@ -798,6 +873,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\t    &existing_kept_packs, &names, geometry);\n \n \t\tret = write_midx_included_packs(&include, geometry,\n+\t\t\t\t\t\trefs_snapshot ? get_tempfile_path(refs_snapshot) : NULL,\n \t\t\t\t\t\tshow_progress, write_bitmaps > 0);\n \n \t\tstring_list_clear(&include, 0);\n-- \n2.33.0.96.g73915697e6\n"},{"id":"435516","messageId":"xmqqsfybn1ud.fsf@gitster.g","threadId":"56482","inReplyTo":"4afa03b972a1885c60fbf3716f22a7ab58056383.1631331139.git.me@ttaylorr.com","subject":"Re: [PATCH 1/8] midx: expose 'write_midx_file_only()' publicly","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-11T05:00:26Z","receivedAt":"2021-09-11T05:00:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n>  \tif (ends_with(file_name, \".idx\")) {\n>  \t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n> -\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n> -\t\t\treturn;\n> +\t\tif (ctx->m) {\n> +\t\t\tif (midx_contains_pack(ctx->m, file_name))\n> +\t\t\t\treturn;\n> +\t\t} else if (ctx->to_include) {\n> +\t\t\tif (!string_list_has_string(ctx->to_include, file_name))\n> +\t\t\t\treturn;\n\nWhat's the expected number of elements on the to_include list?  I am\nwondering about the performance implications of using linear search\nover the string-list, of course.  Is it about the same order of the\nnumber of packfiles in a repository (up to several dozens, or 1000\nat most unless you are insane, or something like that)?\n"},{"id":"435538","messageId":"87ee9va0ie.fsf@evledraar.gmail.com","threadId":"56482","inReplyTo":"2a16f11790b79ab452233b6f28acac607c0abd28.1631331139.git.me@ttaylorr.com","subject":"Re: [PATCH 2/8] builtin/multi-pack-index.c: support --stdin-packs mode","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-11T10:05:05Z","receivedAt":"2021-09-11T10:07:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Sep 10 2021, Taylor Blau wrote:\n\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +\twhile (strbuf_getline(&buf, stdin) != EOF) {\n> +\t\tstring_list_append(to, strbuf_detach(&buf, NULL));\n\nIf you strbuf_detach() it...\n\n\n> +\t\tstruct string_list packs = STRING_LIST_INIT_NODUP;\n\n...init the list with NODUP...\n\n> +\t\tstring_list_clear(&packs, 0);\n\n...and call string_list_clear() you'll leak memory, since the\nstring_list will think that someone else is taking care of this memory.\n\nI had some recent musings about how this API is bad, and I've got local\nWIP patches to fix it, but in the meantime you need to:\n\n    packs.strdup_strings = 1;\n\nBefore calling string_list_clear(). I.e. we didn't strdup(), but during\nfree() we pretend that we did, because we did, just not in\nstring_list_append().\n"},{"id":"435539","messageId":"87a6kja0b2.fsf@evledraar.gmail.com","threadId":"56482","inReplyTo":"4afa03b972a1885c60fbf3716f22a7ab58056383.1631331139.git.me@ttaylorr.com","subject":"Re: [PATCH 1/8] midx: expose 'write_midx_file_only()' publicly","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-11T10:07:57Z","receivedAt":"2021-09-11T10:12:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Sep 10 2021, Taylor Blau wrote:\n\n> -\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n> -\t\t\treturn;\n> +\t\tif (ctx->m) {\n> +\t\t\tif (midx_contains_pack(ctx->m, file_name))\n> +\t\t\t\treturn;\n> +\t\t} else if (ctx->to_include) {\n> +\t\t\tif (!string_list_has_string(ctx->to_include, file_name))\n> +\t\t\t\treturn;\n> +\t\t}\n\nI think this is equivalent to:\n\n\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n\t\t\treturn;\n\t\telse if (!ctx->m && string_list_has_string(...))\n\t\t\treturn;\n\nJust a suggestion/tip to reduce the diff size/nested if/if blocks.\n\n\n> +int write_midx_file_only(const char *object_dir, struct string_list *packs_to_include, const char *preferred_pack_name, unsigned flags);\n\nLet's also line-wrap header changes like in the *.c file.\n"},{"id":"435540","messageId":"875yv79zot.fsf@evledraar.gmail.com","threadId":"56482","inReplyTo":"a790ee5ac6c03d6832599e77f84c352f577d6287.1631331139.git.me@ttaylorr.com","subject":"Re: [PATCH 7/8] builtin/repack.c: make largest pack preferred","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-11T10:17:37Z","receivedAt":"2021-09-11T10:28:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Sep 10 2021, Taylor Blau wrote:\n\n> +\tif (geometry) {\n> +\t\tstruct packed_git *largest = get_largest_active_pack(geometry);\n> +\t\tif (largest)\n> +\t\t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n> +\t\t\t\t     pack_basename(largest));\n> +\t\telse\n> +\t\t\t/*\n> +\t\t\t * The largest pack was repacked, meaning only one pack\n> +\t\t\t * exists (and tautologically, it is the largest).\n> +\t\t\t */\n> +\t\t\t;\n> +\t}\n\nProbably one of those cases where an assignment within an \"if\" is the\nbetter of two evils, despite the CodingGuidelines warning against it\n(but not forbidding it). The added get_largest_active_pack() could also\nlearn to punt on a NULL argument, in which case we wouldn't need an\nassignment):\n\n\tstruct packed_git *largest;\n\t[...]\n        /* If ...[...] */\n\tif (geometry && (largest = get_largest_active_pack(geometry)))\n\t\tstrvec_push(...);\n\nI punted on re-phrasing the comment, because the current code invites\nthe reader to start reading this block, then we see that we may do\nstuff, and then the comment applies to what we *didn't* do.\n\n> -\t\tusage(\"read-midx [--show-objects|--checksum] <object-dir>\");\n> +\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack] <object-dir>\");\n\nJust an aside, but I'm surprised to see this use the older\nnon-parse_options() usage API, which we're generally moving away\nfrom. usage_msg_opt() is generally better.\n\n> +\t\tgit config core.multiPackIndex true &&\n\nI remember a similar pattern from another series, does this test pass\nwithout the config being set? I didn't check...\n"},{"id":"435541","messageId":"871r5v9z3i.fsf@evledraar.gmail.com","threadId":"56482","inReplyTo":"6a1c52181e8c8c9fe2f0e2d7fbeb1057f68c1f3d.1631331139.git.me@ttaylorr.com","subject":"Re: [PATCH 8/8] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-11T10:27:39Z","receivedAt":"2021-09-11T10:38:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Sep 10 2021, Taylor Blau wrote:\n\n> +struct midx_snapshot_ref_data {\n> +\tstruct tempfile *f;\n\nStyle/readability: Spare more than one byte for a variable name in a\nstruct, maybe file, or tmpfile?\n\n> +\tfprintf(data->f->fp, \"%s%s\\n\", data->preferred ? \"+\" : \"\",\n> +\t\toid_to_hex(oid));\n\nJust an idea: Maybe the file could be lines of \"+\\tOID\\n\" instead of\n\"+OID\\n\"? Lends itself more naturally to extension, use with the likes\nof string_list_split() etc.\n\n> +\t\tfor_each_string_list_item(item, preferred) {\n> +\t\t\tfor_each_ref_in(item->string, midx_snapshot_ref_one, &data);\n> +\t\t}\n\nCheap style commenst I know, but throughout this series there's an odd\nmixture of sometimes adding braces for one-lines, sometimes not, or in\nthe case of that \"else ;\" I commented on not for something *much* bigger\n:)\n\n> +\t\tif (midx_snapshot_refs(refs_snapshot) < 0)\n> +\t\t\tdie(_(\"could not take a snapshot of references\"));\n\nSeems like this die() should be moved into the midx_snapshot_refs()\nfunction and that function should use die_errno(), not die(), also it\nuses close_tempfile_gently(), shouldn't save the errno on failure there,\nclean up the tempfile, and then die?\n"},{"id":"435548","messageId":"87sfyb8ihg.fsf@evledraar.gmail.com","threadId":"56482","inReplyTo":"871r5v9z3i.fsf@evledraar.gmail.com","subject":"Re: [PATCH 8/8] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-11T11:19:27Z","receivedAt":"2021-09-11T11:22:25Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Sep 11 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Fri, Sep 10 2021, Taylor Blau wrote:\n>\n>> +struct midx_snapshot_ref_data {\n>> +\tstruct tempfile *f;\n>\n> Style/readability: Spare more than one byte for a variable name in a\n> struct, maybe file, or tmpfile?\n>\n>> +\tfprintf(data->f->fp, \"%s%s\\n\", data->preferred ? \"+\" : \"\",\n>> +\t\toid_to_hex(oid));\n>\n> Just an idea: Maybe the file could be lines of \"+\\tOID\\n\" instead of\n> \"+OID\\n\"? Lends itself more naturally to extension, use with the likes\n> of string_list_split() etc.\n\nActually, even better a format like:\n\n    \"OID[\\t+]\\n\"\n\nOr\n\n    \"OID[\\tpreferred=1]\\n\"\n\nThen you can just use the strbuf API to eat the lines, and only parse\nany optional \\t-delimited \"flags\" if the length of the line you get\ndoesn't match the hash size length.\n\nI don't think it matters for speed or whatever, just ease of parsing &\nworking with this new format.\n\nThe \"+\" v.s. \"preferred=1\" suggestion is just to make it future-proof\nfor comma-delimited key-value options, which we already use in the\nprotocol etc., so it's easy to parse, but I don't know if we're ever\nlikely to add a new flag here (and even if we are, we can use the next\n\"\\t\" positional...\n"},{"id":"435561","messageId":"YTzWmhFxn04Jgk1U@nand.local","threadId":"56482","inReplyTo":"xmqqsfybn1ud.fsf@gitster.g","subject":"Re: [PATCH 1/8] midx: expose 'write_midx_file_only()' publicly","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-11T16:17:30Z","receivedAt":"2021-09-11T16:17:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Sep 10, 2021 at 10:00:26PM -0700, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> >  \tif (ends_with(file_name, \".idx\")) {\n> >  \t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n> > -\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n> > -\t\t\treturn;\n> > +\t\tif (ctx->m) {\n> > +\t\t\tif (midx_contains_pack(ctx->m, file_name))\n> > +\t\t\t\treturn;\n> > +\t\t} else if (ctx->to_include) {\n> > +\t\t\tif (!string_list_has_string(ctx->to_include, file_name))\n> > +\t\t\t\treturn;\n>\n> What's the expected number of elements on the to_include list?  I am\n> wondering about the performance implications of using linear search\n> over the string-list, of course.  Is it about the same order of the\n> number of packfiles in a repository (up to several dozens, or 1000\n> at most unless you are insane, or something like that)?\n\nYou're definitely in the right ballpark. It depends on the repack\nsettings and size of repository, of course, but I imagine that roughly\n1,000 entries would be the most anybody could ever pass (e.g., during a\n`--geometric` repack, the biggest pack would have to contain 2^1000\ntimes as many objects as the smallest pack).\n\nOf course, you could just constantly be adding packs and doing\nincremental `git repack -d --write-midx`. Seems unlikely to me, but if\nit does become a problem we could easily read the values into a hashmap\nand constant-ize the lookup.\n\nBut the scan is logarithmic, not linear, since the string list is\nsorted.\n\nThanks,\nTaylor\n"},{"id":"435562","messageId":"YTzXmAOvGp2OiuYZ@nand.local","threadId":"56482","inReplyTo":"87a6kja0b2.fsf@evledraar.gmail.com","subject":"Re: [PATCH 1/8] midx: expose 'write_midx_file_only()' publicly","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-11T16:21:44Z","receivedAt":"2021-09-11T16:21:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Sep 11, 2021 at 12:07:57PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Fri, Sep 10 2021, Taylor Blau wrote:\n>\n> > -\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n> > -\t\t\treturn;\n> > +\t\tif (ctx->m) {\n> > +\t\t\tif (midx_contains_pack(ctx->m, file_name))\n> > +\t\t\t\treturn;\n> > +\t\t} else if (ctx->to_include) {\n> > +\t\t\tif (!string_list_has_string(ctx->to_include, file_name))\n> > +\t\t\t\treturn;\n> > +\t\t}\n>\n> I think this is equivalent to:\n>\n> \t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n> \t\t\treturn;\n> \t\telse if (!ctx->m && string_list_has_string(...))\n> \t\t\treturn;\n\nThey are equivalent, but it took me a minute to convince myself ;). They\nare only equivalent because *either* ctx->m or ctx->to_include is\nnon-NULL.\n\nSo it wouldn't be the case that ctx->m being set and the subsequent\nmidx_contains_pack() call returning zero would cause us to check the\nto_include list, because to_include will be NULL if ctx->m isn't.\n\nAnyway, that makes them equivalent, so I'm happy to clean it up and\ndecrease the nested-ness.\n\n> > +int write_midx_file_only(const char *object_dir, struct string_list *packs_to_include, const char *preferred_pack_name, unsigned flags);\n>\n> Let's also line-wrap header changes like in the *.c file.\n\nSure.\n\nThanks,\nTaylor\n"},{"id":"435563","messageId":"YTzYYfBGqG87O5mN@nand.local","threadId":"56482","inReplyTo":"87ee9va0ie.fsf@evledraar.gmail.com","subject":"Re: [PATCH 2/8] builtin/multi-pack-index.c: support --stdin-packs mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-11T16:25:05Z","receivedAt":"2021-09-11T16:25:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Sep 11, 2021 at 12:05:05PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Fri, Sep 10 2021, Taylor Blau wrote:\n>\n> > +\tstruct strbuf buf = STRBUF_INIT;\n> > +\twhile (strbuf_getline(&buf, stdin) != EOF) {\n> > +\t\tstring_list_append(to, strbuf_detach(&buf, NULL));\n>\n> If you strbuf_detach() it...\n>\n>\n> > +\t\tstruct string_list packs = STRING_LIST_INIT_NODUP;\n>\n> ...init the list with NODUP...\n>\n> > +\t\tstring_list_clear(&packs, 0);\n>\n> ...and call string_list_clear() you'll leak memory, since the\n> string_list will think that someone else is taking care of this memory.\n>\n> I had some recent musings about how this API is bad, and I've got local\n> WIP patches to fix it, but in the meantime you need to:\n>\n>     packs.strdup_strings = 1;\n>\n> Before calling string_list_clear(). I.e. we didn't strdup(), but during\n> free() we pretend that we did, because we did, just not in\n> string_list_append().\n\nGood catch. It's kind of gross, but the result is:\n\n--- 8< ---\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 77488b6b7b..e6cab975e3 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -64,7 +64,7 @@ static struct option *add_common_options(struct option *prev)\n\n static void read_packs_from_stdin(struct string_list *to)\n {\n-\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct strbuf buf = STRBUF_INIT_NODUP;\n \twhile (strbuf_getline(&buf, stdin) != EOF) {\n \t\tstring_list_append(to, strbuf_detach(&buf, NULL));\n \t}\n@@ -107,6 +107,11 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\tret = write_midx_file_only(opts.object_dir, &packs,\n \t\t\t\t\t   opts.preferred_pack, opts.flags);\n\n+\t\t/*\n+\t\t * pretend strings are duplicated to free the memory allocated\n+\t\t * by read_packs_from_stdin()\n+\t\t */\n+\t\tpacks.strdup_strings = 1;\n \t\tstring_list_clear(&packs, 0);\n\n \t\treturn ret;\n"},{"id":"435564","messageId":"YTzZPti/asQwZC/D@nand.local","threadId":"56482","inReplyTo":"YTzYYfBGqG87O5mN@nand.local","subject":"Re: [PATCH 2/8] builtin/multi-pack-index.c: support --stdin-packs mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-11T16:28:46Z","receivedAt":"2021-09-11T16:28:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Sep 11, 2021 at 12:25:05PM -0400, Taylor Blau wrote:\n> Good catch. It's kind of gross, but the result is:\n>\n> --- 8< ---\n>\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> index 77488b6b7b..e6cab975e3 100644\n> --- a/builtin/multi-pack-index.c\n> +++ b/builtin/multi-pack-index.c\n> @@ -64,7 +64,7 @@ static struct option *add_common_options(struct option *prev)\n>\n>  static void read_packs_from_stdin(struct string_list *to)\n>  {\n> -\tstruct strbuf buf = STRBUF_INIT;\n> +\tstruct strbuf buf = STRBUF_INIT_NODUP;\n\nThanks,\nTaylor\n"},{"id":"435565","messageId":"YTza5MVTcBgOFXRS@nand.local","threadId":"56482","inReplyTo":"875yv79zot.fsf@evledraar.gmail.com","subject":"Re: [PATCH 7/8] builtin/repack.c: make largest pack preferred","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-11T16:35:48Z","receivedAt":"2021-09-11T16:35:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Sep 11, 2021 at 12:17:37PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Fri, Sep 10 2021, Taylor Blau wrote:\n>\n> > +\tif (geometry) {\n> > +\t\tstruct packed_git *largest = get_largest_active_pack(geometry);\n> > +\t\tif (largest)\n> > +\t\t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n> > +\t\t\t\t     pack_basename(largest));\n> > +\t\telse\n> > +\t\t\t/*\n> > +\t\t\t * The largest pack was repacked, meaning only one pack\n> > +\t\t\t * exists (and tautologically, it is the largest).\n> > +\t\t\t */\n> > +\t\t\t;\n> > +\t}\n>\n> Probably one of those cases where an assignment within an \"if\" is the\n> better of two evils, despite the CodingGuidelines warning against it\n> (but not forbidding it). The added get_largest_active_pack() could also\n> learn to punt on a NULL argument, in which case we wouldn't need an\n> assignment):\n>\n> \tstruct packed_git *largest;\n> \t[...]\n>         /* If ...[...] */\n> \tif (geometry && (largest = get_largest_active_pack(geometry)))\n> \t\tstrvec_push(...);\n>\n> I punted on re-phrasing the comment, because the current code invites\n> the reader to start reading this block, then we see that we may do\n> stuff, and then the comment applies to what we *didn't* do.\n\nOf the two, I vastly prefer the form that doesn't require assignment\ninside the conditional expression.\n\nThat comment could use an extra bit about how we don't handle\nincremental repacks (since we don't bother looking at the size after\nwords in the case of `git repack -d` (which is neither all-into-one nor\ngeometric). Fixed both, thanks.\n\n> > -\t\tusage(\"read-midx [--show-objects|--checksum] <object-dir>\");\n> > +\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack] <object-dir>\");\n>\n> Just an aside, but I'm surprised to see this use the older\n> non-parse_options() usage API, which we're generally moving away\n> from. usage_msg_opt() is generally better.\n\nOpportunity for future cleanup. I didn't see any point to change it in\nthe middle of an otherwise unrelated series.\n\n> > +\t\tgit config core.multiPackIndex true &&\n>\n> I remember a similar pattern from another series, does this test pass\n> without the config being set? I didn't check...\n\nIt does! This test was written long before 18e449f86b (midx: enable\ncore.multiPackIndex by default, 2020-09-25), but since 18e449f86b that\nline is no longer necessary. Thanks for noticing!\n\nThanks,\nTaylor\n"},{"id":"435566","messageId":"YTzeCD4AKcgTnkm2@nand.local","threadId":"56482","inReplyTo":"871r5v9z3i.fsf@evledraar.gmail.com","subject":"Re: [PATCH 8/8] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-11T16:49:12Z","receivedAt":"2021-09-11T16:49:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Sep 11, 2021 at 12:27:39PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Fri, Sep 10 2021, Taylor Blau wrote:\n>\n> > +struct midx_snapshot_ref_data {\n> > +\tstruct tempfile *f;\n>\n> Style/readability: Spare more than one byte for a variable name in a\n> struct, maybe file, or tmpfile?\n\nSure.\n\n> > +\tfprintf(data->f->fp, \"%s%s\\n\", data->preferred ? \"+\" : \"\",\n> > +\t\toid_to_hex(oid));\n>\n> Just an idea: Maybe the file could be lines of \"+\\tOID\\n\" instead of\n> \"+OID\\n\"? Lends itself more naturally to extension, use with the likes\n> of string_list_split() etc.\n>\n> > +\t\tfor_each_string_list_item(item, preferred) {\n> > +\t\t\tfor_each_ref_in(item->string, midx_snapshot_ref_one, &data);\n> > +\t\t}\n>\n> Cheap style commenst I know, but throughout this series there's an odd\n> mixture of sometimes adding braces for one-lines, sometimes not, or in\n> the case of that \"else ;\" I commented on not for something *much* bigger\n> :)\n\nI've always been uncomfortable with brace-less for loops when the loop\nis hidden behind a macro. I haven't bothered to check, but my gut\nfeeling is that it is pretty rare to see a for_each_something() macro\nwith a one-line body that doesn't have braces.\n\nBut my gut is wrong, because\n\n    git grep 'for_each_.*($' **/*.c | wc -l\n\nturns up 99 results! So I'm happy to be more consistent with the other\ncode (despite my general discomfort with it). An observation is that\neven this file isn't consistent about braceless macro'd for loops, even\nbefore this series.\n\nSo even cleaning it up will still leave this file in an inconsistent\nstate, but I don't think it's worth spending much time at all thinking\nabout cleaning it up.\n\n>\n> > +\t\tif (midx_snapshot_refs(refs_snapshot) < 0)\n> > +\t\t\tdie(_(\"could not take a snapshot of references\"));\n>\n> Seems like this die() should be moved into the midx_snapshot_refs()\n> function and that function should use die_errno(), not die(), also it\n> uses close_tempfile_gently(), shouldn't save the errno on failure there,\n> clean up the tempfile, and then die?\n\nYeah, makes sense.\n\nThanks,\nTaylor\n"},{"id":"435567","messageId":"YTzejKyYvqZu4v16@nand.local","threadId":"56482","inReplyTo":"87sfyb8ihg.fsf@evledraar.gmail.com","subject":"Re: [PATCH 8/8] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-11T16:51:24Z","receivedAt":"2021-09-11T16:51:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Sep 11, 2021 at 01:19:27PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Sat, Sep 11 2021, Ævar Arnfjörð Bjarmason wrote:\n>\n> > On Fri, Sep 10 2021, Taylor Blau wrote:\n> >\n> >> +struct midx_snapshot_ref_data {\n> >> +\tstruct tempfile *f;\n> >\n> > Style/readability: Spare more than one byte for a variable name in a\n> > struct, maybe file, or tmpfile?\n> >\n> >> +\tfprintf(data->f->fp, \"%s%s\\n\", data->preferred ? \"+\" : \"\",\n> >> +\t\toid_to_hex(oid));\n> >\n> > Just an idea: Maybe the file could be lines of \"+\\tOID\\n\" instead of\n> > \"+OID\\n\"? Lends itself more naturally to extension, use with the likes\n> > of string_list_split() etc.\n>\n> Actually, even better a format like:\n>\n>     \"OID[\\t+]\\n\"\n>\n> Or\n>\n>     \"OID[\\tpreferred=1]\\n\"\n\nSure, but I admit that I'm a little torn on this suggestion. I don't\nwant to be naive and say that we're never going to change this format\nand paint ourselves into a corner.\n\nOn the other hand, changing it does seem extremely unlikely to me, and\nthis tab-delimited thing feels like overkill compared to how simple the\n'+' format is.\n\nSo, I don't know. It's certainly easy enough to change now before we\nlock it in, so I guess we should.\n\nThanks,\nTaylor\n"},{"id":"435618","messageId":"CAPig+cRAKJyzuMwPNdmC+BobrMjk5TF-o3Ort8R0E77ij=g6_A@mail.gmail.com","threadId":"56482","inReplyTo":"YTzYYfBGqG87O5mN@nand.local","subject":"Re: [PATCH 2/8] builtin/multi-pack-index.c: support --stdin-packs mode","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-09-12T02:08:44Z","receivedAt":"2021-09-12T02:14:58Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sat, Sep 11, 2021 at 12:25 PM Taylor Blau <me@ttaylorr.com> wrote:\n> On Sat, Sep 11, 2021 at 12:05:05PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> > Before calling string_list_clear(). I.e. we didn't strdup(), but during\n> > free() we pretend that we did, because we did, just not in\n> > string_list_append().\n>\n> Good catch. It's kind of gross, but the result is:\n>\n>  static void read_packs_from_stdin(struct string_list *to)\n>  {\n> -       struct strbuf buf = STRBUF_INIT;\n> +       struct strbuf buf = STRBUF_INIT_NODUP;\n>         while (strbuf_getline(&buf, stdin) != EOF) {\n>                 string_list_append(to, strbuf_detach(&buf, NULL));\n>         }\n> @@ -107,6 +107,11 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n>                 ret = write_midx_file_only(opts.object_dir, &packs,\n>                                            opts.preferred_pack, opts.flags);\n>\n> +               /*\n> +                * pretend strings are duplicated to free the memory allocated\n> +                * by read_packs_from_stdin()\n> +                */\n> +               packs.strdup_strings = 1;\n>                 string_list_clear(&packs, 0);\n\nAn alternative is to do this:\n\n    struct strbuf buf = STRBUF_INIT;\n    ...\n    while (...) {\n        string_list_append_nodup(to, strbuf_detach(&buf, NULL));\n        ...\n    }\n    ...\n    string_list_clear(&packs, 0);\n\nThat is, use string_list_append_nodup(), which is documented as\nexisting precisely for this sort of use-case:\n\n    Like string_list_append(), except string is never copied.  When\n    list->strdup_strings is set, this function can be used to hand\n    ownership of a malloc()ed string to list without making an extra\n    copy.\n\n(I mention this only for completeness but don't care strongly which\napproach is used.)\n"},{"id":"435619","messageId":"YT1kDVBHtPhxc6Wk@nand.local","threadId":"56482","inReplyTo":"CAPig+cRAKJyzuMwPNdmC+BobrMjk5TF-o3Ort8R0E77ij=g6_A@mail.gmail.com","subject":"Re: [PATCH 2/8] builtin/multi-pack-index.c: support --stdin-packs mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-12T02:21:01Z","receivedAt":"2021-09-12T02:21:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Sep 11, 2021 at 10:08:44PM -0400, Eric Sunshine wrote:\n> On Sat, Sep 11, 2021 at 12:25 PM Taylor Blau <me@ttaylorr.com> wrote:\n> > On Sat, Sep 11, 2021 at 12:05:05PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> > > Before calling string_list_clear(). I.e. we didn't strdup(), but during\n> > > free() we pretend that we did, because we did, just not in\n> > > string_list_append().\n> >\n> > Good catch. It's kind of gross, but the result is:\n> >\n> >  static void read_packs_from_stdin(struct string_list *to)\n> >  {\n> > -       struct strbuf buf = STRBUF_INIT;\n> > +       struct strbuf buf = STRBUF_INIT_NODUP;\n> >         while (strbuf_getline(&buf, stdin) != EOF) {\n> >                 string_list_append(to, strbuf_detach(&buf, NULL));\n> >         }\n> > @@ -107,6 +107,11 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n> >                 ret = write_midx_file_only(opts.object_dir, &packs,\n> >                                            opts.preferred_pack, opts.flags);\n> >\n> > +               /*\n> > +                * pretend strings are duplicated to free the memory allocated\n> > +                * by read_packs_from_stdin()\n> > +                */\n> > +               packs.strdup_strings = 1;\n> >                 string_list_clear(&packs, 0);\n>\n> An alternative is to do this:\n>\n>     struct strbuf buf = STRBUF_INIT;\n>     ...\n>     while (...) {\n>         string_list_append_nodup(to, strbuf_detach(&buf, NULL));\n>         ...\n>     }\n>     ...\n>     string_list_clear(&packs, 0);\n>\n> That is, use string_list_append_nodup(), which is documented as\n> existing precisely for this sort of use-case:\n>\n>     Like string_list_append(), except string is never copied.  When\n>     list->strdup_strings is set, this function can be used to hand\n>     ownership of a malloc()ed string to list without making an extra\n>     copy.\n>\n> (I mention this only for completeness but don't care strongly which\n> approach is used.)\n\nThanks for a great suggestion; I do prefer using the existing APIs where\npossible, and it seems like string_list_append_nodup() was designed\nexactly for this case.\n\nThanks,\nTaylor\n"},{"id":"435648","messageId":"87bl4x94ux.fsf@evledraar.gmail.com","threadId":"56482","inReplyTo":"YT1kDVBHtPhxc6Wk@nand.local","subject":"Re: [PATCH 2/8] builtin/multi-pack-index.c: support --stdin-packs mode","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-12T15:15:18Z","receivedAt":"2021-09-12T15:43:39Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Sep 11 2021, Taylor Blau wrote:\n\n> On Sat, Sep 11, 2021 at 10:08:44PM -0400, Eric Sunshine wrote:\n>> On Sat, Sep 11, 2021 at 12:25 PM Taylor Blau <me@ttaylorr.com> wrote:\n>> > On Sat, Sep 11, 2021 at 12:05:05PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> > > Before calling string_list_clear(). I.e. we didn't strdup(), but during\n>> > > free() we pretend that we did, because we did, just not in\n>> > > string_list_append().\n>> >\n>> > Good catch. It's kind of gross, but the result is:\n>> >\n>> >  static void read_packs_from_stdin(struct string_list *to)\n>> >  {\n>> > -       struct strbuf buf = STRBUF_INIT;\n>> > +       struct strbuf buf = STRBUF_INIT_NODUP;\n>> >         while (strbuf_getline(&buf, stdin) != EOF) {\n>> >                 string_list_append(to, strbuf_detach(&buf, NULL));\n>> >         }\n>> > @@ -107,6 +107,11 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n>> >                 ret = write_midx_file_only(opts.object_dir, &packs,\n>> >                                            opts.preferred_pack, opts.flags);\n>> >\n>> > +               /*\n>> > +                * pretend strings are duplicated to free the memory allocated\n>> > +                * by read_packs_from_stdin()\n>> > +                */\n>> > +               packs.strdup_strings = 1;\n>> >                 string_list_clear(&packs, 0);\n>>\n>> An alternative is to do this:\n>>\n>>     struct strbuf buf = STRBUF_INIT;\n>>     ...\n>>     while (...) {\n>>         string_list_append_nodup(to, strbuf_detach(&buf, NULL));\n>>         ...\n>>     }\n>>     ...\n>>     string_list_clear(&packs, 0);\n>>\n>> That is, use string_list_append_nodup(), which is documented as\n>> existing precisely for this sort of use-case:\n>>\n>>     Like string_list_append(), except string is never copied.  When\n>>     list->strdup_strings is set, this function can be used to hand\n>>     ownership of a malloc()ed string to list without making an extra\n>>     copy.\n>>\n>> (I mention this only for completeness but don't care strongly which\n>> approach is used.)\n>\n> Thanks for a great suggestion; I do prefer using the existing APIs where\n> possible, and it seems like string_list_append_nodup() was designed\n> exactly for this case.\n\nI think Eric's suggestion is better in this case, maybe all cases.\n\nFor what it's worth the alternate WIP approach I was hinting at is some\nversion of this:\nhttps://lore.kernel.org/git/87bl6kq631.fsf@evledraar.gmail.com/\n\nI might change my mind, but I think ultimately running with move of the\nstring_list_append_nodup() approach Eric points out is probably the\nright thing to do.\n\nI.e. it's the difference between declaring that a string list is \"dup'd\"\nupfront, and whenever we insert into it we make sure that's the case one\nway or another. Then when we clear we don't need to pass any options.\n\nIt does mean that instead of extra clear functions we need to have\n*_nodup() versions of all the utility functions for this to work,\ne.g. in trying to convert this now there's no\nstring_list_insert_nodup(), which would be needed.\n\nOr maybe the whole approch of the string_list API is just a dead-end in\nAPI design, i.e. it shouldn't have any \"dup\" mode, just nodup, if you\nneed something dup'd you xstrdup()-it.\n\n"},{"id":"435666","messageId":"xmqqlf41juju.fsf@gitster.g","threadId":"56482","inReplyTo":"87bl4x94ux.fsf@evledraar.gmail.com","subject":"Re: [PATCH 2/8] builtin/multi-pack-index.c: support --stdin-packs mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-12T22:30:45Z","receivedAt":"2021-09-12T22:30:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n\n> Or maybe the whole approch of the string_list API is just a dead-end in\n> API design, i.e. it shouldn't have any \"dup\" mode, just nodup, if you\n> need something dup'd you xstrdup()-it.\n\n;-)\n"},{"id":"435667","messageId":"87czpd5spo.fsf@evledraar.gmail.com","threadId":"56482","inReplyTo":"xmqqlf41juju.fsf@gitster.g","subject":"Re: [PATCH 2/8] builtin/multi-pack-index.c: support --stdin-packs mode","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-12T22:32:40Z","receivedAt":"2021-09-12T22:34:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Sep 12 2021, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n>\n>> Or maybe the whole approch of the string_list API is just a dead-end in\n>> API design, i.e. it shouldn't have any \"dup\" mode, just nodup, if you\n>> need something dup'd you xstrdup()-it.\n>\n> ;-)\n\nI realize in practice that's a bit too much, i.e. it's very useful to be\nable to use a string_list to account for a bunch of already-allocated\nmemory without strduping everything.\n\nIt's just unfortunate that you can't look at any code that works with\nthe API and understand whether it does an implicit xstrdup() or not\nwithout tracking down its current state / where it got allocated at.\n"},{"id":"435903","messageId":"YUDwB0W6IpWWcU81@coredump.intra.peff.net","threadId":"56482","inReplyTo":"YTzejKyYvqZu4v16@nand.local","subject":"Re: [PATCH 8/8] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-14T18:55:03Z","receivedAt":"2021-09-14T18:55:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Sep 11, 2021 at 12:51:24PM -0400, Taylor Blau wrote:\n\n> > >> +\tfprintf(data->f->fp, \"%s%s\\n\", data->preferred ? \"+\" : \"\",\n> > >> +\t\toid_to_hex(oid));\n> > >\n> > > Just an idea: Maybe the file could be lines of \"+\\tOID\\n\" instead of\n> > > \"+OID\\n\"? Lends itself more naturally to extension, use with the likes\n> > > of string_list_split() etc.\n> >\n> > Actually, even better a format like:\n> >\n> >     \"OID[\\t+]\\n\"\n> >\n> > Or\n> >\n> >     \"OID[\\tpreferred=1]\\n\"\n> \n> Sure, but I admit that I'm a little torn on this suggestion. I don't\n> want to be naive and say that we're never going to change this format\n> and paint ourselves into a corner.\n> \n> On the other hand, changing it does seem extremely unlikely to me, and\n> this tab-delimited thing feels like overkill compared to how simple the\n> '+' format is.\n> \n> So, I don't know. It's certainly easy enough to change now before we\n> lock it in, so I guess we should.\n\nI'm not sure I really see the point of making this infinitely\nextensible. This is mostly an internal interface between two Git\nprograms. Sure, it's exposed to the user in the sense that they can use\n--refs-snapshot themselves. But if some writer wants to add a \"foo\"\nflag, do they really want to be able to do it in a way that they're\n_syntactically_ compatible with the older versions of Git, yet have no\nclue if their option was actually recognized and accepted?\n\nI.e., if we later add such a flag, then:\n\n  +OID\\tfoo\n  +\\tOID\\tfoo\n  OID\\t+\\tfoo\n\ndoes not seem to make a big difference. We will likely add a new\ncommand-line option to say \"by the way, I am passing you some extra\ninformation\" at which point the format can be whatever we like.\n\nAnd unless it is really complicated, I'd just as soon use a\nsingle-character marker again, because it's dead-simple to generate and\nparse (and again, this is an internal interface and not something we\nexpect users to have to understand).\n\nUnless we think the \"+\" is too ugly or confusing, but I don't. The \"-\"\nprefix for pack-objects read_object_list_from_stdin() is similar and\nworks just fine. For that matter, the --stdin-packs feature there works\nsimilarly. Everything more feels a bit like over-engineering at this\npoint.\n\n-Peff\n"},{"id":"435906","messageId":"YUDxzhAnzI5Anm6F@coredump.intra.peff.net","threadId":"56482","inReplyTo":"YTzYYfBGqG87O5mN@nand.local","subject":"Re: [PATCH 2/8] builtin/multi-pack-index.c: support --stdin-packs mode","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-14T19:02:38Z","receivedAt":"2021-09-14T19:02:41Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Sep 11, 2021 at 12:25:05PM -0400, Taylor Blau wrote:\n\n> > Before calling string_list_clear(). I.e. we didn't strdup(), but during\n> > free() we pretend that we did, because we did, just not in\n> > string_list_append().\n> \n> Good catch. It's kind of gross, but the result is:\n> \n> --- 8< ---\n> \n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> index 77488b6b7b..e6cab975e3 100644\n> --- a/builtin/multi-pack-index.c\n> +++ b/builtin/multi-pack-index.c\n> @@ -64,7 +64,7 @@ static struct option *add_common_options(struct option *prev)\n> \n>  static void read_packs_from_stdin(struct string_list *to)\n>  {\n> -\tstruct strbuf buf = STRBUF_INIT;\n> +\tstruct strbuf buf = STRBUF_INIT_NODUP;\n\nDid you mean to use STRING_LIST_INIT_NODUP on the string-list\ndeclaration?\n\n>  \twhile (strbuf_getline(&buf, stdin) != EOF) {\n>  \t\tstring_list_append(to, strbuf_detach(&buf, NULL));\n>  \t}\n> @@ -107,6 +107,11 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n>  \t\tret = write_midx_file_only(opts.object_dir, &packs,\n>  \t\t\t\t\t   opts.preferred_pack, opts.flags);\n> \n> +\t\t/*\n> +\t\t * pretend strings are duplicated to free the memory allocated\n> +\t\t * by read_packs_from_stdin()\n> +\t\t */\n> +\t\tpacks.strdup_strings = 1;\n>  \t\tstring_list_clear(&packs, 0);\n> \n>  \t\treturn ret;\n\nI think the root of the problem here is the non-idiomatic use of\nstrbuf_getline(). The usual thing (and in fact the thing done by the\nquite-similar code in read_packs_list_from_stdin() in pack-objects.c)\nis not to detach, because strbuf_getline() will reset the buffer each\ntime. I.e.:\n\n  struct string_list to = STRING_LIST_INIT_DUP;\n  ...\n  struct strbuf buf = STRBUF\n  while (strbuf_getline(&buf, stdin) != EOF)\n\tstring_list_append(to, &buf);\n  strbuf_release(&buf);\n\nThat avoids any clever string-list allocation games. The number of heap\nallocations is about the same (one strbuf and N list items, versus N\nstrbufs and 0 list items). There's a little extra copying (from the\nstrbuf into the list items), but the strings themselves are more\nefficiently allocated (strbuf may over-allocate, and we lock in\nthat choice forever by handing over the string).\n\nNot that efficiency probably matters either way for this spot. I'd do it\nthis way because it's simple and idiomatic for our code base.\n\n-Peff\n"},{"id":"435938","messageId":"YUExaCwMwjfd5h7S@nand.local","threadId":"56482","inReplyTo":"YUDwB0W6IpWWcU81@coredump.intra.peff.net","subject":"Re: [PATCH 8/8] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-14T23:34:00Z","receivedAt":"2021-09-14T23:34:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 14, 2021 at 02:55:03PM -0400, Jeff King wrote:\n> On Sat, Sep 11, 2021 at 12:51:24PM -0400, Taylor Blau wrote:\n>\n> > > >> +\tfprintf(data->f->fp, \"%s%s\\n\", data->preferred ? \"+\" : \"\",\n> > > >> +\t\toid_to_hex(oid));\n> > > >\n> > > > Just an idea: Maybe the file could be lines of \"+\\tOID\\n\" instead of\n> > > > \"+OID\\n\"? Lends itself more naturally to extension, use with the likes\n> > > > of string_list_split() etc.\n> > >\n> > > Actually, even better a format like:\n> > >\n> > >     \"OID[\\t+]\\n\"\n> > >\n> > > Or\n> > >\n> > >     \"OID[\\tpreferred=1]\\n\"\n> >\n> > Sure, but I admit that I'm a little torn on this suggestion. I don't\n> > want to be naive and say that we're never going to change this format\n> > and paint ourselves into a corner.\n> >\n> > On the other hand, changing it does seem extremely unlikely to me, and\n> > this tab-delimited thing feels like overkill compared to how simple the\n> > '+' format is.\n> >\n> > So, I don't know. It's certainly easy enough to change now before we\n> > lock it in, so I guess we should.\n>\n> I'm not sure I really see the point of making this infinitely\n> extensible. This is mostly an internal interface between two Git\n> programs. Sure, it's exposed to the user in the sense that they can use\n> --refs-snapshot themselves. But if some writer wants to add a \"foo\"\n> flag, do they really want to be able to do it in a way that they're\n> _syntactically_ compatible with the older versions of Git, yet have no\n> clue if their option was actually recognized and accepted?\n\nI like this perspective, and tend to agree.\n\n(I'm basically repeating what you're saying but) it seems to me that\ntrying to make the interface of `--refs-snapshot` compatible across\nversions of Git is stretching what we consider to be our compatibility\nguarantee given that:\n\n  - the interface is basically expected to be private between `repack`\n    and `multi-pack-index`, and\n\n  - an infinitely-extensible version of `--refs-snapshot` would still\n    have no way to tell the caller whether or not those new flags were\n    accepted\n\nSo I tend to agree the existing format works fine and we shouldn't spend\na ton of time trying to over-engineer a solution.\n\n(FWIW, the `+` vs `-` thing is intentional; `--stdin-packs` uses `-` to\n_exclude_ packs, but `+` here means \"this one is special\" not \"exclude\nthis ref\").\n\nThanks,\nTaylor\n"},{"id":"435939","messageId":"YUE0u+T/CL8D5WV/@nand.local","threadId":"56482","inReplyTo":"YUDxzhAnzI5Anm6F@coredump.intra.peff.net","subject":"Re: [PATCH 2/8] builtin/multi-pack-index.c: support --stdin-packs mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-14T23:48:11Z","receivedAt":"2021-09-14T23:48:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 14, 2021 at 03:02:38PM -0400, Jeff King wrote:\n> On Sat, Sep 11, 2021 at 12:25:05PM -0400, Taylor Blau wrote:\n>\n> > > Before calling string_list_clear(). I.e. we didn't strdup(), but during\n> > > free() we pretend that we did, because we did, just not in\n> > > string_list_append().\n> >\n> > Good catch. It's kind of gross, but the result is:\n> >\n> > --- 8< ---\n> >\n> > diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> > index 77488b6b7b..e6cab975e3 100644\n> > --- a/builtin/multi-pack-index.c\n> > +++ b/builtin/multi-pack-index.c\n> > @@ -64,7 +64,7 @@ static struct option *add_common_options(struct option *prev)\n> >\n> >  static void read_packs_from_stdin(struct string_list *to)\n> >  {\n> > -\tstruct strbuf buf = STRBUF_INIT;\n> > +\tstruct strbuf buf = STRBUF_INIT_NODUP;\n>\n> Did you mean to use STRING_LIST_INIT_NODUP on the string-list\n> declaration?\n\nYeah, and I thought that I even mentioned it in [1], but I apparently\nsent the contents of my buffer before I saved it. So, I did notice it at\nthe time, but failed to tell anybody about it!\n\n[1]: https://lore.kernel.org/git/YTzZPti%2FasQwZC%2FD@nand.local/\n\n> >  \twhile (strbuf_getline(&buf, stdin) != EOF) {\n> >  \t\tstring_list_append(to, strbuf_detach(&buf, NULL));\n> >  \t}\n> > @@ -107,6 +107,11 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n> >  \t\tret = write_midx_file_only(opts.object_dir, &packs,\n> >  \t\t\t\t\t   opts.preferred_pack, opts.flags);\n> >\n> > +\t\t/*\n> > +\t\t * pretend strings are duplicated to free the memory allocated\n> > +\t\t * by read_packs_from_stdin()\n> > +\t\t */\n> > +\t\tpacks.strdup_strings = 1;\n> >  \t\tstring_list_clear(&packs, 0);\n> >\n> >  \t\treturn ret;\n>\n> I think the root of the problem here is the non-idiomatic use of\n> strbuf_getline(). The usual thing (and in fact the thing done by the\n> quite-similar code in read_packs_list_from_stdin() in pack-objects.c)\n> is not to detach, because strbuf_getline() will reset the buffer each\n> time. I.e.:\n>\n>   struct string_list to = STRING_LIST_INIT_DUP;\n>   ...\n>   struct strbuf buf = STRBUF\n>   while (strbuf_getline(&buf, stdin) != EOF)\n> \tstring_list_append(to, &buf);\n\n(Assuming that you meant `s/&buf/buf.buf/`, but otherwise this makes\nsense). There is no need to call strbuf_reset() inside the body of the\nloop, since strbuf_getline() calls it via strbuf_getwholeline().\n\n>   strbuf_release(&buf);\n\n...But we should still be careful to release the memory used by the\nstrbuf at the end (which is safe to do, since the whole point is that we\ncopy each buffer linewise into the string_list).\n\n> That avoids any clever string-list allocation games. The number of heap\n> allocations is about the same (one strbuf and N list items, versus N\n> strbufs and 0 list items). There's a little extra copying (from the\n> strbuf into the list items), but the strings themselves are more\n> efficiently allocated (strbuf may over-allocate, and we lock in\n> that choice forever by handing over the string).\n>\n> Not that efficiency probably matters either way for this spot. I'd do it\n> this way because it's simple and idiomatic for our code base.\n\nMore than anything, I'm persuaded by that (though I was quite partial to\nEric's suggestion, which I found very clever).\n\nThanks,\nTaylor\n"},{"id":"435952","messageId":"87a6keoglz.fsf@evledraar.gmail.com","threadId":"56482","inReplyTo":"YUExaCwMwjfd5h7S@nand.local","subject":"Re: [PATCH 8/8] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-14T23:56:00Z","receivedAt":"2021-09-14T23:57:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Sep 14 2021, Taylor Blau wrote:\n\n> On Tue, Sep 14, 2021 at 02:55:03PM -0400, Jeff King wrote:\n>> On Sat, Sep 11, 2021 at 12:51:24PM -0400, Taylor Blau wrote:\n>>\n>> > > >> +\tfprintf(data->f->fp, \"%s%s\\n\", data->preferred ? \"+\" : \"\",\n>> > > >> +\t\toid_to_hex(oid));\n>> > > >\n>> > > > Just an idea: Maybe the file could be lines of \"+\\tOID\\n\" instead of\n>> > > > \"+OID\\n\"? Lends itself more naturally to extension, use with the likes\n>> > > > of string_list_split() etc.\n>> > >\n>> > > Actually, even better a format like:\n>> > >\n>> > >     \"OID[\\t+]\\n\"\n>> > >\n>> > > Or\n>> > >\n>> > >     \"OID[\\tpreferred=1]\\n\"\n>> >\n>> > Sure, but I admit that I'm a little torn on this suggestion. I don't\n>> > want to be naive and say that we're never going to change this format\n>> > and paint ourselves into a corner.\n>> >\n>> > On the other hand, changing it does seem extremely unlikely to me, and\n>> > this tab-delimited thing feels like overkill compared to how simple the\n>> > '+' format is.\n>> >\n>> > So, I don't know. It's certainly easy enough to change now before we\n>> > lock it in, so I guess we should.\n>>\n>> I'm not sure I really see the point of making this infinitely\n>> extensible. This is mostly an internal interface between two Git\n>> programs. Sure, it's exposed to the user in the sense that they can use\n>> --refs-snapshot themselves. But if some writer wants to add a \"foo\"\n>> flag, do they really want to be able to do it in a way that they're\n>> _syntactically_ compatible with the older versions of Git, yet have no\n>> clue if their option was actually recognized and accepted?\n>\n> I like this perspective, and tend to agree.\n>\n> (I'm basically repeating what you're saying but) it seems to me that\n> trying to make the interface of `--refs-snapshot` compatible across\n> versions of Git is stretching what we consider to be our compatibility\n> guarantee given that:\n>\n>   - the interface is basically expected to be private between `repack`\n>     and `multi-pack-index`, and\n>\n>   - an infinitely-extensible version of `--refs-snapshot` would still\n>     have no way to tell the caller whether or not those new flags were\n>     accepted\n>\n> So I tend to agree the existing format works fine and we shouldn't spend\n> a ton of time trying to over-engineer a solution.\n>\n> (FWIW, the `+` vs `-` thing is intentional; `--stdin-packs` uses `-` to\n> _exclude_ packs, but `+` here means \"this one is special\" not \"exclude\n> this ref\").\n\nThe suggestion I had offhand was just meant as an offhand \"interesting,\nmaybe easier like...\", i.e. if you found it easier to split on \\t, or\ncheck the hash size in the loop or whatever.\n\nI agree that this doesn't matter and should just be left to whatever\nyou've got a taste for, thanks. This thread became more of a digression\nthan I thought...\n"},{"id":"435957","messageId":"CAPig+cTpdPT6kD9NY2HXgPPvceX=euSGhQXpvKC+WS5-dLxe2A@mail.gmail.com","threadId":"56482","inReplyTo":"YUE0u+T/CL8D5WV/@nand.local","subject":"Re: [PATCH 2/8] builtin/multi-pack-index.c: support --stdin-packs mode","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-09-15T01:55:47Z","receivedAt":"2021-09-15T01:56:01Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Sep 14, 2021 at 7:48 PM Taylor Blau <me@ttaylorr.com> wrote:\n> On Tue, Sep 14, 2021 at 03:02:38PM -0400, Jeff King wrote:\n> > I think the root of the problem here is the non-idiomatic use of\n> > strbuf_getline(). The usual thing (and in fact the thing done by the\n> > quite-similar code in read_packs_list_from_stdin() in pack-objects.c)\n> > is not to detach, because strbuf_getline() will reset the buffer each\n> > time. I.e.:\n> >\n> >   struct string_list to = STRING_LIST_INIT_DUP;\n> >   ...\n> >   struct strbuf buf = STRBUF\n> >   while (strbuf_getline(&buf, stdin) != EOF)\n> >       string_list_append(to, &buf);\n>\n> > That avoids any clever string-list allocation games. The number of heap\n> > allocations is about the same (one strbuf and N list items, versus N\n> > strbufs and 0 list items). There's a little extra copying (from the\n> > strbuf into the list items), but the strings themselves are more\n> > efficiently allocated (strbuf may over-allocate, and we lock in\n> > that choice forever by handing over the string).\n> >\n> > Not that efficiency probably matters either way for this spot. I'd do it\n> > this way because it's simple and idiomatic for our code base.\n>\n> More than anything, I'm persuaded by that (though I was quite partial to\n> Eric's suggestion, which I found very clever).\n\nFor what it's worth, Peff's suggestion seems best of all those\npresented thus far.\n"},{"id":"435960","messageId":"YUF3NXcjo2dWtPOR@nand.local","threadId":"56482","inReplyTo":"87a6keoglz.fsf@evledraar.gmail.com","subject":"Re: [PATCH 8/8] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-15T04:31:49Z","receivedAt":"2021-09-15T04:32:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 15, 2021 at 01:56:00AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> The suggestion I had offhand was just meant as an offhand \"interesting,\n> maybe easier like...\", i.e. if you found it easier to split on \\t, or\n> check the hash size in the loop or whatever.\n\n:-). You did say \"Just an idea\", but I treated your suggestion as if you\nwere concerned about being able to extend the format in the future.\n\n> I agree that this doesn't matter and should just be left to whatever\n> you've got a taste for, thanks. This thread became more of a digression\n> than I thought...\n\nI think that all of the discussion was reasonable if I understood your\noriginal opinion more clearly ;). It may help (me, at least) to say more\nexplicitly, \"This may be easier like [...] but I don't have any concerns\nabout functionality\" or similar.\n\nThanks,\nTaylor\n"},{"id":"436013","messageId":"cover.1631730270.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631331139.git.me@ttaylorr.com","subject":"[PATCH v2 0/8] repack: introduce `--write-midx`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-15T18:24:33Z","receivedAt":"2021-09-15T18:24:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is a small reroll of my series to implement the final component of\nmulti-pack reachability bitamps, which is to be able to write them from `git\nrepack`.\n\nNothing substantial has changed since last time. Mostly re-wrapping lines and\nremoving braces in macro'd for_each_xyz() loops. A larger (but still small)\nchange is how we read input to `--refs-snapshot`. This version contains a more\nidiomatic implementation, but see the sub-thread beginning at [1] for a more\ncomplete discussion.\n\nOtherwise, this series is unchagned. It depends on tb/multi-pack-bitmaps, which\nhas graduated to master. Range-diff below:\n\n[1]: https://lore.kernel.org/git/2a16f11790b79ab452233b6f28acac607c0abd28.1631331139.git.me@ttaylorr.com/\n\nTaylor Blau (8):\n  midx: expose `write_midx_file_only()` publicly\n  builtin/multi-pack-index.c: support `--stdin-packs` mode\n  midx: preliminary support for `--refs-snapshot`\n  builtin/repack.c: keep track of existing packs unconditionally\n  builtin/repack.c: extract showing progress to a variable\n  builtin/repack.c: support writing a MIDX while repacking\n  builtin/repack.c: make largest pack preferred\n  builtin/repack.c: pass `--refs-snapshot` when writing bitmaps\n\n Documentation/git-multi-pack-index.txt |  19 ++\n Documentation/git-repack.txt           |  18 +-\n builtin/multi-pack-index.c             |  36 +++-\n builtin/repack.c                       | 255 ++++++++++++++++++++++---\n midx.c                                 |  96 ++++++++--\n midx.h                                 |  11 +-\n pack-bitmap.c                          |   2 +-\n pack-bitmap.h                          |   1 +\n t/helper/test-read-midx.c              |  25 ++-\n t/lib-midx.sh                          |   8 +\n t/t5319-multi-pack-index.sh            |  15 ++\n t/t5326-multi-pack-bitmaps.sh          |  82 ++++++++\n t/t7700-repack.sh                      |  96 ++++++++++\n t/t7703-repack-geometric.sh            |  22 +++\n 14 files changed, 637 insertions(+), 49 deletions(-)\n create mode 100644 t/lib-midx.sh\n\nRange-diff against v1:\n1:  6d50f46080 ! 1:  03c1b2c4d3 midx: expose 'write_midx_file_only()' publicly\n    @@ Metadata\n     Author: Taylor Blau <me@ttaylorr.com>\n     \n      ## Commit message ##\n    -    midx: expose 'write_midx_file_only()' publicly\n    +    midx: expose `write_midx_file_only()` publicly\n     \n         Expose a variant of the write_midx_file() function which ignores packs\n         that aren't included in an explicit \"allow\" list.\n    @@ midx.c: struct write_midx_context {\n      \n      static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n     @@ midx.c: static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n    - \n    - \tif (ends_with(file_name, \".idx\")) {\n      \t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n    --\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n    --\t\t\treturn;\n    -+\t\tif (ctx->m) {\n    -+\t\t\tif (midx_contains_pack(ctx->m, file_name))\n    -+\t\t\t\treturn;\n    -+\t\t} else if (ctx->to_include) {\n    -+\t\t\tif (!string_list_has_string(ctx->to_include, file_name))\n    -+\t\t\t\treturn;\n    -+\t\t}\n    + \t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n    + \t\t\treturn;\n    ++\t\telse if (ctx->to_include &&\n    ++\t\t\t !string_list_has_string(ctx->to_include, file_name))\n    ++\t\t\treturn;\n      \n      \t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n      \n    @@ midx.h: int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pa\n      int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n      \n      int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n    -+int write_midx_file_only(const char *object_dir, struct string_list *packs_to_include, const char *preferred_pack_name, unsigned flags);\n    ++int write_midx_file_only(const char *object_dir,\n    ++\t\t\t struct string_list *packs_to_include,\n    ++\t\t\t const char *preferred_pack_name,\n    ++\t\t\t unsigned flags);\n      void clear_midx_file(struct repository *r);\n      int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);\n      int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);\n2:  66aa7f4b48 ! 2:  59556e5545 builtin/multi-pack-index.c: support --stdin-packs mode\n    @@ Metadata\n     Author: Taylor Blau <me@ttaylorr.com>\n     \n      ## Commit message ##\n    -    builtin/multi-pack-index.c: support --stdin-packs mode\n    +    builtin/multi-pack-index.c: support `--stdin-packs` mode\n     \n         To power a new `--write-midx` mode, `git repack` will want to write a\n         multi-pack index containing a certain set of packs in the repository.\n    @@ builtin/multi-pack-index.c: static struct option *add_common_options(struct opti\n     +static void read_packs_from_stdin(struct string_list *to)\n     +{\n     +\tstruct strbuf buf = STRBUF_INIT;\n    -+\twhile (strbuf_getline(&buf, stdin) != EOF) {\n    -+\t\tstring_list_append(to, strbuf_detach(&buf, NULL));\n    -+\t}\n    ++\twhile (strbuf_getline(&buf, stdin) != EOF)\n    ++\t\tstring_list_append(to, buf.buf);\n     +\tstring_list_sort(to);\n    ++\n    ++\tstrbuf_release(&buf);\n     +}\n     +\n      static int cmd_multi_pack_index_write(int argc, const char **argv)\n    @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, cons\n      \tFREE_AND_NULL(options);\n      \n     +\tif (opts.stdin_packs) {\n    -+\t\tstruct string_list packs = STRING_LIST_INIT_NODUP;\n    ++\t\tstruct string_list packs = STRING_LIST_INIT_DUP;\n     +\t\tint ret;\n     +\n     +\t\tread_packs_from_stdin(&packs);\n3:  f74a967be3 ! 3:  42f1ae9ede midx: preliminary support for `--refs-snapshot`\n    @@ midx.h: int fill_midx_entry(struct repository *r, const struct object_id *oid, s\n      int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n      \n     -int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n    --int write_midx_file_only(const char *object_dir, struct string_list *packs_to_include, const char *preferred_pack_name, unsigned flags);\n    -+int write_midx_file(const char *object_dir, const char *preferred_pack_name, const char *refs_snapshot, unsigned flags);\n    -+int write_midx_file_only(const char *object_dir, struct string_list *packs_to_include, const char *preferred_pack_name, const char *refs_snapshot, unsigned flags);\n    ++int write_midx_file(const char *object_dir,\n    ++\t\t    const char *preferred_pack_name,\n    ++\t\t    const char *refs_snapshot,\n    ++\t\t    unsigned flags);\n    + int write_midx_file_only(const char *object_dir,\n    + \t\t\t struct string_list *packs_to_include,\n    + \t\t\t const char *preferred_pack_name,\n    ++\t\t\t const char *refs_snapshot,\n    + \t\t\t unsigned flags);\n      void clear_midx_file(struct repository *r);\n      int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);\n    - int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);\n     \n      ## t/t5326-multi-pack-bitmaps.sh ##\n     @@ t/t5326-multi-pack-bitmaps.sh: test_expect_success 'pack.preferBitmapTips' '\n4:  5067c2cc65 = 4:  c0d045a9de builtin/repack.c: keep track of existing packs unconditionally\n5:  d5990e3b69 = 5:  09de03de47 builtin/repack.c: extract showing progress to a variable\n6:  d32d800954 ! 6:  83dfdb8b12 builtin/repack.c: support writing a MIDX while repacking\n    @@ builtin/repack.c: static void clear_pack_geometry(struct pack_geometry *geometry\n     +{\n     +\tstruct string_list_item *item;\n     +\n    -+\tfor_each_string_list_item(item, existing_kept_packs) {\n    ++\tfor_each_string_list_item(item, existing_kept_packs)\n     +\t\tstring_list_insert(include, xstrfmt(\"%s.idx\", item->string));\n    -+\t}\n    -+\tfor_each_string_list_item(item, names) {\n    ++\tfor_each_string_list_item(item, names)\n     +\t\tstring_list_insert(include, xstrfmt(\"pack-%s.idx\", item->string));\n    -+\t}\n     +\tif (geometry) {\n     +\t\tstruct strbuf buf = STRBUF_INIT;\n     +\t\tuint32_t i;\n    @@ builtin/repack.c: static void clear_pack_geometry(struct pack_geometry *geometry\n     +\t\treturn ret;\n     +\n     +\tin = xfdopen(cmd.in, \"w\");\n    -+\tfor_each_string_list_item(item, include) {\n    ++\tfor_each_string_list_item(item, include)\n     +\t\tfprintf(in, \"%s\\n\", item->string);\n    -+\t}\n     +\tfclose(in);\n     +\n     +\treturn finish_command(&cmd);\n7:  515e1d95a6 ! 7:  68bc49d8ae builtin/repack.c: make largest pack preferred\n    @@ builtin/repack.c: static void split_pack_geometry(struct pack_geometry *geometry\n      \n     +static struct packed_git *get_largest_active_pack(struct pack_geometry *geometry)\n     +{\n    ++\tif (!geometry) {\n    ++\t\t/*\n    ++\t\t * No geometry means either an all-into-one repack (in which\n    ++\t\t * case there is only one pack left and it is the largest) or an\n    ++\t\t * incremental one.\n    ++\t\t *\n    ++\t\t * If repacking incrementally, then we could check the size of\n    ++\t\t * all packs to determine which should be preferred, but leave\n    ++\t\t * this for later.\n    ++\t\t */\n    ++\t\treturn NULL;\n    ++\t}\n     +\tif (geometry->split == geometry->pack_nr)\n     +\t\treturn NULL;\n     +\treturn geometry->pack[geometry->pack_nr - 1];\n    @@ builtin/repack.c: static void midx_included_packs(struct string_list *include,\n      \t\t\t\t     int show_progress, int write_bitmaps)\n      {\n      \tstruct child_process cmd = CHILD_PROCESS_INIT;\n    + \tstruct string_list_item *item;\n    ++\tstruct packed_git *largest = get_largest_active_pack(geometry);\n    + \tFILE *in;\n    + \tint ret;\n    + \n     @@ builtin/repack.c: static int write_midx_included_packs(struct string_list *include,\n      \tif (write_bitmaps)\n      \t\tstrvec_push(&cmd.args, \"--bitmap\");\n      \n    -+\tif (geometry) {\n    -+\t\tstruct packed_git *largest = get_largest_active_pack(geometry);\n    -+\t\tif (largest)\n    -+\t\t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n    -+\t\t\t\t     pack_basename(largest));\n    -+\t\telse\n    -+\t\t\t/*\n    -+\t\t\t * The largest pack was repacked, meaning only one pack\n    -+\t\t\t * exists (and tautologically, it is the largest).\n    -+\t\t\t */\n    -+\t\t\t;\n    -+\t}\n    ++\tif (largest)\n    ++\t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n    ++\t\t\t     pack_basename(largest));\n     +\n      \tret = start_command(&cmd);\n      \tif (ret)\n    @@ t/t7703-repack-geometric.sh: test_expect_success '--geometric ignores kept packs\n     +\ttest_when_finished \"rm -fr geometric\" &&\n     +\t(\n     +\t\tcd geometric &&\n    -+\t\tgit config core.multiPackIndex true &&\n     +\n     +\t\t# These packs already form a geometric progression.\n     +\t\ttest_commit_bulk --start=1 1 && # 3 objects\n8:  74a9da0ef0 ! 8:  eb24b308ec builtin/repack.c: pass `--refs-snapshot` when writing bitmaps\n    @@ builtin/repack.c: static void clear_pack_geometry(struct pack_geometry *geometry\n     +\treturn 0;\n     +}\n     +\n    -+static int midx_snapshot_refs(struct tempfile *f)\n    ++static void midx_snapshot_refs(struct tempfile *f)\n     +{\n     +\tstruct midx_snapshot_ref_data data;\n     +\tconst struct string_list *preferred = bitmap_preferred_tips(the_repository);\n    @@ builtin/repack.c: static void clear_pack_geometry(struct pack_geometry *geometry\n     +\t\tstruct string_list_item *item;\n     +\n     +\t\tdata.preferred = 1;\n    -+\t\tfor_each_string_list_item(item, preferred) {\n    ++\t\tfor_each_string_list_item(item, preferred)\n     +\t\t\tfor_each_ref_in(item->string, midx_snapshot_ref_one, &data);\n    -+\t\t}\n     +\t\tdata.preferred = 0;\n     +\t}\n     +\n     +\tfor_each_ref(midx_snapshot_ref_one, &data);\n     +\n    -+\treturn close_tempfile_gently(f);\n    ++\tif (close_tempfile_gently(f)) {\n    ++\t\tint save_errno = errno;\n    ++\t\tdelete_tempfile(&f);\n    ++\t\terrno = save_errno;\n    ++\t\tdie_errno(_(\"could not close refs snapshot tempfile\"));\n    ++\t}\n     +}\n     +\n      static void midx_included_packs(struct string_list *include,\n    @@ builtin/repack.c: static void midx_included_packs(struct string_list *include,\n      {\n      \tstruct child_process cmd = CHILD_PROCESS_INIT;\n     @@ builtin/repack.c: static int write_midx_included_packs(struct string_list *include,\n    - \t\t\t;\n    - \t}\n    + \t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n    + \t\t\t     pack_basename(largest));\n      \n     +\tif (refs_snapshot)\n     +\t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\", refs_snapshot);\n    @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n     +\t\t\t    \"bitmap-ref-tips\");\n     +\n     +\t\trefs_snapshot = xmks_tempfile(path.buf);\n    -+\t\tif (midx_snapshot_refs(refs_snapshot) < 0)\n    -+\t\t\tdie(_(\"could not take a snapshot of references\"));\n    ++\t\tmidx_snapshot_refs(refs_snapshot);\n     +\n     +\t\tstrbuf_release(&path);\n     +\t}\n-- \n2.33.0.96.g73915697e6\n"},{"id":"436014","messageId":"03c1b2c4d3d9d10a6768684e9c4d1c1cc8019844.1631730270.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631730270.git.me@ttaylorr.com","subject":"[PATCH v2 1/8] midx: expose `write_midx_file_only()` publicly","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-15T18:24:35Z","receivedAt":"2021-09-15T18:24:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Expose a variant of the write_midx_file() function which ignores packs\nthat aren't included in an explicit \"allow\" list.\n\nThis will be used in an upcoming patch to power a new `--stdin-packs`\nmode of `git multi-pack-index write` for callers that only want to\ninclude certain packs in a MIDX (and ignore any packs which may have\nhappened to enter the repository independently, e.g., from pushes).\n\nThose patches will provide test coverage for this new function.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx.c | 44 +++++++++++++++++++++++++++++++++++---------\n midx.h |  5 +++++\n 2 files changed, 40 insertions(+), 9 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 864034a6ad..0330202fda 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -475,6 +475,8 @@ struct write_midx_context {\n \tuint32_t num_large_offsets;\n \n \tint preferred_pack_idx;\n+\n+\tstruct string_list *to_include;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -486,6 +488,9 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n \t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n \t\t\treturn;\n+\t\telse if (ctx->to_include &&\n+\t\t\t !string_list_has_string(ctx->to_include, file_name))\n+\t\t\treturn;\n \n \t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n \n@@ -1058,6 +1063,7 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n }\n \n static int write_midx_internal(const char *object_dir,\n+\t\t\t       struct string_list *packs_to_include,\n \t\t\t       struct string_list *packs_to_drop,\n \t\t\t       const char *preferred_pack_name,\n \t\t\t       unsigned flags)\n@@ -1082,10 +1088,17 @@ static int write_midx_internal(const char *object_dir,\n \t\tdie_errno(_(\"unable to create leading directories of %s\"),\n \t\t\t  midx_name);\n \n-\tfor (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {\n-\t\tif (!strcmp(object_dir, cur->object_dir)) {\n-\t\t\tctx.m = cur;\n-\t\t\tbreak;\n+\tif (!packs_to_include) {\n+\t\t/*\n+\t\t * Only reference an existing MIDX when not filtering which\n+\t\t * packs to include, since all packs and objects are copied\n+\t\t * blindly from an existing MIDX if one is present.\n+\t\t */\n+\t\tfor (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {\n+\t\t\tif (!strcmp(object_dir, cur->object_dir)) {\n+\t\t\t\tctx.m = cur;\n+\t\t\t\tbreak;\n+\t\t\t}\n \t\t}\n \t}\n \n@@ -1136,10 +1149,13 @@ static int write_midx_internal(const char *object_dir,\n \telse\n \t\tctx.progress = NULL;\n \n+\tctx.to_include = packs_to_include;\n+\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n-\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop) {\n+\tif ((ctx.m && ctx.nr == ctx.m->num_packs) &&\n+\t    !(packs_to_include || packs_to_drop)) {\n \t\tstruct bitmap_index *bitmap_git;\n \t\tint bitmap_exists;\n \t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n@@ -1237,7 +1253,7 @@ static int write_midx_internal(const char *object_dir,\n \n \tQSORT(ctx.info, ctx.nr, pack_info_compare);\n \n-\tif (packs_to_drop && packs_to_drop->nr) {\n+\tif (ctx.m && packs_to_drop && packs_to_drop->nr) {\n \t\tint drop_index = 0;\n \t\tint missing_drops = 0;\n \n@@ -1380,7 +1396,17 @@ int write_midx_file(const char *object_dir,\n \t\t    const char *preferred_pack_name,\n \t\t    unsigned flags)\n {\n-\treturn write_midx_internal(object_dir, NULL, preferred_pack_name, flags);\n+\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n+\t\t\t\t   flags);\n+}\n+\n+int write_midx_file_only(const char *object_dir,\n+\t\t\t struct string_list *packs_to_include,\n+\t\t\t const char *preferred_pack_name,\n+\t\t\t unsigned flags)\n+{\n+\treturn write_midx_internal(object_dir, packs_to_include, NULL,\n+\t\t\t\t   preferred_pack_name, flags);\n }\n \n struct clear_midx_data {\n@@ -1660,7 +1686,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tfree(count);\n \n \tif (packs_to_drop.nr) {\n-\t\tresult = write_midx_internal(object_dir, &packs_to_drop, NULL, flags);\n+\t\tresult = write_midx_internal(object_dir, NULL, &packs_to_drop, NULL, flags);\n \t\tm = NULL;\n \t}\n \n@@ -1851,7 +1877,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t\tgoto cleanup;\n \t}\n \n-\tresult = write_midx_internal(object_dir, NULL, NULL, flags);\n+\tresult = write_midx_internal(object_dir, NULL, NULL, NULL, flags);\n \tm = NULL;\n \n cleanup:\ndiff --git a/midx.h b/midx.h\nindex aa3da557bb..80f502d39b 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -2,6 +2,7 @@\n #define MIDX_H\n \n #include \"repository.h\"\n+#include \"string-list.h\"\n \n struct object_id;\n struct pack_entry;\n@@ -62,6 +63,10 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n \n int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n+int write_midx_file_only(const char *object_dir,\n+\t\t\t struct string_list *packs_to_include,\n+\t\t\t const char *preferred_pack_name,\n+\t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);\n int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"436015","messageId":"59556e554565120929549239f1aee5a76d80ac8d.1631730270.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631730270.git.me@ttaylorr.com","subject":"[PATCH v2 2/8] builtin/multi-pack-index.c: support `--stdin-packs` mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-15T18:24:38Z","receivedAt":"2021-09-15T18:24:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To power a new `--write-midx` mode, `git repack` will want to write a\nmulti-pack index containing a certain set of packs in the repository.\n\nThis new option will be used by `git repack` to write a MIDX which\ncontains only the packs which will survive after the repack (that is, it\nwill exclude any packs which are about to be deleted).\n\nThis patch effectively exposes the function implemented in the previous\ncommit via the `git multi-pack-index` builtin. An alternative approach\nwould have been to call that function from the `git repack` builtin\ndirectly, but this introduces awkward problems around closing and\nreopening the object store, so the MIDX will be written out-of-process.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-multi-pack-index.txt |  4 ++++\n builtin/multi-pack-index.c             | 27 ++++++++++++++++++++++++++\n t/t5319-multi-pack-index.sh            | 15 ++++++++++++++\n 3 files changed, 46 insertions(+)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex a9df3dbd32..009c989ef8 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -45,6 +45,10 @@ write::\n \n \t--[no-]bitmap::\n \t\tControl whether or not a multi-pack bitmap is written.\n+\n+\t--stdin-packs::\n+\t\tWrite a multi-pack index containing only the set of\n+\t\tline-delimited pack index basenames provided over stdin.\n --\n \n verify::\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 73c0113b48..047647b5f2 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -47,6 +47,7 @@ static struct opts_multi_pack_index {\n \tconst char *preferred_pack;\n \tunsigned long batch_size;\n \tunsigned flags;\n+\tint stdin_packs;\n } opts;\n \n static struct option common_opts[] = {\n@@ -61,6 +62,16 @@ static struct option *add_common_options(struct option *prev)\n \treturn parse_options_concat(common_opts, prev);\n }\n \n+static void read_packs_from_stdin(struct string_list *to)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\twhile (strbuf_getline(&buf, stdin) != EOF)\n+\t\tstring_list_append(to, buf.buf);\n+\tstring_list_sort(to);\n+\n+\tstrbuf_release(&buf);\n+}\n+\n static int cmd_multi_pack_index_write(int argc, const char **argv)\n {\n \tstruct option *options;\n@@ -70,6 +81,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\t\t   N_(\"pack for reuse when computing a multi-pack bitmap\")),\n \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n+\t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n+\t\t\t N_(\"write multi-pack index containing only given indexes\")),\n \t\tOPT_END(),\n \t};\n \n@@ -86,6 +99,20 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \n \tFREE_AND_NULL(options);\n \n+\tif (opts.stdin_packs) {\n+\t\tstruct string_list packs = STRING_LIST_INIT_DUP;\n+\t\tint ret;\n+\n+\t\tread_packs_from_stdin(&packs);\n+\n+\t\tret = write_midx_file_only(opts.object_dir, &packs,\n+\t\t\t\t\t   opts.preferred_pack, opts.flags);\n+\n+\t\tstring_list_clear(&packs, 0);\n+\n+\t\treturn ret;\n+\n+\t}\n \treturn write_midx_file(opts.object_dir, opts.preferred_pack,\n \t\t\t       opts.flags);\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex bb04f0f23b..385f0a3efd 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -168,6 +168,21 @@ test_expect_success 'write midx with two packs' '\n \n compare_results_with_midx \"two packs\"\n \n+test_expect_success 'write midx with --stdin-packs' '\n+\trm -fr $objdir/pack/multi-pack-index &&\n+\n+\tidx=\"$(find $objdir/pack -name \"test-2-*.idx\")\" &&\n+\tbasename \"$idx\" >in &&\n+\n+\tgit multi-pack-index write --stdin-packs <in &&\n+\n+\ttest-tool read-midx $objdir | grep \"\\.idx$\" >packs &&\n+\n+\ttest_cmp packs in\n+'\n+\n+compare_results_with_midx \"mixed mode (one pack + extra)\"\n+\n test_expect_success 'write progress off for redirected stderr' '\n \tgit multi-pack-index --object-dir=$objdir write 2>err &&\n \ttest_line_count = 0 err\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"436016","messageId":"42f1ae9edeb55d8e1b0c8f2c3110e9c8326a5fc1.1631730270.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631730270.git.me@ttaylorr.com","subject":"[PATCH v2 3/8] midx: preliminary support for `--refs-snapshot`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-15T18:24:40Z","receivedAt":"2021-09-15T18:24:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To figure out which commits we can write a bitmap for, the multi-pack\nindex/bitmap code does a reachability traversal, marking any commit\nwhich can be found in the MIDX as eligible to receive a bitmap.\n\nThis approach will cause a problem when multi-pack bitmaps are able to\nbe generated from `git repack`, since the reference tips can change\nduring the repack. Even though we ignore commits that don't exist in\nthe MIDX (when doing a scan of the ref tips), it's possible that a\ncommit in the MIDX reaches something that isn't.\n\nThis can happen when a multi-pack index contains some pack which refers\nto loose objects (which by definition aren't included in the multi-pack\nindex).\n\nBy taking a snapshot of the references before we start repacking, we can\nclose that race window. In the above scenario (where we have a packed\nobject pointing at a loose one), we'll either (a) take a snapshot of the\nreferences before seeing the packed one, or (b) take it after, at which\npoint we can guarantee that the loose object will be packed and included\nin the MIDX.\n\nThis patch does just that. It writes a temporary \"reference snapshot\",\nwhich is a list of OIDs that are at the ref tips before writing a\nmulti-pack bitmap. References that are \"preferred\" (i.e,. are a suffix\nof at least one value of the 'pack.preferBitmapTips' configuration) are\nmarked with a special '+'.\n\nThe format is simple: one line per commit at each tip, with an optional\n'+' at the beginning (for preferred references, as described above).\n\nWhen provided, the reference snapshot is used to drive bitmap selection\ninstead of the MIDX code doing its own traversal. When it isn't\nprovided, the usual traversal takes place instead.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-multi-pack-index.txt | 15 +++++\n builtin/multi-pack-index.c             | 11 +++-\n builtin/repack.c                       |  2 +-\n midx.c                                 | 60 ++++++++++++++++---\n midx.h                                 |  6 +-\n t/t5326-multi-pack-bitmaps.sh          | 82 ++++++++++++++++++++++++++\n 6 files changed, 163 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 009c989ef8..27f83932e4 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -49,6 +49,21 @@ write::\n \t--stdin-packs::\n \t\tWrite a multi-pack index containing only the set of\n \t\tline-delimited pack index basenames provided over stdin.\n+\n+\t--refs-snapshot=<path>::\n+\t\tWith `--bitmap`, optionally specify a file which\n+\t\tcontains a \"refs snapshot\" taken prior to repacking.\n++\n+A reference snapshot is composed of line-delimited OIDs corresponding to\n+the reference tips, usually taken by `git repack` prior to generating a\n+new pack. A line may optionally start with a `+` character to indicate\n+that the reference which corresponds to that OID is \"preferred\" (see\n+linkgit:git-config[1]'s `pack.preferBitmapTips`.)\n++\n+The file given at `<path>` is expected to be readable, and can contain\n+duplicates. (If a given OID is given more than once, it is marked as\n+preferred if at least one instance of it begins with the special `+`\n+marker).\n --\n \n verify::\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 047647b5f2..4b827a07c0 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -7,7 +7,8 @@\n #include \"object-store.h\"\n \n #define BUILTIN_MIDX_WRITE_USAGE \\\n-\tN_(\"git multi-pack-index [<options>] write [--preferred-pack=<pack>]\")\n+\tN_(\"git multi-pack-index [<options>] write [--preferred-pack=<pack>]\" \\\n+\t   \"[--refs-snapshot=<path>]\")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [<options>] verify\")\n@@ -45,6 +46,7 @@ static char const * const builtin_multi_pack_index_usage[] = {\n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n \tconst char *preferred_pack;\n+\tconst char *refs_snapshot;\n \tunsigned long batch_size;\n \tunsigned flags;\n \tint stdin_packs;\n@@ -83,6 +85,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\t N_(\"write multi-pack index containing only given indexes\")),\n+\t\tOPT_FILENAME(0, \"refs-snapshot\", &opts.refs_snapshot,\n+\t\t\t     N_(\"refs snapshot for selecting bitmap commits\")),\n \t\tOPT_END(),\n \t};\n \n@@ -106,7 +110,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\tread_packs_from_stdin(&packs);\n \n \t\tret = write_midx_file_only(opts.object_dir, &packs,\n-\t\t\t\t\t   opts.preferred_pack, opts.flags);\n+\t\t\t\t\t   opts.preferred_pack,\n+\t\t\t\t\t   opts.refs_snapshot, opts.flags);\n \n \t\tstring_list_clear(&packs, 0);\n \n@@ -114,7 +119,7 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \n \t}\n \treturn write_midx_file(opts.object_dir, opts.preferred_pack,\n-\t\t\t       opts.flags);\n+\t\t\t       opts.refs_snapshot, opts.flags);\n }\n \n static int cmd_multi_pack_index_verify(int argc, const char **argv)\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 82ab668272..27158a897b 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -733,7 +733,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tunsigned flags = 0;\n \t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))\n \t\t\tflags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;\n-\t\twrite_midx_file(get_object_directory(), NULL, flags);\n+\t\twrite_midx_file(get_object_directory(), NULL, NULL, flags);\n \t}\n \n \tstring_list_clear(&names, 0);\ndiff --git a/midx.c b/midx.c\nindex 0330202fda..97ba3421f2 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -968,7 +968,42 @@ static void bitmap_show_commit(struct commit *commit, void *_data)\n \tdata->commits[data->commits_nr++] = commit;\n }\n \n+static int read_refs_snapshot(const char *refs_snapshot,\n+\t\t\t      struct rev_info *revs)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct object_id oid;\n+\tFILE *f = xfopen(refs_snapshot, \"r\");\n+\n+\twhile (strbuf_getline(&buf, f) != EOF) {\n+\t\tstruct object *object;\n+\t\tint preferred = 0;\n+\t\tchar *hex = buf.buf;\n+\t\tconst char *end = NULL;\n+\n+\t\tif (buf.len && *buf.buf == '+') {\n+\t\t\tpreferred = 1;\n+\t\t\thex = &buf.buf[1];\n+\t\t}\n+\n+\t\tif (parse_oid_hex(hex, &oid, &end) < 0)\n+\t\t\tdie(_(\"could not parse line: %s\"), buf.buf);\n+\t\tif (*end)\n+\t\t\tdie(_(\"malformed line: %s\"), buf.buf);\n+\n+\t\tobject = parse_object_or_die(&oid, NULL);\n+\t\tif (preferred)\n+\t\t\tobject->flags |= NEEDS_BITMAP;\n+\n+\t\tadd_pending_object(revs, object, \"\");\n+\t}\n+\n+\tfclose(f);\n+\treturn 0;\n+}\n+\n static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p,\n+\t\t\t\t\t\t    const char *refs_snapshot,\n \t\t\t\t\t\t    struct write_midx_context *ctx)\n {\n \tstruct rev_info revs;\n@@ -977,8 +1012,12 @@ static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr\n \tcb.ctx = ctx;\n \n \trepo_init_revisions(the_repository, &revs, NULL);\n-\tsetup_revisions(0, NULL, &revs, NULL);\n-\tfor_each_ref(add_ref_to_pending, &revs);\n+\tif (refs_snapshot) {\n+\t\tread_refs_snapshot(refs_snapshot, &revs);\n+\t} else {\n+\t\tsetup_revisions(0, NULL, &revs, NULL);\n+\t\tfor_each_ref(add_ref_to_pending, &revs);\n+\t}\n \n \t/*\n \t * Skipping promisor objects here is intentional, since it only excludes\n@@ -1007,6 +1046,7 @@ static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr\n \n static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n \t\t\t     struct write_midx_context *ctx,\n+\t\t\t     const char *refs_snapshot,\n \t\t\t     unsigned flags)\n {\n \tstruct packing_data pdata;\n@@ -1018,7 +1058,7 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n \n \tprepare_midx_packing_data(&pdata, ctx);\n \n-\tcommits = find_commits_for_midx_bitmap(&commits_nr, ctx);\n+\tcommits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, ctx);\n \n \t/*\n \t * Build the MIDX-order index based on pdata.objects (which is already\n@@ -1066,6 +1106,7 @@ static int write_midx_internal(const char *object_dir,\n \t\t\t       struct string_list *packs_to_include,\n \t\t\t       struct string_list *packs_to_drop,\n \t\t\t       const char *preferred_pack_name,\n+\t\t\t       const char *refs_snapshot,\n \t\t\t       unsigned flags)\n {\n \tchar *midx_name;\n@@ -1359,7 +1400,8 @@ static int write_midx_internal(const char *object_dir,\n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n \tif (flags & MIDX_WRITE_BITMAP) {\n-\t\tif (write_midx_bitmap(midx_name, midx_hash, &ctx, flags) < 0) {\n+\t\tif (write_midx_bitmap(midx_name, midx_hash, &ctx,\n+\t\t\t\t      refs_snapshot, flags) < 0) {\n \t\t\terror(_(\"could not write multi-pack bitmap\"));\n \t\t\tresult = 1;\n \t\t\tgoto cleanup;\n@@ -1394,19 +1436,21 @@ static int write_midx_internal(const char *object_dir,\n \n int write_midx_file(const char *object_dir,\n \t\t    const char *preferred_pack_name,\n+\t\t    const char *refs_snapshot,\n \t\t    unsigned flags)\n {\n \treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n-\t\t\t\t   flags);\n+\t\t\t\t   refs_snapshot, flags);\n }\n \n int write_midx_file_only(const char *object_dir,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n+\t\t\t const char *refs_snapshot,\n \t\t\t unsigned flags)\n {\n \treturn write_midx_internal(object_dir, packs_to_include, NULL,\n-\t\t\t\t   preferred_pack_name, flags);\n+\t\t\t\t   preferred_pack_name, refs_snapshot, flags);\n }\n \n struct clear_midx_data {\n@@ -1686,7 +1730,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tfree(count);\n \n \tif (packs_to_drop.nr) {\n-\t\tresult = write_midx_internal(object_dir, NULL, &packs_to_drop, NULL, flags);\n+\t\tresult = write_midx_internal(object_dir, NULL, &packs_to_drop, NULL, NULL, flags);\n \t\tm = NULL;\n \t}\n \n@@ -1877,7 +1921,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t\tgoto cleanup;\n \t}\n \n-\tresult = write_midx_internal(object_dir, NULL, NULL, NULL, flags);\n+\tresult = write_midx_internal(object_dir, NULL, NULL, NULL, NULL, flags);\n \tm = NULL;\n \n cleanup:\ndiff --git a/midx.h b/midx.h\nindex 80f502d39b..c0b4e21df4 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -62,10 +62,14 @@ int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pa\n int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n \n-int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n+int write_midx_file(const char *object_dir,\n+\t\t    const char *preferred_pack_name,\n+\t\t    const char *refs_snapshot,\n+\t\t    unsigned flags);\n int write_midx_file_only(const char *object_dir,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n+\t\t\t const char *refs_snapshot,\n \t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 4ad7c2c969..069dab3e17 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -283,4 +283,86 @@ test_expect_success 'pack.preferBitmapTips' '\n \t)\n '\n \n+test_expect_success 'writing a bitmap with --refs-snapshot' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit one &&\n+\t\ttest_commit two &&\n+\n+\t\tgit rev-parse one >snapshot &&\n+\n+\t\tgit repack -ad &&\n+\n+\t\t# First, write a MIDX which see both refs/tags/one and\n+\t\t# refs/tags/two (causing both of those commits to receive\n+\t\t# bitmaps).\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\n+\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n+\t\tgrep \"$(git rev-parse two)\" bitmaps &&\n+\n+\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n+\t\trm -fr $midx-$(midx_checksum $objdir).rev &&\n+\t\trm -fr $midx &&\n+\n+\t\t# Then again, but with a refs snapshot which only sees\n+\t\t# refs/tags/one.\n+\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\n+\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n+\t\t! grep \"$(git rev-parse two)\" bitmaps\n+\t)\n+'\n+\n+test_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\n+\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\tsort <commits.raw >commits &&\n+\n+\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\tgit update-ref --stdin <refs &&\n+\n+\t\tgit multi-pack-index write --bitmap &&\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\n+\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\tcomm -13 bitmaps commits >before &&\n+\t\ttest_line_count = 1 before &&\n+\n+\t\t(\n+\t\t\tgrep -vf before commits.raw &&\n+\t\t\t# mark missing commits as preferred\n+\t\t\tsed \"s/^/+/\" before\n+\t\t) >snapshot &&\n+\n+\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n+\t\trm -fr $midx-$(midx_checksum $objdir).rev &&\n+\t\trm -fr $midx &&\n+\n+\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n+\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t! test_cmp before after\n+\t)\n+'\n+\n test_done\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"436017","messageId":"c0d045a9de1a5e75d684b0dd2009a3137b6e5c59.1631730270.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631730270.git.me@ttaylorr.com","subject":"[PATCH v2 4/8] builtin/repack.c: keep track of existing packs unconditionally","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-15T18:24:43Z","receivedAt":"2021-09-15T18:24:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In order to be able to write a multi-pack index during repacking, `git\nrepack` must keep track of which packs it wants to write into the MIDX.\nThis set is the union of existing packs which will not be deleted,\nnew pack(s) generated as a result of the repack, and .keep packs.\n\nPrior to this patch, `git repack` populated the list of existing packs\nonly when repacking all-into-one (i.e., with `-A` or `-a`), but we will\nsoon need to know this list when repacking when writing a MIDX without\na-i-o.\n\nPopulate the list of existing packs unconditionally, and guard removing\npacks from that list only when repacking a-i-o.\n\nAdditionally, keep track of filenames of kept packs separately, since\nthis, too, will be used in an upcoming patch.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/repack.c | 49 ++++++++++++++++++++++++++----------------------\n 1 file changed, 27 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 27158a897b..e55a650de5 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -98,8 +98,9 @@ static void remove_pack_on_signal(int signo)\n  * have a corresponding .keep file. These packs are not to\n  * be kept if we are going to pack everything into one file.\n  */\n-static void get_non_kept_pack_filenames(struct string_list *fname_list,\n-\t\t\t\t\tconst struct string_list *extra_keep)\n+static void collect_pack_filenames(struct string_list *fname_list,\n+\t\t\t\t   struct string_list *fname_kept_list,\n+\t\t\t\t   const struct string_list *extra_keep)\n {\n \tDIR *dir;\n \tstruct dirent *e;\n@@ -112,21 +113,20 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list,\n \t\tsize_t len;\n \t\tint i;\n \n+\t\tif (!strip_suffix(e->d_name, \".pack\", &len))\n+\t\t\tcontinue;\n+\n \t\tfor (i = 0; i < extra_keep->nr; i++)\n \t\t\tif (!fspathcmp(e->d_name, extra_keep->items[i].string))\n \t\t\t\tbreak;\n-\t\tif (extra_keep->nr > 0 && i < extra_keep->nr)\n-\t\t\tcontinue;\n-\n-\t\tif (!strip_suffix(e->d_name, \".pack\", &len))\n-\t\t\tcontinue;\n \n \t\tfname = xmemdupz(e->d_name, len);\n \n-\t\tif (!file_exists(mkpath(\"%s/%s.keep\", packdir, fname)))\n-\t\t\tstring_list_append_nodup(fname_list, fname);\n+\t\tif ((extra_keep->nr > 0 && i < extra_keep->nr) ||\n+\t\t    (file_exists(mkpath(\"%s/%s.keep\", packdir, fname))))\n+\t\t\tstring_list_append_nodup(fname_kept_list, fname);\n \t\telse\n-\t\t\tfree(fname);\n+\t\t\tstring_list_append_nodup(fname_list, fname);\n \t}\n \tclosedir(dir);\n }\n@@ -440,6 +440,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list names = STRING_LIST_INIT_DUP;\n \tstruct string_list rollback = STRING_LIST_INIT_NODUP;\n \tstruct string_list existing_packs = STRING_LIST_INIT_DUP;\n+\tstruct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n \tstruct pack_geometry *geometry = NULL;\n \tstruct strbuf line = STRBUF_INIT;\n \tint i, ext, ret;\n@@ -572,9 +573,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (use_delta_islands)\n \t\tstrvec_push(&cmd.args, \"--delta-islands\");\n \n+\tcollect_pack_filenames(&existing_packs, &existing_kept_packs,\n+\t\t\t       &keep_pack_list);\n+\n \tif (pack_everything & ALL_INTO_ONE) {\n-\t\tget_non_kept_pack_filenames(&existing_packs, &keep_pack_list);\n-\n \t\trepack_promisor_objects(&po_args, &names);\n \n \t\tif (existing_packs.nr && delete_redundant) {\n@@ -683,17 +685,19 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \treprepare_packed_git(the_repository);\n \n \tif (delete_redundant) {\n-\t\tconst int hexsz = the_hash_algo->hexsz;\n \t\tint opts = 0;\n-\t\tstring_list_sort(&names);\n-\t\tfor_each_string_list_item(item, &existing_packs) {\n-\t\t\tchar *sha1;\n-\t\t\tsize_t len = strlen(item->string);\n-\t\t\tif (len < hexsz)\n-\t\t\t\tcontinue;\n-\t\t\tsha1 = item->string + len - hexsz;\n-\t\t\tif (!string_list_has_string(&names, sha1))\n-\t\t\t\tremove_redundant_pack(packdir, item->string);\n+\t\tif (pack_everything & ALL_INTO_ONE) {\n+\t\t\tconst int hexsz = the_hash_algo->hexsz;\n+\t\t\tstring_list_sort(&names);\n+\t\t\tfor_each_string_list_item(item, &existing_packs) {\n+\t\t\t\tchar *sha1;\n+\t\t\t\tsize_t len = strlen(item->string);\n+\t\t\t\tif (len < hexsz)\n+\t\t\t\t\tcontinue;\n+\t\t\t\tsha1 = item->string + len - hexsz;\n+\t\t\t\tif (!string_list_has_string(&names, sha1))\n+\t\t\t\t\tremove_redundant_pack(packdir, item->string);\n+\t\t\t}\n \t\t}\n \n \t\tif (geometry) {\n@@ -739,6 +743,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstring_list_clear(&names, 0);\n \tstring_list_clear(&rollback, 0);\n \tstring_list_clear(&existing_packs, 0);\n+\tstring_list_clear(&existing_kept_packs, 0);\n \tclear_pack_geometry(geometry);\n \tstrbuf_release(&line);\n \n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"436018","messageId":"09de03de475d9bd38fa1ce500c45df6c40694cbf.1631730270.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631730270.git.me@ttaylorr.com","subject":"[PATCH v2 5/8] builtin/repack.c: extract showing progress to a variable","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-15T18:24:45Z","receivedAt":"2021-09-15T18:24:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"We only ask whether stderr is a tty before calling\n'prune_packed_objects()', but the subsequent patch will add another use.\n\nExtract this check into a variable so that both can use it without\nhaving to call 'isatty()' twice.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/repack.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex e55a650de5..be470546e6 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -445,6 +445,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct strbuf line = STRBUF_INIT;\n \tint i, ext, ret;\n \tFILE *out;\n+\tint show_progress = isatty(2);\n \n \t/* variables to be filled by option parsing */\n \tint pack_everything = 0;\n@@ -718,7 +719,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t}\n \t\t\tstrbuf_release(&buf);\n \t\t}\n-\t\tif (!po_args.quiet && isatty(2))\n+\t\tif (!po_args.quiet && show_progress)\n \t\t\topts |= PRUNE_PACKED_VERBOSE;\n \t\tprune_packed_objects(opts);\n \n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"436019","messageId":"83dfdb8b125efe970659eeea1e5e88b09546c81e.1631730270.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631730270.git.me@ttaylorr.com","subject":"[PATCH v2 6/8] builtin/repack.c: support writing a MIDX while repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-15T18:24:47Z","receivedAt":"2021-09-15T18:24:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Teach `git repack` a new `--write-midx` option for callers that wish to\npersist a multi-pack index in their repository while repacking.\n\nThere are two existing alternatives to this new flag, but they don't\ncover our particular use-case. These alternatives are:\n\n  - Call 'git multi-pack-index write' after running 'git repack', or\n\n  - Set 'GIT_TEST_MULTI_PACK_INDEX=1' in your environment when running\n    'git repack'.\n\nThe former works, but introduces a gap in bitmap coverage between\nrepacking and writing a new MIDX (since the repack may have deleted a\npack included in the existing MIDX, invalidating it altogether).\n\nSetting the 'GIT_TEST_' environment variable is obviously unsupported.\nIn fact, even if it were supported officially, it still wouldn't work,\nbecause it generates the MIDX *after* redundant packs have been dropped,\nleading to the same issue as above.\n\nIntroduce a new option which eliminates this race by teaching `git\nrepack` to generate the MIDX at the critical point: after the new packs\nhave been written and moved into place, but before the redundant packs\nhave been removed.\n\nThis option is compatible with `git repack`'s '--bitmap' option (it\nchanges the interpretation to be: \"write a bitmap corresponding to the\nMIDX after one has been generated\").\n\nThere is a little bit of additional noise in the patch below to avoid\nrepeating ourselves when selecting which packs to delete. Instead of a\nsingle loop as before (where we iterate over 'existing_packs', decide if\na pack is worth deleting, and if so, delete it), we have two loops (the\nfirst where we decide which ones are worth deleting, and the second\nwhere we actually do the deleting). This makes it so we have a single\ncheck we can make consistently when (1) telling the MIDX which packs we\nwant to exclude, and (2) actually unlinking the redundant packs.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-repack.txt |  14 ++--\n builtin/repack.c             | 129 +++++++++++++++++++++++++++++------\n t/lib-midx.sh                |   8 +++\n t/t7700-repack.sh            |  96 ++++++++++++++++++++++++++\n 4 files changed, 224 insertions(+), 23 deletions(-)\n create mode 100644 t/lib-midx.sh\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 24c00c9384..0f2d235ca5 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -9,7 +9,7 @@ git-repack - Pack unpacked objects in a repository\n SYNOPSIS\n --------\n [verse]\n-'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n+'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m] [--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>] [--write-midx]\n \n DESCRIPTION\n -----------\n@@ -128,10 +128,11 @@ depth is 4095.\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\n-\tonly makes sense when used with `-a` or `-A`, as the bitmaps\n+\tonly makes sense when used with `-a`, `-A` or `-m`, as the bitmaps\n \tmust be able to refer to all reachable objects. This option\n-\toverrides the setting of `repack.writeBitmaps`.  This option\n-\thas no effect if multiple packfiles are created.\n+\toverrides the setting of `repack.writeBitmaps`. This option\n+\thas no effect if multiple packfiles are created, unless writing a\n+\tMIDX (in which case a multi-pack bitmap is created).\n \n --pack-kept-objects::\n \tInclude objects in `.keep` files when repacking.  Note that we\n@@ -190,6 +191,11 @@ to change in the future. This option (implying a drastically different\n repack mode) is not guaranteed to work with all other combinations of\n option to `git repack`.\n \n+-m::\n+--write-midx::\n+\tWrite a multi-pack index (see linkgit:git-multi-pack-index[1])\n+\tcontaining the non-redundant packs.\n+\n CONFIGURATION\n -------------\n \ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex be470546e6..ca18d3dd2e 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -433,6 +433,73 @@ static void clear_pack_geometry(struct pack_geometry *geometry)\n \tgeometry->split = 0;\n }\n \n+static void midx_included_packs(struct string_list *include,\n+\t\t\t\tstruct string_list *existing_packs,\n+\t\t\t\tstruct string_list *existing_kept_packs,\n+\t\t\t\tstruct string_list *names,\n+\t\t\t\tstruct pack_geometry *geometry)\n+{\n+\tstruct string_list_item *item;\n+\n+\tfor_each_string_list_item(item, existing_kept_packs)\n+\t\tstring_list_insert(include, xstrfmt(\"%s.idx\", item->string));\n+\tfor_each_string_list_item(item, names)\n+\t\tstring_list_insert(include, xstrfmt(\"pack-%s.idx\", item->string));\n+\tif (geometry) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tuint32_t i;\n+\t\tfor (i = geometry->split; i < geometry->pack_nr; i++) {\n+\t\t\tstruct packed_git *p = geometry->pack[i];\n+\n+\t\t\tstrbuf_addstr(&buf, pack_basename(p));\n+\t\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\t\tstring_list_insert(include, strbuf_detach(&buf, NULL));\n+\t\t}\n+\t} else {\n+\t\tfor_each_string_list_item(item, existing_packs) {\n+\t\t\tif (item->util)\n+\t\t\t\tcontinue;\n+\t\t\tstring_list_insert(include, xstrfmt(\"%s.idx\", item->string));\n+\t\t}\n+\t}\n+}\n+\n+static int write_midx_included_packs(struct string_list *include,\n+\t\t\t\t     int show_progress, int write_bitmaps)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct string_list_item *item;\n+\tFILE *in;\n+\tint ret;\n+\n+\tcmd.in = -1;\n+\tcmd.git_cmd = 1;\n+\n+\tstrvec_push(&cmd.args, \"multi-pack-index\");\n+\tstrvec_pushl(&cmd.args, \"write\", \"--stdin-packs\", NULL);\n+\n+\tif (show_progress)\n+\t\tstrvec_push(&cmd.args, \"--progress\");\n+\telse\n+\t\tstrvec_push(&cmd.args, \"--no-progress\");\n+\n+\tif (write_bitmaps)\n+\t\tstrvec_push(&cmd.args, \"--bitmap\");\n+\n+\tret = start_command(&cmd);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tin = xfdopen(cmd.in, \"w\");\n+\tfor_each_string_list_item(item, include)\n+\t\tfprintf(in, \"%s\\n\", item->string);\n+\tfclose(in);\n+\n+\treturn finish_command(&cmd);\n+}\n+\n int cmd_repack(int argc, const char **argv, const char *prefix)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -456,6 +523,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint no_update_server_info = 0;\n \tstruct pack_objects_args po_args = {NULL};\n \tint geometric_factor = 0;\n+\tint write_midx = 0;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -498,6 +566,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"do not repack this pack\")),\n \t\tOPT_INTEGER('g', \"geometric\", &geometric_factor,\n \t\t\t    N_(\"find a geometric progression with factor <N>\")),\n+\t\tOPT_BOOL('m', \"write-midx\", &write_midx,\n+\t\t\t   N_(\"write a multi-pack index of the resulting packs\")),\n \t\tOPT_END()\n \t};\n \n@@ -514,8 +584,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"--keep-unreachable and -A are incompatible\"));\n \n \tif (write_bitmaps < 0) {\n-\t\tif (!(pack_everything & ALL_INTO_ONE) ||\n-\t\t    !is_bare_repository())\n+\t\tif (!write_midx &&\n+\t\t    (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n \t\t\twrite_bitmaps = 0;\n \t} else if (write_bitmaps &&\n \t\t   git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&\n@@ -525,7 +595,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (pack_kept_objects < 0)\n \t\tpack_kept_objects = write_bitmaps > 0;\n \n-\tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE))\n+\tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)\n \t\tdie(_(incremental_bitmap_conflict_error));\n \n \tif (geometric_factor) {\n@@ -567,10 +637,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t}\n \tif (has_promisor_remote())\n \t\tstrvec_push(&cmd.args, \"--exclude-promisor-objects\");\n-\tif (write_bitmaps > 0)\n-\t\tstrvec_push(&cmd.args, \"--write-bitmap-index\");\n-\telse if (write_bitmaps < 0)\n-\t\tstrvec_push(&cmd.args, \"--write-bitmap-index-quiet\");\n+\tif (!write_midx) {\n+\t\tif (write_bitmaps > 0)\n+\t\t\tstrvec_push(&cmd.args, \"--write-bitmap-index\");\n+\t\telse if (write_bitmaps < 0)\n+\t\t\tstrvec_push(&cmd.args, \"--write-bitmap-index-quiet\");\n+\t}\n \tif (use_delta_islands)\n \t\tstrvec_push(&cmd.args, \"--delta-islands\");\n \n@@ -683,22 +755,41 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t}\n \t/* End of pack replacement. */\n \n+\tif (delete_redundant && pack_everything & ALL_INTO_ONE) {\n+\t\tconst int hexsz = the_hash_algo->hexsz;\n+\t\tstring_list_sort(&names);\n+\t\tfor_each_string_list_item(item, &existing_packs) {\n+\t\t\tchar *sha1;\n+\t\t\tsize_t len = strlen(item->string);\n+\t\t\tif (len < hexsz)\n+\t\t\t\tcontinue;\n+\t\t\tsha1 = item->string + len - hexsz;\n+\t\t\titem->util = (void*)(intptr_t)!string_list_has_string(&names, sha1);\n+\t\t}\n+\t}\n+\n+\tif (write_midx) {\n+\t\tstruct string_list include = STRING_LIST_INIT_NODUP;\n+\t\tmidx_included_packs(&include, &existing_packs,\n+\t\t\t\t    &existing_kept_packs, &names, geometry);\n+\n+\t\tret = write_midx_included_packs(&include,\n+\t\t\t\t\t\tshow_progress, write_bitmaps > 0);\n+\n+\t\tstring_list_clear(&include, 0);\n+\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n \treprepare_packed_git(the_repository);\n \n \tif (delete_redundant) {\n \t\tint opts = 0;\n-\t\tif (pack_everything & ALL_INTO_ONE) {\n-\t\t\tconst int hexsz = the_hash_algo->hexsz;\n-\t\t\tstring_list_sort(&names);\n-\t\t\tfor_each_string_list_item(item, &existing_packs) {\n-\t\t\t\tchar *sha1;\n-\t\t\t\tsize_t len = strlen(item->string);\n-\t\t\t\tif (len < hexsz)\n-\t\t\t\t\tcontinue;\n-\t\t\t\tsha1 = item->string + len - hexsz;\n-\t\t\t\tif (!string_list_has_string(&names, sha1))\n-\t\t\t\t\tremove_redundant_pack(packdir, item->string);\n-\t\t\t}\n+\t\tfor_each_string_list_item(item, &existing_packs) {\n+\t\t\tif (!item->util)\n+\t\t\t\tcontinue;\n+\t\t\tremove_redundant_pack(packdir, item->string);\n \t\t}\n \n \t\tif (geometry) {\ndiff --git a/t/lib-midx.sh b/t/lib-midx.sh\nnew file mode 100644\nindex 0000000000..1261994744\n--- /dev/null\n+++ b/t/lib-midx.sh\n@@ -0,0 +1,8 @@\n+# test_midx_consistent <objdir>\n+test_midx_consistent () {\n+\tls $1/pack/pack-*.idx | xargs -n 1 basename | sort >expect &&\n+\ttest-tool read-midx $1 | grep ^pack-.*\\.idx$ | sort >actual &&\n+\n+\ttest_cmp expect actual &&\n+\tgit multi-pack-index --object-dir=$1 verify\n+}\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 98eda3bfeb..6792531dfd 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -3,6 +3,8 @@\n test_description='git repack works correctly'\n \n . ./test-lib.sh\n+. \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n+. \"${TEST_DIRECTORY}/lib-midx.sh\"\n \n commit_and_pack () {\n \ttest_commit \"$@\" 1>&2 &&\n@@ -234,4 +236,98 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \ttest_must_be_empty actual\n '\n \n+objdir=.git/objects\n+midx=$objdir/pack/multi-pack-index\n+\n+test_expect_success 'setup for --write-midx tests' '\n+\tgit init midx &&\n+\t(\n+\t\tcd midx &&\n+\t\tgit config core.multiPackIndex true &&\n+\n+\t\ttest_commit base\n+\t)\n+'\n+\n+test_expect_success '--write-midx unchanged' '\n+\t(\n+\t\tcd midx &&\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack &&\n+\t\ttest_path_is_missing $midx &&\n+\t\ttest_path_is_missing $midx-*.bitmap &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_missing $midx-*.bitmap &&\n+\t\ttest_midx_consistent $objdir\n+\t)\n+'\n+\n+test_expect_success '--write-midx with a new pack' '\n+\t(\n+\t\tcd midx &&\n+\t\ttest_commit loose &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_missing $midx-*.bitmap &&\n+\t\ttest_midx_consistent $objdir\n+\t)\n+'\n+\n+test_expect_success '--write-midx with -b' '\n+\t(\n+\t\tcd midx &&\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -mb &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-*.bitmap &&\n+\t\ttest_midx_consistent $objdir\n+\t)\n+'\n+\n+test_expect_success '--write-midx with -d' '\n+\t(\n+\t\tcd midx &&\n+\t\ttest_commit repack &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad --write-midx &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_missing $midx-*.bitmap &&\n+\t\ttest_midx_consistent $objdir\n+\t)\n+'\n+\n+test_expect_success 'cleans up MIDX when appropriate' '\n+\t(\n+\t\tcd midx &&\n+\n+\t\ttest_commit repack-2 &&\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&\n+\n+\t\tchecksum=$(midx_checksum $objdir) &&\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$checksum.bitmap &&\n+\t\ttest_path_is_file $midx-$checksum.rev &&\n+\n+\t\ttest_commit repack-3 &&\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_missing $midx-$checksum.bitmap &&\n+\t\ttest_path_is_missing $midx-$checksum.rev &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n+\n+\t\ttest_commit repack-4 &&\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb &&\n+\n+\t\tfind $objdir/pack -type f -name \"multi-pack-index*\" >files &&\n+\t\ttest_must_be_empty files\n+\t)\n+'\n+\n test_done\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"436020","messageId":"68bc49d8aed438303e422530c74f2e970c7d66a4.1631730270.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631730270.git.me@ttaylorr.com","subject":"[PATCH v2 7/8] builtin/repack.c: make largest pack preferred","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-15T18:24:50Z","receivedAt":"2021-09-15T18:24:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When repacking into a geometric series and writing a multi-pack bitmap,\nit is beneficial to have the largest resulting pack be the preferred\nobject source in the bitmap's MIDX, since selecting the large packs can\nlead to fewer broken delta chains and better compression.\n\nTeach 'git repack' to identify this pack and pass it to the MIDX write\nmachinery in order to mark it as preferred.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-repack.txt |  4 ++++\n builtin/repack.c             | 27 ++++++++++++++++++++++++++-\n pack-bitmap.c                |  2 +-\n pack-bitmap.h                |  1 +\n t/helper/test-read-midx.c    | 25 ++++++++++++++++++++++++-\n t/t7703-repack-geometric.sh  | 22 ++++++++++++++++++++++\n 6 files changed, 78 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 0f2d235ca5..7183fb498f 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -190,6 +190,10 @@ this \"roll-up\", without respect to their reachability. This is subject\n to change in the future. This option (implying a drastically different\n repack mode) is not guaranteed to work with all other combinations of\n option to `git repack`.\n++\n+When writing a multi-pack bitmap, `git repack` selects the largest resulting\n+pack as the preferred pack for object selection by the MIDX (see\n+linkgit:git-multi-pack-index[1]).\n \n -m::\n --write-midx::\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex ca18d3dd2e..8c7bc4551e 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -422,6 +422,25 @@ static void split_pack_geometry(struct pack_geometry *geometry, int factor)\n \tgeometry->split = split;\n }\n \n+static struct packed_git *get_largest_active_pack(struct pack_geometry *geometry)\n+{\n+\tif (!geometry) {\n+\t\t/*\n+\t\t * No geometry means either an all-into-one repack (in which\n+\t\t * case there is only one pack left and it is the largest) or an\n+\t\t * incremental one.\n+\t\t *\n+\t\t * If repacking incrementally, then we could check the size of\n+\t\t * all packs to determine which should be preferred, but leave\n+\t\t * this for later.\n+\t\t */\n+\t\treturn NULL;\n+\t}\n+\tif (geometry->split == geometry->pack_nr)\n+\t\treturn NULL;\n+\treturn geometry->pack[geometry->pack_nr - 1];\n+}\n+\n static void clear_pack_geometry(struct pack_geometry *geometry)\n {\n \tif (!geometry)\n@@ -467,10 +486,12 @@ static void midx_included_packs(struct string_list *include,\n }\n \n static int write_midx_included_packs(struct string_list *include,\n+\t\t\t\t     struct pack_geometry *geometry,\n \t\t\t\t     int show_progress, int write_bitmaps)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tstruct string_list_item *item;\n+\tstruct packed_git *largest = get_largest_active_pack(geometry);\n \tFILE *in;\n \tint ret;\n \n@@ -488,6 +509,10 @@ static int write_midx_included_packs(struct string_list *include,\n \tif (write_bitmaps)\n \t\tstrvec_push(&cmd.args, \"--bitmap\");\n \n+\tif (largest)\n+\t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n+\t\t\t     pack_basename(largest));\n+\n \tret = start_command(&cmd);\n \tif (ret)\n \t\treturn ret;\n@@ -773,7 +798,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tmidx_included_packs(&include, &existing_packs,\n \t\t\t\t    &existing_kept_packs, &names, geometry);\n \n-\t\tret = write_midx_included_packs(&include,\n+\t\tret = write_midx_included_packs(&include, geometry,\n \t\t\t\t\t\tshow_progress, write_bitmaps > 0);\n \n \t\tstring_list_clear(&include, 0);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 8504110a4d..67be9be9a6 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1418,7 +1418,7 @@ static int try_partial_reuse(struct packed_git *pack,\n \treturn 0;\n }\n \n-static uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n+uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n {\n \tstruct multi_pack_index *m = bitmap_git->midx;\n \tif (!m)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 469090bad2..7d407c5a4c 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -55,6 +55,7 @@ int test_bitmap_commits(struct repository *r);\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\t\t\t struct list_objects_filter_options *filter,\n \t\t\t\t\t int filter_provided_objects);\n+uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git);\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *,\n \t\t\t\t       struct packed_git **packfile,\n \t\t\t\t       uint32_t *entries,\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex cb0d27049a..0038559129 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -3,6 +3,7 @@\n #include \"midx.h\"\n #include \"repository.h\"\n #include \"object-store.h\"\n+#include \"pack-bitmap.h\"\n \n static int read_midx_file(const char *object_dir, int show_objects)\n {\n@@ -72,14 +73,36 @@ static int read_midx_checksum(const char *object_dir)\n \treturn 0;\n }\n \n+static int read_midx_preferred_pack(const char *object_dir)\n+{\n+\tstruct multi_pack_index *midx = NULL;\n+\tstruct bitmap_index *bitmap = NULL;\n+\n+\tsetup_git_directory();\n+\n+\tmidx = load_multi_pack_index(object_dir, 1);\n+\tif (!midx)\n+\t\treturn 1;\n+\n+\tbitmap = prepare_bitmap_git(the_repository);\n+\tif (!(bitmap && bitmap_is_midx(bitmap)))\n+\t\treturn 1;\n+\n+\n+\tprintf(\"%s\\n\", midx->pack_names[midx_preferred_pack(bitmap)]);\n+\treturn 0;\n+}\n+\n int cmd__read_midx(int argc, const char **argv)\n {\n \tif (!(argc == 2 || argc == 3))\n-\t\tusage(\"read-midx [--show-objects|--checksum] <object-dir>\");\n+\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack] <object-dir>\");\n \n \tif (!strcmp(argv[1], \"--show-objects\"))\n \t\treturn read_midx_file(argv[2], 1);\n \telse if (!strcmp(argv[1], \"--checksum\"))\n \t\treturn read_midx_checksum(argv[2]);\n+\telse if (!strcmp(argv[1], \"--preferred-pack\"))\n+\t\treturn read_midx_preferred_pack(argv[2]);\n \treturn read_midx_file(argv[1], 0);\n }\ndiff --git a/t/t7703-repack-geometric.sh b/t/t7703-repack-geometric.sh\nindex 5ccaa440e0..54360c9878 100755\n--- a/t/t7703-repack-geometric.sh\n+++ b/t/t7703-repack-geometric.sh\n@@ -180,4 +180,26 @@ test_expect_success '--geometric ignores kept packs' '\n \t)\n '\n \n+test_expect_success '--geometric chooses largest MIDX preferred pack' '\n+\tgit init geometric &&\n+\ttest_when_finished \"rm -fr geometric\" &&\n+\t(\n+\t\tcd geometric &&\n+\n+\t\t# These packs already form a geometric progression.\n+\t\ttest_commit_bulk --start=1 1 && # 3 objects\n+\t\ttest_commit_bulk --start=2 2 && # 6 objects\n+\t\tls $objdir/pack/pack-*.idx >before &&\n+\t\ttest_commit_bulk --start=4 4 && # 12 objects\n+\t\tls $objdir/pack/pack-*.idx >after &&\n+\n+\t\tgit repack --geometric 2 -dbm &&\n+\n+\t\tcomm -3 before after | xargs -n 1 basename >expect &&\n+\t\ttest-tool read-midx --preferred-pack $objdir >actual &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"436021","messageId":"eb24b308ecdbd7736fc63927130540abd0a668db.1631730270.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631730270.git.me@ttaylorr.com","subject":"[PATCH v2 8/8] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-15T18:24:52Z","receivedAt":"2021-09-15T18:24:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To prevent the race described in an earlier patch, generate and pass a\nreference snapshot to the multi-pack bitmap code, if we are writing one\nfrom `git repack`.\n\nThis patch is mostly limited to creating a temporary file, and then\ncalling for_each_ref(). Except we try to minimize duplicates, since\ndoing so can drastically reduce the size in network-of-forks style\nrepositories. In the kernel's fork network (the repository containing\nall objects from the kernel and all its forks), deduplicating the\nreferences drops the snapshot size from 934 MB to just 12 MB.\n\nBut since we're handling duplicates in this way, we have to make sure\nthat we preferred references (those listed in pack.preferBitmapTips)\nbefore non-preferred ones (to avoid recording an object which is pointed\nat by a preferred tip as non-preferred).\n\nWe accomplish this by doing separate passes over the references: first\nvisiting each prefix in pack.preferBitmapTips, and then over the rest of\nthe references.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/repack.c | 79 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 79 insertions(+)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 8c7bc4551e..a0b515b139 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -15,6 +15,8 @@\n #include \"promisor-remote.h\"\n #include \"shallow.h\"\n #include \"pack.h\"\n+#include \"pack-bitmap.h\"\n+#include \"refs.h\"\n \n static int delta_base_offset = 1;\n static int pack_kept_objects = -1;\n@@ -452,6 +454,65 @@ static void clear_pack_geometry(struct pack_geometry *geometry)\n \tgeometry->split = 0;\n }\n \n+struct midx_snapshot_ref_data {\n+\tstruct tempfile *f;\n+\tstruct oidset seen;\n+\tint preferred;\n+};\n+\n+static int midx_snapshot_ref_one(const char *refname,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t int flag, void *_data)\n+{\n+\tstruct midx_snapshot_ref_data *data = _data;\n+\tstruct object_id peeled;\n+\n+\tif (!peel_iterated_oid(oid, &peeled))\n+\t\toid = &peeled;\n+\n+\tif (oidset_insert(&data->seen, oid))\n+\t\treturn 0; /* already seen */\n+\n+\tif (oid_object_info(the_repository, oid, NULL) != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tfprintf(data->f->fp, \"%s%s\\n\", data->preferred ? \"+\" : \"\",\n+\t\toid_to_hex(oid));\n+\n+\treturn 0;\n+}\n+\n+static void midx_snapshot_refs(struct tempfile *f)\n+{\n+\tstruct midx_snapshot_ref_data data;\n+\tconst struct string_list *preferred = bitmap_preferred_tips(the_repository);\n+\n+\tdata.f = f;\n+\toidset_init(&data.seen, 0);\n+\n+\tif (!fdopen_tempfile(f, \"w\"))\n+\t\t die(_(\"could not open tempfile %s for writing\"),\n+\t\t     get_tempfile_path(f));\n+\n+\tif (preferred) {\n+\t\tstruct string_list_item *item;\n+\n+\t\tdata.preferred = 1;\n+\t\tfor_each_string_list_item(item, preferred)\n+\t\t\tfor_each_ref_in(item->string, midx_snapshot_ref_one, &data);\n+\t\tdata.preferred = 0;\n+\t}\n+\n+\tfor_each_ref(midx_snapshot_ref_one, &data);\n+\n+\tif (close_tempfile_gently(f)) {\n+\t\tint save_errno = errno;\n+\t\tdelete_tempfile(&f);\n+\t\terrno = save_errno;\n+\t\tdie_errno(_(\"could not close refs snapshot tempfile\"));\n+\t}\n+}\n+\n static void midx_included_packs(struct string_list *include,\n \t\t\t\tstruct string_list *existing_packs,\n \t\t\t\tstruct string_list *existing_kept_packs,\n@@ -487,6 +548,7 @@ static void midx_included_packs(struct string_list *include,\n \n static int write_midx_included_packs(struct string_list *include,\n \t\t\t\t     struct pack_geometry *geometry,\n+\t\t\t\t     const char *refs_snapshot,\n \t\t\t\t     int show_progress, int write_bitmaps)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -513,6 +575,9 @@ static int write_midx_included_packs(struct string_list *include,\n \t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n \t\t\t     pack_basename(largest));\n \n+\tif (refs_snapshot)\n+\t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\", refs_snapshot);\n+\n \tret = start_command(&cmd);\n \tif (ret)\n \t\treturn ret;\n@@ -535,6 +600,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n \tstruct pack_geometry *geometry = NULL;\n \tstruct strbuf line = STRBUF_INIT;\n+\tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n \tFILE *out;\n \tint show_progress = isatty(2);\n@@ -623,6 +689,18 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)\n \t\tdie(_(incremental_bitmap_conflict_error));\n \n+\tif (write_midx && write_bitmaps) {\n+\t\tstruct strbuf path = STRBUF_INIT;\n+\n+\t\tstrbuf_addf(&path, \"%s/%s_XXXXXX\", get_object_directory(),\n+\t\t\t    \"bitmap-ref-tips\");\n+\n+\t\trefs_snapshot = xmks_tempfile(path.buf);\n+\t\tmidx_snapshot_refs(refs_snapshot);\n+\n+\t\tstrbuf_release(&path);\n+\t}\n+\n \tif (geometric_factor) {\n \t\tif (pack_everything)\n \t\t\tdie(_(\"--geometric is incompatible with -A, -a\"));\n@@ -799,6 +877,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\t    &existing_kept_packs, &names, geometry);\n \n \t\tret = write_midx_included_packs(&include, geometry,\n+\t\t\t\t\t\trefs_snapshot ? get_tempfile_path(refs_snapshot) : NULL,\n \t\t\t\t\t\tshow_progress, write_bitmaps > 0);\n \n \t\tstring_list_clear(&include, 0);\n-- \n2.33.0.96.g73915697e6\n"},{"id":"436037","messageId":"xmqqmtodwsnz.fsf@gitster.g","threadId":"56482","inReplyTo":"cover.1631730270.git.me@ttaylorr.com","subject":"Re: [PATCH v2 0/8] repack: introduce `--write-midx`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-15T19:22:08Z","receivedAt":"2021-09-15T19:22:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> Nothing substantial has changed since last time. Mostly re-wrapping lines and\n> removing braces in macro'd for_each_xyz() loops....\n> Otherwise, this series is unchagned. It depends on tb/multi-pack-bitmaps, which\n> has graduated to master. Range-diff below:\n\nDespite the explanation above ...\n\n>      @@ midx.c: static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n>     - \n>     - \tif (ends_with(file_name, \".idx\")) {\n>       \t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n>     --\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n>     --\t\t\treturn;\n>     -+\t\tif (ctx->m) {\n>     -+\t\t\tif (midx_contains_pack(ctx->m, file_name))\n>     -+\t\t\t\treturn;\n>     -+\t\t} else if (ctx->to_include) {\n>     -+\t\t\tif (!string_list_has_string(ctx->to_include, file_name))\n>     -+\t\t\t\treturn;\n>     -+\t\t}\n>     + \t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n>     + \t\t\treturn;\n>     ++\t\telse if (ctx->to_include &&\n>     ++\t\t\t !string_list_has_string(ctx->to_include, file_name))\n>     ++\t\t\treturn;\n\n... this part seems to change what the code does quite a bit.\n\nThe original used to skip .to_include checks when .m is set but it\ndid not pass midx_contains_pack().  Now .to_include check is done\nin a context with .m that does not pass midex_contains_pack() check.\n\nI suspect that this change since v1 is a bugfix.  We make an early\nreturn based on midx_contains_pack() but make sure .m is not NULL,\nin order to be able to run the check.  There is another early return\ncondition that is orthogonal, and we do string_list check, but that\nis only doable if the .to_include is not NULL.  The logic in v2 \nclearly expresses that, but v1 was simply buggy.\n\nBut if it is the case, I'd step back a bit and further question if\n\"else if\" is a good construct to use here.  We'd return if .m passes\nmidx_contains_pack() check, and another check based on .to_include\ngives us an orthogonal chance to return early, so two \"if\" statement\nthat are independent sitting next to each other may have avoided\nsuch a bug from the beginning, perhaps?\n\n\tif (ctx->m && midx_contains...)\n\t\treturn;\n\tif (ctx->to_include && !string_list_has...)\n\t\treturn;\n\nOf course you could\n\n\tif ((ctx->m && midx_contains...) ||\n\t    (ctx->to_include && !string_list_has...))\n\t\treturn;\n\nbut that may not scale as well as two independent if statements.\n\nEverything else in the range-diff looked cosmetic as explained in\nthe cover letter.\n"},{"id":"436038","messageId":"xmqqilz1wsbz.fsf@gitster.g","threadId":"56482","inReplyTo":"xmqqmtodwsnz.fsf@gitster.g","subject":"Re: [PATCH v2 0/8] repack: introduce `--write-midx`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-15T19:29:20Z","receivedAt":"2021-09-15T19:29:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> But if it is the case, I'd step back a bit and further question if\n> \"else if\" is a good construct to use here.  We'd return if .m passes\n> midx_contains_pack() check, and another check based on .to_include\n> gives us an orthogonal chance to return early, so two \"if\" statement\n> that are independent sitting next to each other may have avoided\n> such a bug from the beginning, perhaps?\n\nOK, I went back and checked your response to a review in an earlier\nround.  If .m and .to_include cannot be turned on at the same time,\nthen I think \"else if\" would express the intention more clearly.\n\nBut if we go that route, the whole \"if ... else if\" may deserve a\ncomment that explains why .m and .to_include are fundamentally and\ninherently mutually exclusive.  In other words, is it possible if\nfuture enhancement may want to pass both .m and .to_include to allow\nthe code path to check both conditions and return early?\n\nThanks.\n"},{"id":"436053","messageId":"YUJjUIEmIFTH0+jq@nand.local","threadId":"56482","inReplyTo":"xmqqilz1wsbz.fsf@gitster.g","subject":"Re: [PATCH v2 0/8] repack: introduce `--write-midx`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-15T21:19:12Z","receivedAt":"2021-09-15T21:19:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 15, 2021 at 12:29:20PM -0700, Junio C Hamano wrote:\n> Junio C Hamano <gitster@pobox.com> writes:\n>\n> > But if it is the case, I'd step back a bit and further question if\n> > \"else if\" is a good construct to use here.  We'd return if .m passes\n> > midx_contains_pack() check, and another check based on .to_include\n> > gives us an orthogonal chance to return early, so two \"if\" statement\n> > that are independent sitting next to each other may have avoided\n> > such a bug from the beginning, perhaps?\n>\n> OK, I went back and checked your response to a review in an earlier\n> round.  If .m and .to_include cannot be turned on at the same time,\n> then I think \"else if\" would express the intention more clearly.\n\nYeah, exactly. I didn't think that this would be as interesting to\nreviewers as it ended up being, hence I didn't put much thought into it.\n\n> But if we go that route, the whole \"if ... else if\" may deserve a\n> comment that explains why .m and .to_include are fundamentally and\n> inherently mutually exclusive.  In other words, is it possible if\n> future enhancement may want to pass both .m and .to_include to allow\n> the code path to check both conditions and return early?\n\nThe gist is that they aren't inherently exclusive, but that using an\nexisting MIDX (which is the result of having `.m` point at a MIDX\nstruct) right now blindly reuses all of the existing packs in that MIDX,\nwhich is what to_include is trying to avoid.\n\nWe could reuse an existing MIDX with to_include by filtering down its\npacks before carrying them forward, but don't currently. So that would\ncause this change to produce unexpected results if we ever changed that\nbehavior.\n\nI think all of this is non-obvious enough that it warrants being written\ndown in a comment. Here's a replacement for that first patch that should\napply without conflict. But if you'd rather have a reroll or anything\nelse, I'm happy to do that, too.\n\n--- >8 ---\n\nSubject: [PATCH] midx: expose `write_midx_file_only()` publicly\n\nExpose a variant of the write_midx_file() function which ignores packs\nthat aren't included in an explicit \"allow\" list.\n\nThis will be used in an upcoming patch to power a new `--stdin-packs`\nmode of `git multi-pack-index write` for callers that only want to\ninclude certain packs in a MIDX (and ignore any packs which may have\nhappened to enter the repository independently, e.g., from pushes).\n\nThose patches will provide test coverage for this new function.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++---------\n midx.h |  5 +++++\n 2 files changed, 55 insertions(+), 9 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 864034a6ad..61226eaa9d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -475,6 +475,8 @@ struct write_midx_context {\n \tuint32_t num_large_offsets;\n\n \tint preferred_pack_idx;\n+\n+\tstruct string_list *to_include;\n };\n\n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -484,8 +486,26 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n\n \tif (ends_with(file_name, \".idx\")) {\n \t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n+\t\t/*\n+\t\t * Note that at most one of ctx->m and ctx->to_include are set,\n+\t\t * so we are testing midx_contains_pack() and\n+\t\t * string_list_has_string() independently (guarded by the\n+\t\t * appropriate NULL checks).\n+\t\t *\n+\t\t * We could support passing to_include while reusing an existing\n+\t\t * MIDX, but don't currently since the reuse process drags\n+\t\t * forward all packs from an existing MIDX (without checking\n+\t\t * whether or not they appear in the to_include list).\n+\t\t *\n+\t\t * If we added support for that, these next two conditional\n+\t\t * should be performed independently (likely checking\n+\t\t * to_include before the existing MIDX).\n+\t\t */\n \t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n \t\t\treturn;\n+\t\telse if (ctx->to_include &&\n+\t\t\t !string_list_has_string(ctx->to_include, file_name))\n+\t\t\treturn;\n\n \t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n\n@@ -1058,6 +1078,7 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n }\n\n static int write_midx_internal(const char *object_dir,\n+\t\t\t       struct string_list *packs_to_include,\n \t\t\t       struct string_list *packs_to_drop,\n \t\t\t       const char *preferred_pack_name,\n \t\t\t       unsigned flags)\n@@ -1082,10 +1103,17 @@ static int write_midx_internal(const char *object_dir,\n \t\tdie_errno(_(\"unable to create leading directories of %s\"),\n \t\t\t  midx_name);\n\n-\tfor (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {\n-\t\tif (!strcmp(object_dir, cur->object_dir)) {\n-\t\t\tctx.m = cur;\n-\t\t\tbreak;\n+\tif (!packs_to_include) {\n+\t\t/*\n+\t\t * Only reference an existing MIDX when not filtering which\n+\t\t * packs to include, since all packs and objects are copied\n+\t\t * blindly from an existing MIDX if one is present.\n+\t\t */\n+\t\tfor (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {\n+\t\t\tif (!strcmp(object_dir, cur->object_dir)) {\n+\t\t\t\tctx.m = cur;\n+\t\t\t\tbreak;\n+\t\t\t}\n \t\t}\n \t}\n\n@@ -1136,10 +1164,13 @@ static int write_midx_internal(const char *object_dir,\n \telse\n \t\tctx.progress = NULL;\n\n+\tctx.to_include = packs_to_include;\n+\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n\n-\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop) {\n+\tif ((ctx.m && ctx.nr == ctx.m->num_packs) &&\n+\t    !(packs_to_include || packs_to_drop)) {\n \t\tstruct bitmap_index *bitmap_git;\n \t\tint bitmap_exists;\n \t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n@@ -1237,7 +1268,7 @@ static int write_midx_internal(const char *object_dir,\n\n \tQSORT(ctx.info, ctx.nr, pack_info_compare);\n\n-\tif (packs_to_drop && packs_to_drop->nr) {\n+\tif (ctx.m && packs_to_drop && packs_to_drop->nr) {\n \t\tint drop_index = 0;\n \t\tint missing_drops = 0;\n\n@@ -1380,7 +1411,17 @@ int write_midx_file(const char *object_dir,\n \t\t    const char *preferred_pack_name,\n \t\t    unsigned flags)\n {\n-\treturn write_midx_internal(object_dir, NULL, preferred_pack_name, flags);\n+\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n+\t\t\t\t   flags);\n+}\n+\n+int write_midx_file_only(const char *object_dir,\n+\t\t\t struct string_list *packs_to_include,\n+\t\t\t const char *preferred_pack_name,\n+\t\t\t unsigned flags)\n+{\n+\treturn write_midx_internal(object_dir, packs_to_include, NULL,\n+\t\t\t\t   preferred_pack_name, flags);\n }\n\n struct clear_midx_data {\n@@ -1660,7 +1701,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tfree(count);\n\n \tif (packs_to_drop.nr) {\n-\t\tresult = write_midx_internal(object_dir, &packs_to_drop, NULL, flags);\n+\t\tresult = write_midx_internal(object_dir, NULL, &packs_to_drop, NULL, flags);\n \t\tm = NULL;\n \t}\n\n@@ -1851,7 +1892,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t\tgoto cleanup;\n \t}\n\n-\tresult = write_midx_internal(object_dir, NULL, NULL, flags);\n+\tresult = write_midx_internal(object_dir, NULL, NULL, NULL, flags);\n \tm = NULL;\n\n cleanup:\ndiff --git a/midx.h b/midx.h\nindex aa3da557bb..80f502d39b 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -2,6 +2,7 @@\n #define MIDX_H\n\n #include \"repository.h\"\n+#include \"string-list.h\"\n\n struct object_id;\n struct pack_entry;\n@@ -62,6 +63,10 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n\n int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n+int write_midx_file_only(const char *object_dir,\n+\t\t\t struct string_list *packs_to_include,\n+\t\t\t const char *preferred_pack_name,\n+\t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);\n int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);\n--\n2.33.0.96.g73915697e6\n\n"},{"id":"436176","messageId":"xmqq8rzwrwsc.fsf@gitster.g","threadId":"56482","inReplyTo":"YUJjUIEmIFTH0+jq@nand.local","subject":"Re: [PATCH v2 0/8] repack: introduce `--write-midx`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-16T22:16:35Z","receivedAt":"2021-09-16T22:16:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> I think all of this is non-obvious enough that it warrants being written\n> down in a comment. Here's a replacement for that first patch that should\n> apply without conflict. But if you'd rather have a reroll or anything\n> else, I'm happy to do that, too.\n\nThanks; will replace.  The comment reads well.\n\n"},{"id":"436791","messageId":"YUuuYW3RktlDzskH@google.com","threadId":"56482","inReplyTo":"59556e554565120929549239f1aee5a76d80ac8d.1631730270.git.me@ttaylorr.com","subject":"Re: [PATCH v2 2/8] builtin/multi-pack-index.c: support `--stdin-packs` mode","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2021-09-22T22:29:53Z","receivedAt":"2021-09-22T22:30:04Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Thanks for the series! I have a couple of questions:\n\n\nOn 2021.09.15 14:24, Taylor Blau wrote:\n> To power a new `--write-midx` mode, `git repack` will want to write a\n> multi-pack index containing a certain set of packs in the repository.\n> \n> This new option will be used by `git repack` to write a MIDX which\n> contains only the packs which will survive after the repack (that is, it\n> will exclude any packs which are about to be deleted).\n> \n> This patch effectively exposes the function implemented in the previous\n> commit via the `git multi-pack-index` builtin. An alternative approach\n> would have been to call that function from the `git repack` builtin\n> directly, but this introduces awkward problems around closing and\n> reopening the object store, so the MIDX will be written out-of-process.\n\nCould you elaborate a bit on the \"awkward problems\" here? I'm afraid I'm\nmissing the context here.\n\n\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> index 73c0113b48..047647b5f2 100644\n> --- a/builtin/multi-pack-index.c\n> +++ b/builtin/multi-pack-index.c\n> @@ -47,6 +47,7 @@ static struct opts_multi_pack_index {\n>  \tconst char *preferred_pack;\n>  \tunsigned long batch_size;\n>  \tunsigned flags;\n> +\tint stdin_packs;\n>  } opts;\n>  \n>  static struct option common_opts[] = {\n> @@ -61,6 +62,16 @@ static struct option *add_common_options(struct option *prev)\n>  \treturn parse_options_concat(common_opts, prev);\n>  }\n>  \n> +static void read_packs_from_stdin(struct string_list *to)\n> +{\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +\twhile (strbuf_getline(&buf, stdin) != EOF)\n> +\t\tstring_list_append(to, buf.buf);\n> +\tstring_list_sort(to);\n> +\n> +\tstrbuf_release(&buf);\n> +}\n> +\n\nI'm presuming that the packfile list is going to be generated\nautomatically, but what happens if that becomes corrupt somehow, and we\nskip a packfile that should have been included? Will that cause\nincorrect behavior, or will we just miss out on some of the bitmap\nperformance benefits?\n"},{"id":"436792","messageId":"YUuvjZl43k3G2SBn@google.com","threadId":"56482","inReplyTo":"42f1ae9edeb55d8e1b0c8f2c3110e9c8326a5fc1.1631730270.git.me@ttaylorr.com","subject":"Re: [PATCH v2 3/8] midx: preliminary support for `--refs-snapshot`","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2021-09-22T22:34:53Z","receivedAt":"2021-09-22T22:35:02Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"A small nitpick for this patch:\n\nOn 2021.09.15 14:24, Taylor Blau wrote:\n> diff --git a/midx.c b/midx.c\n> index 0330202fda..97ba3421f2 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -968,7 +968,42 @@ static void bitmap_show_commit(struct commit *commit, void *_data)\n>  \tdata->commits[data->commits_nr++] = commit;\n>  }\n>  \n> +static int read_refs_snapshot(const char *refs_snapshot,\n> +\t\t\t      struct rev_info *revs)\n> +{\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +\tstruct object_id oid;\n> +\tFILE *f = xfopen(refs_snapshot, \"r\");\n> +\n> +\twhile (strbuf_getline(&buf, f) != EOF) {\n> +\t\tstruct object *object;\n> +\t\tint preferred = 0;\n> +\t\tchar *hex = buf.buf;\n> +\t\tconst char *end = NULL;\n> +\n> +\t\tif (buf.len && *buf.buf == '+') {\n> +\t\t\tpreferred = 1;\n> +\t\t\thex = &buf.buf[1];\n> +\t\t}\n> +\n> +\t\tif (parse_oid_hex(hex, &oid, &end) < 0)\n> +\t\t\tdie(_(\"could not parse line: %s\"), buf.buf);\n> +\t\tif (*end)\n> +\t\t\tdie(_(\"malformed line: %s\"), buf.buf);\n> +\n> +\t\tobject = parse_object_or_die(&oid, NULL);\n> +\t\tif (preferred)\n> +\t\t\tobject->flags |= NEEDS_BITMAP;\n> +\n> +\t\tadd_pending_object(revs, object, \"\");\n> +\t}\n> +\n> +\tfclose(f);\n> +\treturn 0;\n> +}\n\n`buf` needs to be released here.\n"},{"id":"436793","messageId":"20210922223936.2606010-1-jonathantanmy@google.com","threadId":"56482","inReplyTo":"83dfdb8b125efe970659eeea1e5e88b09546c81e.1631730270.git.me@ttaylorr.com","subject":"Re: [PATCH v2 6/8] builtin/repack.c: support writing a MIDX while repacking","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-22T22:39:35Z","receivedAt":"2021-09-22T22:39:42Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> @@ -683,22 +755,41 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n>  \t}\n>  \t/* End of pack replacement. */\n>  \n> +\tif (delete_redundant && pack_everything & ALL_INTO_ONE) {\n> +\t\tconst int hexsz = the_hash_algo->hexsz;\n> +\t\tstring_list_sort(&names);\n> +\t\tfor_each_string_list_item(item, &existing_packs) {\n> +\t\t\tchar *sha1;\n> +\t\t\tsize_t len = strlen(item->string);\n> +\t\t\tif (len < hexsz)\n> +\t\t\t\tcontinue;\n> +\t\t\tsha1 = item->string + len - hexsz;\n> +\t\t\titem->util = (void*)(intptr_t)!string_list_has_string(&names, sha1);\n\nOK, here is the tricky part. They are marked for deletion here...\n\n> +\t\t}\n> +\t}\n> +\n> +\tif (write_midx) {\n> +\t\tstruct string_list include = STRING_LIST_INIT_NODUP;\n> +\t\tmidx_included_packs(&include, &existing_packs,\n> +\t\t\t\t    &existing_kept_packs, &names, geometry);\n\n...the mark for deletion is taken into account during the execution of\nmidx_included_packs() (as can be seen by looking at that function)...\n\n> +\n> +\t\tret = write_midx_included_packs(&include,\n> +\t\t\t\t\t\tshow_progress, write_bitmaps > 0);\n> +\n> +\t\tstring_list_clear(&include, 0);\n> +\n> +\t\tif (ret)\n> +\t\t\treturn ret;\n> +\t}\n> +\n>  \treprepare_packed_git(the_repository);\n>  \n>  \tif (delete_redundant) {\n>  \t\tint opts = 0;\n> -\t\tif (pack_everything & ALL_INTO_ONE) {\n> -\t\t\tconst int hexsz = the_hash_algo->hexsz;\n> -\t\t\tstring_list_sort(&names);\n> -\t\t\tfor_each_string_list_item(item, &existing_packs) {\n> -\t\t\t\tchar *sha1;\n> -\t\t\t\tsize_t len = strlen(item->string);\n> -\t\t\t\tif (len < hexsz)\n> -\t\t\t\t\tcontinue;\n> -\t\t\t\tsha1 = item->string + len - hexsz;\n> -\t\t\t\tif (!string_list_has_string(&names, sha1))\n> -\t\t\t\t\tremove_redundant_pack(packdir, item->string);\n> -\t\t\t}\n> +\t\tfor_each_string_list_item(item, &existing_packs) {\n> +\t\t\tif (!item->util)\n> +\t\t\t\tcontinue;\n\n...and the marks are also used here. I was at first confused about why\nthe functionality of midx_included_packs() depended on whether redundant\npacks were marked for deletion - if they are redundant, shouldn't they\nnever be taken into account (regardless of whether we actually delete\nthem)? But I guess it makes sense as an overall design point that we\npass all packs that are to remain (so if they will be deleted, exclude\nthem, and if they will not be, include them).\n\nI think a comment \"mark this pack for deletion\" at the point we write\nthe mark (so, where the cast to intptr_t is) is worthwhile. Other than\nthat, this patch looks good to me.\n\nI (and the others in our review club) only managed to reach this patch.\nI hope to get to the other 2 by the end of the week.\n"},{"id":"436795","messageId":"20210922225605.2608085-1-jonathantanmy@google.com","threadId":"56482","inReplyTo":"c0d045a9de1a5e75d684b0dd2009a3137b6e5c59.1631730270.git.me@ttaylorr.com","subject":"Re: [PATCH v2 4/8] builtin/repack.c: keep track of existing packs unconditionally","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-22T22:56:05Z","receivedAt":"2021-09-22T22:56:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> @@ -98,8 +98,9 @@ static void remove_pack_on_signal(int signo)\n>   * have a corresponding .keep file. These packs are not to\n>   * be kept if we are going to pack everything into one file.\n>   */\n> -static void get_non_kept_pack_filenames(struct string_list *fname_list,\n> -\t\t\t\t\tconst struct string_list *extra_keep)\n> +static void collect_pack_filenames(struct string_list *fname_list,\n> +\t\t\t\t   struct string_list *fname_kept_list,\n> +\t\t\t\t   const struct string_list *extra_keep)\n>  {\n>  \tDIR *dir;\n>  \tstruct dirent *e;\n\nThe comment in the before-context of this hunk needs to be updated.\n\nAlso, I think that fname_list should be renamed to fname_nonkept_list.\nIt does have the same semantics as before, but the name of the function\nhas changed. And also, fname_list sounds like a superset of\nfname_kept_list, but that is not the case.\n\n> @@ -440,6 +440,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n>  \tstruct string_list names = STRING_LIST_INIT_DUP;\n>  \tstruct string_list rollback = STRING_LIST_INIT_NODUP;\n>  \tstruct string_list existing_packs = STRING_LIST_INIT_DUP;\n> +\tstruct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n>  \tstruct pack_geometry *geometry = NULL;\n>  \tstruct strbuf line = STRBUF_INIT;\n>  \tint i, ext, ret;\n\nLikewise here - the introduction of a new similarly-named variable means\nthat existing_packs should be renamed, I think.\n\nOther than that, this patch looks good.\n"},{"id":"436796","messageId":"20210922230012.2608849-1-jonathantanmy@google.com","threadId":"56482","inReplyTo":"42f1ae9edeb55d8e1b0c8f2c3110e9c8326a5fc1.1631730270.git.me@ttaylorr.com","subject":"Re: [PATCH v2 3/8] midx: preliminary support for `--refs-snapshot`","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-22T23:00:12Z","receivedAt":"2021-09-22T23:00:18Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> This approach will cause a problem when multi-pack bitmaps are able to\n> be generated from `git repack`, since the reference tips can change\n> during the repack. Even though we ignore commits that don't exist in\n> the MIDX (when doing a scan of the ref tips), it's possible that a\n> commit in the MIDX reaches something that isn't.\n> \n> This can happen when a multi-pack index contains some pack which refers\n> to loose objects (which by definition aren't included in the multi-pack\n> index).\n> \n> By taking a snapshot of the references before we start repacking, we can\n> close that race window.\n\nI can understand why we want the refs to remain the same both for the\nMIDX generation and the MIDX bitmap generation (one reason that comes to\nmind is how we select the commits for which to generate bitmaps), but I\ndon't understand what referring to loose objects has to do with it. I\nthink that using the same set of refs for MIDX generation and bitmap\ngeneration is a good enough reason to do this, and we don't need to\nmention loose objects.\n\nThe patch itself looks good.\n"},{"id":"436798","messageId":"20210922231137.2610100-1-jonathantanmy@google.com","threadId":"56482","inReplyTo":"59556e554565120929549239f1aee5a76d80ac8d.1631730270.git.me@ttaylorr.com","subject":"Re: [PATCH v2 2/8] builtin/multi-pack-index.c: support `--stdin-packs` mode","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-22T23:11:37Z","receivedAt":"2021-09-22T23:11:41Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> An alternative approach\n> would have been to call that function from the `git repack` builtin\n> directly, but this introduces awkward problems around closing and\n> reopening the object store, so the MIDX will be written out-of-process.\n\nI'm not sure if the implementation direction started by this patch\n(eventually, running \"git multi-pack-index write --stdin-packs\" to\nreplace the midx of a repository while \"git repack\" is running) would\nwork on platforms in which mmap-ing a file means that other processes\ncan't delete it, but if it works on a Windows CI, this should be fine.\n(I don't have a Windows CI handy to test it on, though.)\n\nAssuming it works on platforms like Windows, this patch looks fine.\n"},{"id":"436801","messageId":"20210922231423.2610917-1-jonathantanmy@google.com","threadId":"56482","inReplyTo":"03c1b2c4d3d9d10a6768684e9c4d1c1cc8019844.1631730270.git.me@ttaylorr.com","subject":"Re: [PATCH v2 1/8] midx: expose `write_midx_file_only()` publicly","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-22T23:14:23Z","receivedAt":"2021-09-22T23:14:30Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> @@ -1237,7 +1253,7 @@ static int write_midx_internal(const char *object_dir,\n>  \n>  \tQSORT(ctx.info, ctx.nr, pack_info_compare);\n>  \n> -\tif (packs_to_drop && packs_to_drop->nr) {\n> +\tif (ctx.m && packs_to_drop && packs_to_drop->nr) {\n>  \t\tint drop_index = 0;\n>  \t\tint missing_drops = 0;\n>  \n\nI couldn't figure out why this requires ctx.m now.\n\n> @@ -62,6 +63,10 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n>  int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n>  \n>  int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n> +int write_midx_file_only(const char *object_dir,\n> +\t\t\t struct string_list *packs_to_include,\n> +\t\t\t const char *preferred_pack_name,\n> +\t\t\t unsigned flags);\n\nIt took me a while to figure out that this function doesn't only write a\nMIDX file, but writes an MIDX file only for certain packs. Maybe worth\nadding a comment here (e.g. \"Write an MIDX file only for the given\npacks\").\n\nOther than that, this patch looks good.\n"},{"id":"436815","messageId":"YUvgbxRmUwZF87b1@nand.local","threadId":"56482","inReplyTo":"YUuuYW3RktlDzskH@google.com","subject":"Re: [PATCH v2 2/8] builtin/multi-pack-index.c: support `--stdin-packs` mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-23T02:03:27Z","receivedAt":"2021-09-23T02:03:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 22, 2021 at 03:29:53PM -0700, Josh Steadmon wrote:\n> Thanks for the series! I have a couple of questions:\n>\n>\n> On 2021.09.15 14:24, Taylor Blau wrote:\n> > To power a new `--write-midx` mode, `git repack` will want to write a\n> > multi-pack index containing a certain set of packs in the repository.\n> >\n> > This new option will be used by `git repack` to write a MIDX which\n> > contains only the packs which will survive after the repack (that is, it\n> > will exclude any packs which are about to be deleted).\n> >\n> > This patch effectively exposes the function implemented in the previous\n> > commit via the `git multi-pack-index` builtin. An alternative approach\n> > would have been to call that function from the `git repack` builtin\n> > directly, but this introduces awkward problems around closing and\n> > reopening the object store, so the MIDX will be written out-of-process.\n>\n> Could you elaborate a bit on the \"awkward problems\" here? I'm afraid I'm\n> missing the context here.\n\nA variety of things can go wrong when the object store is closed and\nre-opened in the same process. Many of the symptoms are described\nbeginning at this message:\n\n  https://lore.kernel.org/git/YPf1m01mcdJ3HNBt@coredump.intra.peff.net/\n\nand further down in the sub-thread. Many of those problems have been\nresolved, but I'm not convinced that there aren't others lurking.\n\n> > +static void read_packs_from_stdin(struct string_list *to)\n> > +{\n> > +\tstruct strbuf buf = STRBUF_INIT;\n> > +\twhile (strbuf_getline(&buf, stdin) != EOF)\n> > +\t\tstring_list_append(to, buf.buf);\n> > +\tstring_list_sort(to);\n> > +\n> > +\tstrbuf_release(&buf);\n> > +}\n> > +\n>\n> I'm presuming that the packfile list is going to be generated\n> automatically, but what happens if that becomes corrupt somehow, and we\n> skip a packfile that should have been included? Will that cause\n> incorrect behavior, or will we just miss out on some of the bitmap\n> performance benefits?\n\nA multi-pack bitmap can only refer to objects that are in a pack which\nthe repository's MIDX includes. So if we left off a pack from this list,\nwe'd be unable to cover that pack's objects in the resulting bitmap.\nWe'd also be unable to cover any objects which are reachable from the\nmissing pack's objects, since the set of objects in a bitmap must be\nclosed under reachability.\n\nIf, on the other hand, we read a line which does not correspond to any\npack, we'll simply ignore it. That's because we loop over the results of\nget_all_packs() and try to find a match in this list instead of the\nother way around.\n\nWe could mark the packs we found by abusing the string_list_item's util\npointer, but it's probably not worth it since this is mostly an internal\ninterface.\n\nThanks,\nTaylor\n"},{"id":"436817","messageId":"YUvhPpRzyfBua0wB@nand.local","threadId":"56482","inReplyTo":"20210922231137.2610100-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 2/8] builtin/multi-pack-index.c: support `--stdin-packs` mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-23T02:06:54Z","receivedAt":"2021-09-23T02:06:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 22, 2021 at 04:11:37PM -0700, Jonathan Tan wrote:\n> > An alternative approach\n> > would have been to call that function from the `git repack` builtin\n> > directly, but this introduces awkward problems around closing and\n> > reopening the object store, so the MIDX will be written out-of-process.\n>\n> I'm not sure if the implementation direction started by this patch\n> (eventually, running \"git multi-pack-index write --stdin-packs\" to\n> replace the midx of a repository while \"git repack\" is running) would\n> work on platforms in which mmap-ing a file means that other processes\n> can't delete it, but if it works on a Windows CI, this should be fine.\n> (I don't have a Windows CI handy to test it on, though.)\n>\n> Assuming it works on platforms like Windows, this patch looks fine.\n\nIt definitely passes CI :-). But special care is taken to handle this\ncase, and it works for a couple of reasons. Notably that we only call\n`write_midx_included_packs()` (which in turn invokes the\nmulti-pack-index builtin as a sub-process) *after* repack has called\nclose_object_store(). That means that `repack` won't be holding the MIDX\nopen while the sub-process is trying to write a new one in its place.\n\nThe other reason is that the multi-pack-index process also makes sure to\nclose the old MIDX before writing the new one, so we can be certain that\nneither of these spots are mapping the file or have it opened when\ntrying to write over it.\n\nThanks,\nTaylor\n"},{"id":"436818","messageId":"YUvhp6EVPEPwuKZ4@nand.local","threadId":"56482","inReplyTo":"YUuvjZl43k3G2SBn@google.com","subject":"Re: [PATCH v2 3/8] midx: preliminary support for `--refs-snapshot`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-23T02:08:39Z","receivedAt":"2021-09-23T02:08:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 22, 2021 at 03:34:53PM -0700, Josh Steadmon wrote:\n> > +\t\tobject = parse_object_or_die(&oid, NULL);\n> > +\t\tif (preferred)\n> > +\t\t\tobject->flags |= NEEDS_BITMAP;\n> > +\n> > +\t\tadd_pending_object(revs, object, \"\");\n> > +\t}\n> > +\n> > +\tfclose(f);\n> > +\treturn 0;\n> > +}\n>\n> `buf` needs to be released here.\n\nGood eyes; thanks for noticing :-).\n\nThanks,\nTaylor\n"},{"id":"436820","messageId":"YUvj33QX5fvjNoiJ@nand.local","threadId":"56482","inReplyTo":"20210922230012.2608849-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 3/8] midx: preliminary support for `--refs-snapshot`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-23T02:18:07Z","receivedAt":"2021-09-23T02:18:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 22, 2021 at 04:00:12PM -0700, Jonathan Tan wrote:\n> > This approach will cause a problem when multi-pack bitmaps are able to\n> > be generated from `git repack`, since the reference tips can change\n> > during the repack. Even though we ignore commits that don't exist in\n> > the MIDX (when doing a scan of the ref tips), it's possible that a\n> > commit in the MIDX reaches something that isn't.\n> >\n> > This can happen when a multi-pack index contains some pack which refers\n> > to loose objects (which by definition aren't included in the multi-pack\n> > index).\n> >\n> > By taking a snapshot of the references before we start repacking, we can\n> > close that race window.\n>\n> I can understand why we want the refs to remain the same both for the\n> MIDX generation and the MIDX bitmap generation (one reason that comes to\n> mind is how we select the commits for which to generate bitmaps), but I\n> don't understand what referring to loose objects has to do with it. I\n> think that using the same set of refs for MIDX generation and bitmap\n> generation is a good enough reason to do this, and we don't need to\n> mention loose objects.\n\nThe point there is that a pack which contains objects that are ancestors\nof loose objects can show up at any time, including just before we\nselect which packs to go into a MIDX.\n\nThis is particularly common at GitHub, where all of our test-merges and\nobjects created via the web interface are written loose. So we could\ncompute a test merge, store the result loose, and have somebody push a\npack up on top of it.\n\nIf that pack shows up after the repack, but before we write a MIDX, then\nwithout the refs-snapshot code, we would include that pack, select its\ncommits as candidates for bitmap selection, and then ultimately discover\nthat the bitmap isn't closed, since the loose object won't be included.\n\nBut that may be a little into the weeds for the patch message. Anyway,\nI did have to think about it for a minute, but I'm pretty sure that's\nwhat I was thinking when I wrote this.\n\nThanks,\nTaylor\n"},{"id":"436822","messageId":"YUvpI3vL8iIidMR+@nand.local","threadId":"56482","inReplyTo":"20210922223936.2606010-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 6/8] builtin/repack.c: support writing a MIDX while repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-23T02:40:35Z","receivedAt":"2021-09-23T02:40:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 22, 2021 at 03:39:35PM -0700, Jonathan Tan wrote:\n> > @@ -683,22 +755,41 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n> >  \t}\n> >  \t/* End of pack replacement. */\n> >\n> > +\tif (delete_redundant && pack_everything & ALL_INTO_ONE) {\n> > +\t\tconst int hexsz = the_hash_algo->hexsz;\n> > +\t\tstring_list_sort(&names);\n> > +\t\tfor_each_string_list_item(item, &existing_packs) {\n> > +\t\t\tchar *sha1;\n> > +\t\t\tsize_t len = strlen(item->string);\n> > +\t\t\tif (len < hexsz)\n> > +\t\t\t\tcontinue;\n> > +\t\t\tsha1 = item->string + len - hexsz;\n> > +\t\t\titem->util = (void*)(intptr_t)!string_list_has_string(&names, sha1);\n>\n> OK, here is the tricky part. They are marked for deletion here...\n\nYep, instead of deleting the packs immediately, which would cause an\nexisting MIDX referencing them to become corrupt, we split that into two\nphases: one to figure out which packs need to be deleted (after the new\nMIDX), and then another to actually delete those packs.\n\n> > +\t\t}\n> > +\t}\n> > +\n> > +\tif (write_midx) {\n> > +\t\tstruct string_list include = STRING_LIST_INIT_NODUP;\n> > +\t\tmidx_included_packs(&include, &existing_packs,\n> > +\t\t\t\t    &existing_kept_packs, &names, geometry);\n>\n> ...the mark for deletion is taken into account during the execution of\n> midx_included_packs() (as can be seen by looking at that function)...\n>\n> > +\n> > +\t\tret = write_midx_included_packs(&include,\n> > +\t\t\t\t\t\tshow_progress, write_bitmaps > 0);\n> > +\n> > +\t\tstring_list_clear(&include, 0);\n> > +\n> > +\t\tif (ret)\n> > +\t\t\treturn ret;\n> > +\t}\n> > +\n> >  \treprepare_packed_git(the_repository);\n> >\n> >  \tif (delete_redundant) {\n> >  \t\tint opts = 0;\n> > -\t\tif (pack_everything & ALL_INTO_ONE) {\n> > -\t\t\tconst int hexsz = the_hash_algo->hexsz;\n> > -\t\t\tstring_list_sort(&names);\n> > -\t\t\tfor_each_string_list_item(item, &existing_packs) {\n> > -\t\t\t\tchar *sha1;\n> > -\t\t\t\tsize_t len = strlen(item->string);\n> > -\t\t\t\tif (len < hexsz)\n> > -\t\t\t\t\tcontinue;\n> > -\t\t\t\tsha1 = item->string + len - hexsz;\n> > -\t\t\t\tif (!string_list_has_string(&names, sha1))\n> > -\t\t\t\t\tremove_redundant_pack(packdir, item->string);\n> > -\t\t\t}\n> > +\t\tfor_each_string_list_item(item, &existing_packs) {\n> > +\t\t\tif (!item->util)\n> > +\t\t\t\tcontinue;\n>\n> ...and the marks are also used here. I was at first confused about why\n> the functionality of midx_included_packs() depended on whether redundant\n> packs were marked for deletion - if they are redundant, shouldn't they\n> never be taken into account (regardless of whether we actually delete\n> them)? But I guess it makes sense as an overall design point that we\n> pass all packs that are to remain (so if they will be deleted, exclude\n> them, and if they will not be, include them).\n\nRight; midx_included_packs() is used to determine which packs are going\nto be written into the new MIDX, and we need to make sure that we\nexclude any packs which are about to be deleted.\n\n> I think a comment \"mark this pack for deletion\" at the point we write\n> the mark (so, where the cast to intptr_t is) is worthwhile. Other than\n> that, this patch looks good to me.\n\nSure, that seems like it would be helpful to other readers.\n\n> I (and the others in our review club) only managed to reach this patch.\n> I hope to get to the other 2 by the end of the week.\n\nThanks for reviewing, I look forward to more of your feedback.\n\nThanks,\nTaylor\n"},{"id":"436823","messageId":"YUvtrTF67FW2UGhC@nand.local","threadId":"56482","inReplyTo":"20210922225605.2608085-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 4/8] builtin/repack.c: keep track of existing packs unconditionally","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-23T02:59:57Z","receivedAt":"2021-09-23T03:00:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 22, 2021 at 03:56:05PM -0700, Jonathan Tan wrote:\n> > @@ -98,8 +98,9 @@ static void remove_pack_on_signal(int signo)\n> >   * have a corresponding .keep file. These packs are not to\n> >   * be kept if we are going to pack everything into one file.\n> >   */\n> > -static void get_non_kept_pack_filenames(struct string_list *fname_list,\n> > -\t\t\t\t\tconst struct string_list *extra_keep)\n> > +static void collect_pack_filenames(struct string_list *fname_list,\n> > +\t\t\t\t   struct string_list *fname_kept_list,\n> > +\t\t\t\t   const struct string_list *extra_keep)\n> >  {\n> >  \tDIR *dir;\n> >  \tstruct dirent *e;\n>\n> The comment in the before-context of this hunk needs to be updated.\n\nThanks, updated. Now it reads like this:\n\n  /*\n   * Adds all packs hex strings to either the fname or fname_kept_list\n   * list, based on whether each pack has a corresponding .keep file or\n   * not. Packs without a .keep file are not to be kept if we are going\n   * to pack everything into one file.\n   */\n\n> Also, I think that fname_list should be renamed to fname_nonkept_list.\n> It does have the same semantics as before, but the name of the function\n> has changed. And also, fname_list sounds like a superset of\n> fname_kept_list, but that is not the case.\n\nYeah, I think that is a reasonable suggestion. I'll do it in a separate\npatch, though, to keep the renaming self-contained.\n\nThanks,\nTaylor\n"},{"id":"436824","messageId":"YUvv/HG3jzOwtwzZ@nand.local","threadId":"56482","inReplyTo":"20210922231423.2610917-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 1/8] midx: expose `write_midx_file_only()` publicly","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-23T03:09:48Z","receivedAt":"2021-09-23T03:09:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 22, 2021 at 04:14:23PM -0700, Jonathan Tan wrote:\n> > @@ -1237,7 +1253,7 @@ static int write_midx_internal(const char *object_dir,\n> >\n> >  \tQSORT(ctx.info, ctx.nr, pack_info_compare);\n> >\n> > -\tif (packs_to_drop && packs_to_drop->nr) {\n> > +\tif (ctx.m && packs_to_drop && packs_to_drop->nr) {\n> >  \t\tint drop_index = 0;\n> >  \t\tint missing_drops = 0;\n> >\n>\n> I couldn't figure out why this requires ctx.m now.\n\nMe either; this must have been a stray change that got dragged along. I\ndropped in -- thanks for pointing it out.\n\n> > @@ -62,6 +63,10 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n> >  int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n> >\n> >  int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n> > +int write_midx_file_only(const char *object_dir,\n> > +\t\t\t struct string_list *packs_to_include,\n> > +\t\t\t const char *preferred_pack_name,\n> > +\t\t\t unsigned flags);\n>\n> It took me a while to figure out that this function doesn't only write a\n> MIDX file, but writes an MIDX file only for certain packs. Maybe worth\n> adding a comment here (e.g. \"Write an MIDX file only for the given\n> packs\").\n\nNitpicking a little, it does still write a (single) MIDX file, but that\nMIDX only includes the packs listed in packs_to_include. I can add a\ncomment to that effect.\n\nThanks,\nTaylor\n"},{"id":"436996","messageId":"20210924182247.2922561-1-jonathantanmy@google.com","threadId":"56482","inReplyTo":"eb24b308ecdbd7736fc63927130540abd0a668db.1631730270.git.me@ttaylorr.com","subject":"Re: [PATCH v2 8/8] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-24T18:22:47Z","receivedAt":"2021-09-24T18:22:52Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> +static void midx_snapshot_refs(struct tempfile *f)\n> +{\n> +\tstruct midx_snapshot_ref_data data;\n> +\tconst struct string_list *preferred = bitmap_preferred_tips(the_repository);\n> +\n> +\tdata.f = f;\n> +\toidset_init(&data.seen, 0);\n> +\n> +\tif (!fdopen_tempfile(f, \"w\"))\n> +\t\t die(_(\"could not open tempfile %s for writing\"),\n> +\t\t     get_tempfile_path(f));\n> +\n> +\tif (preferred) {\n> +\t\tstruct string_list_item *item;\n> +\n> +\t\tdata.preferred = 1;\n> +\t\tfor_each_string_list_item(item, preferred)\n> +\t\t\tfor_each_ref_in(item->string, midx_snapshot_ref_one, &data);\n> +\t\tdata.preferred = 0;\n> +\t}\n> +\n> +\tfor_each_ref(midx_snapshot_ref_one, &data);\n\nOne small thing here - I think \"data.preferred = 0;\" needs to be moved\nto right above this for_each_ref line. As it is, if preferred is NULL,\ndata.preferred wouldn't be initialized.\n\nOther than that (and the minor changes I've suggested in previous\npatches), this series looks good to me.\n"},{"id":"437412","messageId":"cover.1632880469.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1631730270.git.me@ttaylorr.com","subject":"[PATCH v3 0/9] repack: introduce `--write-midx`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-29T01:54:37Z","receivedAt":"2021-09-29T01:55:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is another small reroll of my series to implement the final component of\nmulti-pack reachability bitamps, which is to be able to write them from `git\nrepack`.\n\nThis version incorporates feedback from Jonathan Tan and others at Google who\ngraciously provided review. A range-diff is included below, but the major\nchanges are:\n\n  - A comment to explain the relationship between 'ctx->m' and 'ctx->to_include'\n    in midx.c:add_pack_to_midx().\n\n  - A comment to explain the meaning of write_midx_file_only().\n\n  - Clean-up of stray hunks, a missing strbuf_release().\n\n  - Replacing the name `existing_packs` with `existing_nonkept_packs` to\n    emphasize the relationship between it and the similarly-named\n    `existing_kept_packs`.\n\nInstead of depending on tb/multi-pack-bitmaps, this series now depends on the\ntip of master.\n\nTaylor Blau (9):\n  midx: expose `write_midx_file_only()` publicly\n  builtin/multi-pack-index.c: support `--stdin-packs` mode\n  midx: preliminary support for `--refs-snapshot`\n  builtin/repack.c: keep track of existing packs unconditionally\n  builtin/repack.c: rename variables that deal with non-kept packs\n  builtin/repack.c: extract showing progress to a variable\n  builtin/repack.c: support writing a MIDX while repacking\n  builtin/repack.c: make largest pack preferred\n  builtin/repack.c: pass `--refs-snapshot` when writing bitmaps\n\n Documentation/git-multi-pack-index.txt |  19 ++\n Documentation/git-repack.txt           |  18 +-\n builtin/multi-pack-index.c             |  36 +++-\n builtin/repack.c                       | 279 ++++++++++++++++++++++---\n midx.c                                 | 110 ++++++++--\n midx.h                                 |  15 +-\n pack-bitmap.c                          |   2 +-\n pack-bitmap.h                          |   1 +\n t/helper/test-read-midx.c              |  25 ++-\n t/lib-midx.sh                          |   8 +\n t/t5319-multi-pack-index.sh            |  15 ++\n t/t5326-multi-pack-bitmaps.sh          |  82 ++++++++\n t/t7700-repack.sh                      |  96 +++++++++\n t/t7703-repack-geometric.sh            |  24 ++-\n 14 files changed, 674 insertions(+), 56 deletions(-)\n create mode 100644 t/lib-midx.sh\n\nRange-diff against v2:\n 1:  03c1b2c4d3 !  1:  b7c72d0bf5 midx: expose `write_midx_file_only()` publicly\n    @@ midx.c: struct write_midx_context {\n      \n      static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n     @@ midx.c: static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n    + \n    + \tif (ends_with(file_name, \".idx\")) {\n      \t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n    ++\t\t/*\n    ++\t\t * Note that at most one of ctx->m and ctx->to_include are set,\n    ++\t\t * so we are testing midx_contains_pack() and\n    ++\t\t * string_list_has_string() independently (guarded by the\n    ++\t\t * appropriate NULL checks).\n    ++\t\t *\n    ++\t\t * We could support passing to_include while reusing an existing\n    ++\t\t * MIDX, but don't currently since the reuse process drags\n    ++\t\t * forward all packs from an existing MIDX (without checking\n    ++\t\t * whether or not they appear in the to_include list).\n    ++\t\t *\n    ++\t\t * If we added support for that, these next two conditional\n    ++\t\t * should be performed independently (likely checking\n    ++\t\t * to_include before the existing MIDX).\n    ++\t\t */\n      \t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n      \t\t\treturn;\n     +\t\telse if (ctx->to_include &&\n    @@ midx.c: static int write_midx_internal(const char *object_dir,\n      \t\tstruct bitmap_index *bitmap_git;\n      \t\tint bitmap_exists;\n      \t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n    -@@ midx.c: static int write_midx_internal(const char *object_dir,\n    - \n    - \tQSORT(ctx.info, ctx.nr, pack_info_compare);\n    - \n    --\tif (packs_to_drop && packs_to_drop->nr) {\n    -+\tif (ctx.m && packs_to_drop && packs_to_drop->nr) {\n    - \t\tint drop_index = 0;\n    - \t\tint missing_drops = 0;\n    - \n     @@ midx.c: int write_midx_file(const char *object_dir,\n      \t\t    const char *preferred_pack_name,\n      \t\t    unsigned flags)\n    @@ midx.h: int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pa\n      int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n      \n      int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n    ++/*\n    ++ * Variant of write_midx_file which writes a MIDX containing only the packs\n    ++ * specified in packs_to_include.\n    ++ */\n     +int write_midx_file_only(const char *object_dir,\n     +\t\t\t struct string_list *packs_to_include,\n     +\t\t\t const char *preferred_pack_name,\n 2:  59556e5545 =  2:  986ef14f2a builtin/multi-pack-index.c: support `--stdin-packs` mode\n 3:  42f1ae9ede !  3:  4e3769a4f3 midx: preliminary support for `--refs-snapshot`\n    @@ Commit message\n         commit in the MIDX reaches something that isn't.\n     \n         This can happen when a multi-pack index contains some pack which refers\n    -    to loose objects (which by definition aren't included in the multi-pack\n    -    index).\n    +    to loose objects (e.g., if a pack was pushed after starting the repack\n    +    but before generating the MIDX which depends on an object which is\n    +    stored as loose in the repository, and by definition isn't included in\n    +    the multi-pack index).\n     \n         By taking a snapshot of the references before we start repacking, we can\n         close that race window. In the above scenario (where we have a packed\n    @@ midx.c: static void bitmap_show_commit(struct commit *commit, void *_data)\n     +\t}\n     +\n     +\tfclose(f);\n    ++\tstrbuf_release(&buf);\n     +\treturn 0;\n     +}\n     +\n    @@ midx.h: int fill_midx_entry(struct repository *r, const struct object_id *oid, s\n      int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n      \n     -int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n    + /*\n    +  * Variant of write_midx_file which writes a MIDX containing only the packs\n    +  * specified in packs_to_include.\n    +  */\n     +int write_midx_file(const char *object_dir,\n     +\t\t    const char *preferred_pack_name,\n     +\t\t    const char *refs_snapshot,\n 4:  c0d045a9de !  4:  1b3dd331ca builtin/repack.c: keep track of existing packs unconditionally\n    @@ Commit message\n     \n      ## builtin/repack.c ##\n     @@ builtin/repack.c: static void remove_pack_on_signal(int signo)\n    -  * have a corresponding .keep file. These packs are not to\n    -  * be kept if we are going to pack everything into one file.\n    + }\n    + \n    + /*\n    +- * Adds all packs hex strings to the fname list, which do not\n    +- * have a corresponding .keep file. These packs are not to\n    +- * be kept if we are going to pack everything into one file.\n    ++ * Adds all packs hex strings to either fname_list or fname_kept_list\n    ++ * based on whether each pack has a corresponding .keep file or not.\n    ++ * Packs without a .keep file are not to be kept if we are going to\n    ++ * pack everything into one file.\n       */\n     -static void get_non_kept_pack_filenames(struct string_list *fname_list,\n     -\t\t\t\t\tconst struct string_list *extra_keep)\n -:  ---------- >  5:  15831a201a builtin/repack.c: rename variables that deal with non-kept packs\n 5:  09de03de47 =  6:  1a40161baf builtin/repack.c: extract showing progress to a variable\n 6:  83dfdb8b12 !  7:  6854f0751d builtin/repack.c: support writing a MIDX while repacking\n    @@ Commit message\n         check we can make consistently when (1) telling the MIDX which packs we\n         want to exclude, and (2) actually unlinking the redundant packs.\n     \n    +    There is also a tiny change to short-circuit the body of\n    +    write_midx_included_packs() when no packs remain in the case of an empty\n    +    repository. The MIDX code does not handle this, so avoid trying to\n    +    generate a MIDX covering zero packs in the first place.\n    +\n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n     \n      ## Documentation/git-repack.txt ##\n    @@ builtin/repack.c: static void clear_pack_geometry(struct pack_geometry *geometry\n      }\n      \n     +static void midx_included_packs(struct string_list *include,\n    -+\t\t\t\tstruct string_list *existing_packs,\n    ++\t\t\t\tstruct string_list *existing_nonkept_packs,\n     +\t\t\t\tstruct string_list *existing_kept_packs,\n     +\t\t\t\tstruct string_list *names,\n     +\t\t\t\tstruct pack_geometry *geometry)\n    @@ builtin/repack.c: static void clear_pack_geometry(struct pack_geometry *geometry\n     +\t\t\tstring_list_insert(include, strbuf_detach(&buf, NULL));\n     +\t\t}\n     +\t} else {\n    -+\t\tfor_each_string_list_item(item, existing_packs) {\n    ++\t\tfor_each_string_list_item(item, existing_nonkept_packs) {\n     +\t\t\tif (item->util)\n     +\t\t\t\tcontinue;\n     +\t\t\tstring_list_insert(include, xstrfmt(\"%s.idx\", item->string));\n    @@ builtin/repack.c: static void clear_pack_geometry(struct pack_geometry *geometry\n     +\tFILE *in;\n     +\tint ret;\n     +\n    ++\tif (!include->nr)\n    ++\t\treturn 0;\n    ++\n     +\tcmd.in = -1;\n     +\tcmd.git_cmd = 1;\n     +\n    @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n     +\tif (delete_redundant && pack_everything & ALL_INTO_ONE) {\n     +\t\tconst int hexsz = the_hash_algo->hexsz;\n     +\t\tstring_list_sort(&names);\n    -+\t\tfor_each_string_list_item(item, &existing_packs) {\n    ++\t\tfor_each_string_list_item(item, &existing_nonkept_packs) {\n     +\t\t\tchar *sha1;\n     +\t\t\tsize_t len = strlen(item->string);\n     +\t\t\tif (len < hexsz)\n     +\t\t\t\tcontinue;\n     +\t\t\tsha1 = item->string + len - hexsz;\n    ++\t\t\t/*\n    ++\t\t\t * Mark this pack for deletion, which ensures that this\n    ++\t\t\t * pack won't be included in a MIDX (if `--write-midx`\n    ++\t\t\t * was given) and that we will actually delete this pack\n    ++\t\t\t * (if `-d` was given).\n    ++\t\t\t */\n     +\t\t\titem->util = (void*)(intptr_t)!string_list_has_string(&names, sha1);\n     +\t\t}\n     +\t}\n     +\n     +\tif (write_midx) {\n     +\t\tstruct string_list include = STRING_LIST_INIT_NODUP;\n    -+\t\tmidx_included_packs(&include, &existing_packs,\n    ++\t\tmidx_included_packs(&include, &existing_nonkept_packs,\n     +\t\t\t\t    &existing_kept_packs, &names, geometry);\n     +\n     +\t\tret = write_midx_included_packs(&include,\n    @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n     -\t\tif (pack_everything & ALL_INTO_ONE) {\n     -\t\t\tconst int hexsz = the_hash_algo->hexsz;\n     -\t\t\tstring_list_sort(&names);\n    --\t\t\tfor_each_string_list_item(item, &existing_packs) {\n    +-\t\t\tfor_each_string_list_item(item, &existing_nonkept_packs) {\n     -\t\t\t\tchar *sha1;\n     -\t\t\t\tsize_t len = strlen(item->string);\n     -\t\t\t\tif (len < hexsz)\n    @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix\n     -\t\t\t\tif (!string_list_has_string(&names, sha1))\n     -\t\t\t\t\tremove_redundant_pack(packdir, item->string);\n     -\t\t\t}\n    -+\t\tfor_each_string_list_item(item, &existing_packs) {\n    ++\t\tfor_each_string_list_item(item, &existing_nonkept_packs) {\n     +\t\t\tif (!item->util)\n     +\t\t\t\tcontinue;\n     +\t\t\tremove_redundant_pack(packdir, item->string);\n    @@ t/t7700-repack.sh: test_expect_success 'auto-bitmaps do not complain if unavaila\n     +'\n     +\n      test_done\n    +\n    + ## t/t7703-repack-geometric.sh ##\n    +@@ t/t7703-repack-geometric.sh: test_expect_success '--geometric with no packs' '\n    + \t(\n    + \t\tcd geometric &&\n    + \n    +-\t\tgit repack --geometric 2 >out &&\n    ++\t\tgit repack --write-midx --geometric 2 >out &&\n    + \t\ttest_i18ngrep \"Nothing new to pack\" out\n    + \t)\n    + '\n 7:  68bc49d8ae !  8:  3596c76daf builtin/repack.c: make largest pack preferred\n    @@ builtin/repack.c: static int write_midx_included_packs(struct string_list *inclu\n      \tif (ret)\n      \t\treturn ret;\n     @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix)\n    - \t\tmidx_included_packs(&include, &existing_packs,\n    + \t\tmidx_included_packs(&include, &existing_nonkept_packs,\n      \t\t\t\t    &existing_kept_packs, &names, geometry);\n      \n     -\t\tret = write_midx_included_packs(&include,\n 8:  eb24b308ec !  9:  d99f075321 builtin/repack.c: pass `--refs-snapshot` when writing bitmaps\n    @@ builtin/repack.c: static void clear_pack_geometry(struct pack_geometry *geometry\n     +}\n     +\n      static void midx_included_packs(struct string_list *include,\n    - \t\t\t\tstruct string_list *existing_packs,\n    + \t\t\t\tstruct string_list *existing_nonkept_packs,\n      \t\t\t\tstruct string_list *existing_kept_packs,\n     @@ builtin/repack.c: static void midx_included_packs(struct string_list *include,\n      \n-- \n2.33.0.96.g73915697e6\n"},{"id":"437413","messageId":"b7c72d0bf5b9bbe2d4f082ddcbf65f33c10d584b.1632880469.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1632880469.git.me@ttaylorr.com","subject":"[PATCH v3 1/9] midx: expose `write_midx_file_only()` publicly","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-29T01:55:01Z","receivedAt":"2021-09-29T01:55:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Expose a variant of the write_midx_file() function which ignores packs\nthat aren't included in an explicit \"allow\" list.\n\nThis will be used in an upcoming patch to power a new `--stdin-packs`\nmode of `git multi-pack-index write` for callers that only want to\ninclude certain packs in a MIDX (and ignore any packs which may have\nhappened to enter the repository independently, e.g., from pushes).\n\nThose patches will provide test coverage for this new function.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++--------\n midx.h |  9 +++++++++\n 2 files changed, 58 insertions(+), 8 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex f96fb2efee..7ac97e66e0 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -460,6 +460,8 @@ struct write_midx_context {\n \tuint32_t num_large_offsets;\n \n \tint preferred_pack_idx;\n+\n+\tstruct string_list *to_include;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -469,8 +471,26 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n \n \tif (ends_with(file_name, \".idx\")) {\n \t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n+\t\t/*\n+\t\t * Note that at most one of ctx->m and ctx->to_include are set,\n+\t\t * so we are testing midx_contains_pack() and\n+\t\t * string_list_has_string() independently (guarded by the\n+\t\t * appropriate NULL checks).\n+\t\t *\n+\t\t * We could support passing to_include while reusing an existing\n+\t\t * MIDX, but don't currently since the reuse process drags\n+\t\t * forward all packs from an existing MIDX (without checking\n+\t\t * whether or not they appear in the to_include list).\n+\t\t *\n+\t\t * If we added support for that, these next two conditional\n+\t\t * should be performed independently (likely checking\n+\t\t * to_include before the existing MIDX).\n+\t\t */\n \t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n \t\t\treturn;\n+\t\telse if (ctx->to_include &&\n+\t\t\t !string_list_has_string(ctx->to_include, file_name))\n+\t\t\treturn;\n \n \t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n \n@@ -1043,6 +1063,7 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n }\n \n static int write_midx_internal(const char *object_dir,\n+\t\t\t       struct string_list *packs_to_include,\n \t\t\t       struct string_list *packs_to_drop,\n \t\t\t       const char *preferred_pack_name,\n \t\t\t       unsigned flags)\n@@ -1067,10 +1088,17 @@ static int write_midx_internal(const char *object_dir,\n \t\tdie_errno(_(\"unable to create leading directories of %s\"),\n \t\t\t  midx_name);\n \n-\tfor (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {\n-\t\tif (!strcmp(object_dir, cur->object_dir)) {\n-\t\t\tctx.m = cur;\n-\t\t\tbreak;\n+\tif (!packs_to_include) {\n+\t\t/*\n+\t\t * Only reference an existing MIDX when not filtering which\n+\t\t * packs to include, since all packs and objects are copied\n+\t\t * blindly from an existing MIDX if one is present.\n+\t\t */\n+\t\tfor (cur = get_multi_pack_index(the_repository); cur; cur = cur->next) {\n+\t\t\tif (!strcmp(object_dir, cur->object_dir)) {\n+\t\t\t\tctx.m = cur;\n+\t\t\t\tbreak;\n+\t\t\t}\n \t\t}\n \t}\n \n@@ -1121,10 +1149,13 @@ static int write_midx_internal(const char *object_dir,\n \telse\n \t\tctx.progress = NULL;\n \n+\tctx.to_include = packs_to_include;\n+\n \tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n \tstop_progress(&ctx.progress);\n \n-\tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop) {\n+\tif ((ctx.m && ctx.nr == ctx.m->num_packs) &&\n+\t    !(packs_to_include || packs_to_drop)) {\n \t\tstruct bitmap_index *bitmap_git;\n \t\tint bitmap_exists;\n \t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n@@ -1365,7 +1396,17 @@ int write_midx_file(const char *object_dir,\n \t\t    const char *preferred_pack_name,\n \t\t    unsigned flags)\n {\n-\treturn write_midx_internal(object_dir, NULL, preferred_pack_name, flags);\n+\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n+\t\t\t\t   flags);\n+}\n+\n+int write_midx_file_only(const char *object_dir,\n+\t\t\t struct string_list *packs_to_include,\n+\t\t\t const char *preferred_pack_name,\n+\t\t\t unsigned flags)\n+{\n+\treturn write_midx_internal(object_dir, packs_to_include, NULL,\n+\t\t\t\t   preferred_pack_name, flags);\n }\n \n struct clear_midx_data {\n@@ -1645,7 +1686,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tfree(count);\n \n \tif (packs_to_drop.nr) {\n-\t\tresult = write_midx_internal(object_dir, &packs_to_drop, NULL, flags);\n+\t\tresult = write_midx_internal(object_dir, NULL, &packs_to_drop, NULL, flags);\n \t\tm = NULL;\n \t}\n \n@@ -1836,7 +1877,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t\tgoto cleanup;\n \t}\n \n-\tresult = write_midx_internal(object_dir, NULL, NULL, flags);\n+\tresult = write_midx_internal(object_dir, NULL, NULL, NULL, flags);\n \tm = NULL;\n \n cleanup:\ndiff --git a/midx.h b/midx.h\nindex aa3da557bb..3545e327ea 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -2,6 +2,7 @@\n #define MIDX_H\n \n #include \"repository.h\"\n+#include \"string-list.h\"\n \n struct object_id;\n struct pack_entry;\n@@ -62,6 +63,14 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n \n int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n+/*\n+ * Variant of write_midx_file which writes a MIDX containing only the packs\n+ * specified in packs_to_include.\n+ */\n+int write_midx_file_only(const char *object_dir,\n+\t\t\t struct string_list *packs_to_include,\n+\t\t\t const char *preferred_pack_name,\n+\t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);\n int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"437414","messageId":"986ef14f2af3b137b2a67806f3bf96585dbc27f3.1632880469.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1632880469.git.me@ttaylorr.com","subject":"[PATCH v3 2/9] builtin/multi-pack-index.c: support `--stdin-packs` mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-29T01:55:04Z","receivedAt":"2021-09-29T01:55:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To power a new `--write-midx` mode, `git repack` will want to write a\nmulti-pack index containing a certain set of packs in the repository.\n\nThis new option will be used by `git repack` to write a MIDX which\ncontains only the packs which will survive after the repack (that is, it\nwill exclude any packs which are about to be deleted).\n\nThis patch effectively exposes the function implemented in the previous\ncommit via the `git multi-pack-index` builtin. An alternative approach\nwould have been to call that function from the `git repack` builtin\ndirectly, but this introduces awkward problems around closing and\nreopening the object store, so the MIDX will be written out-of-process.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-multi-pack-index.txt |  4 ++++\n builtin/multi-pack-index.c             | 27 ++++++++++++++++++++++++++\n t/t5319-multi-pack-index.sh            | 15 ++++++++++++++\n 3 files changed, 46 insertions(+)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex a9df3dbd32..009c989ef8 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -45,6 +45,10 @@ write::\n \n \t--[no-]bitmap::\n \t\tControl whether or not a multi-pack bitmap is written.\n+\n+\t--stdin-packs::\n+\t\tWrite a multi-pack index containing only the set of\n+\t\tline-delimited pack index basenames provided over stdin.\n --\n \n verify::\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 66de6efd41..03aaf8e7fb 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -47,6 +47,7 @@ static struct opts_multi_pack_index {\n \tconst char *preferred_pack;\n \tunsigned long batch_size;\n \tunsigned flags;\n+\tint stdin_packs;\n } opts;\n \n static struct option common_opts[] = {\n@@ -61,6 +62,16 @@ static struct option *add_common_options(struct option *prev)\n \treturn parse_options_concat(common_opts, prev);\n }\n \n+static void read_packs_from_stdin(struct string_list *to)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\twhile (strbuf_getline(&buf, stdin) != EOF)\n+\t\tstring_list_append(to, buf.buf);\n+\tstring_list_sort(to);\n+\n+\tstrbuf_release(&buf);\n+}\n+\n static int cmd_multi_pack_index_write(int argc, const char **argv)\n {\n \tstruct option *options;\n@@ -70,6 +81,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\t\t   N_(\"pack for reuse when computing a multi-pack bitmap\")),\n \t\tOPT_BIT(0, \"bitmap\", &opts.flags, N_(\"write multi-pack bitmap\"),\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n+\t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n+\t\t\t N_(\"write multi-pack index containing only given indexes\")),\n \t\tOPT_END(),\n \t};\n \n@@ -86,6 +99,20 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \n \tFREE_AND_NULL(options);\n \n+\tif (opts.stdin_packs) {\n+\t\tstruct string_list packs = STRING_LIST_INIT_DUP;\n+\t\tint ret;\n+\n+\t\tread_packs_from_stdin(&packs);\n+\n+\t\tret = write_midx_file_only(opts.object_dir, &packs,\n+\t\t\t\t\t   opts.preferred_pack, opts.flags);\n+\n+\t\tstring_list_clear(&packs, 0);\n+\n+\t\treturn ret;\n+\n+\t}\n \treturn write_midx_file(opts.object_dir, opts.preferred_pack,\n \t\t\t       opts.flags);\n }\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex bb04f0f23b..385f0a3efd 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -168,6 +168,21 @@ test_expect_success 'write midx with two packs' '\n \n compare_results_with_midx \"two packs\"\n \n+test_expect_success 'write midx with --stdin-packs' '\n+\trm -fr $objdir/pack/multi-pack-index &&\n+\n+\tidx=\"$(find $objdir/pack -name \"test-2-*.idx\")\" &&\n+\tbasename \"$idx\" >in &&\n+\n+\tgit multi-pack-index write --stdin-packs <in &&\n+\n+\ttest-tool read-midx $objdir | grep \"\\.idx$\" >packs &&\n+\n+\ttest_cmp packs in\n+'\n+\n+compare_results_with_midx \"mixed mode (one pack + extra)\"\n+\n test_expect_success 'write progress off for redirected stderr' '\n \tgit multi-pack-index --object-dir=$objdir write 2>err &&\n \ttest_line_count = 0 err\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"437415","messageId":"4e3769a4f39f78788be975adcb5b3b57143df2c7.1632880469.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1632880469.git.me@ttaylorr.com","subject":"[PATCH v3 3/9] midx: preliminary support for `--refs-snapshot`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-29T01:55:07Z","receivedAt":"2021-09-29T01:55:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To figure out which commits we can write a bitmap for, the multi-pack\nindex/bitmap code does a reachability traversal, marking any commit\nwhich can be found in the MIDX as eligible to receive a bitmap.\n\nThis approach will cause a problem when multi-pack bitmaps are able to\nbe generated from `git repack`, since the reference tips can change\nduring the repack. Even though we ignore commits that don't exist in\nthe MIDX (when doing a scan of the ref tips), it's possible that a\ncommit in the MIDX reaches something that isn't.\n\nThis can happen when a multi-pack index contains some pack which refers\nto loose objects (e.g., if a pack was pushed after starting the repack\nbut before generating the MIDX which depends on an object which is\nstored as loose in the repository, and by definition isn't included in\nthe multi-pack index).\n\nBy taking a snapshot of the references before we start repacking, we can\nclose that race window. In the above scenario (where we have a packed\nobject pointing at a loose one), we'll either (a) take a snapshot of the\nreferences before seeing the packed one, or (b) take it after, at which\npoint we can guarantee that the loose object will be packed and included\nin the MIDX.\n\nThis patch does just that. It writes a temporary \"reference snapshot\",\nwhich is a list of OIDs that are at the ref tips before writing a\nmulti-pack bitmap. References that are \"preferred\" (i.e,. are a suffix\nof at least one value of the 'pack.preferBitmapTips' configuration) are\nmarked with a special '+'.\n\nThe format is simple: one line per commit at each tip, with an optional\n'+' at the beginning (for preferred references, as described above).\n\nWhen provided, the reference snapshot is used to drive bitmap selection\ninstead of the MIDX code doing its own traversal. When it isn't\nprovided, the usual traversal takes place instead.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-multi-pack-index.txt | 15 +++++\n builtin/multi-pack-index.c             | 11 +++-\n builtin/repack.c                       |  2 +-\n midx.c                                 | 61 ++++++++++++++++---\n midx.h                                 |  6 +-\n t/t5326-multi-pack-bitmaps.sh          | 82 ++++++++++++++++++++++++++\n 6 files changed, 164 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex 009c989ef8..27f83932e4 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -49,6 +49,21 @@ write::\n \t--stdin-packs::\n \t\tWrite a multi-pack index containing only the set of\n \t\tline-delimited pack index basenames provided over stdin.\n+\n+\t--refs-snapshot=<path>::\n+\t\tWith `--bitmap`, optionally specify a file which\n+\t\tcontains a \"refs snapshot\" taken prior to repacking.\n++\n+A reference snapshot is composed of line-delimited OIDs corresponding to\n+the reference tips, usually taken by `git repack` prior to generating a\n+new pack. A line may optionally start with a `+` character to indicate\n+that the reference which corresponds to that OID is \"preferred\" (see\n+linkgit:git-config[1]'s `pack.preferBitmapTips`.)\n++\n+The file given at `<path>` is expected to be readable, and can contain\n+duplicates. (If a given OID is given more than once, it is marked as\n+preferred if at least one instance of it begins with the special `+`\n+marker).\n --\n \n verify::\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 03aaf8e7fb..93869d58c5 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -7,7 +7,8 @@\n #include \"object-store.h\"\n \n #define BUILTIN_MIDX_WRITE_USAGE \\\n-\tN_(\"git multi-pack-index [<options>] write [--preferred-pack=<pack>]\")\n+\tN_(\"git multi-pack-index [<options>] write [--preferred-pack=<pack>]\" \\\n+\t   \"[--refs-snapshot=<path>]\")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [<options>] verify\")\n@@ -45,6 +46,7 @@ static char const * const builtin_multi_pack_index_usage[] = {\n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n \tconst char *preferred_pack;\n+\tconst char *refs_snapshot;\n \tunsigned long batch_size;\n \tunsigned flags;\n \tint stdin_packs;\n@@ -83,6 +85,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\t\tMIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\t N_(\"write multi-pack index containing only given indexes\")),\n+\t\tOPT_FILENAME(0, \"refs-snapshot\", &opts.refs_snapshot,\n+\t\t\t     N_(\"refs snapshot for selecting bitmap commits\")),\n \t\tOPT_END(),\n \t};\n \n@@ -106,7 +110,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\tread_packs_from_stdin(&packs);\n \n \t\tret = write_midx_file_only(opts.object_dir, &packs,\n-\t\t\t\t\t   opts.preferred_pack, opts.flags);\n+\t\t\t\t\t   opts.preferred_pack,\n+\t\t\t\t\t   opts.refs_snapshot, opts.flags);\n \n \t\tstring_list_clear(&packs, 0);\n \n@@ -114,7 +119,7 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \n \t}\n \treturn write_midx_file(opts.object_dir, opts.preferred_pack,\n-\t\t\t       opts.flags);\n+\t\t\t       opts.refs_snapshot, opts.flags);\n }\n \n static int cmd_multi_pack_index_verify(int argc, const char **argv)\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex c1a209013b..dba83eede2 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -733,7 +733,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tunsigned flags = 0;\n \t\tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))\n \t\t\tflags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;\n-\t\twrite_midx_file(get_object_directory(), NULL, flags);\n+\t\twrite_midx_file(get_object_directory(), NULL, NULL, flags);\n \t}\n \n \tstring_list_clear(&names, 0);\ndiff --git a/midx.c b/midx.c\nindex 7ac97e66e0..984dea2dba 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -968,7 +968,43 @@ static void bitmap_show_commit(struct commit *commit, void *_data)\n \tdata->commits[data->commits_nr++] = commit;\n }\n \n+static int read_refs_snapshot(const char *refs_snapshot,\n+\t\t\t      struct rev_info *revs)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct object_id oid;\n+\tFILE *f = xfopen(refs_snapshot, \"r\");\n+\n+\twhile (strbuf_getline(&buf, f) != EOF) {\n+\t\tstruct object *object;\n+\t\tint preferred = 0;\n+\t\tchar *hex = buf.buf;\n+\t\tconst char *end = NULL;\n+\n+\t\tif (buf.len && *buf.buf == '+') {\n+\t\t\tpreferred = 1;\n+\t\t\thex = &buf.buf[1];\n+\t\t}\n+\n+\t\tif (parse_oid_hex(hex, &oid, &end) < 0)\n+\t\t\tdie(_(\"could not parse line: %s\"), buf.buf);\n+\t\tif (*end)\n+\t\t\tdie(_(\"malformed line: %s\"), buf.buf);\n+\n+\t\tobject = parse_object_or_die(&oid, NULL);\n+\t\tif (preferred)\n+\t\t\tobject->flags |= NEEDS_BITMAP;\n+\n+\t\tadd_pending_object(revs, object, \"\");\n+\t}\n+\n+\tfclose(f);\n+\tstrbuf_release(&buf);\n+\treturn 0;\n+}\n+\n static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p,\n+\t\t\t\t\t\t    const char *refs_snapshot,\n \t\t\t\t\t\t    struct write_midx_context *ctx)\n {\n \tstruct rev_info revs;\n@@ -977,8 +1013,12 @@ static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr\n \tcb.ctx = ctx;\n \n \trepo_init_revisions(the_repository, &revs, NULL);\n-\tsetup_revisions(0, NULL, &revs, NULL);\n-\tfor_each_ref(add_ref_to_pending, &revs);\n+\tif (refs_snapshot) {\n+\t\tread_refs_snapshot(refs_snapshot, &revs);\n+\t} else {\n+\t\tsetup_revisions(0, NULL, &revs, NULL);\n+\t\tfor_each_ref(add_ref_to_pending, &revs);\n+\t}\n \n \t/*\n \t * Skipping promisor objects here is intentional, since it only excludes\n@@ -1007,6 +1047,7 @@ static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr\n \n static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n \t\t\t     struct write_midx_context *ctx,\n+\t\t\t     const char *refs_snapshot,\n \t\t\t     unsigned flags)\n {\n \tstruct packing_data pdata;\n@@ -1018,7 +1059,7 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,\n \n \tprepare_midx_packing_data(&pdata, ctx);\n \n-\tcommits = find_commits_for_midx_bitmap(&commits_nr, ctx);\n+\tcommits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, ctx);\n \n \t/*\n \t * Build the MIDX-order index based on pdata.objects (which is already\n@@ -1066,6 +1107,7 @@ static int write_midx_internal(const char *object_dir,\n \t\t\t       struct string_list *packs_to_include,\n \t\t\t       struct string_list *packs_to_drop,\n \t\t\t       const char *preferred_pack_name,\n+\t\t\t       const char *refs_snapshot,\n \t\t\t       unsigned flags)\n {\n \tchar *midx_name;\n@@ -1359,7 +1401,8 @@ static int write_midx_internal(const char *object_dir,\n \tif (flags & MIDX_WRITE_REV_INDEX)\n \t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n \tif (flags & MIDX_WRITE_BITMAP) {\n-\t\tif (write_midx_bitmap(midx_name, midx_hash, &ctx, flags) < 0) {\n+\t\tif (write_midx_bitmap(midx_name, midx_hash, &ctx,\n+\t\t\t\t      refs_snapshot, flags) < 0) {\n \t\t\terror(_(\"could not write multi-pack bitmap\"));\n \t\t\tresult = 1;\n \t\t\tgoto cleanup;\n@@ -1394,19 +1437,21 @@ static int write_midx_internal(const char *object_dir,\n \n int write_midx_file(const char *object_dir,\n \t\t    const char *preferred_pack_name,\n+\t\t    const char *refs_snapshot,\n \t\t    unsigned flags)\n {\n \treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n-\t\t\t\t   flags);\n+\t\t\t\t   refs_snapshot, flags);\n }\n \n int write_midx_file_only(const char *object_dir,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n+\t\t\t const char *refs_snapshot,\n \t\t\t unsigned flags)\n {\n \treturn write_midx_internal(object_dir, packs_to_include, NULL,\n-\t\t\t\t   preferred_pack_name, flags);\n+\t\t\t\t   preferred_pack_name, refs_snapshot, flags);\n }\n \n struct clear_midx_data {\n@@ -1686,7 +1731,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tfree(count);\n \n \tif (packs_to_drop.nr) {\n-\t\tresult = write_midx_internal(object_dir, NULL, &packs_to_drop, NULL, flags);\n+\t\tresult = write_midx_internal(object_dir, NULL, &packs_to_drop, NULL, NULL, flags);\n \t\tm = NULL;\n \t}\n \n@@ -1877,7 +1922,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t\tgoto cleanup;\n \t}\n \n-\tresult = write_midx_internal(object_dir, NULL, NULL, NULL, flags);\n+\tresult = write_midx_internal(object_dir, NULL, NULL, NULL, NULL, flags);\n \tm = NULL;\n \n cleanup:\ndiff --git a/midx.h b/midx.h\nindex 3545e327ea..11ff094a8c 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -62,14 +62,18 @@ int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pa\n int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n \n-int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n /*\n  * Variant of write_midx_file which writes a MIDX containing only the packs\n  * specified in packs_to_include.\n  */\n+int write_midx_file(const char *object_dir,\n+\t\t    const char *preferred_pack_name,\n+\t\t    const char *refs_snapshot,\n+\t\t    unsigned flags);\n int write_midx_file_only(const char *object_dir,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\n+\t\t\t const char *refs_snapshot,\n \t\t\t unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);\ndiff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh\nindex 4ad7c2c969..069dab3e17 100755\n--- a/t/t5326-multi-pack-bitmaps.sh\n+++ b/t/t5326-multi-pack-bitmaps.sh\n@@ -283,4 +283,86 @@ test_expect_success 'pack.preferBitmapTips' '\n \t)\n '\n \n+test_expect_success 'writing a bitmap with --refs-snapshot' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit one &&\n+\t\ttest_commit two &&\n+\n+\t\tgit rev-parse one >snapshot &&\n+\n+\t\tgit repack -ad &&\n+\n+\t\t# First, write a MIDX which see both refs/tags/one and\n+\t\t# refs/tags/two (causing both of those commits to receive\n+\t\t# bitmaps).\n+\t\tgit multi-pack-index write --bitmap &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\n+\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n+\t\tgrep \"$(git rev-parse two)\" bitmaps &&\n+\n+\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n+\t\trm -fr $midx-$(midx_checksum $objdir).rev &&\n+\t\trm -fr $midx &&\n+\n+\t\t# Then again, but with a refs snapshot which only sees\n+\t\t# refs/tags/one.\n+\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\n+\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\tgrep \"$(git rev-parse one)\" bitmaps &&\n+\t\t! grep \"$(git rev-parse two)\" bitmaps\n+\t)\n+'\n+\n+test_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\n+\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\tsort <commits.raw >commits &&\n+\n+\t\tgit log --format=\"create refs/tags/%s %H\" HEAD >refs &&\n+\t\tgit update-ref --stdin <refs &&\n+\n+\t\tgit multi-pack-index write --bitmap &&\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\n+\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\tcomm -13 bitmaps commits >before &&\n+\t\ttest_line_count = 1 before &&\n+\n+\t\t(\n+\t\t\tgrep -vf before commits.raw &&\n+\t\t\t# mark missing commits as preferred\n+\t\t\tsed \"s/^/+/\" before\n+\t\t) >snapshot &&\n+\n+\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n+\t\trm -fr $midx-$(midx_checksum $objdir).rev &&\n+\t\trm -fr $midx &&\n+\n+\t\tgit multi-pack-index write --bitmap --refs-snapshot=snapshot &&\n+\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t! test_cmp before after\n+\t)\n+'\n+\n test_done\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"437416","messageId":"1b3dd331cac3a94747838bfeb6be14c7c5a240ed.1632880469.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1632880469.git.me@ttaylorr.com","subject":"[PATCH v3 4/9] builtin/repack.c: keep track of existing packs unconditionally","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-29T01:55:10Z","receivedAt":"2021-09-29T01:55:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In order to be able to write a multi-pack index during repacking, `git\nrepack` must keep track of which packs it wants to write into the MIDX.\nThis set is the union of existing packs which will not be deleted,\nnew pack(s) generated as a result of the repack, and .keep packs.\n\nPrior to this patch, `git repack` populated the list of existing packs\nonly when repacking all-into-one (i.e., with `-A` or `-a`), but we will\nsoon need to know this list when repacking when writing a MIDX without\na-i-o.\n\nPopulate the list of existing packs unconditionally, and guard removing\npacks from that list only when repacking a-i-o.\n\nAdditionally, keep track of filenames of kept packs separately, since\nthis, too, will be used in an upcoming patch.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/repack.c | 56 +++++++++++++++++++++++++++---------------------\n 1 file changed, 31 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex dba83eede2..39f11df675 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -94,12 +94,14 @@ static void remove_pack_on_signal(int signo)\n }\n \n /*\n- * Adds all packs hex strings to the fname list, which do not\n- * have a corresponding .keep file. These packs are not to\n- * be kept if we are going to pack everything into one file.\n+ * Adds all packs hex strings to either fname_list or fname_kept_list\n+ * based on whether each pack has a corresponding .keep file or not.\n+ * Packs without a .keep file are not to be kept if we are going to\n+ * pack everything into one file.\n  */\n-static void get_non_kept_pack_filenames(struct string_list *fname_list,\n-\t\t\t\t\tconst struct string_list *extra_keep)\n+static void collect_pack_filenames(struct string_list *fname_list,\n+\t\t\t\t   struct string_list *fname_kept_list,\n+\t\t\t\t   const struct string_list *extra_keep)\n {\n \tDIR *dir;\n \tstruct dirent *e;\n@@ -112,21 +114,20 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list,\n \t\tsize_t len;\n \t\tint i;\n \n+\t\tif (!strip_suffix(e->d_name, \".pack\", &len))\n+\t\t\tcontinue;\n+\n \t\tfor (i = 0; i < extra_keep->nr; i++)\n \t\t\tif (!fspathcmp(e->d_name, extra_keep->items[i].string))\n \t\t\t\tbreak;\n-\t\tif (extra_keep->nr > 0 && i < extra_keep->nr)\n-\t\t\tcontinue;\n-\n-\t\tif (!strip_suffix(e->d_name, \".pack\", &len))\n-\t\t\tcontinue;\n \n \t\tfname = xmemdupz(e->d_name, len);\n \n-\t\tif (!file_exists(mkpath(\"%s/%s.keep\", packdir, fname)))\n-\t\t\tstring_list_append_nodup(fname_list, fname);\n+\t\tif ((extra_keep->nr > 0 && i < extra_keep->nr) ||\n+\t\t    (file_exists(mkpath(\"%s/%s.keep\", packdir, fname))))\n+\t\t\tstring_list_append_nodup(fname_kept_list, fname);\n \t\telse\n-\t\t\tfree(fname);\n+\t\t\tstring_list_append_nodup(fname_list, fname);\n \t}\n \tclosedir(dir);\n }\n@@ -440,6 +441,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list names = STRING_LIST_INIT_DUP;\n \tstruct string_list rollback = STRING_LIST_INIT_NODUP;\n \tstruct string_list existing_packs = STRING_LIST_INIT_DUP;\n+\tstruct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n \tstruct pack_geometry *geometry = NULL;\n \tstruct strbuf line = STRBUF_INIT;\n \tint i, ext, ret;\n@@ -572,9 +574,10 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (use_delta_islands)\n \t\tstrvec_push(&cmd.args, \"--delta-islands\");\n \n+\tcollect_pack_filenames(&existing_packs, &existing_kept_packs,\n+\t\t\t       &keep_pack_list);\n+\n \tif (pack_everything & ALL_INTO_ONE) {\n-\t\tget_non_kept_pack_filenames(&existing_packs, &keep_pack_list);\n-\n \t\trepack_promisor_objects(&po_args, &names);\n \n \t\tif (existing_packs.nr && delete_redundant) {\n@@ -683,17 +686,19 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \treprepare_packed_git(the_repository);\n \n \tif (delete_redundant) {\n-\t\tconst int hexsz = the_hash_algo->hexsz;\n \t\tint opts = 0;\n-\t\tstring_list_sort(&names);\n-\t\tfor_each_string_list_item(item, &existing_packs) {\n-\t\t\tchar *sha1;\n-\t\t\tsize_t len = strlen(item->string);\n-\t\t\tif (len < hexsz)\n-\t\t\t\tcontinue;\n-\t\t\tsha1 = item->string + len - hexsz;\n-\t\t\tif (!string_list_has_string(&names, sha1))\n-\t\t\t\tremove_redundant_pack(packdir, item->string);\n+\t\tif (pack_everything & ALL_INTO_ONE) {\n+\t\t\tconst int hexsz = the_hash_algo->hexsz;\n+\t\t\tstring_list_sort(&names);\n+\t\t\tfor_each_string_list_item(item, &existing_packs) {\n+\t\t\t\tchar *sha1;\n+\t\t\t\tsize_t len = strlen(item->string);\n+\t\t\t\tif (len < hexsz)\n+\t\t\t\t\tcontinue;\n+\t\t\t\tsha1 = item->string + len - hexsz;\n+\t\t\t\tif (!string_list_has_string(&names, sha1))\n+\t\t\t\t\tremove_redundant_pack(packdir, item->string);\n+\t\t\t}\n \t\t}\n \n \t\tif (geometry) {\n@@ -739,6 +744,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstring_list_clear(&names, 0);\n \tstring_list_clear(&rollback, 0);\n \tstring_list_clear(&existing_packs, 0);\n+\tstring_list_clear(&existing_kept_packs, 0);\n \tclear_pack_geometry(geometry);\n \tstrbuf_release(&line);\n \n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"437417","messageId":"15831a201a99534063b81f9bba81ed88f766f46e.1632880469.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1632880469.git.me@ttaylorr.com","subject":"[PATCH v3 5/9] builtin/repack.c: rename variables that deal with non-kept packs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-29T01:55:12Z","receivedAt":"2021-09-29T01:55:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The new variable `existing_kept_packs` (and corresponding parameter\n`fname_kept_list`) added by the previous patch make it seem like\n`existing_packs` and `fname_list` are each subsets of the other two\nrespectively.\n\nIn reality, each pair is disjoint: one stores the packs without .keep\nfiles, and the other stores the packs with .keep files. Rename each to\nmore clearly reflect this.\n\nSuggested-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/repack.c | 22 +++++++++++-----------\n 1 file changed, 11 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 39f11df675..5539ec7e89 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -94,12 +94,12 @@ static void remove_pack_on_signal(int signo)\n }\n \n /*\n- * Adds all packs hex strings to either fname_list or fname_kept_list\n- * based on whether each pack has a corresponding .keep file or not.\n- * Packs without a .keep file are not to be kept if we are going to\n- * pack everything into one file.\n+ * Adds all packs hex strings to either fname_nonkept_list or\n+ * fname_kept_list based on whether each pack has a corresponding\n+ * .keep file or not.  Packs without a .keep file are not to be kept\n+ * if we are going to pack everything into one file.\n  */\n-static void collect_pack_filenames(struct string_list *fname_list,\n+static void collect_pack_filenames(struct string_list *fname_nonkept_list,\n \t\t\t\t   struct string_list *fname_kept_list,\n \t\t\t\t   const struct string_list *extra_keep)\n {\n@@ -127,7 +127,7 @@ static void collect_pack_filenames(struct string_list *fname_list,\n \t\t    (file_exists(mkpath(\"%s/%s.keep\", packdir, fname))))\n \t\t\tstring_list_append_nodup(fname_kept_list, fname);\n \t\telse\n-\t\t\tstring_list_append_nodup(fname_list, fname);\n+\t\t\tstring_list_append_nodup(fname_nonkept_list, fname);\n \t}\n \tclosedir(dir);\n }\n@@ -440,7 +440,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list_item *item;\n \tstruct string_list names = STRING_LIST_INIT_DUP;\n \tstruct string_list rollback = STRING_LIST_INIT_NODUP;\n-\tstruct string_list existing_packs = STRING_LIST_INIT_DUP;\n+\tstruct string_list existing_nonkept_packs = STRING_LIST_INIT_DUP;\n \tstruct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n \tstruct pack_geometry *geometry = NULL;\n \tstruct strbuf line = STRBUF_INIT;\n@@ -574,13 +574,13 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (use_delta_islands)\n \t\tstrvec_push(&cmd.args, \"--delta-islands\");\n \n-\tcollect_pack_filenames(&existing_packs, &existing_kept_packs,\n+\tcollect_pack_filenames(&existing_nonkept_packs, &existing_kept_packs,\n \t\t\t       &keep_pack_list);\n \n \tif (pack_everything & ALL_INTO_ONE) {\n \t\trepack_promisor_objects(&po_args, &names);\n \n-\t\tif (existing_packs.nr && delete_redundant) {\n+\t\tif (existing_nonkept_packs.nr && delete_redundant) {\n \t\t\tfor_each_string_list_item(item, &names) {\n \t\t\t\tstrvec_pushf(&cmd.args, \"--keep-pack=%s-%s.pack\",\n \t\t\t\t\t     packtmp_name, item->string);\n@@ -690,7 +690,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tif (pack_everything & ALL_INTO_ONE) {\n \t\t\tconst int hexsz = the_hash_algo->hexsz;\n \t\t\tstring_list_sort(&names);\n-\t\t\tfor_each_string_list_item(item, &existing_packs) {\n+\t\t\tfor_each_string_list_item(item, &existing_nonkept_packs) {\n \t\t\t\tchar *sha1;\n \t\t\t\tsize_t len = strlen(item->string);\n \t\t\t\tif (len < hexsz)\n@@ -743,7 +743,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \n \tstring_list_clear(&names, 0);\n \tstring_list_clear(&rollback, 0);\n-\tstring_list_clear(&existing_packs, 0);\n+\tstring_list_clear(&existing_nonkept_packs, 0);\n \tstring_list_clear(&existing_kept_packs, 0);\n \tclear_pack_geometry(geometry);\n \tstrbuf_release(&line);\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"437418","messageId":"1a40161baf13224228d9c755db8ce2bdeb06917a.1632880469.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1632880469.git.me@ttaylorr.com","subject":"[PATCH v3 6/9] builtin/repack.c: extract showing progress to a variable","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-29T01:55:15Z","receivedAt":"2021-09-29T01:55:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"We only ask whether stderr is a tty before calling\n'prune_packed_objects()', but the subsequent patch will add another use.\n\nExtract this check into a variable so that both can use it without\nhaving to call 'isatty()' twice.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/repack.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5539ec7e89..475677b297 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -446,6 +446,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct strbuf line = STRBUF_INIT;\n \tint i, ext, ret;\n \tFILE *out;\n+\tint show_progress = isatty(2);\n \n \t/* variables to be filled by option parsing */\n \tint pack_everything = 0;\n@@ -719,7 +720,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t}\n \t\t\tstrbuf_release(&buf);\n \t\t}\n-\t\tif (!po_args.quiet && isatty(2))\n+\t\tif (!po_args.quiet && show_progress)\n \t\t\topts |= PRUNE_PACKED_VERBOSE;\n \t\tprune_packed_objects(opts);\n \n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"437419","messageId":"6854f0751de56eb6117b985fc20e13e2a944a989.1632880469.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1632880469.git.me@ttaylorr.com","subject":"[PATCH v3 7/9] builtin/repack.c: support writing a MIDX while repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-29T01:55:18Z","receivedAt":"2021-09-29T01:55:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Teach `git repack` a new `--write-midx` option for callers that wish to\npersist a multi-pack index in their repository while repacking.\n\nThere are two existing alternatives to this new flag, but they don't\ncover our particular use-case. These alternatives are:\n\n  - Call 'git multi-pack-index write' after running 'git repack', or\n\n  - Set 'GIT_TEST_MULTI_PACK_INDEX=1' in your environment when running\n    'git repack'.\n\nThe former works, but introduces a gap in bitmap coverage between\nrepacking and writing a new MIDX (since the repack may have deleted a\npack included in the existing MIDX, invalidating it altogether).\n\nSetting the 'GIT_TEST_' environment variable is obviously unsupported.\nIn fact, even if it were supported officially, it still wouldn't work,\nbecause it generates the MIDX *after* redundant packs have been dropped,\nleading to the same issue as above.\n\nIntroduce a new option which eliminates this race by teaching `git\nrepack` to generate the MIDX at the critical point: after the new packs\nhave been written and moved into place, but before the redundant packs\nhave been removed.\n\nThis option is compatible with `git repack`'s '--bitmap' option (it\nchanges the interpretation to be: \"write a bitmap corresponding to the\nMIDX after one has been generated\").\n\nThere is a little bit of additional noise in the patch below to avoid\nrepeating ourselves when selecting which packs to delete. Instead of a\nsingle loop as before (where we iterate over 'existing_packs', decide if\na pack is worth deleting, and if so, delete it), we have two loops (the\nfirst where we decide which ones are worth deleting, and the second\nwhere we actually do the deleting). This makes it so we have a single\ncheck we can make consistently when (1) telling the MIDX which packs we\nwant to exclude, and (2) actually unlinking the redundant packs.\n\nThere is also a tiny change to short-circuit the body of\nwrite_midx_included_packs() when no packs remain in the case of an empty\nrepository. The MIDX code does not handle this, so avoid trying to\ngenerate a MIDX covering zero packs in the first place.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-repack.txt |  14 +++-\n builtin/repack.c             | 138 ++++++++++++++++++++++++++++++-----\n t/lib-midx.sh                |   8 ++\n t/t7700-repack.sh            |  96 ++++++++++++++++++++++++\n t/t7703-repack-geometric.sh  |   2 +-\n 5 files changed, 234 insertions(+), 24 deletions(-)\n create mode 100644 t/lib-midx.sh\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 24c00c9384..0f2d235ca5 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -9,7 +9,7 @@ git-repack - Pack unpacked objects in a repository\n SYNOPSIS\n --------\n [verse]\n-'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n+'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m] [--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>] [--write-midx]\n \n DESCRIPTION\n -----------\n@@ -128,10 +128,11 @@ depth is 4095.\n -b::\n --write-bitmap-index::\n \tWrite a reachability bitmap index as part of the repack. This\n-\tonly makes sense when used with `-a` or `-A`, as the bitmaps\n+\tonly makes sense when used with `-a`, `-A` or `-m`, as the bitmaps\n \tmust be able to refer to all reachable objects. This option\n-\toverrides the setting of `repack.writeBitmaps`.  This option\n-\thas no effect if multiple packfiles are created.\n+\toverrides the setting of `repack.writeBitmaps`. This option\n+\thas no effect if multiple packfiles are created, unless writing a\n+\tMIDX (in which case a multi-pack bitmap is created).\n \n --pack-kept-objects::\n \tInclude objects in `.keep` files when repacking.  Note that we\n@@ -190,6 +191,11 @@ to change in the future. This option (implying a drastically different\n repack mode) is not guaranteed to work with all other combinations of\n option to `git repack`.\n \n+-m::\n+--write-midx::\n+\tWrite a multi-pack index (see linkgit:git-multi-pack-index[1])\n+\tcontaining the non-redundant packs.\n+\n CONFIGURATION\n -------------\n \ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 475677b297..abb30f89e8 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -434,6 +434,76 @@ static void clear_pack_geometry(struct pack_geometry *geometry)\n \tgeometry->split = 0;\n }\n \n+static void midx_included_packs(struct string_list *include,\n+\t\t\t\tstruct string_list *existing_nonkept_packs,\n+\t\t\t\tstruct string_list *existing_kept_packs,\n+\t\t\t\tstruct string_list *names,\n+\t\t\t\tstruct pack_geometry *geometry)\n+{\n+\tstruct string_list_item *item;\n+\n+\tfor_each_string_list_item(item, existing_kept_packs)\n+\t\tstring_list_insert(include, xstrfmt(\"%s.idx\", item->string));\n+\tfor_each_string_list_item(item, names)\n+\t\tstring_list_insert(include, xstrfmt(\"pack-%s.idx\", item->string));\n+\tif (geometry) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tuint32_t i;\n+\t\tfor (i = geometry->split; i < geometry->pack_nr; i++) {\n+\t\t\tstruct packed_git *p = geometry->pack[i];\n+\n+\t\t\tstrbuf_addstr(&buf, pack_basename(p));\n+\t\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\t\tstring_list_insert(include, strbuf_detach(&buf, NULL));\n+\t\t}\n+\t} else {\n+\t\tfor_each_string_list_item(item, existing_nonkept_packs) {\n+\t\t\tif (item->util)\n+\t\t\t\tcontinue;\n+\t\t\tstring_list_insert(include, xstrfmt(\"%s.idx\", item->string));\n+\t\t}\n+\t}\n+}\n+\n+static int write_midx_included_packs(struct string_list *include,\n+\t\t\t\t     int show_progress, int write_bitmaps)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstruct string_list_item *item;\n+\tFILE *in;\n+\tint ret;\n+\n+\tif (!include->nr)\n+\t\treturn 0;\n+\n+\tcmd.in = -1;\n+\tcmd.git_cmd = 1;\n+\n+\tstrvec_push(&cmd.args, \"multi-pack-index\");\n+\tstrvec_pushl(&cmd.args, \"write\", \"--stdin-packs\", NULL);\n+\n+\tif (show_progress)\n+\t\tstrvec_push(&cmd.args, \"--progress\");\n+\telse\n+\t\tstrvec_push(&cmd.args, \"--no-progress\");\n+\n+\tif (write_bitmaps)\n+\t\tstrvec_push(&cmd.args, \"--bitmap\");\n+\n+\tret = start_command(&cmd);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tin = xfdopen(cmd.in, \"w\");\n+\tfor_each_string_list_item(item, include)\n+\t\tfprintf(in, \"%s\\n\", item->string);\n+\tfclose(in);\n+\n+\treturn finish_command(&cmd);\n+}\n+\n int cmd_repack(int argc, const char **argv, const char *prefix)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -457,6 +527,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tint no_update_server_info = 0;\n \tstruct pack_objects_args po_args = {NULL};\n \tint geometric_factor = 0;\n+\tint write_midx = 0;\n \n \tstruct option builtin_repack_options[] = {\n \t\tOPT_BIT('a', NULL, &pack_everything,\n@@ -499,6 +570,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"do not repack this pack\")),\n \t\tOPT_INTEGER('g', \"geometric\", &geometric_factor,\n \t\t\t    N_(\"find a geometric progression with factor <N>\")),\n+\t\tOPT_BOOL('m', \"write-midx\", &write_midx,\n+\t\t\t   N_(\"write a multi-pack index of the resulting packs\")),\n \t\tOPT_END()\n \t};\n \n@@ -515,8 +588,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"--keep-unreachable and -A are incompatible\"));\n \n \tif (write_bitmaps < 0) {\n-\t\tif (!(pack_everything & ALL_INTO_ONE) ||\n-\t\t    !is_bare_repository())\n+\t\tif (!write_midx &&\n+\t\t    (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))\n \t\t\twrite_bitmaps = 0;\n \t} else if (write_bitmaps &&\n \t\t   git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&\n@@ -526,7 +599,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (pack_kept_objects < 0)\n \t\tpack_kept_objects = write_bitmaps > 0;\n \n-\tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE))\n+\tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)\n \t\tdie(_(incremental_bitmap_conflict_error));\n \n \tif (geometric_factor) {\n@@ -568,10 +641,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t}\n \tif (has_promisor_remote())\n \t\tstrvec_push(&cmd.args, \"--exclude-promisor-objects\");\n-\tif (write_bitmaps > 0)\n-\t\tstrvec_push(&cmd.args, \"--write-bitmap-index\");\n-\telse if (write_bitmaps < 0)\n-\t\tstrvec_push(&cmd.args, \"--write-bitmap-index-quiet\");\n+\tif (!write_midx) {\n+\t\tif (write_bitmaps > 0)\n+\t\t\tstrvec_push(&cmd.args, \"--write-bitmap-index\");\n+\t\telse if (write_bitmaps < 0)\n+\t\t\tstrvec_push(&cmd.args, \"--write-bitmap-index-quiet\");\n+\t}\n \tif (use_delta_islands)\n \t\tstrvec_push(&cmd.args, \"--delta-islands\");\n \n@@ -684,22 +759,47 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t}\n \t/* End of pack replacement. */\n \n+\tif (delete_redundant && pack_everything & ALL_INTO_ONE) {\n+\t\tconst int hexsz = the_hash_algo->hexsz;\n+\t\tstring_list_sort(&names);\n+\t\tfor_each_string_list_item(item, &existing_nonkept_packs) {\n+\t\t\tchar *sha1;\n+\t\t\tsize_t len = strlen(item->string);\n+\t\t\tif (len < hexsz)\n+\t\t\t\tcontinue;\n+\t\t\tsha1 = item->string + len - hexsz;\n+\t\t\t/*\n+\t\t\t * Mark this pack for deletion, which ensures that this\n+\t\t\t * pack won't be included in a MIDX (if `--write-midx`\n+\t\t\t * was given) and that we will actually delete this pack\n+\t\t\t * (if `-d` was given).\n+\t\t\t */\n+\t\t\titem->util = (void*)(intptr_t)!string_list_has_string(&names, sha1);\n+\t\t}\n+\t}\n+\n+\tif (write_midx) {\n+\t\tstruct string_list include = STRING_LIST_INIT_NODUP;\n+\t\tmidx_included_packs(&include, &existing_nonkept_packs,\n+\t\t\t\t    &existing_kept_packs, &names, geometry);\n+\n+\t\tret = write_midx_included_packs(&include,\n+\t\t\t\t\t\tshow_progress, write_bitmaps > 0);\n+\n+\t\tstring_list_clear(&include, 0);\n+\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n \treprepare_packed_git(the_repository);\n \n \tif (delete_redundant) {\n \t\tint opts = 0;\n-\t\tif (pack_everything & ALL_INTO_ONE) {\n-\t\t\tconst int hexsz = the_hash_algo->hexsz;\n-\t\t\tstring_list_sort(&names);\n-\t\t\tfor_each_string_list_item(item, &existing_nonkept_packs) {\n-\t\t\t\tchar *sha1;\n-\t\t\t\tsize_t len = strlen(item->string);\n-\t\t\t\tif (len < hexsz)\n-\t\t\t\t\tcontinue;\n-\t\t\t\tsha1 = item->string + len - hexsz;\n-\t\t\t\tif (!string_list_has_string(&names, sha1))\n-\t\t\t\t\tremove_redundant_pack(packdir, item->string);\n-\t\t\t}\n+\t\tfor_each_string_list_item(item, &existing_nonkept_packs) {\n+\t\t\tif (!item->util)\n+\t\t\t\tcontinue;\n+\t\t\tremove_redundant_pack(packdir, item->string);\n \t\t}\n \n \t\tif (geometry) {\ndiff --git a/t/lib-midx.sh b/t/lib-midx.sh\nnew file mode 100644\nindex 0000000000..1261994744\n--- /dev/null\n+++ b/t/lib-midx.sh\n@@ -0,0 +1,8 @@\n+# test_midx_consistent <objdir>\n+test_midx_consistent () {\n+\tls $1/pack/pack-*.idx | xargs -n 1 basename | sort >expect &&\n+\ttest-tool read-midx $1 | grep ^pack-.*\\.idx$ | sort >actual &&\n+\n+\ttest_cmp expect actual &&\n+\tgit multi-pack-index --object-dir=$1 verify\n+}\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 98eda3bfeb..6792531dfd 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -3,6 +3,8 @@\n test_description='git repack works correctly'\n \n . ./test-lib.sh\n+. \"${TEST_DIRECTORY}/lib-bitmap.sh\"\n+. \"${TEST_DIRECTORY}/lib-midx.sh\"\n \n commit_and_pack () {\n \ttest_commit \"$@\" 1>&2 &&\n@@ -234,4 +236,98 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \ttest_must_be_empty actual\n '\n \n+objdir=.git/objects\n+midx=$objdir/pack/multi-pack-index\n+\n+test_expect_success 'setup for --write-midx tests' '\n+\tgit init midx &&\n+\t(\n+\t\tcd midx &&\n+\t\tgit config core.multiPackIndex true &&\n+\n+\t\ttest_commit base\n+\t)\n+'\n+\n+test_expect_success '--write-midx unchanged' '\n+\t(\n+\t\tcd midx &&\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack &&\n+\t\ttest_path_is_missing $midx &&\n+\t\ttest_path_is_missing $midx-*.bitmap &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_missing $midx-*.bitmap &&\n+\t\ttest_midx_consistent $objdir\n+\t)\n+'\n+\n+test_expect_success '--write-midx with a new pack' '\n+\t(\n+\t\tcd midx &&\n+\t\ttest_commit loose &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_missing $midx-*.bitmap &&\n+\t\ttest_midx_consistent $objdir\n+\t)\n+'\n+\n+test_expect_success '--write-midx with -b' '\n+\t(\n+\t\tcd midx &&\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -mb &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-*.bitmap &&\n+\t\ttest_midx_consistent $objdir\n+\t)\n+'\n+\n+test_expect_success '--write-midx with -d' '\n+\t(\n+\t\tcd midx &&\n+\t\ttest_commit repack &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad --write-midx &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_missing $midx-*.bitmap &&\n+\t\ttest_midx_consistent $objdir\n+\t)\n+'\n+\n+test_expect_success 'cleans up MIDX when appropriate' '\n+\t(\n+\t\tcd midx &&\n+\n+\t\ttest_commit repack-2 &&\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&\n+\n+\t\tchecksum=$(midx_checksum $objdir) &&\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$checksum.bitmap &&\n+\t\ttest_path_is_file $midx-$checksum.rev &&\n+\n+\t\ttest_commit repack-3 &&\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&\n+\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_missing $midx-$checksum.bitmap &&\n+\t\ttest_path_is_missing $midx-$checksum.rev &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).rev &&\n+\n+\t\ttest_commit repack-4 &&\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb &&\n+\n+\t\tfind $objdir/pack -type f -name \"multi-pack-index*\" >files &&\n+\t\ttest_must_be_empty files\n+\t)\n+'\n+\n test_done\ndiff --git a/t/t7703-repack-geometric.sh b/t/t7703-repack-geometric.sh\nindex 5ccaa440e0..67049f7637 100755\n--- a/t/t7703-repack-geometric.sh\n+++ b/t/t7703-repack-geometric.sh\n@@ -15,7 +15,7 @@ test_expect_success '--geometric with no packs' '\n \t(\n \t\tcd geometric &&\n \n-\t\tgit repack --geometric 2 >out &&\n+\t\tgit repack --write-midx --geometric 2 >out &&\n \t\ttest_i18ngrep \"Nothing new to pack\" out\n \t)\n '\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"437420","messageId":"3596c76daf095dc997c3d322ed96875efe9348a7.1632880469.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1632880469.git.me@ttaylorr.com","subject":"[PATCH v3 8/9] builtin/repack.c: make largest pack preferred","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-29T01:55:20Z","receivedAt":"2021-09-29T01:55:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When repacking into a geometric series and writing a multi-pack bitmap,\nit is beneficial to have the largest resulting pack be the preferred\nobject source in the bitmap's MIDX, since selecting the large packs can\nlead to fewer broken delta chains and better compression.\n\nTeach 'git repack' to identify this pack and pass it to the MIDX write\nmachinery in order to mark it as preferred.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Documentation/git-repack.txt |  4 ++++\n builtin/repack.c             | 27 ++++++++++++++++++++++++++-\n pack-bitmap.c                |  2 +-\n pack-bitmap.h                |  1 +\n t/helper/test-read-midx.c    | 25 ++++++++++++++++++++++++-\n t/t7703-repack-geometric.sh  | 22 ++++++++++++++++++++++\n 6 files changed, 78 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex 0f2d235ca5..7183fb498f 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -190,6 +190,10 @@ this \"roll-up\", without respect to their reachability. This is subject\n to change in the future. This option (implying a drastically different\n repack mode) is not guaranteed to work with all other combinations of\n option to `git repack`.\n++\n+When writing a multi-pack bitmap, `git repack` selects the largest resulting\n+pack as the preferred pack for object selection by the MIDX (see\n+linkgit:git-multi-pack-index[1]).\n \n -m::\n --write-midx::\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex abb30f89e8..1577f0d59f 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -423,6 +423,25 @@ static void split_pack_geometry(struct pack_geometry *geometry, int factor)\n \tgeometry->split = split;\n }\n \n+static struct packed_git *get_largest_active_pack(struct pack_geometry *geometry)\n+{\n+\tif (!geometry) {\n+\t\t/*\n+\t\t * No geometry means either an all-into-one repack (in which\n+\t\t * case there is only one pack left and it is the largest) or an\n+\t\t * incremental one.\n+\t\t *\n+\t\t * If repacking incrementally, then we could check the size of\n+\t\t * all packs to determine which should be preferred, but leave\n+\t\t * this for later.\n+\t\t */\n+\t\treturn NULL;\n+\t}\n+\tif (geometry->split == geometry->pack_nr)\n+\t\treturn NULL;\n+\treturn geometry->pack[geometry->pack_nr - 1];\n+}\n+\n static void clear_pack_geometry(struct pack_geometry *geometry)\n {\n \tif (!geometry)\n@@ -468,10 +487,12 @@ static void midx_included_packs(struct string_list *include,\n }\n \n static int write_midx_included_packs(struct string_list *include,\n+\t\t\t\t     struct pack_geometry *geometry,\n \t\t\t\t     int show_progress, int write_bitmaps)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tstruct string_list_item *item;\n+\tstruct packed_git *largest = get_largest_active_pack(geometry);\n \tFILE *in;\n \tint ret;\n \n@@ -492,6 +513,10 @@ static int write_midx_included_packs(struct string_list *include,\n \tif (write_bitmaps)\n \t\tstrvec_push(&cmd.args, \"--bitmap\");\n \n+\tif (largest)\n+\t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n+\t\t\t     pack_basename(largest));\n+\n \tret = start_command(&cmd);\n \tif (ret)\n \t\treturn ret;\n@@ -783,7 +808,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tmidx_included_packs(&include, &existing_nonkept_packs,\n \t\t\t\t    &existing_kept_packs, &names, geometry);\n \n-\t\tret = write_midx_included_packs(&include,\n+\t\tret = write_midx_included_packs(&include, geometry,\n \t\t\t\t\t\tshow_progress, write_bitmaps > 0);\n \n \t\tstring_list_clear(&include, 0);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 8504110a4d..67be9be9a6 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1418,7 +1418,7 @@ static int try_partial_reuse(struct packed_git *pack,\n \treturn 0;\n }\n \n-static uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n+uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git)\n {\n \tstruct multi_pack_index *m = bitmap_git->midx;\n \tif (!m)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 469090bad2..7d407c5a4c 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -55,6 +55,7 @@ int test_bitmap_commits(struct repository *r);\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\t\t\t struct list_objects_filter_options *filter,\n \t\t\t\t\t int filter_provided_objects);\n+uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git);\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *,\n \t\t\t\t       struct packed_git **packfile,\n \t\t\t\t       uint32_t *entries,\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex cb0d27049a..0038559129 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -3,6 +3,7 @@\n #include \"midx.h\"\n #include \"repository.h\"\n #include \"object-store.h\"\n+#include \"pack-bitmap.h\"\n \n static int read_midx_file(const char *object_dir, int show_objects)\n {\n@@ -72,14 +73,36 @@ static int read_midx_checksum(const char *object_dir)\n \treturn 0;\n }\n \n+static int read_midx_preferred_pack(const char *object_dir)\n+{\n+\tstruct multi_pack_index *midx = NULL;\n+\tstruct bitmap_index *bitmap = NULL;\n+\n+\tsetup_git_directory();\n+\n+\tmidx = load_multi_pack_index(object_dir, 1);\n+\tif (!midx)\n+\t\treturn 1;\n+\n+\tbitmap = prepare_bitmap_git(the_repository);\n+\tif (!(bitmap && bitmap_is_midx(bitmap)))\n+\t\treturn 1;\n+\n+\n+\tprintf(\"%s\\n\", midx->pack_names[midx_preferred_pack(bitmap)]);\n+\treturn 0;\n+}\n+\n int cmd__read_midx(int argc, const char **argv)\n {\n \tif (!(argc == 2 || argc == 3))\n-\t\tusage(\"read-midx [--show-objects|--checksum] <object-dir>\");\n+\t\tusage(\"read-midx [--show-objects|--checksum|--preferred-pack] <object-dir>\");\n \n \tif (!strcmp(argv[1], \"--show-objects\"))\n \t\treturn read_midx_file(argv[2], 1);\n \telse if (!strcmp(argv[1], \"--checksum\"))\n \t\treturn read_midx_checksum(argv[2]);\n+\telse if (!strcmp(argv[1], \"--preferred-pack\"))\n+\t\treturn read_midx_preferred_pack(argv[2]);\n \treturn read_midx_file(argv[1], 0);\n }\ndiff --git a/t/t7703-repack-geometric.sh b/t/t7703-repack-geometric.sh\nindex 67049f7637..bdbbcbf1ec 100755\n--- a/t/t7703-repack-geometric.sh\n+++ b/t/t7703-repack-geometric.sh\n@@ -180,4 +180,26 @@ test_expect_success '--geometric ignores kept packs' '\n \t)\n '\n \n+test_expect_success '--geometric chooses largest MIDX preferred pack' '\n+\tgit init geometric &&\n+\ttest_when_finished \"rm -fr geometric\" &&\n+\t(\n+\t\tcd geometric &&\n+\n+\t\t# These packs already form a geometric progression.\n+\t\ttest_commit_bulk --start=1 1 && # 3 objects\n+\t\ttest_commit_bulk --start=2 2 && # 6 objects\n+\t\tls $objdir/pack/pack-*.idx >before &&\n+\t\ttest_commit_bulk --start=4 4 && # 12 objects\n+\t\tls $objdir/pack/pack-*.idx >after &&\n+\n+\t\tgit repack --geometric 2 -dbm &&\n+\n+\t\tcomm -3 before after | xargs -n 1 basename >expect &&\n+\t\ttest-tool read-midx --preferred-pack $objdir >actual &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"437421","messageId":"d99f0753213a8073afbb1213f0e6d082bf3e5195.1632880469.git.me@ttaylorr.com","threadId":"56482","inReplyTo":"cover.1632880469.git.me@ttaylorr.com","subject":"[PATCH v3 9/9] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-29T01:55:23Z","receivedAt":"2021-09-29T01:55:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To prevent the race described in an earlier patch, generate and pass a\nreference snapshot to the multi-pack bitmap code, if we are writing one\nfrom `git repack`.\n\nThis patch is mostly limited to creating a temporary file, and then\ncalling for_each_ref(). Except we try to minimize duplicates, since\ndoing so can drastically reduce the size in network-of-forks style\nrepositories. In the kernel's fork network (the repository containing\nall objects from the kernel and all its forks), deduplicating the\nreferences drops the snapshot size from 934 MB to just 12 MB.\n\nBut since we're handling duplicates in this way, we have to make sure\nthat we preferred references (those listed in pack.preferBitmapTips)\nbefore non-preferred ones (to avoid recording an object which is pointed\nat by a preferred tip as non-preferred).\n\nWe accomplish this by doing separate passes over the references: first\nvisiting each prefix in pack.preferBitmapTips, and then over the rest of\nthe references.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/repack.c | 79 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 79 insertions(+)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 1577f0d59f..5cc0dff77c 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -15,6 +15,8 @@\n #include \"promisor-remote.h\"\n #include \"shallow.h\"\n #include \"pack.h\"\n+#include \"pack-bitmap.h\"\n+#include \"refs.h\"\n \n static int delta_base_offset = 1;\n static int pack_kept_objects = -1;\n@@ -453,6 +455,65 @@ static void clear_pack_geometry(struct pack_geometry *geometry)\n \tgeometry->split = 0;\n }\n \n+struct midx_snapshot_ref_data {\n+\tstruct tempfile *f;\n+\tstruct oidset seen;\n+\tint preferred;\n+};\n+\n+static int midx_snapshot_ref_one(const char *refname,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t int flag, void *_data)\n+{\n+\tstruct midx_snapshot_ref_data *data = _data;\n+\tstruct object_id peeled;\n+\n+\tif (!peel_iterated_oid(oid, &peeled))\n+\t\toid = &peeled;\n+\n+\tif (oidset_insert(&data->seen, oid))\n+\t\treturn 0; /* already seen */\n+\n+\tif (oid_object_info(the_repository, oid, NULL) != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tfprintf(data->f->fp, \"%s%s\\n\", data->preferred ? \"+\" : \"\",\n+\t\toid_to_hex(oid));\n+\n+\treturn 0;\n+}\n+\n+static void midx_snapshot_refs(struct tempfile *f)\n+{\n+\tstruct midx_snapshot_ref_data data;\n+\tconst struct string_list *preferred = bitmap_preferred_tips(the_repository);\n+\n+\tdata.f = f;\n+\toidset_init(&data.seen, 0);\n+\n+\tif (!fdopen_tempfile(f, \"w\"))\n+\t\t die(_(\"could not open tempfile %s for writing\"),\n+\t\t     get_tempfile_path(f));\n+\n+\tif (preferred) {\n+\t\tstruct string_list_item *item;\n+\n+\t\tdata.preferred = 1;\n+\t\tfor_each_string_list_item(item, preferred)\n+\t\t\tfor_each_ref_in(item->string, midx_snapshot_ref_one, &data);\n+\t\tdata.preferred = 0;\n+\t}\n+\n+\tfor_each_ref(midx_snapshot_ref_one, &data);\n+\n+\tif (close_tempfile_gently(f)) {\n+\t\tint save_errno = errno;\n+\t\tdelete_tempfile(&f);\n+\t\terrno = save_errno;\n+\t\tdie_errno(_(\"could not close refs snapshot tempfile\"));\n+\t}\n+}\n+\n static void midx_included_packs(struct string_list *include,\n \t\t\t\tstruct string_list *existing_nonkept_packs,\n \t\t\t\tstruct string_list *existing_kept_packs,\n@@ -488,6 +549,7 @@ static void midx_included_packs(struct string_list *include,\n \n static int write_midx_included_packs(struct string_list *include,\n \t\t\t\t     struct pack_geometry *geometry,\n+\t\t\t\t     const char *refs_snapshot,\n \t\t\t\t     int show_progress, int write_bitmaps)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -517,6 +579,9 @@ static int write_midx_included_packs(struct string_list *include,\n \t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n \t\t\t     pack_basename(largest));\n \n+\tif (refs_snapshot)\n+\t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\", refs_snapshot);\n+\n \tret = start_command(&cmd);\n \tif (ret)\n \t\treturn ret;\n@@ -539,6 +604,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n \tstruct pack_geometry *geometry = NULL;\n \tstruct strbuf line = STRBUF_INIT;\n+\tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n \tFILE *out;\n \tint show_progress = isatty(2);\n@@ -627,6 +693,18 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)\n \t\tdie(_(incremental_bitmap_conflict_error));\n \n+\tif (write_midx && write_bitmaps) {\n+\t\tstruct strbuf path = STRBUF_INIT;\n+\n+\t\tstrbuf_addf(&path, \"%s/%s_XXXXXX\", get_object_directory(),\n+\t\t\t    \"bitmap-ref-tips\");\n+\n+\t\trefs_snapshot = xmks_tempfile(path.buf);\n+\t\tmidx_snapshot_refs(refs_snapshot);\n+\n+\t\tstrbuf_release(&path);\n+\t}\n+\n \tif (geometric_factor) {\n \t\tif (pack_everything)\n \t\t\tdie(_(\"--geometric is incompatible with -A, -a\"));\n@@ -809,6 +887,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\t    &existing_kept_packs, &names, geometry);\n \n \t\tret = write_midx_included_packs(&include, geometry,\n+\t\t\t\t\t\trefs_snapshot ? get_tempfile_path(refs_snapshot) : NULL,\n \t\t\t\t\t\tshow_progress, write_bitmaps > 0);\n \n \t\tstring_list_clear(&include, 0);\n-- \n2.33.0.96.g73915697e6\n"},{"id":"437431","messageId":"xmqqk0j0f1pw.fsf@gitster.g","threadId":"56482","inReplyTo":"cover.1632880469.git.me@ttaylorr.com","subject":"Re: [PATCH v3 0/9] repack: introduce `--write-midx`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-29T04:24:43Z","receivedAt":"2021-09-29T04:24:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> Here is another small reroll of my series to implement the final component of\n> multi-pack reachability bitamps, which is to be able to write them from `git\n> repack`.\n>\n> This version incorporates feedback from Jonathan Tan and others at Google who\n> graciously provided review. A range-diff is included below, but the major\n> changes are:\n>\n>   - A comment to explain the relationship between 'ctx->m' and 'ctx->to_include'\n>     in midx.c:add_pack_to_midx().\n>\n>   - A comment to explain the meaning of write_midx_file_only().\n>\n>   - Clean-up of stray hunks, a missing strbuf_release().\n>\n>   - Replacing the name `existing_packs` with `existing_nonkept_packs` to\n>     emphasize the relationship between it and the similarly-named\n>     `existing_kept_packs`.\n>\n> Instead of depending on tb/multi-pack-bitmaps, this series now depends on the\n> tip of master.\n\n... as the topic already graduated to 'master' ;-)\n\nBut applying them on the same base as the previous round and merging\nthe result to 'master', and applying them on 'master', will give us\nidentical trees.\n\nWill replace; thanks.\n"},{"id":"437721","messageId":"20211001200135.215855-1-jonathantanmy@google.com","threadId":"56482","inReplyTo":"cover.1632880469.git.me@ttaylorr.com","subject":"Re: [PATCH v3 0/9] repack: introduce `--write-midx`","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-10-01T20:01:35Z","receivedAt":"2021-10-01T20:01:44Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Here is another small reroll of my series to implement the final component of\n> multi-pack reachability bitamps, which is to be able to write them from `git\n> repack`.\n\nI see that all my comments up to patch 6 (of v2) have been addressed.\nThere is one outstanding one involving a potentially uninitialized\nvariable [1] from patch 8 (of v2) that you (Taylor) might have missed\nsince I sent it after I reviewed the first 6.\n\nOther than that,\n\nReviewed-by: Jonathan Tan <jonathantanmy@google.com> (conditional on\nthat being resolved)\n\n[1] https://lore.kernel.org/git/20210924182247.2922561-1-jonathantanmy@google.com/\n"},{"id":"437741","messageId":"YVeN0mXqYvTHtNB+@nand.local","threadId":"56482","inReplyTo":"20210924182247.2922561-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 8/8] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-10-01T22:38:10Z","receivedAt":"2021-10-01T22:38:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Sep 24, 2021 at 11:22:47AM -0700, Jonathan Tan wrote:\n> > +static void midx_snapshot_refs(struct tempfile *f)\n> > +{\n> > +\tstruct midx_snapshot_ref_data data;\n> > +\tconst struct string_list *preferred = bitmap_preferred_tips(the_repository);\n> > +\n> > +\tdata.f = f;\n> > +\toidset_init(&data.seen, 0);\n> > +\n> > +\tif (!fdopen_tempfile(f, \"w\"))\n> > +\t\t die(_(\"could not open tempfile %s for writing\"),\n> > +\t\t     get_tempfile_path(f));\n> > +\n> > +\tif (preferred) {\n> > +\t\tstruct string_list_item *item;\n> > +\n> > +\t\tdata.preferred = 1;\n> > +\t\tfor_each_string_list_item(item, preferred)\n> > +\t\t\tfor_each_ref_in(item->string, midx_snapshot_ref_one, &data);\n> > +\t\tdata.preferred = 0;\n> > +\t}\n> > +\n> > +\tfor_each_ref(midx_snapshot_ref_one, &data);\n>\n> One small thing here - I think \"data.preferred = 0;\" needs to be moved\n> to right above this for_each_ref line. As it is, if preferred is NULL,\n> data.preferred wouldn't be initialized.\n>\n> Other than that (and the minor changes I've suggested in previous\n> patches), this series looks good to me.\n\nEek. Great catch. I was wondering why this didn't bite us at GitHub by\nmarking all refs as preferred, but it doesn't because we always get back\na non-NULL string list from bitmap_preferred_tips(), which causes us to\nset data.preferred back to 0.\n\nHere's a replacement patch which adds the missing `data.preferred = 0`\nwhile initializing data in this function, as well as calls\noidset_clear() and adds a test. Unfortunately, the new test in t7700 is\nbasically a smoke test, because of the way that bitmap selection works\n\nIf Junio would rather I send a reroll of the topic instead, I'm happy to\ndo that, too.\n\n--- 8< ---\n\nSubject: [PATCH] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps\n\nTo prevent the race described in an earlier patch, generate and pass a\nreference snapshot to the multi-pack bitmap code, if we are writing one\nfrom `git repack`.\n\nThis patch is mostly limited to creating a temporary file, and then\ncalling for_each_ref(). Except we try to minimize duplicates, since\ndoing so can drastically reduce the size in network-of-forks style\nrepositories. In the kernel's fork network (the repository containing\nall objects from the kernel and all its forks), deduplicating the\nreferences drops the snapshot size from 934 MB to just 12 MB.\n\nBut since we're handling duplicates in this way, we have to make sure\nthat we preferred references (those listed in pack.preferBitmapTips)\nbefore non-preferred ones (to avoid recording an object which is pointed\nat by a preferred tip as non-preferred).\n\nWe accomplish this by doing separate passes over the references: first\nvisiting each prefix in pack.preferBitmapTips, and then over the rest of\nthe references.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/repack.c  | 82 +++++++++++++++++++++++++++++++++++++++++++++++\n t/t7700-repack.sh | 42 ++++++++++++++++++++++++\n 2 files changed, 124 insertions(+)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 1577f0d59f..9354b2bd93 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -15,6 +15,8 @@\n #include \"promisor-remote.h\"\n #include \"shallow.h\"\n #include \"pack.h\"\n+#include \"pack-bitmap.h\"\n+#include \"refs.h\"\n\n static int delta_base_offset = 1;\n static int pack_kept_objects = -1;\n@@ -453,6 +455,68 @@ static void clear_pack_geometry(struct pack_geometry *geometry)\n \tgeometry->split = 0;\n }\n\n+struct midx_snapshot_ref_data {\n+\tstruct tempfile *f;\n+\tstruct oidset seen;\n+\tint preferred;\n+};\n+\n+static int midx_snapshot_ref_one(const char *refname,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t int flag, void *_data)\n+{\n+\tstruct midx_snapshot_ref_data *data = _data;\n+\tstruct object_id peeled;\n+\n+\tif (!peel_iterated_oid(oid, &peeled))\n+\t\toid = &peeled;\n+\n+\tif (oidset_insert(&data->seen, oid))\n+\t\treturn 0; /* already seen */\n+\n+\tif (oid_object_info(the_repository, oid, NULL) != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tfprintf(data->f->fp, \"%s%s\\n\", data->preferred ? \"+\" : \"\",\n+\t\toid_to_hex(oid));\n+\n+\treturn 0;\n+}\n+\n+static void midx_snapshot_refs(struct tempfile *f)\n+{\n+\tstruct midx_snapshot_ref_data data;\n+\tconst struct string_list *preferred = bitmap_preferred_tips(the_repository);\n+\n+\tdata.f = f;\n+\tdata.preferred = 0;\n+\toidset_init(&data.seen, 0);\n+\n+\tif (!fdopen_tempfile(f, \"w\"))\n+\t\t die(_(\"could not open tempfile %s for writing\"),\n+\t\t     get_tempfile_path(f));\n+\n+\tif (preferred) {\n+\t\tstruct string_list_item *item;\n+\n+\t\tdata.preferred = 1;\n+\t\tfor_each_string_list_item(item, preferred)\n+\t\t\tfor_each_ref_in(item->string, midx_snapshot_ref_one, &data);\n+\t\tdata.preferred = 0;\n+\t}\n+\n+\tfor_each_ref(midx_snapshot_ref_one, &data);\n+\n+\tif (close_tempfile_gently(f)) {\n+\t\tint save_errno = errno;\n+\t\tdelete_tempfile(&f);\n+\t\terrno = save_errno;\n+\t\tdie_errno(_(\"could not close refs snapshot tempfile\"));\n+\t}\n+\n+\toidset_clear(&data.seen);\n+}\n+\n static void midx_included_packs(struct string_list *include,\n \t\t\t\tstruct string_list *existing_nonkept_packs,\n \t\t\t\tstruct string_list *existing_kept_packs,\n@@ -488,6 +552,7 @@ static void midx_included_packs(struct string_list *include,\n\n static int write_midx_included_packs(struct string_list *include,\n \t\t\t\t     struct pack_geometry *geometry,\n+\t\t\t\t     const char *refs_snapshot,\n \t\t\t\t     int show_progress, int write_bitmaps)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n@@ -517,6 +582,9 @@ static int write_midx_included_packs(struct string_list *include,\n \t\tstrvec_pushf(&cmd.args, \"--preferred-pack=%s\",\n \t\t\t     pack_basename(largest));\n\n+\tif (refs_snapshot)\n+\t\tstrvec_pushf(&cmd.args, \"--refs-snapshot=%s\", refs_snapshot);\n+\n \tret = start_command(&cmd);\n \tif (ret)\n \t\treturn ret;\n@@ -539,6 +607,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list existing_kept_packs = STRING_LIST_INIT_DUP;\n \tstruct pack_geometry *geometry = NULL;\n \tstruct strbuf line = STRBUF_INIT;\n+\tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n \tFILE *out;\n \tint show_progress = isatty(2);\n@@ -627,6 +696,18 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tif (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx)\n \t\tdie(_(incremental_bitmap_conflict_error));\n\n+\tif (write_midx && write_bitmaps) {\n+\t\tstruct strbuf path = STRBUF_INIT;\n+\n+\t\tstrbuf_addf(&path, \"%s/%s_XXXXXX\", get_object_directory(),\n+\t\t\t    \"bitmap-ref-tips\");\n+\n+\t\trefs_snapshot = xmks_tempfile(path.buf);\n+\t\tmidx_snapshot_refs(refs_snapshot);\n+\n+\t\tstrbuf_release(&path);\n+\t}\n+\n \tif (geometric_factor) {\n \t\tif (pack_everything)\n \t\t\tdie(_(\"--geometric is incompatible with -A, -a\"));\n@@ -809,6 +890,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\t\t\t    &existing_kept_packs, &names, geometry);\n\n \t\tret = write_midx_included_packs(&include, geometry,\n+\t\t\t\t\t\trefs_snapshot ? get_tempfile_path(refs_snapshot) : NULL,\n \t\t\t\t\t\tshow_progress, write_bitmaps > 0);\n\n \t\tstring_list_clear(&include, 0);\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex 6792531dfd..0260ad6f0e 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -330,4 +330,46 @@ test_expect_success 'cleans up MIDX when appropriate' '\n \t)\n '\n\n+test_expect_success '--write-midx with preferred bitmap tips' '\n+\tgit init midx-preferred-tips &&\n+\ttest_when_finished \"rm -fr midx-preferred-tips\" &&\n+\t(\n+\t\tcd midx-preferred-tips &&\n+\n+\t\ttest_commit_bulk --message=\"%s\" 103 &&\n+\n+\t\tgit log --format=\"%H\" >commits.raw &&\n+\t\tsort <commits.raw >commits &&\n+\n+\t\tgit log --format=\"create refs/tags/%s/%s %H\" HEAD >refs &&\n+\t\tgit update-ref --stdin <refs &&\n+\n+\t\tgit repack --write-midx --write-bitmap-index &&\n+\t\ttest_path_is_file $midx &&\n+\t\ttest_path_is_file $midx-$(midx_checksum $objdir).bitmap &&\n+\n+\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\tcomm -13 bitmaps commits >before &&\n+\t\ttest_line_count = 1 before &&\n+\n+\t\trm -fr $midx-$(midx_checksum $objdir).bitmap &&\n+\t\trm -fr $midx-$(midx_checksum $objdir).rev &&\n+\t\trm -fr $midx &&\n+\n+\t\t# instead of constructing the snapshot ourselves (c.f., the test\n+\t\t# \"write a bitmap with --refs-snapshot (preferred tips)\" in\n+\t\t# t5326), mark the missing commit as preferred by adding it to\n+\t\t# the pack.preferBitmapTips configuration.\n+\t\tgit for-each-ref --format=\"%(refname:rstrip=1)\" \\\n+\t\t\t--points-at=\"$(cat before)\" >missing &&\n+\t\tgit config pack.preferBitmapTips \"$(cat missing)\" &&\n+\t\tgit repack --write-midx --write-bitmap-index &&\n+\n+\t\ttest-tool bitmap list-commits | sort >bitmaps &&\n+\t\tcomm -13 bitmaps commits >after &&\n+\n+\t\t! test_cmp before after\n+\t)\n+'\n+\n test_done\n--\n2.33.0.96.g73915697e6\n\n"},{"id":"437742","messageId":"YVeOcL7/kPxt5XVu@nand.local","threadId":"56482","inReplyTo":"20211001200135.215855-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 0/9] repack: introduce `--write-midx`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-10-01T22:40:48Z","receivedAt":"2021-10-01T22:40:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Oct 01, 2021 at 01:01:35PM -0700, Jonathan Tan wrote:\n> > Here is another small reroll of my series to implement the final component of\n> > multi-pack reachability bitamps, which is to be able to write them from `git\n> > repack`.\n>\n> I see that all my comments up to patch 6 (of v2) have been addressed.\n> There is one outstanding one involving a potentially uninitialized\n> variable [1] from patch 8 (of v2) that you (Taylor) might have missed\n> since I sent it after I reviewed the first 6.\n\nThanks! I must have missed your review of v2 when I sent v3 (indeed, my\nreroll came 5 days after you sent [1], so I missed it).\n\nI sent a replacement patch here:\n\n    https://lore.kernel.org/git/YVeN0mXqYvTHtNB+@nand.local/\n\nwhich Junio can take instead of [v3 8/8]. But I'm happy to reroll the\nwhole series, too, whatever is easier.\n\n> Reviewed-by: Jonathan Tan <jonathantanmy@google.com> (conditional on\n> that being resolved)\n\nThanks again :-).\n\n> [1] https://lore.kernel.org/git/20210924182247.2922561-1-jonathantanmy@google.com/\n\nThanks,\nTaylor\n"}]}