{"thread":{"id":"61199","subject":"[PATCH 00/11] midx: split MIDX writing routines into midx-write.c, cleanup","startedAt":"2024-03-25T17:24:18Z","lastAt":"2024-04-02T11:47:48Z","messageCount":30,"participants":["Taylor Blau","Junio C Hamano","Jeff King","Patrick Steinhardt"],"isPatch":true,"patchVersion":1,"patchTotal":11},"messages":[{"id":"491469","messageId":"cover.1711387439.git.me@ttaylorr.com","threadId":"61199","inReplyTo":null,"subject":"[PATCH 00/11] midx: split MIDX writing routines into midx-write.c, cleanup","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-25T17:24:11Z","receivedAt":"2024-03-25T17:24:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This is a small-ish series that I worked on to split the functions from\nmidx.c which are responsible for writing MIDX files into a separate\ncompilation unit, midx-write.c.\n\nThis is done for a couple of reasons:\n\n  - It reduces the size of midx.c, which is already quite large, thus\n    making it easier to read.\n\n  - It more clearly separates responsibility between the two, similar to\n    the division between pack-bitmap.c, and pack-bitmap-write.c.\n\nMost importantly, it makes midx.c easier to read in preparation for\nfuture changes that I hope to make to midx.c in order to support\nincremental MIDXs in a repository.\n\nThe series is structured as follows:\n\n  - The first seven patches move all writing-related functions from\n    midx.c to their new home in midx-write.c.\n\n  - The remaining four patches rewrite `git multi-pack-index repack` in\n    terms of `git pack-objects --stdin-packs`, yielding more\n    tightly-packed output as a result of the supplemental traversal\n    performed by `--stdin-packs`.\n\nThe first seven are the main goal of this series, and the remaining four\ncould be queued separately, depending on how folks feel about them.\n\nThanks in advance for your review!\n\nTaylor Blau (11):\n  midx-write: initial commit\n  midx: extern a pair of shared functions\n  midx: move `midx_repack` (and related functions) to midx-write.c\n  midx: move `expire_midx_packs` to midx-write.c\n  midx: move `write_midx_file_only` to midx-write.c\n  midx: move `write_midx_file` to midx-write.c\n  midx: move `write_midx_internal` (and related functions) to\n    midx-write.c\n  midx-write.c: avoid directly managed temporary strbuf\n  midx-write.c: factor out common want_included_pack() routine\n  midx-write.c: check count of packs to repack after grouping\n  midx-write.c: use `--stdin-packs` when repacking\n\n Makefile     |    1 +\n midx-write.c | 1525 ++++++++++++++++++++++++++++++++++++++++++++++++\n midx.c       | 1558 +-------------------------------------------------\n midx.h       |   19 +\n 4 files changed, 1559 insertions(+), 1544 deletions(-)\n create mode 100644 midx-write.c\n\n\nbase-commit: 3bd955d26919e149552f34aacf8a4e6368c26cec\n-- \n2.44.0.290.g736be63234b\n"},{"id":"491470","messageId":"ffa8ba18de8be68eb8dbb1e17b5bf8800f564505.1711387439.git.me@ttaylorr.com","threadId":"61199","inReplyTo":"cover.1711387439.git.me@ttaylorr.com","subject":"[PATCH 01/11] midx-write: initial commit","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-25T17:24:18Z","receivedAt":"2024-03-25T17:24:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a new empty midx-write.c source file. Similar to the\nrelationship between \"pack-bitmap.c\" and \"pack-bitmap-write.c\", this\nsource file will hold code that is specific to writing MIDX files as\nopposed to reading them (the latter will remain in midx.c).\n\nThis is a preparatory step which will reduce the overall size of midx.c\nand make it easier to read as we prepare for future changes to that file\n(outside the immediate scope of this series).\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Makefile     | 1 +\n midx-write.c | 2 ++\n 2 files changed, 3 insertions(+)\n create mode 100644 midx-write.c\n\ndiff --git a/Makefile b/Makefile\nindex 4e255c81f2..cf44a964c0 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1072,6 +1072,7 @@ LIB_OBJS += merge-ort-wrappers.o\n LIB_OBJS += merge-recursive.o\n LIB_OBJS += merge.o\n LIB_OBJS += midx.o\n+LIB_OBJS += midx-write.o\n LIB_OBJS += name-hash.o\n LIB_OBJS += negotiator/default.o\n LIB_OBJS += negotiator/noop.o\ndiff --git a/midx-write.c b/midx-write.c\nnew file mode 100644\nindex 0000000000..214179d308\n--- /dev/null\n+++ b/midx-write.c\n@@ -0,0 +1,2 @@\n+#include \"git-compat-util.h\"\n+#include \"midx.h\"\n-- \n2.44.0.290.g736be63234b\n\n"},{"id":"491471","messageId":"b776fd528d39144780e50750bdc1013e7eb3477a.1711387439.git.me@ttaylorr.com","threadId":"61199","inReplyTo":"cover.1711387439.git.me@ttaylorr.com","subject":"[PATCH 02/11] midx: extern a pair of shared functions","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-25T17:24:22Z","receivedAt":"2024-03-25T17:24:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The following commits will incrementally move code specific to writing\nMIDXs from midx.c to their new home in midx-write.c.\n\nPrepare for that by externing a pair of functions which:\n\n  - will (temporarily) need to be called from both midx.c and\n    midx-write.c, but\n\n  - are implementation details that should not be exposed via the midx.h\n    header.\n\nDeclare these functions as extern within midx-write.c, and introduce a\nsimilar (non-extern) declaration within midx.c. This change will be\neffectively reverted by the end of this series.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx-write.c | 10 ++++++++++\n midx.c       | 24 +++++++++++++++++-------\n 2 files changed, 27 insertions(+), 7 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 214179d308..4aab273243 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1,2 +1,12 @@\n #include \"git-compat-util.h\"\n #include \"midx.h\"\n+\n+extern int write_midx_internal(const char *object_dir,\n+\t\t\t       struct string_list *packs_to_include,\n+\t\t\t       struct string_list *packs_to_drop,\n+\t\t\t       const char *preferred_pack_name,\n+\t\t\t       const char *refs_snapshot,\n+\t\t\t       unsigned flags);\n+\n+extern struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n+\t\t\t\t\t\t\tconst char *object_dir);\ndiff --git a/midx.c b/midx.c\nindex 85e1c2cd12..5f22f01716 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -23,6 +23,16 @@\n #include \"list-objects.h\"\n #include \"pack-revindex.h\"\n \n+struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n+\t\t\t\t\t\t\tconst char *object_dir);\n+\n+int write_midx_internal(const char *object_dir,\n+\t\t\t       struct string_list *packs_to_include,\n+\t\t\t       struct string_list *packs_to_drop,\n+\t\t\t       const char *preferred_pack_name,\n+\t\t\t       const char *refs_snapshot,\n+\t\t\t       unsigned flags);\n+\n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n #define MIDX_BYTE_FILE_VERSION 4\n@@ -1347,7 +1357,7 @@ static int write_midx_bitmap(const char *midx_name,\n \treturn ret;\n }\n \n-static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n+struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n \t\t\t\t\t\t\tconst char *object_dir)\n {\n \tstruct multi_pack_index *result = NULL;\n@@ -1372,12 +1382,12 @@ static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n \treturn result;\n }\n \n-static int write_midx_internal(const char *object_dir,\n-\t\t\t       struct string_list *packs_to_include,\n-\t\t\t       struct string_list *packs_to_drop,\n-\t\t\t       const char *preferred_pack_name,\n-\t\t\t       const char *refs_snapshot,\n-\t\t\t       unsigned flags)\n+int write_midx_internal(const char *object_dir,\n+\t\t\tstruct string_list *packs_to_include,\n+\t\t\tstruct string_list *packs_to_drop,\n+\t\t\tconst char *preferred_pack_name,\n+\t\t\tconst char *refs_snapshot,\n+\t\t\tunsigned flags)\n {\n \tstruct strbuf midx_name = STRBUF_INIT;\n \tunsigned char midx_hash[GIT_MAX_RAWSZ];\n-- \n2.44.0.290.g736be63234b\n\n"},{"id":"491472","messageId":"487a0ccda8c781a4e7cfdd14d32b0466a867ddff.1711387439.git.me@ttaylorr.com","threadId":"61199","inReplyTo":"cover.1711387439.git.me@ttaylorr.com","subject":"[PATCH 03/11] midx: move `midx_repack` (and related functions) to midx-write.c","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-25T17:24:25Z","receivedAt":"2024-03-25T17:24:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Move `midx_repack()`, the main function which implements the sub-command\n'git multi-pack-index repack' into midx-write.c.\n\nThis patch does not introduce any behavioral changes and is best viewed\nwith `--color-moved`.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx-write.c | 202 +++++++++++++++++++++++++++++++++++++++++++++++++++\n midx.c       | 196 -------------------------------------------------\n 2 files changed, 202 insertions(+), 196 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 4aab273243..6dd58be7e0 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1,5 +1,11 @@\n #include \"git-compat-util.h\"\n+#include \"config.h\"\n+#include \"hex.h\"\n+#include \"packfile.h\"\n #include \"midx.h\"\n+#include \"run-command.h\"\n+#include \"pack-bitmap.h\"\n+#include \"revision.h\"\n \n extern int write_midx_internal(const char *object_dir,\n \t\t\t       struct string_list *packs_to_include,\n@@ -10,3 +16,199 @@ extern int write_midx_internal(const char *object_dir,\n \n extern struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n \t\t\t\t\t\t\tconst char *object_dir);\n+\n+struct repack_info {\n+\ttimestamp_t mtime;\n+\tuint32_t referenced_objects;\n+\tuint32_t pack_int_id;\n+};\n+\n+static int compare_by_mtime(const void *a_, const void *b_)\n+{\n+\tconst struct repack_info *a, *b;\n+\n+\ta = (const struct repack_info *)a_;\n+\tb = (const struct repack_info *)b_;\n+\n+\tif (a->mtime < b->mtime)\n+\t\treturn -1;\n+\tif (a->mtime > b->mtime)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static int fill_included_packs_all(struct repository *r,\n+\t\t\t\t   struct multi_pack_index *m,\n+\t\t\t\t   unsigned char *include_pack)\n+{\n+\tuint32_t i, count = 0;\n+\tint pack_kept_objects = 0;\n+\n+\trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tif (prepare_midx_pack(r, m, i))\n+\t\t\tcontinue;\n+\t\tif (!pack_kept_objects && m->packs[i]->pack_keep)\n+\t\t\tcontinue;\n+\t\tif (m->packs[i]->is_cruft)\n+\t\t\tcontinue;\n+\n+\t\tinclude_pack[i] = 1;\n+\t\tcount++;\n+\t}\n+\n+\treturn count < 2;\n+}\n+\n+static int fill_included_packs_batch(struct repository *r,\n+\t\t\t\t     struct multi_pack_index *m,\n+\t\t\t\t     unsigned char *include_pack,\n+\t\t\t\t     size_t batch_size)\n+{\n+\tuint32_t i, packs_to_repack;\n+\tsize_t total_size;\n+\tstruct repack_info *pack_info;\n+\tint pack_kept_objects = 0;\n+\n+\tCALLOC_ARRAY(pack_info, m->num_packs);\n+\n+\trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tpack_info[i].pack_int_id = i;\n+\n+\t\tif (prepare_midx_pack(r, m, i))\n+\t\t\tcontinue;\n+\n+\t\tpack_info[i].mtime = m->packs[i]->mtime;\n+\t}\n+\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n+\t\tpack_info[pack_int_id].referenced_objects++;\n+\t}\n+\n+\tQSORT(pack_info, m->num_packs, compare_by_mtime);\n+\n+\ttotal_size = 0;\n+\tpacks_to_repack = 0;\n+\tfor (i = 0; total_size < batch_size && i < m->num_packs; i++) {\n+\t\tint pack_int_id = pack_info[i].pack_int_id;\n+\t\tstruct packed_git *p = m->packs[pack_int_id];\n+\t\tsize_t expected_size;\n+\n+\t\tif (!p)\n+\t\t\tcontinue;\n+\t\tif (!pack_kept_objects && p->pack_keep)\n+\t\t\tcontinue;\n+\t\tif (p->is_cruft)\n+\t\t\tcontinue;\n+\t\tif (open_pack_index(p) || !p->num_objects)\n+\t\t\tcontinue;\n+\n+\t\texpected_size = st_mult(p->pack_size,\n+\t\t\t\t\tpack_info[i].referenced_objects);\n+\t\texpected_size /= p->num_objects;\n+\n+\t\tif (expected_size >= batch_size)\n+\t\t\tcontinue;\n+\n+\t\tpacks_to_repack++;\n+\t\ttotal_size += expected_size;\n+\t\tinclude_pack[pack_int_id] = 1;\n+\t}\n+\n+\tfree(pack_info);\n+\n+\tif (packs_to_repack < 2)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+int midx_repack(struct repository *r, const char *object_dir, size_t batch_size, unsigned flags)\n+{\n+\tint result = 0;\n+\tuint32_t i;\n+\tunsigned char *include_pack;\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tFILE *cmd_in;\n+\tstruct strbuf base_name = STRBUF_INIT;\n+\tstruct multi_pack_index *m = lookup_multi_pack_index(r, object_dir);\n+\n+\t/*\n+\t * When updating the default for these configuration\n+\t * variables in builtin/repack.c, these must be adjusted\n+\t * to match.\n+\t */\n+\tint delta_base_offset = 1;\n+\tint use_delta_islands = 0;\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tCALLOC_ARRAY(include_pack, m->num_packs);\n+\n+\tif (batch_size) {\n+\t\tif (fill_included_packs_batch(r, m, include_pack, batch_size))\n+\t\t\tgoto cleanup;\n+\t} else if (fill_included_packs_all(r, m, include_pack))\n+\t\tgoto cleanup;\n+\n+\trepo_config_get_bool(r, \"repack.usedeltabaseoffset\", &delta_base_offset);\n+\trepo_config_get_bool(r, \"repack.usedeltaislands\", &use_delta_islands);\n+\n+\tstrvec_push(&cmd.args, \"pack-objects\");\n+\n+\tstrbuf_addstr(&base_name, object_dir);\n+\tstrbuf_addstr(&base_name, \"/pack/pack\");\n+\tstrvec_push(&cmd.args, base_name.buf);\n+\n+\tif (delta_base_offset)\n+\t\tstrvec_push(&cmd.args, \"--delta-base-offset\");\n+\tif (use_delta_islands)\n+\t\tstrvec_push(&cmd.args, \"--delta-islands\");\n+\n+\tif (flags & MIDX_PROGRESS)\n+\t\tstrvec_push(&cmd.args, \"--progress\");\n+\telse\n+\t\tstrvec_push(&cmd.args, \"-q\");\n+\n+\tstrbuf_release(&base_name);\n+\n+\tcmd.git_cmd = 1;\n+\tcmd.in = cmd.out = -1;\n+\n+\tif (start_command(&cmd)) {\n+\t\terror(_(\"could not start pack-objects\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tcmd_in = xfdopen(cmd.in, \"w\");\n+\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tstruct object_id oid;\n+\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n+\n+\t\tif (!include_pack[pack_int_id])\n+\t\t\tcontinue;\n+\n+\t\tnth_midxed_object_oid(&oid, m, i);\n+\t\tfprintf(cmd_in, \"%s\\n\", oid_to_hex(&oid));\n+\t}\n+\tfclose(cmd_in);\n+\n+\tif (finish_command(&cmd)) {\n+\t\terror(_(\"could not finish pack-objects\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tresult = write_midx_internal(object_dir, NULL, NULL, NULL, NULL, flags);\n+\n+cleanup:\n+\tfree(include_pack);\n+\treturn result;\n+}\ndiff --git a/midx.c b/midx.c\nindex 5f22f01716..3bd8c58642 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -2055,199 +2055,3 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \n \treturn result;\n }\n-\n-struct repack_info {\n-\ttimestamp_t mtime;\n-\tuint32_t referenced_objects;\n-\tuint32_t pack_int_id;\n-};\n-\n-static int compare_by_mtime(const void *a_, const void *b_)\n-{\n-\tconst struct repack_info *a, *b;\n-\n-\ta = (const struct repack_info *)a_;\n-\tb = (const struct repack_info *)b_;\n-\n-\tif (a->mtime < b->mtime)\n-\t\treturn -1;\n-\tif (a->mtime > b->mtime)\n-\t\treturn 1;\n-\treturn 0;\n-}\n-\n-static int fill_included_packs_all(struct repository *r,\n-\t\t\t\t   struct multi_pack_index *m,\n-\t\t\t\t   unsigned char *include_pack)\n-{\n-\tuint32_t i, count = 0;\n-\tint pack_kept_objects = 0;\n-\n-\trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n-\n-\tfor (i = 0; i < m->num_packs; i++) {\n-\t\tif (prepare_midx_pack(r, m, i))\n-\t\t\tcontinue;\n-\t\tif (!pack_kept_objects && m->packs[i]->pack_keep)\n-\t\t\tcontinue;\n-\t\tif (m->packs[i]->is_cruft)\n-\t\t\tcontinue;\n-\n-\t\tinclude_pack[i] = 1;\n-\t\tcount++;\n-\t}\n-\n-\treturn count < 2;\n-}\n-\n-static int fill_included_packs_batch(struct repository *r,\n-\t\t\t\t     struct multi_pack_index *m,\n-\t\t\t\t     unsigned char *include_pack,\n-\t\t\t\t     size_t batch_size)\n-{\n-\tuint32_t i, packs_to_repack;\n-\tsize_t total_size;\n-\tstruct repack_info *pack_info;\n-\tint pack_kept_objects = 0;\n-\n-\tCALLOC_ARRAY(pack_info, m->num_packs);\n-\n-\trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n-\n-\tfor (i = 0; i < m->num_packs; i++) {\n-\t\tpack_info[i].pack_int_id = i;\n-\n-\t\tif (prepare_midx_pack(r, m, i))\n-\t\t\tcontinue;\n-\n-\t\tpack_info[i].mtime = m->packs[i]->mtime;\n-\t}\n-\n-\tfor (i = 0; i < m->num_objects; i++) {\n-\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n-\t\tpack_info[pack_int_id].referenced_objects++;\n-\t}\n-\n-\tQSORT(pack_info, m->num_packs, compare_by_mtime);\n-\n-\ttotal_size = 0;\n-\tpacks_to_repack = 0;\n-\tfor (i = 0; total_size < batch_size && i < m->num_packs; i++) {\n-\t\tint pack_int_id = pack_info[i].pack_int_id;\n-\t\tstruct packed_git *p = m->packs[pack_int_id];\n-\t\tsize_t expected_size;\n-\n-\t\tif (!p)\n-\t\t\tcontinue;\n-\t\tif (!pack_kept_objects && p->pack_keep)\n-\t\t\tcontinue;\n-\t\tif (p->is_cruft)\n-\t\t\tcontinue;\n-\t\tif (open_pack_index(p) || !p->num_objects)\n-\t\t\tcontinue;\n-\n-\t\texpected_size = st_mult(p->pack_size,\n-\t\t\t\t\tpack_info[i].referenced_objects);\n-\t\texpected_size /= p->num_objects;\n-\n-\t\tif (expected_size >= batch_size)\n-\t\t\tcontinue;\n-\n-\t\tpacks_to_repack++;\n-\t\ttotal_size += expected_size;\n-\t\tinclude_pack[pack_int_id] = 1;\n-\t}\n-\n-\tfree(pack_info);\n-\n-\tif (packs_to_repack < 2)\n-\t\treturn 1;\n-\n-\treturn 0;\n-}\n-\n-int midx_repack(struct repository *r, const char *object_dir, size_t batch_size, unsigned flags)\n-{\n-\tint result = 0;\n-\tuint32_t i;\n-\tunsigned char *include_pack;\n-\tstruct child_process cmd = CHILD_PROCESS_INIT;\n-\tFILE *cmd_in;\n-\tstruct strbuf base_name = STRBUF_INIT;\n-\tstruct multi_pack_index *m = lookup_multi_pack_index(r, object_dir);\n-\n-\t/*\n-\t * When updating the default for these configuration\n-\t * variables in builtin/repack.c, these must be adjusted\n-\t * to match.\n-\t */\n-\tint delta_base_offset = 1;\n-\tint use_delta_islands = 0;\n-\n-\tif (!m)\n-\t\treturn 0;\n-\n-\tCALLOC_ARRAY(include_pack, m->num_packs);\n-\n-\tif (batch_size) {\n-\t\tif (fill_included_packs_batch(r, m, include_pack, batch_size))\n-\t\t\tgoto cleanup;\n-\t} else if (fill_included_packs_all(r, m, include_pack))\n-\t\tgoto cleanup;\n-\n-\trepo_config_get_bool(r, \"repack.usedeltabaseoffset\", &delta_base_offset);\n-\trepo_config_get_bool(r, \"repack.usedeltaislands\", &use_delta_islands);\n-\n-\tstrvec_push(&cmd.args, \"pack-objects\");\n-\n-\tstrbuf_addstr(&base_name, object_dir);\n-\tstrbuf_addstr(&base_name, \"/pack/pack\");\n-\tstrvec_push(&cmd.args, base_name.buf);\n-\n-\tif (delta_base_offset)\n-\t\tstrvec_push(&cmd.args, \"--delta-base-offset\");\n-\tif (use_delta_islands)\n-\t\tstrvec_push(&cmd.args, \"--delta-islands\");\n-\n-\tif (flags & MIDX_PROGRESS)\n-\t\tstrvec_push(&cmd.args, \"--progress\");\n-\telse\n-\t\tstrvec_push(&cmd.args, \"-q\");\n-\n-\tstrbuf_release(&base_name);\n-\n-\tcmd.git_cmd = 1;\n-\tcmd.in = cmd.out = -1;\n-\n-\tif (start_command(&cmd)) {\n-\t\terror(_(\"could not start pack-objects\"));\n-\t\tresult = 1;\n-\t\tgoto cleanup;\n-\t}\n-\n-\tcmd_in = xfdopen(cmd.in, \"w\");\n-\n-\tfor (i = 0; i < m->num_objects; i++) {\n-\t\tstruct object_id oid;\n-\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n-\n-\t\tif (!include_pack[pack_int_id])\n-\t\t\tcontinue;\n-\n-\t\tnth_midxed_object_oid(&oid, m, i);\n-\t\tfprintf(cmd_in, \"%s\\n\", oid_to_hex(&oid));\n-\t}\n-\tfclose(cmd_in);\n-\n-\tif (finish_command(&cmd)) {\n-\t\terror(_(\"could not finish pack-objects\"));\n-\t\tresult = 1;\n-\t\tgoto cleanup;\n-\t}\n-\n-\tresult = write_midx_internal(object_dir, NULL, NULL, NULL, NULL, flags);\n-\n-cleanup:\n-\tfree(include_pack);\n-\treturn result;\n-}\n-- \n2.44.0.290.g736be63234b\n\n"},{"id":"491473","messageId":"e2b6459aa8f4ecedf9a7b1df3e7e56739254a8d1.1711387439.git.me@ttaylorr.com","threadId":"61199","inReplyTo":"cover.1711387439.git.me@ttaylorr.com","subject":"[PATCH 04/11] midx: move `expire_midx_packs` to midx-write.c","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-25T17:24:29Z","receivedAt":"2024-03-25T17:24:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Move `expire_midx_packs()`, the main function which implements the\nsub-command 'git multi-pack-index expire' into midx-write.c.\n\nSimilar to the previous patch, this patch does not introduce any\nbehavioral changes and is best viewed with `--color-moved`.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx-write.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n midx.c       | 57 ---------------------------------------------------\n 2 files changed, 58 insertions(+), 57 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 6dd58be7e0..d679e0a131 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -3,6 +3,7 @@\n #include \"hex.h\"\n #include \"packfile.h\"\n #include \"midx.h\"\n+#include \"progress.h\"\n #include \"run-command.h\"\n #include \"pack-bitmap.h\"\n #include \"revision.h\"\n@@ -17,6 +18,63 @@ extern int write_midx_internal(const char *object_dir,\n extern struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n \t\t\t\t\t\t\tconst char *object_dir);\n \n+int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags)\n+{\n+\tuint32_t i, *count, result = 0;\n+\tstruct string_list packs_to_drop = STRING_LIST_INIT_DUP;\n+\tstruct multi_pack_index *m = lookup_multi_pack_index(r, object_dir);\n+\tstruct progress *progress = NULL;\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tCALLOC_ARRAY(count, m->num_packs);\n+\n+\tif (flags & MIDX_PROGRESS)\n+\t\tprogress = start_delayed_progress(_(\"Counting referenced objects\"),\n+\t\t\t\t\t  m->num_objects);\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tint pack_int_id = nth_midxed_pack_int_id(m, i);\n+\t\tcount[pack_int_id]++;\n+\t\tdisplay_progress(progress, i + 1);\n+\t}\n+\tstop_progress(&progress);\n+\n+\tif (flags & MIDX_PROGRESS)\n+\t\tprogress = start_delayed_progress(_(\"Finding and deleting unreferenced packfiles\"),\n+\t\t\t\t\t  m->num_packs);\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tchar *pack_name;\n+\t\tdisplay_progress(progress, i + 1);\n+\n+\t\tif (count[i])\n+\t\t\tcontinue;\n+\n+\t\tif (prepare_midx_pack(r, m, i))\n+\t\t\tcontinue;\n+\n+\t\tif (m->packs[i]->pack_keep || m->packs[i]->is_cruft)\n+\t\t\tcontinue;\n+\n+\t\tpack_name = xstrdup(m->packs[i]->pack_name);\n+\t\tclose_pack(m->packs[i]);\n+\n+\t\tstring_list_insert(&packs_to_drop, m->pack_names[i]);\n+\t\tunlink_pack_path(pack_name, 0);\n+\t\tfree(pack_name);\n+\t}\n+\tstop_progress(&progress);\n+\n+\tfree(count);\n+\n+\tif (packs_to_drop.nr)\n+\t\tresult = write_midx_internal(object_dir, NULL, &packs_to_drop, NULL, NULL, flags);\n+\n+\tstring_list_clear(&packs_to_drop, 0);\n+\n+\treturn result;\n+}\n+\n struct repack_info {\n \ttimestamp_t mtime;\n \tuint32_t referenced_objects;\ndiff --git a/midx.c b/midx.c\nindex 3bd8c58642..5936bc5b9e 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1998,60 +1998,3 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \n \treturn verify_midx_error;\n }\n-\n-int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags)\n-{\n-\tuint32_t i, *count, result = 0;\n-\tstruct string_list packs_to_drop = STRING_LIST_INIT_DUP;\n-\tstruct multi_pack_index *m = lookup_multi_pack_index(r, object_dir);\n-\tstruct progress *progress = NULL;\n-\n-\tif (!m)\n-\t\treturn 0;\n-\n-\tCALLOC_ARRAY(count, m->num_packs);\n-\n-\tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_delayed_progress(_(\"Counting referenced objects\"),\n-\t\t\t\t\t  m->num_objects);\n-\tfor (i = 0; i < m->num_objects; i++) {\n-\t\tint pack_int_id = nth_midxed_pack_int_id(m, i);\n-\t\tcount[pack_int_id]++;\n-\t\tdisplay_progress(progress, i + 1);\n-\t}\n-\tstop_progress(&progress);\n-\n-\tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_delayed_progress(_(\"Finding and deleting unreferenced packfiles\"),\n-\t\t\t\t\t  m->num_packs);\n-\tfor (i = 0; i < m->num_packs; i++) {\n-\t\tchar *pack_name;\n-\t\tdisplay_progress(progress, i + 1);\n-\n-\t\tif (count[i])\n-\t\t\tcontinue;\n-\n-\t\tif (prepare_midx_pack(r, m, i))\n-\t\t\tcontinue;\n-\n-\t\tif (m->packs[i]->pack_keep || m->packs[i]->is_cruft)\n-\t\t\tcontinue;\n-\n-\t\tpack_name = xstrdup(m->packs[i]->pack_name);\n-\t\tclose_pack(m->packs[i]);\n-\n-\t\tstring_list_insert(&packs_to_drop, m->pack_names[i]);\n-\t\tunlink_pack_path(pack_name, 0);\n-\t\tfree(pack_name);\n-\t}\n-\tstop_progress(&progress);\n-\n-\tfree(count);\n-\n-\tif (packs_to_drop.nr)\n-\t\tresult = write_midx_internal(object_dir, NULL, &packs_to_drop, NULL, NULL, flags);\n-\n-\tstring_list_clear(&packs_to_drop, 0);\n-\n-\treturn result;\n-}\n-- \n2.44.0.290.g736be63234b\n\n"},{"id":"491474","messageId":"31d2e074fbeb81dc856a66fdaee455a7fa70e9c6.1711387439.git.me@ttaylorr.com","threadId":"61199","inReplyTo":"cover.1711387439.git.me@ttaylorr.com","subject":"[PATCH 05/11] midx: move `write_midx_file_only` to midx-write.c","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-25T17:24:32Z","receivedAt":"2024-03-25T17:24:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to move the last substantial function related to writing from\nmidx.c to to midx-write.c by first moving a thin wrapper around it.\n\nLike previous changes, this patch does not introduce any behavioral\nchanges and is best viewed with `--color-moved`.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx-write.c | 10 ++++++++++\n midx.c       | 10 ----------\n 2 files changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex d679e0a131..635e6af193 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -18,6 +18,16 @@ extern int write_midx_internal(const char *object_dir,\n extern struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n \t\t\t\t\t\t\tconst char *object_dir);\n \n+int write_midx_file_only(const char *object_dir,\n+\t\t\t struct string_list *packs_to_include,\n+\t\t\t const char *preferred_pack_name,\n+\t\t\t const char *refs_snapshot,\n+\t\t\t unsigned flags)\n+{\n+\treturn write_midx_internal(object_dir, packs_to_include, NULL,\n+\t\t\t\t   preferred_pack_name, refs_snapshot, flags);\n+}\n+\n int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags)\n {\n \tuint32_t i, *count, result = 0;\ndiff --git a/midx.c b/midx.c\nindex 5936bc5b9e..702eca805a 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1764,16 +1764,6 @@ int write_midx_file(const char *object_dir,\n \t\t\t\t   refs_snapshot, flags);\n }\n \n-int write_midx_file_only(const char *object_dir,\n-\t\t\t struct string_list *packs_to_include,\n-\t\t\t const char *preferred_pack_name,\n-\t\t\t const char *refs_snapshot,\n-\t\t\t unsigned flags)\n-{\n-\treturn write_midx_internal(object_dir, packs_to_include, NULL,\n-\t\t\t\t   preferred_pack_name, refs_snapshot, flags);\n-}\n-\n struct clear_midx_data {\n \tchar *keep;\n \tconst char *ext;\n-- \n2.44.0.290.g736be63234b\n\n"},{"id":"491475","messageId":"73977036d7a8cc8f84a5dbc54e462477d33f2c1e.1711387439.git.me@ttaylorr.com","threadId":"61199","inReplyTo":"cover.1711387439.git.me@ttaylorr.com","subject":"[PATCH 06/11] midx: move `write_midx_file` to midx-write.c","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-25T17:24:35Z","receivedAt":"2024-03-25T17:24:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to move the last substantial function related to writing from\nmidx.c to to midx-write.c by moving another thin wrapper around it.\n\nLike previous changes, this patch does not introduce any behavioral\nchanges and is best viewed with `--color-moved`.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx-write.c | 9 +++++++++\n midx.c       | 9 ---------\n 2 files changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 635e6af193..3d7697d8a2 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -18,6 +18,15 @@ extern int write_midx_internal(const char *object_dir,\n extern struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n \t\t\t\t\t\t\tconst char *object_dir);\n \n+int write_midx_file(const char *object_dir,\n+\t\t    const char *preferred_pack_name,\n+\t\t    const char *refs_snapshot,\n+\t\t    unsigned flags)\n+{\n+\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n+\t\t\t\t   refs_snapshot, flags);\n+}\n+\n int write_midx_file_only(const char *object_dir,\n \t\t\t struct string_list *packs_to_include,\n \t\t\t const char *preferred_pack_name,\ndiff --git a/midx.c b/midx.c\nindex 702eca805a..39b5c86736 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1755,15 +1755,6 @@ int write_midx_internal(const char *object_dir,\n \treturn result;\n }\n \n-int write_midx_file(const char *object_dir,\n-\t\t    const char *preferred_pack_name,\n-\t\t    const char *refs_snapshot,\n-\t\t    unsigned flags)\n-{\n-\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n-\t\t\t\t   refs_snapshot, flags);\n-}\n-\n struct clear_midx_data {\n \tchar *keep;\n \tconst char *ext;\n-- \n2.44.0.290.g736be63234b\n\n"},{"id":"491477","messageId":"e83ef66cf5945e4ff02ac09123c5857d58a898e6.1711387439.git.me@ttaylorr.com","threadId":"61199","inReplyTo":"cover.1711387439.git.me@ttaylorr.com","subject":"[PATCH 07/11] midx: move `write_midx_internal` (and related functions) to midx-write.c","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-25T17:24:38Z","receivedAt":"2024-03-25T17:24:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Move the last writing-related function from midx.c to midx-write.c. This\npatch moves `write_midx_internal()`, along with all of the functions\nused to implement it into midx-write.c.\n\nLike previous patch, this patch too does not introduce any functional\nchanges, and is best moved with `--color-moved`.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx-write.c | 1243 ++++++++++++++++++++++++++++++++++++++++++++++-\n midx.c       | 1296 +-------------------------------------------------\n midx.h       |   19 +\n 3 files changed, 1272 insertions(+), 1286 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 3d7697d8a2..c812156cbd 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1,22 +1,1257 @@\n #include \"git-compat-util.h\"\n+#include \"abspath.h\"\n #include \"config.h\"\n #include \"hex.h\"\n+#include \"lockfile.h\"\n #include \"packfile.h\"\n+#include \"object-file.h\"\n+#include \"hash-lookup.h\"\n #include \"midx.h\"\n #include \"progress.h\"\n+#include \"trace2.h\"\n #include \"run-command.h\"\n+#include \"chunk-format.h\"\n #include \"pack-bitmap.h\"\n+#include \"refs.h\"\n #include \"revision.h\"\n+#include \"list-objects.h\"\n \n-extern int write_midx_internal(const char *object_dir,\n+#define PACK_EXPIRED UINT_MAX\n+#define BITMAP_POS_UNKNOWN (~((uint32_t)0))\n+#define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n+#define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n+\n+extern int midx_checksum_valid(struct multi_pack_index *m);\n+extern void clear_midx_files_ext(const char *object_dir, const char *ext,\n+\t\t\t\t unsigned char *keep_hash);\n+extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n+\t\t\t\tconst char *idx_name);\n+\n+static size_t write_midx_header(struct hashfile *f,\n+\t\t\t\tunsigned char num_chunks,\n+\t\t\t\tuint32_t num_packs)\n+{\n+\thashwrite_be32(f, MIDX_SIGNATURE);\n+\thashwrite_u8(f, MIDX_VERSION);\n+\thashwrite_u8(f, oid_version(the_hash_algo));\n+\thashwrite_u8(f, num_chunks);\n+\thashwrite_u8(f, 0); /* unused */\n+\thashwrite_be32(f, num_packs);\n+\n+\treturn MIDX_HEADER_SIZE;\n+}\n+\n+struct pack_info {\n+\tuint32_t orig_pack_int_id;\n+\tchar *pack_name;\n+\tstruct packed_git *p;\n+\n+\tuint32_t bitmap_pos;\n+\tuint32_t bitmap_nr;\n+\n+\tunsigned expired : 1;\n+};\n+\n+static void fill_pack_info(struct pack_info *info,\n+\t\t\t   struct packed_git *p, const char *pack_name,\n+\t\t\t   uint32_t orig_pack_int_id)\n+{\n+\tmemset(info, 0, sizeof(struct pack_info));\n+\n+\tinfo->orig_pack_int_id = orig_pack_int_id;\n+\tinfo->pack_name = xstrdup(pack_name);\n+\tinfo->p = p;\n+\tinfo->bitmap_pos = BITMAP_POS_UNKNOWN;\n+}\n+\n+static int pack_info_compare(const void *_a, const void *_b)\n+{\n+\tstruct pack_info *a = (struct pack_info *)_a;\n+\tstruct pack_info *b = (struct pack_info *)_b;\n+\treturn strcmp(a->pack_name, b->pack_name);\n+}\n+\n+static int idx_or_pack_name_cmp(const void *_va, const void *_vb)\n+{\n+\tconst char *pack_name = _va;\n+\tconst struct pack_info *compar = _vb;\n+\n+\treturn cmp_idx_or_pack_name(pack_name, compar->pack_name);\n+}\n+\n+struct write_midx_context {\n+\tstruct pack_info *info;\n+\tsize_t nr;\n+\tsize_t alloc;\n+\tstruct multi_pack_index *m;\n+\tstruct progress *progress;\n+\tunsigned pack_paths_checked;\n+\n+\tstruct pack_midx_entry *entries;\n+\tsize_t entries_nr;\n+\n+\tuint32_t *pack_perm;\n+\tuint32_t *pack_order;\n+\tunsigned large_offsets_needed:1;\n+\tuint32_t num_large_offsets;\n+\n+\tint preferred_pack_idx;\n+\n+\tstruct string_list *to_include;\n+};\n+\n+static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n+\t\t\t     const char *file_name, void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tstruct packed_git *p;\n+\n+\tif (ends_with(file_name, \".idx\")) {\n+\t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n+\t\t/*\n+\t\t * Note that at most one of ctx->m and ctx->to_include are set,\n+\t\t * so we are testing midx_contains_pack() and\n+\t\t * string_list_has_string() independently (guarded by the\n+\t\t * appropriate NULL checks).\n+\t\t *\n+\t\t * We could support passing to_include while reusing an existing\n+\t\t * MIDX, but don't currently since the reuse process drags\n+\t\t * forward all packs from an existing MIDX (without checking\n+\t\t * whether or not they appear in the to_include list).\n+\t\t *\n+\t\t * If we added support for that, these next two conditional\n+\t\t * should be performed independently (likely checking\n+\t\t * to_include before the existing MIDX).\n+\t\t */\n+\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n+\t\t\treturn;\n+\t\telse if (ctx->to_include &&\n+\t\t\t !string_list_has_string(ctx->to_include, file_name))\n+\t\t\treturn;\n+\n+\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n+\n+\t\tp = add_packed_git(full_path, full_path_len, 0);\n+\t\tif (!p) {\n+\t\t\twarning(_(\"failed to add packfile '%s'\"),\n+\t\t\t\tfull_path);\n+\t\t\treturn;\n+\t\t}\n+\n+\t\tif (open_pack_index(p)) {\n+\t\t\twarning(_(\"failed to open pack-index '%s'\"),\n+\t\t\t\tfull_path);\n+\t\t\tclose_pack(p);\n+\t\t\tfree(p);\n+\t\t\treturn;\n+\t\t}\n+\n+\t\tfill_pack_info(&ctx->info[ctx->nr], p, file_name, ctx->nr);\n+\t\tctx->nr++;\n+\t}\n+}\n+\n+struct pack_midx_entry {\n+\tstruct object_id oid;\n+\tuint32_t pack_int_id;\n+\ttime_t pack_mtime;\n+\tuint64_t offset;\n+\tunsigned preferred : 1;\n+};\n+\n+static int midx_oid_compare(const void *_a, const void *_b)\n+{\n+\tconst struct pack_midx_entry *a = (const struct pack_midx_entry *)_a;\n+\tconst struct pack_midx_entry *b = (const struct pack_midx_entry *)_b;\n+\tint cmp = oidcmp(&a->oid, &b->oid);\n+\n+\tif (cmp)\n+\t\treturn cmp;\n+\n+\t/* Sort objects in a preferred pack first when multiple copies exist. */\n+\tif (a->preferred > b->preferred)\n+\t\treturn -1;\n+\tif (a->preferred < b->preferred)\n+\t\treturn 1;\n+\n+\tif (a->pack_mtime > b->pack_mtime)\n+\t\treturn -1;\n+\telse if (a->pack_mtime < b->pack_mtime)\n+\t\treturn 1;\n+\n+\treturn a->pack_int_id - b->pack_int_id;\n+}\n+\n+static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n+\t\t\t\t      struct pack_midx_entry *e,\n+\t\t\t\t      uint32_t pos)\n+{\n+\tif (pos >= m->num_objects)\n+\t\treturn 1;\n+\n+\tnth_midxed_object_oid(&e->oid, m, pos);\n+\te->pack_int_id = nth_midxed_pack_int_id(m, pos);\n+\te->offset = nth_midxed_offset(m, pos);\n+\n+\t/* consider objects in midx to be from \"old\" packs */\n+\te->pack_mtime = 0;\n+\treturn 0;\n+}\n+\n+static void fill_pack_entry(uint32_t pack_int_id,\n+\t\t\t    struct packed_git *p,\n+\t\t\t    uint32_t cur_object,\n+\t\t\t    struct pack_midx_entry *entry,\n+\t\t\t    int preferred)\n+{\n+\tif (nth_packed_object_id(&entry->oid, p, cur_object) < 0)\n+\t\tdie(_(\"failed to locate object %d in packfile\"), cur_object);\n+\n+\tentry->pack_int_id = pack_int_id;\n+\tentry->pack_mtime = p->mtime;\n+\n+\tentry->offset = nth_packed_object_offset(p, cur_object);\n+\tentry->preferred = !!preferred;\n+}\n+\n+struct midx_fanout {\n+\tstruct pack_midx_entry *entries;\n+\tsize_t nr, alloc;\n+};\n+\n+static void midx_fanout_grow(struct midx_fanout *fanout, size_t nr)\n+{\n+\tif (nr < fanout->nr)\n+\t\tBUG(\"negative growth in midx_fanout_grow() (%\"PRIuMAX\" < %\"PRIuMAX\")\",\n+\t\t    (uintmax_t)nr, (uintmax_t)fanout->nr);\n+\tALLOC_GROW(fanout->entries, nr, fanout->alloc);\n+}\n+\n+static void midx_fanout_sort(struct midx_fanout *fanout)\n+{\n+\tQSORT(fanout->entries, fanout->nr, midx_oid_compare);\n+}\n+\n+static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n+\t\t\t\t\tstruct multi_pack_index *m,\n+\t\t\t\t\tuint32_t cur_fanout,\n+\t\t\t\t\tint preferred_pack)\n+{\n+\tuint32_t start = 0, end;\n+\tuint32_t cur_object;\n+\n+\tif (cur_fanout)\n+\t\tstart = ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n+\tend = ntohl(m->chunk_oid_fanout[cur_fanout]);\n+\n+\tfor (cur_object = start; cur_object < end; cur_object++) {\n+\t\tif ((preferred_pack > -1) &&\n+\t\t    (preferred_pack == nth_midxed_pack_int_id(m, cur_object))) {\n+\t\t\t/*\n+\t\t\t * Objects from preferred packs are added\n+\t\t\t * separately.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tmidx_fanout_grow(fanout, fanout->nr + 1);\n+\t\tnth_midxed_pack_midx_entry(m,\n+\t\t\t\t\t   &fanout->entries[fanout->nr],\n+\t\t\t\t\t   cur_object);\n+\t\tfanout->entries[fanout->nr].preferred = 0;\n+\t\tfanout->nr++;\n+\t}\n+}\n+\n+static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout,\n+\t\t\t\t\tstruct pack_info *info,\n+\t\t\t\t\tuint32_t cur_pack,\n+\t\t\t\t\tint preferred,\n+\t\t\t\t\tuint32_t cur_fanout)\n+{\n+\tstruct packed_git *pack = info[cur_pack].p;\n+\tuint32_t start = 0, end;\n+\tuint32_t cur_object;\n+\n+\tif (cur_fanout)\n+\t\tstart = get_pack_fanout(pack, cur_fanout - 1);\n+\tend = get_pack_fanout(pack, cur_fanout);\n+\n+\tfor (cur_object = start; cur_object < end; cur_object++) {\n+\t\tmidx_fanout_grow(fanout, fanout->nr + 1);\n+\t\tfill_pack_entry(cur_pack,\n+\t\t\t\tinfo[cur_pack].p,\n+\t\t\t\tcur_object,\n+\t\t\t\t&fanout->entries[fanout->nr],\n+\t\t\t\tpreferred);\n+\t\tfanout->nr++;\n+\t}\n+}\n+\n+/*\n+ * It is possible to artificially get into a state where there are many\n+ * duplicate copies of objects. That can create high memory pressure if\n+ * we are to create a list of all objects before de-duplication. To reduce\n+ * this memory pressure without a significant performance drop, automatically\n+ * group objects by the first byte of their object id. Use the IDX fanout\n+ * tables to group the data, copy to a local array, then sort.\n+ *\n+ * Copy only the de-duplicated entries (selected by most-recent modified time\n+ * of a packfile containing the object).\n+ */\n+static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n+\t\t\t\t\t\t  struct pack_info *info,\n+\t\t\t\t\t\t  uint32_t nr_packs,\n+\t\t\t\t\t\t  size_t *nr_objects,\n+\t\t\t\t\t\t  int preferred_pack)\n+{\n+\tuint32_t cur_fanout, cur_pack, cur_object;\n+\tsize_t alloc_objects, total_objects = 0;\n+\tstruct midx_fanout fanout = { 0 };\n+\tstruct pack_midx_entry *deduplicated_entries = NULL;\n+\tuint32_t start_pack = m ? m->num_packs : 0;\n+\n+\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++)\n+\t\ttotal_objects = st_add(total_objects,\n+\t\t\t\t       info[cur_pack].p->num_objects);\n+\n+\t/*\n+\t * As we de-duplicate by fanout value, we expect the fanout\n+\t * slices to be evenly distributed, with some noise. Hence,\n+\t * allocate slightly more than one 256th.\n+\t */\n+\talloc_objects = fanout.alloc = total_objects > 3200 ? total_objects / 200 : 16;\n+\n+\tALLOC_ARRAY(fanout.entries, fanout.alloc);\n+\tALLOC_ARRAY(deduplicated_entries, alloc_objects);\n+\t*nr_objects = 0;\n+\n+\tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n+\t\tfanout.nr = 0;\n+\n+\t\tif (m)\n+\t\t\tmidx_fanout_add_midx_fanout(&fanout, m, cur_fanout,\n+\t\t\t\t\t\t    preferred_pack);\n+\n+\t\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) {\n+\t\t\tint preferred = cur_pack == preferred_pack;\n+\t\t\tmidx_fanout_add_pack_fanout(&fanout,\n+\t\t\t\t\t\t    info, cur_pack,\n+\t\t\t\t\t\t    preferred, cur_fanout);\n+\t\t}\n+\n+\t\tif (-1 < preferred_pack && preferred_pack < start_pack)\n+\t\t\tmidx_fanout_add_pack_fanout(&fanout, info,\n+\t\t\t\t\t\t    preferred_pack, 1,\n+\t\t\t\t\t\t    cur_fanout);\n+\n+\t\tmidx_fanout_sort(&fanout);\n+\n+\t\t/*\n+\t\t * The batch is now sorted by OID and then mtime (descending).\n+\t\t * Take only the first duplicate.\n+\t\t */\n+\t\tfor (cur_object = 0; cur_object < fanout.nr; cur_object++) {\n+\t\t\tif (cur_object && oideq(&fanout.entries[cur_object - 1].oid,\n+\t\t\t\t\t\t&fanout.entries[cur_object].oid))\n+\t\t\t\tcontinue;\n+\n+\t\t\tALLOC_GROW(deduplicated_entries, st_add(*nr_objects, 1),\n+\t\t\t\t   alloc_objects);\n+\t\t\tmemcpy(&deduplicated_entries[*nr_objects],\n+\t\t\t       &fanout.entries[cur_object],\n+\t\t\t       sizeof(struct pack_midx_entry));\n+\t\t\t(*nr_objects)++;\n+\t\t}\n+\t}\n+\n+\tfree(fanout.entries);\n+\treturn deduplicated_entries;\n+}\n+\n+static int write_midx_pack_names(struct hashfile *f, void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tuint32_t i;\n+\tunsigned char padding[MIDX_CHUNK_ALIGNMENT];\n+\tsize_t written = 0;\n+\n+\tfor (i = 0; i < ctx->nr; i++) {\n+\t\tsize_t writelen;\n+\n+\t\tif (ctx->info[i].expired)\n+\t\t\tcontinue;\n+\n+\t\tif (i && strcmp(ctx->info[i].pack_name, ctx->info[i - 1].pack_name) <= 0)\n+\t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n+\t\t\t    ctx->info[i - 1].pack_name,\n+\t\t\t    ctx->info[i].pack_name);\n+\n+\t\twritelen = strlen(ctx->info[i].pack_name) + 1;\n+\t\thashwrite(f, ctx->info[i].pack_name, writelen);\n+\t\twritten += writelen;\n+\t}\n+\n+\t/* add padding to be aligned */\n+\ti = MIDX_CHUNK_ALIGNMENT - (written % MIDX_CHUNK_ALIGNMENT);\n+\tif (i < MIDX_CHUNK_ALIGNMENT) {\n+\t\tmemset(padding, 0, sizeof(padding));\n+\t\thashwrite(f, padding, i);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int write_midx_bitmapped_packs(struct hashfile *f, void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tsize_t i;\n+\n+\tfor (i = 0; i < ctx->nr; i++) {\n+\t\tstruct pack_info *pack = &ctx->info[i];\n+\t\tif (pack->expired)\n+\t\t\tcontinue;\n+\n+\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN && pack->bitmap_nr)\n+\t\t\tBUG(\"pack '%s' has no bitmap position, but has %d bitmapped object(s)\",\n+\t\t\t    pack->pack_name, pack->bitmap_nr);\n+\n+\t\thashwrite_be32(f, pack->bitmap_pos);\n+\t\thashwrite_be32(f, pack->bitmap_nr);\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_midx_oid_fanout(struct hashfile *f,\n+\t\t\t\t void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n+\tstruct pack_midx_entry *last = ctx->entries + ctx->entries_nr;\n+\tuint32_t count = 0;\n+\tuint32_t i;\n+\n+\t/*\n+\t* Write the first-level table (the list is sorted,\n+\t* but we use a 256-entry lookup to be able to avoid\n+\t* having to do eight extra binary search iterations).\n+\t*/\n+\tfor (i = 0; i < 256; i++) {\n+\t\tstruct pack_midx_entry *next = list;\n+\n+\t\twhile (next < last && next->oid.hash[0] == i) {\n+\t\t\tcount++;\n+\t\t\tnext++;\n+\t\t}\n+\n+\t\thashwrite_be32(f, count);\n+\t\tlist = next;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int write_midx_oid_lookup(struct hashfile *f,\n+\t\t\t\t void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tunsigned char hash_len = the_hash_algo->rawsz;\n+\tstruct pack_midx_entry *list = ctx->entries;\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *obj = list++;\n+\n+\t\tif (i < ctx->entries_nr - 1) {\n+\t\t\tstruct pack_midx_entry *next = list;\n+\t\t\tif (oidcmp(&obj->oid, &next->oid) >= 0)\n+\t\t\t\tBUG(\"OIDs not in order: %s >= %s\",\n+\t\t\t\t    oid_to_hex(&obj->oid),\n+\t\t\t\t    oid_to_hex(&next->oid));\n+\t\t}\n+\n+\t\thashwrite(f, obj->oid.hash, (int)hash_len);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int write_midx_object_offsets(struct hashfile *f,\n+\t\t\t\t     void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n+\tuint32_t i, nr_large_offset = 0;\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *obj = list++;\n+\n+\t\tif (ctx->pack_perm[obj->pack_int_id] == PACK_EXPIRED)\n+\t\t\tBUG(\"object %s is in an expired pack with int-id %d\",\n+\t\t\t    oid_to_hex(&obj->oid),\n+\t\t\t    obj->pack_int_id);\n+\n+\t\thashwrite_be32(f, ctx->pack_perm[obj->pack_int_id]);\n+\n+\t\tif (ctx->large_offsets_needed && obj->offset >> 31)\n+\t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n+\t\telse if (!ctx->large_offsets_needed && obj->offset >> 32)\n+\t\t\tBUG(\"object %s requires a large offset (%\"PRIx64\") but the MIDX is not writing large offsets!\",\n+\t\t\t    oid_to_hex(&obj->oid),\n+\t\t\t    obj->offset);\n+\t\telse\n+\t\t\thashwrite_be32(f, (uint32_t)obj->offset);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int write_midx_large_offsets(struct hashfile *f,\n+\t\t\t\t    void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n+\tstruct pack_midx_entry *end = ctx->entries + ctx->entries_nr;\n+\tuint32_t nr_large_offset = ctx->num_large_offsets;\n+\n+\twhile (nr_large_offset) {\n+\t\tstruct pack_midx_entry *obj;\n+\t\tuint64_t offset;\n+\n+\t\tif (list >= end)\n+\t\t\tBUG(\"too many large-offset objects\");\n+\n+\t\tobj = list++;\n+\t\toffset = obj->offset;\n+\n+\t\tif (!(offset >> 31))\n+\t\t\tcontinue;\n+\n+\t\thashwrite_be64(f, offset);\n+\n+\t\tnr_large_offset--;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int write_midx_revindex(struct hashfile *f,\n+\t\t\t       void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++)\n+\t\thashwrite_be32(f, ctx->pack_order[i]);\n+\n+\treturn 0;\n+}\n+\n+struct midx_pack_order_data {\n+\tuint32_t nr;\n+\tuint32_t pack;\n+\toff_t offset;\n+};\n+\n+static int midx_pack_order_cmp(const void *va, const void *vb)\n+{\n+\tconst struct midx_pack_order_data *a = va, *b = vb;\n+\tif (a->pack < b->pack)\n+\t\treturn -1;\n+\telse if (a->pack > b->pack)\n+\t\treturn 1;\n+\telse if (a->offset < b->offset)\n+\t\treturn -1;\n+\telse if (a->offset > b->offset)\n+\t\treturn 1;\n+\telse\n+\t\treturn 0;\n+}\n+\n+static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n+{\n+\tstruct midx_pack_order_data *data;\n+\tuint32_t *pack_order;\n+\tuint32_t i;\n+\n+\ttrace2_region_enter(\"midx\", \"midx_pack_order\", the_repository);\n+\n+\tALLOC_ARRAY(data, ctx->entries_nr);\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *e = &ctx->entries[i];\n+\t\tdata[i].nr = i;\n+\t\tdata[i].pack = ctx->pack_perm[e->pack_int_id];\n+\t\tif (!e->preferred)\n+\t\t\tdata[i].pack |= (1U << 31);\n+\t\tdata[i].offset = e->offset;\n+\t}\n+\n+\tQSORT(data, ctx->entries_nr, midx_pack_order_cmp);\n+\n+\tALLOC_ARRAY(pack_order, ctx->entries_nr);\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *e = &ctx->entries[data[i].nr];\n+\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]];\n+\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n+\t\t\tpack->bitmap_pos = i;\n+\t\tpack->bitmap_nr++;\n+\t\tpack_order[i] = data[i].nr;\n+\t}\n+\tfor (i = 0; i < ctx->nr; i++) {\n+\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[i]];\n+\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n+\t\t\tpack->bitmap_pos = 0;\n+\t}\n+\tfree(data);\n+\n+\ttrace2_region_leave(\"midx\", \"midx_pack_order\", the_repository);\n+\n+\treturn pack_order;\n+}\n+\n+static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n+\t\t\t\t     struct write_midx_context *ctx)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *tmp_file;\n+\n+\ttrace2_region_enter(\"midx\", \"write_midx_reverse_index\", the_repository);\n+\n+\tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex(midx_hash));\n+\n+\ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n+\t\t\t\t\tmidx_hash, WRITE_REV);\n+\n+\tif (finalize_object_file(tmp_file, buf.buf))\n+\t\tdie(_(\"cannot store reverse index file\"));\n+\n+\tstrbuf_release(&buf);\n+\n+\ttrace2_region_leave(\"midx\", \"write_midx_reverse_index\", the_repository);\n+}\n+\n+static void prepare_midx_packing_data(struct packing_data *pdata,\n+\t\t\t\t      struct write_midx_context *ctx)\n+{\n+\tuint32_t i;\n+\n+\ttrace2_region_enter(\"midx\", \"prepare_midx_packing_data\", the_repository);\n+\n+\tmemset(pdata, 0, sizeof(struct packing_data));\n+\tprepare_packing_data(the_repository, pdata);\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n+\t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n+\n+\t\toe_set_in_pack(pdata, to,\n+\t\t\t       ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n+\t}\n+\n+\ttrace2_region_leave(\"midx\", \"prepare_midx_packing_data\", the_repository);\n+}\n+\n+static int add_ref_to_pending(const char *refname,\n+\t\t\t      const struct object_id *oid,\n+\t\t\t      int flag, void *cb_data)\n+{\n+\tstruct rev_info *revs = (struct rev_info*)cb_data;\n+\tstruct object_id peeled;\n+\tstruct object *object;\n+\n+\tif ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {\n+\t\twarning(\"symbolic ref is dangling: %s\", refname);\n+\t\treturn 0;\n+\t}\n+\n+\tif (!peel_iterated_oid(oid, &peeled))\n+\t\toid = &peeled;\n+\n+\tobject = parse_object_or_die(oid, refname);\n+\tif (object->type != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tadd_pending_object(revs, object, \"\");\n+\tif (bitmap_is_preferred_refname(revs->repo, refname))\n+\t\tobject->flags |= NEEDS_BITMAP;\n+\treturn 0;\n+}\n+\n+struct bitmap_commit_cb {\n+\tstruct commit **commits;\n+\tsize_t commits_nr, commits_alloc;\n+\n+\tstruct write_midx_context *ctx;\n+};\n+\n+static const struct object_id *bitmap_oid_access(size_t index,\n+\t\t\t\t\t\t const void *_entries)\n+{\n+\tconst struct pack_midx_entry *entries = _entries;\n+\treturn &entries[index].oid;\n+}\n+\n+static void bitmap_show_commit(struct commit *commit, void *_data)\n+{\n+\tstruct bitmap_commit_cb *data = _data;\n+\tint pos = oid_pos(&commit->object.oid, data->ctx->entries,\n+\t\t\t  data->ctx->entries_nr,\n+\t\t\t  bitmap_oid_access);\n+\tif (pos < 0)\n+\t\treturn;\n+\n+\tALLOC_GROW(data->commits, data->commits_nr + 1, data->commits_alloc);\n+\tdata->commits[data->commits_nr++] = commit;\n+}\n+\n+static int read_refs_snapshot(const char *refs_snapshot,\n+\t\t\t      struct rev_info *revs)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct object_id oid;\n+\tFILE *f = xfopen(refs_snapshot, \"r\");\n+\n+\twhile (strbuf_getline(&buf, f) != EOF) {\n+\t\tstruct object *object;\n+\t\tint preferred = 0;\n+\t\tchar *hex = buf.buf;\n+\t\tconst char *end = NULL;\n+\n+\t\tif (buf.len && *buf.buf == '+') {\n+\t\t\tpreferred = 1;\n+\t\t\thex = &buf.buf[1];\n+\t\t}\n+\n+\t\tif (parse_oid_hex(hex, &oid, &end) < 0)\n+\t\t\tdie(_(\"could not parse line: %s\"), buf.buf);\n+\t\tif (*end)\n+\t\t\tdie(_(\"malformed line: %s\"), buf.buf);\n+\n+\t\tobject = parse_object_or_die(&oid, NULL);\n+\t\tif (preferred)\n+\t\t\tobject->flags |= NEEDS_BITMAP;\n+\n+\t\tadd_pending_object(revs, object, \"\");\n+\t}\n+\n+\tfclose(f);\n+\tstrbuf_release(&buf);\n+\treturn 0;\n+}\n+static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p,\n+\t\t\t\t\t\t    const char *refs_snapshot,\n+\t\t\t\t\t\t    struct write_midx_context *ctx)\n+{\n+\tstruct rev_info revs;\n+\tstruct bitmap_commit_cb cb = {0};\n+\n+\ttrace2_region_enter(\"midx\", \"find_commits_for_midx_bitmap\",\n+\t\t\t    the_repository);\n+\n+\tcb.ctx = ctx;\n+\n+\trepo_init_revisions(the_repository, &revs, NULL);\n+\tif (refs_snapshot) {\n+\t\tread_refs_snapshot(refs_snapshot, &revs);\n+\t} else {\n+\t\tsetup_revisions(0, NULL, &revs, NULL);\n+\t\tfor_each_ref(add_ref_to_pending, &revs);\n+\t}\n+\n+\t/*\n+\t * Skipping promisor objects here is intentional, since it only excludes\n+\t * them from the list of reachable commits that we want to select from\n+\t * when computing the selection of MIDX'd commits to receive bitmaps.\n+\t *\n+\t * Reachability bitmaps do require that their objects be closed under\n+\t * reachability, but fetching any objects missing from promisors at this\n+\t * point is too late. But, if one of those objects can be reached from\n+\t * an another object that is included in the bitmap, then we will\n+\t * complain later that we don't have reachability closure (and fail\n+\t * appropriately).\n+\t */\n+\tfetch_if_missing = 0;\n+\trevs.exclude_promisor_objects = 1;\n+\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(_(\"revision walk setup failed\"));\n+\n+\ttraverse_commit_list(&revs, bitmap_show_commit, NULL, &cb);\n+\tif (indexed_commits_nr_p)\n+\t\t*indexed_commits_nr_p = cb.commits_nr;\n+\n+\trelease_revisions(&revs);\n+\n+\ttrace2_region_leave(\"midx\", \"find_commits_for_midx_bitmap\",\n+\t\t\t    the_repository);\n+\n+\treturn cb.commits;\n+}\n+\n+static int write_midx_bitmap(const char *midx_name,\n+\t\t\t     const unsigned char *midx_hash,\n+\t\t\t     struct packing_data *pdata,\n+\t\t\t     struct commit **commits,\n+\t\t\t     uint32_t commits_nr,\n+\t\t\t     uint32_t *pack_order,\n+\t\t\t     unsigned flags)\n+{\n+\tint ret, i;\n+\tuint16_t options = 0;\n+\tstruct pack_idx_entry **index;\n+\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name,\n+\t\t\t\t\thash_to_hex(midx_hash));\n+\n+\ttrace2_region_enter(\"midx\", \"write_midx_bitmap\", the_repository);\n+\n+\tif (flags & MIDX_WRITE_BITMAP_HASH_CACHE)\n+\t\toptions |= BITMAP_OPT_HASH_CACHE;\n+\n+\tif (flags & MIDX_WRITE_BITMAP_LOOKUP_TABLE)\n+\t\toptions |= BITMAP_OPT_LOOKUP_TABLE;\n+\n+\t/*\n+\t * Build the MIDX-order index based on pdata.objects (which is already\n+\t * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of\n+\t * this order).\n+\t */\n+\tALLOC_ARRAY(index, pdata->nr_objects);\n+\tfor (i = 0; i < pdata->nr_objects; i++)\n+\t\tindex[i] = &pdata->objects[i].idx;\n+\n+\tbitmap_writer_show_progress(flags & MIDX_PROGRESS);\n+\tbitmap_writer_build_type_index(pdata, index, pdata->nr_objects);\n+\n+\t/*\n+\t * bitmap_writer_finish expects objects in lex order, but pack_order\n+\t * gives us exactly that. use it directly instead of re-sorting the\n+\t * array.\n+\t *\n+\t * This changes the order of objects in 'index' between\n+\t * bitmap_writer_build_type_index and bitmap_writer_finish.\n+\t *\n+\t * The same re-ordering takes place in the single-pack bitmap code via\n+\t * write_idx_file(), which is called by finish_tmp_packfile(), which\n+\t * happens between bitmap_writer_build_type_index() and\n+\t * bitmap_writer_finish().\n+\t */\n+\tfor (i = 0; i < pdata->nr_objects; i++)\n+\t\tindex[pack_order[i]] = &pdata->objects[i].idx;\n+\n+\tbitmap_writer_select_commits(commits, commits_nr, -1);\n+\tret = bitmap_writer_build(pdata);\n+\tif (ret < 0)\n+\t\tgoto cleanup;\n+\n+\tbitmap_writer_set_checksum(midx_hash);\n+\tbitmap_writer_finish(index, pdata->nr_objects, bitmap_name, options);\n+\n+cleanup:\n+\tfree(index);\n+\tfree(bitmap_name);\n+\n+\ttrace2_region_leave(\"midx\", \"write_midx_bitmap\", the_repository);\n+\n+\treturn ret;\n+}\n+\n+static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n+\t\t\t\t\t\t\tconst char *object_dir)\n+{\n+\tstruct multi_pack_index *result = NULL;\n+\tstruct multi_pack_index *cur;\n+\tchar *obj_dir_real = real_pathdup(object_dir, 1);\n+\tstruct strbuf cur_path_real = STRBUF_INIT;\n+\n+\t/* Ensure the given object_dir is local, or a known alternate. */\n+\tfind_odb(r, obj_dir_real);\n+\n+\tfor (cur = get_multi_pack_index(r); cur; cur = cur->next) {\n+\t\tstrbuf_realpath(&cur_path_real, cur->object_dir, 1);\n+\t\tif (!strcmp(obj_dir_real, cur_path_real.buf)) {\n+\t\t\tresult = cur;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n+cleanup:\n+\tfree(obj_dir_real);\n+\tstrbuf_release(&cur_path_real);\n+\treturn result;\n+}\n+\n+static int write_midx_internal(const char *object_dir,\n \t\t\t       struct string_list *packs_to_include,\n \t\t\t       struct string_list *packs_to_drop,\n \t\t\t       const char *preferred_pack_name,\n \t\t\t       const char *refs_snapshot,\n-\t\t\t       unsigned flags);\n+\t\t\t       unsigned flags)\n+{\n+\tstruct strbuf midx_name = STRBUF_INIT;\n+\tunsigned char midx_hash[GIT_MAX_RAWSZ];\n+\tuint32_t i;\n+\tstruct hashfile *f = NULL;\n+\tstruct lock_file lk;\n+\tstruct write_midx_context ctx = { 0 };\n+\tint bitmapped_packs_concat_len = 0;\n+\tint pack_name_concat_len = 0;\n+\tint dropped_packs = 0;\n+\tint result = 0;\n+\tstruct chunkfile *cf;\n \n-extern struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n-\t\t\t\t\t\t\tconst char *object_dir);\n+\ttrace2_region_enter(\"midx\", \"write_midx_internal\", the_repository);\n+\n+\tget_midx_filename(&midx_name, object_dir);\n+\tif (safe_create_leading_directories(midx_name.buf))\n+\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n+\t\t\t  midx_name.buf);\n+\n+\tif (!packs_to_include) {\n+\t\t/*\n+\t\t * Only reference an existing MIDX when not filtering which\n+\t\t * packs to include, since all packs and objects are copied\n+\t\t * blindly from an existing MIDX if one is present.\n+\t\t */\n+\t\tctx.m = lookup_multi_pack_index(the_repository, object_dir);\n+\t}\n+\n+\tif (ctx.m && !midx_checksum_valid(ctx.m)) {\n+\t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n+\t\tctx.m = NULL;\n+\t}\n+\n+\tctx.nr = 0;\n+\tctx.alloc = ctx.m ? ctx.m->num_packs : 16;\n+\tctx.info = NULL;\n+\tALLOC_ARRAY(ctx.info, ctx.alloc);\n+\n+\tif (ctx.m) {\n+\t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n+\t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n+\n+\t\t\tif (flags & MIDX_WRITE_REV_INDEX) {\n+\t\t\t\t/*\n+\t\t\t\t * If generating a reverse index, need to have\n+\t\t\t\t * packed_git's loaded to compare their\n+\t\t\t\t * mtimes and object count.\n+\t\t\t\t */\n+\t\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n+\t\t\t\t\terror(_(\"could not load pack\"));\n+\t\t\t\t\tresult = 1;\n+\t\t\t\t\tgoto cleanup;\n+\t\t\t\t}\n+\n+\t\t\t\tif (open_pack_index(ctx.m->packs[i]))\n+\t\t\t\t\tdie(_(\"could not open index for %s\"),\n+\t\t\t\t\t    ctx.m->packs[i]->pack_name);\n+\t\t\t}\n+\n+\t\t\tfill_pack_info(&ctx.info[ctx.nr++], ctx.m->packs[i],\n+\t\t\t\t       ctx.m->pack_names[i], i);\n+\t\t}\n+\t}\n+\n+\tctx.pack_paths_checked = 0;\n+\tif (flags & MIDX_PROGRESS)\n+\t\tctx.progress = start_delayed_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n+\telse\n+\t\tctx.progress = NULL;\n+\n+\tctx.to_include = packs_to_include;\n+\n+\tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n+\tstop_progress(&ctx.progress);\n+\n+\tif ((ctx.m && ctx.nr == ctx.m->num_packs) &&\n+\t    !(packs_to_include || packs_to_drop)) {\n+\t\tstruct bitmap_index *bitmap_git;\n+\t\tint bitmap_exists;\n+\t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n+\n+\t\tbitmap_git = prepare_midx_bitmap_git(ctx.m);\n+\t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n+\t\tfree_bitmap_index(bitmap_git);\n+\n+\t\tif (bitmap_exists || !want_bitmap) {\n+\t\t\t/*\n+\t\t\t * The correct MIDX already exists, and so does a\n+\t\t\t * corresponding bitmap (or one wasn't requested).\n+\t\t\t */\n+\t\t\tif (!want_bitmap)\n+\t\t\t\tclear_midx_files_ext(object_dir, \".bitmap\",\n+\t\t\t\t\t\t     NULL);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n+\tif (preferred_pack_name) {\n+\t\tctx.preferred_pack_idx = -1;\n+\n+\t\tfor (i = 0; i < ctx.nr; i++) {\n+\t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n+\t\t\t\t\t\t  ctx.info[i].pack_name)) {\n+\t\t\t\tctx.preferred_pack_idx = i;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (ctx.preferred_pack_idx == -1)\n+\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n+\t\t\t\tpreferred_pack_name);\n+\t} else if (ctx.nr &&\n+\t\t   (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))) {\n+\t\tstruct packed_git *oldest = ctx.info[ctx.preferred_pack_idx].p;\n+\t\tctx.preferred_pack_idx = 0;\n+\n+\t\tif (packs_to_drop && packs_to_drop->nr)\n+\t\t\tBUG(\"cannot write a MIDX bitmap during expiration\");\n+\n+\t\t/*\n+\t\t * set a preferred pack when writing a bitmap to ensure that\n+\t\t * the pack from which the first object is selected in pseudo\n+\t\t * pack-order has all of its objects selected from that pack\n+\t\t * (and not another pack containing a duplicate)\n+\t\t */\n+\t\tfor (i = 1; i < ctx.nr; i++) {\n+\t\t\tstruct packed_git *p = ctx.info[i].p;\n+\n+\t\t\tif (!oldest->num_objects || p->mtime < oldest->mtime) {\n+\t\t\t\toldest = p;\n+\t\t\t\tctx.preferred_pack_idx = i;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!oldest->num_objects) {\n+\t\t\t/*\n+\t\t\t * If all packs are empty; unset the preferred index.\n+\t\t\t * This is acceptable since there will be no duplicate\n+\t\t\t * objects to resolve, so the preferred value doesn't\n+\t\t\t * matter.\n+\t\t\t */\n+\t\t\tctx.preferred_pack_idx = -1;\n+\t\t}\n+\t} else {\n+\t\t/*\n+\t\t * otherwise don't mark any pack as preferred to avoid\n+\t\t * interfering with expiration logic below\n+\t\t */\n+\t\tctx.preferred_pack_idx = -1;\n+\t}\n+\n+\tif (ctx.preferred_pack_idx > -1) {\n+\t\tstruct packed_git *preferred = ctx.info[ctx.preferred_pack_idx].p;\n+\t\tif (!preferred->num_objects) {\n+\t\t\terror(_(\"cannot select preferred pack %s with no objects\"),\n+\t\t\t      preferred->pack_name);\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n+\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,\n+\t\t\t\t\t ctx.preferred_pack_idx);\n+\n+\tctx.large_offsets_needed = 0;\n+\tfor (i = 0; i < ctx.entries_nr; i++) {\n+\t\tif (ctx.entries[i].offset > 0x7fffffff)\n+\t\t\tctx.num_large_offsets++;\n+\t\tif (ctx.entries[i].offset > 0xffffffff)\n+\t\t\tctx.large_offsets_needed = 1;\n+\t}\n+\n+\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n+\n+\tif (packs_to_drop && packs_to_drop->nr) {\n+\t\tint drop_index = 0;\n+\t\tint missing_drops = 0;\n+\n+\t\tfor (i = 0; i < ctx.nr && drop_index < packs_to_drop->nr; i++) {\n+\t\t\tint cmp = strcmp(ctx.info[i].pack_name,\n+\t\t\t\t\t packs_to_drop->items[drop_index].string);\n+\n+\t\t\tif (!cmp) {\n+\t\t\t\tdrop_index++;\n+\t\t\t\tctx.info[i].expired = 1;\n+\t\t\t} else if (cmp > 0) {\n+\t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n+\t\t\t\t      packs_to_drop->items[drop_index].string);\n+\t\t\t\tdrop_index++;\n+\t\t\t\tmissing_drops++;\n+\t\t\t\ti--;\n+\t\t\t} else {\n+\t\t\t\tctx.info[i].expired = 0;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (missing_drops) {\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * pack_perm stores a permutation between pack-int-ids from the\n+\t * previous multi-pack-index to the new one we are writing:\n+\t *\n+\t * pack_perm[old_id] = new_id\n+\t */\n+\tALLOC_ARRAY(ctx.pack_perm, ctx.nr);\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (ctx.info[i].expired) {\n+\t\t\tdropped_packs++;\n+\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n+\t\t} else {\n+\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (ctx.info[i].expired)\n+\t\t\tcontinue;\n+\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n+\t\tbitmapped_packs_concat_len += 2 * sizeof(uint32_t);\n+\t}\n+\n+\t/* Check that the preferred pack wasn't expired (if given). */\n+\tif (preferred_pack_name) {\n+\t\tstruct pack_info *preferred = bsearch(preferred_pack_name,\n+\t\t\t\t\t\t      ctx.info, ctx.nr,\n+\t\t\t\t\t\t      sizeof(*ctx.info),\n+\t\t\t\t\t\t      idx_or_pack_name_cmp);\n+\t\tif (preferred) {\n+\t\t\tuint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];\n+\t\t\tif (perm == PACK_EXPIRED)\n+\t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n+\t\t\t\t\tpreferred_pack_name);\n+\t\t}\n+\t}\n+\n+\tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n+\t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n+\t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n+\n+\thold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR);\n+\tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n+\n+\tif (ctx.nr - dropped_packs == 0) {\n+\t\terror(_(\"no pack files to index.\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (!ctx.entries_nr) {\n+\t\tif (flags & MIDX_WRITE_BITMAP)\n+\t\t\twarning(_(\"refusing to write multi-pack .bitmap without any objects\"));\n+\t\tflags &= ~(MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP);\n+\t}\n+\n+\tcf = init_chunkfile(f);\n+\n+\tadd_chunk(cf, MIDX_CHUNKID_PACKNAMES, pack_name_concat_len,\n+\t\t  write_midx_pack_names);\n+\tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT, MIDX_CHUNK_FANOUT_SIZE,\n+\t\t  write_midx_oid_fanout);\n+\tadd_chunk(cf, MIDX_CHUNKID_OIDLOOKUP,\n+\t\t  st_mult(ctx.entries_nr, the_hash_algo->rawsz),\n+\t\t  write_midx_oid_lookup);\n+\tadd_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS,\n+\t\t  st_mult(ctx.entries_nr, MIDX_CHUNK_OFFSET_WIDTH),\n+\t\t  write_midx_object_offsets);\n+\n+\tif (ctx.large_offsets_needed)\n+\t\tadd_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS,\n+\t\t\tst_mult(ctx.num_large_offsets,\n+\t\t\t\tMIDX_CHUNK_LARGE_OFFSET_WIDTH),\n+\t\t\twrite_midx_large_offsets);\n+\n+\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) {\n+\t\tctx.pack_order = midx_pack_order(&ctx);\n+\t\tadd_chunk(cf, MIDX_CHUNKID_REVINDEX,\n+\t\t\t  st_mult(ctx.entries_nr, sizeof(uint32_t)),\n+\t\t\t  write_midx_revindex);\n+\t\tadd_chunk(cf, MIDX_CHUNKID_BITMAPPEDPACKS,\n+\t\t\t  bitmapped_packs_concat_len,\n+\t\t\t  write_midx_bitmapped_packs);\n+\t}\n+\n+\twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n+\twrite_chunkfile(cf, &ctx);\n+\n+\tfinalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,\n+\t\t\t  CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n+\tfree_chunkfile(cf);\n+\n+\tif (flags & MIDX_WRITE_REV_INDEX &&\n+\t    git_env_bool(\"GIT_TEST_MIDX_WRITE_REV\", 0))\n+\t\twrite_midx_reverse_index(midx_name.buf, midx_hash, &ctx);\n+\n+\tif (flags & MIDX_WRITE_BITMAP) {\n+\t\tstruct packing_data pdata;\n+\t\tstruct commit **commits;\n+\t\tuint32_t commits_nr;\n+\n+\t\tif (!ctx.entries_nr)\n+\t\t\tBUG(\"cannot write a bitmap without any objects\");\n+\n+\t\tprepare_midx_packing_data(&pdata, &ctx);\n+\n+\t\tcommits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, &ctx);\n+\n+\t\t/*\n+\t\t * The previous steps translated the information from\n+\t\t * 'entries' into information suitable for constructing\n+\t\t * bitmaps. We no longer need that array, so clear it to\n+\t\t * reduce memory pressure.\n+\t\t */\n+\t\tFREE_AND_NULL(ctx.entries);\n+\t\tctx.entries_nr = 0;\n+\n+\t\tif (write_midx_bitmap(midx_name.buf, midx_hash, &pdata,\n+\t\t\t\t      commits, commits_nr, ctx.pack_order,\n+\t\t\t\t      flags) < 0) {\n+\t\t\terror(_(\"could not write multi-pack bitmap\"));\n+\t\t\tresult = 1;\n+\t\t\tclear_packing_data(&pdata);\n+\t\t\tfree(commits);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\n+\t\tclear_packing_data(&pdata);\n+\t\tfree(commits);\n+\t}\n+\t/*\n+\t * NOTE: Do not use ctx.entries beyond this point, since it might\n+\t * have been freed in the previous if block.\n+\t */\n+\n+\tif (ctx.m)\n+\t\tclose_object_store(the_repository->objects);\n+\n+\tif (commit_lock_file(&lk) < 0)\n+\t\tdie_errno(_(\"could not write multi-pack-index\"));\n+\n+\tclear_midx_files_ext(object_dir, \".bitmap\", midx_hash);\n+\tclear_midx_files_ext(object_dir, \".rev\", midx_hash);\n+\n+cleanup:\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (ctx.info[i].p) {\n+\t\t\tclose_pack(ctx.info[i].p);\n+\t\t\tfree(ctx.info[i].p);\n+\t\t}\n+\t\tfree(ctx.info[i].pack_name);\n+\t}\n+\n+\tfree(ctx.info);\n+\tfree(ctx.entries);\n+\tfree(ctx.pack_perm);\n+\tfree(ctx.pack_order);\n+\tstrbuf_release(&midx_name);\n+\n+\ttrace2_region_leave(\"midx\", \"write_midx_internal\", the_repository);\n+\n+\treturn result;\n+}\n \n int write_midx_file(const char *object_dir,\n \t\t    const char *preferred_pack_name,\ndiff --git a/midx.c b/midx.c\nindex 39b5c86736..ae3b49166c 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1,62 +1,22 @@\n #include \"git-compat-util.h\"\n-#include \"abspath.h\"\n #include \"config.h\"\n-#include \"csum-file.h\"\n #include \"dir.h\"\n-#include \"gettext.h\"\n #include \"hex.h\"\n-#include \"lockfile.h\"\n #include \"packfile.h\"\n #include \"object-file.h\"\n-#include \"object-store-ll.h\"\n #include \"hash-lookup.h\"\n #include \"midx.h\"\n #include \"progress.h\"\n #include \"trace2.h\"\n-#include \"run-command.h\"\n-#include \"repository.h\"\n #include \"chunk-format.h\"\n-#include \"pack.h\"\n #include \"pack-bitmap.h\"\n-#include \"refs.h\"\n-#include \"revision.h\"\n-#include \"list-objects.h\"\n #include \"pack-revindex.h\"\n \n-struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n-\t\t\t\t\t\t\tconst char *object_dir);\n-\n-int write_midx_internal(const char *object_dir,\n-\t\t\t       struct string_list *packs_to_include,\n-\t\t\t       struct string_list *packs_to_drop,\n-\t\t\t       const char *preferred_pack_name,\n-\t\t\t       const char *refs_snapshot,\n-\t\t\t       unsigned flags);\n-\n-#define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n-#define MIDX_VERSION 1\n-#define MIDX_BYTE_FILE_VERSION 4\n-#define MIDX_BYTE_HASH_VERSION 5\n-#define MIDX_BYTE_NUM_CHUNKS 6\n-#define MIDX_BYTE_NUM_PACKS 8\n-#define MIDX_HEADER_SIZE 12\n-#define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + the_hash_algo->rawsz)\n-\n-#define MIDX_CHUNK_ALIGNMENT 4\n-#define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n-#define MIDX_CHUNKID_BITMAPPEDPACKS 0x42544d50 /* \"BTMP\" */\n-#define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n-#define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n-#define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n-#define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n-#define MIDX_CHUNKID_REVINDEX 0x52494458 /* \"RIDX\" */\n-#define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n-#define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n-#define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n-#define MIDX_CHUNK_BITMAPPED_PACKS_WIDTH (2 * sizeof(uint32_t))\n-#define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n-\n-#define PACK_EXPIRED UINT_MAX\n+int midx_checksum_valid(struct multi_pack_index *m);\n+void clear_midx_files_ext(const char *object_dir, const char *ext,\n+\t\t\t  unsigned char *keep_hash);\n+int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n+\t\t\t const char *idx_name);\n \n const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n {\n@@ -125,6 +85,8 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+#define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + the_hash_algo->rawsz)\n+\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local)\n {\n \tstruct multi_pack_index *m = NULL;\n@@ -304,6 +266,8 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t\n \treturn 0;\n }\n \n+#define MIDX_CHUNK_BITMAPPED_PACKS_WIDTH (2 * sizeof(uint32_t))\n+\n int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n \t\t       struct bitmapped_pack *bp, uint32_t pack_int_id)\n {\n@@ -410,8 +374,8 @@ int fill_midx_entry(struct repository *r,\n }\n \n /* Match \"foo.idx\" against either \"foo.pack\" _or_ \"foo.idx\". */\n-static int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n-\t\t\t\tconst char *idx_name)\n+int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n+\t\t\t const char *idx_name)\n {\n \t/* Skip past any initial matching prefix. */\n \twhile (*idx_name && *idx_name == *idx_or_pack_name) {\n@@ -518,1243 +482,11 @@ int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, i\n \treturn 0;\n }\n \n-static size_t write_midx_header(struct hashfile *f,\n-\t\t\t\tunsigned char num_chunks,\n-\t\t\t\tuint32_t num_packs)\n-{\n-\thashwrite_be32(f, MIDX_SIGNATURE);\n-\thashwrite_u8(f, MIDX_VERSION);\n-\thashwrite_u8(f, oid_version(the_hash_algo));\n-\thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0); /* unused */\n-\thashwrite_be32(f, num_packs);\n-\n-\treturn MIDX_HEADER_SIZE;\n-}\n-\n-#define BITMAP_POS_UNKNOWN (~((uint32_t)0))\n-\n-struct pack_info {\n-\tuint32_t orig_pack_int_id;\n-\tchar *pack_name;\n-\tstruct packed_git *p;\n-\n-\tuint32_t bitmap_pos;\n-\tuint32_t bitmap_nr;\n-\n-\tunsigned expired : 1;\n-};\n-\n-static void fill_pack_info(struct pack_info *info,\n-\t\t\t   struct packed_git *p, const char *pack_name,\n-\t\t\t   uint32_t orig_pack_int_id)\n-{\n-\tmemset(info, 0, sizeof(struct pack_info));\n-\n-\tinfo->orig_pack_int_id = orig_pack_int_id;\n-\tinfo->pack_name = xstrdup(pack_name);\n-\tinfo->p = p;\n-\tinfo->bitmap_pos = BITMAP_POS_UNKNOWN;\n-}\n-\n-static int pack_info_compare(const void *_a, const void *_b)\n-{\n-\tstruct pack_info *a = (struct pack_info *)_a;\n-\tstruct pack_info *b = (struct pack_info *)_b;\n-\treturn strcmp(a->pack_name, b->pack_name);\n-}\n-\n-static int idx_or_pack_name_cmp(const void *_va, const void *_vb)\n-{\n-\tconst char *pack_name = _va;\n-\tconst struct pack_info *compar = _vb;\n-\n-\treturn cmp_idx_or_pack_name(pack_name, compar->pack_name);\n-}\n-\n-struct write_midx_context {\n-\tstruct pack_info *info;\n-\tsize_t nr;\n-\tsize_t alloc;\n-\tstruct multi_pack_index *m;\n-\tstruct progress *progress;\n-\tunsigned pack_paths_checked;\n-\n-\tstruct pack_midx_entry *entries;\n-\tsize_t entries_nr;\n-\n-\tuint32_t *pack_perm;\n-\tuint32_t *pack_order;\n-\tunsigned large_offsets_needed:1;\n-\tuint32_t num_large_offsets;\n-\n-\tint preferred_pack_idx;\n-\n-\tstruct string_list *to_include;\n-};\n-\n-static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n-\t\t\t     const char *file_name, void *data)\n-{\n-\tstruct write_midx_context *ctx = data;\n-\tstruct packed_git *p;\n-\n-\tif (ends_with(file_name, \".idx\")) {\n-\t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n-\t\t/*\n-\t\t * Note that at most one of ctx->m and ctx->to_include are set,\n-\t\t * so we are testing midx_contains_pack() and\n-\t\t * string_list_has_string() independently (guarded by the\n-\t\t * appropriate NULL checks).\n-\t\t *\n-\t\t * We could support passing to_include while reusing an existing\n-\t\t * MIDX, but don't currently since the reuse process drags\n-\t\t * forward all packs from an existing MIDX (without checking\n-\t\t * whether or not they appear in the to_include list).\n-\t\t *\n-\t\t * If we added support for that, these next two conditional\n-\t\t * should be performed independently (likely checking\n-\t\t * to_include before the existing MIDX).\n-\t\t */\n-\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n-\t\t\treturn;\n-\t\telse if (ctx->to_include &&\n-\t\t\t !string_list_has_string(ctx->to_include, file_name))\n-\t\t\treturn;\n-\n-\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n-\n-\t\tp = add_packed_git(full_path, full_path_len, 0);\n-\t\tif (!p) {\n-\t\t\twarning(_(\"failed to add packfile '%s'\"),\n-\t\t\t\tfull_path);\n-\t\t\treturn;\n-\t\t}\n-\n-\t\tif (open_pack_index(p)) {\n-\t\t\twarning(_(\"failed to open pack-index '%s'\"),\n-\t\t\t\tfull_path);\n-\t\t\tclose_pack(p);\n-\t\t\tfree(p);\n-\t\t\treturn;\n-\t\t}\n-\n-\t\tfill_pack_info(&ctx->info[ctx->nr], p, file_name, ctx->nr);\n-\t\tctx->nr++;\n-\t}\n-}\n-\n-struct pack_midx_entry {\n-\tstruct object_id oid;\n-\tuint32_t pack_int_id;\n-\ttime_t pack_mtime;\n-\tuint64_t offset;\n-\tunsigned preferred : 1;\n-};\n-\n-static int midx_oid_compare(const void *_a, const void *_b)\n-{\n-\tconst struct pack_midx_entry *a = (const struct pack_midx_entry *)_a;\n-\tconst struct pack_midx_entry *b = (const struct pack_midx_entry *)_b;\n-\tint cmp = oidcmp(&a->oid, &b->oid);\n-\n-\tif (cmp)\n-\t\treturn cmp;\n-\n-\t/* Sort objects in a preferred pack first when multiple copies exist. */\n-\tif (a->preferred > b->preferred)\n-\t\treturn -1;\n-\tif (a->preferred < b->preferred)\n-\t\treturn 1;\n-\n-\tif (a->pack_mtime > b->pack_mtime)\n-\t\treturn -1;\n-\telse if (a->pack_mtime < b->pack_mtime)\n-\t\treturn 1;\n-\n-\treturn a->pack_int_id - b->pack_int_id;\n-}\n-\n-static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n-\t\t\t\t      struct pack_midx_entry *e,\n-\t\t\t\t      uint32_t pos)\n-{\n-\tif (pos >= m->num_objects)\n-\t\treturn 1;\n-\n-\tnth_midxed_object_oid(&e->oid, m, pos);\n-\te->pack_int_id = nth_midxed_pack_int_id(m, pos);\n-\te->offset = nth_midxed_offset(m, pos);\n-\n-\t/* consider objects in midx to be from \"old\" packs */\n-\te->pack_mtime = 0;\n-\treturn 0;\n-}\n-\n-static void fill_pack_entry(uint32_t pack_int_id,\n-\t\t\t    struct packed_git *p,\n-\t\t\t    uint32_t cur_object,\n-\t\t\t    struct pack_midx_entry *entry,\n-\t\t\t    int preferred)\n-{\n-\tif (nth_packed_object_id(&entry->oid, p, cur_object) < 0)\n-\t\tdie(_(\"failed to locate object %d in packfile\"), cur_object);\n-\n-\tentry->pack_int_id = pack_int_id;\n-\tentry->pack_mtime = p->mtime;\n-\n-\tentry->offset = nth_packed_object_offset(p, cur_object);\n-\tentry->preferred = !!preferred;\n-}\n-\n-struct midx_fanout {\n-\tstruct pack_midx_entry *entries;\n-\tsize_t nr, alloc;\n-};\n-\n-static void midx_fanout_grow(struct midx_fanout *fanout, size_t nr)\n-{\n-\tif (nr < fanout->nr)\n-\t\tBUG(\"negative growth in midx_fanout_grow() (%\"PRIuMAX\" < %\"PRIuMAX\")\",\n-\t\t    (uintmax_t)nr, (uintmax_t)fanout->nr);\n-\tALLOC_GROW(fanout->entries, nr, fanout->alloc);\n-}\n-\n-static void midx_fanout_sort(struct midx_fanout *fanout)\n-{\n-\tQSORT(fanout->entries, fanout->nr, midx_oid_compare);\n-}\n-\n-static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n-\t\t\t\t\tstruct multi_pack_index *m,\n-\t\t\t\t\tuint32_t cur_fanout,\n-\t\t\t\t\tint preferred_pack)\n-{\n-\tuint32_t start = 0, end;\n-\tuint32_t cur_object;\n-\n-\tif (cur_fanout)\n-\t\tstart = ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n-\tend = ntohl(m->chunk_oid_fanout[cur_fanout]);\n-\n-\tfor (cur_object = start; cur_object < end; cur_object++) {\n-\t\tif ((preferred_pack > -1) &&\n-\t\t    (preferred_pack == nth_midxed_pack_int_id(m, cur_object))) {\n-\t\t\t/*\n-\t\t\t * Objects from preferred packs are added\n-\t\t\t * separately.\n-\t\t\t */\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tmidx_fanout_grow(fanout, fanout->nr + 1);\n-\t\tnth_midxed_pack_midx_entry(m,\n-\t\t\t\t\t   &fanout->entries[fanout->nr],\n-\t\t\t\t\t   cur_object);\n-\t\tfanout->entries[fanout->nr].preferred = 0;\n-\t\tfanout->nr++;\n-\t}\n-}\n-\n-static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout,\n-\t\t\t\t\tstruct pack_info *info,\n-\t\t\t\t\tuint32_t cur_pack,\n-\t\t\t\t\tint preferred,\n-\t\t\t\t\tuint32_t cur_fanout)\n-{\n-\tstruct packed_git *pack = info[cur_pack].p;\n-\tuint32_t start = 0, end;\n-\tuint32_t cur_object;\n-\n-\tif (cur_fanout)\n-\t\tstart = get_pack_fanout(pack, cur_fanout - 1);\n-\tend = get_pack_fanout(pack, cur_fanout);\n-\n-\tfor (cur_object = start; cur_object < end; cur_object++) {\n-\t\tmidx_fanout_grow(fanout, fanout->nr + 1);\n-\t\tfill_pack_entry(cur_pack,\n-\t\t\t\tinfo[cur_pack].p,\n-\t\t\t\tcur_object,\n-\t\t\t\t&fanout->entries[fanout->nr],\n-\t\t\t\tpreferred);\n-\t\tfanout->nr++;\n-\t}\n-}\n-\n-/*\n- * It is possible to artificially get into a state where there are many\n- * duplicate copies of objects. That can create high memory pressure if\n- * we are to create a list of all objects before de-duplication. To reduce\n- * this memory pressure without a significant performance drop, automatically\n- * group objects by the first byte of their object id. Use the IDX fanout\n- * tables to group the data, copy to a local array, then sort.\n- *\n- * Copy only the de-duplicated entries (selected by most-recent modified time\n- * of a packfile containing the object).\n- */\n-static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n-\t\t\t\t\t\t  struct pack_info *info,\n-\t\t\t\t\t\t  uint32_t nr_packs,\n-\t\t\t\t\t\t  size_t *nr_objects,\n-\t\t\t\t\t\t  int preferred_pack)\n-{\n-\tuint32_t cur_fanout, cur_pack, cur_object;\n-\tsize_t alloc_objects, total_objects = 0;\n-\tstruct midx_fanout fanout = { 0 };\n-\tstruct pack_midx_entry *deduplicated_entries = NULL;\n-\tuint32_t start_pack = m ? m->num_packs : 0;\n-\n-\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++)\n-\t\ttotal_objects = st_add(total_objects,\n-\t\t\t\t       info[cur_pack].p->num_objects);\n-\n-\t/*\n-\t * As we de-duplicate by fanout value, we expect the fanout\n-\t * slices to be evenly distributed, with some noise. Hence,\n-\t * allocate slightly more than one 256th.\n-\t */\n-\talloc_objects = fanout.alloc = total_objects > 3200 ? total_objects / 200 : 16;\n-\n-\tALLOC_ARRAY(fanout.entries, fanout.alloc);\n-\tALLOC_ARRAY(deduplicated_entries, alloc_objects);\n-\t*nr_objects = 0;\n-\n-\tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n-\t\tfanout.nr = 0;\n-\n-\t\tif (m)\n-\t\t\tmidx_fanout_add_midx_fanout(&fanout, m, cur_fanout,\n-\t\t\t\t\t\t    preferred_pack);\n-\n-\t\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) {\n-\t\t\tint preferred = cur_pack == preferred_pack;\n-\t\t\tmidx_fanout_add_pack_fanout(&fanout,\n-\t\t\t\t\t\t    info, cur_pack,\n-\t\t\t\t\t\t    preferred, cur_fanout);\n-\t\t}\n-\n-\t\tif (-1 < preferred_pack && preferred_pack < start_pack)\n-\t\t\tmidx_fanout_add_pack_fanout(&fanout, info,\n-\t\t\t\t\t\t    preferred_pack, 1,\n-\t\t\t\t\t\t    cur_fanout);\n-\n-\t\tmidx_fanout_sort(&fanout);\n-\n-\t\t/*\n-\t\t * The batch is now sorted by OID and then mtime (descending).\n-\t\t * Take only the first duplicate.\n-\t\t */\n-\t\tfor (cur_object = 0; cur_object < fanout.nr; cur_object++) {\n-\t\t\tif (cur_object && oideq(&fanout.entries[cur_object - 1].oid,\n-\t\t\t\t\t\t&fanout.entries[cur_object].oid))\n-\t\t\t\tcontinue;\n-\n-\t\t\tALLOC_GROW(deduplicated_entries, st_add(*nr_objects, 1),\n-\t\t\t\t   alloc_objects);\n-\t\t\tmemcpy(&deduplicated_entries[*nr_objects],\n-\t\t\t       &fanout.entries[cur_object],\n-\t\t\t       sizeof(struct pack_midx_entry));\n-\t\t\t(*nr_objects)++;\n-\t\t}\n-\t}\n-\n-\tfree(fanout.entries);\n-\treturn deduplicated_entries;\n-}\n-\n-static int write_midx_pack_names(struct hashfile *f, void *data)\n-{\n-\tstruct write_midx_context *ctx = data;\n-\tuint32_t i;\n-\tunsigned char padding[MIDX_CHUNK_ALIGNMENT];\n-\tsize_t written = 0;\n-\n-\tfor (i = 0; i < ctx->nr; i++) {\n-\t\tsize_t writelen;\n-\n-\t\tif (ctx->info[i].expired)\n-\t\t\tcontinue;\n-\n-\t\tif (i && strcmp(ctx->info[i].pack_name, ctx->info[i - 1].pack_name) <= 0)\n-\t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n-\t\t\t    ctx->info[i - 1].pack_name,\n-\t\t\t    ctx->info[i].pack_name);\n-\n-\t\twritelen = strlen(ctx->info[i].pack_name) + 1;\n-\t\thashwrite(f, ctx->info[i].pack_name, writelen);\n-\t\twritten += writelen;\n-\t}\n-\n-\t/* add padding to be aligned */\n-\ti = MIDX_CHUNK_ALIGNMENT - (written % MIDX_CHUNK_ALIGNMENT);\n-\tif (i < MIDX_CHUNK_ALIGNMENT) {\n-\t\tmemset(padding, 0, sizeof(padding));\n-\t\thashwrite(f, padding, i);\n-\t}\n-\n-\treturn 0;\n-}\n-\n-static int write_midx_bitmapped_packs(struct hashfile *f, void *data)\n-{\n-\tstruct write_midx_context *ctx = data;\n-\tsize_t i;\n-\n-\tfor (i = 0; i < ctx->nr; i++) {\n-\t\tstruct pack_info *pack = &ctx->info[i];\n-\t\tif (pack->expired)\n-\t\t\tcontinue;\n-\n-\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN && pack->bitmap_nr)\n-\t\t\tBUG(\"pack '%s' has no bitmap position, but has %d bitmapped object(s)\",\n-\t\t\t    pack->pack_name, pack->bitmap_nr);\n-\n-\t\thashwrite_be32(f, pack->bitmap_pos);\n-\t\thashwrite_be32(f, pack->bitmap_nr);\n-\t}\n-\treturn 0;\n-}\n-\n-static int write_midx_oid_fanout(struct hashfile *f,\n-\t\t\t\t void *data)\n-{\n-\tstruct write_midx_context *ctx = data;\n-\tstruct pack_midx_entry *list = ctx->entries;\n-\tstruct pack_midx_entry *last = ctx->entries + ctx->entries_nr;\n-\tuint32_t count = 0;\n-\tuint32_t i;\n-\n-\t/*\n-\t* Write the first-level table (the list is sorted,\n-\t* but we use a 256-entry lookup to be able to avoid\n-\t* having to do eight extra binary search iterations).\n-\t*/\n-\tfor (i = 0; i < 256; i++) {\n-\t\tstruct pack_midx_entry *next = list;\n-\n-\t\twhile (next < last && next->oid.hash[0] == i) {\n-\t\t\tcount++;\n-\t\t\tnext++;\n-\t\t}\n-\n-\t\thashwrite_be32(f, count);\n-\t\tlist = next;\n-\t}\n-\n-\treturn 0;\n-}\n-\n-static int write_midx_oid_lookup(struct hashfile *f,\n-\t\t\t\t void *data)\n-{\n-\tstruct write_midx_context *ctx = data;\n-\tunsigned char hash_len = the_hash_algo->rawsz;\n-\tstruct pack_midx_entry *list = ctx->entries;\n-\tuint32_t i;\n-\n-\tfor (i = 0; i < ctx->entries_nr; i++) {\n-\t\tstruct pack_midx_entry *obj = list++;\n-\n-\t\tif (i < ctx->entries_nr - 1) {\n-\t\t\tstruct pack_midx_entry *next = list;\n-\t\t\tif (oidcmp(&obj->oid, &next->oid) >= 0)\n-\t\t\t\tBUG(\"OIDs not in order: %s >= %s\",\n-\t\t\t\t    oid_to_hex(&obj->oid),\n-\t\t\t\t    oid_to_hex(&next->oid));\n-\t\t}\n-\n-\t\thashwrite(f, obj->oid.hash, (int)hash_len);\n-\t}\n-\n-\treturn 0;\n-}\n-\n-static int write_midx_object_offsets(struct hashfile *f,\n-\t\t\t\t     void *data)\n-{\n-\tstruct write_midx_context *ctx = data;\n-\tstruct pack_midx_entry *list = ctx->entries;\n-\tuint32_t i, nr_large_offset = 0;\n-\n-\tfor (i = 0; i < ctx->entries_nr; i++) {\n-\t\tstruct pack_midx_entry *obj = list++;\n-\n-\t\tif (ctx->pack_perm[obj->pack_int_id] == PACK_EXPIRED)\n-\t\t\tBUG(\"object %s is in an expired pack with int-id %d\",\n-\t\t\t    oid_to_hex(&obj->oid),\n-\t\t\t    obj->pack_int_id);\n-\n-\t\thashwrite_be32(f, ctx->pack_perm[obj->pack_int_id]);\n-\n-\t\tif (ctx->large_offsets_needed && obj->offset >> 31)\n-\t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n-\t\telse if (!ctx->large_offsets_needed && obj->offset >> 32)\n-\t\t\tBUG(\"object %s requires a large offset (%\"PRIx64\") but the MIDX is not writing large offsets!\",\n-\t\t\t    oid_to_hex(&obj->oid),\n-\t\t\t    obj->offset);\n-\t\telse\n-\t\t\thashwrite_be32(f, (uint32_t)obj->offset);\n-\t}\n-\n-\treturn 0;\n-}\n-\n-static int write_midx_large_offsets(struct hashfile *f,\n-\t\t\t\t    void *data)\n-{\n-\tstruct write_midx_context *ctx = data;\n-\tstruct pack_midx_entry *list = ctx->entries;\n-\tstruct pack_midx_entry *end = ctx->entries + ctx->entries_nr;\n-\tuint32_t nr_large_offset = ctx->num_large_offsets;\n-\n-\twhile (nr_large_offset) {\n-\t\tstruct pack_midx_entry *obj;\n-\t\tuint64_t offset;\n-\n-\t\tif (list >= end)\n-\t\t\tBUG(\"too many large-offset objects\");\n-\n-\t\tobj = list++;\n-\t\toffset = obj->offset;\n-\n-\t\tif (!(offset >> 31))\n-\t\t\tcontinue;\n-\n-\t\thashwrite_be64(f, offset);\n-\n-\t\tnr_large_offset--;\n-\t}\n-\n-\treturn 0;\n-}\n-\n-static int write_midx_revindex(struct hashfile *f,\n-\t\t\t       void *data)\n-{\n-\tstruct write_midx_context *ctx = data;\n-\tuint32_t i;\n-\n-\tfor (i = 0; i < ctx->entries_nr; i++)\n-\t\thashwrite_be32(f, ctx->pack_order[i]);\n-\n-\treturn 0;\n-}\n-\n-struct midx_pack_order_data {\n-\tuint32_t nr;\n-\tuint32_t pack;\n-\toff_t offset;\n-};\n-\n-static int midx_pack_order_cmp(const void *va, const void *vb)\n-{\n-\tconst struct midx_pack_order_data *a = va, *b = vb;\n-\tif (a->pack < b->pack)\n-\t\treturn -1;\n-\telse if (a->pack > b->pack)\n-\t\treturn 1;\n-\telse if (a->offset < b->offset)\n-\t\treturn -1;\n-\telse if (a->offset > b->offset)\n-\t\treturn 1;\n-\telse\n-\t\treturn 0;\n-}\n-\n-static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n-{\n-\tstruct midx_pack_order_data *data;\n-\tuint32_t *pack_order;\n-\tuint32_t i;\n-\n-\ttrace2_region_enter(\"midx\", \"midx_pack_order\", the_repository);\n-\n-\tALLOC_ARRAY(data, ctx->entries_nr);\n-\tfor (i = 0; i < ctx->entries_nr; i++) {\n-\t\tstruct pack_midx_entry *e = &ctx->entries[i];\n-\t\tdata[i].nr = i;\n-\t\tdata[i].pack = ctx->pack_perm[e->pack_int_id];\n-\t\tif (!e->preferred)\n-\t\t\tdata[i].pack |= (1U << 31);\n-\t\tdata[i].offset = e->offset;\n-\t}\n-\n-\tQSORT(data, ctx->entries_nr, midx_pack_order_cmp);\n-\n-\tALLOC_ARRAY(pack_order, ctx->entries_nr);\n-\tfor (i = 0; i < ctx->entries_nr; i++) {\n-\t\tstruct pack_midx_entry *e = &ctx->entries[data[i].nr];\n-\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]];\n-\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n-\t\t\tpack->bitmap_pos = i;\n-\t\tpack->bitmap_nr++;\n-\t\tpack_order[i] = data[i].nr;\n-\t}\n-\tfor (i = 0; i < ctx->nr; i++) {\n-\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[i]];\n-\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n-\t\t\tpack->bitmap_pos = 0;\n-\t}\n-\tfree(data);\n-\n-\ttrace2_region_leave(\"midx\", \"midx_pack_order\", the_repository);\n-\n-\treturn pack_order;\n-}\n-\n-static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n-\t\t\t\t     struct write_midx_context *ctx)\n-{\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tconst char *tmp_file;\n-\n-\ttrace2_region_enter(\"midx\", \"write_midx_reverse_index\", the_repository);\n-\n-\tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex(midx_hash));\n-\n-\ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n-\t\t\t\t\tmidx_hash, WRITE_REV);\n-\n-\tif (finalize_object_file(tmp_file, buf.buf))\n-\t\tdie(_(\"cannot store reverse index file\"));\n-\n-\tstrbuf_release(&buf);\n-\n-\ttrace2_region_leave(\"midx\", \"write_midx_reverse_index\", the_repository);\n-}\n-\n-static void clear_midx_files_ext(const char *object_dir, const char *ext,\n-\t\t\t\t unsigned char *keep_hash);\n-\n-static int midx_checksum_valid(struct multi_pack_index *m)\n+int midx_checksum_valid(struct multi_pack_index *m)\n {\n \treturn hashfile_checksum_valid(m->data, m->data_len);\n }\n \n-static void prepare_midx_packing_data(struct packing_data *pdata,\n-\t\t\t\t      struct write_midx_context *ctx)\n-{\n-\tuint32_t i;\n-\n-\ttrace2_region_enter(\"midx\", \"prepare_midx_packing_data\", the_repository);\n-\n-\tmemset(pdata, 0, sizeof(struct packing_data));\n-\tprepare_packing_data(the_repository, pdata);\n-\n-\tfor (i = 0; i < ctx->entries_nr; i++) {\n-\t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n-\t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n-\n-\t\toe_set_in_pack(pdata, to,\n-\t\t\t       ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n-\t}\n-\n-\ttrace2_region_leave(\"midx\", \"prepare_midx_packing_data\", the_repository);\n-}\n-\n-static int add_ref_to_pending(const char *refname,\n-\t\t\t      const struct object_id *oid,\n-\t\t\t      int flag, void *cb_data)\n-{\n-\tstruct rev_info *revs = (struct rev_info*)cb_data;\n-\tstruct object_id peeled;\n-\tstruct object *object;\n-\n-\tif ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {\n-\t\twarning(\"symbolic ref is dangling: %s\", refname);\n-\t\treturn 0;\n-\t}\n-\n-\tif (!peel_iterated_oid(oid, &peeled))\n-\t\toid = &peeled;\n-\n-\tobject = parse_object_or_die(oid, refname);\n-\tif (object->type != OBJ_COMMIT)\n-\t\treturn 0;\n-\n-\tadd_pending_object(revs, object, \"\");\n-\tif (bitmap_is_preferred_refname(revs->repo, refname))\n-\t\tobject->flags |= NEEDS_BITMAP;\n-\treturn 0;\n-}\n-\n-struct bitmap_commit_cb {\n-\tstruct commit **commits;\n-\tsize_t commits_nr, commits_alloc;\n-\n-\tstruct write_midx_context *ctx;\n-};\n-\n-static const struct object_id *bitmap_oid_access(size_t index,\n-\t\t\t\t\t\t const void *_entries)\n-{\n-\tconst struct pack_midx_entry *entries = _entries;\n-\treturn &entries[index].oid;\n-}\n-\n-static void bitmap_show_commit(struct commit *commit, void *_data)\n-{\n-\tstruct bitmap_commit_cb *data = _data;\n-\tint pos = oid_pos(&commit->object.oid, data->ctx->entries,\n-\t\t\t  data->ctx->entries_nr,\n-\t\t\t  bitmap_oid_access);\n-\tif (pos < 0)\n-\t\treturn;\n-\n-\tALLOC_GROW(data->commits, data->commits_nr + 1, data->commits_alloc);\n-\tdata->commits[data->commits_nr++] = commit;\n-}\n-\n-static int read_refs_snapshot(const char *refs_snapshot,\n-\t\t\t      struct rev_info *revs)\n-{\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tstruct object_id oid;\n-\tFILE *f = xfopen(refs_snapshot, \"r\");\n-\n-\twhile (strbuf_getline(&buf, f) != EOF) {\n-\t\tstruct object *object;\n-\t\tint preferred = 0;\n-\t\tchar *hex = buf.buf;\n-\t\tconst char *end = NULL;\n-\n-\t\tif (buf.len && *buf.buf == '+') {\n-\t\t\tpreferred = 1;\n-\t\t\thex = &buf.buf[1];\n-\t\t}\n-\n-\t\tif (parse_oid_hex(hex, &oid, &end) < 0)\n-\t\t\tdie(_(\"could not parse line: %s\"), buf.buf);\n-\t\tif (*end)\n-\t\t\tdie(_(\"malformed line: %s\"), buf.buf);\n-\n-\t\tobject = parse_object_or_die(&oid, NULL);\n-\t\tif (preferred)\n-\t\t\tobject->flags |= NEEDS_BITMAP;\n-\n-\t\tadd_pending_object(revs, object, \"\");\n-\t}\n-\n-\tfclose(f);\n-\tstrbuf_release(&buf);\n-\treturn 0;\n-}\n-\n-static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p,\n-\t\t\t\t\t\t    const char *refs_snapshot,\n-\t\t\t\t\t\t    struct write_midx_context *ctx)\n-{\n-\tstruct rev_info revs;\n-\tstruct bitmap_commit_cb cb = {0};\n-\n-\ttrace2_region_enter(\"midx\", \"find_commits_for_midx_bitmap\",\n-\t\t\t    the_repository);\n-\n-\tcb.ctx = ctx;\n-\n-\trepo_init_revisions(the_repository, &revs, NULL);\n-\tif (refs_snapshot) {\n-\t\tread_refs_snapshot(refs_snapshot, &revs);\n-\t} else {\n-\t\tsetup_revisions(0, NULL, &revs, NULL);\n-\t\tfor_each_ref(add_ref_to_pending, &revs);\n-\t}\n-\n-\t/*\n-\t * Skipping promisor objects here is intentional, since it only excludes\n-\t * them from the list of reachable commits that we want to select from\n-\t * when computing the selection of MIDX'd commits to receive bitmaps.\n-\t *\n-\t * Reachability bitmaps do require that their objects be closed under\n-\t * reachability, but fetching any objects missing from promisors at this\n-\t * point is too late. But, if one of those objects can be reached from\n-\t * an another object that is included in the bitmap, then we will\n-\t * complain later that we don't have reachability closure (and fail\n-\t * appropriately).\n-\t */\n-\tfetch_if_missing = 0;\n-\trevs.exclude_promisor_objects = 1;\n-\n-\tif (prepare_revision_walk(&revs))\n-\t\tdie(_(\"revision walk setup failed\"));\n-\n-\ttraverse_commit_list(&revs, bitmap_show_commit, NULL, &cb);\n-\tif (indexed_commits_nr_p)\n-\t\t*indexed_commits_nr_p = cb.commits_nr;\n-\n-\trelease_revisions(&revs);\n-\n-\ttrace2_region_leave(\"midx\", \"find_commits_for_midx_bitmap\",\n-\t\t\t    the_repository);\n-\n-\treturn cb.commits;\n-}\n-\n-static int write_midx_bitmap(const char *midx_name,\n-\t\t\t     const unsigned char *midx_hash,\n-\t\t\t     struct packing_data *pdata,\n-\t\t\t     struct commit **commits,\n-\t\t\t     uint32_t commits_nr,\n-\t\t\t     uint32_t *pack_order,\n-\t\t\t     unsigned flags)\n-{\n-\tint ret, i;\n-\tuint16_t options = 0;\n-\tstruct pack_idx_entry **index;\n-\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name,\n-\t\t\t\t\thash_to_hex(midx_hash));\n-\n-\ttrace2_region_enter(\"midx\", \"write_midx_bitmap\", the_repository);\n-\n-\tif (flags & MIDX_WRITE_BITMAP_HASH_CACHE)\n-\t\toptions |= BITMAP_OPT_HASH_CACHE;\n-\n-\tif (flags & MIDX_WRITE_BITMAP_LOOKUP_TABLE)\n-\t\toptions |= BITMAP_OPT_LOOKUP_TABLE;\n-\n-\t/*\n-\t * Build the MIDX-order index based on pdata.objects (which is already\n-\t * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of\n-\t * this order).\n-\t */\n-\tALLOC_ARRAY(index, pdata->nr_objects);\n-\tfor (i = 0; i < pdata->nr_objects; i++)\n-\t\tindex[i] = &pdata->objects[i].idx;\n-\n-\tbitmap_writer_show_progress(flags & MIDX_PROGRESS);\n-\tbitmap_writer_build_type_index(pdata, index, pdata->nr_objects);\n-\n-\t/*\n-\t * bitmap_writer_finish expects objects in lex order, but pack_order\n-\t * gives us exactly that. use it directly instead of re-sorting the\n-\t * array.\n-\t *\n-\t * This changes the order of objects in 'index' between\n-\t * bitmap_writer_build_type_index and bitmap_writer_finish.\n-\t *\n-\t * The same re-ordering takes place in the single-pack bitmap code via\n-\t * write_idx_file(), which is called by finish_tmp_packfile(), which\n-\t * happens between bitmap_writer_build_type_index() and\n-\t * bitmap_writer_finish().\n-\t */\n-\tfor (i = 0; i < pdata->nr_objects; i++)\n-\t\tindex[pack_order[i]] = &pdata->objects[i].idx;\n-\n-\tbitmap_writer_select_commits(commits, commits_nr, -1);\n-\tret = bitmap_writer_build(pdata);\n-\tif (ret < 0)\n-\t\tgoto cleanup;\n-\n-\tbitmap_writer_set_checksum(midx_hash);\n-\tbitmap_writer_finish(index, pdata->nr_objects, bitmap_name, options);\n-\n-cleanup:\n-\tfree(index);\n-\tfree(bitmap_name);\n-\n-\ttrace2_region_leave(\"midx\", \"write_midx_bitmap\", the_repository);\n-\n-\treturn ret;\n-}\n-\n-struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n-\t\t\t\t\t\t\tconst char *object_dir)\n-{\n-\tstruct multi_pack_index *result = NULL;\n-\tstruct multi_pack_index *cur;\n-\tchar *obj_dir_real = real_pathdup(object_dir, 1);\n-\tstruct strbuf cur_path_real = STRBUF_INIT;\n-\n-\t/* Ensure the given object_dir is local, or a known alternate. */\n-\tfind_odb(r, obj_dir_real);\n-\n-\tfor (cur = get_multi_pack_index(r); cur; cur = cur->next) {\n-\t\tstrbuf_realpath(&cur_path_real, cur->object_dir, 1);\n-\t\tif (!strcmp(obj_dir_real, cur_path_real.buf)) {\n-\t\t\tresult = cur;\n-\t\t\tgoto cleanup;\n-\t\t}\n-\t}\n-\n-cleanup:\n-\tfree(obj_dir_real);\n-\tstrbuf_release(&cur_path_real);\n-\treturn result;\n-}\n-\n-int write_midx_internal(const char *object_dir,\n-\t\t\tstruct string_list *packs_to_include,\n-\t\t\tstruct string_list *packs_to_drop,\n-\t\t\tconst char *preferred_pack_name,\n-\t\t\tconst char *refs_snapshot,\n-\t\t\tunsigned flags)\n-{\n-\tstruct strbuf midx_name = STRBUF_INIT;\n-\tunsigned char midx_hash[GIT_MAX_RAWSZ];\n-\tuint32_t i;\n-\tstruct hashfile *f = NULL;\n-\tstruct lock_file lk;\n-\tstruct write_midx_context ctx = { 0 };\n-\tint bitmapped_packs_concat_len = 0;\n-\tint pack_name_concat_len = 0;\n-\tint dropped_packs = 0;\n-\tint result = 0;\n-\tstruct chunkfile *cf;\n-\n-\ttrace2_region_enter(\"midx\", \"write_midx_internal\", the_repository);\n-\n-\tget_midx_filename(&midx_name, object_dir);\n-\tif (safe_create_leading_directories(midx_name.buf))\n-\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n-\t\t\t  midx_name.buf);\n-\n-\tif (!packs_to_include) {\n-\t\t/*\n-\t\t * Only reference an existing MIDX when not filtering which\n-\t\t * packs to include, since all packs and objects are copied\n-\t\t * blindly from an existing MIDX if one is present.\n-\t\t */\n-\t\tctx.m = lookup_multi_pack_index(the_repository, object_dir);\n-\t}\n-\n-\tif (ctx.m && !midx_checksum_valid(ctx.m)) {\n-\t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n-\t\tctx.m = NULL;\n-\t}\n-\n-\tctx.nr = 0;\n-\tctx.alloc = ctx.m ? ctx.m->num_packs : 16;\n-\tctx.info = NULL;\n-\tALLOC_ARRAY(ctx.info, ctx.alloc);\n-\n-\tif (ctx.m) {\n-\t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n-\t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n-\n-\t\t\tif (flags & MIDX_WRITE_REV_INDEX) {\n-\t\t\t\t/*\n-\t\t\t\t * If generating a reverse index, need to have\n-\t\t\t\t * packed_git's loaded to compare their\n-\t\t\t\t * mtimes and object count.\n-\t\t\t\t */\n-\t\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n-\t\t\t\t\terror(_(\"could not load pack\"));\n-\t\t\t\t\tresult = 1;\n-\t\t\t\t\tgoto cleanup;\n-\t\t\t\t}\n-\n-\t\t\t\tif (open_pack_index(ctx.m->packs[i]))\n-\t\t\t\t\tdie(_(\"could not open index for %s\"),\n-\t\t\t\t\t    ctx.m->packs[i]->pack_name);\n-\t\t\t}\n-\n-\t\t\tfill_pack_info(&ctx.info[ctx.nr++], ctx.m->packs[i],\n-\t\t\t\t       ctx.m->pack_names[i], i);\n-\t\t}\n-\t}\n-\n-\tctx.pack_paths_checked = 0;\n-\tif (flags & MIDX_PROGRESS)\n-\t\tctx.progress = start_delayed_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n-\telse\n-\t\tctx.progress = NULL;\n-\n-\tctx.to_include = packs_to_include;\n-\n-\tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n-\tstop_progress(&ctx.progress);\n-\n-\tif ((ctx.m && ctx.nr == ctx.m->num_packs) &&\n-\t    !(packs_to_include || packs_to_drop)) {\n-\t\tstruct bitmap_index *bitmap_git;\n-\t\tint bitmap_exists;\n-\t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n-\n-\t\tbitmap_git = prepare_midx_bitmap_git(ctx.m);\n-\t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n-\t\tfree_bitmap_index(bitmap_git);\n-\n-\t\tif (bitmap_exists || !want_bitmap) {\n-\t\t\t/*\n-\t\t\t * The correct MIDX already exists, and so does a\n-\t\t\t * corresponding bitmap (or one wasn't requested).\n-\t\t\t */\n-\t\t\tif (!want_bitmap)\n-\t\t\t\tclear_midx_files_ext(object_dir, \".bitmap\",\n-\t\t\t\t\t\t     NULL);\n-\t\t\tgoto cleanup;\n-\t\t}\n-\t}\n-\n-\tif (preferred_pack_name) {\n-\t\tctx.preferred_pack_idx = -1;\n-\n-\t\tfor (i = 0; i < ctx.nr; i++) {\n-\t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n-\t\t\t\t\t\t  ctx.info[i].pack_name)) {\n-\t\t\t\tctx.preferred_pack_idx = i;\n-\t\t\t\tbreak;\n-\t\t\t}\n-\t\t}\n-\n-\t\tif (ctx.preferred_pack_idx == -1)\n-\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n-\t\t\t\tpreferred_pack_name);\n-\t} else if (ctx.nr &&\n-\t\t   (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))) {\n-\t\tstruct packed_git *oldest = ctx.info[ctx.preferred_pack_idx].p;\n-\t\tctx.preferred_pack_idx = 0;\n-\n-\t\tif (packs_to_drop && packs_to_drop->nr)\n-\t\t\tBUG(\"cannot write a MIDX bitmap during expiration\");\n-\n-\t\t/*\n-\t\t * set a preferred pack when writing a bitmap to ensure that\n-\t\t * the pack from which the first object is selected in pseudo\n-\t\t * pack-order has all of its objects selected from that pack\n-\t\t * (and not another pack containing a duplicate)\n-\t\t */\n-\t\tfor (i = 1; i < ctx.nr; i++) {\n-\t\t\tstruct packed_git *p = ctx.info[i].p;\n-\n-\t\t\tif (!oldest->num_objects || p->mtime < oldest->mtime) {\n-\t\t\t\toldest = p;\n-\t\t\t\tctx.preferred_pack_idx = i;\n-\t\t\t}\n-\t\t}\n-\n-\t\tif (!oldest->num_objects) {\n-\t\t\t/*\n-\t\t\t * If all packs are empty; unset the preferred index.\n-\t\t\t * This is acceptable since there will be no duplicate\n-\t\t\t * objects to resolve, so the preferred value doesn't\n-\t\t\t * matter.\n-\t\t\t */\n-\t\t\tctx.preferred_pack_idx = -1;\n-\t\t}\n-\t} else {\n-\t\t/*\n-\t\t * otherwise don't mark any pack as preferred to avoid\n-\t\t * interfering with expiration logic below\n-\t\t */\n-\t\tctx.preferred_pack_idx = -1;\n-\t}\n-\n-\tif (ctx.preferred_pack_idx > -1) {\n-\t\tstruct packed_git *preferred = ctx.info[ctx.preferred_pack_idx].p;\n-\t\tif (!preferred->num_objects) {\n-\t\t\terror(_(\"cannot select preferred pack %s with no objects\"),\n-\t\t\t      preferred->pack_name);\n-\t\t\tresult = 1;\n-\t\t\tgoto cleanup;\n-\t\t}\n-\t}\n-\n-\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,\n-\t\t\t\t\t ctx.preferred_pack_idx);\n-\n-\tctx.large_offsets_needed = 0;\n-\tfor (i = 0; i < ctx.entries_nr; i++) {\n-\t\tif (ctx.entries[i].offset > 0x7fffffff)\n-\t\t\tctx.num_large_offsets++;\n-\t\tif (ctx.entries[i].offset > 0xffffffff)\n-\t\t\tctx.large_offsets_needed = 1;\n-\t}\n-\n-\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n-\n-\tif (packs_to_drop && packs_to_drop->nr) {\n-\t\tint drop_index = 0;\n-\t\tint missing_drops = 0;\n-\n-\t\tfor (i = 0; i < ctx.nr && drop_index < packs_to_drop->nr; i++) {\n-\t\t\tint cmp = strcmp(ctx.info[i].pack_name,\n-\t\t\t\t\t packs_to_drop->items[drop_index].string);\n-\n-\t\t\tif (!cmp) {\n-\t\t\t\tdrop_index++;\n-\t\t\t\tctx.info[i].expired = 1;\n-\t\t\t} else if (cmp > 0) {\n-\t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n-\t\t\t\t      packs_to_drop->items[drop_index].string);\n-\t\t\t\tdrop_index++;\n-\t\t\t\tmissing_drops++;\n-\t\t\t\ti--;\n-\t\t\t} else {\n-\t\t\t\tctx.info[i].expired = 0;\n-\t\t\t}\n-\t\t}\n-\n-\t\tif (missing_drops) {\n-\t\t\tresult = 1;\n-\t\t\tgoto cleanup;\n-\t\t}\n-\t}\n-\n-\t/*\n-\t * pack_perm stores a permutation between pack-int-ids from the\n-\t * previous multi-pack-index to the new one we are writing:\n-\t *\n-\t * pack_perm[old_id] = new_id\n-\t */\n-\tALLOC_ARRAY(ctx.pack_perm, ctx.nr);\n-\tfor (i = 0; i < ctx.nr; i++) {\n-\t\tif (ctx.info[i].expired) {\n-\t\t\tdropped_packs++;\n-\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n-\t\t} else {\n-\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n-\t\t}\n-\t}\n-\n-\tfor (i = 0; i < ctx.nr; i++) {\n-\t\tif (ctx.info[i].expired)\n-\t\t\tcontinue;\n-\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n-\t\tbitmapped_packs_concat_len += 2 * sizeof(uint32_t);\n-\t}\n-\n-\t/* Check that the preferred pack wasn't expired (if given). */\n-\tif (preferred_pack_name) {\n-\t\tstruct pack_info *preferred = bsearch(preferred_pack_name,\n-\t\t\t\t\t\t      ctx.info, ctx.nr,\n-\t\t\t\t\t\t      sizeof(*ctx.info),\n-\t\t\t\t\t\t      idx_or_pack_name_cmp);\n-\t\tif (preferred) {\n-\t\t\tuint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];\n-\t\t\tif (perm == PACK_EXPIRED)\n-\t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n-\t\t\t\t\tpreferred_pack_name);\n-\t\t}\n-\t}\n-\n-\tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n-\t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n-\t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n-\n-\thold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR);\n-\tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n-\n-\tif (ctx.nr - dropped_packs == 0) {\n-\t\terror(_(\"no pack files to index.\"));\n-\t\tresult = 1;\n-\t\tgoto cleanup;\n-\t}\n-\n-\tif (!ctx.entries_nr) {\n-\t\tif (flags & MIDX_WRITE_BITMAP)\n-\t\t\twarning(_(\"refusing to write multi-pack .bitmap without any objects\"));\n-\t\tflags &= ~(MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP);\n-\t}\n-\n-\tcf = init_chunkfile(f);\n-\n-\tadd_chunk(cf, MIDX_CHUNKID_PACKNAMES, pack_name_concat_len,\n-\t\t  write_midx_pack_names);\n-\tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT, MIDX_CHUNK_FANOUT_SIZE,\n-\t\t  write_midx_oid_fanout);\n-\tadd_chunk(cf, MIDX_CHUNKID_OIDLOOKUP,\n-\t\t  st_mult(ctx.entries_nr, the_hash_algo->rawsz),\n-\t\t  write_midx_oid_lookup);\n-\tadd_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS,\n-\t\t  st_mult(ctx.entries_nr, MIDX_CHUNK_OFFSET_WIDTH),\n-\t\t  write_midx_object_offsets);\n-\n-\tif (ctx.large_offsets_needed)\n-\t\tadd_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS,\n-\t\t\tst_mult(ctx.num_large_offsets,\n-\t\t\t\tMIDX_CHUNK_LARGE_OFFSET_WIDTH),\n-\t\t\twrite_midx_large_offsets);\n-\n-\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) {\n-\t\tctx.pack_order = midx_pack_order(&ctx);\n-\t\tadd_chunk(cf, MIDX_CHUNKID_REVINDEX,\n-\t\t\t  st_mult(ctx.entries_nr, sizeof(uint32_t)),\n-\t\t\t  write_midx_revindex);\n-\t\tadd_chunk(cf, MIDX_CHUNKID_BITMAPPEDPACKS,\n-\t\t\t  bitmapped_packs_concat_len,\n-\t\t\t  write_midx_bitmapped_packs);\n-\t}\n-\n-\twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n-\twrite_chunkfile(cf, &ctx);\n-\n-\tfinalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,\n-\t\t\t  CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n-\tfree_chunkfile(cf);\n-\n-\tif (flags & MIDX_WRITE_REV_INDEX &&\n-\t    git_env_bool(\"GIT_TEST_MIDX_WRITE_REV\", 0))\n-\t\twrite_midx_reverse_index(midx_name.buf, midx_hash, &ctx);\n-\n-\tif (flags & MIDX_WRITE_BITMAP) {\n-\t\tstruct packing_data pdata;\n-\t\tstruct commit **commits;\n-\t\tuint32_t commits_nr;\n-\n-\t\tif (!ctx.entries_nr)\n-\t\t\tBUG(\"cannot write a bitmap without any objects\");\n-\n-\t\tprepare_midx_packing_data(&pdata, &ctx);\n-\n-\t\tcommits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, &ctx);\n-\n-\t\t/*\n-\t\t * The previous steps translated the information from\n-\t\t * 'entries' into information suitable for constructing\n-\t\t * bitmaps. We no longer need that array, so clear it to\n-\t\t * reduce memory pressure.\n-\t\t */\n-\t\tFREE_AND_NULL(ctx.entries);\n-\t\tctx.entries_nr = 0;\n-\n-\t\tif (write_midx_bitmap(midx_name.buf, midx_hash, &pdata,\n-\t\t\t\t      commits, commits_nr, ctx.pack_order,\n-\t\t\t\t      flags) < 0) {\n-\t\t\terror(_(\"could not write multi-pack bitmap\"));\n-\t\t\tresult = 1;\n-\t\t\tclear_packing_data(&pdata);\n-\t\t\tfree(commits);\n-\t\t\tgoto cleanup;\n-\t\t}\n-\n-\t\tclear_packing_data(&pdata);\n-\t\tfree(commits);\n-\t}\n-\t/*\n-\t * NOTE: Do not use ctx.entries beyond this point, since it might\n-\t * have been freed in the previous if block.\n-\t */\n-\n-\tif (ctx.m)\n-\t\tclose_object_store(the_repository->objects);\n-\n-\tif (commit_lock_file(&lk) < 0)\n-\t\tdie_errno(_(\"could not write multi-pack-index\"));\n-\n-\tclear_midx_files_ext(object_dir, \".bitmap\", midx_hash);\n-\tclear_midx_files_ext(object_dir, \".rev\", midx_hash);\n-\n-cleanup:\n-\tfor (i = 0; i < ctx.nr; i++) {\n-\t\tif (ctx.info[i].p) {\n-\t\t\tclose_pack(ctx.info[i].p);\n-\t\t\tfree(ctx.info[i].p);\n-\t\t}\n-\t\tfree(ctx.info[i].pack_name);\n-\t}\n-\n-\tfree(ctx.info);\n-\tfree(ctx.entries);\n-\tfree(ctx.pack_perm);\n-\tfree(ctx.pack_order);\n-\tstrbuf_release(&midx_name);\n-\n-\ttrace2_region_leave(\"midx\", \"write_midx_internal\", the_repository);\n-\n-\treturn result;\n-}\n-\n struct clear_midx_data {\n \tchar *keep;\n \tconst char *ext;\n@@ -1775,8 +507,8 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n \t\tdie_errno(_(\"failed to remove %s\"), full_path);\n }\n \n-static void clear_midx_files_ext(const char *object_dir, const char *ext,\n-\t\t\t\t unsigned char *keep_hash)\n+void clear_midx_files_ext(const char *object_dir, const char *ext,\n+\t\t\t  unsigned char *keep_hash)\n {\n \tstruct clear_midx_data data;\n \tmemset(&data, 0, sizeof(struct clear_midx_data));\ndiff --git a/midx.h b/midx.h\nindex b374a7afaf..dc477dff44 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -8,6 +8,25 @@ struct pack_entry;\n struct repository;\n struct bitmapped_pack;\n \n+#define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n+#define MIDX_VERSION 1\n+#define MIDX_BYTE_FILE_VERSION 4\n+#define MIDX_BYTE_HASH_VERSION 5\n+#define MIDX_BYTE_NUM_CHUNKS 6\n+#define MIDX_BYTE_NUM_PACKS 8\n+#define MIDX_HEADER_SIZE 12\n+\n+#define MIDX_CHUNK_ALIGNMENT 4\n+#define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n+#define MIDX_CHUNKID_BITMAPPEDPACKS 0x42544d50 /* \"BTMP\" */\n+#define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n+#define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n+#define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n+#define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n+#define MIDX_CHUNKID_REVINDEX 0x52494458 /* \"RIDX\" */\n+#define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n+#define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n+\n #define GIT_TEST_MULTI_PACK_INDEX \"GIT_TEST_MULTI_PACK_INDEX\"\n #define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \\\n \t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\"\n-- \n2.44.0.290.g736be63234b\n\n"},{"id":"491476","messageId":"8e32755c492d20eec02c81351d249ce34cc6d7b9.1711387439.git.me@ttaylorr.com","threadId":"61199","inReplyTo":"cover.1711387439.git.me@ttaylorr.com","subject":"[PATCH 08/11] midx-write.c: avoid directly managed temporary strbuf","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-25T17:24:41Z","receivedAt":"2024-03-25T17:24:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In midx-write.c::midx_repack(), we construct the command-line arguments\nfor a pack-objects invocation which will combine objects from the packs\nbelow our `--batch-size` option.\n\nTo construct the base name of the output pack, we use a temporary\nstrbuf, and then push the result of that onto the strvec which holds the\ncommand-line arguments, after which point we release the strbuf.\n\nWe could replace this by doing something like:\n\n    struct strbuf buf = STRBUF_INIT;\n    strbuf_addf(&buf, \"%s/pack/pack\", object_dir);\n    strvec_push_nodup(&cmd.args, strbuf_detach(&buf));\n\n(combining the two separate `strbuf_addstr()` calls into a single\n`strbuf_addf()`). But that is more or less an open-coded version of\nstrvec_pushf(), which we could use directly instead.\n\n(Note that at the time this code was written back in ce1e4a105b4 (midx:\nimplement midx_repack(), 2019-06-10), strvec did not yet exist, so the\nabove example would have replaced the last line with:\n\n    argv_array_push_nodup(&cmd.args, strbuf_detach(&buf));\n\n, but the code is otherwise unchanged).\n\nAvoid directly managing the temporary strbuf used to construct the\nbase name for pack-object's command-line arguments, and instead use the\npurpose-built `strvec_pushf()` instead.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx-write.c | 7 +------\n 1 file changed, 1 insertion(+), 6 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex c812156cbd..89e325d08e 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1446,7 +1446,6 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \tunsigned char *include_pack;\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tFILE *cmd_in;\n-\tstruct strbuf base_name = STRBUF_INIT;\n \tstruct multi_pack_index *m = lookup_multi_pack_index(r, object_dir);\n \n \t/*\n@@ -1473,10 +1472,6 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \n \tstrvec_push(&cmd.args, \"pack-objects\");\n \n-\tstrbuf_addstr(&base_name, object_dir);\n-\tstrbuf_addstr(&base_name, \"/pack/pack\");\n-\tstrvec_push(&cmd.args, base_name.buf);\n-\n \tif (delta_base_offset)\n \t\tstrvec_push(&cmd.args, \"--delta-base-offset\");\n \tif (use_delta_islands)\n@@ -1487,7 +1482,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \telse\n \t\tstrvec_push(&cmd.args, \"-q\");\n \n-\tstrbuf_release(&base_name);\n+\tstrvec_pushf(&cmd.args, \"%s/pack/pack\", object_dir);\n \n \tcmd.git_cmd = 1;\n \tcmd.in = cmd.out = -1;\n-- \n2.44.0.290.g736be63234b\n\n"},{"id":"491478","messageId":"5475b09a7afc4d55a8e1a1a72f20fa9109447cec.1711387439.git.me@ttaylorr.com","threadId":"61199","inReplyTo":"cover.1711387439.git.me@ttaylorr.com","subject":"[PATCH 09/11] midx-write.c: factor out common want_included_pack() routine","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-25T17:24:44Z","receivedAt":"2024-03-25T17:24:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When performing a 'git multi-pack-index repack', the MIDX machinery\ntries to aggregate MIDX'd packs together either to (a) fill the given\n`--batch-size` argument, or (b) combine all packs together.\n\nIn either case (using the `midx-write.c::fill_included_packs_batch()` or\n`midx-write.c::fill_included_packs_all()` function, respectively), we\nevaluate whether or not we want to repack each MIDX'd pack, according to\nwhether or it is loadable, kept, cruft, or non-empty.\n\nBetween the two `fill_included_packs_` callers, they both care about the\nsame conditions, except for `fill_included_packs_batch()` which also\ncares that the pack is non-empty.\n\nWe could extract two functions (say, `want_included_pack()` and a\n`_nonempty()` variant), but this is not necessary. For the case in\n`fill_included_packs_all()` which does not check the pack size, we add\nall of the pack's objects assuming that the pack meets all other\ncriteria. But if the pack is empty in the first place, we add all of its\nzero objects, so whether or not we \"accept\" or \"reject\" it in the first\nplace is irrelevant.\n\nThis change improves the readability in both `fill_included_packs_`\nfunctions.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx-write.c | 32 ++++++++++++++++++++------------\n 1 file changed, 20 insertions(+), 12 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 89e325d08e..2f0f5d133f 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1349,6 +1349,24 @@ static int compare_by_mtime(const void *a_, const void *b_)\n \treturn 0;\n }\n \n+static int want_included_pack(struct repository *r,\n+\t\t\t      struct multi_pack_index *m,\n+\t\t\t      int pack_kept_objects,\n+\t\t\t      uint32_t pack_int_id)\n+{\n+\tstruct packed_git *p;\n+\tif (prepare_midx_pack(r, m, pack_int_id))\n+\t\treturn 0;\n+\tp = m->packs[pack_int_id];\n+\tif (!pack_kept_objects && p->pack_keep)\n+\t\treturn 0;\n+\tif (p->is_cruft)\n+\t\treturn 0;\n+\tif (open_pack_index(p) || !p->num_objects)\n+\t\treturn 0;\n+\treturn 1;\n+}\n+\n static int fill_included_packs_all(struct repository *r,\n \t\t\t\t   struct multi_pack_index *m,\n \t\t\t\t   unsigned char *include_pack)\n@@ -1359,11 +1377,7 @@ static int fill_included_packs_all(struct repository *r,\n \trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n \n \tfor (i = 0; i < m->num_packs; i++) {\n-\t\tif (prepare_midx_pack(r, m, i))\n-\t\t\tcontinue;\n-\t\tif (!pack_kept_objects && m->packs[i]->pack_keep)\n-\t\t\tcontinue;\n-\t\tif (m->packs[i]->is_cruft)\n+\t\tif (!want_included_pack(r, m, pack_kept_objects, i))\n \t\t\tcontinue;\n \n \t\tinclude_pack[i] = 1;\n@@ -1410,13 +1424,7 @@ static int fill_included_packs_batch(struct repository *r,\n \t\tstruct packed_git *p = m->packs[pack_int_id];\n \t\tsize_t expected_size;\n \n-\t\tif (!p)\n-\t\t\tcontinue;\n-\t\tif (!pack_kept_objects && p->pack_keep)\n-\t\t\tcontinue;\n-\t\tif (p->is_cruft)\n-\t\t\tcontinue;\n-\t\tif (open_pack_index(p) || !p->num_objects)\n+\t\tif (!want_included_pack(r, m, pack_kept_objects, pack_int_id))\n \t\t\tcontinue;\n \n \t\texpected_size = st_mult(p->pack_size,\n-- \n2.44.0.290.g736be63234b\n\n"},{"id":"491479","messageId":"f77e3167aad4b60c381cf1def2ee1eeb26218d06.1711387439.git.me@ttaylorr.com","threadId":"61199","inReplyTo":"cover.1711387439.git.me@ttaylorr.com","subject":"[PATCH 10/11] midx-write.c: check count of packs to repack after grouping","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-25T17:24:47Z","receivedAt":"2024-03-25T17:24:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In both fill_included_packs_all() and fill_included_packs_batch(), we\naccumulate a list of packs whose contents we want to repack together,\nand then use that information to feed a list of objects as input to\npack-objects.\n\nIn both cases, the `fill_included_packs_` functions keep track of how\nmany packs they want to repack together, and only execute pack-objects\nif there are at least two packs that need repacking.\n\nHaving both of these functions keep track of this information themselves\nis not strictly necessary, since they also log which packs to repack via\nthe `include_pack` array, so we can simply count the non-zero entries in\nthat array after either function is done executing, reducing the overall\namount of code necessary.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx-write.c | 44 ++++++++++++++++++++------------------------\n 1 file changed, 20 insertions(+), 24 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 2f0f5d133f..4f1d649aa6 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1367,11 +1367,11 @@ static int want_included_pack(struct repository *r,\n \treturn 1;\n }\n \n-static int fill_included_packs_all(struct repository *r,\n-\t\t\t\t   struct multi_pack_index *m,\n-\t\t\t\t   unsigned char *include_pack)\n+static void fill_included_packs_all(struct repository *r,\n+\t\t\t\t    struct multi_pack_index *m,\n+\t\t\t\t    unsigned char *include_pack)\n {\n-\tuint32_t i, count = 0;\n+\tuint32_t i;\n \tint pack_kept_objects = 0;\n \n \trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n@@ -1381,18 +1381,15 @@ static int fill_included_packs_all(struct repository *r,\n \t\t\tcontinue;\n \n \t\tinclude_pack[i] = 1;\n-\t\tcount++;\n \t}\n-\n-\treturn count < 2;\n }\n \n-static int fill_included_packs_batch(struct repository *r,\n-\t\t\t\t     struct multi_pack_index *m,\n-\t\t\t\t     unsigned char *include_pack,\n-\t\t\t\t     size_t batch_size)\n+static void fill_included_packs_batch(struct repository *r,\n+\t\t\t\t      struct multi_pack_index *m,\n+\t\t\t\t      unsigned char *include_pack,\n+\t\t\t\t      size_t batch_size)\n {\n-\tuint32_t i, packs_to_repack;\n+\tuint32_t i;\n \tsize_t total_size;\n \tstruct repack_info *pack_info;\n \tint pack_kept_objects = 0;\n@@ -1418,7 +1415,6 @@ static int fill_included_packs_batch(struct repository *r,\n \tQSORT(pack_info, m->num_packs, compare_by_mtime);\n \n \ttotal_size = 0;\n-\tpacks_to_repack = 0;\n \tfor (i = 0; total_size < batch_size && i < m->num_packs; i++) {\n \t\tint pack_int_id = pack_info[i].pack_int_id;\n \t\tstruct packed_git *p = m->packs[pack_int_id];\n@@ -1434,23 +1430,17 @@ static int fill_included_packs_batch(struct repository *r,\n \t\tif (expected_size >= batch_size)\n \t\t\tcontinue;\n \n-\t\tpacks_to_repack++;\n \t\ttotal_size += expected_size;\n \t\tinclude_pack[pack_int_id] = 1;\n \t}\n \n \tfree(pack_info);\n-\n-\tif (packs_to_repack < 2)\n-\t\treturn 1;\n-\n-\treturn 0;\n }\n \n int midx_repack(struct repository *r, const char *object_dir, size_t batch_size, unsigned flags)\n {\n \tint result = 0;\n-\tuint32_t i;\n+\tuint32_t i, packs_to_repack = 0;\n \tunsigned char *include_pack;\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tFILE *cmd_in;\n@@ -1469,10 +1459,16 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \n \tCALLOC_ARRAY(include_pack, m->num_packs);\n \n-\tif (batch_size) {\n-\t\tif (fill_included_packs_batch(r, m, include_pack, batch_size))\n-\t\t\tgoto cleanup;\n-\t} else if (fill_included_packs_all(r, m, include_pack))\n+\tif (batch_size)\n+\t\tfill_included_packs_batch(r, m, include_pack, batch_size);\n+\telse\n+\t\tfill_included_packs_all(r, m, include_pack);\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tif (include_pack[i])\n+\t\t\tpacks_to_repack++;\n+\t}\n+\tif (packs_to_repack <= 1)\n \t\tgoto cleanup;\n \n \trepo_config_get_bool(r, \"repack.usedeltabaseoffset\", &delta_base_offset);\n-- \n2.44.0.290.g736be63234b\n\n"},{"id":"491480","messageId":"736be63234baf7fc6df8259d9bb7298858b2bc74.1711387439.git.me@ttaylorr.com","threadId":"61199","inReplyTo":"cover.1711387439.git.me@ttaylorr.com","subject":"[PATCH 11/11] midx-write.c: use `--stdin-packs` when repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-25T17:24:50Z","receivedAt":"2024-03-25T17:24:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When constructing a new pack `git multi-pack-index repack` provides a\nlist of objects which is the union of objects in all MIDX'd packs which\nwere \"included\" in the repack.\n\nThough correct, this typically yields a poorly structured pack, since\nproviding the objects list over stdin does not give pack-objects a\nchance to discover the namehash values for each object, leading to\nsub-optimal delta selection.\n\nWe can use `--stdin-packs` instead, which has a couple of benefits:\n\n  - it does a supplemental walk over objects in the supplied list of\n    packs to discover their namehash, leading to higher-quality delta\n    selection\n\n  - it requires us to list far less data over stdin; instead of listing\n    each object in the resulting pack, we need only list the\n    constituent packs from which those objects were selected in the MIDX\n\nOf course, this comes at a slight cost: though we save time on listing\npacks versus objects over stdin[^1] (around ~650 milliseconds), we add a\nnon-trivial amount of time walking over the given objects in order to\nfind better deltas.\n\nIn general, this is likely to more closely match the user's expectations\n(i.e. that packs generated via `git multi-pack-index repack` are written\nwith high-quality deltas). But if not, we can always introduce a new\noption in pack-objects to disable the supplemental object walk, which\nwould yield a pure CPU-time savings, at the cost of the on-disk size of\nthe resulting pack.\n\n[^1]: In a patched version of Git that doesn't perform the supplemental\n  object walk in `pack-objects --stdin-packs`, we save around ~650ms\n  (from 5.968 to 5.325 seconds) when running `git multi-pack-index\n  repack --batch-size=0` on git.git with all objects packed, and all\n  packs in a MIDX.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx-write.c | 18 +++++++++---------\n 1 file changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 4f1d649aa6..d341b9c628 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1474,7 +1474,8 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \trepo_config_get_bool(r, \"repack.usedeltabaseoffset\", &delta_base_offset);\n \trepo_config_get_bool(r, \"repack.usedeltaislands\", &use_delta_islands);\n \n-\tstrvec_push(&cmd.args, \"pack-objects\");\n+\tstrvec_pushl(&cmd.args, \"pack-objects\", \"--stdin-packs\", \"--non-empty\",\n+\t\t     NULL);\n \n \tif (delta_base_offset)\n \t\tstrvec_push(&cmd.args, \"--delta-base-offset\");\n@@ -1498,16 +1499,15 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t}\n \n \tcmd_in = xfdopen(cmd.in, \"w\");\n-\n-\tfor (i = 0; i < m->num_objects; i++) {\n-\t\tstruct object_id oid;\n-\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n-\n-\t\tif (!include_pack[pack_int_id])\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tstruct packed_git *p = m->packs[i];\n+\t\tif (!p)\n \t\t\tcontinue;\n \n-\t\tnth_midxed_object_oid(&oid, m, i);\n-\t\tfprintf(cmd_in, \"%s\\n\", oid_to_hex(&oid));\n+\t\tif (include_pack[i])\n+\t\t\tfprintf(cmd_in, \"%s\\n\", pack_basename(p));\n+\t\telse\n+\t\t\tfprintf(cmd_in, \"^%s\\n\", pack_basename(p));\n \t}\n \tfclose(cmd_in);\n \n-- \n2.44.0.290.g736be63234b\n"},{"id":"491498","messageId":"xmqq1q7yjbpz.fsf@gitster.g","threadId":"61199","inReplyTo":"ffa8ba18de8be68eb8dbb1e17b5bf8800f564505.1711387439.git.me@ttaylorr.com","subject":"Re: [PATCH 01/11] midx-write: initial commit","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-03-25T20:30:32Z","receivedAt":"2024-03-25T20:30:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> Introduce a new empty midx-write.c source file. Similar to the\n> relationship between \"pack-bitmap.c\" and \"pack-bitmap-write.c\", this\n> source file will hold code that is specific to writing MIDX files as\n> opposed to reading them (the latter will remain in midx.c).\n>\n> This is a preparatory step which will reduce the overall size of midx.c\n> and make it easier to read as we prepare for future changes to that file\n> (outside the immediate scope of this series).\n>\n> Signed-off-by: Taylor Blau <me@ttaylorr.com>\n> ---\n>  Makefile     | 1 +\n>  midx-write.c | 2 ++\n>  2 files changed, 3 insertions(+)\n>  create mode 100644 midx-write.c\n>\n> diff --git a/Makefile b/Makefile\n> index 4e255c81f2..cf44a964c0 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1072,6 +1072,7 @@ LIB_OBJS += merge-ort-wrappers.o\n>  LIB_OBJS += merge-recursive.o\n>  LIB_OBJS += merge.o\n>  LIB_OBJS += midx.o\n> +LIB_OBJS += midx-write.o\n>  LIB_OBJS += name-hash.o\n>  LIB_OBJS += negotiator/default.o\n>  LIB_OBJS += negotiator/noop.o\n> diff --git a/midx-write.c b/midx-write.c\n> new file mode 100644\n> index 0000000000..214179d308\n> --- /dev/null\n> +++ b/midx-write.c\n> @@ -0,0 +1,2 @@\n> +#include \"git-compat-util.h\"\n> +#include \"midx.h\"\n\nI noticed that \"nm midx-write.o\" after this step gives us nothing.\nA source that produces absolutely nothing may not successfully\ncompile everywhere.  It is unlikely we will stop the series at this\nstep and it won't break bisection, though.\n\nI do not quite see why the first 3 patches need to be split this\nway, rather than being done as a single step.  Is it making the\nreview any simpler?\n"},{"id":"491499","messageId":"xmqqv85ahx14.fsf@gitster.g","threadId":"61199","inReplyTo":"8e32755c492d20eec02c81351d249ce34cc6d7b9.1711387439.git.me@ttaylorr.com","subject":"Re: [PATCH 08/11] midx-write.c: avoid directly managed temporary strbuf","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-03-25T20:33:11Z","receivedAt":"2024-03-25T20:33:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> In midx-write.c::midx_repack(), we construct the command-line arguments\n> for a pack-objects invocation which will combine objects from the packs\n> below our `--batch-size` option.\n>\n> To construct the base name of the output pack, we use a temporary\n> strbuf, and then push the result of that onto the strvec which holds the\n> command-line arguments, after which point we release the strbuf.\n>\n> We could replace this by doing something like:\n>\n>     struct strbuf buf = STRBUF_INIT;\n>     strbuf_addf(&buf, \"%s/pack/pack\", object_dir);\n>     strvec_push_nodup(&cmd.args, strbuf_detach(&buf));\n\nHmph, I thought I saw another patch recently that uses\nstrvec_pushf() to simplify such a sequence.  Does the technique\napply here as well?\n\nAh, yes, exactly.  See <9483038c-9529-4243-9b9a-97254fac29c1@web.de>\n\n> @@ -1473,10 +1472,6 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n>  \n>  \tstrvec_push(&cmd.args, \"pack-objects\");\n>  \n> -\tstrbuf_addstr(&base_name, object_dir);\n> -\tstrbuf_addstr(&base_name, \"/pack/pack\");\n> -\tstrvec_push(&cmd.args, base_name.buf);\n> -\n>  \tif (delta_base_offset)\n>  \t\tstrvec_push(&cmd.args, \"--delta-base-offset\");\n>  \tif (use_delta_islands)\n> @@ -1487,7 +1482,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n>  \telse\n>  \t\tstrvec_push(&cmd.args, \"-q\");\n>  \n> -\tstrbuf_release(&base_name);\n> +\tstrvec_pushf(&cmd.args, \"%s/pack/pack\", object_dir);\n>  \n>  \tcmd.git_cmd = 1;\n>  \tcmd.in = cmd.out = -1;\n"},{"id":"491505","messageId":"xmqqo7b2hwv3.fsf@gitster.g","threadId":"61199","inReplyTo":"5475b09a7afc4d55a8e1a1a72f20fa9109447cec.1711387439.git.me@ttaylorr.com","subject":"Re: [PATCH 09/11] midx-write.c: factor out common want_included_pack() routine","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-03-25T20:36:48Z","receivedAt":"2024-03-25T20:36:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> diff --git a/midx-write.c b/midx-write.c\n> index 89e325d08e..2f0f5d133f 100644\n> --- a/midx-write.c\n> +++ b/midx-write.c\n> @@ -1349,6 +1349,24 @@ static int compare_by_mtime(const void *a_, const void *b_)\n>  \treturn 0;\n>  }\n>  \n> +static int want_included_pack(struct repository *r,\n> +\t\t\t      struct multi_pack_index *m,\n> +\t\t\t      int pack_kept_objects,\n> +\t\t\t      uint32_t pack_int_id)\n> +{\n> +\tstruct packed_git *p;\n> +\tif (prepare_midx_pack(r, m, pack_int_id))\n> +\t\treturn 0;\n> +\tp = m->packs[pack_int_id];\n> +\tif (!pack_kept_objects && p->pack_keep)\n> +\t\treturn 0;\n> +\tif (p->is_cruft)\n> +\t\treturn 0;\n> +\tif (open_pack_index(p) || !p->num_objects)\n> +\t\treturn 0;\n> +\treturn 1;\n> +}\n\nOK, makes sense to return false when we do not want the caller\nproceed with the pack for a function whose name is want_*_pack().\n\n"},{"id":"491508","messageId":"xmqqjzlqhwnc.fsf@gitster.g","threadId":"61199","inReplyTo":"f77e3167aad4b60c381cf1def2ee1eeb26218d06.1711387439.git.me@ttaylorr.com","subject":"Re: [PATCH 10/11] midx-write.c: check count of packs to repack after grouping","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-03-25T20:41:27Z","receivedAt":"2024-03-25T20:41:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> In both fill_included_packs_all() and fill_included_packs_batch(), we\n> accumulate a list of packs whose contents we want to repack together,\n> and then use that information to feed a list of objects as input to\n> pack-objects.\n>\n> In both cases, the `fill_included_packs_` functions keep track of how\n> many packs they want to repack together, and only execute pack-objects\n> if there are at least two packs that need repacking.\n>\n> Having both of these functions keep track of this information themselves\n> is not strictly necessary, since they also log which packs to repack via\n> the `include_pack` array, so we can simply count the non-zero entries in\n> that array after either function is done executing, reducing the overall\n> amount of code necessary.\n\nIt does make the logic at the caller simpler to follow.\n\n> -\tif (batch_size) {\n> -\t\tif (fill_included_packs_batch(r, m, include_pack, batch_size))\n> -\t\t\tgoto cleanup;\n> -\t} else if (fill_included_packs_all(r, m, include_pack))\n> +\tif (batch_size)\n> +\t\tfill_included_packs_batch(r, m, include_pack, batch_size);\n> +\telse\n> +\t\tfill_included_packs_all(r, m, include_pack);\n> +\n> +\tfor (i = 0; i < m->num_packs; i++) {\n> +\t\tif (include_pack[i])\n> +\t\t\tpacks_to_repack++;\n> +\t}\n> +\tif (packs_to_repack <= 1)\n>  \t\tgoto cleanup;\n>  \n>  \trepo_config_get_bool(r, \"repack.usedeltabaseoffset\", &delta_base_offset);\n\nQueued.  Thanks.\n\n"},{"id":"491533","messageId":"ZgH2JJZ1sdVeVne2@nand.local","threadId":"61199","inReplyTo":"xmqq1q7yjbpz.fsf@gitster.g","subject":"Re: [PATCH 01/11] midx-write: initial commit","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-25T22:09:40Z","receivedAt":"2024-03-25T22:09:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 25, 2024 at 01:30:32PM -0700, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> > Introduce a new empty midx-write.c source file. Similar to the\n> > relationship between \"pack-bitmap.c\" and \"pack-bitmap-write.c\", this\n> > source file will hold code that is specific to writing MIDX files as\n> > opposed to reading them (the latter will remain in midx.c).\n> >\n> > This is a preparatory step which will reduce the overall size of midx.c\n> > and make it easier to read as we prepare for future changes to that file\n> > (outside the immediate scope of this series).\n> >\n> > Signed-off-by: Taylor Blau <me@ttaylorr.com>\n> > ---\n> >  Makefile     | 1 +\n> >  midx-write.c | 2 ++\n> >  2 files changed, 3 insertions(+)\n> >  create mode 100644 midx-write.c\n> >\n> > diff --git a/Makefile b/Makefile\n> > index 4e255c81f2..cf44a964c0 100644\n> > --- a/Makefile\n> > +++ b/Makefile\n> > @@ -1072,6 +1072,7 @@ LIB_OBJS += merge-ort-wrappers.o\n> >  LIB_OBJS += merge-recursive.o\n> >  LIB_OBJS += merge.o\n> >  LIB_OBJS += midx.o\n> > +LIB_OBJS += midx-write.o\n> >  LIB_OBJS += name-hash.o\n> >  LIB_OBJS += negotiator/default.o\n> >  LIB_OBJS += negotiator/noop.o\n> > diff --git a/midx-write.c b/midx-write.c\n> > new file mode 100644\n> > index 0000000000..214179d308\n> > --- /dev/null\n> > +++ b/midx-write.c\n> > @@ -0,0 +1,2 @@\n> > +#include \"git-compat-util.h\"\n> > +#include \"midx.h\"\n>\n> I noticed that \"nm midx-write.o\" after this step gives us nothing.\n> A source that produces absolutely nothing may not successfully\n> compile everywhere.  It is unlikely we will stop the series at this\n> step and it won't break bisection, though.\n>\n> I do not quite see why the first 3 patches need to be split this\n> way, rather than being done as a single step.  Is it making the\n> review any simpler?\n\nI was hoping that it would make review simpler. If you're concerned\nabout an empty compilation unit, we could:\n\n  - combine the first three patches into one, so that we start by just\n    porting `midx_repack()`\n\n  - combine the first seven patches into one, so that the move is done in\n    a single step, or\n\n  - leave it as-is\n\nWhatever you think makes most sense is fine with me. The original\nmotivation behind splitting them was to make the steps easier to review,\nbut if that doesn't seem to be the case, I'm happy to combine them.\n\nThanks,\nTaylor\n"},{"id":"491534","messageId":"ZgH2dhjECOTr0GMG@nand.local","threadId":"61199","inReplyTo":"xmqqv85ahx14.fsf@gitster.g","subject":"Re: [PATCH 08/11] midx-write.c: avoid directly managed temporary strbuf","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-25T22:11:02Z","receivedAt":"2024-03-25T22:11:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 25, 2024 at 01:33:11PM -0700, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> > In midx-write.c::midx_repack(), we construct the command-line arguments\n> > for a pack-objects invocation which will combine objects from the packs\n> > below our `--batch-size` option.\n> >\n> > To construct the base name of the output pack, we use a temporary\n> > strbuf, and then push the result of that onto the strvec which holds the\n> > command-line arguments, after which point we release the strbuf.\n> >\n> > We could replace this by doing something like:\n> >\n> >     struct strbuf buf = STRBUF_INIT;\n> >     strbuf_addf(&buf, \"%s/pack/pack\", object_dir);\n> >     strvec_push_nodup(&cmd.args, strbuf_detach(&buf));\n>\n> Hmph, I thought I saw another patch recently that uses\n> strvec_pushf() to simplify such a sequence.  Does the technique\n> apply here as well?\n>\n> Ah, yes, exactly.  See <9483038c-9529-4243-9b9a-97254fac29c1@web.de>\n\nHah. I wrote this patch on Saturday but didn't read the list before\nsending it on Monday. Serves me right ;-). Feel free to drop this one,\nor I can reroll based on René's patch.\n\nEither is fine, just let me know :-).\n\nThanks,\nTaylor\n"},{"id":"491535","messageId":"ZgH2rQYWbj+UkNIc@nand.local","threadId":"61199","inReplyTo":"xmqqjzlqhwnc.fsf@gitster.g","subject":"Re: [PATCH 10/11] midx-write.c: check count of packs to repack after grouping","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-25T22:11:57Z","receivedAt":"2024-03-25T22:12:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 25, 2024 at 01:41:27PM -0700, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> [...]\n>\n> Queued.  Thanks.\n\nThanks very much!\n\nThanks,\nTaylor\n"},{"id":"491651","messageId":"20240327082949.GB830163@coredump.intra.peff.net","threadId":"61199","inReplyTo":"5475b09a7afc4d55a8e1a1a72f20fa9109447cec.1711387439.git.me@ttaylorr.com","subject":"Re: [PATCH 09/11] midx-write.c: factor out common want_included_pack() routine","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-03-27T08:29:49Z","receivedAt":"2024-03-27T08:29:50Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 25, 2024 at 01:24:44PM -0400, Taylor Blau wrote:\n\n> We could extract two functions (say, `want_included_pack()` and a\n> `_nonempty()` variant), but this is not necessary. For the case in\n> `fill_included_packs_all()` which does not check the pack size, we add\n> all of the pack's objects assuming that the pack meets all other\n> criteria. But if the pack is empty in the first place, we add all of its\n> zero objects, so whether or not we \"accept\" or \"reject\" it in the first\n> place is irrelevant.\n\nOK, that makes sense. It does mean that we call the expensive-ish\nopen_pack_index() just to find out that it has 0 objects. But I guess if\nwe didn't reject it at this point, we'd soon open it anyway, so it's\nprobably not a big deal.\n\n> +static int want_included_pack(struct repository *r,\n> +\t\t\t      struct multi_pack_index *m,\n> +\t\t\t      int pack_kept_objects,\n> +\t\t\t      uint32_t pack_int_id)\n\nI wondered about this funky pack_int_id interface, rather than just\nhaving the caller pass in the pack struct. But we need it because\none of the callers needs to load the pack struct by calling\nprepare_midx_pack().\n\nThat should be a quick noop for the other caller, since the entry in\nm->packs[] will already be filled in (and if it's not, then you've just\nfixed a bug!).\n\nSo this all looks good to me.\n\n-Peff\n"},{"id":"491652","messageId":"20240327083725.GC830163@coredump.intra.peff.net","threadId":"61199","inReplyTo":"736be63234baf7fc6df8259d9bb7298858b2bc74.1711387439.git.me@ttaylorr.com","subject":"Re: [PATCH 11/11] midx-write.c: use `--stdin-packs` when repacking","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-03-27T08:37:25Z","receivedAt":"2024-03-27T08:37:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 25, 2024 at 01:24:50PM -0400, Taylor Blau wrote:\n\n> Though correct, this typically yields a poorly structured pack, since\n> providing the objects list over stdin does not give pack-objects a\n> chance to discover the namehash values for each object, leading to\n> sub-optimal delta selection.\n> \n> We can use `--stdin-packs` instead, which has a couple of benefits:\n> \n>   - it does a supplemental walk over objects in the supplied list of\n>     packs to discover their namehash, leading to higher-quality delta\n>     selection\n> \n>   - it requires us to list far less data over stdin; instead of listing\n>     each object in the resulting pack, we need only list the\n>     constituent packs from which those objects were selected in the MIDX\n\nYeah, using --stdin-packs makes much more sense to me. I have to admit\nthat I do not really see the point \"multi-pack-index repack\" in the\nfirst place. You'd presumably be better off with geometric repacking,\nand I think \"repack --geometric --write-midx\" will do things in the\nright order (new pack, then write midx, then delete redundant packs).\n\nI guess \"multi-pack-index repack\" came before geometric repacking, and\nis maybe redundant-ish now? If that's the case, then I guess I don't\ncare that much about optimizing its packs. ;) But we can't really delete\nit without a deprecation period, so making it more sensible in the\nmeantime is reasonable to me.\n\n-Peff\n"},{"id":"491653","messageId":"20240327083911.GD830163@coredump.intra.peff.net","threadId":"61199","inReplyTo":"cover.1711387439.git.me@ttaylorr.com","subject":"Re: [PATCH 00/11] midx: split MIDX writing routines into midx-write.c, cleanup","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-03-27T08:39:11Z","receivedAt":"2024-03-27T08:39:13Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 25, 2024 at 01:24:11PM -0400, Taylor Blau wrote:\n\n> This is a small-ish series that I worked on to split the functions from\n> midx.c which are responsible for writing MIDX files into a separate\n> compilation unit, midx-write.c.\n> \n> This is done for a couple of reasons:\n> \n>   - It reduces the size of midx.c, which is already quite large, thus\n>     making it easier to read.\n> \n>   - It more clearly separates responsibility between the two, similar to\n>     the division between pack-bitmap.c, and pack-bitmap-write.c.\n\nYeah, I think this is a good cleanup. Like Junio, I did wonder about the\npiece-meal movement. I kind of wonder if one big patch moving everything\nwould have been just as easy to review. But I'm happy either way.\n\nThe other refactoring patches all looked good to me.\n\n-Peff\n"},{"id":"491987","messageId":"cover.1712006190.git.me@ttaylorr.com","threadId":"61199","inReplyTo":"cover.1711387439.git.me@ttaylorr.com","subject":"[PATCH v2 0/4] midx: split MIDX writing routines into midx-write.c, cleanup","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-01T21:16:30Z","receivedAt":"2024-04-01T21:16:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is a reroll of my series 'tb/midx-write', now based on\n'rs/midx-use-strvec-pushf', and reworked to incorporate feedback.\n\nThe series is mostly unchanged since last time, and the goal remains to\nsplit out the writing-related functions from midx.c into a separate\ncompilation unit, midx-write.c.\n\nNotable changes since last time include:\n\n  - Rebasing on top of 'rs/midx-use-strvec-pushf', and dropping my patch\n    which does the same thing as René's.\n\n  - Combining the piece-meal migration patches from midx.c ->\n    midx-write.c into a single patch.\n\nThanks in advance for your review!\n\nTaylor Blau (4):\n  midx-write: move writing-related functions from midx.c\n  midx-write.c: factor out common want_included_pack() routine\n  midx-write.c: check count of packs to repack after grouping\n  midx-write.c: use `--stdin-packs` when repacking\n\n Makefile     |    1 +\n midx-write.c | 1525 +++++++++++++++++++++++++++++++++++++++++++++++++\n midx.c       | 1553 +-------------------------------------------------\n midx.h       |   19 +\n 4 files changed, 1559 insertions(+), 1539 deletions(-)\n create mode 100644 midx-write.c\n\nRange-diff against v1:\n 1:  ffa8ba18de <  -:  ---------- midx-write: initial commit\n 2:  b776fd528d <  -:  ---------- midx: extern a pair of shared functions\n 3:  487a0ccda8 <  -:  ---------- midx: move `midx_repack` (and related functions) to midx-write.c\n 4:  e2b6459aa8 <  -:  ---------- midx: move `expire_midx_packs` to midx-write.c\n 5:  31d2e074fb <  -:  ---------- midx: move `write_midx_file_only` to midx-write.c\n 6:  73977036d7 <  -:  ---------- midx: move `write_midx_file` to midx-write.c\n 7:  e83ef66cf5 !  1:  8637ee3d0e midx: move `write_midx_internal` (and related functions) to midx-write.c\n    @@ Metadata\n     Author: Taylor Blau <me@ttaylorr.com>\n     \n      ## Commit message ##\n    -    midx: move `write_midx_internal` (and related functions) to midx-write.c\n    +    midx-write: move writing-related functions from midx.c\n     \n    -    Move the last writing-related function from midx.c to midx-write.c. This\n    -    patch moves `write_midx_internal()`, along with all of the functions\n    -    used to implement it into midx-write.c.\n    +    Introduce a new midx-write.c source file, which holds all of the\n    +    functionality from the MIDX sub-system related to writing new MIDX files.\n     \n    -    Like previous patch, this patch too does not introduce any functional\n    -    changes, and is best moved with `--color-moved`.\n    +    Similar to the relationship between \"pack-bitmap.c\" and\n    +    \"pack-bitmap-write.c\", this source file will hold code that is specific\n    +    to writing MIDX files as opposed to reading them (the latter will remain\n    +    in midx.c).\n     \n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n     \n    - ## midx-write.c ##\n    + ## Makefile ##\n    +@@ Makefile: LIB_OBJS += merge-ort-wrappers.o\n    + LIB_OBJS += merge-recursive.o\n    + LIB_OBJS += merge.o\n    + LIB_OBJS += midx.o\n    ++LIB_OBJS += midx-write.o\n    + LIB_OBJS += name-hash.o\n    + LIB_OBJS += negotiator/default.o\n    + LIB_OBJS += negotiator/noop.o\n    +\n    + ## midx-write.c (new) ##\n     @@\n    - #include \"git-compat-util.h\"\n    ++#include \"git-compat-util.h\"\n     +#include \"abspath.h\"\n    - #include \"config.h\"\n    - #include \"hex.h\"\n    ++#include \"config.h\"\n    ++#include \"hex.h\"\n     +#include \"lockfile.h\"\n    - #include \"packfile.h\"\n    ++#include \"packfile.h\"\n     +#include \"object-file.h\"\n     +#include \"hash-lookup.h\"\n    - #include \"midx.h\"\n    - #include \"progress.h\"\n    ++#include \"midx.h\"\n    ++#include \"progress.h\"\n     +#include \"trace2.h\"\n    - #include \"run-command.h\"\n    ++#include \"run-command.h\"\n     +#include \"chunk-format.h\"\n    - #include \"pack-bitmap.h\"\n    ++#include \"pack-bitmap.h\"\n     +#include \"refs.h\"\n    - #include \"revision.h\"\n    ++#include \"revision.h\"\n     +#include \"list-objects.h\"\n    - \n    --extern int write_midx_internal(const char *object_dir,\n    ++\n     +#define PACK_EXPIRED UINT_MAX\n     +#define BITMAP_POS_UNKNOWN (~((uint32_t)0))\n     +#define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n    @@ midx-write.c\n     +}\n     +\n     +static int write_midx_internal(const char *object_dir,\n    - \t\t\t       struct string_list *packs_to_include,\n    - \t\t\t       struct string_list *packs_to_drop,\n    - \t\t\t       const char *preferred_pack_name,\n    - \t\t\t       const char *refs_snapshot,\n    --\t\t\t       unsigned flags);\n    ++\t\t\t       struct string_list *packs_to_include,\n    ++\t\t\t       struct string_list *packs_to_drop,\n    ++\t\t\t       const char *preferred_pack_name,\n    ++\t\t\t       const char *refs_snapshot,\n     +\t\t\t       unsigned flags)\n     +{\n     +\tstruct strbuf midx_name = STRBUF_INIT;\n    @@ midx-write.c\n     +\tint dropped_packs = 0;\n     +\tint result = 0;\n     +\tstruct chunkfile *cf;\n    - \n    --extern struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n    --\t\t\t\t\t\t\tconst char *object_dir);\n    ++\n     +\ttrace2_region_enter(\"midx\", \"write_midx_internal\", the_repository);\n     +\n     +\tget_midx_filename(&midx_name, object_dir);\n    @@ midx-write.c\n     +\n     +\treturn result;\n     +}\n    - \n    - int write_midx_file(const char *object_dir,\n    - \t\t    const char *preferred_pack_name,\n    ++\n    ++int write_midx_file(const char *object_dir,\n    ++\t\t    const char *preferred_pack_name,\n    ++\t\t    const char *refs_snapshot,\n    ++\t\t    unsigned flags)\n    ++{\n    ++\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n    ++\t\t\t\t   refs_snapshot, flags);\n    ++}\n    ++\n    ++int write_midx_file_only(const char *object_dir,\n    ++\t\t\t struct string_list *packs_to_include,\n    ++\t\t\t const char *preferred_pack_name,\n    ++\t\t\t const char *refs_snapshot,\n    ++\t\t\t unsigned flags)\n    ++{\n    ++\treturn write_midx_internal(object_dir, packs_to_include, NULL,\n    ++\t\t\t\t   preferred_pack_name, refs_snapshot, flags);\n    ++}\n    ++\n    ++int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags)\n    ++{\n    ++\tuint32_t i, *count, result = 0;\n    ++\tstruct string_list packs_to_drop = STRING_LIST_INIT_DUP;\n    ++\tstruct multi_pack_index *m = lookup_multi_pack_index(r, object_dir);\n    ++\tstruct progress *progress = NULL;\n    ++\n    ++\tif (!m)\n    ++\t\treturn 0;\n    ++\n    ++\tCALLOC_ARRAY(count, m->num_packs);\n    ++\n    ++\tif (flags & MIDX_PROGRESS)\n    ++\t\tprogress = start_delayed_progress(_(\"Counting referenced objects\"),\n    ++\t\t\t\t\t  m->num_objects);\n    ++\tfor (i = 0; i < m->num_objects; i++) {\n    ++\t\tint pack_int_id = nth_midxed_pack_int_id(m, i);\n    ++\t\tcount[pack_int_id]++;\n    ++\t\tdisplay_progress(progress, i + 1);\n    ++\t}\n    ++\tstop_progress(&progress);\n    ++\n    ++\tif (flags & MIDX_PROGRESS)\n    ++\t\tprogress = start_delayed_progress(_(\"Finding and deleting unreferenced packfiles\"),\n    ++\t\t\t\t\t  m->num_packs);\n    ++\tfor (i = 0; i < m->num_packs; i++) {\n    ++\t\tchar *pack_name;\n    ++\t\tdisplay_progress(progress, i + 1);\n    ++\n    ++\t\tif (count[i])\n    ++\t\t\tcontinue;\n    ++\n    ++\t\tif (prepare_midx_pack(r, m, i))\n    ++\t\t\tcontinue;\n    ++\n    ++\t\tif (m->packs[i]->pack_keep || m->packs[i]->is_cruft)\n    ++\t\t\tcontinue;\n    ++\n    ++\t\tpack_name = xstrdup(m->packs[i]->pack_name);\n    ++\t\tclose_pack(m->packs[i]);\n    ++\n    ++\t\tstring_list_insert(&packs_to_drop, m->pack_names[i]);\n    ++\t\tunlink_pack_path(pack_name, 0);\n    ++\t\tfree(pack_name);\n    ++\t}\n    ++\tstop_progress(&progress);\n    ++\n    ++\tfree(count);\n    ++\n    ++\tif (packs_to_drop.nr)\n    ++\t\tresult = write_midx_internal(object_dir, NULL, &packs_to_drop, NULL, NULL, flags);\n    ++\n    ++\tstring_list_clear(&packs_to_drop, 0);\n    ++\n    ++\treturn result;\n    ++}\n    ++\n    ++struct repack_info {\n    ++\ttimestamp_t mtime;\n    ++\tuint32_t referenced_objects;\n    ++\tuint32_t pack_int_id;\n    ++};\n    ++\n    ++static int compare_by_mtime(const void *a_, const void *b_)\n    ++{\n    ++\tconst struct repack_info *a, *b;\n    ++\n    ++\ta = (const struct repack_info *)a_;\n    ++\tb = (const struct repack_info *)b_;\n    ++\n    ++\tif (a->mtime < b->mtime)\n    ++\t\treturn -1;\n    ++\tif (a->mtime > b->mtime)\n    ++\t\treturn 1;\n    ++\treturn 0;\n    ++}\n    ++\n    ++static int fill_included_packs_all(struct repository *r,\n    ++\t\t\t\t   struct multi_pack_index *m,\n    ++\t\t\t\t   unsigned char *include_pack)\n    ++{\n    ++\tuint32_t i, count = 0;\n    ++\tint pack_kept_objects = 0;\n    ++\n    ++\trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n    ++\n    ++\tfor (i = 0; i < m->num_packs; i++) {\n    ++\t\tif (prepare_midx_pack(r, m, i))\n    ++\t\t\tcontinue;\n    ++\t\tif (!pack_kept_objects && m->packs[i]->pack_keep)\n    ++\t\t\tcontinue;\n    ++\t\tif (m->packs[i]->is_cruft)\n    ++\t\t\tcontinue;\n    ++\n    ++\t\tinclude_pack[i] = 1;\n    ++\t\tcount++;\n    ++\t}\n    ++\n    ++\treturn count < 2;\n    ++}\n    ++\n    ++static int fill_included_packs_batch(struct repository *r,\n    ++\t\t\t\t     struct multi_pack_index *m,\n    ++\t\t\t\t     unsigned char *include_pack,\n    ++\t\t\t\t     size_t batch_size)\n    ++{\n    ++\tuint32_t i, packs_to_repack;\n    ++\tsize_t total_size;\n    ++\tstruct repack_info *pack_info;\n    ++\tint pack_kept_objects = 0;\n    ++\n    ++\tCALLOC_ARRAY(pack_info, m->num_packs);\n    ++\n    ++\trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n    ++\n    ++\tfor (i = 0; i < m->num_packs; i++) {\n    ++\t\tpack_info[i].pack_int_id = i;\n    ++\n    ++\t\tif (prepare_midx_pack(r, m, i))\n    ++\t\t\tcontinue;\n    ++\n    ++\t\tpack_info[i].mtime = m->packs[i]->mtime;\n    ++\t}\n    ++\n    ++\tfor (i = 0; i < m->num_objects; i++) {\n    ++\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n    ++\t\tpack_info[pack_int_id].referenced_objects++;\n    ++\t}\n    ++\n    ++\tQSORT(pack_info, m->num_packs, compare_by_mtime);\n    ++\n    ++\ttotal_size = 0;\n    ++\tpacks_to_repack = 0;\n    ++\tfor (i = 0; total_size < batch_size && i < m->num_packs; i++) {\n    ++\t\tint pack_int_id = pack_info[i].pack_int_id;\n    ++\t\tstruct packed_git *p = m->packs[pack_int_id];\n    ++\t\tsize_t expected_size;\n    ++\n    ++\t\tif (!p)\n    ++\t\t\tcontinue;\n    ++\t\tif (!pack_kept_objects && p->pack_keep)\n    ++\t\t\tcontinue;\n    ++\t\tif (p->is_cruft)\n    ++\t\t\tcontinue;\n    ++\t\tif (open_pack_index(p) || !p->num_objects)\n    ++\t\t\tcontinue;\n    ++\n    ++\t\texpected_size = st_mult(p->pack_size,\n    ++\t\t\t\t\tpack_info[i].referenced_objects);\n    ++\t\texpected_size /= p->num_objects;\n    ++\n    ++\t\tif (expected_size >= batch_size)\n    ++\t\t\tcontinue;\n    ++\n    ++\t\tpacks_to_repack++;\n    ++\t\ttotal_size += expected_size;\n    ++\t\tinclude_pack[pack_int_id] = 1;\n    ++\t}\n    ++\n    ++\tfree(pack_info);\n    ++\n    ++\tif (packs_to_repack < 2)\n    ++\t\treturn 1;\n    ++\n    ++\treturn 0;\n    ++}\n    ++\n    ++int midx_repack(struct repository *r, const char *object_dir, size_t batch_size, unsigned flags)\n    ++{\n    ++\tint result = 0;\n    ++\tuint32_t i;\n    ++\tunsigned char *include_pack;\n    ++\tstruct child_process cmd = CHILD_PROCESS_INIT;\n    ++\tFILE *cmd_in;\n    ++\tstruct multi_pack_index *m = lookup_multi_pack_index(r, object_dir);\n    ++\n    ++\t/*\n    ++\t * When updating the default for these configuration\n    ++\t * variables in builtin/repack.c, these must be adjusted\n    ++\t * to match.\n    ++\t */\n    ++\tint delta_base_offset = 1;\n    ++\tint use_delta_islands = 0;\n    ++\n    ++\tif (!m)\n    ++\t\treturn 0;\n    ++\n    ++\tCALLOC_ARRAY(include_pack, m->num_packs);\n    ++\n    ++\tif (batch_size) {\n    ++\t\tif (fill_included_packs_batch(r, m, include_pack, batch_size))\n    ++\t\t\tgoto cleanup;\n    ++\t} else if (fill_included_packs_all(r, m, include_pack))\n    ++\t\tgoto cleanup;\n    ++\n    ++\trepo_config_get_bool(r, \"repack.usedeltabaseoffset\", &delta_base_offset);\n    ++\trepo_config_get_bool(r, \"repack.usedeltaislands\", &use_delta_islands);\n    ++\n    ++\tstrvec_push(&cmd.args, \"pack-objects\");\n    ++\n    ++\tstrvec_pushf(&cmd.args, \"%s/pack/pack\", object_dir);\n    ++\n    ++\tif (delta_base_offset)\n    ++\t\tstrvec_push(&cmd.args, \"--delta-base-offset\");\n    ++\tif (use_delta_islands)\n    ++\t\tstrvec_push(&cmd.args, \"--delta-islands\");\n    ++\n    ++\tif (flags & MIDX_PROGRESS)\n    ++\t\tstrvec_push(&cmd.args, \"--progress\");\n    ++\telse\n    ++\t\tstrvec_push(&cmd.args, \"-q\");\n    ++\n    ++\tcmd.git_cmd = 1;\n    ++\tcmd.in = cmd.out = -1;\n    ++\n    ++\tif (start_command(&cmd)) {\n    ++\t\terror(_(\"could not start pack-objects\"));\n    ++\t\tresult = 1;\n    ++\t\tgoto cleanup;\n    ++\t}\n    ++\n    ++\tcmd_in = xfdopen(cmd.in, \"w\");\n    ++\n    ++\tfor (i = 0; i < m->num_objects; i++) {\n    ++\t\tstruct object_id oid;\n    ++\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n    ++\n    ++\t\tif (!include_pack[pack_int_id])\n    ++\t\t\tcontinue;\n    ++\n    ++\t\tnth_midxed_object_oid(&oid, m, i);\n    ++\t\tfprintf(cmd_in, \"%s\\n\", oid_to_hex(&oid));\n    ++\t}\n    ++\tfclose(cmd_in);\n    ++\n    ++\tif (finish_command(&cmd)) {\n    ++\t\terror(_(\"could not finish pack-objects\"));\n    ++\t\tresult = 1;\n    ++\t\tgoto cleanup;\n    ++\t}\n    ++\n    ++\tresult = write_midx_internal(object_dir, NULL, NULL, NULL, NULL, flags);\n    ++\n    ++cleanup:\n    ++\tfree(include_pack);\n    ++\treturn result;\n    ++}\n     \n      ## midx.c ##\n     @@\n    @@ midx.c\n     -#include \"list-objects.h\"\n      #include \"pack-revindex.h\"\n      \n    --struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n    --\t\t\t\t\t\t\tconst char *object_dir);\n    --\n    --int write_midx_internal(const char *object_dir,\n    --\t\t\t       struct string_list *packs_to_include,\n    --\t\t\t       struct string_list *packs_to_drop,\n    --\t\t\t       const char *preferred_pack_name,\n    --\t\t\t       const char *refs_snapshot,\n    --\t\t\t       unsigned flags);\n    --\n     -#define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n     -#define MIDX_VERSION 1\n     -#define MIDX_BYTE_FILE_VERSION 4\n    @@ midx.c: int prepare_multi_pack_index_one(struct repository *r, const char *objec\n     -\treturn ret;\n     -}\n     -\n    --struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n    +-static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n     -\t\t\t\t\t\t\tconst char *object_dir)\n     -{\n     -\tstruct multi_pack_index *result = NULL;\n    @@ midx.c: int prepare_multi_pack_index_one(struct repository *r, const char *objec\n     -\treturn result;\n     -}\n     -\n    --int write_midx_internal(const char *object_dir,\n    --\t\t\tstruct string_list *packs_to_include,\n    --\t\t\tstruct string_list *packs_to_drop,\n    --\t\t\tconst char *preferred_pack_name,\n    --\t\t\tconst char *refs_snapshot,\n    --\t\t\tunsigned flags)\n    +-static int write_midx_internal(const char *object_dir,\n    +-\t\t\t       struct string_list *packs_to_include,\n    +-\t\t\t       struct string_list *packs_to_drop,\n    +-\t\t\t       const char *preferred_pack_name,\n    +-\t\t\t       const char *refs_snapshot,\n    +-\t\t\t       unsigned flags)\n     -{\n     -\tstruct strbuf midx_name = STRBUF_INIT;\n     -\tunsigned char midx_hash[GIT_MAX_RAWSZ];\n    @@ midx.c: int prepare_multi_pack_index_one(struct repository *r, const char *objec\n     -\n     -\treturn result;\n     -}\n    +-\n    +-int write_midx_file(const char *object_dir,\n    +-\t\t    const char *preferred_pack_name,\n    +-\t\t    const char *refs_snapshot,\n    +-\t\t    unsigned flags)\n    +-{\n    +-\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n    +-\t\t\t\t   refs_snapshot, flags);\n    +-}\n    +-\n    +-int write_midx_file_only(const char *object_dir,\n    +-\t\t\t struct string_list *packs_to_include,\n    +-\t\t\t const char *preferred_pack_name,\n    +-\t\t\t const char *refs_snapshot,\n    +-\t\t\t unsigned flags)\n    +-{\n    +-\treturn write_midx_internal(object_dir, packs_to_include, NULL,\n    +-\t\t\t\t   preferred_pack_name, refs_snapshot, flags);\n    +-}\n     -\n      struct clear_midx_data {\n      \tchar *keep;\n    @@ midx.c: static void clear_midx_file_ext(const char *full_path, size_t full_path_\n      {\n      \tstruct clear_midx_data data;\n      \tmemset(&data, 0, sizeof(struct clear_midx_data));\n    +@@ midx.c: int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n    + \n    + \treturn verify_midx_error;\n    + }\n    +-\n    +-int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags)\n    +-{\n    +-\tuint32_t i, *count, result = 0;\n    +-\tstruct string_list packs_to_drop = STRING_LIST_INIT_DUP;\n    +-\tstruct multi_pack_index *m = lookup_multi_pack_index(r, object_dir);\n    +-\tstruct progress *progress = NULL;\n    +-\n    +-\tif (!m)\n    +-\t\treturn 0;\n    +-\n    +-\tCALLOC_ARRAY(count, m->num_packs);\n    +-\n    +-\tif (flags & MIDX_PROGRESS)\n    +-\t\tprogress = start_delayed_progress(_(\"Counting referenced objects\"),\n    +-\t\t\t\t\t  m->num_objects);\n    +-\tfor (i = 0; i < m->num_objects; i++) {\n    +-\t\tint pack_int_id = nth_midxed_pack_int_id(m, i);\n    +-\t\tcount[pack_int_id]++;\n    +-\t\tdisplay_progress(progress, i + 1);\n    +-\t}\n    +-\tstop_progress(&progress);\n    +-\n    +-\tif (flags & MIDX_PROGRESS)\n    +-\t\tprogress = start_delayed_progress(_(\"Finding and deleting unreferenced packfiles\"),\n    +-\t\t\t\t\t  m->num_packs);\n    +-\tfor (i = 0; i < m->num_packs; i++) {\n    +-\t\tchar *pack_name;\n    +-\t\tdisplay_progress(progress, i + 1);\n    +-\n    +-\t\tif (count[i])\n    +-\t\t\tcontinue;\n    +-\n    +-\t\tif (prepare_midx_pack(r, m, i))\n    +-\t\t\tcontinue;\n    +-\n    +-\t\tif (m->packs[i]->pack_keep || m->packs[i]->is_cruft)\n    +-\t\t\tcontinue;\n    +-\n    +-\t\tpack_name = xstrdup(m->packs[i]->pack_name);\n    +-\t\tclose_pack(m->packs[i]);\n    +-\n    +-\t\tstring_list_insert(&packs_to_drop, m->pack_names[i]);\n    +-\t\tunlink_pack_path(pack_name, 0);\n    +-\t\tfree(pack_name);\n    +-\t}\n    +-\tstop_progress(&progress);\n    +-\n    +-\tfree(count);\n    +-\n    +-\tif (packs_to_drop.nr)\n    +-\t\tresult = write_midx_internal(object_dir, NULL, &packs_to_drop, NULL, NULL, flags);\n    +-\n    +-\tstring_list_clear(&packs_to_drop, 0);\n    +-\n    +-\treturn result;\n    +-}\n    +-\n    +-struct repack_info {\n    +-\ttimestamp_t mtime;\n    +-\tuint32_t referenced_objects;\n    +-\tuint32_t pack_int_id;\n    +-};\n    +-\n    +-static int compare_by_mtime(const void *a_, const void *b_)\n    +-{\n    +-\tconst struct repack_info *a, *b;\n    +-\n    +-\ta = (const struct repack_info *)a_;\n    +-\tb = (const struct repack_info *)b_;\n    +-\n    +-\tif (a->mtime < b->mtime)\n    +-\t\treturn -1;\n    +-\tif (a->mtime > b->mtime)\n    +-\t\treturn 1;\n    +-\treturn 0;\n    +-}\n    +-\n    +-static int fill_included_packs_all(struct repository *r,\n    +-\t\t\t\t   struct multi_pack_index *m,\n    +-\t\t\t\t   unsigned char *include_pack)\n    +-{\n    +-\tuint32_t i, count = 0;\n    +-\tint pack_kept_objects = 0;\n    +-\n    +-\trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n    +-\n    +-\tfor (i = 0; i < m->num_packs; i++) {\n    +-\t\tif (prepare_midx_pack(r, m, i))\n    +-\t\t\tcontinue;\n    +-\t\tif (!pack_kept_objects && m->packs[i]->pack_keep)\n    +-\t\t\tcontinue;\n    +-\t\tif (m->packs[i]->is_cruft)\n    +-\t\t\tcontinue;\n    +-\n    +-\t\tinclude_pack[i] = 1;\n    +-\t\tcount++;\n    +-\t}\n    +-\n    +-\treturn count < 2;\n    +-}\n    +-\n    +-static int fill_included_packs_batch(struct repository *r,\n    +-\t\t\t\t     struct multi_pack_index *m,\n    +-\t\t\t\t     unsigned char *include_pack,\n    +-\t\t\t\t     size_t batch_size)\n    +-{\n    +-\tuint32_t i, packs_to_repack;\n    +-\tsize_t total_size;\n    +-\tstruct repack_info *pack_info;\n    +-\tint pack_kept_objects = 0;\n    +-\n    +-\tCALLOC_ARRAY(pack_info, m->num_packs);\n    +-\n    +-\trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n    +-\n    +-\tfor (i = 0; i < m->num_packs; i++) {\n    +-\t\tpack_info[i].pack_int_id = i;\n    +-\n    +-\t\tif (prepare_midx_pack(r, m, i))\n    +-\t\t\tcontinue;\n    +-\n    +-\t\tpack_info[i].mtime = m->packs[i]->mtime;\n    +-\t}\n    +-\n    +-\tfor (i = 0; i < m->num_objects; i++) {\n    +-\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n    +-\t\tpack_info[pack_int_id].referenced_objects++;\n    +-\t}\n    +-\n    +-\tQSORT(pack_info, m->num_packs, compare_by_mtime);\n    +-\n    +-\ttotal_size = 0;\n    +-\tpacks_to_repack = 0;\n    +-\tfor (i = 0; total_size < batch_size && i < m->num_packs; i++) {\n    +-\t\tint pack_int_id = pack_info[i].pack_int_id;\n    +-\t\tstruct packed_git *p = m->packs[pack_int_id];\n    +-\t\tsize_t expected_size;\n    +-\n    +-\t\tif (!p)\n    +-\t\t\tcontinue;\n    +-\t\tif (!pack_kept_objects && p->pack_keep)\n    +-\t\t\tcontinue;\n    +-\t\tif (p->is_cruft)\n    +-\t\t\tcontinue;\n    +-\t\tif (open_pack_index(p) || !p->num_objects)\n    +-\t\t\tcontinue;\n    +-\n    +-\t\texpected_size = st_mult(p->pack_size,\n    +-\t\t\t\t\tpack_info[i].referenced_objects);\n    +-\t\texpected_size /= p->num_objects;\n    +-\n    +-\t\tif (expected_size >= batch_size)\n    +-\t\t\tcontinue;\n    +-\n    +-\t\tpacks_to_repack++;\n    +-\t\ttotal_size += expected_size;\n    +-\t\tinclude_pack[pack_int_id] = 1;\n    +-\t}\n    +-\n    +-\tfree(pack_info);\n    +-\n    +-\tif (packs_to_repack < 2)\n    +-\t\treturn 1;\n    +-\n    +-\treturn 0;\n    +-}\n    +-\n    +-int midx_repack(struct repository *r, const char *object_dir, size_t batch_size, unsigned flags)\n    +-{\n    +-\tint result = 0;\n    +-\tuint32_t i;\n    +-\tunsigned char *include_pack;\n    +-\tstruct child_process cmd = CHILD_PROCESS_INIT;\n    +-\tFILE *cmd_in;\n    +-\tstruct multi_pack_index *m = lookup_multi_pack_index(r, object_dir);\n    +-\n    +-\t/*\n    +-\t * When updating the default for these configuration\n    +-\t * variables in builtin/repack.c, these must be adjusted\n    +-\t * to match.\n    +-\t */\n    +-\tint delta_base_offset = 1;\n    +-\tint use_delta_islands = 0;\n    +-\n    +-\tif (!m)\n    +-\t\treturn 0;\n    +-\n    +-\tCALLOC_ARRAY(include_pack, m->num_packs);\n    +-\n    +-\tif (batch_size) {\n    +-\t\tif (fill_included_packs_batch(r, m, include_pack, batch_size))\n    +-\t\t\tgoto cleanup;\n    +-\t} else if (fill_included_packs_all(r, m, include_pack))\n    +-\t\tgoto cleanup;\n    +-\n    +-\trepo_config_get_bool(r, \"repack.usedeltabaseoffset\", &delta_base_offset);\n    +-\trepo_config_get_bool(r, \"repack.usedeltaislands\", &use_delta_islands);\n    +-\n    +-\tstrvec_push(&cmd.args, \"pack-objects\");\n    +-\n    +-\tstrvec_pushf(&cmd.args, \"%s/pack/pack\", object_dir);\n    +-\n    +-\tif (delta_base_offset)\n    +-\t\tstrvec_push(&cmd.args, \"--delta-base-offset\");\n    +-\tif (use_delta_islands)\n    +-\t\tstrvec_push(&cmd.args, \"--delta-islands\");\n    +-\n    +-\tif (flags & MIDX_PROGRESS)\n    +-\t\tstrvec_push(&cmd.args, \"--progress\");\n    +-\telse\n    +-\t\tstrvec_push(&cmd.args, \"-q\");\n    +-\n    +-\tcmd.git_cmd = 1;\n    +-\tcmd.in = cmd.out = -1;\n    +-\n    +-\tif (start_command(&cmd)) {\n    +-\t\terror(_(\"could not start pack-objects\"));\n    +-\t\tresult = 1;\n    +-\t\tgoto cleanup;\n    +-\t}\n    +-\n    +-\tcmd_in = xfdopen(cmd.in, \"w\");\n    +-\n    +-\tfor (i = 0; i < m->num_objects; i++) {\n    +-\t\tstruct object_id oid;\n    +-\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n    +-\n    +-\t\tif (!include_pack[pack_int_id])\n    +-\t\t\tcontinue;\n    +-\n    +-\t\tnth_midxed_object_oid(&oid, m, i);\n    +-\t\tfprintf(cmd_in, \"%s\\n\", oid_to_hex(&oid));\n    +-\t}\n    +-\tfclose(cmd_in);\n    +-\n    +-\tif (finish_command(&cmd)) {\n    +-\t\terror(_(\"could not finish pack-objects\"));\n    +-\t\tresult = 1;\n    +-\t\tgoto cleanup;\n    +-\t}\n    +-\n    +-\tresult = write_midx_internal(object_dir, NULL, NULL, NULL, NULL, flags);\n    +-\n    +-cleanup:\n    +-\tfree(include_pack);\n    +-\treturn result;\n    +-}\n     \n      ## midx.h ##\n     @@ midx.h: struct pack_entry;\n 8:  8e32755c49 <  -:  ---------- midx-write.c: avoid directly managed temporary strbuf\n 9:  5475b09a7a =  2:  0064e363c0 midx-write.c: factor out common want_included_pack() routine\n10:  f77e3167aa =  3:  b121f05a32 midx-write.c: check count of packs to repack after grouping\n11:  736be63234 !  4:  b5d6ba5802 midx-write.c: use `--stdin-packs` when repacking\n    @@ midx-write.c: int midx_repack(struct repository *r, const char *object_dir, size\n     +\tstrvec_pushl(&cmd.args, \"pack-objects\", \"--stdin-packs\", \"--non-empty\",\n     +\t\t     NULL);\n      \n    - \tif (delta_base_offset)\n    - \t\tstrvec_push(&cmd.args, \"--delta-base-offset\");\n    + \tstrvec_pushf(&cmd.args, \"%s/pack/pack\", object_dir);\n    + \n     @@ midx-write.c: int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n      \t}\n      \n\nbase-commit: 11c821f2f2a31e70fb5cc449f9a29401c333aad2\nprerequisite-patch-id: 3f7553c1b52071c935dd9ad3f94f7e675247498b\n-- \n2.44.0.330.g158d2a670b4\n"},{"id":"491989","messageId":"8637ee3d0e29d65dd9cde3d40b5987b18a12b270.1712006190.git.me@ttaylorr.com","threadId":"61199","inReplyTo":"cover.1712006190.git.me@ttaylorr.com","subject":"[PATCH v2 1/4] midx-write: move writing-related functions from midx.c","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-01T21:16:34Z","receivedAt":"2024-04-01T21:16:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a new midx-write.c source file, which holds all of the\nfunctionality from the MIDX sub-system related to writing new MIDX files.\n\nSimilar to the relationship between \"pack-bitmap.c\" and\n\"pack-bitmap-write.c\", this source file will hold code that is specific\nto writing MIDX files as opposed to reading them (the latter will remain\nin midx.c).\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n Makefile     |    1 +\n midx-write.c | 1521 ++++++++++++++++++++++++++++++++++++++++++++++++\n midx.c       | 1553 +-------------------------------------------------\n midx.h       |   19 +\n 4 files changed, 1555 insertions(+), 1539 deletions(-)\n create mode 100644 midx-write.c\n\ndiff --git a/Makefile b/Makefile\nindex 4e255c81f2..cf44a964c0 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1072,6 +1072,7 @@ LIB_OBJS += merge-ort-wrappers.o\n LIB_OBJS += merge-recursive.o\n LIB_OBJS += merge.o\n LIB_OBJS += midx.o\n+LIB_OBJS += midx-write.o\n LIB_OBJS += name-hash.o\n LIB_OBJS += negotiator/default.o\n LIB_OBJS += negotiator/noop.o\ndiff --git a/midx-write.c b/midx-write.c\nnew file mode 100644\nindex 0000000000..5242d2a724\n--- /dev/null\n+++ b/midx-write.c\n@@ -0,0 +1,1521 @@\n+#include \"git-compat-util.h\"\n+#include \"abspath.h\"\n+#include \"config.h\"\n+#include \"hex.h\"\n+#include \"lockfile.h\"\n+#include \"packfile.h\"\n+#include \"object-file.h\"\n+#include \"hash-lookup.h\"\n+#include \"midx.h\"\n+#include \"progress.h\"\n+#include \"trace2.h\"\n+#include \"run-command.h\"\n+#include \"chunk-format.h\"\n+#include \"pack-bitmap.h\"\n+#include \"refs.h\"\n+#include \"revision.h\"\n+#include \"list-objects.h\"\n+\n+#define PACK_EXPIRED UINT_MAX\n+#define BITMAP_POS_UNKNOWN (~((uint32_t)0))\n+#define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n+#define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n+\n+extern int midx_checksum_valid(struct multi_pack_index *m);\n+extern void clear_midx_files_ext(const char *object_dir, const char *ext,\n+\t\t\t\t unsigned char *keep_hash);\n+extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n+\t\t\t\tconst char *idx_name);\n+\n+static size_t write_midx_header(struct hashfile *f,\n+\t\t\t\tunsigned char num_chunks,\n+\t\t\t\tuint32_t num_packs)\n+{\n+\thashwrite_be32(f, MIDX_SIGNATURE);\n+\thashwrite_u8(f, MIDX_VERSION);\n+\thashwrite_u8(f, oid_version(the_hash_algo));\n+\thashwrite_u8(f, num_chunks);\n+\thashwrite_u8(f, 0); /* unused */\n+\thashwrite_be32(f, num_packs);\n+\n+\treturn MIDX_HEADER_SIZE;\n+}\n+\n+struct pack_info {\n+\tuint32_t orig_pack_int_id;\n+\tchar *pack_name;\n+\tstruct packed_git *p;\n+\n+\tuint32_t bitmap_pos;\n+\tuint32_t bitmap_nr;\n+\n+\tunsigned expired : 1;\n+};\n+\n+static void fill_pack_info(struct pack_info *info,\n+\t\t\t   struct packed_git *p, const char *pack_name,\n+\t\t\t   uint32_t orig_pack_int_id)\n+{\n+\tmemset(info, 0, sizeof(struct pack_info));\n+\n+\tinfo->orig_pack_int_id = orig_pack_int_id;\n+\tinfo->pack_name = xstrdup(pack_name);\n+\tinfo->p = p;\n+\tinfo->bitmap_pos = BITMAP_POS_UNKNOWN;\n+}\n+\n+static int pack_info_compare(const void *_a, const void *_b)\n+{\n+\tstruct pack_info *a = (struct pack_info *)_a;\n+\tstruct pack_info *b = (struct pack_info *)_b;\n+\treturn strcmp(a->pack_name, b->pack_name);\n+}\n+\n+static int idx_or_pack_name_cmp(const void *_va, const void *_vb)\n+{\n+\tconst char *pack_name = _va;\n+\tconst struct pack_info *compar = _vb;\n+\n+\treturn cmp_idx_or_pack_name(pack_name, compar->pack_name);\n+}\n+\n+struct write_midx_context {\n+\tstruct pack_info *info;\n+\tsize_t nr;\n+\tsize_t alloc;\n+\tstruct multi_pack_index *m;\n+\tstruct progress *progress;\n+\tunsigned pack_paths_checked;\n+\n+\tstruct pack_midx_entry *entries;\n+\tsize_t entries_nr;\n+\n+\tuint32_t *pack_perm;\n+\tuint32_t *pack_order;\n+\tunsigned large_offsets_needed:1;\n+\tuint32_t num_large_offsets;\n+\n+\tint preferred_pack_idx;\n+\n+\tstruct string_list *to_include;\n+};\n+\n+static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n+\t\t\t     const char *file_name, void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tstruct packed_git *p;\n+\n+\tif (ends_with(file_name, \".idx\")) {\n+\t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n+\t\t/*\n+\t\t * Note that at most one of ctx->m and ctx->to_include are set,\n+\t\t * so we are testing midx_contains_pack() and\n+\t\t * string_list_has_string() independently (guarded by the\n+\t\t * appropriate NULL checks).\n+\t\t *\n+\t\t * We could support passing to_include while reusing an existing\n+\t\t * MIDX, but don't currently since the reuse process drags\n+\t\t * forward all packs from an existing MIDX (without checking\n+\t\t * whether or not they appear in the to_include list).\n+\t\t *\n+\t\t * If we added support for that, these next two conditional\n+\t\t * should be performed independently (likely checking\n+\t\t * to_include before the existing MIDX).\n+\t\t */\n+\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n+\t\t\treturn;\n+\t\telse if (ctx->to_include &&\n+\t\t\t !string_list_has_string(ctx->to_include, file_name))\n+\t\t\treturn;\n+\n+\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n+\n+\t\tp = add_packed_git(full_path, full_path_len, 0);\n+\t\tif (!p) {\n+\t\t\twarning(_(\"failed to add packfile '%s'\"),\n+\t\t\t\tfull_path);\n+\t\t\treturn;\n+\t\t}\n+\n+\t\tif (open_pack_index(p)) {\n+\t\t\twarning(_(\"failed to open pack-index '%s'\"),\n+\t\t\t\tfull_path);\n+\t\t\tclose_pack(p);\n+\t\t\tfree(p);\n+\t\t\treturn;\n+\t\t}\n+\n+\t\tfill_pack_info(&ctx->info[ctx->nr], p, file_name, ctx->nr);\n+\t\tctx->nr++;\n+\t}\n+}\n+\n+struct pack_midx_entry {\n+\tstruct object_id oid;\n+\tuint32_t pack_int_id;\n+\ttime_t pack_mtime;\n+\tuint64_t offset;\n+\tunsigned preferred : 1;\n+};\n+\n+static int midx_oid_compare(const void *_a, const void *_b)\n+{\n+\tconst struct pack_midx_entry *a = (const struct pack_midx_entry *)_a;\n+\tconst struct pack_midx_entry *b = (const struct pack_midx_entry *)_b;\n+\tint cmp = oidcmp(&a->oid, &b->oid);\n+\n+\tif (cmp)\n+\t\treturn cmp;\n+\n+\t/* Sort objects in a preferred pack first when multiple copies exist. */\n+\tif (a->preferred > b->preferred)\n+\t\treturn -1;\n+\tif (a->preferred < b->preferred)\n+\t\treturn 1;\n+\n+\tif (a->pack_mtime > b->pack_mtime)\n+\t\treturn -1;\n+\telse if (a->pack_mtime < b->pack_mtime)\n+\t\treturn 1;\n+\n+\treturn a->pack_int_id - b->pack_int_id;\n+}\n+\n+static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n+\t\t\t\t      struct pack_midx_entry *e,\n+\t\t\t\t      uint32_t pos)\n+{\n+\tif (pos >= m->num_objects)\n+\t\treturn 1;\n+\n+\tnth_midxed_object_oid(&e->oid, m, pos);\n+\te->pack_int_id = nth_midxed_pack_int_id(m, pos);\n+\te->offset = nth_midxed_offset(m, pos);\n+\n+\t/* consider objects in midx to be from \"old\" packs */\n+\te->pack_mtime = 0;\n+\treturn 0;\n+}\n+\n+static void fill_pack_entry(uint32_t pack_int_id,\n+\t\t\t    struct packed_git *p,\n+\t\t\t    uint32_t cur_object,\n+\t\t\t    struct pack_midx_entry *entry,\n+\t\t\t    int preferred)\n+{\n+\tif (nth_packed_object_id(&entry->oid, p, cur_object) < 0)\n+\t\tdie(_(\"failed to locate object %d in packfile\"), cur_object);\n+\n+\tentry->pack_int_id = pack_int_id;\n+\tentry->pack_mtime = p->mtime;\n+\n+\tentry->offset = nth_packed_object_offset(p, cur_object);\n+\tentry->preferred = !!preferred;\n+}\n+\n+struct midx_fanout {\n+\tstruct pack_midx_entry *entries;\n+\tsize_t nr, alloc;\n+};\n+\n+static void midx_fanout_grow(struct midx_fanout *fanout, size_t nr)\n+{\n+\tif (nr < fanout->nr)\n+\t\tBUG(\"negative growth in midx_fanout_grow() (%\"PRIuMAX\" < %\"PRIuMAX\")\",\n+\t\t    (uintmax_t)nr, (uintmax_t)fanout->nr);\n+\tALLOC_GROW(fanout->entries, nr, fanout->alloc);\n+}\n+\n+static void midx_fanout_sort(struct midx_fanout *fanout)\n+{\n+\tQSORT(fanout->entries, fanout->nr, midx_oid_compare);\n+}\n+\n+static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n+\t\t\t\t\tstruct multi_pack_index *m,\n+\t\t\t\t\tuint32_t cur_fanout,\n+\t\t\t\t\tint preferred_pack)\n+{\n+\tuint32_t start = 0, end;\n+\tuint32_t cur_object;\n+\n+\tif (cur_fanout)\n+\t\tstart = ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n+\tend = ntohl(m->chunk_oid_fanout[cur_fanout]);\n+\n+\tfor (cur_object = start; cur_object < end; cur_object++) {\n+\t\tif ((preferred_pack > -1) &&\n+\t\t    (preferred_pack == nth_midxed_pack_int_id(m, cur_object))) {\n+\t\t\t/*\n+\t\t\t * Objects from preferred packs are added\n+\t\t\t * separately.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tmidx_fanout_grow(fanout, fanout->nr + 1);\n+\t\tnth_midxed_pack_midx_entry(m,\n+\t\t\t\t\t   &fanout->entries[fanout->nr],\n+\t\t\t\t\t   cur_object);\n+\t\tfanout->entries[fanout->nr].preferred = 0;\n+\t\tfanout->nr++;\n+\t}\n+}\n+\n+static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout,\n+\t\t\t\t\tstruct pack_info *info,\n+\t\t\t\t\tuint32_t cur_pack,\n+\t\t\t\t\tint preferred,\n+\t\t\t\t\tuint32_t cur_fanout)\n+{\n+\tstruct packed_git *pack = info[cur_pack].p;\n+\tuint32_t start = 0, end;\n+\tuint32_t cur_object;\n+\n+\tif (cur_fanout)\n+\t\tstart = get_pack_fanout(pack, cur_fanout - 1);\n+\tend = get_pack_fanout(pack, cur_fanout);\n+\n+\tfor (cur_object = start; cur_object < end; cur_object++) {\n+\t\tmidx_fanout_grow(fanout, fanout->nr + 1);\n+\t\tfill_pack_entry(cur_pack,\n+\t\t\t\tinfo[cur_pack].p,\n+\t\t\t\tcur_object,\n+\t\t\t\t&fanout->entries[fanout->nr],\n+\t\t\t\tpreferred);\n+\t\tfanout->nr++;\n+\t}\n+}\n+\n+/*\n+ * It is possible to artificially get into a state where there are many\n+ * duplicate copies of objects. That can create high memory pressure if\n+ * we are to create a list of all objects before de-duplication. To reduce\n+ * this memory pressure without a significant performance drop, automatically\n+ * group objects by the first byte of their object id. Use the IDX fanout\n+ * tables to group the data, copy to a local array, then sort.\n+ *\n+ * Copy only the de-duplicated entries (selected by most-recent modified time\n+ * of a packfile containing the object).\n+ */\n+static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n+\t\t\t\t\t\t  struct pack_info *info,\n+\t\t\t\t\t\t  uint32_t nr_packs,\n+\t\t\t\t\t\t  size_t *nr_objects,\n+\t\t\t\t\t\t  int preferred_pack)\n+{\n+\tuint32_t cur_fanout, cur_pack, cur_object;\n+\tsize_t alloc_objects, total_objects = 0;\n+\tstruct midx_fanout fanout = { 0 };\n+\tstruct pack_midx_entry *deduplicated_entries = NULL;\n+\tuint32_t start_pack = m ? m->num_packs : 0;\n+\n+\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++)\n+\t\ttotal_objects = st_add(total_objects,\n+\t\t\t\t       info[cur_pack].p->num_objects);\n+\n+\t/*\n+\t * As we de-duplicate by fanout value, we expect the fanout\n+\t * slices to be evenly distributed, with some noise. Hence,\n+\t * allocate slightly more than one 256th.\n+\t */\n+\talloc_objects = fanout.alloc = total_objects > 3200 ? total_objects / 200 : 16;\n+\n+\tALLOC_ARRAY(fanout.entries, fanout.alloc);\n+\tALLOC_ARRAY(deduplicated_entries, alloc_objects);\n+\t*nr_objects = 0;\n+\n+\tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n+\t\tfanout.nr = 0;\n+\n+\t\tif (m)\n+\t\t\tmidx_fanout_add_midx_fanout(&fanout, m, cur_fanout,\n+\t\t\t\t\t\t    preferred_pack);\n+\n+\t\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) {\n+\t\t\tint preferred = cur_pack == preferred_pack;\n+\t\t\tmidx_fanout_add_pack_fanout(&fanout,\n+\t\t\t\t\t\t    info, cur_pack,\n+\t\t\t\t\t\t    preferred, cur_fanout);\n+\t\t}\n+\n+\t\tif (-1 < preferred_pack && preferred_pack < start_pack)\n+\t\t\tmidx_fanout_add_pack_fanout(&fanout, info,\n+\t\t\t\t\t\t    preferred_pack, 1,\n+\t\t\t\t\t\t    cur_fanout);\n+\n+\t\tmidx_fanout_sort(&fanout);\n+\n+\t\t/*\n+\t\t * The batch is now sorted by OID and then mtime (descending).\n+\t\t * Take only the first duplicate.\n+\t\t */\n+\t\tfor (cur_object = 0; cur_object < fanout.nr; cur_object++) {\n+\t\t\tif (cur_object && oideq(&fanout.entries[cur_object - 1].oid,\n+\t\t\t\t\t\t&fanout.entries[cur_object].oid))\n+\t\t\t\tcontinue;\n+\n+\t\t\tALLOC_GROW(deduplicated_entries, st_add(*nr_objects, 1),\n+\t\t\t\t   alloc_objects);\n+\t\t\tmemcpy(&deduplicated_entries[*nr_objects],\n+\t\t\t       &fanout.entries[cur_object],\n+\t\t\t       sizeof(struct pack_midx_entry));\n+\t\t\t(*nr_objects)++;\n+\t\t}\n+\t}\n+\n+\tfree(fanout.entries);\n+\treturn deduplicated_entries;\n+}\n+\n+static int write_midx_pack_names(struct hashfile *f, void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tuint32_t i;\n+\tunsigned char padding[MIDX_CHUNK_ALIGNMENT];\n+\tsize_t written = 0;\n+\n+\tfor (i = 0; i < ctx->nr; i++) {\n+\t\tsize_t writelen;\n+\n+\t\tif (ctx->info[i].expired)\n+\t\t\tcontinue;\n+\n+\t\tif (i && strcmp(ctx->info[i].pack_name, ctx->info[i - 1].pack_name) <= 0)\n+\t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n+\t\t\t    ctx->info[i - 1].pack_name,\n+\t\t\t    ctx->info[i].pack_name);\n+\n+\t\twritelen = strlen(ctx->info[i].pack_name) + 1;\n+\t\thashwrite(f, ctx->info[i].pack_name, writelen);\n+\t\twritten += writelen;\n+\t}\n+\n+\t/* add padding to be aligned */\n+\ti = MIDX_CHUNK_ALIGNMENT - (written % MIDX_CHUNK_ALIGNMENT);\n+\tif (i < MIDX_CHUNK_ALIGNMENT) {\n+\t\tmemset(padding, 0, sizeof(padding));\n+\t\thashwrite(f, padding, i);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int write_midx_bitmapped_packs(struct hashfile *f, void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tsize_t i;\n+\n+\tfor (i = 0; i < ctx->nr; i++) {\n+\t\tstruct pack_info *pack = &ctx->info[i];\n+\t\tif (pack->expired)\n+\t\t\tcontinue;\n+\n+\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN && pack->bitmap_nr)\n+\t\t\tBUG(\"pack '%s' has no bitmap position, but has %d bitmapped object(s)\",\n+\t\t\t    pack->pack_name, pack->bitmap_nr);\n+\n+\t\thashwrite_be32(f, pack->bitmap_pos);\n+\t\thashwrite_be32(f, pack->bitmap_nr);\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_midx_oid_fanout(struct hashfile *f,\n+\t\t\t\t void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n+\tstruct pack_midx_entry *last = ctx->entries + ctx->entries_nr;\n+\tuint32_t count = 0;\n+\tuint32_t i;\n+\n+\t/*\n+\t* Write the first-level table (the list is sorted,\n+\t* but we use a 256-entry lookup to be able to avoid\n+\t* having to do eight extra binary search iterations).\n+\t*/\n+\tfor (i = 0; i < 256; i++) {\n+\t\tstruct pack_midx_entry *next = list;\n+\n+\t\twhile (next < last && next->oid.hash[0] == i) {\n+\t\t\tcount++;\n+\t\t\tnext++;\n+\t\t}\n+\n+\t\thashwrite_be32(f, count);\n+\t\tlist = next;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int write_midx_oid_lookup(struct hashfile *f,\n+\t\t\t\t void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tunsigned char hash_len = the_hash_algo->rawsz;\n+\tstruct pack_midx_entry *list = ctx->entries;\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *obj = list++;\n+\n+\t\tif (i < ctx->entries_nr - 1) {\n+\t\t\tstruct pack_midx_entry *next = list;\n+\t\t\tif (oidcmp(&obj->oid, &next->oid) >= 0)\n+\t\t\t\tBUG(\"OIDs not in order: %s >= %s\",\n+\t\t\t\t    oid_to_hex(&obj->oid),\n+\t\t\t\t    oid_to_hex(&next->oid));\n+\t\t}\n+\n+\t\thashwrite(f, obj->oid.hash, (int)hash_len);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int write_midx_object_offsets(struct hashfile *f,\n+\t\t\t\t     void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n+\tuint32_t i, nr_large_offset = 0;\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *obj = list++;\n+\n+\t\tif (ctx->pack_perm[obj->pack_int_id] == PACK_EXPIRED)\n+\t\t\tBUG(\"object %s is in an expired pack with int-id %d\",\n+\t\t\t    oid_to_hex(&obj->oid),\n+\t\t\t    obj->pack_int_id);\n+\n+\t\thashwrite_be32(f, ctx->pack_perm[obj->pack_int_id]);\n+\n+\t\tif (ctx->large_offsets_needed && obj->offset >> 31)\n+\t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n+\t\telse if (!ctx->large_offsets_needed && obj->offset >> 32)\n+\t\t\tBUG(\"object %s requires a large offset (%\"PRIx64\") but the MIDX is not writing large offsets!\",\n+\t\t\t    oid_to_hex(&obj->oid),\n+\t\t\t    obj->offset);\n+\t\telse\n+\t\t\thashwrite_be32(f, (uint32_t)obj->offset);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int write_midx_large_offsets(struct hashfile *f,\n+\t\t\t\t    void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tstruct pack_midx_entry *list = ctx->entries;\n+\tstruct pack_midx_entry *end = ctx->entries + ctx->entries_nr;\n+\tuint32_t nr_large_offset = ctx->num_large_offsets;\n+\n+\twhile (nr_large_offset) {\n+\t\tstruct pack_midx_entry *obj;\n+\t\tuint64_t offset;\n+\n+\t\tif (list >= end)\n+\t\t\tBUG(\"too many large-offset objects\");\n+\n+\t\tobj = list++;\n+\t\toffset = obj->offset;\n+\n+\t\tif (!(offset >> 31))\n+\t\t\tcontinue;\n+\n+\t\thashwrite_be64(f, offset);\n+\n+\t\tnr_large_offset--;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int write_midx_revindex(struct hashfile *f,\n+\t\t\t       void *data)\n+{\n+\tstruct write_midx_context *ctx = data;\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++)\n+\t\thashwrite_be32(f, ctx->pack_order[i]);\n+\n+\treturn 0;\n+}\n+\n+struct midx_pack_order_data {\n+\tuint32_t nr;\n+\tuint32_t pack;\n+\toff_t offset;\n+};\n+\n+static int midx_pack_order_cmp(const void *va, const void *vb)\n+{\n+\tconst struct midx_pack_order_data *a = va, *b = vb;\n+\tif (a->pack < b->pack)\n+\t\treturn -1;\n+\telse if (a->pack > b->pack)\n+\t\treturn 1;\n+\telse if (a->offset < b->offset)\n+\t\treturn -1;\n+\telse if (a->offset > b->offset)\n+\t\treturn 1;\n+\telse\n+\t\treturn 0;\n+}\n+\n+static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n+{\n+\tstruct midx_pack_order_data *data;\n+\tuint32_t *pack_order;\n+\tuint32_t i;\n+\n+\ttrace2_region_enter(\"midx\", \"midx_pack_order\", the_repository);\n+\n+\tALLOC_ARRAY(data, ctx->entries_nr);\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *e = &ctx->entries[i];\n+\t\tdata[i].nr = i;\n+\t\tdata[i].pack = ctx->pack_perm[e->pack_int_id];\n+\t\tif (!e->preferred)\n+\t\t\tdata[i].pack |= (1U << 31);\n+\t\tdata[i].offset = e->offset;\n+\t}\n+\n+\tQSORT(data, ctx->entries_nr, midx_pack_order_cmp);\n+\n+\tALLOC_ARRAY(pack_order, ctx->entries_nr);\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *e = &ctx->entries[data[i].nr];\n+\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]];\n+\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n+\t\t\tpack->bitmap_pos = i;\n+\t\tpack->bitmap_nr++;\n+\t\tpack_order[i] = data[i].nr;\n+\t}\n+\tfor (i = 0; i < ctx->nr; i++) {\n+\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[i]];\n+\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n+\t\t\tpack->bitmap_pos = 0;\n+\t}\n+\tfree(data);\n+\n+\ttrace2_region_leave(\"midx\", \"midx_pack_order\", the_repository);\n+\n+\treturn pack_order;\n+}\n+\n+static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n+\t\t\t\t     struct write_midx_context *ctx)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *tmp_file;\n+\n+\ttrace2_region_enter(\"midx\", \"write_midx_reverse_index\", the_repository);\n+\n+\tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex(midx_hash));\n+\n+\ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n+\t\t\t\t\tmidx_hash, WRITE_REV);\n+\n+\tif (finalize_object_file(tmp_file, buf.buf))\n+\t\tdie(_(\"cannot store reverse index file\"));\n+\n+\tstrbuf_release(&buf);\n+\n+\ttrace2_region_leave(\"midx\", \"write_midx_reverse_index\", the_repository);\n+}\n+\n+static void prepare_midx_packing_data(struct packing_data *pdata,\n+\t\t\t\t      struct write_midx_context *ctx)\n+{\n+\tuint32_t i;\n+\n+\ttrace2_region_enter(\"midx\", \"prepare_midx_packing_data\", the_repository);\n+\n+\tmemset(pdata, 0, sizeof(struct packing_data));\n+\tprepare_packing_data(the_repository, pdata);\n+\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n+\t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n+\n+\t\toe_set_in_pack(pdata, to,\n+\t\t\t       ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n+\t}\n+\n+\ttrace2_region_leave(\"midx\", \"prepare_midx_packing_data\", the_repository);\n+}\n+\n+static int add_ref_to_pending(const char *refname,\n+\t\t\t      const struct object_id *oid,\n+\t\t\t      int flag, void *cb_data)\n+{\n+\tstruct rev_info *revs = (struct rev_info*)cb_data;\n+\tstruct object_id peeled;\n+\tstruct object *object;\n+\n+\tif ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {\n+\t\twarning(\"symbolic ref is dangling: %s\", refname);\n+\t\treturn 0;\n+\t}\n+\n+\tif (!peel_iterated_oid(oid, &peeled))\n+\t\toid = &peeled;\n+\n+\tobject = parse_object_or_die(oid, refname);\n+\tif (object->type != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tadd_pending_object(revs, object, \"\");\n+\tif (bitmap_is_preferred_refname(revs->repo, refname))\n+\t\tobject->flags |= NEEDS_BITMAP;\n+\treturn 0;\n+}\n+\n+struct bitmap_commit_cb {\n+\tstruct commit **commits;\n+\tsize_t commits_nr, commits_alloc;\n+\n+\tstruct write_midx_context *ctx;\n+};\n+\n+static const struct object_id *bitmap_oid_access(size_t index,\n+\t\t\t\t\t\t const void *_entries)\n+{\n+\tconst struct pack_midx_entry *entries = _entries;\n+\treturn &entries[index].oid;\n+}\n+\n+static void bitmap_show_commit(struct commit *commit, void *_data)\n+{\n+\tstruct bitmap_commit_cb *data = _data;\n+\tint pos = oid_pos(&commit->object.oid, data->ctx->entries,\n+\t\t\t  data->ctx->entries_nr,\n+\t\t\t  bitmap_oid_access);\n+\tif (pos < 0)\n+\t\treturn;\n+\n+\tALLOC_GROW(data->commits, data->commits_nr + 1, data->commits_alloc);\n+\tdata->commits[data->commits_nr++] = commit;\n+}\n+\n+static int read_refs_snapshot(const char *refs_snapshot,\n+\t\t\t      struct rev_info *revs)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct object_id oid;\n+\tFILE *f = xfopen(refs_snapshot, \"r\");\n+\n+\twhile (strbuf_getline(&buf, f) != EOF) {\n+\t\tstruct object *object;\n+\t\tint preferred = 0;\n+\t\tchar *hex = buf.buf;\n+\t\tconst char *end = NULL;\n+\n+\t\tif (buf.len && *buf.buf == '+') {\n+\t\t\tpreferred = 1;\n+\t\t\thex = &buf.buf[1];\n+\t\t}\n+\n+\t\tif (parse_oid_hex(hex, &oid, &end) < 0)\n+\t\t\tdie(_(\"could not parse line: %s\"), buf.buf);\n+\t\tif (*end)\n+\t\t\tdie(_(\"malformed line: %s\"), buf.buf);\n+\n+\t\tobject = parse_object_or_die(&oid, NULL);\n+\t\tif (preferred)\n+\t\t\tobject->flags |= NEEDS_BITMAP;\n+\n+\t\tadd_pending_object(revs, object, \"\");\n+\t}\n+\n+\tfclose(f);\n+\tstrbuf_release(&buf);\n+\treturn 0;\n+}\n+static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p,\n+\t\t\t\t\t\t    const char *refs_snapshot,\n+\t\t\t\t\t\t    struct write_midx_context *ctx)\n+{\n+\tstruct rev_info revs;\n+\tstruct bitmap_commit_cb cb = {0};\n+\n+\ttrace2_region_enter(\"midx\", \"find_commits_for_midx_bitmap\",\n+\t\t\t    the_repository);\n+\n+\tcb.ctx = ctx;\n+\n+\trepo_init_revisions(the_repository, &revs, NULL);\n+\tif (refs_snapshot) {\n+\t\tread_refs_snapshot(refs_snapshot, &revs);\n+\t} else {\n+\t\tsetup_revisions(0, NULL, &revs, NULL);\n+\t\tfor_each_ref(add_ref_to_pending, &revs);\n+\t}\n+\n+\t/*\n+\t * Skipping promisor objects here is intentional, since it only excludes\n+\t * them from the list of reachable commits that we want to select from\n+\t * when computing the selection of MIDX'd commits to receive bitmaps.\n+\t *\n+\t * Reachability bitmaps do require that their objects be closed under\n+\t * reachability, but fetching any objects missing from promisors at this\n+\t * point is too late. But, if one of those objects can be reached from\n+\t * an another object that is included in the bitmap, then we will\n+\t * complain later that we don't have reachability closure (and fail\n+\t * appropriately).\n+\t */\n+\tfetch_if_missing = 0;\n+\trevs.exclude_promisor_objects = 1;\n+\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(_(\"revision walk setup failed\"));\n+\n+\ttraverse_commit_list(&revs, bitmap_show_commit, NULL, &cb);\n+\tif (indexed_commits_nr_p)\n+\t\t*indexed_commits_nr_p = cb.commits_nr;\n+\n+\trelease_revisions(&revs);\n+\n+\ttrace2_region_leave(\"midx\", \"find_commits_for_midx_bitmap\",\n+\t\t\t    the_repository);\n+\n+\treturn cb.commits;\n+}\n+\n+static int write_midx_bitmap(const char *midx_name,\n+\t\t\t     const unsigned char *midx_hash,\n+\t\t\t     struct packing_data *pdata,\n+\t\t\t     struct commit **commits,\n+\t\t\t     uint32_t commits_nr,\n+\t\t\t     uint32_t *pack_order,\n+\t\t\t     unsigned flags)\n+{\n+\tint ret, i;\n+\tuint16_t options = 0;\n+\tstruct pack_idx_entry **index;\n+\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name,\n+\t\t\t\t\thash_to_hex(midx_hash));\n+\n+\ttrace2_region_enter(\"midx\", \"write_midx_bitmap\", the_repository);\n+\n+\tif (flags & MIDX_WRITE_BITMAP_HASH_CACHE)\n+\t\toptions |= BITMAP_OPT_HASH_CACHE;\n+\n+\tif (flags & MIDX_WRITE_BITMAP_LOOKUP_TABLE)\n+\t\toptions |= BITMAP_OPT_LOOKUP_TABLE;\n+\n+\t/*\n+\t * Build the MIDX-order index based on pdata.objects (which is already\n+\t * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of\n+\t * this order).\n+\t */\n+\tALLOC_ARRAY(index, pdata->nr_objects);\n+\tfor (i = 0; i < pdata->nr_objects; i++)\n+\t\tindex[i] = &pdata->objects[i].idx;\n+\n+\tbitmap_writer_show_progress(flags & MIDX_PROGRESS);\n+\tbitmap_writer_build_type_index(pdata, index, pdata->nr_objects);\n+\n+\t/*\n+\t * bitmap_writer_finish expects objects in lex order, but pack_order\n+\t * gives us exactly that. use it directly instead of re-sorting the\n+\t * array.\n+\t *\n+\t * This changes the order of objects in 'index' between\n+\t * bitmap_writer_build_type_index and bitmap_writer_finish.\n+\t *\n+\t * The same re-ordering takes place in the single-pack bitmap code via\n+\t * write_idx_file(), which is called by finish_tmp_packfile(), which\n+\t * happens between bitmap_writer_build_type_index() and\n+\t * bitmap_writer_finish().\n+\t */\n+\tfor (i = 0; i < pdata->nr_objects; i++)\n+\t\tindex[pack_order[i]] = &pdata->objects[i].idx;\n+\n+\tbitmap_writer_select_commits(commits, commits_nr, -1);\n+\tret = bitmap_writer_build(pdata);\n+\tif (ret < 0)\n+\t\tgoto cleanup;\n+\n+\tbitmap_writer_set_checksum(midx_hash);\n+\tbitmap_writer_finish(index, pdata->nr_objects, bitmap_name, options);\n+\n+cleanup:\n+\tfree(index);\n+\tfree(bitmap_name);\n+\n+\ttrace2_region_leave(\"midx\", \"write_midx_bitmap\", the_repository);\n+\n+\treturn ret;\n+}\n+\n+static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n+\t\t\t\t\t\t\tconst char *object_dir)\n+{\n+\tstruct multi_pack_index *result = NULL;\n+\tstruct multi_pack_index *cur;\n+\tchar *obj_dir_real = real_pathdup(object_dir, 1);\n+\tstruct strbuf cur_path_real = STRBUF_INIT;\n+\n+\t/* Ensure the given object_dir is local, or a known alternate. */\n+\tfind_odb(r, obj_dir_real);\n+\n+\tfor (cur = get_multi_pack_index(r); cur; cur = cur->next) {\n+\t\tstrbuf_realpath(&cur_path_real, cur->object_dir, 1);\n+\t\tif (!strcmp(obj_dir_real, cur_path_real.buf)) {\n+\t\t\tresult = cur;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n+cleanup:\n+\tfree(obj_dir_real);\n+\tstrbuf_release(&cur_path_real);\n+\treturn result;\n+}\n+\n+static int write_midx_internal(const char *object_dir,\n+\t\t\t       struct string_list *packs_to_include,\n+\t\t\t       struct string_list *packs_to_drop,\n+\t\t\t       const char *preferred_pack_name,\n+\t\t\t       const char *refs_snapshot,\n+\t\t\t       unsigned flags)\n+{\n+\tstruct strbuf midx_name = STRBUF_INIT;\n+\tunsigned char midx_hash[GIT_MAX_RAWSZ];\n+\tuint32_t i;\n+\tstruct hashfile *f = NULL;\n+\tstruct lock_file lk;\n+\tstruct write_midx_context ctx = { 0 };\n+\tint bitmapped_packs_concat_len = 0;\n+\tint pack_name_concat_len = 0;\n+\tint dropped_packs = 0;\n+\tint result = 0;\n+\tstruct chunkfile *cf;\n+\n+\ttrace2_region_enter(\"midx\", \"write_midx_internal\", the_repository);\n+\n+\tget_midx_filename(&midx_name, object_dir);\n+\tif (safe_create_leading_directories(midx_name.buf))\n+\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n+\t\t\t  midx_name.buf);\n+\n+\tif (!packs_to_include) {\n+\t\t/*\n+\t\t * Only reference an existing MIDX when not filtering which\n+\t\t * packs to include, since all packs and objects are copied\n+\t\t * blindly from an existing MIDX if one is present.\n+\t\t */\n+\t\tctx.m = lookup_multi_pack_index(the_repository, object_dir);\n+\t}\n+\n+\tif (ctx.m && !midx_checksum_valid(ctx.m)) {\n+\t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n+\t\tctx.m = NULL;\n+\t}\n+\n+\tctx.nr = 0;\n+\tctx.alloc = ctx.m ? ctx.m->num_packs : 16;\n+\tctx.info = NULL;\n+\tALLOC_ARRAY(ctx.info, ctx.alloc);\n+\n+\tif (ctx.m) {\n+\t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n+\t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n+\n+\t\t\tif (flags & MIDX_WRITE_REV_INDEX) {\n+\t\t\t\t/*\n+\t\t\t\t * If generating a reverse index, need to have\n+\t\t\t\t * packed_git's loaded to compare their\n+\t\t\t\t * mtimes and object count.\n+\t\t\t\t */\n+\t\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n+\t\t\t\t\terror(_(\"could not load pack\"));\n+\t\t\t\t\tresult = 1;\n+\t\t\t\t\tgoto cleanup;\n+\t\t\t\t}\n+\n+\t\t\t\tif (open_pack_index(ctx.m->packs[i]))\n+\t\t\t\t\tdie(_(\"could not open index for %s\"),\n+\t\t\t\t\t    ctx.m->packs[i]->pack_name);\n+\t\t\t}\n+\n+\t\t\tfill_pack_info(&ctx.info[ctx.nr++], ctx.m->packs[i],\n+\t\t\t\t       ctx.m->pack_names[i], i);\n+\t\t}\n+\t}\n+\n+\tctx.pack_paths_checked = 0;\n+\tif (flags & MIDX_PROGRESS)\n+\t\tctx.progress = start_delayed_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n+\telse\n+\t\tctx.progress = NULL;\n+\n+\tctx.to_include = packs_to_include;\n+\n+\tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n+\tstop_progress(&ctx.progress);\n+\n+\tif ((ctx.m && ctx.nr == ctx.m->num_packs) &&\n+\t    !(packs_to_include || packs_to_drop)) {\n+\t\tstruct bitmap_index *bitmap_git;\n+\t\tint bitmap_exists;\n+\t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n+\n+\t\tbitmap_git = prepare_midx_bitmap_git(ctx.m);\n+\t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n+\t\tfree_bitmap_index(bitmap_git);\n+\n+\t\tif (bitmap_exists || !want_bitmap) {\n+\t\t\t/*\n+\t\t\t * The correct MIDX already exists, and so does a\n+\t\t\t * corresponding bitmap (or one wasn't requested).\n+\t\t\t */\n+\t\t\tif (!want_bitmap)\n+\t\t\t\tclear_midx_files_ext(object_dir, \".bitmap\",\n+\t\t\t\t\t\t     NULL);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n+\tif (preferred_pack_name) {\n+\t\tctx.preferred_pack_idx = -1;\n+\n+\t\tfor (i = 0; i < ctx.nr; i++) {\n+\t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n+\t\t\t\t\t\t  ctx.info[i].pack_name)) {\n+\t\t\t\tctx.preferred_pack_idx = i;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (ctx.preferred_pack_idx == -1)\n+\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n+\t\t\t\tpreferred_pack_name);\n+\t} else if (ctx.nr &&\n+\t\t   (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))) {\n+\t\tstruct packed_git *oldest = ctx.info[ctx.preferred_pack_idx].p;\n+\t\tctx.preferred_pack_idx = 0;\n+\n+\t\tif (packs_to_drop && packs_to_drop->nr)\n+\t\t\tBUG(\"cannot write a MIDX bitmap during expiration\");\n+\n+\t\t/*\n+\t\t * set a preferred pack when writing a bitmap to ensure that\n+\t\t * the pack from which the first object is selected in pseudo\n+\t\t * pack-order has all of its objects selected from that pack\n+\t\t * (and not another pack containing a duplicate)\n+\t\t */\n+\t\tfor (i = 1; i < ctx.nr; i++) {\n+\t\t\tstruct packed_git *p = ctx.info[i].p;\n+\n+\t\t\tif (!oldest->num_objects || p->mtime < oldest->mtime) {\n+\t\t\t\toldest = p;\n+\t\t\t\tctx.preferred_pack_idx = i;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!oldest->num_objects) {\n+\t\t\t/*\n+\t\t\t * If all packs are empty; unset the preferred index.\n+\t\t\t * This is acceptable since there will be no duplicate\n+\t\t\t * objects to resolve, so the preferred value doesn't\n+\t\t\t * matter.\n+\t\t\t */\n+\t\t\tctx.preferred_pack_idx = -1;\n+\t\t}\n+\t} else {\n+\t\t/*\n+\t\t * otherwise don't mark any pack as preferred to avoid\n+\t\t * interfering with expiration logic below\n+\t\t */\n+\t\tctx.preferred_pack_idx = -1;\n+\t}\n+\n+\tif (ctx.preferred_pack_idx > -1) {\n+\t\tstruct packed_git *preferred = ctx.info[ctx.preferred_pack_idx].p;\n+\t\tif (!preferred->num_objects) {\n+\t\t\terror(_(\"cannot select preferred pack %s with no objects\"),\n+\t\t\t      preferred->pack_name);\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n+\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,\n+\t\t\t\t\t ctx.preferred_pack_idx);\n+\n+\tctx.large_offsets_needed = 0;\n+\tfor (i = 0; i < ctx.entries_nr; i++) {\n+\t\tif (ctx.entries[i].offset > 0x7fffffff)\n+\t\t\tctx.num_large_offsets++;\n+\t\tif (ctx.entries[i].offset > 0xffffffff)\n+\t\t\tctx.large_offsets_needed = 1;\n+\t}\n+\n+\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n+\n+\tif (packs_to_drop && packs_to_drop->nr) {\n+\t\tint drop_index = 0;\n+\t\tint missing_drops = 0;\n+\n+\t\tfor (i = 0; i < ctx.nr && drop_index < packs_to_drop->nr; i++) {\n+\t\t\tint cmp = strcmp(ctx.info[i].pack_name,\n+\t\t\t\t\t packs_to_drop->items[drop_index].string);\n+\n+\t\t\tif (!cmp) {\n+\t\t\t\tdrop_index++;\n+\t\t\t\tctx.info[i].expired = 1;\n+\t\t\t} else if (cmp > 0) {\n+\t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n+\t\t\t\t      packs_to_drop->items[drop_index].string);\n+\t\t\t\tdrop_index++;\n+\t\t\t\tmissing_drops++;\n+\t\t\t\ti--;\n+\t\t\t} else {\n+\t\t\t\tctx.info[i].expired = 0;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (missing_drops) {\n+\t\t\tresult = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * pack_perm stores a permutation between pack-int-ids from the\n+\t * previous multi-pack-index to the new one we are writing:\n+\t *\n+\t * pack_perm[old_id] = new_id\n+\t */\n+\tALLOC_ARRAY(ctx.pack_perm, ctx.nr);\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (ctx.info[i].expired) {\n+\t\t\tdropped_packs++;\n+\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n+\t\t} else {\n+\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (ctx.info[i].expired)\n+\t\t\tcontinue;\n+\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n+\t\tbitmapped_packs_concat_len += 2 * sizeof(uint32_t);\n+\t}\n+\n+\t/* Check that the preferred pack wasn't expired (if given). */\n+\tif (preferred_pack_name) {\n+\t\tstruct pack_info *preferred = bsearch(preferred_pack_name,\n+\t\t\t\t\t\t      ctx.info, ctx.nr,\n+\t\t\t\t\t\t      sizeof(*ctx.info),\n+\t\t\t\t\t\t      idx_or_pack_name_cmp);\n+\t\tif (preferred) {\n+\t\t\tuint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];\n+\t\t\tif (perm == PACK_EXPIRED)\n+\t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n+\t\t\t\t\tpreferred_pack_name);\n+\t\t}\n+\t}\n+\n+\tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n+\t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n+\t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n+\n+\thold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR);\n+\tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n+\n+\tif (ctx.nr - dropped_packs == 0) {\n+\t\terror(_(\"no pack files to index.\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (!ctx.entries_nr) {\n+\t\tif (flags & MIDX_WRITE_BITMAP)\n+\t\t\twarning(_(\"refusing to write multi-pack .bitmap without any objects\"));\n+\t\tflags &= ~(MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP);\n+\t}\n+\n+\tcf = init_chunkfile(f);\n+\n+\tadd_chunk(cf, MIDX_CHUNKID_PACKNAMES, pack_name_concat_len,\n+\t\t  write_midx_pack_names);\n+\tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT, MIDX_CHUNK_FANOUT_SIZE,\n+\t\t  write_midx_oid_fanout);\n+\tadd_chunk(cf, MIDX_CHUNKID_OIDLOOKUP,\n+\t\t  st_mult(ctx.entries_nr, the_hash_algo->rawsz),\n+\t\t  write_midx_oid_lookup);\n+\tadd_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS,\n+\t\t  st_mult(ctx.entries_nr, MIDX_CHUNK_OFFSET_WIDTH),\n+\t\t  write_midx_object_offsets);\n+\n+\tif (ctx.large_offsets_needed)\n+\t\tadd_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS,\n+\t\t\tst_mult(ctx.num_large_offsets,\n+\t\t\t\tMIDX_CHUNK_LARGE_OFFSET_WIDTH),\n+\t\t\twrite_midx_large_offsets);\n+\n+\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) {\n+\t\tctx.pack_order = midx_pack_order(&ctx);\n+\t\tadd_chunk(cf, MIDX_CHUNKID_REVINDEX,\n+\t\t\t  st_mult(ctx.entries_nr, sizeof(uint32_t)),\n+\t\t\t  write_midx_revindex);\n+\t\tadd_chunk(cf, MIDX_CHUNKID_BITMAPPEDPACKS,\n+\t\t\t  bitmapped_packs_concat_len,\n+\t\t\t  write_midx_bitmapped_packs);\n+\t}\n+\n+\twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n+\twrite_chunkfile(cf, &ctx);\n+\n+\tfinalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,\n+\t\t\t  CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n+\tfree_chunkfile(cf);\n+\n+\tif (flags & MIDX_WRITE_REV_INDEX &&\n+\t    git_env_bool(\"GIT_TEST_MIDX_WRITE_REV\", 0))\n+\t\twrite_midx_reverse_index(midx_name.buf, midx_hash, &ctx);\n+\n+\tif (flags & MIDX_WRITE_BITMAP) {\n+\t\tstruct packing_data pdata;\n+\t\tstruct commit **commits;\n+\t\tuint32_t commits_nr;\n+\n+\t\tif (!ctx.entries_nr)\n+\t\t\tBUG(\"cannot write a bitmap without any objects\");\n+\n+\t\tprepare_midx_packing_data(&pdata, &ctx);\n+\n+\t\tcommits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, &ctx);\n+\n+\t\t/*\n+\t\t * The previous steps translated the information from\n+\t\t * 'entries' into information suitable for constructing\n+\t\t * bitmaps. We no longer need that array, so clear it to\n+\t\t * reduce memory pressure.\n+\t\t */\n+\t\tFREE_AND_NULL(ctx.entries);\n+\t\tctx.entries_nr = 0;\n+\n+\t\tif (write_midx_bitmap(midx_name.buf, midx_hash, &pdata,\n+\t\t\t\t      commits, commits_nr, ctx.pack_order,\n+\t\t\t\t      flags) < 0) {\n+\t\t\terror(_(\"could not write multi-pack bitmap\"));\n+\t\t\tresult = 1;\n+\t\t\tclear_packing_data(&pdata);\n+\t\t\tfree(commits);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\n+\t\tclear_packing_data(&pdata);\n+\t\tfree(commits);\n+\t}\n+\t/*\n+\t * NOTE: Do not use ctx.entries beyond this point, since it might\n+\t * have been freed in the previous if block.\n+\t */\n+\n+\tif (ctx.m)\n+\t\tclose_object_store(the_repository->objects);\n+\n+\tif (commit_lock_file(&lk) < 0)\n+\t\tdie_errno(_(\"could not write multi-pack-index\"));\n+\n+\tclear_midx_files_ext(object_dir, \".bitmap\", midx_hash);\n+\tclear_midx_files_ext(object_dir, \".rev\", midx_hash);\n+\n+cleanup:\n+\tfor (i = 0; i < ctx.nr; i++) {\n+\t\tif (ctx.info[i].p) {\n+\t\t\tclose_pack(ctx.info[i].p);\n+\t\t\tfree(ctx.info[i].p);\n+\t\t}\n+\t\tfree(ctx.info[i].pack_name);\n+\t}\n+\n+\tfree(ctx.info);\n+\tfree(ctx.entries);\n+\tfree(ctx.pack_perm);\n+\tfree(ctx.pack_order);\n+\tstrbuf_release(&midx_name);\n+\n+\ttrace2_region_leave(\"midx\", \"write_midx_internal\", the_repository);\n+\n+\treturn result;\n+}\n+\n+int write_midx_file(const char *object_dir,\n+\t\t    const char *preferred_pack_name,\n+\t\t    const char *refs_snapshot,\n+\t\t    unsigned flags)\n+{\n+\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n+\t\t\t\t   refs_snapshot, flags);\n+}\n+\n+int write_midx_file_only(const char *object_dir,\n+\t\t\t struct string_list *packs_to_include,\n+\t\t\t const char *preferred_pack_name,\n+\t\t\t const char *refs_snapshot,\n+\t\t\t unsigned flags)\n+{\n+\treturn write_midx_internal(object_dir, packs_to_include, NULL,\n+\t\t\t\t   preferred_pack_name, refs_snapshot, flags);\n+}\n+\n+int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags)\n+{\n+\tuint32_t i, *count, result = 0;\n+\tstruct string_list packs_to_drop = STRING_LIST_INIT_DUP;\n+\tstruct multi_pack_index *m = lookup_multi_pack_index(r, object_dir);\n+\tstruct progress *progress = NULL;\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tCALLOC_ARRAY(count, m->num_packs);\n+\n+\tif (flags & MIDX_PROGRESS)\n+\t\tprogress = start_delayed_progress(_(\"Counting referenced objects\"),\n+\t\t\t\t\t  m->num_objects);\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tint pack_int_id = nth_midxed_pack_int_id(m, i);\n+\t\tcount[pack_int_id]++;\n+\t\tdisplay_progress(progress, i + 1);\n+\t}\n+\tstop_progress(&progress);\n+\n+\tif (flags & MIDX_PROGRESS)\n+\t\tprogress = start_delayed_progress(_(\"Finding and deleting unreferenced packfiles\"),\n+\t\t\t\t\t  m->num_packs);\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tchar *pack_name;\n+\t\tdisplay_progress(progress, i + 1);\n+\n+\t\tif (count[i])\n+\t\t\tcontinue;\n+\n+\t\tif (prepare_midx_pack(r, m, i))\n+\t\t\tcontinue;\n+\n+\t\tif (m->packs[i]->pack_keep || m->packs[i]->is_cruft)\n+\t\t\tcontinue;\n+\n+\t\tpack_name = xstrdup(m->packs[i]->pack_name);\n+\t\tclose_pack(m->packs[i]);\n+\n+\t\tstring_list_insert(&packs_to_drop, m->pack_names[i]);\n+\t\tunlink_pack_path(pack_name, 0);\n+\t\tfree(pack_name);\n+\t}\n+\tstop_progress(&progress);\n+\n+\tfree(count);\n+\n+\tif (packs_to_drop.nr)\n+\t\tresult = write_midx_internal(object_dir, NULL, &packs_to_drop, NULL, NULL, flags);\n+\n+\tstring_list_clear(&packs_to_drop, 0);\n+\n+\treturn result;\n+}\n+\n+struct repack_info {\n+\ttimestamp_t mtime;\n+\tuint32_t referenced_objects;\n+\tuint32_t pack_int_id;\n+};\n+\n+static int compare_by_mtime(const void *a_, const void *b_)\n+{\n+\tconst struct repack_info *a, *b;\n+\n+\ta = (const struct repack_info *)a_;\n+\tb = (const struct repack_info *)b_;\n+\n+\tif (a->mtime < b->mtime)\n+\t\treturn -1;\n+\tif (a->mtime > b->mtime)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static int fill_included_packs_all(struct repository *r,\n+\t\t\t\t   struct multi_pack_index *m,\n+\t\t\t\t   unsigned char *include_pack)\n+{\n+\tuint32_t i, count = 0;\n+\tint pack_kept_objects = 0;\n+\n+\trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tif (prepare_midx_pack(r, m, i))\n+\t\t\tcontinue;\n+\t\tif (!pack_kept_objects && m->packs[i]->pack_keep)\n+\t\t\tcontinue;\n+\t\tif (m->packs[i]->is_cruft)\n+\t\t\tcontinue;\n+\n+\t\tinclude_pack[i] = 1;\n+\t\tcount++;\n+\t}\n+\n+\treturn count < 2;\n+}\n+\n+static int fill_included_packs_batch(struct repository *r,\n+\t\t\t\t     struct multi_pack_index *m,\n+\t\t\t\t     unsigned char *include_pack,\n+\t\t\t\t     size_t batch_size)\n+{\n+\tuint32_t i, packs_to_repack;\n+\tsize_t total_size;\n+\tstruct repack_info *pack_info;\n+\tint pack_kept_objects = 0;\n+\n+\tCALLOC_ARRAY(pack_info, m->num_packs);\n+\n+\trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tpack_info[i].pack_int_id = i;\n+\n+\t\tif (prepare_midx_pack(r, m, i))\n+\t\t\tcontinue;\n+\n+\t\tpack_info[i].mtime = m->packs[i]->mtime;\n+\t}\n+\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n+\t\tpack_info[pack_int_id].referenced_objects++;\n+\t}\n+\n+\tQSORT(pack_info, m->num_packs, compare_by_mtime);\n+\n+\ttotal_size = 0;\n+\tpacks_to_repack = 0;\n+\tfor (i = 0; total_size < batch_size && i < m->num_packs; i++) {\n+\t\tint pack_int_id = pack_info[i].pack_int_id;\n+\t\tstruct packed_git *p = m->packs[pack_int_id];\n+\t\tsize_t expected_size;\n+\n+\t\tif (!p)\n+\t\t\tcontinue;\n+\t\tif (!pack_kept_objects && p->pack_keep)\n+\t\t\tcontinue;\n+\t\tif (p->is_cruft)\n+\t\t\tcontinue;\n+\t\tif (open_pack_index(p) || !p->num_objects)\n+\t\t\tcontinue;\n+\n+\t\texpected_size = st_mult(p->pack_size,\n+\t\t\t\t\tpack_info[i].referenced_objects);\n+\t\texpected_size /= p->num_objects;\n+\n+\t\tif (expected_size >= batch_size)\n+\t\t\tcontinue;\n+\n+\t\tpacks_to_repack++;\n+\t\ttotal_size += expected_size;\n+\t\tinclude_pack[pack_int_id] = 1;\n+\t}\n+\n+\tfree(pack_info);\n+\n+\tif (packs_to_repack < 2)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+int midx_repack(struct repository *r, const char *object_dir, size_t batch_size, unsigned flags)\n+{\n+\tint result = 0;\n+\tuint32_t i;\n+\tunsigned char *include_pack;\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tFILE *cmd_in;\n+\tstruct multi_pack_index *m = lookup_multi_pack_index(r, object_dir);\n+\n+\t/*\n+\t * When updating the default for these configuration\n+\t * variables in builtin/repack.c, these must be adjusted\n+\t * to match.\n+\t */\n+\tint delta_base_offset = 1;\n+\tint use_delta_islands = 0;\n+\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tCALLOC_ARRAY(include_pack, m->num_packs);\n+\n+\tif (batch_size) {\n+\t\tif (fill_included_packs_batch(r, m, include_pack, batch_size))\n+\t\t\tgoto cleanup;\n+\t} else if (fill_included_packs_all(r, m, include_pack))\n+\t\tgoto cleanup;\n+\n+\trepo_config_get_bool(r, \"repack.usedeltabaseoffset\", &delta_base_offset);\n+\trepo_config_get_bool(r, \"repack.usedeltaislands\", &use_delta_islands);\n+\n+\tstrvec_push(&cmd.args, \"pack-objects\");\n+\n+\tstrvec_pushf(&cmd.args, \"%s/pack/pack\", object_dir);\n+\n+\tif (delta_base_offset)\n+\t\tstrvec_push(&cmd.args, \"--delta-base-offset\");\n+\tif (use_delta_islands)\n+\t\tstrvec_push(&cmd.args, \"--delta-islands\");\n+\n+\tif (flags & MIDX_PROGRESS)\n+\t\tstrvec_push(&cmd.args, \"--progress\");\n+\telse\n+\t\tstrvec_push(&cmd.args, \"-q\");\n+\n+\tcmd.git_cmd = 1;\n+\tcmd.in = cmd.out = -1;\n+\n+\tif (start_command(&cmd)) {\n+\t\terror(_(\"could not start pack-objects\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tcmd_in = xfdopen(cmd.in, \"w\");\n+\n+\tfor (i = 0; i < m->num_objects; i++) {\n+\t\tstruct object_id oid;\n+\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n+\n+\t\tif (!include_pack[pack_int_id])\n+\t\t\tcontinue;\n+\n+\t\tnth_midxed_object_oid(&oid, m, i);\n+\t\tfprintf(cmd_in, \"%s\\n\", oid_to_hex(&oid));\n+\t}\n+\tfclose(cmd_in);\n+\n+\tif (finish_command(&cmd)) {\n+\t\terror(_(\"could not finish pack-objects\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tresult = write_midx_internal(object_dir, NULL, NULL, NULL, NULL, flags);\n+\n+cleanup:\n+\tfree(include_pack);\n+\treturn result;\n+}\ndiff --git a/midx.c b/midx.c\nindex 41521e019c..ae3b49166c 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1,52 +1,22 @@\n #include \"git-compat-util.h\"\n-#include \"abspath.h\"\n #include \"config.h\"\n-#include \"csum-file.h\"\n #include \"dir.h\"\n-#include \"gettext.h\"\n #include \"hex.h\"\n-#include \"lockfile.h\"\n #include \"packfile.h\"\n #include \"object-file.h\"\n-#include \"object-store-ll.h\"\n #include \"hash-lookup.h\"\n #include \"midx.h\"\n #include \"progress.h\"\n #include \"trace2.h\"\n-#include \"run-command.h\"\n-#include \"repository.h\"\n #include \"chunk-format.h\"\n-#include \"pack.h\"\n #include \"pack-bitmap.h\"\n-#include \"refs.h\"\n-#include \"revision.h\"\n-#include \"list-objects.h\"\n #include \"pack-revindex.h\"\n \n-#define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n-#define MIDX_VERSION 1\n-#define MIDX_BYTE_FILE_VERSION 4\n-#define MIDX_BYTE_HASH_VERSION 5\n-#define MIDX_BYTE_NUM_CHUNKS 6\n-#define MIDX_BYTE_NUM_PACKS 8\n-#define MIDX_HEADER_SIZE 12\n-#define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + the_hash_algo->rawsz)\n-\n-#define MIDX_CHUNK_ALIGNMENT 4\n-#define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n-#define MIDX_CHUNKID_BITMAPPEDPACKS 0x42544d50 /* \"BTMP\" */\n-#define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n-#define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n-#define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n-#define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n-#define MIDX_CHUNKID_REVINDEX 0x52494458 /* \"RIDX\" */\n-#define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)\n-#define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n-#define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))\n-#define MIDX_CHUNK_BITMAPPED_PACKS_WIDTH (2 * sizeof(uint32_t))\n-#define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n-\n-#define PACK_EXPIRED UINT_MAX\n+int midx_checksum_valid(struct multi_pack_index *m);\n+void clear_midx_files_ext(const char *object_dir, const char *ext,\n+\t\t\t  unsigned char *keep_hash);\n+int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n+\t\t\t const char *idx_name);\n \n const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n {\n@@ -115,6 +85,8 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+#define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + the_hash_algo->rawsz)\n+\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local)\n {\n \tstruct multi_pack_index *m = NULL;\n@@ -294,6 +266,8 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t\n \treturn 0;\n }\n \n+#define MIDX_CHUNK_BITMAPPED_PACKS_WIDTH (2 * sizeof(uint32_t))\n+\n int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,\n \t\t       struct bitmapped_pack *bp, uint32_t pack_int_id)\n {\n@@ -400,8 +374,8 @@ int fill_midx_entry(struct repository *r,\n }\n \n /* Match \"foo.idx\" against either \"foo.pack\" _or_ \"foo.idx\". */\n-static int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n-\t\t\t\tconst char *idx_name)\n+int cmp_idx_or_pack_name(const char *idx_or_pack_name,\n+\t\t\t const char *idx_name)\n {\n \t/* Skip past any initial matching prefix. */\n \twhile (*idx_name && *idx_name == *idx_or_pack_name) {\n@@ -508,1262 +482,11 @@ int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, i\n \treturn 0;\n }\n \n-static size_t write_midx_header(struct hashfile *f,\n-\t\t\t\tunsigned char num_chunks,\n-\t\t\t\tuint32_t num_packs)\n-{\n-\thashwrite_be32(f, MIDX_SIGNATURE);\n-\thashwrite_u8(f, MIDX_VERSION);\n-\thashwrite_u8(f, oid_version(the_hash_algo));\n-\thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0); /* unused */\n-\thashwrite_be32(f, num_packs);\n-\n-\treturn MIDX_HEADER_SIZE;\n-}\n-\n-#define BITMAP_POS_UNKNOWN (~((uint32_t)0))\n-\n-struct pack_info {\n-\tuint32_t orig_pack_int_id;\n-\tchar *pack_name;\n-\tstruct packed_git *p;\n-\n-\tuint32_t bitmap_pos;\n-\tuint32_t bitmap_nr;\n-\n-\tunsigned expired : 1;\n-};\n-\n-static void fill_pack_info(struct pack_info *info,\n-\t\t\t   struct packed_git *p, const char *pack_name,\n-\t\t\t   uint32_t orig_pack_int_id)\n-{\n-\tmemset(info, 0, sizeof(struct pack_info));\n-\n-\tinfo->orig_pack_int_id = orig_pack_int_id;\n-\tinfo->pack_name = xstrdup(pack_name);\n-\tinfo->p = p;\n-\tinfo->bitmap_pos = BITMAP_POS_UNKNOWN;\n-}\n-\n-static int pack_info_compare(const void *_a, const void *_b)\n-{\n-\tstruct pack_info *a = (struct pack_info *)_a;\n-\tstruct pack_info *b = (struct pack_info *)_b;\n-\treturn strcmp(a->pack_name, b->pack_name);\n-}\n-\n-static int idx_or_pack_name_cmp(const void *_va, const void *_vb)\n-{\n-\tconst char *pack_name = _va;\n-\tconst struct pack_info *compar = _vb;\n-\n-\treturn cmp_idx_or_pack_name(pack_name, compar->pack_name);\n-}\n-\n-struct write_midx_context {\n-\tstruct pack_info *info;\n-\tsize_t nr;\n-\tsize_t alloc;\n-\tstruct multi_pack_index *m;\n-\tstruct progress *progress;\n-\tunsigned pack_paths_checked;\n-\n-\tstruct pack_midx_entry *entries;\n-\tsize_t entries_nr;\n-\n-\tuint32_t *pack_perm;\n-\tuint32_t *pack_order;\n-\tunsigned large_offsets_needed:1;\n-\tuint32_t num_large_offsets;\n-\n-\tint preferred_pack_idx;\n-\n-\tstruct string_list *to_include;\n-};\n-\n-static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n-\t\t\t     const char *file_name, void *data)\n-{\n-\tstruct write_midx_context *ctx = data;\n-\tstruct packed_git *p;\n-\n-\tif (ends_with(file_name, \".idx\")) {\n-\t\tdisplay_progress(ctx->progress, ++ctx->pack_paths_checked);\n-\t\t/*\n-\t\t * Note that at most one of ctx->m and ctx->to_include are set,\n-\t\t * so we are testing midx_contains_pack() and\n-\t\t * string_list_has_string() independently (guarded by the\n-\t\t * appropriate NULL checks).\n-\t\t *\n-\t\t * We could support passing to_include while reusing an existing\n-\t\t * MIDX, but don't currently since the reuse process drags\n-\t\t * forward all packs from an existing MIDX (without checking\n-\t\t * whether or not they appear in the to_include list).\n-\t\t *\n-\t\t * If we added support for that, these next two conditional\n-\t\t * should be performed independently (likely checking\n-\t\t * to_include before the existing MIDX).\n-\t\t */\n-\t\tif (ctx->m && midx_contains_pack(ctx->m, file_name))\n-\t\t\treturn;\n-\t\telse if (ctx->to_include &&\n-\t\t\t !string_list_has_string(ctx->to_include, file_name))\n-\t\t\treturn;\n-\n-\t\tALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);\n-\n-\t\tp = add_packed_git(full_path, full_path_len, 0);\n-\t\tif (!p) {\n-\t\t\twarning(_(\"failed to add packfile '%s'\"),\n-\t\t\t\tfull_path);\n-\t\t\treturn;\n-\t\t}\n-\n-\t\tif (open_pack_index(p)) {\n-\t\t\twarning(_(\"failed to open pack-index '%s'\"),\n-\t\t\t\tfull_path);\n-\t\t\tclose_pack(p);\n-\t\t\tfree(p);\n-\t\t\treturn;\n-\t\t}\n-\n-\t\tfill_pack_info(&ctx->info[ctx->nr], p, file_name, ctx->nr);\n-\t\tctx->nr++;\n-\t}\n-}\n-\n-struct pack_midx_entry {\n-\tstruct object_id oid;\n-\tuint32_t pack_int_id;\n-\ttime_t pack_mtime;\n-\tuint64_t offset;\n-\tunsigned preferred : 1;\n-};\n-\n-static int midx_oid_compare(const void *_a, const void *_b)\n-{\n-\tconst struct pack_midx_entry *a = (const struct pack_midx_entry *)_a;\n-\tconst struct pack_midx_entry *b = (const struct pack_midx_entry *)_b;\n-\tint cmp = oidcmp(&a->oid, &b->oid);\n-\n-\tif (cmp)\n-\t\treturn cmp;\n-\n-\t/* Sort objects in a preferred pack first when multiple copies exist. */\n-\tif (a->preferred > b->preferred)\n-\t\treturn -1;\n-\tif (a->preferred < b->preferred)\n-\t\treturn 1;\n-\n-\tif (a->pack_mtime > b->pack_mtime)\n-\t\treturn -1;\n-\telse if (a->pack_mtime < b->pack_mtime)\n-\t\treturn 1;\n-\n-\treturn a->pack_int_id - b->pack_int_id;\n-}\n-\n-static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n-\t\t\t\t      struct pack_midx_entry *e,\n-\t\t\t\t      uint32_t pos)\n-{\n-\tif (pos >= m->num_objects)\n-\t\treturn 1;\n-\n-\tnth_midxed_object_oid(&e->oid, m, pos);\n-\te->pack_int_id = nth_midxed_pack_int_id(m, pos);\n-\te->offset = nth_midxed_offset(m, pos);\n-\n-\t/* consider objects in midx to be from \"old\" packs */\n-\te->pack_mtime = 0;\n-\treturn 0;\n-}\n-\n-static void fill_pack_entry(uint32_t pack_int_id,\n-\t\t\t    struct packed_git *p,\n-\t\t\t    uint32_t cur_object,\n-\t\t\t    struct pack_midx_entry *entry,\n-\t\t\t    int preferred)\n-{\n-\tif (nth_packed_object_id(&entry->oid, p, cur_object) < 0)\n-\t\tdie(_(\"failed to locate object %d in packfile\"), cur_object);\n-\n-\tentry->pack_int_id = pack_int_id;\n-\tentry->pack_mtime = p->mtime;\n-\n-\tentry->offset = nth_packed_object_offset(p, cur_object);\n-\tentry->preferred = !!preferred;\n-}\n-\n-struct midx_fanout {\n-\tstruct pack_midx_entry *entries;\n-\tsize_t nr, alloc;\n-};\n-\n-static void midx_fanout_grow(struct midx_fanout *fanout, size_t nr)\n-{\n-\tif (nr < fanout->nr)\n-\t\tBUG(\"negative growth in midx_fanout_grow() (%\"PRIuMAX\" < %\"PRIuMAX\")\",\n-\t\t    (uintmax_t)nr, (uintmax_t)fanout->nr);\n-\tALLOC_GROW(fanout->entries, nr, fanout->alloc);\n-}\n-\n-static void midx_fanout_sort(struct midx_fanout *fanout)\n-{\n-\tQSORT(fanout->entries, fanout->nr, midx_oid_compare);\n-}\n-\n-static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,\n-\t\t\t\t\tstruct multi_pack_index *m,\n-\t\t\t\t\tuint32_t cur_fanout,\n-\t\t\t\t\tint preferred_pack)\n-{\n-\tuint32_t start = 0, end;\n-\tuint32_t cur_object;\n-\n-\tif (cur_fanout)\n-\t\tstart = ntohl(m->chunk_oid_fanout[cur_fanout - 1]);\n-\tend = ntohl(m->chunk_oid_fanout[cur_fanout]);\n-\n-\tfor (cur_object = start; cur_object < end; cur_object++) {\n-\t\tif ((preferred_pack > -1) &&\n-\t\t    (preferred_pack == nth_midxed_pack_int_id(m, cur_object))) {\n-\t\t\t/*\n-\t\t\t * Objects from preferred packs are added\n-\t\t\t * separately.\n-\t\t\t */\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tmidx_fanout_grow(fanout, fanout->nr + 1);\n-\t\tnth_midxed_pack_midx_entry(m,\n-\t\t\t\t\t   &fanout->entries[fanout->nr],\n-\t\t\t\t\t   cur_object);\n-\t\tfanout->entries[fanout->nr].preferred = 0;\n-\t\tfanout->nr++;\n-\t}\n-}\n-\n-static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout,\n-\t\t\t\t\tstruct pack_info *info,\n-\t\t\t\t\tuint32_t cur_pack,\n-\t\t\t\t\tint preferred,\n-\t\t\t\t\tuint32_t cur_fanout)\n-{\n-\tstruct packed_git *pack = info[cur_pack].p;\n-\tuint32_t start = 0, end;\n-\tuint32_t cur_object;\n-\n-\tif (cur_fanout)\n-\t\tstart = get_pack_fanout(pack, cur_fanout - 1);\n-\tend = get_pack_fanout(pack, cur_fanout);\n-\n-\tfor (cur_object = start; cur_object < end; cur_object++) {\n-\t\tmidx_fanout_grow(fanout, fanout->nr + 1);\n-\t\tfill_pack_entry(cur_pack,\n-\t\t\t\tinfo[cur_pack].p,\n-\t\t\t\tcur_object,\n-\t\t\t\t&fanout->entries[fanout->nr],\n-\t\t\t\tpreferred);\n-\t\tfanout->nr++;\n-\t}\n-}\n-\n-/*\n- * It is possible to artificially get into a state where there are many\n- * duplicate copies of objects. That can create high memory pressure if\n- * we are to create a list of all objects before de-duplication. To reduce\n- * this memory pressure without a significant performance drop, automatically\n- * group objects by the first byte of their object id. Use the IDX fanout\n- * tables to group the data, copy to a local array, then sort.\n- *\n- * Copy only the de-duplicated entries (selected by most-recent modified time\n- * of a packfile containing the object).\n- */\n-static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n-\t\t\t\t\t\t  struct pack_info *info,\n-\t\t\t\t\t\t  uint32_t nr_packs,\n-\t\t\t\t\t\t  size_t *nr_objects,\n-\t\t\t\t\t\t  int preferred_pack)\n-{\n-\tuint32_t cur_fanout, cur_pack, cur_object;\n-\tsize_t alloc_objects, total_objects = 0;\n-\tstruct midx_fanout fanout = { 0 };\n-\tstruct pack_midx_entry *deduplicated_entries = NULL;\n-\tuint32_t start_pack = m ? m->num_packs : 0;\n-\n-\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++)\n-\t\ttotal_objects = st_add(total_objects,\n-\t\t\t\t       info[cur_pack].p->num_objects);\n-\n-\t/*\n-\t * As we de-duplicate by fanout value, we expect the fanout\n-\t * slices to be evenly distributed, with some noise. Hence,\n-\t * allocate slightly more than one 256th.\n-\t */\n-\talloc_objects = fanout.alloc = total_objects > 3200 ? total_objects / 200 : 16;\n-\n-\tALLOC_ARRAY(fanout.entries, fanout.alloc);\n-\tALLOC_ARRAY(deduplicated_entries, alloc_objects);\n-\t*nr_objects = 0;\n-\n-\tfor (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {\n-\t\tfanout.nr = 0;\n-\n-\t\tif (m)\n-\t\t\tmidx_fanout_add_midx_fanout(&fanout, m, cur_fanout,\n-\t\t\t\t\t\t    preferred_pack);\n-\n-\t\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) {\n-\t\t\tint preferred = cur_pack == preferred_pack;\n-\t\t\tmidx_fanout_add_pack_fanout(&fanout,\n-\t\t\t\t\t\t    info, cur_pack,\n-\t\t\t\t\t\t    preferred, cur_fanout);\n-\t\t}\n-\n-\t\tif (-1 < preferred_pack && preferred_pack < start_pack)\n-\t\t\tmidx_fanout_add_pack_fanout(&fanout, info,\n-\t\t\t\t\t\t    preferred_pack, 1,\n-\t\t\t\t\t\t    cur_fanout);\n-\n-\t\tmidx_fanout_sort(&fanout);\n-\n-\t\t/*\n-\t\t * The batch is now sorted by OID and then mtime (descending).\n-\t\t * Take only the first duplicate.\n-\t\t */\n-\t\tfor (cur_object = 0; cur_object < fanout.nr; cur_object++) {\n-\t\t\tif (cur_object && oideq(&fanout.entries[cur_object - 1].oid,\n-\t\t\t\t\t\t&fanout.entries[cur_object].oid))\n-\t\t\t\tcontinue;\n-\n-\t\t\tALLOC_GROW(deduplicated_entries, st_add(*nr_objects, 1),\n-\t\t\t\t   alloc_objects);\n-\t\t\tmemcpy(&deduplicated_entries[*nr_objects],\n-\t\t\t       &fanout.entries[cur_object],\n-\t\t\t       sizeof(struct pack_midx_entry));\n-\t\t\t(*nr_objects)++;\n-\t\t}\n-\t}\n-\n-\tfree(fanout.entries);\n-\treturn deduplicated_entries;\n-}\n-\n-static int write_midx_pack_names(struct hashfile *f, void *data)\n-{\n-\tstruct write_midx_context *ctx = data;\n-\tuint32_t i;\n-\tunsigned char padding[MIDX_CHUNK_ALIGNMENT];\n-\tsize_t written = 0;\n-\n-\tfor (i = 0; i < ctx->nr; i++) {\n-\t\tsize_t writelen;\n-\n-\t\tif (ctx->info[i].expired)\n-\t\t\tcontinue;\n-\n-\t\tif (i && strcmp(ctx->info[i].pack_name, ctx->info[i - 1].pack_name) <= 0)\n-\t\t\tBUG(\"incorrect pack-file order: %s before %s\",\n-\t\t\t    ctx->info[i - 1].pack_name,\n-\t\t\t    ctx->info[i].pack_name);\n-\n-\t\twritelen = strlen(ctx->info[i].pack_name) + 1;\n-\t\thashwrite(f, ctx->info[i].pack_name, writelen);\n-\t\twritten += writelen;\n-\t}\n-\n-\t/* add padding to be aligned */\n-\ti = MIDX_CHUNK_ALIGNMENT - (written % MIDX_CHUNK_ALIGNMENT);\n-\tif (i < MIDX_CHUNK_ALIGNMENT) {\n-\t\tmemset(padding, 0, sizeof(padding));\n-\t\thashwrite(f, padding, i);\n-\t}\n-\n-\treturn 0;\n-}\n-\n-static int write_midx_bitmapped_packs(struct hashfile *f, void *data)\n-{\n-\tstruct write_midx_context *ctx = data;\n-\tsize_t i;\n-\n-\tfor (i = 0; i < ctx->nr; i++) {\n-\t\tstruct pack_info *pack = &ctx->info[i];\n-\t\tif (pack->expired)\n-\t\t\tcontinue;\n-\n-\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN && pack->bitmap_nr)\n-\t\t\tBUG(\"pack '%s' has no bitmap position, but has %d bitmapped object(s)\",\n-\t\t\t    pack->pack_name, pack->bitmap_nr);\n-\n-\t\thashwrite_be32(f, pack->bitmap_pos);\n-\t\thashwrite_be32(f, pack->bitmap_nr);\n-\t}\n-\treturn 0;\n-}\n-\n-static int write_midx_oid_fanout(struct hashfile *f,\n-\t\t\t\t void *data)\n-{\n-\tstruct write_midx_context *ctx = data;\n-\tstruct pack_midx_entry *list = ctx->entries;\n-\tstruct pack_midx_entry *last = ctx->entries + ctx->entries_nr;\n-\tuint32_t count = 0;\n-\tuint32_t i;\n-\n-\t/*\n-\t* Write the first-level table (the list is sorted,\n-\t* but we use a 256-entry lookup to be able to avoid\n-\t* having to do eight extra binary search iterations).\n-\t*/\n-\tfor (i = 0; i < 256; i++) {\n-\t\tstruct pack_midx_entry *next = list;\n-\n-\t\twhile (next < last && next->oid.hash[0] == i) {\n-\t\t\tcount++;\n-\t\t\tnext++;\n-\t\t}\n-\n-\t\thashwrite_be32(f, count);\n-\t\tlist = next;\n-\t}\n-\n-\treturn 0;\n-}\n-\n-static int write_midx_oid_lookup(struct hashfile *f,\n-\t\t\t\t void *data)\n-{\n-\tstruct write_midx_context *ctx = data;\n-\tunsigned char hash_len = the_hash_algo->rawsz;\n-\tstruct pack_midx_entry *list = ctx->entries;\n-\tuint32_t i;\n-\n-\tfor (i = 0; i < ctx->entries_nr; i++) {\n-\t\tstruct pack_midx_entry *obj = list++;\n-\n-\t\tif (i < ctx->entries_nr - 1) {\n-\t\t\tstruct pack_midx_entry *next = list;\n-\t\t\tif (oidcmp(&obj->oid, &next->oid) >= 0)\n-\t\t\t\tBUG(\"OIDs not in order: %s >= %s\",\n-\t\t\t\t    oid_to_hex(&obj->oid),\n-\t\t\t\t    oid_to_hex(&next->oid));\n-\t\t}\n-\n-\t\thashwrite(f, obj->oid.hash, (int)hash_len);\n-\t}\n-\n-\treturn 0;\n-}\n-\n-static int write_midx_object_offsets(struct hashfile *f,\n-\t\t\t\t     void *data)\n-{\n-\tstruct write_midx_context *ctx = data;\n-\tstruct pack_midx_entry *list = ctx->entries;\n-\tuint32_t i, nr_large_offset = 0;\n-\n-\tfor (i = 0; i < ctx->entries_nr; i++) {\n-\t\tstruct pack_midx_entry *obj = list++;\n-\n-\t\tif (ctx->pack_perm[obj->pack_int_id] == PACK_EXPIRED)\n-\t\t\tBUG(\"object %s is in an expired pack with int-id %d\",\n-\t\t\t    oid_to_hex(&obj->oid),\n-\t\t\t    obj->pack_int_id);\n-\n-\t\thashwrite_be32(f, ctx->pack_perm[obj->pack_int_id]);\n-\n-\t\tif (ctx->large_offsets_needed && obj->offset >> 31)\n-\t\t\thashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);\n-\t\telse if (!ctx->large_offsets_needed && obj->offset >> 32)\n-\t\t\tBUG(\"object %s requires a large offset (%\"PRIx64\") but the MIDX is not writing large offsets!\",\n-\t\t\t    oid_to_hex(&obj->oid),\n-\t\t\t    obj->offset);\n-\t\telse\n-\t\t\thashwrite_be32(f, (uint32_t)obj->offset);\n-\t}\n-\n-\treturn 0;\n-}\n-\n-static int write_midx_large_offsets(struct hashfile *f,\n-\t\t\t\t    void *data)\n-{\n-\tstruct write_midx_context *ctx = data;\n-\tstruct pack_midx_entry *list = ctx->entries;\n-\tstruct pack_midx_entry *end = ctx->entries + ctx->entries_nr;\n-\tuint32_t nr_large_offset = ctx->num_large_offsets;\n-\n-\twhile (nr_large_offset) {\n-\t\tstruct pack_midx_entry *obj;\n-\t\tuint64_t offset;\n-\n-\t\tif (list >= end)\n-\t\t\tBUG(\"too many large-offset objects\");\n-\n-\t\tobj = list++;\n-\t\toffset = obj->offset;\n-\n-\t\tif (!(offset >> 31))\n-\t\t\tcontinue;\n-\n-\t\thashwrite_be64(f, offset);\n-\n-\t\tnr_large_offset--;\n-\t}\n-\n-\treturn 0;\n-}\n-\n-static int write_midx_revindex(struct hashfile *f,\n-\t\t\t       void *data)\n-{\n-\tstruct write_midx_context *ctx = data;\n-\tuint32_t i;\n-\n-\tfor (i = 0; i < ctx->entries_nr; i++)\n-\t\thashwrite_be32(f, ctx->pack_order[i]);\n-\n-\treturn 0;\n-}\n-\n-struct midx_pack_order_data {\n-\tuint32_t nr;\n-\tuint32_t pack;\n-\toff_t offset;\n-};\n-\n-static int midx_pack_order_cmp(const void *va, const void *vb)\n-{\n-\tconst struct midx_pack_order_data *a = va, *b = vb;\n-\tif (a->pack < b->pack)\n-\t\treturn -1;\n-\telse if (a->pack > b->pack)\n-\t\treturn 1;\n-\telse if (a->offset < b->offset)\n-\t\treturn -1;\n-\telse if (a->offset > b->offset)\n-\t\treturn 1;\n-\telse\n-\t\treturn 0;\n-}\n-\n-static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n-{\n-\tstruct midx_pack_order_data *data;\n-\tuint32_t *pack_order;\n-\tuint32_t i;\n-\n-\ttrace2_region_enter(\"midx\", \"midx_pack_order\", the_repository);\n-\n-\tALLOC_ARRAY(data, ctx->entries_nr);\n-\tfor (i = 0; i < ctx->entries_nr; i++) {\n-\t\tstruct pack_midx_entry *e = &ctx->entries[i];\n-\t\tdata[i].nr = i;\n-\t\tdata[i].pack = ctx->pack_perm[e->pack_int_id];\n-\t\tif (!e->preferred)\n-\t\t\tdata[i].pack |= (1U << 31);\n-\t\tdata[i].offset = e->offset;\n-\t}\n-\n-\tQSORT(data, ctx->entries_nr, midx_pack_order_cmp);\n-\n-\tALLOC_ARRAY(pack_order, ctx->entries_nr);\n-\tfor (i = 0; i < ctx->entries_nr; i++) {\n-\t\tstruct pack_midx_entry *e = &ctx->entries[data[i].nr];\n-\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]];\n-\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n-\t\t\tpack->bitmap_pos = i;\n-\t\tpack->bitmap_nr++;\n-\t\tpack_order[i] = data[i].nr;\n-\t}\n-\tfor (i = 0; i < ctx->nr; i++) {\n-\t\tstruct pack_info *pack = &ctx->info[ctx->pack_perm[i]];\n-\t\tif (pack->bitmap_pos == BITMAP_POS_UNKNOWN)\n-\t\t\tpack->bitmap_pos = 0;\n-\t}\n-\tfree(data);\n-\n-\ttrace2_region_leave(\"midx\", \"midx_pack_order\", the_repository);\n-\n-\treturn pack_order;\n-}\n-\n-static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n-\t\t\t\t     struct write_midx_context *ctx)\n-{\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tconst char *tmp_file;\n-\n-\ttrace2_region_enter(\"midx\", \"write_midx_reverse_index\", the_repository);\n-\n-\tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex(midx_hash));\n-\n-\ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n-\t\t\t\t\tmidx_hash, WRITE_REV);\n-\n-\tif (finalize_object_file(tmp_file, buf.buf))\n-\t\tdie(_(\"cannot store reverse index file\"));\n-\n-\tstrbuf_release(&buf);\n-\n-\ttrace2_region_leave(\"midx\", \"write_midx_reverse_index\", the_repository);\n-}\n-\n-static void clear_midx_files_ext(const char *object_dir, const char *ext,\n-\t\t\t\t unsigned char *keep_hash);\n-\n-static int midx_checksum_valid(struct multi_pack_index *m)\n+int midx_checksum_valid(struct multi_pack_index *m)\n {\n \treturn hashfile_checksum_valid(m->data, m->data_len);\n }\n \n-static void prepare_midx_packing_data(struct packing_data *pdata,\n-\t\t\t\t      struct write_midx_context *ctx)\n-{\n-\tuint32_t i;\n-\n-\ttrace2_region_enter(\"midx\", \"prepare_midx_packing_data\", the_repository);\n-\n-\tmemset(pdata, 0, sizeof(struct packing_data));\n-\tprepare_packing_data(the_repository, pdata);\n-\n-\tfor (i = 0; i < ctx->entries_nr; i++) {\n-\t\tstruct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];\n-\t\tstruct object_entry *to = packlist_alloc(pdata, &from->oid);\n-\n-\t\toe_set_in_pack(pdata, to,\n-\t\t\t       ctx->info[ctx->pack_perm[from->pack_int_id]].p);\n-\t}\n-\n-\ttrace2_region_leave(\"midx\", \"prepare_midx_packing_data\", the_repository);\n-}\n-\n-static int add_ref_to_pending(const char *refname,\n-\t\t\t      const struct object_id *oid,\n-\t\t\t      int flag, void *cb_data)\n-{\n-\tstruct rev_info *revs = (struct rev_info*)cb_data;\n-\tstruct object_id peeled;\n-\tstruct object *object;\n-\n-\tif ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {\n-\t\twarning(\"symbolic ref is dangling: %s\", refname);\n-\t\treturn 0;\n-\t}\n-\n-\tif (!peel_iterated_oid(oid, &peeled))\n-\t\toid = &peeled;\n-\n-\tobject = parse_object_or_die(oid, refname);\n-\tif (object->type != OBJ_COMMIT)\n-\t\treturn 0;\n-\n-\tadd_pending_object(revs, object, \"\");\n-\tif (bitmap_is_preferred_refname(revs->repo, refname))\n-\t\tobject->flags |= NEEDS_BITMAP;\n-\treturn 0;\n-}\n-\n-struct bitmap_commit_cb {\n-\tstruct commit **commits;\n-\tsize_t commits_nr, commits_alloc;\n-\n-\tstruct write_midx_context *ctx;\n-};\n-\n-static const struct object_id *bitmap_oid_access(size_t index,\n-\t\t\t\t\t\t const void *_entries)\n-{\n-\tconst struct pack_midx_entry *entries = _entries;\n-\treturn &entries[index].oid;\n-}\n-\n-static void bitmap_show_commit(struct commit *commit, void *_data)\n-{\n-\tstruct bitmap_commit_cb *data = _data;\n-\tint pos = oid_pos(&commit->object.oid, data->ctx->entries,\n-\t\t\t  data->ctx->entries_nr,\n-\t\t\t  bitmap_oid_access);\n-\tif (pos < 0)\n-\t\treturn;\n-\n-\tALLOC_GROW(data->commits, data->commits_nr + 1, data->commits_alloc);\n-\tdata->commits[data->commits_nr++] = commit;\n-}\n-\n-static int read_refs_snapshot(const char *refs_snapshot,\n-\t\t\t      struct rev_info *revs)\n-{\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tstruct object_id oid;\n-\tFILE *f = xfopen(refs_snapshot, \"r\");\n-\n-\twhile (strbuf_getline(&buf, f) != EOF) {\n-\t\tstruct object *object;\n-\t\tint preferred = 0;\n-\t\tchar *hex = buf.buf;\n-\t\tconst char *end = NULL;\n-\n-\t\tif (buf.len && *buf.buf == '+') {\n-\t\t\tpreferred = 1;\n-\t\t\thex = &buf.buf[1];\n-\t\t}\n-\n-\t\tif (parse_oid_hex(hex, &oid, &end) < 0)\n-\t\t\tdie(_(\"could not parse line: %s\"), buf.buf);\n-\t\tif (*end)\n-\t\t\tdie(_(\"malformed line: %s\"), buf.buf);\n-\n-\t\tobject = parse_object_or_die(&oid, NULL);\n-\t\tif (preferred)\n-\t\t\tobject->flags |= NEEDS_BITMAP;\n-\n-\t\tadd_pending_object(revs, object, \"\");\n-\t}\n-\n-\tfclose(f);\n-\tstrbuf_release(&buf);\n-\treturn 0;\n-}\n-\n-static struct commit **find_commits_for_midx_bitmap(uint32_t *indexed_commits_nr_p,\n-\t\t\t\t\t\t    const char *refs_snapshot,\n-\t\t\t\t\t\t    struct write_midx_context *ctx)\n-{\n-\tstruct rev_info revs;\n-\tstruct bitmap_commit_cb cb = {0};\n-\n-\ttrace2_region_enter(\"midx\", \"find_commits_for_midx_bitmap\",\n-\t\t\t    the_repository);\n-\n-\tcb.ctx = ctx;\n-\n-\trepo_init_revisions(the_repository, &revs, NULL);\n-\tif (refs_snapshot) {\n-\t\tread_refs_snapshot(refs_snapshot, &revs);\n-\t} else {\n-\t\tsetup_revisions(0, NULL, &revs, NULL);\n-\t\tfor_each_ref(add_ref_to_pending, &revs);\n-\t}\n-\n-\t/*\n-\t * Skipping promisor objects here is intentional, since it only excludes\n-\t * them from the list of reachable commits that we want to select from\n-\t * when computing the selection of MIDX'd commits to receive bitmaps.\n-\t *\n-\t * Reachability bitmaps do require that their objects be closed under\n-\t * reachability, but fetching any objects missing from promisors at this\n-\t * point is too late. But, if one of those objects can be reached from\n-\t * an another object that is included in the bitmap, then we will\n-\t * complain later that we don't have reachability closure (and fail\n-\t * appropriately).\n-\t */\n-\tfetch_if_missing = 0;\n-\trevs.exclude_promisor_objects = 1;\n-\n-\tif (prepare_revision_walk(&revs))\n-\t\tdie(_(\"revision walk setup failed\"));\n-\n-\ttraverse_commit_list(&revs, bitmap_show_commit, NULL, &cb);\n-\tif (indexed_commits_nr_p)\n-\t\t*indexed_commits_nr_p = cb.commits_nr;\n-\n-\trelease_revisions(&revs);\n-\n-\ttrace2_region_leave(\"midx\", \"find_commits_for_midx_bitmap\",\n-\t\t\t    the_repository);\n-\n-\treturn cb.commits;\n-}\n-\n-static int write_midx_bitmap(const char *midx_name,\n-\t\t\t     const unsigned char *midx_hash,\n-\t\t\t     struct packing_data *pdata,\n-\t\t\t     struct commit **commits,\n-\t\t\t     uint32_t commits_nr,\n-\t\t\t     uint32_t *pack_order,\n-\t\t\t     unsigned flags)\n-{\n-\tint ret, i;\n-\tuint16_t options = 0;\n-\tstruct pack_idx_entry **index;\n-\tchar *bitmap_name = xstrfmt(\"%s-%s.bitmap\", midx_name,\n-\t\t\t\t\thash_to_hex(midx_hash));\n-\n-\ttrace2_region_enter(\"midx\", \"write_midx_bitmap\", the_repository);\n-\n-\tif (flags & MIDX_WRITE_BITMAP_HASH_CACHE)\n-\t\toptions |= BITMAP_OPT_HASH_CACHE;\n-\n-\tif (flags & MIDX_WRITE_BITMAP_LOOKUP_TABLE)\n-\t\toptions |= BITMAP_OPT_LOOKUP_TABLE;\n-\n-\t/*\n-\t * Build the MIDX-order index based on pdata.objects (which is already\n-\t * in MIDX order; c.f., 'midx_pack_order_cmp()' for the definition of\n-\t * this order).\n-\t */\n-\tALLOC_ARRAY(index, pdata->nr_objects);\n-\tfor (i = 0; i < pdata->nr_objects; i++)\n-\t\tindex[i] = &pdata->objects[i].idx;\n-\n-\tbitmap_writer_show_progress(flags & MIDX_PROGRESS);\n-\tbitmap_writer_build_type_index(pdata, index, pdata->nr_objects);\n-\n-\t/*\n-\t * bitmap_writer_finish expects objects in lex order, but pack_order\n-\t * gives us exactly that. use it directly instead of re-sorting the\n-\t * array.\n-\t *\n-\t * This changes the order of objects in 'index' between\n-\t * bitmap_writer_build_type_index and bitmap_writer_finish.\n-\t *\n-\t * The same re-ordering takes place in the single-pack bitmap code via\n-\t * write_idx_file(), which is called by finish_tmp_packfile(), which\n-\t * happens between bitmap_writer_build_type_index() and\n-\t * bitmap_writer_finish().\n-\t */\n-\tfor (i = 0; i < pdata->nr_objects; i++)\n-\t\tindex[pack_order[i]] = &pdata->objects[i].idx;\n-\n-\tbitmap_writer_select_commits(commits, commits_nr, -1);\n-\tret = bitmap_writer_build(pdata);\n-\tif (ret < 0)\n-\t\tgoto cleanup;\n-\n-\tbitmap_writer_set_checksum(midx_hash);\n-\tbitmap_writer_finish(index, pdata->nr_objects, bitmap_name, options);\n-\n-cleanup:\n-\tfree(index);\n-\tfree(bitmap_name);\n-\n-\ttrace2_region_leave(\"midx\", \"write_midx_bitmap\", the_repository);\n-\n-\treturn ret;\n-}\n-\n-static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,\n-\t\t\t\t\t\t\tconst char *object_dir)\n-{\n-\tstruct multi_pack_index *result = NULL;\n-\tstruct multi_pack_index *cur;\n-\tchar *obj_dir_real = real_pathdup(object_dir, 1);\n-\tstruct strbuf cur_path_real = STRBUF_INIT;\n-\n-\t/* Ensure the given object_dir is local, or a known alternate. */\n-\tfind_odb(r, obj_dir_real);\n-\n-\tfor (cur = get_multi_pack_index(r); cur; cur = cur->next) {\n-\t\tstrbuf_realpath(&cur_path_real, cur->object_dir, 1);\n-\t\tif (!strcmp(obj_dir_real, cur_path_real.buf)) {\n-\t\t\tresult = cur;\n-\t\t\tgoto cleanup;\n-\t\t}\n-\t}\n-\n-cleanup:\n-\tfree(obj_dir_real);\n-\tstrbuf_release(&cur_path_real);\n-\treturn result;\n-}\n-\n-static int write_midx_internal(const char *object_dir,\n-\t\t\t       struct string_list *packs_to_include,\n-\t\t\t       struct string_list *packs_to_drop,\n-\t\t\t       const char *preferred_pack_name,\n-\t\t\t       const char *refs_snapshot,\n-\t\t\t       unsigned flags)\n-{\n-\tstruct strbuf midx_name = STRBUF_INIT;\n-\tunsigned char midx_hash[GIT_MAX_RAWSZ];\n-\tuint32_t i;\n-\tstruct hashfile *f = NULL;\n-\tstruct lock_file lk;\n-\tstruct write_midx_context ctx = { 0 };\n-\tint bitmapped_packs_concat_len = 0;\n-\tint pack_name_concat_len = 0;\n-\tint dropped_packs = 0;\n-\tint result = 0;\n-\tstruct chunkfile *cf;\n-\n-\ttrace2_region_enter(\"midx\", \"write_midx_internal\", the_repository);\n-\n-\tget_midx_filename(&midx_name, object_dir);\n-\tif (safe_create_leading_directories(midx_name.buf))\n-\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n-\t\t\t  midx_name.buf);\n-\n-\tif (!packs_to_include) {\n-\t\t/*\n-\t\t * Only reference an existing MIDX when not filtering which\n-\t\t * packs to include, since all packs and objects are copied\n-\t\t * blindly from an existing MIDX if one is present.\n-\t\t */\n-\t\tctx.m = lookup_multi_pack_index(the_repository, object_dir);\n-\t}\n-\n-\tif (ctx.m && !midx_checksum_valid(ctx.m)) {\n-\t\twarning(_(\"ignoring existing multi-pack-index; checksum mismatch\"));\n-\t\tctx.m = NULL;\n-\t}\n-\n-\tctx.nr = 0;\n-\tctx.alloc = ctx.m ? ctx.m->num_packs : 16;\n-\tctx.info = NULL;\n-\tALLOC_ARRAY(ctx.info, ctx.alloc);\n-\n-\tif (ctx.m) {\n-\t\tfor (i = 0; i < ctx.m->num_packs; i++) {\n-\t\t\tALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc);\n-\n-\t\t\tif (flags & MIDX_WRITE_REV_INDEX) {\n-\t\t\t\t/*\n-\t\t\t\t * If generating a reverse index, need to have\n-\t\t\t\t * packed_git's loaded to compare their\n-\t\t\t\t * mtimes and object count.\n-\t\t\t\t */\n-\t\t\t\tif (prepare_midx_pack(the_repository, ctx.m, i)) {\n-\t\t\t\t\terror(_(\"could not load pack\"));\n-\t\t\t\t\tresult = 1;\n-\t\t\t\t\tgoto cleanup;\n-\t\t\t\t}\n-\n-\t\t\t\tif (open_pack_index(ctx.m->packs[i]))\n-\t\t\t\t\tdie(_(\"could not open index for %s\"),\n-\t\t\t\t\t    ctx.m->packs[i]->pack_name);\n-\t\t\t}\n-\n-\t\t\tfill_pack_info(&ctx.info[ctx.nr++], ctx.m->packs[i],\n-\t\t\t\t       ctx.m->pack_names[i], i);\n-\t\t}\n-\t}\n-\n-\tctx.pack_paths_checked = 0;\n-\tif (flags & MIDX_PROGRESS)\n-\t\tctx.progress = start_delayed_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n-\telse\n-\t\tctx.progress = NULL;\n-\n-\tctx.to_include = packs_to_include;\n-\n-\tfor_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);\n-\tstop_progress(&ctx.progress);\n-\n-\tif ((ctx.m && ctx.nr == ctx.m->num_packs) &&\n-\t    !(packs_to_include || packs_to_drop)) {\n-\t\tstruct bitmap_index *bitmap_git;\n-\t\tint bitmap_exists;\n-\t\tint want_bitmap = flags & MIDX_WRITE_BITMAP;\n-\n-\t\tbitmap_git = prepare_midx_bitmap_git(ctx.m);\n-\t\tbitmap_exists = bitmap_git && bitmap_is_midx(bitmap_git);\n-\t\tfree_bitmap_index(bitmap_git);\n-\n-\t\tif (bitmap_exists || !want_bitmap) {\n-\t\t\t/*\n-\t\t\t * The correct MIDX already exists, and so does a\n-\t\t\t * corresponding bitmap (or one wasn't requested).\n-\t\t\t */\n-\t\t\tif (!want_bitmap)\n-\t\t\t\tclear_midx_files_ext(object_dir, \".bitmap\",\n-\t\t\t\t\t\t     NULL);\n-\t\t\tgoto cleanup;\n-\t\t}\n-\t}\n-\n-\tif (preferred_pack_name) {\n-\t\tctx.preferred_pack_idx = -1;\n-\n-\t\tfor (i = 0; i < ctx.nr; i++) {\n-\t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n-\t\t\t\t\t\t  ctx.info[i].pack_name)) {\n-\t\t\t\tctx.preferred_pack_idx = i;\n-\t\t\t\tbreak;\n-\t\t\t}\n-\t\t}\n-\n-\t\tif (ctx.preferred_pack_idx == -1)\n-\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n-\t\t\t\tpreferred_pack_name);\n-\t} else if (ctx.nr &&\n-\t\t   (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))) {\n-\t\tstruct packed_git *oldest = ctx.info[ctx.preferred_pack_idx].p;\n-\t\tctx.preferred_pack_idx = 0;\n-\n-\t\tif (packs_to_drop && packs_to_drop->nr)\n-\t\t\tBUG(\"cannot write a MIDX bitmap during expiration\");\n-\n-\t\t/*\n-\t\t * set a preferred pack when writing a bitmap to ensure that\n-\t\t * the pack from which the first object is selected in pseudo\n-\t\t * pack-order has all of its objects selected from that pack\n-\t\t * (and not another pack containing a duplicate)\n-\t\t */\n-\t\tfor (i = 1; i < ctx.nr; i++) {\n-\t\t\tstruct packed_git *p = ctx.info[i].p;\n-\n-\t\t\tif (!oldest->num_objects || p->mtime < oldest->mtime) {\n-\t\t\t\toldest = p;\n-\t\t\t\tctx.preferred_pack_idx = i;\n-\t\t\t}\n-\t\t}\n-\n-\t\tif (!oldest->num_objects) {\n-\t\t\t/*\n-\t\t\t * If all packs are empty; unset the preferred index.\n-\t\t\t * This is acceptable since there will be no duplicate\n-\t\t\t * objects to resolve, so the preferred value doesn't\n-\t\t\t * matter.\n-\t\t\t */\n-\t\t\tctx.preferred_pack_idx = -1;\n-\t\t}\n-\t} else {\n-\t\t/*\n-\t\t * otherwise don't mark any pack as preferred to avoid\n-\t\t * interfering with expiration logic below\n-\t\t */\n-\t\tctx.preferred_pack_idx = -1;\n-\t}\n-\n-\tif (ctx.preferred_pack_idx > -1) {\n-\t\tstruct packed_git *preferred = ctx.info[ctx.preferred_pack_idx].p;\n-\t\tif (!preferred->num_objects) {\n-\t\t\terror(_(\"cannot select preferred pack %s with no objects\"),\n-\t\t\t      preferred->pack_name);\n-\t\t\tresult = 1;\n-\t\t\tgoto cleanup;\n-\t\t}\n-\t}\n-\n-\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,\n-\t\t\t\t\t ctx.preferred_pack_idx);\n-\n-\tctx.large_offsets_needed = 0;\n-\tfor (i = 0; i < ctx.entries_nr; i++) {\n-\t\tif (ctx.entries[i].offset > 0x7fffffff)\n-\t\t\tctx.num_large_offsets++;\n-\t\tif (ctx.entries[i].offset > 0xffffffff)\n-\t\t\tctx.large_offsets_needed = 1;\n-\t}\n-\n-\tQSORT(ctx.info, ctx.nr, pack_info_compare);\n-\n-\tif (packs_to_drop && packs_to_drop->nr) {\n-\t\tint drop_index = 0;\n-\t\tint missing_drops = 0;\n-\n-\t\tfor (i = 0; i < ctx.nr && drop_index < packs_to_drop->nr; i++) {\n-\t\t\tint cmp = strcmp(ctx.info[i].pack_name,\n-\t\t\t\t\t packs_to_drop->items[drop_index].string);\n-\n-\t\t\tif (!cmp) {\n-\t\t\t\tdrop_index++;\n-\t\t\t\tctx.info[i].expired = 1;\n-\t\t\t} else if (cmp > 0) {\n-\t\t\t\terror(_(\"did not see pack-file %s to drop\"),\n-\t\t\t\t      packs_to_drop->items[drop_index].string);\n-\t\t\t\tdrop_index++;\n-\t\t\t\tmissing_drops++;\n-\t\t\t\ti--;\n-\t\t\t} else {\n-\t\t\t\tctx.info[i].expired = 0;\n-\t\t\t}\n-\t\t}\n-\n-\t\tif (missing_drops) {\n-\t\t\tresult = 1;\n-\t\t\tgoto cleanup;\n-\t\t}\n-\t}\n-\n-\t/*\n-\t * pack_perm stores a permutation between pack-int-ids from the\n-\t * previous multi-pack-index to the new one we are writing:\n-\t *\n-\t * pack_perm[old_id] = new_id\n-\t */\n-\tALLOC_ARRAY(ctx.pack_perm, ctx.nr);\n-\tfor (i = 0; i < ctx.nr; i++) {\n-\t\tif (ctx.info[i].expired) {\n-\t\t\tdropped_packs++;\n-\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = PACK_EXPIRED;\n-\t\t} else {\n-\t\t\tctx.pack_perm[ctx.info[i].orig_pack_int_id] = i - dropped_packs;\n-\t\t}\n-\t}\n-\n-\tfor (i = 0; i < ctx.nr; i++) {\n-\t\tif (ctx.info[i].expired)\n-\t\t\tcontinue;\n-\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n-\t\tbitmapped_packs_concat_len += 2 * sizeof(uint32_t);\n-\t}\n-\n-\t/* Check that the preferred pack wasn't expired (if given). */\n-\tif (preferred_pack_name) {\n-\t\tstruct pack_info *preferred = bsearch(preferred_pack_name,\n-\t\t\t\t\t\t      ctx.info, ctx.nr,\n-\t\t\t\t\t\t      sizeof(*ctx.info),\n-\t\t\t\t\t\t      idx_or_pack_name_cmp);\n-\t\tif (preferred) {\n-\t\t\tuint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];\n-\t\t\tif (perm == PACK_EXPIRED)\n-\t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n-\t\t\t\t\tpreferred_pack_name);\n-\t\t}\n-\t}\n-\n-\tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n-\t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n-\t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n-\n-\thold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR);\n-\tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n-\n-\tif (ctx.nr - dropped_packs == 0) {\n-\t\terror(_(\"no pack files to index.\"));\n-\t\tresult = 1;\n-\t\tgoto cleanup;\n-\t}\n-\n-\tif (!ctx.entries_nr) {\n-\t\tif (flags & MIDX_WRITE_BITMAP)\n-\t\t\twarning(_(\"refusing to write multi-pack .bitmap without any objects\"));\n-\t\tflags &= ~(MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP);\n-\t}\n-\n-\tcf = init_chunkfile(f);\n-\n-\tadd_chunk(cf, MIDX_CHUNKID_PACKNAMES, pack_name_concat_len,\n-\t\t  write_midx_pack_names);\n-\tadd_chunk(cf, MIDX_CHUNKID_OIDFANOUT, MIDX_CHUNK_FANOUT_SIZE,\n-\t\t  write_midx_oid_fanout);\n-\tadd_chunk(cf, MIDX_CHUNKID_OIDLOOKUP,\n-\t\t  st_mult(ctx.entries_nr, the_hash_algo->rawsz),\n-\t\t  write_midx_oid_lookup);\n-\tadd_chunk(cf, MIDX_CHUNKID_OBJECTOFFSETS,\n-\t\t  st_mult(ctx.entries_nr, MIDX_CHUNK_OFFSET_WIDTH),\n-\t\t  write_midx_object_offsets);\n-\n-\tif (ctx.large_offsets_needed)\n-\t\tadd_chunk(cf, MIDX_CHUNKID_LARGEOFFSETS,\n-\t\t\tst_mult(ctx.num_large_offsets,\n-\t\t\t\tMIDX_CHUNK_LARGE_OFFSET_WIDTH),\n-\t\t\twrite_midx_large_offsets);\n-\n-\tif (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) {\n-\t\tctx.pack_order = midx_pack_order(&ctx);\n-\t\tadd_chunk(cf, MIDX_CHUNKID_REVINDEX,\n-\t\t\t  st_mult(ctx.entries_nr, sizeof(uint32_t)),\n-\t\t\t  write_midx_revindex);\n-\t\tadd_chunk(cf, MIDX_CHUNKID_BITMAPPEDPACKS,\n-\t\t\t  bitmapped_packs_concat_len,\n-\t\t\t  write_midx_bitmapped_packs);\n-\t}\n-\n-\twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n-\twrite_chunkfile(cf, &ctx);\n-\n-\tfinalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,\n-\t\t\t  CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n-\tfree_chunkfile(cf);\n-\n-\tif (flags & MIDX_WRITE_REV_INDEX &&\n-\t    git_env_bool(\"GIT_TEST_MIDX_WRITE_REV\", 0))\n-\t\twrite_midx_reverse_index(midx_name.buf, midx_hash, &ctx);\n-\n-\tif (flags & MIDX_WRITE_BITMAP) {\n-\t\tstruct packing_data pdata;\n-\t\tstruct commit **commits;\n-\t\tuint32_t commits_nr;\n-\n-\t\tif (!ctx.entries_nr)\n-\t\t\tBUG(\"cannot write a bitmap without any objects\");\n-\n-\t\tprepare_midx_packing_data(&pdata, &ctx);\n-\n-\t\tcommits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, &ctx);\n-\n-\t\t/*\n-\t\t * The previous steps translated the information from\n-\t\t * 'entries' into information suitable for constructing\n-\t\t * bitmaps. We no longer need that array, so clear it to\n-\t\t * reduce memory pressure.\n-\t\t */\n-\t\tFREE_AND_NULL(ctx.entries);\n-\t\tctx.entries_nr = 0;\n-\n-\t\tif (write_midx_bitmap(midx_name.buf, midx_hash, &pdata,\n-\t\t\t\t      commits, commits_nr, ctx.pack_order,\n-\t\t\t\t      flags) < 0) {\n-\t\t\terror(_(\"could not write multi-pack bitmap\"));\n-\t\t\tresult = 1;\n-\t\t\tclear_packing_data(&pdata);\n-\t\t\tfree(commits);\n-\t\t\tgoto cleanup;\n-\t\t}\n-\n-\t\tclear_packing_data(&pdata);\n-\t\tfree(commits);\n-\t}\n-\t/*\n-\t * NOTE: Do not use ctx.entries beyond this point, since it might\n-\t * have been freed in the previous if block.\n-\t */\n-\n-\tif (ctx.m)\n-\t\tclose_object_store(the_repository->objects);\n-\n-\tif (commit_lock_file(&lk) < 0)\n-\t\tdie_errno(_(\"could not write multi-pack-index\"));\n-\n-\tclear_midx_files_ext(object_dir, \".bitmap\", midx_hash);\n-\tclear_midx_files_ext(object_dir, \".rev\", midx_hash);\n-\n-cleanup:\n-\tfor (i = 0; i < ctx.nr; i++) {\n-\t\tif (ctx.info[i].p) {\n-\t\t\tclose_pack(ctx.info[i].p);\n-\t\t\tfree(ctx.info[i].p);\n-\t\t}\n-\t\tfree(ctx.info[i].pack_name);\n-\t}\n-\n-\tfree(ctx.info);\n-\tfree(ctx.entries);\n-\tfree(ctx.pack_perm);\n-\tfree(ctx.pack_order);\n-\tstrbuf_release(&midx_name);\n-\n-\ttrace2_region_leave(\"midx\", \"write_midx_internal\", the_repository);\n-\n-\treturn result;\n-}\n-\n-int write_midx_file(const char *object_dir,\n-\t\t    const char *preferred_pack_name,\n-\t\t    const char *refs_snapshot,\n-\t\t    unsigned flags)\n-{\n-\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n-\t\t\t\t   refs_snapshot, flags);\n-}\n-\n-int write_midx_file_only(const char *object_dir,\n-\t\t\t struct string_list *packs_to_include,\n-\t\t\t const char *preferred_pack_name,\n-\t\t\t const char *refs_snapshot,\n-\t\t\t unsigned flags)\n-{\n-\treturn write_midx_internal(object_dir, packs_to_include, NULL,\n-\t\t\t\t   preferred_pack_name, refs_snapshot, flags);\n-}\n-\n struct clear_midx_data {\n \tchar *keep;\n \tconst char *ext;\n@@ -1784,8 +507,8 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS\n \t\tdie_errno(_(\"failed to remove %s\"), full_path);\n }\n \n-static void clear_midx_files_ext(const char *object_dir, const char *ext,\n-\t\t\t\t unsigned char *keep_hash)\n+void clear_midx_files_ext(const char *object_dir, const char *ext,\n+\t\t\t  unsigned char *keep_hash)\n {\n \tstruct clear_midx_data data;\n \tmemset(&data, 0, sizeof(struct clear_midx_data));\n@@ -1988,251 +711,3 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \n \treturn verify_midx_error;\n }\n-\n-int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags)\n-{\n-\tuint32_t i, *count, result = 0;\n-\tstruct string_list packs_to_drop = STRING_LIST_INIT_DUP;\n-\tstruct multi_pack_index *m = lookup_multi_pack_index(r, object_dir);\n-\tstruct progress *progress = NULL;\n-\n-\tif (!m)\n-\t\treturn 0;\n-\n-\tCALLOC_ARRAY(count, m->num_packs);\n-\n-\tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_delayed_progress(_(\"Counting referenced objects\"),\n-\t\t\t\t\t  m->num_objects);\n-\tfor (i = 0; i < m->num_objects; i++) {\n-\t\tint pack_int_id = nth_midxed_pack_int_id(m, i);\n-\t\tcount[pack_int_id]++;\n-\t\tdisplay_progress(progress, i + 1);\n-\t}\n-\tstop_progress(&progress);\n-\n-\tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_delayed_progress(_(\"Finding and deleting unreferenced packfiles\"),\n-\t\t\t\t\t  m->num_packs);\n-\tfor (i = 0; i < m->num_packs; i++) {\n-\t\tchar *pack_name;\n-\t\tdisplay_progress(progress, i + 1);\n-\n-\t\tif (count[i])\n-\t\t\tcontinue;\n-\n-\t\tif (prepare_midx_pack(r, m, i))\n-\t\t\tcontinue;\n-\n-\t\tif (m->packs[i]->pack_keep || m->packs[i]->is_cruft)\n-\t\t\tcontinue;\n-\n-\t\tpack_name = xstrdup(m->packs[i]->pack_name);\n-\t\tclose_pack(m->packs[i]);\n-\n-\t\tstring_list_insert(&packs_to_drop, m->pack_names[i]);\n-\t\tunlink_pack_path(pack_name, 0);\n-\t\tfree(pack_name);\n-\t}\n-\tstop_progress(&progress);\n-\n-\tfree(count);\n-\n-\tif (packs_to_drop.nr)\n-\t\tresult = write_midx_internal(object_dir, NULL, &packs_to_drop, NULL, NULL, flags);\n-\n-\tstring_list_clear(&packs_to_drop, 0);\n-\n-\treturn result;\n-}\n-\n-struct repack_info {\n-\ttimestamp_t mtime;\n-\tuint32_t referenced_objects;\n-\tuint32_t pack_int_id;\n-};\n-\n-static int compare_by_mtime(const void *a_, const void *b_)\n-{\n-\tconst struct repack_info *a, *b;\n-\n-\ta = (const struct repack_info *)a_;\n-\tb = (const struct repack_info *)b_;\n-\n-\tif (a->mtime < b->mtime)\n-\t\treturn -1;\n-\tif (a->mtime > b->mtime)\n-\t\treturn 1;\n-\treturn 0;\n-}\n-\n-static int fill_included_packs_all(struct repository *r,\n-\t\t\t\t   struct multi_pack_index *m,\n-\t\t\t\t   unsigned char *include_pack)\n-{\n-\tuint32_t i, count = 0;\n-\tint pack_kept_objects = 0;\n-\n-\trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n-\n-\tfor (i = 0; i < m->num_packs; i++) {\n-\t\tif (prepare_midx_pack(r, m, i))\n-\t\t\tcontinue;\n-\t\tif (!pack_kept_objects && m->packs[i]->pack_keep)\n-\t\t\tcontinue;\n-\t\tif (m->packs[i]->is_cruft)\n-\t\t\tcontinue;\n-\n-\t\tinclude_pack[i] = 1;\n-\t\tcount++;\n-\t}\n-\n-\treturn count < 2;\n-}\n-\n-static int fill_included_packs_batch(struct repository *r,\n-\t\t\t\t     struct multi_pack_index *m,\n-\t\t\t\t     unsigned char *include_pack,\n-\t\t\t\t     size_t batch_size)\n-{\n-\tuint32_t i, packs_to_repack;\n-\tsize_t total_size;\n-\tstruct repack_info *pack_info;\n-\tint pack_kept_objects = 0;\n-\n-\tCALLOC_ARRAY(pack_info, m->num_packs);\n-\n-\trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n-\n-\tfor (i = 0; i < m->num_packs; i++) {\n-\t\tpack_info[i].pack_int_id = i;\n-\n-\t\tif (prepare_midx_pack(r, m, i))\n-\t\t\tcontinue;\n-\n-\t\tpack_info[i].mtime = m->packs[i]->mtime;\n-\t}\n-\n-\tfor (i = 0; i < m->num_objects; i++) {\n-\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n-\t\tpack_info[pack_int_id].referenced_objects++;\n-\t}\n-\n-\tQSORT(pack_info, m->num_packs, compare_by_mtime);\n-\n-\ttotal_size = 0;\n-\tpacks_to_repack = 0;\n-\tfor (i = 0; total_size < batch_size && i < m->num_packs; i++) {\n-\t\tint pack_int_id = pack_info[i].pack_int_id;\n-\t\tstruct packed_git *p = m->packs[pack_int_id];\n-\t\tsize_t expected_size;\n-\n-\t\tif (!p)\n-\t\t\tcontinue;\n-\t\tif (!pack_kept_objects && p->pack_keep)\n-\t\t\tcontinue;\n-\t\tif (p->is_cruft)\n-\t\t\tcontinue;\n-\t\tif (open_pack_index(p) || !p->num_objects)\n-\t\t\tcontinue;\n-\n-\t\texpected_size = st_mult(p->pack_size,\n-\t\t\t\t\tpack_info[i].referenced_objects);\n-\t\texpected_size /= p->num_objects;\n-\n-\t\tif (expected_size >= batch_size)\n-\t\t\tcontinue;\n-\n-\t\tpacks_to_repack++;\n-\t\ttotal_size += expected_size;\n-\t\tinclude_pack[pack_int_id] = 1;\n-\t}\n-\n-\tfree(pack_info);\n-\n-\tif (packs_to_repack < 2)\n-\t\treturn 1;\n-\n-\treturn 0;\n-}\n-\n-int midx_repack(struct repository *r, const char *object_dir, size_t batch_size, unsigned flags)\n-{\n-\tint result = 0;\n-\tuint32_t i;\n-\tunsigned char *include_pack;\n-\tstruct child_process cmd = CHILD_PROCESS_INIT;\n-\tFILE *cmd_in;\n-\tstruct multi_pack_index *m = lookup_multi_pack_index(r, object_dir);\n-\n-\t/*\n-\t * When updating the default for these configuration\n-\t * variables in builtin/repack.c, these must be adjusted\n-\t * to match.\n-\t */\n-\tint delta_base_offset = 1;\n-\tint use_delta_islands = 0;\n-\n-\tif (!m)\n-\t\treturn 0;\n-\n-\tCALLOC_ARRAY(include_pack, m->num_packs);\n-\n-\tif (batch_size) {\n-\t\tif (fill_included_packs_batch(r, m, include_pack, batch_size))\n-\t\t\tgoto cleanup;\n-\t} else if (fill_included_packs_all(r, m, include_pack))\n-\t\tgoto cleanup;\n-\n-\trepo_config_get_bool(r, \"repack.usedeltabaseoffset\", &delta_base_offset);\n-\trepo_config_get_bool(r, \"repack.usedeltaislands\", &use_delta_islands);\n-\n-\tstrvec_push(&cmd.args, \"pack-objects\");\n-\n-\tstrvec_pushf(&cmd.args, \"%s/pack/pack\", object_dir);\n-\n-\tif (delta_base_offset)\n-\t\tstrvec_push(&cmd.args, \"--delta-base-offset\");\n-\tif (use_delta_islands)\n-\t\tstrvec_push(&cmd.args, \"--delta-islands\");\n-\n-\tif (flags & MIDX_PROGRESS)\n-\t\tstrvec_push(&cmd.args, \"--progress\");\n-\telse\n-\t\tstrvec_push(&cmd.args, \"-q\");\n-\n-\tcmd.git_cmd = 1;\n-\tcmd.in = cmd.out = -1;\n-\n-\tif (start_command(&cmd)) {\n-\t\terror(_(\"could not start pack-objects\"));\n-\t\tresult = 1;\n-\t\tgoto cleanup;\n-\t}\n-\n-\tcmd_in = xfdopen(cmd.in, \"w\");\n-\n-\tfor (i = 0; i < m->num_objects; i++) {\n-\t\tstruct object_id oid;\n-\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n-\n-\t\tif (!include_pack[pack_int_id])\n-\t\t\tcontinue;\n-\n-\t\tnth_midxed_object_oid(&oid, m, i);\n-\t\tfprintf(cmd_in, \"%s\\n\", oid_to_hex(&oid));\n-\t}\n-\tfclose(cmd_in);\n-\n-\tif (finish_command(&cmd)) {\n-\t\terror(_(\"could not finish pack-objects\"));\n-\t\tresult = 1;\n-\t\tgoto cleanup;\n-\t}\n-\n-\tresult = write_midx_internal(object_dir, NULL, NULL, NULL, NULL, flags);\n-\n-cleanup:\n-\tfree(include_pack);\n-\treturn result;\n-}\ndiff --git a/midx.h b/midx.h\nindex b374a7afaf..dc477dff44 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -8,6 +8,25 @@ struct pack_entry;\n struct repository;\n struct bitmapped_pack;\n \n+#define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n+#define MIDX_VERSION 1\n+#define MIDX_BYTE_FILE_VERSION 4\n+#define MIDX_BYTE_HASH_VERSION 5\n+#define MIDX_BYTE_NUM_CHUNKS 6\n+#define MIDX_BYTE_NUM_PACKS 8\n+#define MIDX_HEADER_SIZE 12\n+\n+#define MIDX_CHUNK_ALIGNMENT 4\n+#define MIDX_CHUNKID_PACKNAMES 0x504e414d /* \"PNAM\" */\n+#define MIDX_CHUNKID_BITMAPPEDPACKS 0x42544d50 /* \"BTMP\" */\n+#define MIDX_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n+#define MIDX_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n+#define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* \"OOFF\" */\n+#define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* \"LOFF\" */\n+#define MIDX_CHUNKID_REVINDEX 0x52494458 /* \"RIDX\" */\n+#define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))\n+#define MIDX_LARGE_OFFSET_NEEDED 0x80000000\n+\n #define GIT_TEST_MULTI_PACK_INDEX \"GIT_TEST_MULTI_PACK_INDEX\"\n #define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \\\n \t\"GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP\"\n-- \n2.44.0.330.g158d2a670b4\n\n"},{"id":"491988","messageId":"0064e363c0cc3288346585a6b4340444ce7b863c.1712006190.git.me@ttaylorr.com","threadId":"61199","inReplyTo":"cover.1712006190.git.me@ttaylorr.com","subject":"[PATCH v2 2/4] midx-write.c: factor out common want_included_pack() routine","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-01T21:16:38Z","receivedAt":"2024-04-01T21:16:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When performing a 'git multi-pack-index repack', the MIDX machinery\ntries to aggregate MIDX'd packs together either to (a) fill the given\n`--batch-size` argument, or (b) combine all packs together.\n\nIn either case (using the `midx-write.c::fill_included_packs_batch()` or\n`midx-write.c::fill_included_packs_all()` function, respectively), we\nevaluate whether or not we want to repack each MIDX'd pack, according to\nwhether or it is loadable, kept, cruft, or non-empty.\n\nBetween the two `fill_included_packs_` callers, they both care about the\nsame conditions, except for `fill_included_packs_batch()` which also\ncares that the pack is non-empty.\n\nWe could extract two functions (say, `want_included_pack()` and a\n`_nonempty()` variant), but this is not necessary. For the case in\n`fill_included_packs_all()` which does not check the pack size, we add\nall of the pack's objects assuming that the pack meets all other\ncriteria. But if the pack is empty in the first place, we add all of its\nzero objects, so whether or not we \"accept\" or \"reject\" it in the first\nplace is irrelevant.\n\nThis change improves the readability in both `fill_included_packs_`\nfunctions.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx-write.c | 32 ++++++++++++++++++++------------\n 1 file changed, 20 insertions(+), 12 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 5242d2a724..906efa2169 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1349,6 +1349,24 @@ static int compare_by_mtime(const void *a_, const void *b_)\n \treturn 0;\n }\n \n+static int want_included_pack(struct repository *r,\n+\t\t\t      struct multi_pack_index *m,\n+\t\t\t      int pack_kept_objects,\n+\t\t\t      uint32_t pack_int_id)\n+{\n+\tstruct packed_git *p;\n+\tif (prepare_midx_pack(r, m, pack_int_id))\n+\t\treturn 0;\n+\tp = m->packs[pack_int_id];\n+\tif (!pack_kept_objects && p->pack_keep)\n+\t\treturn 0;\n+\tif (p->is_cruft)\n+\t\treturn 0;\n+\tif (open_pack_index(p) || !p->num_objects)\n+\t\treturn 0;\n+\treturn 1;\n+}\n+\n static int fill_included_packs_all(struct repository *r,\n \t\t\t\t   struct multi_pack_index *m,\n \t\t\t\t   unsigned char *include_pack)\n@@ -1359,11 +1377,7 @@ static int fill_included_packs_all(struct repository *r,\n \trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n \n \tfor (i = 0; i < m->num_packs; i++) {\n-\t\tif (prepare_midx_pack(r, m, i))\n-\t\t\tcontinue;\n-\t\tif (!pack_kept_objects && m->packs[i]->pack_keep)\n-\t\t\tcontinue;\n-\t\tif (m->packs[i]->is_cruft)\n+\t\tif (!want_included_pack(r, m, pack_kept_objects, i))\n \t\t\tcontinue;\n \n \t\tinclude_pack[i] = 1;\n@@ -1410,13 +1424,7 @@ static int fill_included_packs_batch(struct repository *r,\n \t\tstruct packed_git *p = m->packs[pack_int_id];\n \t\tsize_t expected_size;\n \n-\t\tif (!p)\n-\t\t\tcontinue;\n-\t\tif (!pack_kept_objects && p->pack_keep)\n-\t\t\tcontinue;\n-\t\tif (p->is_cruft)\n-\t\t\tcontinue;\n-\t\tif (open_pack_index(p) || !p->num_objects)\n+\t\tif (!want_included_pack(r, m, pack_kept_objects, pack_int_id))\n \t\t\tcontinue;\n \n \t\texpected_size = st_mult(p->pack_size,\n-- \n2.44.0.330.g158d2a670b4\n\n"},{"id":"491990","messageId":"b121f05a32b6639e6d8d28c8c7eb8fef1c7bfb59.1712006190.git.me@ttaylorr.com","threadId":"61199","inReplyTo":"cover.1712006190.git.me@ttaylorr.com","subject":"[PATCH v2 3/4] midx-write.c: check count of packs to repack after grouping","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-01T21:16:41Z","receivedAt":"2024-04-01T21:16:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In both fill_included_packs_all() and fill_included_packs_batch(), we\naccumulate a list of packs whose contents we want to repack together,\nand then use that information to feed a list of objects as input to\npack-objects.\n\nIn both cases, the `fill_included_packs_` functions keep track of how\nmany packs they want to repack together, and only execute pack-objects\nif there are at least two packs that need repacking.\n\nHaving both of these functions keep track of this information themselves\nis not strictly necessary, since they also log which packs to repack via\nthe `include_pack` array, so we can simply count the non-zero entries in\nthat array after either function is done executing, reducing the overall\namount of code necessary.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx-write.c | 44 ++++++++++++++++++++------------------------\n 1 file changed, 20 insertions(+), 24 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 906efa2169..960cc46250 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1367,11 +1367,11 @@ static int want_included_pack(struct repository *r,\n \treturn 1;\n }\n \n-static int fill_included_packs_all(struct repository *r,\n-\t\t\t\t   struct multi_pack_index *m,\n-\t\t\t\t   unsigned char *include_pack)\n+static void fill_included_packs_all(struct repository *r,\n+\t\t\t\t    struct multi_pack_index *m,\n+\t\t\t\t    unsigned char *include_pack)\n {\n-\tuint32_t i, count = 0;\n+\tuint32_t i;\n \tint pack_kept_objects = 0;\n \n \trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n@@ -1381,18 +1381,15 @@ static int fill_included_packs_all(struct repository *r,\n \t\t\tcontinue;\n \n \t\tinclude_pack[i] = 1;\n-\t\tcount++;\n \t}\n-\n-\treturn count < 2;\n }\n \n-static int fill_included_packs_batch(struct repository *r,\n-\t\t\t\t     struct multi_pack_index *m,\n-\t\t\t\t     unsigned char *include_pack,\n-\t\t\t\t     size_t batch_size)\n+static void fill_included_packs_batch(struct repository *r,\n+\t\t\t\t      struct multi_pack_index *m,\n+\t\t\t\t      unsigned char *include_pack,\n+\t\t\t\t      size_t batch_size)\n {\n-\tuint32_t i, packs_to_repack;\n+\tuint32_t i;\n \tsize_t total_size;\n \tstruct repack_info *pack_info;\n \tint pack_kept_objects = 0;\n@@ -1418,7 +1415,6 @@ static int fill_included_packs_batch(struct repository *r,\n \tQSORT(pack_info, m->num_packs, compare_by_mtime);\n \n \ttotal_size = 0;\n-\tpacks_to_repack = 0;\n \tfor (i = 0; total_size < batch_size && i < m->num_packs; i++) {\n \t\tint pack_int_id = pack_info[i].pack_int_id;\n \t\tstruct packed_git *p = m->packs[pack_int_id];\n@@ -1434,23 +1430,17 @@ static int fill_included_packs_batch(struct repository *r,\n \t\tif (expected_size >= batch_size)\n \t\t\tcontinue;\n \n-\t\tpacks_to_repack++;\n \t\ttotal_size += expected_size;\n \t\tinclude_pack[pack_int_id] = 1;\n \t}\n \n \tfree(pack_info);\n-\n-\tif (packs_to_repack < 2)\n-\t\treturn 1;\n-\n-\treturn 0;\n }\n \n int midx_repack(struct repository *r, const char *object_dir, size_t batch_size, unsigned flags)\n {\n \tint result = 0;\n-\tuint32_t i;\n+\tuint32_t i, packs_to_repack = 0;\n \tunsigned char *include_pack;\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \tFILE *cmd_in;\n@@ -1469,10 +1459,16 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \n \tCALLOC_ARRAY(include_pack, m->num_packs);\n \n-\tif (batch_size) {\n-\t\tif (fill_included_packs_batch(r, m, include_pack, batch_size))\n-\t\t\tgoto cleanup;\n-\t} else if (fill_included_packs_all(r, m, include_pack))\n+\tif (batch_size)\n+\t\tfill_included_packs_batch(r, m, include_pack, batch_size);\n+\telse\n+\t\tfill_included_packs_all(r, m, include_pack);\n+\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tif (include_pack[i])\n+\t\t\tpacks_to_repack++;\n+\t}\n+\tif (packs_to_repack <= 1)\n \t\tgoto cleanup;\n \n \trepo_config_get_bool(r, \"repack.usedeltabaseoffset\", &delta_base_offset);\n-- \n2.44.0.330.g158d2a670b4\n\n"},{"id":"491991","messageId":"b5d6ba5802aef6ddf1542f1b0efffe43c22436ba.1712006190.git.me@ttaylorr.com","threadId":"61199","inReplyTo":"cover.1712006190.git.me@ttaylorr.com","subject":"[PATCH v2 4/4] midx-write.c: use `--stdin-packs` when repacking","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-01T21:16:44Z","receivedAt":"2024-04-01T21:16:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When constructing a new pack `git multi-pack-index repack` provides a\nlist of objects which is the union of objects in all MIDX'd packs which\nwere \"included\" in the repack.\n\nThough correct, this typically yields a poorly structured pack, since\nproviding the objects list over stdin does not give pack-objects a\nchance to discover the namehash values for each object, leading to\nsub-optimal delta selection.\n\nWe can use `--stdin-packs` instead, which has a couple of benefits:\n\n  - it does a supplemental walk over objects in the supplied list of\n    packs to discover their namehash, leading to higher-quality delta\n    selection\n\n  - it requires us to list far less data over stdin; instead of listing\n    each object in the resulting pack, we need only list the\n    constituent packs from which those objects were selected in the MIDX\n\nOf course, this comes at a slight cost: though we save time on listing\npacks versus objects over stdin[^1] (around ~650 milliseconds), we add a\nnon-trivial amount of time walking over the given objects in order to\nfind better deltas.\n\nIn general, this is likely to more closely match the user's expectations\n(i.e. that packs generated via `git multi-pack-index repack` are written\nwith high-quality deltas). But if not, we can always introduce a new\noption in pack-objects to disable the supplemental object walk, which\nwould yield a pure CPU-time savings, at the cost of the on-disk size of\nthe resulting pack.\n\n[^1]: In a patched version of Git that doesn't perform the supplemental\n  object walk in `pack-objects --stdin-packs`, we save around ~650ms\n  (from 5.968 to 5.325 seconds) when running `git multi-pack-index\n  repack --batch-size=0` on git.git with all objects packed, and all\n  packs in a MIDX.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n midx-write.c | 18 +++++++++---------\n 1 file changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/midx-write.c b/midx-write.c\nindex 960cc46250..65e69d2de7 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -1474,7 +1474,8 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \trepo_config_get_bool(r, \"repack.usedeltabaseoffset\", &delta_base_offset);\n \trepo_config_get_bool(r, \"repack.usedeltaislands\", &use_delta_islands);\n \n-\tstrvec_push(&cmd.args, \"pack-objects\");\n+\tstrvec_pushl(&cmd.args, \"pack-objects\", \"--stdin-packs\", \"--non-empty\",\n+\t\t     NULL);\n \n \tstrvec_pushf(&cmd.args, \"%s/pack/pack\", object_dir);\n \n@@ -1498,16 +1499,15 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t}\n \n \tcmd_in = xfdopen(cmd.in, \"w\");\n-\n-\tfor (i = 0; i < m->num_objects; i++) {\n-\t\tstruct object_id oid;\n-\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n-\n-\t\tif (!include_pack[pack_int_id])\n+\tfor (i = 0; i < m->num_packs; i++) {\n+\t\tstruct packed_git *p = m->packs[i];\n+\t\tif (!p)\n \t\t\tcontinue;\n \n-\t\tnth_midxed_object_oid(&oid, m, i);\n-\t\tfprintf(cmd_in, \"%s\\n\", oid_to_hex(&oid));\n+\t\tif (include_pack[i])\n+\t\t\tfprintf(cmd_in, \"%s\\n\", pack_basename(p));\n+\t\telse\n+\t\t\tfprintf(cmd_in, \"^%s\\n\", pack_basename(p));\n \t}\n \tfclose(cmd_in);\n \n-- \n2.44.0.330.g158d2a670b4\n"},{"id":"491996","messageId":"xmqqbk6s92p4.fsf@gitster.g","threadId":"61199","inReplyTo":"b5d6ba5802aef6ddf1542f1b0efffe43c22436ba.1712006190.git.me@ttaylorr.com","subject":"Re: [PATCH v2 4/4] midx-write.c: use `--stdin-packs` when repacking","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-01T21:45:59Z","receivedAt":"2024-04-01T21:46:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> When constructing a new pack `git multi-pack-index repack` provides a\n> list of objects which is the union of objects in all MIDX'd packs which\n> were \"included\" in the repack.\n>\n> Though correct, this typically yields a poorly structured pack, since\n> providing the objects list over stdin does not give pack-objects a\n> chance to discover the namehash values for each object, leading to\n> sub-optimal delta selection.\n>\n> We can use `--stdin-packs` instead, which has a couple of benefits:\n>\n>   - it does a supplemental walk over objects in the supplied list of\n>     packs to discover their namehash, leading to higher-quality delta\n>     selection\n>\n>   - it requires us to list far less data over stdin; instead of listing\n>     each object in the resulting pack, we need only list the\n>     constituent packs from which those objects were selected in the MIDX\n>\n> Of course, this comes at a slight cost: though we save time on listing\n> packs versus objects over stdin[^1] (around ~650 milliseconds), we add a\n> non-trivial amount of time walking over the given objects in order to\n> find better deltas.\n>\n> In general, this is likely to more closely match the user's expectations\n> (i.e. that packs generated via `git multi-pack-index repack` are written\n> with high-quality deltas). But if not, we can always introduce a new\n> option in pack-objects to disable the supplemental object walk, which\n> would yield a pure CPU-time savings, at the cost of the on-disk size of\n> the resulting pack.\n>\n> [^1]: In a patched version of Git that doesn't perform the supplemental\n>   object walk in `pack-objects --stdin-packs`, we save around ~650ms\n>   (from 5.968 to 5.325 seconds) when running `git multi-pack-index\n>   repack --batch-size=0` on git.git with all objects packed, and all\n>   packs in a MIDX.\n\nThere are some measures in the mind of readers' who have read the\nexplanation so far.\n\n - So, this gives us a resulting pack with better delta selection.\n   How much better would it get in a sample repository?  10%?  40%?\n\n - Of course, the better delta selection comes with cost.  How much\n   more time do we spend?  20%?  150%?\n\n - As we do not enumerate all the object names, we save some time.\n   Around 0.65 seconds in a sample repository.\n\nI think among the three, the first two are more interesting numbers,\nno?\n\nI wonder if we can leverage the trick that reuses existing packdata\nwhen we stream packs to feed the \"git fetch\" clients---we rely on\nthe fact that existing packs are tightly packed with good delta\nselection, and using bitmap stream contiguous section(s) as much as\npossible without disturbing the existing delta chain.  Wouldn't the\n\"we have many packs, let's repack them into one\" workload benefit\nthe same way?\n\n> -\tstrvec_push(&cmd.args, \"pack-objects\");\n> +\tstrvec_pushl(&cmd.args, \"pack-objects\", \"--stdin-packs\", \"--non-empty\",\n> +\t\t     NULL);\n>  \n>  \tstrvec_pushf(&cmd.args, \"%s/pack/pack\", object_dir);\n>  \n> @@ -1498,16 +1499,15 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n>  \t}\n>  \n>  \tcmd_in = xfdopen(cmd.in, \"w\");\n> -\n> -\tfor (i = 0; i < m->num_objects; i++) {\n> -\t\tstruct object_id oid;\n> -\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n> -\n> -\t\tif (!include_pack[pack_int_id])\n> +\tfor (i = 0; i < m->num_packs; i++) {\n> +\t\tstruct packed_git *p = m->packs[i];\n> +\t\tif (!p)\n>  \t\t\tcontinue;\n>  \n> -\t\tnth_midxed_object_oid(&oid, m, i);\n> -\t\tfprintf(cmd_in, \"%s\\n\", oid_to_hex(&oid));\n> +\t\tif (include_pack[i])\n> +\t\t\tfprintf(cmd_in, \"%s\\n\", pack_basename(p));\n> +\t\telse\n> +\t\t\tfprintf(cmd_in, \"^%s\\n\", pack_basename(p));\n>  \t}\n>  \tfclose(cmd_in);\n\nLooking very straight-forward.  Will queue.\n\nThanks.\n"},{"id":"492044","messageId":"ZgvwWlvrw8MpC_AZ@tanuki","threadId":"61199","inReplyTo":"0064e363c0cc3288346585a6b4340444ce7b863c.1712006190.git.me@ttaylorr.com","subject":"Re: [PATCH v2 2/4] midx-write.c: factor out common want_included_pack() routine","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-02T11:47:38Z","receivedAt":"2024-04-02T11:47:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 01, 2024 at 05:16:38PM -0400, Taylor Blau wrote:\n> When performing a 'git multi-pack-index repack', the MIDX machinery\n> tries to aggregate MIDX'd packs together either to (a) fill the given\n> `--batch-size` argument, or (b) combine all packs together.\n> \n> In either case (using the `midx-write.c::fill_included_packs_batch()` or\n> `midx-write.c::fill_included_packs_all()` function, respectively), we\n> evaluate whether or not we want to repack each MIDX'd pack, according to\n> whether or it is loadable, kept, cruft, or non-empty.\n> \n> Between the two `fill_included_packs_` callers, they both care about the\n> same conditions, except for `fill_included_packs_batch()` which also\n> cares that the pack is non-empty.\n> \n> We could extract two functions (say, `want_included_pack()` and a\n> `_nonempty()` variant), but this is not necessary. For the case in\n> `fill_included_packs_all()` which does not check the pack size, we add\n> all of the pack's objects assuming that the pack meets all other\n> criteria. But if the pack is empty in the first place, we add all of its\n> zero objects, so whether or not we \"accept\" or \"reject\" it in the first\n> place is irrelevant.\n> \n> This change improves the readability in both `fill_included_packs_`\n> functions.\n> \n> Signed-off-by: Taylor Blau <me@ttaylorr.com>\n> ---\n>  midx-write.c | 32 ++++++++++++++++++++------------\n>  1 file changed, 20 insertions(+), 12 deletions(-)\n> \n> diff --git a/midx-write.c b/midx-write.c\n> index 5242d2a724..906efa2169 100644\n> --- a/midx-write.c\n> +++ b/midx-write.c\n> @@ -1349,6 +1349,24 @@ static int compare_by_mtime(const void *a_, const void *b_)\n>  \treturn 0;\n>  }\n>  \n> +static int want_included_pack(struct repository *r,\n> +\t\t\t      struct multi_pack_index *m,\n> +\t\t\t      int pack_kept_objects,\n> +\t\t\t      uint32_t pack_int_id)\n> +{\n> +\tstruct packed_git *p;\n> +\tif (prepare_midx_pack(r, m, pack_int_id))\n> +\t\treturn 0;\n> +\tp = m->packs[pack_int_id];\n> +\tif (!pack_kept_objects && p->pack_keep)\n> +\t\treturn 0;\n> +\tif (p->is_cruft)\n> +\t\treturn 0;\n> +\tif (open_pack_index(p) || !p->num_objects)\n> +\t\treturn 0;\n> +\treturn 1;\n> +}\n> +\n>  static int fill_included_packs_all(struct repository *r,\n>  \t\t\t\t   struct multi_pack_index *m,\n>  \t\t\t\t   unsigned char *include_pack)\n> @@ -1359,11 +1377,7 @@ static int fill_included_packs_all(struct repository *r,\n>  \trepo_config_get_bool(r, \"repack.packkeptobjects\", &pack_kept_objects);\n>  \n>  \tfor (i = 0; i < m->num_packs; i++) {\n> -\t\tif (prepare_midx_pack(r, m, i))\n> -\t\t\tcontinue;\n> -\t\tif (!pack_kept_objects && m->packs[i]->pack_keep)\n> -\t\t\tcontinue;\n> -\t\tif (m->packs[i]->is_cruft)\n> +\t\tif (!want_included_pack(r, m, pack_kept_objects, i))\n>  \t\t\tcontinue;\n>  \n>  \t\tinclude_pack[i] = 1;\n> @@ -1410,13 +1424,7 @@ static int fill_included_packs_batch(struct repository *r,\n>  \t\tstruct packed_git *p = m->packs[pack_int_id];\n>  \t\tsize_t expected_size;\n\nI was briefly wondering whether `m->packs[pack_int_id]` could change\nafter the above assignment. But there's a loop a bit further up here\nthat already calls `prepare_midx_pack()`, so this shouldn't happen.\n\n> -\t\tif (!p)\n> -\t\t\tcontinue;\n> -\t\tif (!pack_kept_objects && p->pack_keep)\n> -\t\t\tcontinue;\n> -\t\tif (p->is_cruft)\n> -\t\t\tcontinue;\n> -\t\tif (open_pack_index(p) || !p->num_objects)\n> +\t\tif (!want_included_pack(r, m, pack_kept_objects, pack_int_id))\n>  \t\t\tcontinue;\n\nAnother thing I wondered was whether it hurts performance that we now\ncall `prepare_midx_pack()` twice. But this shouldn't matter either as we\nexit early from that function in case `m->packs[pack_int_id]` is already\npopulated.\n\nSo this patch looks good to me.\n\nPatrick\n\n>  \t\texpected_size = st_mult(p->pack_size,\n> -- \n> 2.44.0.330.g158d2a670b4\n> \n> \n"},{"id":"492045","messageId":"ZgvwXyMMUKxfP9ni@tanuki","threadId":"61199","inReplyTo":"b5d6ba5802aef6ddf1542f1b0efffe43c22436ba.1712006190.git.me@ttaylorr.com","subject":"Re: [PATCH v2 4/4] midx-write.c: use `--stdin-packs` when repacking","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-02T11:47:43Z","receivedAt":"2024-04-02T11:47:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 01, 2024 at 05:16:44PM -0400, Taylor Blau wrote:\n[snip]\n> @@ -1498,16 +1499,15 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n>  \t}\n>  \n>  \tcmd_in = xfdopen(cmd.in, \"w\");\n> -\n> -\tfor (i = 0; i < m->num_objects; i++) {\n> -\t\tstruct object_id oid;\n> -\t\tuint32_t pack_int_id = nth_midxed_pack_int_id(m, i);\n> -\n> -\t\tif (!include_pack[pack_int_id])\n> +\tfor (i = 0; i < m->num_packs; i++) {\n> +\t\tstruct packed_git *p = m->packs[i];\n> +\t\tif (!p)\n>  \t\t\tcontinue;\n>  \n> -\t\tnth_midxed_object_oid(&oid, m, i);\n> -\t\tfprintf(cmd_in, \"%s\\n\", oid_to_hex(&oid));\n> +\t\tif (include_pack[i])\n> +\t\t\tfprintf(cmd_in, \"%s\\n\", pack_basename(p));\n> +\t\telse\n> +\t\t\tfprintf(cmd_in, \"^%s\\n\", pack_basename(p));\n>  \t}\n\nThe patch looks straight-forward to me overall, but this last part\nconfused me a bit. Why do we explicitly exclude those packs instead of\nmerely skipping over them? Is this so that we don't repack objects which\nare part of both included and not-included packs?\n\nI was mostly wondering whether it can ever happen here that we decide to\nnot include a pack because all of its objects are already contained in\nother packs. In that case, explicitly excluding it would lead to repo\ncorruption if we deleted such a not-included pack.\n\nPatrick\n"}]}