{"thread":{"id":"55135","subject":"[PATCH 0/9] midx: implement a multi-pack reverse index","startedAt":"2021-02-10T23:03:21Z","lastAt":"2021-09-18T15:58:13Z","messageCount":171,"participants":["Taylor Blau","Derrick Stolee","Junio C Hamano","SZEDER Gábor","Ævar Arnfjörð Bjarmason","Jonathan Tan","Jeff King","Carlo Arenas"],"isPatch":true,"patchVersion":1,"patchTotal":9},"messages":[{"id":"416671","messageId":"cover.1612998106.git.me@ttaylorr.com","threadId":"55135","inReplyTo":null,"subject":"[PATCH 0/9] midx: implement a multi-pack reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-10T23:02:18Z","receivedAt":"2021-02-10T23:03:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This series describes and implements a reverse index for the multi-pack index,\nbased on a \"pseudo-pack\" which can be uniquely described by the multi-pack\nindex.\n\nThe details of the pseudo-pack, and multi-pack reverse index are laid out in\ndetail in the sixth patch.\n\nThis is in support of multi-pack reachability bitmaps, which contain objects\nfrom the multi-pack index. Likewise, an object's bit position in a multi-pack\nreachability bitmap is determined by its position with that multi-pack index's\npseudo pack.\n\nIn this series, there are no users of the multi-pack index, so this series is\nmainly about laying the groundwork for implementing multi-pack bitmaps. This\nseries is the final prerequisite needed before we can implement multi-pack\nbitmaps, which will come in the next series[1].\n\nSince tb/pack-revindex-on-disk is queued to be merged to 'master', but hasn't\nyet been merged, this series is based on that branch.\n\nThanks in advance for your review of this series, and all of the many other\nseries in support of multi-pack bitmaps.\n\n[1]: If you're curious, you can find the patches in the tb/multi-pack-bitmaps\nbranch of my fork at https://github.com/ttaylorr/git.\n\nTaylor Blau (9):\n t/helper/test-read-midx.c: add '--show-objects'\n midx: allow marking a pack as preferred\n midx: don't free midx_name early\n midx: keep track of the checksum\n midx: make some functions non-static\n Documentation/technical: describe multi-pack reverse indexes\n pack-revindex: read multi-pack reverse indexes\n pack-write.c: extract 'write_rev_file_order'\n pack-revindex: write multi-pack reverse indexes\n\n Documentation/git-multi-pack-index.txt | 11 +-\n Documentation/technical/multi-pack-index.txt | 5 +-\n Documentation/technical/pack-format.txt | 83 +++++++\n builtin/multi-pack-index.c | 10 +-\n builtin/repack.c | 2 +-\n midx.c | 239 ++++++++++++++++++-\n midx.h | 11 +-\n pack-revindex.c | 112 +++++++++\n pack-revindex.h | 46 ++++\n pack-write.c | 39 ++-\n pack.h | 1 +\n packfile.c | 3 +\n t/helper/test-read-midx.c | 24 +-\n t/t5319-multi-pack-index.sh | 39 +++\n 14 files changed, 591 insertions(+), 34 deletions(-)\n\n-- \n2.30.0.667.g81c0cbc6fd\n"},{"id":"416672","messageId":"e36acb005d3563db772711207d2af3df03685bd4.1612998106.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1612998106.git.me@ttaylorr.com","subject":"[PATCH 1/9] t/helper/test-read-midx.c: add '--show-objects'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-10T23:02:29Z","receivedAt":"2021-02-10T23:03:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The 'read-midx' helper is used in places like t5319 to display basic\ninformation about a multi-pack-index.\n\nIn the next patch, the MIDX writing machinery will learn a new way to\nchoose from which pack an object is selected when multiple copies of\nthat object exist.\n\nTo disambiguate which pack introduces an object so that this feature can\nbe tested, add a '--show-objects' option which displays additional\ninformation about each object in the MIDX.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-midx.c | 24 ++++++++++++++++++++----\n 1 file changed, 20 insertions(+), 4 deletions(-)\n\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 2430880f78..4ec12f77a0 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -4,7 +4,7 @@\n #include \"repository.h\"\n #include \"object-store.h\"\n \n-static int read_midx_file(const char *object_dir)\n+static int read_midx_file(const char *object_dir, int show_objects)\n {\n \tuint32_t i;\n \tstruct multi_pack_index *m;\n@@ -15,6 +15,20 @@ static int read_midx_file(const char *object_dir)\n \tif (!m)\n \t\treturn 1;\n \n+\tif (show_objects) {\n+\t\tstruct object_id oid;\n+\t\tstruct pack_entry e;\n+\n+\t\tfor (i = 0; i < m->num_objects; i++) {\n+\t\t\tnth_midxed_object_oid(&oid, m, i);\n+\t\t\tfill_midx_entry(the_repository, &oid, &e, m);\n+\n+\t\t\tprintf(\"%s %\"PRIu64\"\\t%s\\n\",\n+\t\t\t oid_to_hex(&oid), e.offset, e.p->pack_name);\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t m->signature,\n \t m->version,\n@@ -48,8 +62,10 @@ static int read_midx_file(const char *object_dir)\n \n int cmd__read_midx(int argc, const char **argv)\n {\n-\tif (argc != 2)\n-\t\tusage(\"read-midx \");\n+\tif (!(argc == 2 || argc == 3))\n+\t\tusage(\"read-midx [--show-objects] \");\n \n-\treturn read_midx_file(argv[1]);\n+\tif (!strcmp(argv[1], \"--show-objects\"))\n+\t\treturn read_midx_file(argv[2], 1);\n+\treturn read_midx_file(argv[1], 0);\n }\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"416673","messageId":"4a358d57cf6e834cd1756e70bf713d4d104f321e.1612998106.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1612998106.git.me@ttaylorr.com","subject":"[PATCH 2/9] midx: allow marking a pack as preferred","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-10T23:02:42Z","receivedAt":"2021-02-10T23:03:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When multiple packs in the multi-pack index contain the same object, the\nMIDX machinery must make a choice about which pack it associates with\nthat object. Prior to this patch, the lowest-ordered[1] pack was always\nselected.\n\nPack selection for duplicate objects is relatively unimportant today,\nbut it will become important for multi-pack bitmaps. This is because we\ncan only invoke the pack-reuse mechanism when all of the bits for reused\nobjects come from the reuse pack (in order to ensure that all reused\ndeltas can find their base objects in the same pack).\n\nTo encourage the pack selection process to prefer one pack over another\n(the pack to be preferred is the one a caller would like to later use as\na reuse pack), introduce the concept of a \"preferred pack\". When\nprovided, the MIDX code will always prefer an object found in a\npreferred pack over any other.\n\nNo format changes are required to store the preferred pack, since it\nwill be able to be inferred with a corresponding MIDX bitmap, by looking\nup the pack associated with the object in the first bit position (this\nordering is described in detail in a subsequent commit).\n\n[1]: the ordering is specified by MIDX internals; for our purposes we\ncan consider the \"lowest ordered\" pack to be \"the one with the\nmost-recent mtime.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 11 ++-\n Documentation/technical/multi-pack-index.txt | 5 +-\n builtin/multi-pack-index.c | 10 +-\n builtin/repack.c | 2 +-\n midx.c | 97 ++++++++++++++++++--\n midx.h | 2 +-\n t/t5319-multi-pack-index.sh | 39 ++++++++\n 7 files changed, 151 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex eb0caa0439..dd14eab781 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -9,7 +9,8 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n SYNOPSIS\n --------\n [verse]\n-'git multi-pack-index' [--object-dir=] [--[no-]progress] \n+'git multi-pack-index' [--object-dir=] [--[no-]progress]\n+\t[--preferred-pack=] \n \n DESCRIPTION\n -----------\n@@ -27,6 +28,14 @@ OPTIONS\n \tTurn progress on/off explicitly. If neither is specified, progress is\n \tshown if standard error is connected to a terminal.\n \n+--preferred-pack=::\n+\tWhen using the `write` subcommand, optionally specify the\n+\ttie-breaking pack used when multiple packs contain the same\n+\tobject. Incompatible with other subcommands, including `repack`,\n+\twhich may repack the pack marked as preferred. If not given, the\n+\tpreferred pack is inferred from an existing `multi-pack-index`,\n+\tif one exists, otherwise the pack with the lowest mtime.\n+\n The following subcommands are available:\n \n write::\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex e8e377a59f..fb688976c4 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -43,8 +43,9 @@ Design Details\n a change in format.\n \n - The MIDX keeps only one record per object ID. If an object appears\n- in multiple packfiles, then the MIDX selects the copy in the most-\n- recently modified packfile.\n+ in multiple packfiles, then the MIDX selects the copy in the\n+ preferred packfile, otherwise selecting from the most-recently\n+ modified packfile.\n \n - If there exist packfiles in the pack directory not registered in\n the MIDX, then those packfiles are loaded into the `packed_git`\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5bf88cd2a8..4d1ea3fe84 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -4,6 +4,7 @@\n #include \"parse-options.h\"\n #include \"midx.h\"\n #include \"trace2.h\"\n+#include \"object-store.h\"\n \n static char const * const builtin_multi_pack_index_usage[] = {\n \tN_(\"git multi-pack-index [] (write|verify|expire|repack --batch-size=)\"),\n@@ -12,6 +13,7 @@ static char const * const builtin_multi_pack_index_usage[] = {\n \n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n+\tconst char *preferred_pack;\n \tunsigned long batch_size;\n \tint progress;\n } opts;\n@@ -24,6 +26,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tstatic struct option builtin_multi_pack_index_options[] = {\n \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n+\t\tOPT_STRING(0, \"preferred-pack\", &opts.preferred_pack, N_(\"preferred-pack\"),\n+\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n \t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n \t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n@@ -51,6 +55,9 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\treturn 1;\n \t}\n \n+\tif (strcmp(argv[0], \"write\") && opts.preferred_pack)\n+\t\tdie(_(\"'--preferred-pack' requires 'write'\"));\n+\n \ttrace2_cmd_mode(argv[0]);\n \n \tif (!strcmp(argv[0], \"repack\"))\n@@ -60,7 +67,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n \n \tif (!strcmp(argv[0], \"write\"))\n-\t\treturn write_midx_file(opts.object_dir, flags);\n+\t\treturn write_midx_file(opts.object_dir, opts.preferred_pack,\n+\t\t\t\t flags);\n \tif (!strcmp(argv[0], \"verify\"))\n \t\treturn verify_midx_file(the_repository, opts.object_dir, flags);\n \tif (!strcmp(argv[0], \"expire\"))\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 01440de2d5..9f00806805 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -523,7 +523,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tremove_temporary_files();\n \n \tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n-\t\twrite_midx_file(get_object_directory(), 0);\n+\t\twrite_midx_file(get_object_directory(), NULL, 0);\n \n \tstring_list_clear(&names, 0);\n \tstring_list_clear(&rollback, 0);\ndiff --git a/midx.c b/midx.c\nindex 05c40a98e0..064670c0c0 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -451,6 +451,24 @@ static int pack_info_compare(const void *_a, const void *_b)\n \treturn strcmp(a->pack_name, b->pack_name);\n }\n \n+static int lookup_idx_or_pack_name(struct pack_info *info,\n+\t\t\t\t uint32_t nr,\n+\t\t\t\t const char *pack_name)\n+{\n+\tuint32_t lo = 0, hi = nr;\n+\twhile (lo < hi) {\n+\t\tuint32_t mi = lo + (hi - lo) / 2;\n+\t\tint cmp = cmp_idx_or_pack_name(pack_name, info[mi].pack_name);\n+\t\tif (cmp < 0)\n+\t\t\thi = mi;\n+\t\telse if (cmp > 0)\n+\t\t\tlo = mi + 1;\n+\t\telse\n+\t\t\treturn mi;\n+\t}\n+\treturn -1;\n+}\n+\n struct pack_list {\n \tstruct pack_info *info;\n \tuint32_t nr;\n@@ -502,6 +520,7 @@ struct pack_midx_entry {\n \tuint32_t pack_int_id;\n \ttime_t pack_mtime;\n \tuint64_t offset;\n+\tunsigned preferred : 1;\n };\n \n static int midx_oid_compare(const void *_a, const void *_b)\n@@ -513,6 +532,12 @@ static int midx_oid_compare(const void *_a, const void *_b)\n \tif (cmp)\n \t\treturn cmp;\n \n+\t/* Sort objects in a preferred pack first when multiple copies exist. */\n+\tif (a->preferred > b->preferred)\n+\t\treturn -1;\n+\tif (a->preferred < b->preferred)\n+\t\treturn 1;\n+\n \tif (a->pack_mtime > b->pack_mtime)\n \t\treturn -1;\n \telse if (a->pack_mtime < b->pack_mtime)\n@@ -540,7 +565,8 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n static void fill_pack_entry(uint32_t pack_int_id,\n \t\t\t struct packed_git *p,\n \t\t\t uint32_t cur_object,\n-\t\t\t struct pack_midx_entry *entry)\n+\t\t\t struct pack_midx_entry *entry,\n+\t\t\t int preferred)\n {\n \tif (nth_packed_object_id(&entry->oid, p, cur_object) < 0)\n \t\tdie(_(\"failed to locate object %d in packfile\"), cur_object);\n@@ -549,6 +575,7 @@ static void fill_pack_entry(uint32_t pack_int_id,\n \tentry->pack_mtime = p->mtime;\n \n \tentry->offset = nth_packed_object_offset(p, cur_object);\n+\tentry->preferred = !!preferred;\n }\n \n /*\n@@ -565,7 +592,8 @@ static void fill_pack_entry(uint32_t pack_int_id,\n static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \t\t\t\t\t\t struct pack_info *info,\n \t\t\t\t\t\t uint32_t nr_packs,\n-\t\t\t\t\t\t uint32_t *nr_objects)\n+\t\t\t\t\t\t uint32_t *nr_objects,\n+\t\t\t\t\t\t uint32_t preferred_pack)\n {\n \tuint32_t cur_fanout, cur_pack, cur_object;\n \tuint32_t alloc_fanout, alloc_objects, total_objects = 0;\n@@ -602,12 +630,17 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \t\t\t\tnth_midxed_pack_midx_entry(m,\n \t\t\t\t\t\t\t &entries_by_fanout[nr_fanout],\n \t\t\t\t\t\t\t cur_object);\n+\t\t\t\tif (nth_midxed_pack_int_id(m, cur_object) == preferred_pack)\n+\t\t\t\t\tentries_by_fanout[nr_fanout].preferred = 1;\n+\t\t\t\telse\n+\t\t\t\t\tentries_by_fanout[nr_fanout].preferred = 0;\n \t\t\t\tnr_fanout++;\n \t\t\t}\n \t\t}\n \n \t\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) {\n \t\t\tuint32_t start = 0, end;\n+\t\t\tint preferred = cur_pack == preferred_pack;\n \n \t\t\tif (cur_fanout)\n \t\t\t\tstart = get_pack_fanout(info[cur_pack].p, cur_fanout - 1);\n@@ -615,7 +648,11 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \n \t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n \t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n-\t\t\t\tfill_pack_entry(cur_pack, info[cur_pack].p, cur_object, &entries_by_fanout[nr_fanout]);\n+\t\t\t\tfill_pack_entry(cur_pack,\n+\t\t\t\t\t\tinfo[cur_pack].p,\n+\t\t\t\t\t\tcur_object,\n+\t\t\t\t\t\t&entries_by_fanout[nr_fanout],\n+\t\t\t\t\t\tpreferred);\n \t\t\t\tnr_fanout++;\n \t\t\t}\n \t\t}\n@@ -794,7 +831,9 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off\n }\n \n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n-\t\t\t struct string_list *packs_to_drop, unsigned flags)\n+\t\t\t struct string_list *packs_to_drop,\n+\t\t\t const char *preferred_pack_name,\n+\t\t\t unsigned flags)\n {\n \tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n@@ -813,6 +852,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tint pack_name_concat_len = 0;\n \tint dropped_packs = 0;\n \tint result = 0;\n+\tint preferred_pack_idx = -1;\n \n \tmidx_name = get_midx_filename(object_dir);\n \tif (safe_create_leading_directories(midx_name))\n@@ -853,7 +893,18 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n-\tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);\n+\tif (preferred_pack_name) {\n+\t\tfor (i = 0; i < packs.nr; i++) {\n+\t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n+\t\t\t\t\t\t packs.info[i].pack_name)) {\n+\t\t\t\tpreferred_pack_idx = i;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries,\n+\t\t\t\t preferred_pack_idx);\n \n \tfor (i = 0; i < nr_entries; i++) {\n \t\tif (entries[i].offset > 0x7fffffff)\n@@ -913,6 +964,31 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n \t}\n \n+\t/*\n+\t * Recompute the preferred_pack_idx (if applicable) according to the\n+\t * permuted pack order.\n+\t */\n+\tpreferred_pack_idx = -1;\n+\tif (preferred_pack_name) {\n+\t\tpreferred_pack_idx = lookup_idx_or_pack_name(packs.info,\n+\t\t\t\t\t\t\t packs.nr,\n+\t\t\t\t\t\t\t preferred_pack_name);\n+\t\tif (preferred_pack_idx < 0)\n+\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n+\t\t\t\tpreferred_pack_name);\n+\t\telse {\n+\t\t\tuint32_t orig = packs.info[preferred_pack_idx].orig_pack_int_id;\n+\t\t\tuint32_t perm = pack_perm[orig];\n+\n+\t\t\tif (perm == PACK_EXPIRED) {\n+\t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n+\t\t\t\t\tpreferred_pack_name);\n+\t\t\t\tpreferred_pack_idx = -1;\n+\t\t\t} else\n+\t\t\t\tpreferred_pack_idx = perm;\n+\t\t}\n+\t}\n+\n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n \t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n@@ -1042,9 +1118,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \treturn result;\n }\n \n-int write_midx_file(const char *object_dir, unsigned flags)\n+int write_midx_file(const char *object_dir,\n+\t\t const char *preferred_pack_name,\n+\t\t unsigned flags)\n {\n-\treturn write_midx_internal(object_dir, NULL, NULL, flags);\n+\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n+\t\t\t\t flags);\n }\n \n void clear_midx_file(struct repository *r)\n@@ -1279,7 +1358,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tfree(count);\n \n \tif (packs_to_drop.nr)\n-\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop, flags);\n+\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop, NULL, flags);\n \n \tstring_list_clear(&packs_to_drop, 0);\n \treturn result;\n@@ -1468,7 +1547,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t\tgoto cleanup;\n \t}\n \n-\tresult = write_midx_internal(object_dir, m, NULL, flags);\n+\tresult = write_midx_internal(object_dir, m, NULL, NULL, flags);\n \tm = NULL;\n \n cleanup:\ndiff --git a/midx.h b/midx.h\nindex b18cf53bc4..e7fea61109 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -47,7 +47,7 @@ int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pa\n int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n \n-int write_midx_file(const char *object_dir, unsigned flags);\n+int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);\n int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 2fc3aadbd1..9304b33484 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -31,6 +31,14 @@ midx_read_expect () {\n \ttest_cmp expect actual\n }\n \n+midx_expect_object_offset () {\n+\tOID=\"$1\"\n+\tOFFSET=\"$2\"\n+\tOBJECT_DIR=\"$3\"\n+\ttest-tool read-midx --show-objects $OBJECT_DIR >actual &&\n+\tgrep \"^$OID $OFFSET\" actual\n+}\n+\n test_expect_success 'setup' '\n \ttest_oid_cache <<-EOF\n \tidxoff sha1:2999\n@@ -234,6 +242,37 @@ test_expect_success 'warn on improper hash version' '\n \t)\n '\n \n+test_expect_success 'midx picks objects from preferred pack' '\n+\ttest_when_finished rm -rf preferred.git &&\n+\tgit init --bare preferred.git &&\n+\t(\n+\t\tcd preferred.git &&\n+\n+\t\ta=$(echo \"a\" | git hash-object -w --stdin) &&\n+\t\tb=$(echo \"b\" | git hash-object -w --stdin) &&\n+\t\tc=$(echo \"c\" | git hash-object -w --stdin) &&\n+\n+\t\t# Set up two packs, duplicating the object \"B\" at different\n+\t\t# offsets.\n+\t\tgit pack-objects objects/pack/test-AB <<-EOF &&\n+\t\t$a\n+\t\t$b\n+\t\tEOF\n+\t\tbc=$(git pack-objects objects/pack/test-BC <<-EOF\n+\t\t$b\n+\t\t$c\n+\t\tEOF\n+\t\t) &&\n+\n+\t\tgit multi-pack-index --object-dir=objects \\\n+\t\t\t--preferred-pack=test-BC-$bc.idx write 2>err &&\n+\t\ttest_must_be_empty err &&\n+\n+\t\tofs=$(git show-index \n---\n midx.c | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 064670c0c0..34fb9de3f3 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -995,7 +995,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n-\tFREE_AND_NULL(midx_name);\n \n \tif (packs.m)\n \t\tclose_midx(packs.m);\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"416675","messageId":"b4b842fa1e8c711af30cc93b6f1100661791416c.1612998106.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1612998106.git.me@ttaylorr.com","subject":"[PATCH 4/9] midx: keep track of the checksum","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-10T23:02:55Z","receivedAt":"2021-02-10T23:03:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"write_midx_internal() uses a hashfile to write the multi-pack index, but\ndiscards its checksum. This makes sense, since nothing that takes place\nafter writing the MIDX cares about its checksum.\n\nThat is about to change in a subsequent patch, when the optional\nreverse index corresponding to the MIDX will want to include the MIDX's\nchecksum.\n\nStore the checksum of the MIDX in preparation for that.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 34fb9de3f3..6e47c726af 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -837,6 +837,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n {\n \tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n+\tunsigned char midx_hash[GIT_MAX_RAWSZ];\n \tuint32_t i;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n@@ -1098,7 +1099,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t written,\n \t\t chunk_offsets[num_chunks]);\n \n-\tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n+\tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tcommit_lock_file(&lk);\n \n cleanup:\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"416676","messageId":"953beabaa493eb0869b0b1f1f24c05067e333f8d.1612998106.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1612998106.git.me@ttaylorr.com","subject":"[PATCH 5/9] midx: make some functions non-static","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-10T23:03:05Z","receivedAt":"2021-02-10T23:04:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a subsequent commit, pack-revindex.c will become responsible for\nsorting a list of objects in the \"MIDX pack order\" (which will be\ndefined in the following patch). To do so, it will need to be know the\npack identifier and offset within that pack for each object in the MIDX.\n\nThe MIDX code already has functions for doing just that\n(nth_midxed_offset() and nth_midxed_pack_int_id()), but they are\nstatically declared.\n\nSince there is no reason that they couldn't be exposed publicly, and\nbecause they are already doing exactly what the caller in\npack-revindex.c will want, expose them publicly so that they can be\nreused there.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 4 ++--\n midx.h | 2 ++\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 6e47c726af..bf258c4fde 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -260,7 +260,7 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \treturn oid;\n }\n \n-static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n+off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n {\n \tconst unsigned char *offset_data;\n \tuint32_t offset32;\n@@ -279,7 +279,7 @@ static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n \treturn offset32;\n }\n \n-static uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n+uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n {\n \treturn get_be32(m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH);\n }\ndiff --git a/midx.h b/midx.h\nindex e7fea61109..93bd68189e 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -40,6 +40,8 @@ struct multi_pack_index {\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n+off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);\n+uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);\n struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct multi_pack_index *m,\n \t\t\t\t\tuint32_t n);\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"416677","messageId":"e64504bad6e181522946a8f234e12f569bede89e.1612998106.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1612998106.git.me@ttaylorr.com","subject":"[PATCH 6/9] Documentation/technical: describe multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-10T23:03:11Z","receivedAt":"2021-02-10T23:04:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"As a prerequisite to implementing multi-pack bitmaps, motivate and\ndescribe the format and ordering of the multi-pack reverse index.\n\nThe subsequent patch will implement reading this format, and the patch\nafter that will implement writing it while producing a multi-pack index.\n\nCo-authored-by: Jeff King \nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n Documentation/technical/pack-format.txt | 83 +++++++++++++++++++++++++\n 1 file changed, 83 insertions(+)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 8833b71c8b..a14722f119 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -376,3 +376,86 @@ CHUNK DATA:\n TRAILER:\n \n \tIndex checksum of the above contents.\n+\n+== multi-pack-index reverse indexes\n+\n+Similar to the pack-based reverse index, the multi-pack index can also\n+be used to generate a reverse index.\n+\n+Instead of mapping between offset, pack-, and index position, this\n+reverse index maps between an object's position within the midx, and\n+that object's position within a pseudo-pack that the midx describes.\n+Crucially, the objects' positions within this pseudo-pack are the same\n+as their bit positions in a multi-pack reachability bitmap.\n+\n+As a motivating example, consider the multi-pack reachability bitmap\n+(which does not yet exist, but is what we are building towards here). We\n+need each bit to correspond to an object covered by the midx, and we\n+need to be able to convert bit positions back to index positions (from\n+which we can get the oid, etc).\n+\n+One solution is to let each bit position in the index correspond to\n+the same position in the oid-sorted index stored by the midx. But\n+because oids are effectively random, there resulting reachability\n+bitmaps would have no locality, and thus compress poorly. (This is the\n+reason that single-pack bitmaps use the pack ordering, and not the .idx\n+ordering, for the same purpose.)\n+\n+So we'd like to define an ordering for the whole midx based around\n+pack ordering. We can think of it as a pseudo-pack created by the\n+concatenation of all of the packs in the midx. E.g., if we had a midx\n+with three packs (a, b, c), with 10, 15, and 20 objects respectively, we\n+can imagine an ordering of the objects like:\n+\n+ |a,0|a,1|...|a,9|b,0|b,1|...|b,14|c,0|c,1|...|c,19|\n+\n+where the ordering of the packs is defined by the midx's pack list,\n+and then the ordering of objects within each pack is the same as the\n+order in the actual packfile.\n+\n+Given the list of packs and their counts of objects, you can\n+naïvely reconstruct that pseudo-pack ordering (e.g., the object at\n+position 27 must be (c,1) because packs \"a\" and \"b\" consumed 25 of the\n+slots). But there's a catch. Objects may be duplicated between packs, in\n+which case the midx only stores one pointer to the object (and thus we'd\n+want only one slot in the bitmap).\n+\n+Callers could handle duplicates themselves by reading objects in order\n+of their bit-position, but that's linear in the number of objects, and\n+much too expensive for ordinary bitmap lookups. Building a reverse index\n+solves this, since it is the logical inverse of the index, and that\n+index has already removed duplicates. But, building a reverse index on\n+the fly can be expensive. Since we already have an on-disk format for\n+pack-based reverse indexes, let's reuse it for the midx's pseudo-pack,\n+too.\n+\n+Objects from the midx are ordered as follows to string together the\n+pseudo-pack. Let _pack(o)_ return the pack from which _o_ was selected\n+by the midx, and define an ordering of packs based on their numeric ID\n+(as stored by the midx). Let _offset(o)_ return the object offset of _o_\n+within _pack(o)_. Then, compare _o~1~_ and _o~2~_ as follows:\n+\n+ - If one of _pack(o~1~)_ and _pack(o~2~)_ is preferred and the other\n+ is not, then the preferred one sorts first.\n++\n+(This is a detail that allows the midx bitmap to determine which\n+pack should be used by the pack-reuse mechanism, since it can ask\n+the midx for the pack containing the object at bit position 0).\n+\n+ - If _pack(o~1~) ≠ pack(o~2~)_, then sort the two objects in\n+ descending order based on the pack ID.\n+\n+ - Otherwise, _pack(o~1~) = pack(o~2~)_, and the objects are\n+ sorted in pack-order (i.e., _o~1~_ sorts ahead of _o~2~_ exactly\n+ when _offset(o~1~) < offset(o~2~)_).\n+\n+In short, a midx's pseudo-pack is the de-duplicated concatenation of\n+objects in packs stored by the midx, laid out in pack order, and the\n+packs arranged in midx order (with the preferred pack coming first).\n+\n+Finally, note that the midx's reverse index is not stored as a chunk in\n+the multi-pack-index itself. This is done because the reverse index\n+includes the checksum of the pack or midx to which it belongs, which\n+makes it impossible to write in the midx. To avoid races when rewriting\n+the midx, a midx reverse index includes the midx's checksum in its\n+filename (e.g., `multi-pack-index-xyz.rev`).\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"416678","messageId":"4c5e64c5fcb2948a0598d6e0c70097a9561716a9.1612998106.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1612998106.git.me@ttaylorr.com","subject":"[PATCH 7/9] pack-revindex: read multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-10T23:03:20Z","receivedAt":"2021-02-10T23:05:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement reading for multi-pack reverse indexes, as described in the\nprevious patch.\n\nNote that these functions don't yet have any callers, and won't until\nmulti-pack reachability bitmaps are introduced in a later patch series.\nIn the meantime, this patch implements some of the infrastructure\nnecessary to support multi-pack bitmaps.\n\nThere are three new functions exposed by the revindex API:\n\n - load_midx_revindex(): loads the reverse index corresponding to the\n given multi-pack index.\n\n - midx_to_pack_pos() and pack_pos_to_midx(): these convert between the\n multi-pack index and pseudo-pack order.\n\nload_midx_revindex() and pack_pos_to_midx() are both relatively\nstraightforward.\n\nload_midx_revindex() needs a few functions to be exposed from the midx\nAPI. One to get the checksum of a midx, and another to get the .rev's\nfilename. Similar to recent changes in the packed_git struct, three new\nfields are added to the multi_pack_index struct: one to keep track of\nthe size, one to keep track of the mmap'd pointer, and another to point\npast the header and at the reverse index's data.\n\npack_pos_to_midx() simply reads the corresponding entry out of the\ntable.\n\nmidx_to_pack_pos() is the trickiest, since it needs to find an object's\nposition in the psuedo-pack order, but that order can only be recovered\nin the .rev file itself. This mapping can be implemented with a binary\nsearch, but note that the thing we're binary searching over isn't an\narray, but rather a _permutation_.\n\nSo, when comparing two items, it's helpful to keep in mind the\ndifference. Instead of a traditional binary search, where you are\ncomparing two things directly, here we're comparing a (pack, offset)\ntuple with an index into the multi-pack index. That index describes\nanother (pack, offset) tuple, and it is _those_ two tuples that are\ncompared.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 11 +++++\n midx.h | 6 +++\n pack-revindex.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++++\n pack-revindex.h | 46 ++++++++++++++++++++\n packfile.c | 3 ++\n 5 files changed, 178 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex bf258c4fde..12bfce8bb1 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -48,11 +48,22 @@ static uint8_t oid_version(void)\n \t}\n }\n \n+static const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n+{\n+\treturn m->data + m->data_len - the_hash_algo->rawsz;\n+}\n+\n static char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\n \n+char *get_midx_rev_filename(struct multi_pack_index *m)\n+{\n+\treturn xstrfmt(\"%s/pack/multi-pack-index-%s.rev\",\n+\t\t m->object_dir, hash_to_hex(get_midx_checksum(m)));\n+}\n+\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local)\n {\n \tstruct multi_pack_index *m = NULL;\ndiff --git a/midx.h b/midx.h\nindex 93bd68189e..0a8294d2ee 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -15,6 +15,10 @@ struct multi_pack_index {\n \tconst unsigned char *data;\n \tsize_t data_len;\n \n+\tconst uint32_t *revindex_data;\n+\tconst uint32_t *revindex_map;\n+\tsize_t revindex_len;\n+\n \tuint32_t signature;\n \tunsigned char version;\n \tunsigned char hash_len;\n@@ -37,6 +41,8 @@ struct multi_pack_index {\n \n #define MIDX_PROGRESS (1 << 0)\n \n+char *get_midx_rev_filename(struct multi_pack_index *m);\n+\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 83fe4de773..da4101f4b2 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -3,6 +3,7 @@\n #include \"object-store.h\"\n #include \"packfile.h\"\n #include \"config.h\"\n+#include \"midx.h\"\n \n struct revindex_entry {\n \toff_t offset;\n@@ -292,6 +293,29 @@ int load_pack_revindex(struct packed_git *p)\n \treturn -1;\n }\n \n+int load_midx_revindex(struct multi_pack_index *m)\n+{\n+\tchar *revindex_name;\n+\tint ret;\n+\tif (m->revindex_data)\n+\t\treturn 0;\n+\n+\trevindex_name = get_midx_rev_filename(m);\n+\n+\tret = load_revindex_from_disk(revindex_name,\n+\t\t\t\t m->num_objects,\n+\t\t\t\t &m->revindex_map,\n+\t\t\t\t &m->revindex_len);\n+\tif (ret)\n+\t\tgoto cleanup;\n+\n+\tm->revindex_data = (const uint32_t *)((const char *)m->revindex_map + RIDX_HEADER_SIZE);\n+\n+cleanup:\n+\tfree(revindex_name);\n+\treturn ret;\n+}\n+\n int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n {\n \tunsigned lo, hi;\n@@ -346,3 +370,91 @@ off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n \telse\n \t\treturn nth_packed_object_offset(p, pack_pos_to_index(p, pos));\n }\n+\n+uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos)\n+{\n+\tif (!m->revindex_data)\n+\t\tBUG(\"pack_pos_to_midx: reverse index not yet loaded\");\n+\tif (m->num_objects <= pos)\n+\t\tBUG(\"pack_pos_to_midx: out-of-bounds object at %\"PRIu32, pos);\n+\treturn get_be32((const char *)m->revindex_data + (pos * sizeof(uint32_t)));\n+}\n+\n+struct midx_pack_key {\n+\tuint32_t pack;\n+\toff_t offset;\n+\n+\tuint32_t preferred_pack;\n+\tstruct multi_pack_index *midx;\n+};\n+\n+static int midx_pack_order_cmp(const void *va, const void *vb)\n+{\n+\tconst struct midx_pack_key *key = va;\n+\tstruct multi_pack_index *midx = key->midx;\n+\n+\tuint32_t versus = pack_pos_to_midx(midx, (uint32_t*)vb - (const uint32_t *)midx->revindex_data);\n+\tuint32_t versus_pack = nth_midxed_pack_int_id(midx, versus);\n+\toff_t versus_offset;\n+\n+\tuint32_t key_preferred = key->pack == key->preferred_pack;\n+\tuint32_t versus_preferred = versus_pack == key->preferred_pack;\n+\n+\t/*\n+\t * First, compare the preferred-ness, noting that the preferred pack\n+\t * comes first.\n+\t */\n+\tif (key_preferred && !versus_preferred)\n+\t\treturn -1;\n+\telse if (!key_preferred && versus_preferred)\n+\t\treturn 1;\n+\n+\t/* Then, break ties first by comparing the pack IDs. */\n+\tif (key->pack < versus_pack)\n+\t\treturn -1;\n+\telse if (key->pack > versus_pack)\n+\t\treturn 1;\n+\n+\t/* Finally, break ties by comparing offsets within a pack. */\n+\tversus_offset = nth_midxed_offset(midx, versus);\n+\tif (key->offset < versus_offset)\n+\t\treturn -1;\n+\telse if (key->offset > versus_offset)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)\n+{\n+\tstruct midx_pack_key key;\n+\tuint32_t *found;\n+\n+\tif (!m->revindex_data)\n+\t\tBUG(\"midx_to_pack_pos: reverse index not yet loaded\");\n+\tif (m->num_objects <= at)\n+\t\tBUG(\"midx_to_pack_pos: out-of-bounds object at %\"PRIu32, at);\n+\n+\tkey.pack = nth_midxed_pack_int_id(m, at);\n+\tkey.offset = nth_midxed_offset(m, at);\n+\tkey.midx = m;\n+\t/*\n+\t * The preferred pack sorts first, so determine its identifier by\n+\t * looking at the first object in pseudo-pack order.\n+\t *\n+\t * Note that if no --preferred-pack is explicitly given when writing a\n+\t * multi-pack index, then whichever pack has the lowest identifier\n+\t * implicitly is preferred (and includes all its objects, since ties are\n+\t * broken first by pack identifier).\n+\t */\n+\tkey.preferred_pack = nth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));\n+\n+\tfound = bsearch(&key, m->revindex_data, m->num_objects,\n+\t\t\tsizeof(uint32_t), midx_pack_order_cmp);\n+\n+\tif (!found)\n+\t\treturn error(\"bad offset for revindex\");\n+\n+\t*pos = found - m->revindex_data;\n+\treturn 0;\n+}\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex ba7c82c125..49d604aa9a 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -14,6 +14,20 @@\n *\n * - offset: the byte offset within the .pack file at which the object contents\n * can be found\n+ *\n+ * The revindex can also be used with a multi-pack index (MIDX). In this\n+ * setting:\n+ *\n+ * - index position refers to an object's numeric position within the MIDX\n+ *\n+ * - pack position refers to an object's position within a non-existent pack\n+ * described by the MIDX. The pack structure is described in\n+ * Documentation/technical/pack-format.txt.\n+ *\n+ * It is effectively a concatanation of all packs in the MIDX (ordered by\n+ * their numeric ID within the MIDX) in their original order within each\n+ * pack), removing duplicates, and placing the preferred pack (if any)\n+ * first.\n */\n \n \n@@ -24,6 +38,7 @@\n #define GIT_TEST_REV_INDEX_DIE_IN_MEMORY \"GIT_TEST_REV_INDEX_DIE_IN_MEMORY\"\n \n struct packed_git;\n+struct multi_pack_index;\n \n /*\n * load_pack_revindex populates the revindex's internal data-structures for the\n@@ -34,6 +49,15 @@ struct packed_git;\n */\n int load_pack_revindex(struct packed_git *p);\n \n+/*\n+ * load_midx_revindex loads the '.rev' file corresponding to the given\n+ * multi-pack index by mmap-ing it and assigning pointers in the\n+ * multi_pack_index to point at it.\n+ *\n+ * A negative number is returned on error.\n+ */\n+int load_midx_revindex(struct multi_pack_index *m);\n+\n /*\n * offset_to_pack_pos converts an object offset to a pack position. This\n * function returns zero on success, and a negative number otherwise. The\n@@ -71,4 +95,26 @@ uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);\n */\n off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos);\n \n+/*\n+ * pack_pos_to_midx converts the object at position \"pos\" within the MIDX\n+ * pseudo-pack into a MIDX position.\n+ *\n+ * If the reverse index has not yet been loaded, or the position is out of\n+ * bounds, this function aborts.\n+ *\n+ * This function runs in time O(log N) with the number of objects in the MIDX.\n+ */\n+uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos);\n+\n+/*\n+ * midx_to_pack_pos converts from the MIDX-relative position at \"at\" to the\n+ * corresponding pack position.\n+ *\n+ * If the reverse index has not yet been loaded, or the position is out of\n+ * bounds, this function aborts.\n+ *\n+ * This function runs in constant time.\n+ */\n+int midx_to_pack_pos(struct multi_pack_index *midx, uint32_t at, uint32_t *pos);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 1fec12ac5f..82623e0cb4 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -862,6 +862,9 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \n \tif (!strcmp(file_name, \"multi-pack-index\"))\n \t\treturn;\n+\tif (starts_with(file_name, \"multi-pack-index\") &&\n+\t ends_with(file_name, \".rev\"))\n+\t\treturn;\n \tif (ends_with(file_name, \".idx\") ||\n \t ends_with(file_name, \".rev\") ||\n \t ends_with(file_name, \".pack\") ||\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"416679","messageId":"4829b93f42955f417d7ccdfffeee2393313a347e.1612998106.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1612998106.git.me@ttaylorr.com","subject":"[PATCH 8/9] pack-write.c: extract 'write_rev_file_order'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-10T23:03:26Z","receivedAt":"2021-02-10T23:05:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Existing callers provide the reverse index code with an array of 'struct\npack_idx_entry *'s, which is then sorted by pack order (comparing the\noffsets of each object within the pack).\n\nPrepare for the multi-pack index to write a .rev file by providing a way\nto write the reverse index without an array of pack_idx_entry (which the\nMIDX code does not have).\n\nInstead, callers can invoke 'write_rev_index_positions()', which takes\nan array of uint32_t's. The ith entry in this array specifies the ith\nobject's (in index order) position within the pack (in pack order).\n\nExpose this new function for use in a later patch, and rewrite the\nexisting write_rev_file() in terms of this new function.\n\nSigned-off-by: Taylor Blau \n---\n pack-write.c | 39 ++++++++++++++++++++++++++++-----------\n pack.h | 1 +\n 2 files changed, 29 insertions(+), 11 deletions(-)\n\ndiff --git a/pack-write.c b/pack-write.c\nindex 680c36755d..75fcf70db1 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -201,21 +201,12 @@ static void write_rev_header(struct hashfile *f)\n }\n \n static void write_rev_index_positions(struct hashfile *f,\n-\t\t\t\t struct pack_idx_entry **objects,\n+\t\t\t\t uint32_t *pack_order,\n \t\t\t\t uint32_t nr_objects)\n {\n-\tuint32_t *pack_order;\n \tuint32_t i;\n-\n-\tALLOC_ARRAY(pack_order, nr_objects);\n-\tfor (i = 0; i < nr_objects; i++)\n-\t\tpack_order[i] = i;\n-\tQSORT_S(pack_order, nr_objects, pack_order_cmp, objects);\n-\n \tfor (i = 0; i < nr_objects; i++)\n \t\thashwrite_be32(f, pack_order[i]);\n-\n-\tfree(pack_order);\n }\n \n static void write_rev_trailer(struct hashfile *f, const unsigned char *hash)\n@@ -228,6 +219,32 @@ const char *write_rev_file(const char *rev_name,\n \t\t\t uint32_t nr_objects,\n \t\t\t const unsigned char *hash,\n \t\t\t unsigned flags)\n+{\n+\tuint32_t *pack_order;\n+\tuint32_t i;\n+\tconst char *ret;\n+\n+\tALLOC_ARRAY(pack_order, nr_objects);\n+\tfor (i = 0; i < nr_objects; i++)\n+\t\tpack_order[i] = i;\n+\tQSORT_S(pack_order, nr_objects, pack_order_cmp, objects);\n+\n+\tif (!(flags & (WRITE_REV | WRITE_REV_VERIFY)))\n+\t\treturn NULL;\n+\n+\tret = write_rev_file_order(rev_name, pack_order, nr_objects, hash,\n+\t\t\t\t flags);\n+\n+\tfree(pack_order);\n+\n+\treturn ret;\n+}\n+\n+const char *write_rev_file_order(const char *rev_name,\n+\t\t\t\t uint32_t *pack_order,\n+\t\t\t\t uint32_t nr_objects,\n+\t\t\t\t const unsigned char *hash,\n+\t\t\t\t unsigned flags)\n {\n \tstruct hashfile *f;\n \tint fd;\n@@ -262,7 +279,7 @@ const char *write_rev_file(const char *rev_name,\n \n \twrite_rev_header(f);\n \n-\twrite_rev_index_positions(f, objects, nr_objects);\n+\twrite_rev_index_positions(f, pack_order, nr_objects);\n \twrite_rev_trailer(f, hash);\n \n \tif (rev_name && adjust_shared_perm(rev_name) < 0)\ndiff --git a/pack.h b/pack.h\nindex afdcf8f5c7..09c2a7dd3a 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -94,6 +94,7 @@ struct ref;\n void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought);\n \n const char *write_rev_file(const char *rev_name, struct pack_idx_entry **objects, uint32_t nr_objects, const unsigned char *hash, unsigned flags);\n+const char *write_rev_file_order(const char *rev_name, uint32_t *pack_order, uint32_t nr_objects, const unsigned char *hash, unsigned flags);\n \n /*\n * The \"hdr\" output buffer should be at least this big, which will handle sizes\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"416680","messageId":"fb5954b769004e623e5cd49893b16e6d2adb0a54.1612998106.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1612998106.git.me@ttaylorr.com","subject":"[PATCH 9/9] pack-revindex: write multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-10T23:03:32Z","receivedAt":"2021-02-10T23:05:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement the writing half of multi-pack reverse indexes. This is\nnothing more than the format describe a few patches ago, with a new set\nof helper functions that will be used to clear out stale .rev files\ncorresponding to old MIDXs.\n\nUnfortunately, a very similar comparison function as the one implemented\nrecently in pack-revindex.c is reimplemented here, this time accepting a\nMIDX-internal type. An effort to DRY these up would create more\nindirection and overhead than is necessary, so it isn't pursued here.\n\nCurrently, there are no callers which pass the MIDX_WRITE_REV_INDEX\nflag, meaning that this is all dead code. But, that won't be the case\nfor long, since subsequent patches will introduce the multi-pack bitmap,\nwhich will begin passing this field.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 123 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n midx.h | 1 +\n 2 files changed, 124 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex 12bfce8bb1..3d79a1930a 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -11,6 +11,7 @@\n #include \"trace2.h\"\n #include \"run-command.h\"\n #include \"repository.h\"\n+#include \"pack.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -841,6 +842,78 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off\n \treturn written;\n }\n \n+struct midx_pack_order_data {\n+\tstruct pack_midx_entry *entries;\n+\tuint32_t *pack_perm;\n+};\n+\n+static int midx_pack_order_cmp(const void *va, const void *vb, void *_data)\n+{\n+\tstruct midx_pack_order_data *data = _data;\n+\n+\tstruct pack_midx_entry *a = &data->entries[*(const uint32_t *)va];\n+\tstruct pack_midx_entry *b = &data->entries[*(const uint32_t *)vb];\n+\n+\tuint32_t perm_a = data->pack_perm[a->pack_int_id];\n+\tuint32_t perm_b = data->pack_perm[b->pack_int_id];\n+\n+\t/* Sort objects in the preferred pack ahead of any others. */\n+\tif (a->preferred > b->preferred)\n+\t\treturn -1;\n+\tif (a->preferred < b->preferred)\n+\t\treturn 1;\n+\n+\t/* Then, order objects by which packs they appear in. */\n+\tif (perm_a < perm_b)\n+\t\treturn -1;\n+\tif (perm_a > perm_b)\n+\t\treturn 1;\n+\n+\t/* Then, disambiguate by their offset within each pack. */\n+\tif (a->offset < b->offset)\n+\t\treturn -1;\n+\tif (a->offset > b->offset)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t *midx_pack_order(struct pack_midx_entry *entries,\n+\t\t\t\t uint32_t *pack_perm,\n+\t\t\t\t uint32_t entries_nr)\n+{\n+\tstruct midx_pack_order_data data;\n+\tuint32_t *pack_order;\n+\tuint32_t i;\n+\n+\tdata.entries = entries;\n+\tdata.pack_perm = pack_perm;\n+\n+\tALLOC_ARRAY(pack_order, entries_nr);\n+\tfor (i = 0; i < entries_nr; i++)\n+\t\tpack_order[i] = i;\n+\tQSORT_S(pack_order, entries_nr, midx_pack_order_cmp, &data);\n+\n+\treturn pack_order;\n+}\n+\n+static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n+\t\t\t\t uint32_t *pack_order,\n+\t\t\t\t uint32_t entries_nr)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex(midx_hash));\n+\n+\twrite_rev_file_order(buf.buf, pack_order, entries_nr, midx_hash,\n+\t\t\t WRITE_REV);\n+\n+\tstrbuf_release(&buf);\n+}\n+\n+static void clear_midx_files_ext(struct repository *r, const char *ext,\n+\t\t\t\t unsigned char *keep_hash);\n+\n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n \t\t\t struct string_list *packs_to_drop,\n \t\t\t const char *preferred_pack_name,\n@@ -854,6 +927,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tstruct lock_file lk;\n \tstruct pack_list packs;\n \tuint32_t *pack_perm = NULL;\n+\tuint32_t *pack_order = NULL;\n \tuint64_t written = 0;\n \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n@@ -1111,6 +1185,14 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t chunk_offsets[num_chunks]);\n \n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n+\tif (flags & MIDX_WRITE_REV_INDEX)\n+\t\tpack_order = midx_pack_order(entries, pack_perm, nr_entries);\n+\n+\tif (flags & MIDX_WRITE_REV_INDEX)\n+\t\twrite_midx_reverse_index(midx_name, midx_hash, pack_order,\n+\t\t\t\t\t nr_entries);\n+\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n+\n \tcommit_lock_file(&lk);\n \n cleanup:\n@@ -1125,6 +1207,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tfree(packs.info);\n \tfree(entries);\n \tfree(pack_perm);\n+\tfree(pack_order);\n \tfree(midx_name);\n \treturn result;\n }\n@@ -1137,6 +1220,44 @@ int write_midx_file(const char *object_dir,\n \t\t\t\t flags);\n }\n \n+struct clear_midx_data {\n+\tchar *keep;\n+\tconst char *ext;\n+};\n+\n+static void clear_midx_file_ext(const char *full_path, size_t full_path_len,\n+\t\t\t\tconst char *file_name, void *_data)\n+{\n+\tstruct clear_midx_data *data = _data;\n+\n+\tif (!(starts_with(file_name, \"multi-pack-index-\") &&\n+\t ends_with(file_name, data->ext)))\n+\t\treturn;\n+\tif (data->keep && !strcmp(data->keep, file_name))\n+\t\treturn;\n+\n+\tif (unlink(full_path))\n+\t\tdie_errno(_(\"failed to remove %s\"), full_path);\n+}\n+\n+static void clear_midx_files_ext(struct repository *r, const char *ext,\n+\t\t\t\t unsigned char *keep_hash)\n+{\n+\tstruct clear_midx_data data;\n+\tmemset(&data, 0, sizeof(struct clear_midx_data));\n+\n+\tif (keep_hash)\n+\t\tdata.keep = xstrfmt(\"multi-pack-index-%s%s\",\n+\t\t\t\t hash_to_hex(keep_hash), ext);\n+\tdata.ext = ext;\n+\n+\tfor_each_file_in_pack_dir(r->objects->odb->path,\n+\t\t\t\t clear_midx_file_ext,\n+\t\t\t\t &data);\n+\n+\tfree(data.keep);\n+}\n+\n void clear_midx_file(struct repository *r)\n {\n \tchar *midx = get_midx_filename(r->objects->odb->path);\n@@ -1149,6 +1270,8 @@ void clear_midx_file(struct repository *r)\n \tif (remove_path(midx))\n \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n \n+\tclear_midx_files_ext(r, \".rev\", NULL);\n+\n \tfree(midx);\n }\n \ndiff --git a/midx.h b/midx.h\nindex 0a8294d2ee..8684cf0fef 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -40,6 +40,7 @@ struct multi_pack_index {\n };\n \n #define MIDX_PROGRESS (1 << 0)\n+#define MIDX_WRITE_REV_INDEX (1 << 1)\n \n char *get_midx_rev_filename(struct multi_pack_index *m);\n \n-- \n2.30.0.667.g81c0cbc6fd\n"},{"id":"416701","messageId":"7b422082-d6a9-5e64-17a0-559da7777a08@gmail.com","threadId":"55135","inReplyTo":"e36acb005d3563db772711207d2af3df03685bd4.1612998106.git.me@ttaylorr.com","subject":"Re: [PATCH 1/9] t/helper/test-read-midx.c: add '--show-objects'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-11T02:27:29Z","receivedAt":"2021-02-11T02:28:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/10/21 6:02 PM, Taylor Blau wrote:\n> +\tif (show_objects) {\n> +\t\tstruct object_id oid;\n> +\t\tstruct pack_entry e;\n> +\n> +\t\tfor (i = 0; i < m->num_objects; i++) {\n> +\t\t\tnth_midxed_object_oid(&oid, m, i);\n> +\t\t\tfill_midx_entry(the_repository, &oid, &e, m);\n> +\n> +\t\t\tprintf(\"%s %\"PRIu64\"\\t%s\\n\",\n> +\t\t\t oid_to_hex(&oid), e.offset, e.p->pack_name);\n> +\t\t}\n> +\t\treturn 0;\n> +\t}\n> +\n> \tprintf(\"header: %08x %d %d %d %d\\n\",\n> \t m->signature,\n> \t m->version,\n\nIt seems a little odd to me that the list of objects happens after\nthe header information. Probably doesn't matter in your test cases,\nbut I sometimes use the test helpers to diagnose data during development\nand could see piping this output into 'less' and wanting the header\nat the top.\n\nThanks,\n-Stolee\n"},{"id":"416702","messageId":"d03263c7-56d3-575c-4ab7-56a617daf34a@gmail.com","threadId":"55135","inReplyTo":"b4b842fa1e8c711af30cc93b6f1100661791416c.1612998106.git.me@ttaylorr.com","subject":"Re: [PATCH 4/9] midx: keep track of the checksum","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-11T02:33:36Z","receivedAt":"2021-02-11T02:34:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/10/21 6:02 PM, Taylor Blau wrote:\n> +\tunsigned char midx_hash[GIT_MAX_RAWSZ];\n\nI was initially thinking we should use something like\n'struct object_id' here, but the hash we are storing\ndoesn't correspond to an object, which would be\nconfusing. I suppose this is the most correct thing\nto do.\n\n> -\tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n> +\tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n\nAnd this is of course correct. The issue will be what\nhappens to 'midx_hash' in the later changes. Will\nkeep reading.\n\nThanks,\n-Stolee\n"},{"id":"416703","messageId":"YCSXp+YgDVCLL67Q@nand.local","threadId":"55135","inReplyTo":"7b422082-d6a9-5e64-17a0-559da7777a08@gmail.com","subject":"Re: [PATCH 1/9] t/helper/test-read-midx.c: add '--show-objects'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-11T02:34:15Z","receivedAt":"2021-02-11T02:35:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Feb 10, 2021 at 09:27:29PM -0500, Derrick Stolee wrote:\n> On 2/10/21 6:02 PM, Taylor Blau wrote:\n> > +\tif (show_objects) {\n> > +\t\tstruct object_id oid;\n> > +\t\tstruct pack_entry e;\n> > +\n> > +\t\tfor (i = 0; i < m->num_objects; i++) {\n> > +\t\t\tnth_midxed_object_oid(&oid, m, i);\n> > +\t\t\tfill_midx_entry(the_repository, &oid, &e, m);\n> > +\n> > +\t\t\tprintf(\"%s %\"PRIu64\"\\t%s\\n\",\n> > +\t\t\t oid_to_hex(&oid), e.offset, e.p->pack_name);\n> > +\t\t}\n> > +\t\treturn 0;\n> > +\t}\n> > +\n> > \tprintf(\"header: %08x %d %d %d %d\\n\",\n> > \t m->signature,\n> > \t m->version,\n>\n> It seems a little odd to me that the list of objects happens after\n> the header information. Probably doesn't matter in your test cases,\n> but I sometimes use the test helpers to diagnose data during development\n> and could see piping this output into 'less' and wanting the header\n> at the top.\n\nIndeed. In theory you could pipe to tail instead (or to less and\nimmediately hit 'G'), but I can't think of a good reason that this would\nhave appeared above the header when I originally wrote the patch.\n\nAnyway, it doesn't seem that the tests care about where this is (they're\njust looking for for a line that begins with the object id and ends with\nits offset), so I think this could probably be moved without thinking\ntoo hard about it.\n\n> Thanks,\n> -Stolee\n\nThanks,\nTaylor\n"},{"id":"416704","messageId":"YCSX9FeUu17v13Al@nand.local","threadId":"55135","inReplyTo":"d03263c7-56d3-575c-4ab7-56a617daf34a@gmail.com","subject":"Re: [PATCH 4/9] midx: keep track of the checksum","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-11T02:35:32Z","receivedAt":"2021-02-11T02:36:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Feb 10, 2021 at 09:33:36PM -0500, Derrick Stolee wrote:\n> On 2/10/21 6:02 PM, Taylor Blau wrote:\n> > +\tunsigned char midx_hash[GIT_MAX_RAWSZ];\n>\n> I was initially thinking we should use something like\n> 'struct object_id' here, but the hash we are storing\n> doesn't correspond to an object, which would be\n> confusing. I suppose this is the most correct thing\n> to do.\n\nYeah. There are a number of places that abuse the unsigned char array\ninside of object_id, but there's no good reason to.\n\nThanks,\nTaylor\n"},{"id":"416705","messageId":"320ef364-05ac-7d51-a6f9-770d32590766@gmail.com","threadId":"55135","inReplyTo":"e64504bad6e181522946a8f234e12f569bede89e.1612998106.git.me@ttaylorr.com","subject":"Re: [PATCH 6/9] Documentation/technical: describe multi-pack reverse indexes","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-11T02:48:20Z","receivedAt":"2021-02-11T02:49:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/10/21 6:03 PM, Taylor Blau wrote:> +Instead of mapping between offset, pack-, and index position, this\n\nThe \"pack-,\" should be paired with \"index-position\" or drop the\nhyphen in both cases. Perhaps just be explicit, especially since\n\"position\" doesn't match with \"offset\":\n\n Instead of mapping between pack offset, pack position, and index\n position, ...\n\n> +reverse index maps between an object's position within the midx, and\n> +that object's position within a pseudo-pack that the midx describes.\n\nnit: use multi-pack-index or MIDX, not lower-case 'midx'.\n\n> +Crucially, the objects' positions within this pseudo-pack are the same\n> +as their bit positions in a multi-pack reachability bitmap.\n> +\n> +As a motivating example, consider the multi-pack reachability bitmap\n> +(which does not yet exist, but is what we are building towards here). We\n> +need each bit to correspond to an object covered by the midx, and we\n> +need to be able to convert bit positions back to index positions (from\n> +which we can get the oid, etc).\n\nThese paragraphs are awkward. Instead of operating in the hypothetical\nworld of reachability bitmaps, focus on the fact that bitmaps need\na bidirectional mapping between \"bit position\" and an object ID.\n\nHere is an attempt to reword some of the context you are using here.\nFeel free to take as much or as little as you want.\n\n The multi-pack-index stores the object IDs in lexicographical order\n (lex-order) to allow binary search. To allow compressible reachability\n bitmaps to pair with a multi-pack-index, a different ordering is\n required. When paired with a single packfile, the order used is the\n object order within the packfile (called the pack-order). Construct\n a \"pseudo-pack\" by concatenating all tracked packfiles in the\n multi-pack-index. We now need a mapping between the lex-order and the\n pseudo-pack-order.\n\n> +One solution is to let each bit position in the index correspond to\n> +the same position in the oid-sorted index stored by the midx. But\n> +because oids are effectively random, there resulting reachability\n> +bitmaps would have no locality, and thus compress poorly. (This is the\n> +reason that single-pack bitmaps use the pack ordering, and not the .idx\n> +ordering, for the same purpose.)\n> +\n> +So we'd like to define an ordering for the whole midx based around\n> +pack ordering. We can think of it as a pseudo-pack created by the\n> +concatenation of all of the packs in the midx. E.g., if we had a midx\n> +with three packs (a, b, c), with 10, 15, and 20 objects respectively, we\n> +can imagine an ordering of the objects like:\n> +\n> + |a,0|a,1|...|a,9|b,0|b,1|...|b,14|c,0|c,1|...|c,19|\n> +\n> +where the ordering of the packs is defined by the midx's pack list,\n> +and then the ordering of objects within each pack is the same as the\n> +order in the actual packfile.\n> +\n> +Given the list of packs and their counts of objects, you can\n> +naïvely reconstruct that pseudo-pack ordering (e.g., the object at\n> +position 27 must be (c,1) because packs \"a\" and \"b\" consumed 25 of the\n> +slots). But there's a catch. Objects may be duplicated between packs, in\n> +which case the midx only stores one pointer to the object (and thus we'd\n> +want only one slot in the bitmap).\n> +\n> +Callers could handle duplicates themselves by reading objects in order\n> +of their bit-position, but that's linear in the number of objects, and\n> +much too expensive for ordinary bitmap lookups. Building a reverse index\n> +solves this, since it is the logical inverse of the index, and that\n> +index has already removed duplicates. But, building a reverse index on\n> +the fly can be expensive. Since we already have an on-disk format for\n> +pack-based reverse indexes, let's reuse it for the midx's pseudo-pack,\n> +too.\n> +\n> +Objects from the midx are ordered as follows to string together the\n> +pseudo-pack. Let _pack(o)_ return the pack from which _o_ was selected\n> +by the midx, and define an ordering of packs based on their numeric ID\n> +(as stored by the midx). Let _offset(o)_ return the object offset of _o_\n> +within _pack(o)_. Then, compare _o~1~_ and _o~2~_ as follows:\n> +\n> + - If one of _pack(o~1~)_ and _pack(o~2~)_ is preferred and the other\n> + is not, then the preferred one sorts first.\n> ++\n> +(This is a detail that allows the midx bitmap to determine which\n> +pack should be used by the pack-reuse mechanism, since it can ask\n> +the midx for the pack containing the object at bit position 0).\n> +\n> + - If _pack(o~1~) ≠ pack(o~2~)_, then sort the two objects in\n> + descending order based on the pack ID.\n> +\n> + - Otherwise, _pack(o~1~) = pack(o~2~)_, and the objects are\n> + sorted in pack-order (i.e., _o~1~_ sorts ahead of _o~2~_ exactly\n> + when _offset(o~1~) < offset(o~2~)_).\n> +\n> +In short, a midx's pseudo-pack is the de-duplicated concatenation of\n> +objects in packs stored by the midx, laid out in pack order, and the\n> +packs arranged in midx order (with the preferred pack coming first).\n> +\n> +Finally, note that the midx's reverse index is not stored as a chunk in\n> +the multi-pack-index itself. This is done because the reverse index\n> +includes the checksum of the pack or midx to which it belongs, which\n> +makes it impossible to write in the midx. To avoid races when rewriting\n> +the midx, a midx reverse index includes the midx's checksum in its\n> +filename (e.g., `multi-pack-index-xyz.rev`).\n\nThe rest of these details make sense and sufficiently motivate the\nordering, once the concept is clear.\n\nThanks,\n-Stolee\n"},{"id":"416706","messageId":"3680ce51-81ba-d6a3-a0b6-6f3a7cc14503@gmail.com","threadId":"55135","inReplyTo":"4c5e64c5fcb2948a0598d6e0c70097a9561716a9.1612998106.git.me@ttaylorr.com","subject":"Re: [PATCH 7/9] pack-revindex: read multi-pack reverse indexes","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-11T02:53:23Z","receivedAt":"2021-02-11T02:54:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/10/21 6:03 PM, Taylor Blau wrote:\n> Implement reading for multi-pack reverse indexes, as described in the\n> previous patch.\n> \n> Note that these functions don't yet have any callers, and won't until\n> multi-pack reachability bitmaps are introduced in a later patch series.\n> In the meantime, this patch implements some of the infrastructure\n> necessary to support multi-pack bitmaps.\n> \n> There are three new functions exposed by the revindex API:\n> \n> - load_midx_revindex(): loads the reverse index corresponding to the\n> given multi-pack index.\n> \n> - midx_to_pack_pos() and pack_pos_to_midx(): these convert between the\n> multi-pack index and pseudo-pack order.\n> \n> load_midx_revindex() and pack_pos_to_midx() are both relatively\n> straightforward.\n> \n> load_midx_revindex() needs a few functions to be exposed from the midx\n> API. One to get the checksum of a midx, and another to get the .rev's\n> filename. Similar to recent changes in the packed_git struct, three new\n> fields are added to the multi_pack_index struct: one to keep track of\n> the size, one to keep track of the mmap'd pointer, and another to point\n> past the header and at the reverse index's data.\n> \n> pack_pos_to_midx() simply reads the corresponding entry out of the\n> table.\n> \n> midx_to_pack_pos() is the trickiest, since it needs to find an object's\n> position in the psuedo-pack order, but that order can only be recovered\n> in the .rev file itself. This mapping can be implemented with a binary\n> search, but note that the thing we're binary searching over isn't an\n> array, but rather a _permutation_.\n> \n> So, when comparing two items, it's helpful to keep in mind the\n> difference. Instead of a traditional binary search, where you are\n> comparing two things directly, here we're comparing a (pack, offset)\n> tuple with an index into the multi-pack index. That index describes\n> another (pack, offset) tuple, and it is _those_ two tuples that are\n> compared.\n> \n> Signed-off-by: Taylor Blau \n> ---\n> midx.c | 11 +++++\n> midx.h | 6 +++\n> pack-revindex.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++++\n> pack-revindex.h | 46 ++++++++++++++++++++\n> packfile.c | 3 ++\n> 5 files changed, 178 insertions(+)\n> \n> diff --git a/midx.c b/midx.c\n> index bf258c4fde..12bfce8bb1 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -48,11 +48,22 @@ static uint8_t oid_version(void)\n> \t}\n> }\n> \n> +static const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n> +{\n> +\treturn m->data + m->data_len - the_hash_algo->rawsz;\n\n'struct multi_pack_index' has a 'hash_len' member that you could\nuse here. It would allow a different hash length in the stored\nfile than the one required by the repository. Except...\n\n> +}\n> +\n> static char *get_midx_filename(const char *object_dir)\n> {\n> \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n> }\n> \n> +char *get_midx_rev_filename(struct multi_pack_index *m)\n> +{\n> +\treturn xstrfmt(\"%s/pack/multi-pack-index-%s.rev\",\n> +\t\t m->object_dir, hash_to_hex(get_midx_checksum(m)));\n\n...this assumes the hash is of the same length as the_hash_algo,\nso you are doing the right thing. Currently, I think we check\nthat 'm->hash_len == the_hash_algo->rawsz' on load. We'll need\nto check this again later when in the transition phase of the\nnew hash work.\n\n(No changes are needed to your patch.)\n\nThanks,\n-Stolee\n"},{"id":"416707","messageId":"f783c173-6a0f-71d0-174c-78e310d2edfd@gmail.com","threadId":"55135","inReplyTo":"cover.1612998106.git.me@ttaylorr.com","subject":"Re: [PATCH 0/9] midx: implement a multi-pack reverse index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-11T02:58:18Z","receivedAt":"2021-02-11T02:59:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/10/21 6:02 PM, Taylor Blau wrote:\n> This series describes and implements a reverse index for the multi-pack index,\n> based on a \"pseudo-pack\" which can be uniquely described by the multi-pack\n> index.\n> \n> The details of the pseudo-pack, and multi-pack reverse index are laid out in\n> detail in the sixth patch.\n> \n> This is in support of multi-pack reachability bitmaps, which contain objects\n> from the multi-pack index. Likewise, an object's bit position in a multi-pack\n> reachability bitmap is determined by its position with that multi-pack index's\n> pseudo pack.\n\nThis has been a lot of work, but I'm impressed with the progress here.\n\nThis series is good prep, and my comments are very minor.\n\nSince the need for these multi-pack-index-.rev files doesn't show\nup until the reachability bitmaps can be paired with MIDX files, it\nwould make sense to hold this series in 'next' until that one also\nstabilizes and they merge to 'master' together.\n\nThanks,\n-Stolee\n"},{"id":"416709","messageId":"YCSeami0pkLy20tK@nand.local","threadId":"55135","inReplyTo":"320ef364-05ac-7d51-a6f9-770d32590766@gmail.com","subject":"Re: [PATCH 6/9] Documentation/technical: describe multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-11T03:03:06Z","receivedAt":"2021-02-11T03:04:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Feb 10, 2021 at 09:48:20PM -0500, Derrick Stolee wrote:\n> nit: use multi-pack-index or MIDX, not lower-case 'midx'.\n\nThanks.\n\n> > +Crucially, the objects' positions within this pseudo-pack are the same\n> > +as their bit positions in a multi-pack reachability bitmap.\n> > +\n> > +As a motivating example, consider the multi-pack reachability bitmap\n> > +(which does not yet exist, but is what we are building towards here). We\n> > +need each bit to correspond to an object covered by the midx, and we\n> > +need to be able to convert bit positions back to index positions (from\n> > +which we can get the oid, etc).\n>\n> These paragraphs are awkward. Instead of operating in the hypothetical\n> world of reachability bitmaps, focus on the fact that bitmaps need\n> a bidirectional mapping between \"bit position\" and an object ID.\n\nHmm. I could buy that these paragraphs are awkward, but I'm not sure\nthat what you proposed makes it less so.\n\nI may be a bad person to judge what you wrote, since I am familiar with\nthe details of what it's describing. But my thoughts on that second and\nthird paragraph are basically:\n\n - define the valid orderings we might consider objects in a MIDX by,\n indicating which of those orderings we're going to use for\n multi-pack bitmaps\n\n - motivate the need for a mapping between lexicographic order and\n pseudo-pack order\n\n> Here is an attempt to reword some of the context you are using here.\n> Feel free to take as much or as little as you want.\n>\n> The multi-pack-index stores the object IDs in lexicographical order\n> (lex-order) to allow binary search. To allow compressible reachability\n> bitmaps to pair with a multi-pack-index, a different ordering is\n> required. When paired with a single packfile, the order used is the\n> object order within the packfile (called the pack-order). Construct\n> a \"pseudo-pack\" by concatenating all tracked packfiles in the\n> multi-pack-index. We now need a mapping between the lex-order and the\n> pseudo-pack-order.\n\nI struggled with what you wrote because I couldn't seem to neatly\nplace/replace that paragraph in with the existing text without referring\nto yet-undefined concepts.\n\nMaybe the confusion lies in the fact that we stray too far from the\npoint in the second and third paragraphs. What if we reordered the\nsecond, third, and fourth paragraph like this:\n\n\t\tInstead of mapping between offset, pack-, and index position, this\n\t\treverse index maps between an object's position within the MIDX, and\n\t\tthat object's position within a pseudo-pack that the MIDX describes.\n\n\t\tTo clarify these three orderings, consider a multi-pack reachability\n\t\tbitmap (which does not yet exist, but is what we are building towards\n\t\there). Each bit needs to correspond to an object in the MIDX, and so we\n\t\tneed an efficient mapping from bit position to MIDX position.\n\n\t\tOne solution is to let bits occupy the same position in the oid-sorted\n\t\tindex stored by the MIDX. But because oids are effectively random, there\n\t\tresulting reachability bitmaps would have no locality, and thus compress\n\t\tpoorly. (This is the reason that single-pack bitmaps use the pack\n\t\tordering, and not the .idx ordering, for the same purpose.)\n\n\t\tSo we'd like to define an ordering for the whole MIDX based around\n\t\tpack ordering, which has far better locality (and thus compresses more\n\t\tefficiently). We can think of a pseudo-pack created by the concatenation\n\t\tof all of the packs in the MIDX. E.g., if we had a MIDX with three packs\n\t\t(a, b, c), with 10, 15, and 20 objects respectively, we can imagine an\n\t\tordering of the objects like:\n\n> [snip]\n>\n> The rest of these details make sense and sufficiently motivate the\n> ordering, once the concept is clear.\n>\n> Thanks,\n> -Stolee\n\nThanks,\nTaylor\n"},{"id":"416710","messageId":"YCSe0/OIXisrgNP6@nand.local","threadId":"55135","inReplyTo":"3680ce51-81ba-d6a3-a0b6-6f3a7cc14503@gmail.com","subject":"Re: [PATCH 7/9] pack-revindex: read multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-11T03:04:51Z","receivedAt":"2021-02-11T03:05:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Feb 10, 2021 at 09:53:23PM -0500, Derrick Stolee wrote:\n> ...this assumes the hash is of the same length as the_hash_algo,\n> so you are doing the right thing. Currently, I think we check\n> that 'm->hash_len == the_hash_algo->rawsz' on load. We'll need\n> to check this again later when in the transition phase of the\n> new hash work.\n>\n> (No changes are needed to your patch.)\n\nAll makes sense, thanks.\n\nTaylor\n"},{"id":"416711","messageId":"YCSfQredivkRj1Es@nand.local","threadId":"55135","inReplyTo":"f783c173-6a0f-71d0-174c-78e310d2edfd@gmail.com","subject":"Re: [PATCH 0/9] midx: implement a multi-pack reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-11T03:06:42Z","receivedAt":"2021-02-11T03:07:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Feb 10, 2021 at 09:58:18PM -0500, Derrick Stolee wrote:\n> This series is good prep, and my comments are very minor.\n\nThanks for your review, especially on a series like this one that adds\nlots of code without any users :).\n\n> Since the need for these multi-pack-index-.rev files doesn't show\n> up until the reachability bitmaps can be paired with MIDX files, it\n> would make sense to hold this series in 'next' until that one also\n> stabilizes and they merge to 'master' together.\n\nThat's fine with me. It would be OK to merge this down to 'master', too,\nsince this is all dead code. In fact, that may be easier to work with,\nsince the next topic can be based directly off 'master' instead of\nhaving to keep this branch around forever.\n\nEither is fine, though.\n\nThanks,\nTaylor\n"},{"id":"416723","messageId":"xmqqpn17xcnk.fsf@gitster.c.googlers.com","threadId":"55135","inReplyTo":"4c5e64c5fcb2948a0598d6e0c70097a9561716a9.1612998106.git.me@ttaylorr.com","subject":"Re: [PATCH 7/9] pack-revindex: read multi-pack reverse indexes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-11T07:54:23Z","receivedAt":"2021-02-11T07:58:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> diff --git a/pack-revindex.c b/pack-revindex.c\n> index 83fe4de773..da4101f4b2 100644\n> --- a/pack-revindex.c\n> +++ b/pack-revindex.c\n> @@ -3,6 +3,7 @@\n> #include \"object-store.h\"\n> #include \"packfile.h\"\n> #include \"config.h\"\n> +#include \"midx.h\"\n\nThis seems to assume that the topic tb/pack-revindex-on-disk is\nalready there?\n\nJust trying to establish what dependencies the bunch of topics have\namong themselves.\n\nThanks.\n"},{"id":"416727","messageId":"xmqqlfbvxbrc.fsf@gitster.c.googlers.com","threadId":"55135","inReplyTo":"cover.1612998106.git.me@ttaylorr.com","subject":"Re: [PATCH 0/9] midx: implement a multi-pack reverse index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-11T08:13:43Z","receivedAt":"2021-02-11T08:14:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Since tb/pack-revindex-on-disk is queued to be merged to 'master', but hasn't\n> yet been merged, this series is based on that branch.\n\nThis seems to have a light conflict with Derrick's chunked file\nformat work in midx.c where pack_info is renamed and extended so the\nnew pack_order variable now needs to become a member in it.\n\nI think I resolved it OK, but without any callers that actually\nutilize the new code or tests, it is almost impossible to have any\nconfidence in the result of the conflict resolution X-<.\n\nCould you two please look over to see if I made any silly mistakes,\nwhen I pushe it out later?\n\nThanks.\n\n\n"},{"id":"416744","messageId":"YCVFOcQ7Uo1ujvAt@nand.local","threadId":"55135","inReplyTo":"xmqqpn17xcnk.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 7/9] pack-revindex: read multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-11T14:54:49Z","receivedAt":"2021-02-11T15:02:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Feb 10, 2021 at 11:54:23PM -0800, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > diff --git a/pack-revindex.c b/pack-revindex.c\n> > index 83fe4de773..da4101f4b2 100644\n> > --- a/pack-revindex.c\n> > +++ b/pack-revindex.c\n> > @@ -3,6 +3,7 @@\n> > #include \"object-store.h\"\n> > #include \"packfile.h\"\n> > #include \"config.h\"\n> > +#include \"midx.h\"\n>\n> This seems to assume that the topic tb/pack-revindex-on-disk is\n> already there?\n\nYeah, this topic depends on tb/pack-revindex-on-disk, which looks like\nwill be merged to master according to your last What's Cooking email.\n\n> Just trying to establish what dependencies the bunch of topics have\n> among themselves.\n\nThanks, I hope that it wasn't too much trouble.\n\nTaylor\n"},{"id":"416752","messageId":"10c76063-f6b9-d88e-cb2e-731dec204bba@gmail.com","threadId":"55135","inReplyTo":"xmqqlfbvxbrc.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 0/9] midx: implement a multi-pack reverse index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-11T18:37:06Z","receivedAt":"2021-02-11T18:54:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/11/2021 3:13 AM, Junio C Hamano wrote:\n> Taylor Blau writes:\n> \n>> Since tb/pack-revindex-on-disk is queued to be merged to 'master', but hasn't\n>> yet been merged, this series is based on that branch.\n> \n> This seems to have a light conflict with Derrick's chunked file\n> format work in midx.c where pack_info is renamed and extended so the\n> new pack_order variable now needs to become a member in it.\n> \n> I think I resolved it OK, but without any callers that actually\n> utilize the new code or tests, it is almost impossible to have any\n> confidence in the result of the conflict resolution X-<.\n> \n> Could you two please look over to see if I made any silly mistakes,\n> when I pushe it out later?\n\nI reproduced your merge and got very similar results. The differences\nthat happened between my result and yours are not meaningful in any\nway.\n\nDefinitely a very subtle merge, so thank you for doing that!\n\n-Stolee\n\n"},{"id":"416753","messageId":"xmqqwnvewi1p.fsf@gitster.c.googlers.com","threadId":"55135","inReplyTo":"10c76063-f6b9-d88e-cb2e-731dec204bba@gmail.com","subject":"Re: [PATCH 0/9] midx: implement a multi-pack reverse index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-02-11T18:55:30Z","receivedAt":"2021-02-11T19:04:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>> Could you two please look over to see if I made any silly mistakes,\n>> when I pushe it out later?\n>\n> I reproduced your merge and got very similar results. The differences\n> that happened between my result and yours are not meaningful in any\n> way.\n>\n> Definitely a very subtle merge, so thank you for doing that!\n\nResolving this kind of conflict is like a quick quiz in the\nclassroom to check my understanding of what is happening in all the\ntopics involved. It is like solving a puzzle, and I do not dislike\ndoing it (otherwise I won't be working as a maintainer), but having\na second set of eyes, helping with independent verification, is\nalways appreciated.\n\nThanks.\n"},{"id":"416757","messageId":"20210211193314.GG1015009@szeder.dev","threadId":"55135","inReplyTo":"4a358d57cf6e834cd1756e70bf713d4d104f321e.1612998106.git.me@ttaylorr.com","subject":"Re: [PATCH 2/9] midx: allow marking a pack as preferred","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-02-11T19:33:14Z","receivedAt":"2021-02-11T19:34:26Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Feb 10, 2021 at 06:02:42PM -0500, Taylor Blau wrote:\n> diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\n> index eb0caa0439..dd14eab781 100644\n> --- a/Documentation/git-multi-pack-index.txt\n> +++ b/Documentation/git-multi-pack-index.txt\n> @@ -9,7 +9,8 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n> SYNOPSIS\n> --------\n> [verse]\n> -'git multi-pack-index' [--object-dir=] [--[no-]progress] \n> +'git multi-pack-index' [--object-dir=] [--[no-]progress]\n> +\t[--preferred-pack=] \n> \n> DESCRIPTION\n> -----------\n> @@ -27,6 +28,14 @@ OPTIONS\n> \tTurn progress on/off explicitly. If neither is specified, progress is\n> \tshown if standard error is connected to a terminal.\n> \n> +--preferred-pack=::\n> +\tWhen using the `write` subcommand, optionally specify the\n> +\ttie-breaking pack used when multiple packs contain the same\n> +\tobject. Incompatible with other subcommands, including `repack`,\n\nI think this shouldn't be an option of the 'git multi-pack-index'\ncommand but an option of its 'write' subcommand.\n\n"},{"id":"416937","messageId":"YCqX7hxWWKzmvaUO@nand.local","threadId":"55135","inReplyTo":"20210211193314.GG1015009@szeder.dev","subject":"Re: [PATCH 2/9] midx: allow marking a pack as preferred","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-15T15:49:02Z","receivedAt":"2021-02-15T16:05:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Feb 11, 2021 at 08:33:14PM +0100, SZEDER Gábor wrote:\n> > +--preferred-pack=::\n> > +\tWhen using the `write` subcommand, optionally specify the\n> > +\ttie-breaking pack used when multiple packs contain the same\n> > +\tobject. Incompatible with other subcommands, including `repack`,\n>\n> I think this shouldn't be an option of the 'git multi-pack-index'\n> command but an option of its 'write' subcommand.\n\n:-/. I wrote a lengthy response on Friday, but Gmail must have eaten it.\n\nThe gist of my response was that the intermingling of sub-commands with\noptions from other sub-commands goes deeper than just the documentation,\nsince command-line arguments are only parsed once in\nbuiltin/multi-pack-index.c.\n\nI explored whether or not it would be worth it to parse the common\noptions first, and then have separate options for each of the\nsub-commands (as is done in the commit-graph builtin). But, this is\ntricky, since we accept common options on either side of the sub-command\n(i.e., we'd expect both 'git multi-pack-index --object-dir=... write' to\nbehave the same as 'git multi-pack-index write --object-dir=...').\n\nSo you could let the first call to parse_options() parse all of the\narguments, but then specialized arguments (e.g., 'repack --batch-size')\nwould cause the parse-options API to barf because the first call to\nparse_options() doesn't recognize '--batch-size'.\n\nI think the easiest way to do it would be to pass\nPARSE_OPT_STOP_AT_NON_OPTION, and then let the subsequent calls to\nparse_options() pass an array of option structs that also includes the\ncommon options so they can be parsed on either side of the sub-command.\n\nObviously this leads to a lot of rather unfortunate duplication. So,\nI'm content to leave it all as-is, and let the multi-pack-index\nbuiltin check the disallowed combinations itself (e.g., if you passed\n'--preferred-pack' but aren't in 'write' mode, then we should complain).\n\nI can certainly move this piece of documentation into the 'write'\nsection, though, which should alleviate your immediate concern.\n\nThanks,\nTaylor\n"},{"id":"416947","messageId":"87r1lhb6z7.fsf@evledraar.gmail.com","threadId":"55135","inReplyTo":"YCqX7hxWWKzmvaUO@nand.local","subject":"Re: [PATCH 2/9] midx: allow marking a pack as preferred","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-15T17:01:32Z","receivedAt":"2021-02-15T17:05:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 15 2021, Taylor Blau wrote:\n\n> On Thu, Feb 11, 2021 at 08:33:14PM +0100, SZEDER Gábor wrote:\n>> > +--preferred-pack=::\n>> > +\tWhen using the `write` subcommand, optionally specify the\n>> > +\ttie-breaking pack used when multiple packs contain the same\n>> > +\tobject. Incompatible with other subcommands, including `repack`,\n>>\n>> I think this shouldn't be an option of the 'git multi-pack-index'\n>> command but an option of its 'write' subcommand.\n>\n> :-/. I wrote a lengthy response on Friday, but Gmail must have eaten it.\n>\n> The gist of my response was that the intermingling of sub-commands with\n> options from other sub-commands goes deeper than just the documentation,\n> since command-line arguments are only parsed once in\n> builtin/multi-pack-index.c.\n>\n> I explored whether or not it would be worth it to parse the common\n> options first, and then have separate options for each of the\n> sub-commands (as is done in the commit-graph builtin). But, this is\n> tricky, since we accept common options on either side of the sub-command\n> (i.e., we'd expect both 'git multi-pack-index --object-dir=... write' to\n> behave the same as 'git multi-pack-index write --object-dir=...').\n>\n> So you could let the first call to parse_options() parse all of the\n> arguments, but then specialized arguments (e.g., 'repack --batch-size')\n> would cause the parse-options API to barf because the first call to\n> parse_options() doesn't recognize '--batch-size'.\n>\n> I think the easiest way to do it would be to pass\n> PARSE_OPT_STOP_AT_NON_OPTION, and then let the subsequent calls to\n> parse_options() pass an array of option structs that also includes the\n> common options so they can be parsed on either side of the sub-command.\n>\n> Obviously this leads to a lot of rather unfortunate duplication. So,\n> I'm content to leave it all as-is, and let the multi-pack-index\n> builtin check the disallowed combinations itself (e.g., if you passed\n> '--preferred-pack' but aren't in 'write' mode, then we should complain).\n>\n> I can certainly move this piece of documentation into the 'write'\n> section, though, which should alleviate your immediate concern.\n\nI may be missing something, but...\n\nIt sounds to me like you're imagining this is more complex than it is\nbecause you don't know about some/all of parse_options_concat() or\nPARSE_OPT_KEEP_*.\n\nSee e.g. cmd_{switch,restore} in builti/checkout.c, or the entire family\nof diff-like commands where we do parse_options() followed by\nsetup_revisions(). We've got a lot of commands that parse options in a\npiecemeal manner.\n\nAt no point do you need to re-parse the options. You just have the\ncommon command parse as far as it gets, and leave anything else in\nargv/argc for sub-commands like \"write\".\n\nI think the problem is you read the builtin/commit-graph.c code, it\ncould really benefit from using parse_options_concat(), now things like\n\"object-directory\" are copy/pasted in that file. See 2087182272\n(checkout: split options[] array in three pieces, 2019-03-29) for a\ncommit which simplified that sort of code.\n\nIn this case you'd share the \"opts_multi_pack_index\" struct between the\nvarious commands, it would just have unused fields for \"write\" that\naren't used by \"verify\" or whatever.\n\nThe PARSE_OPT_STOP_AT_NON_OPTION flag isn't for what you're doing with\n\"write\" here, since as your test shows you're doing:\n\n git multi-pack-index \n\nBut PARSE_OPT_STOP_AT_NON_OPTION is for cases like \"git-remote\" that do:\n\n git multi-pack-index \n\n(Or maybe you really want the latter, and the test change isn't\nrepresentative).\n\nSo then we want to stop at the first non-option, i.e. the subcommand. I\nthink it's good practice not to emulate how \"git remote\" works for new\ncommands, which makes things a bit simpler to implement.\n\nYou say \"since we accept common options on either side of the\nsub-command\" but without PARSE_OPT_STOP_AT_NON_OPTION this works, since\nif you can parse everything you'll have \"write\" left, but if you truly\nhave unknown options you'll have more than that in argv.\n\nAll of the above shouldn't be taken as a \"your patch should change\"\ncomment, maybe it's fine as-is.\n\nI just replied because it sounded like you didn't spot how to easily use\nparse_options() to do this sort of thing. It's actually rather easy.\n"},{"id":"416957","messageId":"20210215184118.11306-1-avarab@gmail.com","threadId":"55135","inReplyTo":"87r1lhb6z7.fsf@evledraar.gmail.com","subject":"[PATCH 0/5] commit-graph: parse_options() cleanup","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-15T18:41:13Z","receivedAt":"2021-02-15T18:43:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A trivial cleanup series as a follow-up to my comments in\nhttps://lore.kernel.org/git/87r1lhb6z7.fsf@evledraar.gmail.com/\n\nÆvar Arnfjörð Bjarmason (5):\n commit-graph: define common usage with a macro\n commit-graph: remove redundant handling of -h\n commit-graph: use parse_options_concat()\n commit-graph: refactor dispatch loop for style\n commit-graph: show usage on \"commit-graph [write|verify] garbage\"\n\n builtin/commit-graph.c | 102 ++++++++++++++++++++++------------------\n t/t5318-commit-graph.sh | 7 +++\n 2 files changed, 62 insertions(+), 47 deletions(-)\n\n-- \n2.30.0.284.gd98b1dd5eaa7\n\n"},{"id":"416958","messageId":"20210215184118.11306-2-avarab@gmail.com","threadId":"55135","inReplyTo":"87r1lhb6z7.fsf@evledraar.gmail.com","subject":"[PATCH 1/5] commit-graph: define common usage with a macro","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-15T18:41:14Z","receivedAt":"2021-02-15T18:43:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Share the usage message between these three variables by using a\nmacro. Before this new options needed to copy/paste the usage\ninformation, see e.g. 809e0327f5 (builtin/commit-graph.c: introduce\n'--max-new-filters=', 2020-09-18).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/commit-graph.c | 29 +++++++++++++++--------------\n 1 file changed, 15 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex cd86315221..c3fa4fde3e 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -9,26 +9,27 @@\n #include \"progress.h\"\n #include \"tag.h\"\n \n-static char const * const builtin_commit_graph_usage[] = {\n-\tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n-\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n-\t \"\"),\n+static const char * builtin_commit_graph_verify_usage[] = {\n+#define BUILTIN_COMMIT_GRAPH_VERIFY_USAGE \\\n+\tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\")\n+\tBUILTIN_COMMIT_GRAPH_VERIFY_USAGE,\n \tNULL\n };\n \n-static const char * const builtin_commit_graph_verify_usage[] = {\n-\tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n+static const char * builtin_commit_graph_write_usage[] = {\n+#define BUILTIN_COMMIT_GRAPH_WRITE_USAGE \\\n+\tN_(\"git commit-graph write [--object-dir ] [--append] \" \\\n+\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \" \\\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \" \\\n+\t \"\")\n+\tBUILTIN_COMMIT_GRAPH_WRITE_USAGE,\n \tNULL\n };\n \n-static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n-\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n-\t \"\"),\n-\tNULL\n+static char const * const builtin_commit_graph_usage[] = {\n+\tBUILTIN_COMMIT_GRAPH_VERIFY_USAGE,\n+\tBUILTIN_COMMIT_GRAPH_WRITE_USAGE,\n+\tNULL,\n };\n \n static struct opts_commit_graph {\n-- \n2.30.0.284.gd98b1dd5eaa7\n\n"},{"id":"416959","messageId":"20210215184118.11306-3-avarab@gmail.com","threadId":"55135","inReplyTo":"87r1lhb6z7.fsf@evledraar.gmail.com","subject":"[PATCH 2/5] commit-graph: remove redundant handling of -h","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-15T18:41:15Z","receivedAt":"2021-02-15T18:43:39Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"If we don't handle the -h option here like most parse_options() users\nwe'll fall through and it'll do the right thing for us.\n\nI think this code added in 4ce58ee38d (commit-graph: create\ngit-commit-graph builtin, 2018-04-02) was always redundant,\nparse_options() did this at the time, and the commit-graph code never\nused PARSE_OPT_NO_INTERNAL_HELP.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/commit-graph.c | 4 ----\n t/t5318-commit-graph.sh | 5 +++++\n 2 files changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex c3fa4fde3e..baead04a03 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -319,10 +319,6 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\tOPT_END(),\n \t};\n \n-\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n-\t\tusage_with_options(builtin_commit_graph_usage,\n-\t\t\t\t builtin_commit_graph_options);\n-\n \tgit_config(git_default_config, NULL);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_commit_graph_options,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 2ed0c1544d..567e68bd93 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -5,6 +5,11 @@ test_description='commit graph'\n \n GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=0\n \n+test_expect_success 'usage' '\n+\ttest_expect_code 129 git commit-graph -h 2>err &&\n+\t! grep error: err\n+'\n+\n test_expect_success 'setup full repo' '\n \tmkdir full &&\n \tcd \"$TRASH_DIRECTORY/full\" &&\n-- \n2.30.0.284.gd98b1dd5eaa7\n\n"},{"id":"416960","messageId":"20210215184118.11306-5-avarab@gmail.com","threadId":"55135","inReplyTo":"87r1lhb6z7.fsf@evledraar.gmail.com","subject":"[PATCH 4/5] commit-graph: refactor dispatch loop for style","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-15T18:41:17Z","receivedAt":"2021-02-15T18:44:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"I think it's more readable to have one if/elsif/else chain here than\nthe code this replaces.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/commit-graph.c | 16 +++++++---------\n 1 file changed, 7 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex a7718b2025..66fbdb7cb1 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -334,13 +334,11 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \n \tsave_commit_buffer = 0;\n \n-\tif (argc > 0) {\n-\t\tif (!strcmp(argv[0], \"verify\"))\n-\t\t\treturn graph_verify(argc, argv);\n-\t\tif (!strcmp(argv[0], \"write\"))\n-\t\t\treturn graph_write(argc, argv);\n-\t}\n-\n-\tusage_with_options(builtin_commit_graph_usage,\n-\t\t\t builtin_commit_graph_options);\n+\tif (argc && !strcmp(argv[0], \"verify\"))\n+\t\treturn graph_verify(argc, argv);\n+\telse if (argc && !strcmp(argv[0], \"write\"))\n+\t\treturn graph_write(argc, argv);\n+\telse\n+\t\tusage_with_options(builtin_commit_graph_usage,\n+\t\t\t\t builtin_commit_graph_options);\n }\n-- \n2.30.0.284.gd98b1dd5eaa7\n\n"},{"id":"416961","messageId":"20210215184118.11306-4-avarab@gmail.com","threadId":"55135","inReplyTo":"87r1lhb6z7.fsf@evledraar.gmail.com","subject":"[PATCH 3/5] commit-graph: use parse_options_concat()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-15T18:41:16Z","receivedAt":"2021-02-15T18:44:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make use of the parse_options_concat() so we don't need to copy/paste\ncommon options like --object-dir. This is inspired by a similar change\nto \"checkout\" in 2087182272\n(checkout: split options[] array in three pieces, 2019-03-29).\n\nA minor behavior change here is that now we're going to list both\n--object-dir and --progress first, before we'd list --progress along\nwith other options.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/commit-graph.c | 43 ++++++++++++++++++++++++------------------\n 1 file changed, 25 insertions(+), 18 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex baead04a03..a7718b2025 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -44,6 +44,21 @@ static struct opts_commit_graph {\n \tint enable_changed_paths;\n } opts;\n \n+static struct option *add_common_options(struct option *prevopts)\n+{\n+\tstruct option options[] = {\n+\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n+\t\t\t N_(\"dir\"),\n+\t\t\t N_(\"the object directory to store the graph\")),\n+\t\tOPT_BOOL(0, \"progress\", &opts.progress,\n+\t\t\t N_(\"force progress reporting\")),\n+\t\tOPT_END()\n+\t};\n+\tstruct option *newopts = parse_options_concat(options, prevopts);\n+\tfree(prevopts);\n+\treturn newopts;\n+}\n+\n static struct object_directory *find_odb(struct repository *r,\n \t\t\t\t\t const char *obj_dir)\n {\n@@ -75,22 +90,20 @@ static int graph_verify(int argc, const char **argv)\n \tint fd;\n \tstruct stat st;\n \tint flags = 0;\n-\n+\tstruct option *options = NULL;\n \tstatic struct option builtin_commit_graph_verify_options[] = {\n-\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n-\t\t\t N_(\"dir\"),\n-\t\t\t N_(\"the object directory to store the graph\")),\n \t\tOPT_BOOL(0, \"shallow\", &opts.shallow,\n \t\t\t N_(\"if the commit-graph is split, only verify the tip file\")),\n-\t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n \t\tOPT_END(),\n \t};\n+\toptions = parse_options_dup(builtin_commit_graph_verify_options);\n+\toptions = add_common_options(options);\n \n \ttrace2_cmd_mode(\"verify\");\n \n \topts.progress = isatty(2);\n \targc = parse_options(argc, argv, NULL,\n-\t\t\t builtin_commit_graph_verify_options,\n+\t\t\t options,\n \t\t\t builtin_commit_graph_verify_usage, 0);\n \n \tif (!opts.obj_dir)\n@@ -205,11 +218,8 @@ static int graph_write(int argc, const char **argv)\n \tint result = 0;\n \tenum commit_graph_write_flags flags = 0;\n \tstruct progress *progress = NULL;\n-\n+\tstruct option *options = NULL;\n \tstatic struct option builtin_commit_graph_write_options[] = {\n-\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n-\t\t\tN_(\"dir\"),\n-\t\t\tN_(\"the object directory to store the graph\")),\n \t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n \t\t\tN_(\"start walk at all refs\")),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n@@ -220,7 +230,6 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n \t\tOPT_BOOL(0, \"changed-paths\", &opts.enable_changed_paths,\n \t\t\tN_(\"enable computation for changed paths\")),\n-\t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n \t\tOPT_CALLBACK_F(0, \"split\", &write_opts.split_flags, NULL,\n \t\t\tN_(\"allow writing an incremental commit-graph file\"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n@@ -236,6 +245,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\t0, write_option_max_new_filters),\n \t\tOPT_END(),\n \t};\n+\toptions = parse_options_dup(builtin_commit_graph_write_options);\n+\toptions = add_common_options(options);\n \n \topts.progress = isatty(2);\n \topts.enable_changed_paths = -1;\n@@ -249,7 +260,7 @@ static int graph_write(int argc, const char **argv)\n \tgit_config(git_commit_graph_write_config, &opts);\n \n \targc = parse_options(argc, argv, NULL,\n-\t\t\t builtin_commit_graph_write_options,\n+\t\t\t options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n \n \tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n@@ -312,12 +323,8 @@ static int graph_write(int argc, const char **argv)\n \n int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n {\n-\tstatic struct option builtin_commit_graph_options[] = {\n-\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n-\t\t\tN_(\"dir\"),\n-\t\t\tN_(\"the object directory to store the graph\")),\n-\t\tOPT_END(),\n-\t};\n+\tstruct option *no_options = parse_options_dup(NULL);\n+\tstruct option *builtin_commit_graph_options = add_common_options(no_options);\n \n \tgit_config(git_default_config, NULL);\n \targc = parse_options(argc, argv, prefix,\n-- \n2.30.0.284.gd98b1dd5eaa7\n\n"},{"id":"416962","messageId":"20210215184118.11306-6-avarab@gmail.com","threadId":"55135","inReplyTo":"87r1lhb6z7.fsf@evledraar.gmail.com","subject":"[PATCH 5/5] commit-graph: show usage on \"commit-graph [write|verify] garbage\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-15T18:41:18Z","receivedAt":"2021-02-15T18:44:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the parse_options() invocation in the commit-graph code to make\nsense. We're calling it twice, once for common options parsing, and\nthen for the sub-commands.\n\nBut we never checked if we had something leftover in argc in \"write\"\nor \"verify\", as a result we'd silently accept garbage in these\nsubcommands. Let's not do that.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/commit-graph.c | 10 ++++++++--\n t/t5318-commit-graph.sh | 4 +++-\n 2 files changed, 11 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 66fbdb7cb1..cb57771026 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -104,7 +104,10 @@ static int graph_verify(int argc, const char **argv)\n \topts.progress = isatty(2);\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options,\n-\t\t\t builtin_commit_graph_verify_usage, 0);\n+\t\t\t builtin_commit_graph_verify_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_commit_graph_verify_usage, options);\n \n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n@@ -261,7 +264,10 @@ static int graph_write(int argc, const char **argv)\n \n \targc = parse_options(argc, argv, NULL,\n \t\t\t options,\n-\t\t\t builtin_commit_graph_write_usage, 0);\n+\t\t\t builtin_commit_graph_write_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_commit_graph_write_usage, options);\n \n \tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n \t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 567e68bd93..3f1c6dbc8f 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -7,7 +7,9 @@ GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=0\n \n test_expect_success 'usage' '\n \ttest_expect_code 129 git commit-graph -h 2>err &&\n-\t! grep error: err\n+\t! grep error: err &&\n+\ttest_expect_code 129 git commit-graph write blah &&\n+\ttest_expect_code 129 git commit-graph write verify\n '\n \n test_expect_success 'setup full repo' '\n-- \n2.30.0.284.gd98b1dd5eaa7\n\n"},{"id":"416965","messageId":"YCrCt8sEFJSPE+28@nand.local","threadId":"55135","inReplyTo":"20210215184118.11306-4-avarab@gmail.com","subject":"Re: [PATCH 3/5] commit-graph: use parse_options_concat()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-15T18:51:35Z","receivedAt":"2021-02-15T18:53:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Feb 15, 2021 at 07:41:16PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> Make use of the parse_options_concat() so we don't need to copy/paste\n> common options like --object-dir. This is inspired by a similar change\n> to \"checkout\" in 2087182272\n> (checkout: split options[] array in three pieces, 2019-03-29).\n>\n> A minor behavior change here is that now we're going to list both\n> --object-dir and --progress first, before we'd list --progress along\n> with other options.\n\n\"Behavior change\" referring only to the output of `git commit-graph -h`,\nno?\n\nLooking at the code (and understanding this whole situation a little bit\nbetter), I'd think that this wouldn't cause us to parse anything\ndifferently before or after this change, right?\n\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> builtin/commit-graph.c | 43 ++++++++++++++++++++++++------------------\n> 1 file changed, 25 insertions(+), 18 deletions(-)\n>\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index baead04a03..a7718b2025 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -44,6 +44,21 @@ static struct opts_commit_graph {\n> \tint enable_changed_paths;\n> } opts;\n>\n> +static struct option *add_common_options(struct option *prevopts)\n> +{\n> +\tstruct option options[] = {\n> +\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n> +\t\t\t N_(\"dir\"),\n> +\t\t\t N_(\"the object directory to store the graph\")),\n> +\t\tOPT_BOOL(0, \"progress\", &opts.progress,\n> +\t\t\t N_(\"force progress reporting\")),\n> +\t\tOPT_END()\n> +\t};\n\nI'm nitpicking, but I wouldn't be sad to see this called \"common\"\ninstead\".\n\nCan't this also be declared statically?\n\n> +\tstruct option *newopts = parse_options_concat(options, prevopts);\n> +\tfree(prevopts);\n> +\treturn newopts;\n> +}\n> +\n> static struct object_directory *find_odb(struct repository *r,\n> \t\t\t\t\t const char *obj_dir)\n> {\n> @@ -75,22 +90,20 @@ static int graph_verify(int argc, const char **argv)\n> \tint fd;\n> \tstruct stat st;\n> \tint flags = 0;\n> -\n> +\tstruct option *options = NULL;\n> \tstatic struct option builtin_commit_graph_verify_options[] = {\n> -\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n> -\t\t\t N_(\"dir\"),\n> -\t\t\t N_(\"the object directory to store the graph\")),\n> \t\tOPT_BOOL(0, \"shallow\", &opts.shallow,\n> \t\t\t N_(\"if the commit-graph is split, only verify the tip file\")),\n> -\t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n> \t\tOPT_END(),\n> \t};\n> +\toptions = parse_options_dup(builtin_commit_graph_verify_options);\n\nAnother nitpick, but I'd rather see the initialization of \"options\" and\nits declaration be on the same line, after declaring\nbuiltin_commit_graph_verify_options.\n\n> +\toptions = add_common_options(options);\n>\n> \ttrace2_cmd_mode(\"verify\");\n>\n> \topts.progress = isatty(2);\n> \targc = parse_options(argc, argv, NULL,\n> -\t\t\t builtin_commit_graph_verify_options,\n> +\t\t\t options,\n> \t\t\t builtin_commit_graph_verify_usage, 0);\n>\n> \tif (!opts.obj_dir)\n> @@ -205,11 +218,8 @@ static int graph_write(int argc, const char **argv)\n> \tint result = 0;\n> \tenum commit_graph_write_flags flags = 0;\n> \tstruct progress *progress = NULL;\n> -\n> +\tstruct option *options = NULL;\n> \tstatic struct option builtin_commit_graph_write_options[] = {\n> -\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n> -\t\t\tN_(\"dir\"),\n> -\t\t\tN_(\"the object directory to store the graph\")),\n> \t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n> \t\t\tN_(\"start walk at all refs\")),\n> \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n> @@ -220,7 +230,6 @@ static int graph_write(int argc, const char **argv)\n> \t\t\tN_(\"include all commits already in the commit-graph file\")),\n> \t\tOPT_BOOL(0, \"changed-paths\", &opts.enable_changed_paths,\n> \t\t\tN_(\"enable computation for changed paths\")),\n> -\t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n> \t\tOPT_CALLBACK_F(0, \"split\", &write_opts.split_flags, NULL,\n> \t\t\tN_(\"allow writing an incremental commit-graph file\"),\n> \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n> @@ -236,6 +245,8 @@ static int graph_write(int argc, const char **argv)\n> \t\t\t0, write_option_max_new_filters),\n> \t\tOPT_END(),\n> \t};\n> +\toptions = parse_options_dup(builtin_commit_graph_write_options);\n> +\toptions = add_common_options(options);\n>\n> \topts.progress = isatty(2);\n> \topts.enable_changed_paths = -1;\n> @@ -249,7 +260,7 @@ static int graph_write(int argc, const char **argv)\n> \tgit_config(git_commit_graph_write_config, &opts);\n>\n> \targc = parse_options(argc, argv, NULL,\n> -\t\t\t builtin_commit_graph_write_options,\n> +\t\t\t options,\n> \t\t\t builtin_commit_graph_write_usage, 0);\n>\n> \tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n> @@ -312,12 +323,8 @@ static int graph_write(int argc, const char **argv)\n>\n> int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> {\n> -\tstatic struct option builtin_commit_graph_options[] = {\n> -\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n> -\t\t\tN_(\"dir\"),\n> -\t\t\tN_(\"the object directory to store the graph\")),\n> -\t\tOPT_END(),\n> -\t};\n> +\tstruct option *no_options = parse_options_dup(NULL);\n\nHmm. Why bother calling add_common_options at all here?\n\nThanks,\nTaylor\n"},{"id":"416966","messageId":"YCrDGhIq7kU57p1s@nand.local","threadId":"55135","inReplyTo":"20210215184118.11306-5-avarab@gmail.com","subject":"Re: [PATCH 4/5] commit-graph: refactor dispatch loop for style","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-15T18:53:14Z","receivedAt":"2021-02-15T18:54:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Feb 15, 2021 at 07:41:17PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> I think it's more readable to have one if/elsif/else chain here than\n> the code this replaces.\n\nFWIW, I find the pre-image more readable than what you are proposing\nreplacing it with here.\n\nOf course, I have no doubts about the obvious correctness of this patch;\nI'm merely suggesting that I wouldn't be sad to see us apply the first\nthree patches, and the fifth patch, but drop this one.\n\nThanks,\nTaylor\n"},{"id":"416968","messageId":"YCrGQPaAs2KGeCBG@nand.local","threadId":"55135","inReplyTo":"20210215184118.11306-6-avarab@gmail.com","subject":"Re: [PATCH 5/5] commit-graph: show usage on \"commit-graph [write|verify] garbage\"","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-15T19:06:40Z","receivedAt":"2021-02-15T19:07:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Feb 15, 2021 at 07:41:18PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> Change the parse_options() invocation in the commit-graph code to make\n> sense. We're calling it twice, once for common options parsing, and\n> then for the sub-commands.\n>\n> But we never checked if we had something leftover in argc in \"write\"\n> or \"verify\", as a result we'd silently accept garbage in these\n> subcommands. Let's not do that.\n\n...Implicit in all of this is that we need to pass\nPARSE_OPT_KEEP_UNKNOWN to have the sub-commands' call to parse_options()\nleave extra cruft alone so we can check for its existence with an \"if (argc)\".\n\nMakes sense, thanks.\n\nThanks,\nTaylor\n"},{"id":"416971","messageId":"YCrRTXFkn1lXt4Mw@nand.local","threadId":"55135","inReplyTo":"YCrCt8sEFJSPE+28@nand.local","subject":"Re: [PATCH 3/5] commit-graph: use parse_options_concat()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-15T19:53:49Z","receivedAt":"2021-02-15T19:54:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Feb 15, 2021 at 01:51:35PM -0500, Taylor Blau wrote:\n> Another nitpick, but I'd rather see the initialization of \"options\" and\n> its declaration be on the same line, after declaring\n> builtin_commit_graph_verify_options.\n\nIgnore me; the NULL initialization is important.\n\nThanks,\nTaylor\n"},{"id":"416976","messageId":"87czx1awwg.fsf@evledraar.gmail.com","threadId":"55135","inReplyTo":"YCrCt8sEFJSPE+28@nand.local","subject":"Re: [PATCH 3/5] commit-graph: use parse_options_concat()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-15T20:39:11Z","receivedAt":"2021-02-15T20:40:07Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 15 2021, Taylor Blau wrote:\n\n> On Mon, Feb 15, 2021 at 07:41:16PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> Make use of the parse_options_concat() so we don't need to copy/paste\n>> common options like --object-dir. This is inspired by a similar change\n>> to \"checkout\" in 2087182272\n>> (checkout: split options[] array in three pieces, 2019-03-29).\n>>\n>> A minor behavior change here is that now we're going to list both\n>> --object-dir and --progress first, before we'd list --progress along\n>> with other options.\n>\n> \"Behavior change\" referring only to the output of `git commit-graph -h`,\n> no?\n>\n> Looking at the code (and understanding this whole situation a little bit\n> better), I'd think that this wouldn't cause us to parse anything\n> differently before or after this change, right?\n\nIndeed, I just mean the \"-h\" or \"--invalid-opt\" output changed in the\norder we show the options in.\n\n>> Signed-off-by: Ævar Arnfjörð Bjarmason \n>> ---\n>> builtin/commit-graph.c | 43 ++++++++++++++++++++++++------------------\n>> 1 file changed, 25 insertions(+), 18 deletions(-)\n>>\n>> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n>> index baead04a03..a7718b2025 100644\n>> --- a/builtin/commit-graph.c\n>> +++ b/builtin/commit-graph.c\n>> @@ -44,6 +44,21 @@ static struct opts_commit_graph {\n>> \tint enable_changed_paths;\n>> } opts;\n>>\n>> +static struct option *add_common_options(struct option *prevopts)\n>> +{\n>> +\tstruct option options[] = {\n>> +\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n>> +\t\t\t N_(\"dir\"),\n>> +\t\t\t N_(\"the object directory to store the graph\")),\n>> +\t\tOPT_BOOL(0, \"progress\", &opts.progress,\n>> +\t\t\t N_(\"force progress reporting\")),\n>> +\t\tOPT_END()\n>> +\t};\n>\n> I'm nitpicking, but I wouldn't be sad to see this called \"common\"\n> instead\".\n>\n> Can't this also be declared statically?\n\nIt happens to work now to do that, but try it in builtin/checkout.c and\nyou'll see it blows up with a wall of \"initializer element is not\nconstant\".\n\nProbably better to be consistent in parse_options() usage than make it\nsafe for that sort of use...\n\n>> +\tstruct option *newopts = parse_options_concat(options, prevopts);\n>> +\tfree(prevopts);\n>> +\treturn newopts;\n>> +}\n>> +\n>> static struct object_directory *find_odb(struct repository *r,\n>> \t\t\t\t\t const char *obj_dir)\n>> {\n>> @@ -75,22 +90,20 @@ static int graph_verify(int argc, const char **argv)\n>> \tint fd;\n>> \tstruct stat st;\n>> \tint flags = 0;\n>> -\n>> +\tstruct option *options = NULL;\n>> \tstatic struct option builtin_commit_graph_verify_options[] = {\n>> -\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n>> -\t\t\t N_(\"dir\"),\n>> -\t\t\t N_(\"the object directory to store the graph\")),\n>> \t\tOPT_BOOL(0, \"shallow\", &opts.shallow,\n>> \t\t\t N_(\"if the commit-graph is split, only verify the tip file\")),\n>> -\t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n>> \t\tOPT_END(),\n>> \t};\n>> +\toptions = parse_options_dup(builtin_commit_graph_verify_options);\n>\n> Another nitpick, but I'd rather see the initialization of \"options\" and\n> its declaration be on the same line, after declaring\n> builtin_commit_graph_verify_options.\n\nAs you noted in your own reply \"the NULL initialization is important\",\nor more specifically: We're doing this dance here (and in other existing\ncode, e.g. checkout.c) to trampoline from the stack ot the heap.\n\n>> +\toptions = add_common_options(options);\n>>\n>> \ttrace2_cmd_mode(\"verify\");\n>>\n>> \topts.progress = isatty(2);\n>> \targc = parse_options(argc, argv, NULL,\n>> -\t\t\t builtin_commit_graph_verify_options,\n>> +\t\t\t options,\n>> \t\t\t builtin_commit_graph_verify_usage, 0);\n>>\n>> \tif (!opts.obj_dir)\n>> @@ -205,11 +218,8 @@ static int graph_write(int argc, const char **argv)\n>> \tint result = 0;\n>> \tenum commit_graph_write_flags flags = 0;\n>> \tstruct progress *progress = NULL;\n>> -\n>> +\tstruct option *options = NULL;\n>> \tstatic struct option builtin_commit_graph_write_options[] = {\n>> -\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n>> -\t\t\tN_(\"dir\"),\n>> -\t\t\tN_(\"the object directory to store the graph\")),\n>> \t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n>> \t\t\tN_(\"start walk at all refs\")),\n>> \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n>> @@ -220,7 +230,6 @@ static int graph_write(int argc, const char **argv)\n>> \t\t\tN_(\"include all commits already in the commit-graph file\")),\n>> \t\tOPT_BOOL(0, \"changed-paths\", &opts.enable_changed_paths,\n>> \t\t\tN_(\"enable computation for changed paths\")),\n>> -\t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n>> \t\tOPT_CALLBACK_F(0, \"split\", &write_opts.split_flags, NULL,\n>> \t\t\tN_(\"allow writing an incremental commit-graph file\"),\n>> \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n>> @@ -236,6 +245,8 @@ static int graph_write(int argc, const char **argv)\n>> \t\t\t0, write_option_max_new_filters),\n>> \t\tOPT_END(),\n>> \t};\n>> +\toptions = parse_options_dup(builtin_commit_graph_write_options);\n>> +\toptions = add_common_options(options);\n>>\n>> \topts.progress = isatty(2);\n>> \topts.enable_changed_paths = -1;\n>> @@ -249,7 +260,7 @@ static int graph_write(int argc, const char **argv)\n>> \tgit_config(git_commit_graph_write_config, &opts);\n>>\n>> \targc = parse_options(argc, argv, NULL,\n>> -\t\t\t builtin_commit_graph_write_options,\n>> +\t\t\t options,\n>> \t\t\t builtin_commit_graph_write_usage, 0);\n>>\n>> \tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n>> @@ -312,12 +323,8 @@ static int graph_write(int argc, const char **argv)\n>>\n>> int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n>> {\n>> -\tstatic struct option builtin_commit_graph_options[] = {\n>> -\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n>> -\t\t\tN_(\"dir\"),\n>> -\t\t\tN_(\"the object directory to store the graph\")),\n>> -\t\tOPT_END(),\n>> -\t};\n>> +\tstruct option *no_options = parse_options_dup(NULL);\n>\n> Hmm. Why bother calling add_common_options at all here?\n\nI assume you mean in this line just below what you quoted:\n\n struct option *builtin_commit_graph_options = add_common_options(no_options);\n\nDo you mean why not do the whole thing in graph_{verify,write}() and\nonly show the usage if we fail here?\n\nYeah arguably that makes more sense, but I wanted to just focus on\nrefactoring existing behavior & get rid of the copy/pasted options\nrather than start a bigger rewrite of \"maybe we shouldn't show this\nrather useless help info if we die here....\".\n"},{"id":"416977","messageId":"15c5acf3e03337f9b04b4c99591fb54dc4546771.1613422804.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1613422804.git.me@ttaylorr.com","subject":"[PATCH v2 1/4] builtin/multi-pack-index.c: inline 'flags' with options","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-15T21:01:38Z","receivedAt":"2021-02-15T21:02:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Subcommands of the 'git multi-pack-index' command (e.g., 'write',\n'verify', etc.) will want to optionally change a set of shared flags\nthat are eventually passed to the MIDX libraries.\n\nRight now, options and flags are handled separately. Inline them into\nthe same structure so that sub-commands can more easily share the\n'flags' data.\n\nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 13 ++++++-------\n 1 file changed, 6 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5bf88cd2a8..4a0ddb06c4 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -14,13 +14,12 @@ static struct opts_multi_pack_index {\n \tconst char *object_dir;\n \tunsigned long batch_size;\n \tint progress;\n+\tunsigned flags;\n } opts;\n \n int cmd_multi_pack_index(int argc, const char **argv,\n \t\t\t const char *prefix)\n {\n-\tunsigned flags = 0;\n-\n \tstatic struct option builtin_multi_pack_index_options[] = {\n \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n@@ -40,7 +39,7 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n \tif (opts.progress)\n-\t\tflags |= MIDX_PROGRESS;\n+\t\topts.flags |= MIDX_PROGRESS;\n \n \tif (argc == 0)\n \t\tusage_with_options(builtin_multi_pack_index_usage,\n@@ -55,16 +54,16 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \n \tif (!strcmp(argv[0], \"repack\"))\n \t\treturn midx_repack(the_repository, opts.object_dir,\n-\t\t\t(size_t)opts.batch_size, flags);\n+\t\t\t(size_t)opts.batch_size, opts.flags);\n \tif (opts.batch_size)\n \t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n \n \tif (!strcmp(argv[0], \"write\"))\n-\t\treturn write_midx_file(opts.object_dir, flags);\n+\t\treturn write_midx_file(opts.object_dir, opts.flags);\n \tif (!strcmp(argv[0], \"verify\"))\n-\t\treturn verify_midx_file(the_repository, opts.object_dir, flags);\n+\t\treturn verify_midx_file(the_repository, opts.object_dir, opts.flags);\n \tif (!strcmp(argv[0], \"expire\"))\n-\t\treturn expire_midx_packs(the_repository, opts.object_dir, flags);\n+\t\treturn expire_midx_packs(the_repository, opts.object_dir, opts.flags);\n \n \tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n }\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"416978","messageId":"cover.1613422804.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"87r1lhb6z7.fsf@evledraar.gmail.com","subject":"[PATCH v2 0/4] midx: split out sub-commands","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-15T21:01:28Z","receivedAt":"2021-02-15T21:02:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here's a few patches that we could add to the beginning of this series,\nor queue up separately.\n\nI think that these are all fairly straightforward, but it would be good\nto have Ævar take a look and make sure I'm not doing anything wrong\nhere.\n\nI'll plan to send a v2 of the reverse index series in a few days with\nthese four new patches at the beginning.\n\nTaylor Blau (4):\n builtin/multi-pack-index.c: inline 'flags' with options\n builtin/multi-pack-index.c: don't handle 'progress' separately\n builtin/multi-pack-index.c: define common usage with a macro\n builtin/multi-pack-index.c: split sub-commands\n\n builtin/multi-pack-index.c | 155 +++++++++++++++++++++++++++++--------\n 1 file changed, 124 insertions(+), 31 deletions(-)\n\n--\n2.30.0.667.g81c0cbc6fd\n"},{"id":"416980","messageId":"abcc7fb7312b349562fe6d13e2250496e107c9ed.1613422804.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1613422804.git.me@ttaylorr.com","subject":"[PATCH v2 2/4] builtin/multi-pack-index.c: don't handle 'progress' separately","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-15T21:01:42Z","receivedAt":"2021-02-15T21:02:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that there is a shared 'flags' member in the options structure,\nthere is no need to keep track of whether to force progress or not,\nsince ultimately the decision of whether or not to show a progress meter\nis controlled by a bit in the flags member.\n\nManipulate that bit directly, and drop the now-unnecessary 'progress'\nfield while we're at it.\n\nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 8 +++-----\n 1 file changed, 3 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 4a0ddb06c4..c70f020d8f 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -13,7 +13,6 @@ static char const * const builtin_multi_pack_index_usage[] = {\n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n \tunsigned long batch_size;\n-\tint progress;\n \tunsigned flags;\n } opts;\n \n@@ -23,7 +22,7 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tstatic struct option builtin_multi_pack_index_options[] = {\n \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n-\t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n+\t\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"), MIDX_PROGRESS),\n \t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n \t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n \t\tOPT_END(),\n@@ -31,15 +30,14 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \n \tgit_config(git_default_config, NULL);\n \n-\topts.progress = isatty(2);\n+\tif (isatty(2))\n+\t\topts.flags |= MIDX_PROGRESS;\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_multi_pack_index_options,\n \t\t\t builtin_multi_pack_index_usage, 0);\n \n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n-\tif (opts.progress)\n-\t\topts.flags |= MIDX_PROGRESS;\n \n \tif (argc == 0)\n \t\tusage_with_options(builtin_multi_pack_index_usage,\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"416979","messageId":"b369262cf9a49454d2c9c9dd809fac2314e041c7.1613422804.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1613422804.git.me@ttaylorr.com","subject":"[PATCH v2 3/4] builtin/multi-pack-index.c: define common usage with a macro","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-15T21:01:46Z","receivedAt":"2021-02-15T21:02:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Factor out the usage message into pieces corresponding to each mode.\nThis avoids options specific to one sub-command from being shared with\nanother in the usage.\n\nA subsequent commit will use these #define macros to have usage\nvariables for each sub-command without duplicating their contents.\n\nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 17 ++++++++++++++++-\n 1 file changed, 16 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex c70f020d8f..eea498e026 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,8 +5,23 @@\n #include \"midx.h\"\n #include \"trace2.h\"\n \n+#define BUILTIN_MIDX_WRITE_USAGE \\\n+\tN_(\"git multi-pack-index [] write\")\n+\n+#define BUILTIN_MIDX_VERIFY_USAGE \\\n+\tN_(\"git multi-pack-index [] verify\")\n+\n+#define BUILTIN_MIDX_EXPIRE_USAGE \\\n+\tN_(\"git multi-pack-index [] expire\")\n+\n+#define BUILTIN_MIDX_REPACK_USAGE \\\n+\tN_(\"git multi-pack-index [] repack [--batch-size=]\")\n+\n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [] (write|verify|expire|repack --batch-size=)\"),\n+\tBUILTIN_MIDX_WRITE_USAGE,\n+\tBUILTIN_MIDX_VERIFY_USAGE,\n+\tBUILTIN_MIDX_EXPIRE_USAGE,\n+\tBUILTIN_MIDX_REPACK_USAGE,\n \tNULL\n };\n \n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"416981","messageId":"16f33e41388ed431f70e09ef68717bd30fbee67f.1613422804.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1613422804.git.me@ttaylorr.com","subject":"[PATCH v2 4/4] builtin/multi-pack-index.c: split sub-commands","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-15T21:01:49Z","receivedAt":"2021-02-15T21:02:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Handle sub-commands of the 'git multi-pack-index' builtin (e.g.,\n\"write\", \"repack\", etc.) separately from one another. This allows\nsub-commands with unique options, without forcing cmd_multi_pack_index()\nto reject invalid combinations itself.\n\nThis comes at the cost of some duplication and boilerplate. Luckily, the\nduplication is reduced to a minimum, since common options are shared\namong sub-commands due to a suggestion by Ævar. (Sub-commands do have to\nretain the common options, too, since this builtin accepts common\noptions on either side of the sub-command).\n\nRoughly speaking, cmd_multi_pack_index() parses options (including\ncommon ones), and stops at the first non-option, which is the\nsub-command. It then dispatches to the appropriate sub-command, which\nparses the remaining options (also including common options).\n\nUnknown options are kept by the sub-commands in order to detect their\npresence (and complain that too many arguments were given).\n\nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 131 ++++++++++++++++++++++++++++++-------\n 1 file changed, 106 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex eea498e026..caf0248a98 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,17 +5,33 @@\n #include \"midx.h\"\n #include \"trace2.h\"\n \n+static char const * const builtin_multi_pack_index_write_usage[] = {\n #define BUILTIN_MIDX_WRITE_USAGE \\\n \tN_(\"git multi-pack-index [] write\")\n+\tBUILTIN_MIDX_WRITE_USAGE,\n+\tNULL\n+};\n \n+static char const * const builtin_multi_pack_index_verify_usage[] = {\n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n+\tBUILTIN_MIDX_VERIFY_USAGE,\n+\tNULL\n+};\n \n+static char const * const builtin_multi_pack_index_expire_usage[] = {\n #define BUILTIN_MIDX_EXPIRE_USAGE \\\n \tN_(\"git multi-pack-index [] expire\")\n+\tBUILTIN_MIDX_EXPIRE_USAGE,\n+\tNULL\n+};\n \n+static char const * const builtin_multi_pack_index_repack_usage[] = {\n #define BUILTIN_MIDX_REPACK_USAGE \\\n \tN_(\"git multi-pack-index [] repack [--batch-size=]\")\n+\tBUILTIN_MIDX_REPACK_USAGE,\n+\tNULL\n+};\n \n static char const * const builtin_multi_pack_index_usage[] = {\n \tBUILTIN_MIDX_WRITE_USAGE,\n@@ -31,25 +47,99 @@ static struct opts_multi_pack_index {\n \tunsigned flags;\n } opts;\n \n-int cmd_multi_pack_index(int argc, const char **argv,\n-\t\t\t const char *prefix)\n+static struct option common_opts[] = {\n+\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n+\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n+\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"), MIDX_PROGRESS),\n+\tOPT_END(),\n+};\n+\n+static struct option *add_common_options(struct option *prev)\n {\n-\tstatic struct option builtin_multi_pack_index_options[] = {\n-\t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n-\t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n-\t\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"), MIDX_PROGRESS),\n+\tstruct option *with_common = parse_options_concat(common_opts, prev);\n+\tfree(prev);\n+\treturn with_common;\n+}\n+\n+static int cmd_multi_pack_index_write(int argc, const char **argv)\n+{\n+\tstruct option *options = common_opts;\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options, builtin_multi_pack_index_write_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n+\t\t\t\t options);\n+\n+\treturn write_midx_file(opts.object_dir, opts.flags);\n+}\n+\n+static int cmd_multi_pack_index_verify(int argc, const char **argv)\n+{\n+\tstruct option *options = common_opts;\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options, builtin_multi_pack_index_verify_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_multi_pack_index_verify_usage,\n+\t\t\t\t options);\n+\n+\treturn verify_midx_file(the_repository, opts.object_dir, opts.flags);\n+}\n+\n+static int cmd_multi_pack_index_expire(int argc, const char **argv)\n+{\n+\tstruct option *options = common_opts;\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options, builtin_multi_pack_index_expire_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_multi_pack_index_expire_usage,\n+\t\t\t\t options);\n+\n+\treturn expire_midx_packs(the_repository, opts.object_dir, opts.flags);\n+}\n+\n+static int cmd_multi_pack_index_repack(int argc, const char **argv)\n+{\n+\tstruct option *options;\n+\tstatic struct option builtin_multi_pack_index_repack_options[] = {\n \t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n \t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n \t\tOPT_END(),\n \t};\n \n+\toptions = parse_options_dup(builtin_multi_pack_index_repack_options);\n+\toptions = add_common_options(options);\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options,\n+\t\t\t builtin_multi_pack_index_repack_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_multi_pack_index_repack_usage,\n+\t\t\t\t options);\n+\n+\treturn midx_repack(the_repository, opts.object_dir,\n+\t\t\t (size_t)opts.batch_size, opts.flags);\n+}\n+\n+int cmd_multi_pack_index(int argc, const char **argv,\n+\t\t\t const char *prefix)\n+{\n+\tstruct option *builtin_multi_pack_index_options = common_opts;\n+\n \tgit_config(git_default_config, NULL);\n \n \tif (isatty(2))\n \t\topts.flags |= MIDX_PROGRESS;\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_multi_pack_index_options,\n-\t\t\t builtin_multi_pack_index_usage, 0);\n+\t\t\t builtin_multi_pack_index_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n@@ -58,25 +148,16 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\tusage_with_options(builtin_multi_pack_index_usage,\n \t\t\t\t builtin_multi_pack_index_options);\n \n-\tif (argc > 1) {\n-\t\tdie(_(\"too many arguments\"));\n-\t\treturn 1;\n-\t}\n-\n \ttrace2_cmd_mode(argv[0]);\n \n \tif (!strcmp(argv[0], \"repack\"))\n-\t\treturn midx_repack(the_repository, opts.object_dir,\n-\t\t\t(size_t)opts.batch_size, opts.flags);\n-\tif (opts.batch_size)\n-\t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n-\n-\tif (!strcmp(argv[0], \"write\"))\n-\t\treturn write_midx_file(opts.object_dir, opts.flags);\n-\tif (!strcmp(argv[0], \"verify\"))\n-\t\treturn verify_midx_file(the_repository, opts.object_dir, opts.flags);\n-\tif (!strcmp(argv[0], \"expire\"))\n-\t\treturn expire_midx_packs(the_repository, opts.object_dir, opts.flags);\n-\n-\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n+\t\treturn cmd_multi_pack_index_repack(argc, argv);\n+\telse if (!strcmp(argv[0], \"write\"))\n+\t\treturn cmd_multi_pack_index_write(argc, argv);\n+\telse if (!strcmp(argv[0], \"verify\"))\n+\t\treturn cmd_multi_pack_index_verify(argc, argv);\n+\telse if (!strcmp(argv[0], \"expire\"))\n+\t\treturn cmd_multi_pack_index_expire(argc, argv);\n+\telse\n+\t\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n }\n-- \n2.30.0.667.g81c0cbc6fd\n"},{"id":"416983","messageId":"87a6s5au4b.fsf@evledraar.gmail.com","threadId":"55135","inReplyTo":"abcc7fb7312b349562fe6d13e2250496e107c9ed.1613422804.git.me@ttaylorr.com","subject":"Re: [PATCH v2 2/4] builtin/multi-pack-index.c: don't handle 'progress' separately","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-15T21:39:16Z","receivedAt":"2021-02-15T21:40:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 15 2021, Taylor Blau wrote:\n\n> @@ -31,15 +30,14 @@ int cmd_multi_pack_index(int argc, const char **argv,\n> \n> \tgit_config(git_default_config, NULL);\n> \n> -\topts.progress = isatty(2);\n> +\tif (isatty(2))\n> +\t\topts.flags |= MIDX_PROGRESS;\n> \targc = parse_options(argc, argv, prefix,\n> \t\t\t builtin_multi_pack_index_options,\n> \t\t\t builtin_multi_pack_index_usage, 0);\n> \n> \tif (!opts.object_dir)\n> \t\topts.object_dir = get_object_directory();\n> -\tif (opts.progress)\n> -\t\topts.flags |= MIDX_PROGRESS;\n\n\nFunnily enough we could also just do:\n\n opts.flags = isatty(2);\n\nSince there's a grand total of one flag it knows about, and\nMIDX_PROGRESS is defined as 1.\n\nNot the problem of this series really, just a nit: In efbc3aee08d (midx:\nadd MIDX_PROGRESS flag, 2019-10-21) we added this flag, and around the\nsame time the similar commit-graph code got refactored to have an enum\nof flags in 5af80394521 (commit-graph: collapse parameters into flags,\n2019-06-12).\n\nI prefer the commit-graph way of having a clean boundary between the two\na bit more, and then just setting a flag based on an OPT_BOOL...\n"},{"id":"416984","messageId":"YCrrauGOiVkpM2fP@nand.local","threadId":"55135","inReplyTo":"87a6s5au4b.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 2/4] builtin/multi-pack-index.c: don't handle 'progress' separately","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-15T21:45:14Z","receivedAt":"2021-02-15T21:46:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Feb 15, 2021 at 10:39:16PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Mon, Feb 15 2021, Taylor Blau wrote:\n>\n> > @@ -31,15 +30,14 @@ int cmd_multi_pack_index(int argc, const char **argv,\n> >\n> > \tgit_config(git_default_config, NULL);\n> >\n> > -\topts.progress = isatty(2);\n> > +\tif (isatty(2))\n> > +\t\topts.flags |= MIDX_PROGRESS;\n> > \targc = parse_options(argc, argv, prefix,\n> > \t\t\t builtin_multi_pack_index_options,\n> > \t\t\t builtin_multi_pack_index_usage, 0);\n> >\n> > \tif (!opts.object_dir)\n> > \t\topts.object_dir = get_object_directory();\n> > -\tif (opts.progress)\n> > -\t\topts.flags |= MIDX_PROGRESS;\n>\n>\n> Funnily enough we could also just do:\n>\n> opts.flags = isatty(2);\n>\n> Since there's a grand total of one flag it knows about, and\n> MIDX_PROGRESS is defined as 1.\n\n:-). I have a handful of branches that add some new flags (including the\noriginal series I sent down-thread), so I'm not sure that I'm in favor\nof this (admittedly cute) hack.\n\n> Not the problem of this series really, just a nit: In efbc3aee08d (midx:\n> add MIDX_PROGRESS flag, 2019-10-21) we added this flag, and around the\n> same time the similar commit-graph code got refactored to have an enum\n> of flags in 5af80394521 (commit-graph: collapse parameters into flags,\n> 2019-06-12).\n\nHmm. I don't really have a strong opinion either way. I'd like to avoid\nsteering too far away from my original goal of multi-pack reverse\nindexes, at least for now...\n\n> I prefer the commit-graph way of having a clean boundary between the two\n> a bit more, and then just setting a flag based on an OPT_BOOL...\n\nMe too. But if you can part ways with it, it cuts down on the code\nduplication (since callers in the sub-commands don't have to set that\nbit on the flags themselves).\n\nOTOH, we could keep half of this change and store the flags in the\noptions structure in addition to the progress field, then set the\nappropriate bit in \"flags\" in cmd_builtin_multi_pack_index().\n\nBut I think at that point you're already sharing the flags field\neverywhere, so you're just as well off to have something like what's\nwritten in this patch here.\n\nI don't have strong feelings either way.\n\nThanks,\nTaylor\n"},{"id":"416985","messageId":"877dn9atew.fsf@evledraar.gmail.com","threadId":"55135","inReplyTo":"16f33e41388ed431f70e09ef68717bd30fbee67f.1613422804.git.me@ttaylorr.com","subject":"Re: [PATCH v2 4/4] builtin/multi-pack-index.c: split sub-commands","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-15T21:54:31Z","receivedAt":"2021-02-15T21:55:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 15 2021, Taylor Blau wrote:\n\n> \ttrace2_cmd_mode(argv[0]);\n\nNot a new issue, but curious that in the commit-graph.c code we'll first\nvalidate, but here write garbage into the trace2_cmd_mode() before\npotentially dying.\n\n> \n> \tif (!strcmp(argv[0], \"repack\"))\n> -\t\treturn midx_repack(the_repository, opts.object_dir,\n> -\t\t\t(size_t)opts.batch_size, opts.flags);\n> -\tif (opts.batch_size)\n> -\t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n> -\n> -\tif (!strcmp(argv[0], \"write\"))\n> -\t\treturn write_midx_file(opts.object_dir, opts.flags);\n> -\tif (!strcmp(argv[0], \"verify\"))\n> -\t\treturn verify_midx_file(the_repository, opts.object_dir, opts.flags);\n> -\tif (!strcmp(argv[0], \"expire\"))\n> -\t\treturn expire_midx_packs(the_repository, opts.object_dir, opts.flags);\n> -\n> -\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n> +\t\treturn cmd_multi_pack_index_repack(argc, argv);\n> +\telse if (!strcmp(argv[0], \"write\"))\n> +\t\treturn cmd_multi_pack_index_write(argc, argv);\n> +\telse if (!strcmp(argv[0], \"verify\"))\n> +\t\treturn cmd_multi_pack_index_verify(argc, argv);\n> +\telse if (!strcmp(argv[0], \"expire\"))\n> +\t\treturn cmd_multi_pack_index_expire(argc, argv);\n> +\telse\n> +\t\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n\nI realize this is the existing behavior, but let's just make this die()\nbe the usage_with_options() we emit above in this case?\n\nSo maybe this on top?\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex caf0248a98..6f9223d538 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -65,6 +65,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n {\n \tstruct option *options = common_opts;\n \n+\ttrace2_cmd_mode(argv[0]);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options, builtin_multi_pack_index_write_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n@@ -79,6 +81,8 @@ static int cmd_multi_pack_index_verify(int argc, const char **argv)\n {\n \tstruct option *options = common_opts;\n \n+\ttrace2_cmd_mode(argv[0]);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options, builtin_multi_pack_index_verify_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n@@ -93,6 +97,8 @@ static int cmd_multi_pack_index_expire(int argc, const char **argv)\n {\n \tstruct option *options = common_opts;\n \n+\ttrace2_cmd_mode(argv[0]);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options, builtin_multi_pack_index_expire_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n@@ -112,6 +118,8 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv)\n \t\tOPT_END(),\n \t};\n \n+\ttrace2_cmd_mode(argv[0]);\n+\n \toptions = parse_options_dup(builtin_multi_pack_index_repack_options);\n \toptions = add_common_options(options);\n \n@@ -144,20 +152,15 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n \n-\tif (argc == 0)\n-\t\tusage_with_options(builtin_multi_pack_index_usage,\n-\t\t\t\t builtin_multi_pack_index_options);\n-\n-\ttrace2_cmd_mode(argv[0]);\n-\n-\tif (!strcmp(argv[0], \"repack\"))\n+\tif (argc && !strcmp(argv[0], \"repack\"))\n \t\treturn cmd_multi_pack_index_repack(argc, argv);\n-\telse if (!strcmp(argv[0], \"write\"))\n+\telse if (argc && !strcmp(argv[0], \"write\"))\n \t\treturn cmd_multi_pack_index_write(argc, argv);\n-\telse if (!strcmp(argv[0], \"verify\"))\n+\telse if (argc && !strcmp(argv[0], \"verify\"))\n \t\treturn cmd_multi_pack_index_verify(argc, argv);\n-\telse if (!strcmp(argv[0], \"expire\"))\n+\telse if (argc && !strcmp(argv[0], \"expire\"))\n \t\treturn cmd_multi_pack_index_expire(argc, argv);\n \telse\n-\t\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n+\t\tusage_with_options(builtin_multi_pack_index_usage,\n+\t\t\t\t builtin_multi_pack_index_options);\n }\n\n"},{"id":"416987","messageId":"YCr3DY5gawPC+G38@nand.local","threadId":"55135","inReplyTo":"877dn9atew.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 4/4] builtin/multi-pack-index.c: split sub-commands","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-15T22:34:53Z","receivedAt":"2021-02-15T22:35:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Feb 15, 2021 at 10:54:31PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Mon, Feb 15 2021, Taylor Blau wrote:\n>\n> > \ttrace2_cmd_mode(argv[0]);\n>\n> Not a new issue, but curious that in the commit-graph.c code we'll first\n> validate, but here write garbage into the trace2_cmd_mode() before\n> potentially dying.\n\nYeah, that's a good catch.\n\n> I realize this is the existing behavior, but let's just make this die()\n> be the usage_with_options() we emit above in this case?\n>\n> So maybe this on top?\n\nI split this into two patches: one to move the trace2_cmd_mode() calls\naround, and another to replace the final 'die()' with the usage text.\n\nLike I said in my review of your patches to the commit-graph builtin\nhere:\n\n https://lore.kernel.org/git/YCrDGhIq7kU57p1s@nand.local/\n\nI don't find the 'if (argc && ...)' style more readable, so the second\npatch looks like this instead:\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5ab80bc722..ce4f1a0bcb 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -165,5 +165,6 @@ int cmd_multi_pack_index(int argc, const char **argv,\n else if (!strcmp(argv[0], \"expire\"))\n return cmd_multi_pack_index_expire(argc, argv);\n else\n- die(_(\"unrecognized subcommand: %s\"), argv[0]);\n+ usage_with_options(builtin_multi_pack_index_usage,\n+ builtin_multi_pack_index_options);\n }\n\nIs it OK if I use your Signed-off-by on both of those two new patches?\n\nThanks,\nTaylor\n"},{"id":"416988","messageId":"874kidapv7.fsf@evledraar.gmail.com","threadId":"55135","inReplyTo":"YCr3DY5gawPC+G38@nand.local","subject":"Re: [PATCH v2 4/4] builtin/multi-pack-index.c: split sub-commands","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-15T23:11:08Z","receivedAt":"2021-02-15T23:12:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 15 2021, Taylor Blau wrote:\n\n> On Mon, Feb 15, 2021 at 10:54:31PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Mon, Feb 15 2021, Taylor Blau wrote:\n>>\n>> > \ttrace2_cmd_mode(argv[0]);\n>>\n>> Not a new issue, but curious that in the commit-graph.c code we'll first\n>> validate, but here write garbage into the trace2_cmd_mode() before\n>> potentially dying.\n>\n> Yeah, that's a good catch.\n>\n>> I realize this is the existing behavior, but let's just make this die()\n>> be the usage_with_options() we emit above in this case?\n>>\n>> So maybe this on top?\n>\n> I split this into two patches: one to move the trace2_cmd_mode() calls\n> around, and another to replace the final 'die()' with the usage text.\n\nThanks for picking it up.\n\n> Like I said in my review of your patches to the commit-graph builtin\n> here:\n>\n> https://lore.kernel.org/git/YCrDGhIq7kU57p1s@nand.local/\n>\n> I don't find the 'if (argc && ...)' style more readable, so the second\n> patch looks like this instead:\n\n*Nod* FWIW (and this is getting way to nit-y) I don't disagree with you\nabout the \"argc &&\" being not very readable,\n\nI just lean more on the side of getting rid of duplicate branches,\nyou'll still need the if (!argc) usage(...) case above without that\npattern, or some replacement for it.\n\nBut we can have our cake (not re-check argc all the time) and eat it too\n(not copy/paste usage_with_options()). Isn't it beautiful?\n \n diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n index caf0248a98..7ff50439f8 100644\n --- a/builtin/multi-pack-index.c\n +++ b/builtin/multi-pack-index.c\n @@ -144,12 +144,8 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n \n -\tif (argc == 0)\n -\t\tusage_with_options(builtin_multi_pack_index_usage,\n -\t\t\t\t builtin_multi_pack_index_options);\n -\n -\ttrace2_cmd_mode(argv[0]);\n -\n +\tif (!argc)\n +\t\tgoto usage;\n \tif (!strcmp(argv[0], \"repack\"))\n \t\treturn cmd_multi_pack_index_repack(argc, argv);\n \telse if (!strcmp(argv[0], \"write\"))\n @@ -159,5 +155,7 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \telse if (!strcmp(argv[0], \"expire\"))\n \t\treturn cmd_multi_pack_index_expire(argc, argv);\n \telse\n -\t\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n +usage:\n +\t\tusage_with_options(builtin_multi_pack_index_usage,\n +\t\t\t\t builtin_multi_pack_index_options);\n }\n \n:)\n\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> index 5ab80bc722..ce4f1a0bcb 100644\n> --- a/builtin/multi-pack-index.c\n> +++ b/builtin/multi-pack-index.c\n> @@ -165,5 +165,6 @@ int cmd_multi_pack_index(int argc, const char **argv,\n> else if (!strcmp(argv[0], \"expire\"))\n> return cmd_multi_pack_index_expire(argc, argv);\n> else\n> - die(_(\"unrecognized subcommand: %s\"), argv[0]);\n> + usage_with_options(builtin_multi_pack_index_usage,\n> + builtin_multi_pack_index_options);\n> }\n>\n> Is it OK if I use your Signed-off-by on both of those two new patches?\n\nYes please, should have included it to begin with.\n"},{"id":"416992","messageId":"YCsIfbgaA+NwHZOS@nand.local","threadId":"55135","inReplyTo":"874kidapv7.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 4/4] builtin/multi-pack-index.c: split sub-commands","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-15T23:49:17Z","receivedAt":"2021-02-15T23:52:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Feb 16, 2021 at 12:11:08AM +0100, Ævar Arnfjörð Bjarmason wrote:\n> > I split this into two patches: one to move the trace2_cmd_mode() calls\n> > around, and another to replace the final 'die()' with the usage text.\n>\n> Thanks for picking it up.\n\nOf course. This has been quite a fun digression :-).\n\n> > Like I said in my review of your patches to the commit-graph builtin\n> > here:\n> >\n> > https://lore.kernel.org/git/YCrDGhIq7kU57p1s@nand.local/\n> >\n> > I don't find the 'if (argc && ...)' style more readable, so the second\n> > patch looks like this instead:\n>\n> *Nod* FWIW (and this is getting way to nit-y) I don't disagree with you\n> about the \"argc &&\" being not very readable,\n>\n> I just lean more on the side of getting rid of duplicate branches,\n> you'll still need the if (!argc) usage(...) case above without that\n> pattern, or some replacement for it.\n>\n> But we can have our cake (not re-check argc all the time) and eat it too\n> (not copy/paste usage_with_options()). Isn't it beautiful?\n\nHeh; I'm not sure that I'd call adding a goto \"beautiful\", but I\nactually do find this one more readable. I dunno, honestly, I'm happy to\nsquash it in to the last commit on top, but honestly I don't really care\nstrongly one way or another ;).\n\n> > Is it OK if I use your Signed-off-by on both of those two new patches?\n>\n> Yes please, should have included it to begin with.\n\nThanks, and no worries.\n\nThanks,\nTaylor\n"},{"id":"417013","messageId":"8ac1dcc5-c21c-4c93-b762-1cc0a27bbeae@gmail.com","threadId":"55135","inReplyTo":"20210215184118.11306-2-avarab@gmail.com","subject":"Re: [PATCH 1/5] commit-graph: define common usage with a macro","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-16T11:33:16Z","receivedAt":"2021-02-16T11:35:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2021 1:41 PM, Ævar Arnfjörð Bjarmason wrote:\n> +static const char * builtin_commit_graph_verify_usage[] = {\n> +#define BUILTIN_COMMIT_GRAPH_VERIFY_USAGE \\\n> +\tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\")\n> +\tBUILTIN_COMMIT_GRAPH_VERIFY_USAGE,\n> \tNULL\n> };\n> \n> +static const char * builtin_commit_graph_write_usage[] = {\n> +#define BUILTIN_COMMIT_GRAPH_WRITE_USAGE \\\n> +\tN_(\"git commit-graph write [--object-dir ] [--append] \" \\\n> +\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \" \\\n> +\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \" \\\n> +\t \"\")\n> +\tBUILTIN_COMMIT_GRAPH_WRITE_USAGE,\n> \tNULL\n> };\n\nThis seemed very unnatural to me, but it all makes sense in the end:\n\n> +static char const * const builtin_commit_graph_usage[] = {\n> +\tBUILTIN_COMMIT_GRAPH_VERIFY_USAGE,\n> +\tBUILTIN_COMMIT_GRAPH_WRITE_USAGE,\n> +\tNULL,\n> };\n\nClever!\n\nThanks,\n-Stolee\n"},{"id":"417014","messageId":"57d86d79-e33e-053d-3282-0b36145358fb@gmail.com","threadId":"55135","inReplyTo":"20210215184118.11306-3-avarab@gmail.com","subject":"Re: [PATCH 2/5] commit-graph: remove redundant handling of -h","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-16T11:35:26Z","receivedAt":"2021-02-16T11:36:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2021 1:41 PM, Ævar Arnfjörð Bjarmason wrote:\n> +test_expect_success 'usage' '\n> +\ttest_expect_code 129 git commit-graph -h 2>err &&\n> +\t! grep error: err\n> +'\n\nI think this test already exists in t0012-help.sh, since it\ntests bogus options for all of the builtins. (I can guarantee\nthat I wouldn't have thought to add the check without some\ninstance like that.)\n\nThanks,\n-Stolee\n"},{"id":"417015","messageId":"4d0bd095-5593-33cc-4eec-24f06cc3cf2e@gmail.com","threadId":"55135","inReplyTo":"YCrDGhIq7kU57p1s@nand.local","subject":"Re: [PATCH 4/5] commit-graph: refactor dispatch loop for style","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-16T11:40:39Z","receivedAt":"2021-02-16T11:43:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2021 1:53 PM, Taylor Blau wrote:\n> On Mon, Feb 15, 2021 at 07:41:17PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> I think it's more readable to have one if/elsif/else chain here than\n>> the code this replaces.\n> \n> FWIW, I find the pre-image more readable than what you are proposing\n> replacing it with here.\n> \n> Of course, I have no doubts about the obvious correctness of this patch;\n> I'm merely suggesting that I wouldn't be sad to see us apply the first\n> three patches, and the fifth patch, but drop this one.\n\nI agree with all of your points here. I think that compared to the\ncurrent code at-rest, the new version might be preferred. It's a little\ndense, which is my only complaint.\n\nThe issue comes for the future: what if we need to add a third verb\nto 'git commit-graph'? Then extending this new option looks worse since\nwe would check 'argc' three times.\n\nThe other patches solve real readability problems or reorganize the code\nto use other concepts within the codebase. This one is much more optional.\n\nThanks,\n-Stolee\n"},{"id":"417016","messageId":"e54a532e-713c-f01e-cafe-a826230abdbf@gmail.com","threadId":"55135","inReplyTo":"20210215184118.11306-6-avarab@gmail.com","subject":"Re: [PATCH 5/5] commit-graph: show usage on \"commit-graph [write|verify] garbage\"","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-16T11:43:49Z","receivedAt":"2021-02-16T11:46:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2021 1:41 PM, Ævar Arnfjörð Bjarmason wrote:\n> test_expect_success 'usage' '\n> \ttest_expect_code 129 git commit-graph -h 2>err &&\n> -\t! grep error: err\n> +\t! grep error: err &&\n> +\ttest_expect_code 129 git commit-graph write blah &&\n> \ttest_expect_code 129 git commit-graph write verify\n> '\n\nThis extension of the test justifies its existence, so ignore my earlier\ncomment about it being redundant.\n\nThanks,\n-Stolee\n"},{"id":"417017","messageId":"030f1d2f-62eb-03c0-b23e-d491fac50463@gmail.com","threadId":"55135","inReplyTo":"YCrrauGOiVkpM2fP@nand.local","subject":"Re: [PATCH v2 2/4] builtin/multi-pack-index.c: don't handle 'progress' separately","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-16T11:47:06Z","receivedAt":"2021-02-16T11:50:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2021 4:45 PM, Taylor Blau wrote:\n> On Mon, Feb 15, 2021 at 10:39:16PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> Funnily enough we could also just do:\n>>\n>> opts.flags = isatty(2);\n>>\n>> Since there's a grand total of one flag it knows about, and\n>> MIDX_PROGRESS is defined as 1.\n> \n> :-). I have a handful of branches that add some new flags (including the\n> original series I sent down-thread), so I'm not sure that I'm in favor\n> of this (admittedly cute) hack.\n\nIt's also _wrong_ if the user passes in '--progress' but redirects stderr\nto a file. I don't know why someone would want to do that, but they\ncould, and we honor that throughout all commands.\n\nThanks,\n-Stolee\n"},{"id":"417018","messageId":"29d29633-ccbb-e534-24bc-c8b16c7df38d@gmail.com","threadId":"55135","inReplyTo":"cover.1613422804.git.me@ttaylorr.com","subject":"Re: [PATCH v2 0/4] midx: split out sub-commands","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-16T11:50:13Z","receivedAt":"2021-02-16T11:53:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2021 4:01 PM, Taylor Blau wrote:\n> Here's a few patches that we could add to the beginning of this series,\n> or queue up separately.\n> \n> I think that these are all fairly straightforward, but it would be good\n> to have Ævar take a look and make sure I'm not doing anything wrong\n> here.\n> \n> I'll plan to send a v2 of the reverse index series in a few days with\n> these four new patches at the beginning.\n\nThanks, both, for cleaning up a mess I made as a new contributor. These\npatches have been enlightening and definitely move the code into a\ncleaner and more extensible direction. Thanks!\n\n-Stolee\n"},{"id":"417042","messageId":"87sg5w9q64.fsf@evledraar.gmail.com","threadId":"55135","inReplyTo":"4d0bd095-5593-33cc-4eec-24f06cc3cf2e@gmail.com","subject":"Re: [PATCH 4/5] commit-graph: refactor dispatch loop for style","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-16T12:02:11Z","receivedAt":"2021-02-16T12:06:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 16 2021, Derrick Stolee wrote:\n\n> On 2/15/2021 1:53 PM, Taylor Blau wrote:\n>> On Mon, Feb 15, 2021 at 07:41:17PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>>> I think it's more readable to have one if/elsif/else chain here than\n>>> the code this replaces.\n>> \n>> FWIW, I find the pre-image more readable than what you are proposing\n>> replacing it with here.\n>> \n>> Of course, I have no doubts about the obvious correctness of this patch;\n>> I'm merely suggesting that I wouldn't be sad to see us apply the first\n>> three patches, and the fifth patch, but drop this one.\n>\n> I agree with all of your points here. I think that compared to the\n> current code at-rest, the new version might be preferred. It's a little\n> dense, which is my only complaint.\n>\n> The issue comes for the future: what if we need to add a third verb\n> to 'git commit-graph'? Then extending this new option looks worse since\n> we would check 'argc' three times.\n>\n> The other patches solve real readability problems or reorganize the code\n> to use other concepts within the codebase. This one is much more optional.\n\nWhat do you think about\nhttps://lore.kernel.org/git/874kidapv7.fsf@evledraar.gmail.com/ ? :)\n"},{"id":"417054","messageId":"YCvWnt3a5X7eZBwm@nand.local","threadId":"55135","inReplyTo":"29d29633-ccbb-e534-24bc-c8b16c7df38d@gmail.com","subject":"Re: [PATCH v2 0/4] midx: split out sub-commands","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-16T14:28:46Z","receivedAt":"2021-02-16T14:29:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Feb 16, 2021 at 06:50:13AM -0500, Derrick Stolee wrote:\n> On 2/15/2021 4:01 PM, Taylor Blau wrote:\n> > Here's a few patches that we could add to the beginning of this series,\n> > or queue up separately.\n> >\n> > I think that these are all fairly straightforward, but it would be good\n> > to have Ævar take a look and make sure I'm not doing anything wrong\n> > here.\n> >\n> > I'll plan to send a v2 of the reverse index series in a few days with\n> > these four new patches at the beginning.\n>\n> Thanks, both, for cleaning up a mess I made as a new contributor. These\n> patches have been enlightening and definitely move the code into a\n> cleaner and more extensible direction. Thanks!\n\nThere was hardly a mess to clean-up, and clearly this pattern is new to\nme, too :).\n\nI'm planning on resubmitting my tb/reverse-midx series as soon as it\ngets another set of reviewer eyes with these four or five patches as new\nat the beginning.\n\nI do wonder about the merge conflicts caused between this and your\nchunk-format API series. I'd rather not create such conflicts for Junio,\nand last I recall there were still some outstanding comments on that\nseries. So long as you don't think that you resolving those comments\nwould cause new conflicts, I would assume that Junio's rerere cache\nwould make applying both easy enough.\n\nIf you do think it would cause new conflicts, it may make sense for you\nto rebase your branch on mine, but I'm not sure if that's something\nyou'd want to do or not.\n\n> -Stolee\n\nThanks,\nTaylor\n"},{"id":"417082","messageId":"35448e9d-5054-43be-5dc8-beb6eb1d8339@gmail.com","threadId":"55135","inReplyTo":"87sg5w9q64.fsf@evledraar.gmail.com","subject":"Re: [PATCH 4/5] commit-graph: refactor dispatch loop for style","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-16T18:28:37Z","receivedAt":"2021-02-16T18:29:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/16/2021 7:02 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Feb 16 2021, Derrick Stolee wrote:\n>> The other patches solve real readability problems or reorganize the code\n>> to use other concepts within the codebase. This one is much more optional.\n> \n> What do you think about\n> https://lore.kernel.org/git/874kidapv7.fsf@evledraar.gmail.com/ ? :)\n\nUsing a 'goto' is a fine way to avoid a nesting level, but I'm not sure\nit \"improves readability.\" Having the tab level makes it clear that that\ncode is executed only when some condition is met, in this case \"if (argc),\"\nwhile with the 'goto' we need to know that execution was redirected.\n\nI don't feel too strongly either way. If the code was presented one way or\nthe other, I probably wouldn't recommend changing to the other mode. In\nthat sense, the change isn't necessary and causes me to break the tie in\nfavor of leaving it where it is.\n\nOf course, if someone else says \"I like this and would prefer it be used\nas an example for future contributors\" then the tie is broken in the other\nway.\n\nThanks,\n-Stolee\n"},{"id":"417694","messageId":"cover.1614193703.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1612998106.git.me@ttaylorr.com","subject":"[PATCH v2 00/15] midx: implement a multi-pack reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-24T19:09:12Z","receivedAt":"2021-02-24T19:10:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is a reroll of my series to implement a reverse index in preparation for\nmulti-pack reachability bitmaps. This version is based on a merge of\n'ds/chunked-file-api' (currently on 'next') with the tip of 'master'.\n\nThe range-diff below looks complicated, but it isn't. The main changes from last\ntime are:\n\n - A handful of new patches at the front inspired by Ævar to handle\n sub-command options separately from the main 'git multi-pack-index' builtin.\n\n - Minor tweaks to the '--show-objects' mode for the multi-pack index helper\n (the information now follows the existing output instead of preceeding it).\n\n - Many fields have been folded into the new 'write_midx_context' struct, which\n allowed for some related cleanup.\n\n - A new 'close_midx_revindex()' was added, which I discovered I need for the\n multi-pack bitmaps topic.\n\nI know that we're getting close to an -rc0, so I'm torn about what should happen\nwith this branch assuming that no major issues are uncovered during review. I\nwould be happy if this were queued soon so that we can start working on the main\ntopic which all of this is in support of, but I'm not sure how that would\ninteract with the upcoming release.\n\nIn fact, I think that the first six patches couple make it into 2.31 as-is,\nsince they have not much to do with the substance of this series and should be\nfairly uncontroversial.\n\nIn any case, thanks in advance for your review.\n\nTaylor Blau (15):\n builtin/multi-pack-index.c: inline 'flags' with options\n builtin/multi-pack-index.c: don't handle 'progress' separately\n builtin/multi-pack-index.c: define common usage with a macro\n builtin/multi-pack-index.c: split sub-commands\n builtin/multi-pack-index.c: don't enter bogus cmd_mode\n builtin/multi-pack-index.c: display usage on unrecognized command\n t/helper/test-read-midx.c: add '--show-objects'\n midx: allow marking a pack as preferred\n midx: don't free midx_name early\n midx: keep track of the checksum\n midx: make some functions non-static\n Documentation/technical: describe multi-pack reverse indexes\n pack-revindex: read multi-pack reverse indexes\n pack-write.c: extract 'write_rev_file_order'\n pack-revindex: write multi-pack reverse indexes\n\n Documentation/git-multi-pack-index.txt | 14 +-\n Documentation/technical/multi-pack-index.txt | 5 +-\n Documentation/technical/pack-format.txt | 80 +++++++\n builtin/multi-pack-index.c | 182 ++++++++++++---\n builtin/repack.c | 2 +-\n midx.c | 229 +++++++++++++++++--\n midx.h | 11 +-\n pack-revindex.c | 127 ++++++++++\n pack-revindex.h | 53 +++++\n pack-write.c | 39 +++-\n pack.h | 1 +\n packfile.c | 3 +\n t/helper/test-read-midx.c | 24 +-\n t/t5319-multi-pack-index.sh | 39 ++++\n 14 files changed, 740 insertions(+), 69 deletions(-)\n\nRange-diff against v1:\n -: ---------- > 1: 0527fa89a9 builtin/multi-pack-index.c: inline 'flags' with options\n -: ---------- > 2: a4e107b1f8 builtin/multi-pack-index.c: don't handle 'progress' separately\n -: ---------- > 3: 8679dfd212 builtin/multi-pack-index.c: define common usage with a macro\n -: ---------- > 4: bc42b56ea2 builtin/multi-pack-index.c: split sub-commands\n -: ---------- > 5: 5daa2946d3 builtin/multi-pack-index.c: don't enter bogus cmd_mode\n -: ---------- > 6: 98d9ea0770 builtin/multi-pack-index.c: display usage on unrecognized command\n 1: e36acb005d ! 7: 2fd9f4debf t/helper/test-read-midx.c: add '--show-objects'\n @@ t/helper/test-read-midx.c\n \tuint32_t i;\n \tstruct multi_pack_index *m;\n @@ t/helper/test-read-midx.c: static int read_midx_file(const char *object_dir)\n - \tif (!m)\n - \t\treturn 1;\n + \n + \tprintf(\"object-dir: %s\\n\", m->object_dir);\n \n +\tif (show_objects) {\n +\t\tstruct object_id oid;\n @@ t/helper/test-read-midx.c: static int read_midx_file(const char *object_dir)\n +\t\treturn 0;\n +\t}\n +\n - \tprintf(\"header: %08x %d %d %d %d\\n\",\n - \t m->signature,\n - \t m->version,\n -@@ t/helper/test-read-midx.c: static int read_midx_file(const char *object_dir)\n + \treturn 0;\n + }\n \n int cmd__read_midx(int argc, const char **argv)\n {\n 2: 4a358d57cf ! 8: 223b899094 midx: allow marking a pack as preferred\n @@ Documentation/git-multi-pack-index.txt: git-multi-pack-index - Write and verify\n DESCRIPTION\n -----------\n @@ Documentation/git-multi-pack-index.txt: OPTIONS\n - \tTurn progress on/off explicitly. If neither is specified, progress is\n - \tshown if standard error is connected to a terminal.\n - \n -+--preferred-pack=::\n -+\tWhen using the `write` subcommand, optionally specify the\n -+\ttie-breaking pack used when multiple packs contain the same\n -+\tobject. Incompatible with other subcommands, including `repack`,\n -+\twhich may repack the pack marked as preferred. If not given, the\n -+\tpreferred pack is inferred from an existing `multi-pack-index`,\n -+\tif one exists, otherwise the pack with the lowest mtime.\n -+\n The following subcommands are available:\n \n write::\n +-\tWrite a new MIDX file.\n ++\tWrite a new MIDX file. The following options are available for\n ++\tthe `write` sub-command:\n +++\n ++--\n ++\t--preferred-pack=::\n ++\t\tOptionally specify the tie-breaking pack used when\n ++\t\tmultiple packs contain the same object. If not given,\n ++\t\tties are broken in favor of the pack with the lowest\n ++\t\tmtime.\n ++--\n + \n + verify::\n + \tVerify the contents of the MIDX file.\n \n ## Documentation/technical/multi-pack-index.txt ##\n @@ Documentation/technical/multi-pack-index.txt: Design Details\n @@ builtin/multi-pack-index.c\n #include \"trace2.h\"\n +#include \"object-store.h\"\n \n - static char const * const builtin_multi_pack_index_usage[] = {\n - \tN_(\"git multi-pack-index [] (write|verify|expire|repack --batch-size=)\"),\n + static char const * const builtin_multi_pack_index_write_usage[] = {\n + #define BUILTIN_MIDX_WRITE_USAGE \\\n +-\tN_(\"git multi-pack-index [] write\")\n ++\tN_(\"git multi-pack-index [] write [--preferred-pack=]\")\n + \tBUILTIN_MIDX_WRITE_USAGE,\n + \tNULL\n + };\n @@ builtin/multi-pack-index.c: static char const * const builtin_multi_pack_index_usage[] = {\n \n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n +\tconst char *preferred_pack;\n \tunsigned long batch_size;\n - \tint progress;\n + \tunsigned flags;\n } opts;\n -@@ builtin/multi-pack-index.c: int cmd_multi_pack_index(int argc, const char **argv,\n - \tstatic struct option builtin_multi_pack_index_options[] = {\n - \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n - \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n -+\t\tOPT_STRING(0, \"preferred-pack\", &opts.preferred_pack, N_(\"preferred-pack\"),\n -+\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n - \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n - \t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n - \t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n -@@ builtin/multi-pack-index.c: int cmd_multi_pack_index(int argc, const char **argv,\n - \t\treturn 1;\n - \t}\n +@@ builtin/multi-pack-index.c: static struct option *add_common_options(struct option *prev)\n \n -+\tif (strcmp(argv[0], \"write\") && opts.preferred_pack)\n -+\t\tdie(_(\"'--preferred-pack' requires 'write'\"));\n + static int cmd_multi_pack_index_write(int argc, const char **argv)\n + {\n +-\tstruct option *options = common_opts;\n ++\tstruct option *options;\n ++\tstatic struct option builtin_multi_pack_index_write_options[] = {\n ++\t\tOPT_STRING(0, \"preferred-pack\", &opts.preferred_pack,\n ++\t\t\t N_(\"preferred-pack\"),\n ++\t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n ++\t\tOPT_END(),\n ++\t};\n +\n ++\toptions = parse_options_dup(builtin_multi_pack_index_write_options);\n ++\toptions = add_common_options(options);\n + \n \ttrace2_cmd_mode(argv[0]);\n \n - \tif (!strcmp(argv[0], \"repack\"))\n -@@ builtin/multi-pack-index.c: int cmd_multi_pack_index(int argc, const char **argv,\n - \t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n +@@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, const char **argv)\n + \t\tusage_with_options(builtin_multi_pack_index_write_usage,\n + \t\t\t\t options);\n \n - \tif (!strcmp(argv[0], \"write\"))\n --\t\treturn write_midx_file(opts.object_dir, flags);\n -+\t\treturn write_midx_file(opts.object_dir, opts.preferred_pack,\n -+\t\t\t\t flags);\n - \tif (!strcmp(argv[0], \"verify\"))\n - \t\treturn verify_midx_file(the_repository, opts.object_dir, flags);\n - \tif (!strcmp(argv[0], \"expire\"))\n +-\treturn write_midx_file(opts.object_dir, opts.flags);\n ++\treturn write_midx_file(opts.object_dir, opts.preferred_pack,\n ++\t\t\t opts.flags);\n + }\n + \n + static int cmd_multi_pack_index_verify(int argc, const char **argv)\n \n ## builtin/repack.c ##\n @@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix)\n @@ midx.c: static int pack_info_compare(const void *_a, const void *_b)\n +\treturn -1;\n +}\n +\n - struct pack_list {\n + struct write_midx_context {\n \tstruct pack_info *info;\n \tuint32_t nr;\n +@@ midx.c: struct write_midx_context {\n + \tuint32_t *pack_perm;\n + \tunsigned large_offsets_needed:1;\n + \tuint32_t num_large_offsets;\n ++\n ++\tint preferred_pack_idx;\n + };\n + \n + static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n @@ midx.c: struct pack_midx_entry {\n \tuint32_t pack_int_id;\n \ttime_t pack_mtime;\n @@ midx.c: static struct pack_midx_entry *get_sorted_entries(struct multi_pack_inde\n \t\t\t\tnr_fanout++;\n \t\t\t}\n \t\t}\n -@@ midx.c: static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off\n +@@ midx.c: static int write_midx_large_offsets(struct hashfile *f,\n }\n \n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n @@ midx.c: static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_l\n +\t\t\t const char *preferred_pack_name,\n +\t\t\t unsigned flags)\n {\n - \tunsigned char cur_chunk, num_chunks = 0;\n \tchar *midx_name;\n + \tuint32_t i;\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n - \tint pack_name_concat_len = 0;\n - \tint dropped_packs = 0;\n - \tint result = 0;\n -+\tint preferred_pack_idx = -1;\n - \n - \tmidx_name = get_midx_filename(object_dir);\n - \tif (safe_create_leading_directories(midx_name))\n -@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n - \tif (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop)\n + \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n --\tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);\n +-\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);\n +\tif (preferred_pack_name) {\n -+\t\tfor (i = 0; i < packs.nr; i++) {\n ++\t\tfor (i = 0; i < ctx.nr; i++) {\n +\t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n -+\t\t\t\t\t\t packs.info[i].pack_name)) {\n -+\t\t\t\tpreferred_pack_idx = i;\n ++\t\t\t\t\t\t ctx.info[i].pack_name)) {\n ++\t\t\t\tctx.preferred_pack_idx = i;\n +\t\t\t\tbreak;\n +\t\t\t}\n +\t\t}\n -+\t}\n ++\t} else\n ++\t\tctx.preferred_pack_idx = -1;\n +\n -+\tentries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries,\n -+\t\t\t\t preferred_pack_idx);\n ++\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,\n ++\t\t\t\t\t ctx.preferred_pack_idx);\n \n - \tfor (i = 0; i < nr_entries; i++) {\n - \t\tif (entries[i].offset > 0x7fffffff)\n + \tctx.large_offsets_needed = 0;\n + \tfor (i = 0; i < ctx.entries_nr; i++) {\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n - \t\t\tpack_name_concat_len += strlen(packs.info[i].pack_name) + 1;\n + \t\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n \t}\n \n +\t/*\n +\t * Recompute the preferred_pack_idx (if applicable) according to the\n +\t * permuted pack order.\n +\t */\n -+\tpreferred_pack_idx = -1;\n ++\tctx.preferred_pack_idx = -1;\n +\tif (preferred_pack_name) {\n -+\t\tpreferred_pack_idx = lookup_idx_or_pack_name(packs.info,\n -+\t\t\t\t\t\t\t packs.nr,\n ++\t\tctx.preferred_pack_idx = lookup_idx_or_pack_name(ctx.info,\n ++\t\t\t\t\t\t\t ctx.nr,\n +\t\t\t\t\t\t\t preferred_pack_name);\n -+\t\tif (preferred_pack_idx < 0)\n ++\t\tif (ctx.preferred_pack_idx < 0)\n +\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n +\t\t\t\tpreferred_pack_name);\n +\t\telse {\n -+\t\t\tuint32_t orig = packs.info[preferred_pack_idx].orig_pack_int_id;\n -+\t\t\tuint32_t perm = pack_perm[orig];\n ++\t\t\tuint32_t orig = ctx.info[ctx.preferred_pack_idx].orig_pack_int_id;\n ++\t\t\tuint32_t perm = ctx.pack_perm[orig];\n +\n +\t\t\tif (perm == PACK_EXPIRED) {\n +\t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n +\t\t\t\t\tpreferred_pack_name);\n -+\t\t\t\tpreferred_pack_idx = -1;\n ++\t\t\t\tctx.preferred_pack_idx = -1;\n +\t\t\t} else\n -+\t\t\t\tpreferred_pack_idx = perm;\n ++\t\t\t\tctx.preferred_pack_idx = perm;\n +\t\t}\n +\t}\n +\n @@ t/t5319-multi-pack-index.sh: test_expect_success 'warn on improper hash version'\n +\t\t) &&\n +\n +\t\tgit multi-pack-index --object-dir=objects \\\n -+\t\t\t--preferred-pack=test-BC-$bc.idx write 2>err &&\n ++\t\t\twrite --preferred-pack=test-BC-$bc.idx 2>err &&\n +\t\ttest_must_be_empty err &&\n +\n +\t\tofs=$(git show-index 10: 5ed47f7e3a midx: keep track of the checksum\n 5: 953beabaa4 ! 11: 0292508e12 midx: make some functions non-static\n @@ midx.c: static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n -static uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n +uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n {\n - \treturn get_be32(m->chunk_object_offsets + pos * MIDX_CHUNK_OFFSET_WIDTH);\n - }\n + \treturn get_be32(m->chunk_object_offsets +\n + \t\t\t(off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);\n \n ## midx.h ##\n @@ midx.h: struct multi_pack_index {\n 6: e64504bad6 < -: ---------- Documentation/technical: describe multi-pack reverse indexes\n -: ---------- > 12: 404d730498 Documentation/technical: describe multi-pack reverse indexes\n 7: 4c5e64c5fc ! 13: d4e01a44e7 pack-revindex: read multi-pack reverse indexes\n @@ midx.c: static uint8_t oid_version(void)\n +\t\t m->object_dir, hash_to_hex(get_midx_checksum(m)));\n +}\n +\n - struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local)\n + static int midx_read_oid_fanout(const unsigned char *chunk_start,\n + \t\t\t\tsize_t chunk_size, void *data)\n {\n - \tstruct multi_pack_index *m = NULL;\n \n ## midx.h ##\n @@ midx.h: struct multi_pack_index {\n @@ pack-revindex.c: int load_pack_revindex(struct packed_git *p)\n +\tfree(revindex_name);\n +\treturn ret;\n +}\n ++\n ++int close_midx_revindex(struct multi_pack_index *m)\n ++{\n ++\tif (!m)\n ++\t\treturn 0;\n ++\n ++\tif (munmap((void*)m->revindex_map, m->revindex_len))\n ++\t\treturn -1;\n ++\n ++\tm->revindex_map = NULL;\n ++\tm->revindex_data = NULL;\n ++\tm->revindex_len = 0;\n ++\n ++\treturn 0;\n ++}\n +\n int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n {\n @@ pack-revindex.h: struct packed_git;\n + * A negative number is returned on error.\n + */\n +int load_midx_revindex(struct multi_pack_index *m);\n ++\n ++/*\n ++ * Frees resources associated with a multi-pack reverse index.\n ++ *\n ++ * A negative number is returned on error.\n ++ */\n ++int close_midx_revindex(struct multi_pack_index *m);\n +\n /*\n * offset_to_pack_pos converts an object offset to a pack position. This\n 8: 4829b93f42 = 14: ab7012b283 pack-write.c: extract 'write_rev_file_order'\n 9: fb5954b769 ! 15: 01bd6a35c6 pack-revindex: write multi-pack reverse indexes\n @@ Commit message\n \n ## midx.c ##\n @@\n - #include \"trace2.h\"\n #include \"run-command.h\"\n #include \"repository.h\"\n + #include \"chunk-format.h\"\n +#include \"pack.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n -@@ midx.c: static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off\n - \treturn written;\n +@@ midx.c: struct write_midx_context {\n + \tuint32_t entries_nr;\n + \n + \tuint32_t *pack_perm;\n ++\tuint32_t *pack_order;\n + \tunsigned large_offsets_needed:1;\n + \tuint32_t num_large_offsets;\n + \n +@@ midx.c: static int write_midx_large_offsets(struct hashfile *f,\n + \treturn 0;\n }\n \n -+struct midx_pack_order_data {\n -+\tstruct pack_midx_entry *entries;\n -+\tuint32_t *pack_perm;\n -+};\n -+\n -+static int midx_pack_order_cmp(const void *va, const void *vb, void *_data)\n ++static int midx_pack_order_cmp(const void *va, const void *vb, void *_ctx)\n +{\n -+\tstruct midx_pack_order_data *data = _data;\n ++\tstruct write_midx_context *ctx = _ctx;\n +\n -+\tstruct pack_midx_entry *a = &data->entries[*(const uint32_t *)va];\n -+\tstruct pack_midx_entry *b = &data->entries[*(const uint32_t *)vb];\n ++\tstruct pack_midx_entry *a = &ctx->entries[*(const uint32_t *)va];\n ++\tstruct pack_midx_entry *b = &ctx->entries[*(const uint32_t *)vb];\n +\n -+\tuint32_t perm_a = data->pack_perm[a->pack_int_id];\n -+\tuint32_t perm_b = data->pack_perm[b->pack_int_id];\n ++\tuint32_t perm_a = ctx->pack_perm[a->pack_int_id];\n ++\tuint32_t perm_b = ctx->pack_perm[b->pack_int_id];\n +\n +\t/* Sort objects in the preferred pack ahead of any others. */\n +\tif (a->preferred > b->preferred)\n @@ midx.c: static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_l\n +\treturn 0;\n +}\n +\n -+static uint32_t *midx_pack_order(struct pack_midx_entry *entries,\n -+\t\t\t\t uint32_t *pack_perm,\n -+\t\t\t\t uint32_t entries_nr)\n ++static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n +{\n -+\tstruct midx_pack_order_data data;\n +\tuint32_t *pack_order;\n +\tuint32_t i;\n +\n -+\tdata.entries = entries;\n -+\tdata.pack_perm = pack_perm;\n -+\n -+\tALLOC_ARRAY(pack_order, entries_nr);\n -+\tfor (i = 0; i < entries_nr; i++)\n ++\tALLOC_ARRAY(pack_order, ctx->entries_nr);\n ++\tfor (i = 0; i < ctx->entries_nr; i++)\n +\t\tpack_order[i] = i;\n -+\tQSORT_S(pack_order, entries_nr, midx_pack_order_cmp, &data);\n ++\tQSORT_S(pack_order, ctx->entries_nr, midx_pack_order_cmp, ctx);\n +\n +\treturn pack_order;\n +}\n +\n +static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n -+\t\t\t\t uint32_t *pack_order,\n -+\t\t\t\t uint32_t entries_nr)\n ++\t\t\t\t struct write_midx_context *ctx)\n +{\n +\tstruct strbuf buf = STRBUF_INIT;\n +\n +\tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex(midx_hash));\n +\n -+\twrite_rev_file_order(buf.buf, pack_order, entries_nr, midx_hash,\n -+\t\t\t WRITE_REV);\n ++\twrite_rev_file_order(buf.buf, ctx->pack_order, ctx->entries_nr,\n ++\t\t\t midx_hash, WRITE_REV);\n +\n +\tstrbuf_release(&buf);\n +}\n @@ midx.c: static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_l\n \t\t\t struct string_list *packs_to_drop,\n \t\t\t const char *preferred_pack_name,\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n - \tstruct lock_file lk;\n - \tstruct pack_list packs;\n - \tuint32_t *pack_perm = NULL;\n -+\tuint32_t *pack_order = NULL;\n - \tuint64_t written = 0;\n - \tuint32_t chunk_ids[MIDX_MAX_CHUNKS + 1];\n - \tuint64_t chunk_offsets[MIDX_MAX_CHUNKS + 1];\n -@@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n - \t\t chunk_offsets[num_chunks]);\n \n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n + \tfree_chunkfile(cf);\n ++\n +\tif (flags & MIDX_WRITE_REV_INDEX)\n -+\t\tpack_order = midx_pack_order(entries, pack_perm, nr_entries);\n ++\t\tctx.pack_order = midx_pack_order(&ctx);\n +\n +\tif (flags & MIDX_WRITE_REV_INDEX)\n -+\t\twrite_midx_reverse_index(midx_name, midx_hash, pack_order,\n -+\t\t\t\t\t nr_entries);\n ++\t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n +\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n +\n \tcommit_lock_file(&lk);\n \n cleanup:\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n - \tfree(packs.info);\n - \tfree(entries);\n - \tfree(pack_perm);\n -+\tfree(pack_order);\n + \tfree(ctx.info);\n + \tfree(ctx.entries);\n + \tfree(ctx.pack_perm);\n ++\tfree(ctx.pack_order);\n \tfree(midx_name);\n \treturn result;\n }\n-- \n2.30.0.667.g81c0cbc6fd\n"},{"id":"417695","messageId":"0527fa89a9a8a44df4d046f4efd04fd9fc23bb91.1614193703.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1614193703.git.me@ttaylorr.com","subject":"[PATCH v2 01/15] builtin/multi-pack-index.c: inline 'flags' with options","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-24T19:09:25Z","receivedAt":"2021-02-24T19:11:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Subcommands of the 'git multi-pack-index' command (e.g., 'write',\n'verify', etc.) will want to optionally change a set of shared flags\nthat are eventually passed to the MIDX libraries.\n\nRight now, options and flags are handled separately. Inline them into\nthe same structure so that sub-commands can more easily share the\n'flags' data.\n\nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 13 ++++++-------\n 1 file changed, 6 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5bf88cd2a8..4a0ddb06c4 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -14,13 +14,12 @@ static struct opts_multi_pack_index {\n \tconst char *object_dir;\n \tunsigned long batch_size;\n \tint progress;\n+\tunsigned flags;\n } opts;\n \n int cmd_multi_pack_index(int argc, const char **argv,\n \t\t\t const char *prefix)\n {\n-\tunsigned flags = 0;\n-\n \tstatic struct option builtin_multi_pack_index_options[] = {\n \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n@@ -40,7 +39,7 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n \tif (opts.progress)\n-\t\tflags |= MIDX_PROGRESS;\n+\t\topts.flags |= MIDX_PROGRESS;\n \n \tif (argc == 0)\n \t\tusage_with_options(builtin_multi_pack_index_usage,\n@@ -55,16 +54,16 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \n \tif (!strcmp(argv[0], \"repack\"))\n \t\treturn midx_repack(the_repository, opts.object_dir,\n-\t\t\t(size_t)opts.batch_size, flags);\n+\t\t\t(size_t)opts.batch_size, opts.flags);\n \tif (opts.batch_size)\n \t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n \n \tif (!strcmp(argv[0], \"write\"))\n-\t\treturn write_midx_file(opts.object_dir, flags);\n+\t\treturn write_midx_file(opts.object_dir, opts.flags);\n \tif (!strcmp(argv[0], \"verify\"))\n-\t\treturn verify_midx_file(the_repository, opts.object_dir, flags);\n+\t\treturn verify_midx_file(the_repository, opts.object_dir, opts.flags);\n \tif (!strcmp(argv[0], \"expire\"))\n-\t\treturn expire_midx_packs(the_repository, opts.object_dir, flags);\n+\t\treturn expire_midx_packs(the_repository, opts.object_dir, opts.flags);\n \n \tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n }\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"417696","messageId":"8679dfd2121cbc75818e97eb40a27dd3af021e38.1614193703.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1614193703.git.me@ttaylorr.com","subject":"[PATCH v2 03/15] builtin/multi-pack-index.c: define common usage with a macro","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-24T19:09:34Z","receivedAt":"2021-02-24T19:12:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Factor out the usage message into pieces corresponding to each mode.\nThis avoids options specific to one sub-command from being shared with\nanother in the usage.\n\nA subsequent commit will use these #define macros to have usage\nvariables for each sub-command without duplicating their contents.\n\nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 17 ++++++++++++++++-\n 1 file changed, 16 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex c70f020d8f..eea498e026 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,8 +5,23 @@\n #include \"midx.h\"\n #include \"trace2.h\"\n \n+#define BUILTIN_MIDX_WRITE_USAGE \\\n+\tN_(\"git multi-pack-index [] write\")\n+\n+#define BUILTIN_MIDX_VERIFY_USAGE \\\n+\tN_(\"git multi-pack-index [] verify\")\n+\n+#define BUILTIN_MIDX_EXPIRE_USAGE \\\n+\tN_(\"git multi-pack-index [] expire\")\n+\n+#define BUILTIN_MIDX_REPACK_USAGE \\\n+\tN_(\"git multi-pack-index [] repack [--batch-size=]\")\n+\n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [] (write|verify|expire|repack --batch-size=)\"),\n+\tBUILTIN_MIDX_WRITE_USAGE,\n+\tBUILTIN_MIDX_VERIFY_USAGE,\n+\tBUILTIN_MIDX_EXPIRE_USAGE,\n+\tBUILTIN_MIDX_REPACK_USAGE,\n \tNULL\n };\n \n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"417697","messageId":"a4e107b1f8308b9e8e779e31fc27dcbc082ab545.1614193703.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1614193703.git.me@ttaylorr.com","subject":"[PATCH v2 02/15] builtin/multi-pack-index.c: don't handle 'progress' separately","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-24T19:09:30Z","receivedAt":"2021-02-24T19:12:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that there is a shared 'flags' member in the options structure,\nthere is no need to keep track of whether to force progress or not,\nsince ultimately the decision of whether or not to show a progress meter\nis controlled by a bit in the flags member.\n\nManipulate that bit directly, and drop the now-unnecessary 'progress'\nfield while we're at it.\n\nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 8 +++-----\n 1 file changed, 3 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 4a0ddb06c4..c70f020d8f 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -13,7 +13,6 @@ static char const * const builtin_multi_pack_index_usage[] = {\n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n \tunsigned long batch_size;\n-\tint progress;\n \tunsigned flags;\n } opts;\n \n@@ -23,7 +22,7 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tstatic struct option builtin_multi_pack_index_options[] = {\n \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n-\t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n+\t\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"), MIDX_PROGRESS),\n \t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n \t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n \t\tOPT_END(),\n@@ -31,15 +30,14 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \n \tgit_config(git_default_config, NULL);\n \n-\topts.progress = isatty(2);\n+\tif (isatty(2))\n+\t\topts.flags |= MIDX_PROGRESS;\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_multi_pack_index_options,\n \t\t\t builtin_multi_pack_index_usage, 0);\n \n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n-\tif (opts.progress)\n-\t\topts.flags |= MIDX_PROGRESS;\n \n \tif (argc == 0)\n \t\tusage_with_options(builtin_multi_pack_index_usage,\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"417698","messageId":"bc42b56ea2c3ac7ead1cad2a95ac01a7adfbb28f.1614193703.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1614193703.git.me@ttaylorr.com","subject":"[PATCH v2 04/15] builtin/multi-pack-index.c: split sub-commands","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-24T19:09:38Z","receivedAt":"2021-02-24T19:12:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Handle sub-commands of the 'git multi-pack-index' builtin (e.g.,\n\"write\", \"repack\", etc.) separately from one another. This allows\nsub-commands with unique options, without forcing cmd_multi_pack_index()\nto reject invalid combinations itself.\n\nThis comes at the cost of some duplication and boilerplate. Luckily, the\nduplication is reduced to a minimum, since common options are shared\namong sub-commands due to a suggestion by Ævar. (Sub-commands do have to\nretain the common options, too, since this builtin accepts common\noptions on either side of the sub-command).\n\nRoughly speaking, cmd_multi_pack_index() parses options (including\ncommon ones), and stops at the first non-option, which is the\nsub-command. It then dispatches to the appropriate sub-command, which\nparses the remaining options (also including common options).\n\nUnknown options are kept by the sub-commands in order to detect their\npresence (and complain that too many arguments were given).\n\nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 131 ++++++++++++++++++++++++++++++-------\n 1 file changed, 106 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex eea498e026..caf0248a98 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,17 +5,33 @@\n #include \"midx.h\"\n #include \"trace2.h\"\n \n+static char const * const builtin_multi_pack_index_write_usage[] = {\n #define BUILTIN_MIDX_WRITE_USAGE \\\n \tN_(\"git multi-pack-index [] write\")\n+\tBUILTIN_MIDX_WRITE_USAGE,\n+\tNULL\n+};\n \n+static char const * const builtin_multi_pack_index_verify_usage[] = {\n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n+\tBUILTIN_MIDX_VERIFY_USAGE,\n+\tNULL\n+};\n \n+static char const * const builtin_multi_pack_index_expire_usage[] = {\n #define BUILTIN_MIDX_EXPIRE_USAGE \\\n \tN_(\"git multi-pack-index [] expire\")\n+\tBUILTIN_MIDX_EXPIRE_USAGE,\n+\tNULL\n+};\n \n+static char const * const builtin_multi_pack_index_repack_usage[] = {\n #define BUILTIN_MIDX_REPACK_USAGE \\\n \tN_(\"git multi-pack-index [] repack [--batch-size=]\")\n+\tBUILTIN_MIDX_REPACK_USAGE,\n+\tNULL\n+};\n \n static char const * const builtin_multi_pack_index_usage[] = {\n \tBUILTIN_MIDX_WRITE_USAGE,\n@@ -31,25 +47,99 @@ static struct opts_multi_pack_index {\n \tunsigned flags;\n } opts;\n \n-int cmd_multi_pack_index(int argc, const char **argv,\n-\t\t\t const char *prefix)\n+static struct option common_opts[] = {\n+\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n+\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n+\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"), MIDX_PROGRESS),\n+\tOPT_END(),\n+};\n+\n+static struct option *add_common_options(struct option *prev)\n {\n-\tstatic struct option builtin_multi_pack_index_options[] = {\n-\t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n-\t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n-\t\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"), MIDX_PROGRESS),\n+\tstruct option *with_common = parse_options_concat(common_opts, prev);\n+\tfree(prev);\n+\treturn with_common;\n+}\n+\n+static int cmd_multi_pack_index_write(int argc, const char **argv)\n+{\n+\tstruct option *options = common_opts;\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options, builtin_multi_pack_index_write_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n+\t\t\t\t options);\n+\n+\treturn write_midx_file(opts.object_dir, opts.flags);\n+}\n+\n+static int cmd_multi_pack_index_verify(int argc, const char **argv)\n+{\n+\tstruct option *options = common_opts;\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options, builtin_multi_pack_index_verify_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_multi_pack_index_verify_usage,\n+\t\t\t\t options);\n+\n+\treturn verify_midx_file(the_repository, opts.object_dir, opts.flags);\n+}\n+\n+static int cmd_multi_pack_index_expire(int argc, const char **argv)\n+{\n+\tstruct option *options = common_opts;\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options, builtin_multi_pack_index_expire_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_multi_pack_index_expire_usage,\n+\t\t\t\t options);\n+\n+\treturn expire_midx_packs(the_repository, opts.object_dir, opts.flags);\n+}\n+\n+static int cmd_multi_pack_index_repack(int argc, const char **argv)\n+{\n+\tstruct option *options;\n+\tstatic struct option builtin_multi_pack_index_repack_options[] = {\n \t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n \t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n \t\tOPT_END(),\n \t};\n \n+\toptions = parse_options_dup(builtin_multi_pack_index_repack_options);\n+\toptions = add_common_options(options);\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options,\n+\t\t\t builtin_multi_pack_index_repack_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_multi_pack_index_repack_usage,\n+\t\t\t\t options);\n+\n+\treturn midx_repack(the_repository, opts.object_dir,\n+\t\t\t (size_t)opts.batch_size, opts.flags);\n+}\n+\n+int cmd_multi_pack_index(int argc, const char **argv,\n+\t\t\t const char *prefix)\n+{\n+\tstruct option *builtin_multi_pack_index_options = common_opts;\n+\n \tgit_config(git_default_config, NULL);\n \n \tif (isatty(2))\n \t\topts.flags |= MIDX_PROGRESS;\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_multi_pack_index_options,\n-\t\t\t builtin_multi_pack_index_usage, 0);\n+\t\t\t builtin_multi_pack_index_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n@@ -58,25 +148,16 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\tusage_with_options(builtin_multi_pack_index_usage,\n \t\t\t\t builtin_multi_pack_index_options);\n \n-\tif (argc > 1) {\n-\t\tdie(_(\"too many arguments\"));\n-\t\treturn 1;\n-\t}\n-\n \ttrace2_cmd_mode(argv[0]);\n \n \tif (!strcmp(argv[0], \"repack\"))\n-\t\treturn midx_repack(the_repository, opts.object_dir,\n-\t\t\t(size_t)opts.batch_size, opts.flags);\n-\tif (opts.batch_size)\n-\t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n-\n-\tif (!strcmp(argv[0], \"write\"))\n-\t\treturn write_midx_file(opts.object_dir, opts.flags);\n-\tif (!strcmp(argv[0], \"verify\"))\n-\t\treturn verify_midx_file(the_repository, opts.object_dir, opts.flags);\n-\tif (!strcmp(argv[0], \"expire\"))\n-\t\treturn expire_midx_packs(the_repository, opts.object_dir, opts.flags);\n-\n-\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n+\t\treturn cmd_multi_pack_index_repack(argc, argv);\n+\telse if (!strcmp(argv[0], \"write\"))\n+\t\treturn cmd_multi_pack_index_write(argc, argv);\n+\telse if (!strcmp(argv[0], \"verify\"))\n+\t\treturn cmd_multi_pack_index_verify(argc, argv);\n+\telse if (!strcmp(argv[0], \"expire\"))\n+\t\treturn cmd_multi_pack_index_expire(argc, argv);\n+\telse\n+\t\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n }\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"417699","messageId":"98d9ea0770ea38caaa71c6b9bf234c8bdde2639b.1614193703.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1614193703.git.me@ttaylorr.com","subject":"[PATCH v2 06/15] builtin/multi-pack-index.c: display usage on unrecognized command","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-24T19:09:46Z","receivedAt":"2021-02-24T19:12:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When given a sub-command that it doesn't understand, 'git\nmulti-pack-index' dies with the following message:\n\n $ git multi-pack-index bogus\n fatal: unrecognized subcommand: bogus\n\nInstead of 'die()'-ing, we can display the usage text, which is much\nmore helpful:\n\n $ git.compile multi-pack-index bogus\n usage: git multi-pack-index [] write\n or: git multi-pack-index [] verify\n or: git multi-pack-index [] expire\n or: git multi-pack-index [] repack [--batch-size=]\n\n\t--object-dir object directory containing set of packfile and pack-index pairs\n\t--progress force progress reporting\n\nWhile we're at it, clean up some duplication between the \"no sub-command\"\nand \"unrecognized sub-command\" conditionals.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 7 ++++---\n 1 file changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 9fdfe168c2..5b05e5ce39 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -153,8 +153,7 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\topts.object_dir = get_object_directory();\n \n \tif (argc == 0)\n-\t\tusage_with_options(builtin_multi_pack_index_usage,\n-\t\t\t\t builtin_multi_pack_index_options);\n+\t\tgoto usage;\n \n \tif (!strcmp(argv[0], \"repack\"))\n \t\treturn cmd_multi_pack_index_repack(argc, argv);\n@@ -165,5 +164,7 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \telse if (!strcmp(argv[0], \"expire\"))\n \t\treturn cmd_multi_pack_index_expire(argc, argv);\n \telse\n-\t\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n+usage:\n+\t\tusage_with_options(builtin_multi_pack_index_usage,\n+\t\t\t\t builtin_multi_pack_index_options);\n }\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"417700","messageId":"5daa2946d37b5662274f5565ec35ef4d169b55bf.1614193703.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1614193703.git.me@ttaylorr.com","subject":"[PATCH v2 05/15] builtin/multi-pack-index.c: don't enter bogus cmd_mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-24T19:09:42Z","receivedAt":"2021-02-24T19:12:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Even before the recent refactoring, 'git multi-pack-index' calls\n'trace2_cmd_mode()' before verifying that the sub-command is recognized.\n\nPush this call down into the individual sub-commands so that we don't\nenter a bogus command mode.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex caf0248a98..9fdfe168c2 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -65,6 +65,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n {\n \tstruct option *options = common_opts;\n \n+\ttrace2_cmd_mode(argv[0]);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options, builtin_multi_pack_index_write_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n@@ -79,6 +81,8 @@ static int cmd_multi_pack_index_verify(int argc, const char **argv)\n {\n \tstruct option *options = common_opts;\n \n+\ttrace2_cmd_mode(argv[0]);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options, builtin_multi_pack_index_verify_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n@@ -93,6 +97,8 @@ static int cmd_multi_pack_index_expire(int argc, const char **argv)\n {\n \tstruct option *options = common_opts;\n \n+\ttrace2_cmd_mode(argv[0]);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options, builtin_multi_pack_index_expire_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n@@ -115,6 +121,8 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv)\n \toptions = parse_options_dup(builtin_multi_pack_index_repack_options);\n \toptions = add_common_options(options);\n \n+\ttrace2_cmd_mode(argv[0]);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options,\n \t\t\t builtin_multi_pack_index_repack_usage,\n@@ -148,8 +156,6 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\tusage_with_options(builtin_multi_pack_index_usage,\n \t\t\t\t builtin_multi_pack_index_options);\n \n-\ttrace2_cmd_mode(argv[0]);\n-\n \tif (!strcmp(argv[0], \"repack\"))\n \t\treturn cmd_multi_pack_index_repack(argc, argv);\n \telse if (!strcmp(argv[0], \"write\"))\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"417701","messageId":"2fd9f4debff480e18c902a919750cb7b7aba66bf.1614193703.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1614193703.git.me@ttaylorr.com","subject":"[PATCH v2 07/15] t/helper/test-read-midx.c: add '--show-objects'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-24T19:09:50Z","receivedAt":"2021-02-24T19:13:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The 'read-midx' helper is used in places like t5319 to display basic\ninformation about a multi-pack-index.\n\nIn the next patch, the MIDX writing machinery will learn a new way to\nchoose from which pack an object is selected when multiple copies of\nthat object exist.\n\nTo disambiguate which pack introduces an object so that this feature can\nbe tested, add a '--show-objects' option which displays additional\ninformation about each object in the MIDX.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-midx.c | 24 ++++++++++++++++++++----\n 1 file changed, 20 insertions(+), 4 deletions(-)\n\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 2430880f78..7c2eb11a8e 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -4,7 +4,7 @@\n #include \"repository.h\"\n #include \"object-store.h\"\n \n-static int read_midx_file(const char *object_dir)\n+static int read_midx_file(const char *object_dir, int show_objects)\n {\n \tuint32_t i;\n \tstruct multi_pack_index *m;\n@@ -43,13 +43,29 @@ static int read_midx_file(const char *object_dir)\n \n \tprintf(\"object-dir: %s\\n\", m->object_dir);\n \n+\tif (show_objects) {\n+\t\tstruct object_id oid;\n+\t\tstruct pack_entry e;\n+\n+\t\tfor (i = 0; i < m->num_objects; i++) {\n+\t\t\tnth_midxed_object_oid(&oid, m, i);\n+\t\t\tfill_midx_entry(the_repository, &oid, &e, m);\n+\n+\t\t\tprintf(\"%s %\"PRIu64\"\\t%s\\n\",\n+\t\t\t oid_to_hex(&oid), e.offset, e.p->pack_name);\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\n \treturn 0;\n }\n \n int cmd__read_midx(int argc, const char **argv)\n {\n-\tif (argc != 2)\n-\t\tusage(\"read-midx \");\n+\tif (!(argc == 2 || argc == 3))\n+\t\tusage(\"read-midx [--show-objects] \");\n \n-\treturn read_midx_file(argv[1]);\n+\tif (!strcmp(argv[1], \"--show-objects\"))\n+\t\treturn read_midx_file(argv[2], 1);\n+\treturn read_midx_file(argv[1], 0);\n }\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"417702","messageId":"223b89909416ec7c5505f9cedaa80bf86ecc7b2e.1614193703.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1614193703.git.me@ttaylorr.com","subject":"[PATCH v2 08/15] midx: allow marking a pack as preferred","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-24T19:09:54Z","receivedAt":"2021-02-24T19:13:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When multiple packs in the multi-pack index contain the same object, the\nMIDX machinery must make a choice about which pack it associates with\nthat object. Prior to this patch, the lowest-ordered[1] pack was always\nselected.\n\nPack selection for duplicate objects is relatively unimportant today,\nbut it will become important for multi-pack bitmaps. This is because we\ncan only invoke the pack-reuse mechanism when all of the bits for reused\nobjects come from the reuse pack (in order to ensure that all reused\ndeltas can find their base objects in the same pack).\n\nTo encourage the pack selection process to prefer one pack over another\n(the pack to be preferred is the one a caller would like to later use as\na reuse pack), introduce the concept of a \"preferred pack\". When\nprovided, the MIDX code will always prefer an object found in a\npreferred pack over any other.\n\nNo format changes are required to store the preferred pack, since it\nwill be able to be inferred with a corresponding MIDX bitmap, by looking\nup the pack associated with the object in the first bit position (this\nordering is described in detail in a subsequent commit).\n\n[1]: the ordering is specified by MIDX internals; for our purposes we\ncan consider the \"lowest ordered\" pack to be \"the one with the\nmost-recent mtime.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 14 ++-\n Documentation/technical/multi-pack-index.txt | 5 +-\n builtin/multi-pack-index.c | 18 +++-\n builtin/repack.c | 2 +-\n midx.c | 99 ++++++++++++++++++--\n midx.h | 2 +-\n t/t5319-multi-pack-index.sh | 39 ++++++++\n 7 files changed, 161 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex eb0caa0439..ffd601bc17 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -9,7 +9,8 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n SYNOPSIS\n --------\n [verse]\n-'git multi-pack-index' [--object-dir=] [--[no-]progress] \n+'git multi-pack-index' [--object-dir=] [--[no-]progress]\n+\t[--preferred-pack=] \n \n DESCRIPTION\n -----------\n@@ -30,7 +31,16 @@ OPTIONS\n The following subcommands are available:\n \n write::\n-\tWrite a new MIDX file.\n+\tWrite a new MIDX file. The following options are available for\n+\tthe `write` sub-command:\n++\n+--\n+\t--preferred-pack=::\n+\t\tOptionally specify the tie-breaking pack used when\n+\t\tmultiple packs contain the same object. If not given,\n+\t\tties are broken in favor of the pack with the lowest\n+\t\tmtime.\n+--\n \n verify::\n \tVerify the contents of the MIDX file.\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex e8e377a59f..fb688976c4 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -43,8 +43,9 @@ Design Details\n a change in format.\n \n - The MIDX keeps only one record per object ID. If an object appears\n- in multiple packfiles, then the MIDX selects the copy in the most-\n- recently modified packfile.\n+ in multiple packfiles, then the MIDX selects the copy in the\n+ preferred packfile, otherwise selecting from the most-recently\n+ modified packfile.\n \n - If there exist packfiles in the pack directory not registered in\n the MIDX, then those packfiles are loaded into the `packed_git`\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5b05e5ce39..2329dc5ec0 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -4,10 +4,11 @@\n #include \"parse-options.h\"\n #include \"midx.h\"\n #include \"trace2.h\"\n+#include \"object-store.h\"\n \n static char const * const builtin_multi_pack_index_write_usage[] = {\n #define BUILTIN_MIDX_WRITE_USAGE \\\n-\tN_(\"git multi-pack-index [] write\")\n+\tN_(\"git multi-pack-index [] write [--preferred-pack=]\")\n \tBUILTIN_MIDX_WRITE_USAGE,\n \tNULL\n };\n@@ -43,6 +44,7 @@ static char const * const builtin_multi_pack_index_usage[] = {\n \n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n+\tconst char *preferred_pack;\n \tunsigned long batch_size;\n \tunsigned flags;\n } opts;\n@@ -63,7 +65,16 @@ static struct option *add_common_options(struct option *prev)\n \n static int cmd_multi_pack_index_write(int argc, const char **argv)\n {\n-\tstruct option *options = common_opts;\n+\tstruct option *options;\n+\tstatic struct option builtin_multi_pack_index_write_options[] = {\n+\t\tOPT_STRING(0, \"preferred-pack\", &opts.preferred_pack,\n+\t\t\t N_(\"preferred-pack\"),\n+\t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\toptions = parse_options_dup(builtin_multi_pack_index_write_options);\n+\toptions = add_common_options(options);\n \n \ttrace2_cmd_mode(argv[0]);\n \n@@ -74,7 +85,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\tusage_with_options(builtin_multi_pack_index_write_usage,\n \t\t\t\t options);\n \n-\treturn write_midx_file(opts.object_dir, opts.flags);\n+\treturn write_midx_file(opts.object_dir, opts.preferred_pack,\n+\t\t\t opts.flags);\n }\n \n static int cmd_multi_pack_index_verify(int argc, const char **argv)\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 01440de2d5..9f00806805 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -523,7 +523,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tremove_temporary_files();\n \n \tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n-\t\twrite_midx_file(get_object_directory(), 0);\n+\t\twrite_midx_file(get_object_directory(), NULL, 0);\n \n \tstring_list_clear(&names, 0);\n \tstring_list_clear(&rollback, 0);\ndiff --git a/midx.c b/midx.c\nindex 971faa8cfc..d2c56c4bc6 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -431,6 +431,24 @@ static int pack_info_compare(const void *_a, const void *_b)\n \treturn strcmp(a->pack_name, b->pack_name);\n }\n \n+static int lookup_idx_or_pack_name(struct pack_info *info,\n+\t\t\t\t uint32_t nr,\n+\t\t\t\t const char *pack_name)\n+{\n+\tuint32_t lo = 0, hi = nr;\n+\twhile (lo < hi) {\n+\t\tuint32_t mi = lo + (hi - lo) / 2;\n+\t\tint cmp = cmp_idx_or_pack_name(pack_name, info[mi].pack_name);\n+\t\tif (cmp < 0)\n+\t\t\thi = mi;\n+\t\telse if (cmp > 0)\n+\t\t\tlo = mi + 1;\n+\t\telse\n+\t\t\treturn mi;\n+\t}\n+\treturn -1;\n+}\n+\n struct write_midx_context {\n \tstruct pack_info *info;\n \tuint32_t nr;\n@@ -445,6 +463,8 @@ struct write_midx_context {\n \tuint32_t *pack_perm;\n \tunsigned large_offsets_needed:1;\n \tuint32_t num_large_offsets;\n+\n+\tint preferred_pack_idx;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -489,6 +509,7 @@ struct pack_midx_entry {\n \tuint32_t pack_int_id;\n \ttime_t pack_mtime;\n \tuint64_t offset;\n+\tunsigned preferred : 1;\n };\n \n static int midx_oid_compare(const void *_a, const void *_b)\n@@ -500,6 +521,12 @@ static int midx_oid_compare(const void *_a, const void *_b)\n \tif (cmp)\n \t\treturn cmp;\n \n+\t/* Sort objects in a preferred pack first when multiple copies exist. */\n+\tif (a->preferred > b->preferred)\n+\t\treturn -1;\n+\tif (a->preferred < b->preferred)\n+\t\treturn 1;\n+\n \tif (a->pack_mtime > b->pack_mtime)\n \t\treturn -1;\n \telse if (a->pack_mtime < b->pack_mtime)\n@@ -527,7 +554,8 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n static void fill_pack_entry(uint32_t pack_int_id,\n \t\t\t struct packed_git *p,\n \t\t\t uint32_t cur_object,\n-\t\t\t struct pack_midx_entry *entry)\n+\t\t\t struct pack_midx_entry *entry,\n+\t\t\t int preferred)\n {\n \tif (nth_packed_object_id(&entry->oid, p, cur_object) < 0)\n \t\tdie(_(\"failed to locate object %d in packfile\"), cur_object);\n@@ -536,6 +564,7 @@ static void fill_pack_entry(uint32_t pack_int_id,\n \tentry->pack_mtime = p->mtime;\n \n \tentry->offset = nth_packed_object_offset(p, cur_object);\n+\tentry->preferred = !!preferred;\n }\n \n /*\n@@ -552,7 +581,8 @@ static void fill_pack_entry(uint32_t pack_int_id,\n static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \t\t\t\t\t\t struct pack_info *info,\n \t\t\t\t\t\t uint32_t nr_packs,\n-\t\t\t\t\t\t uint32_t *nr_objects)\n+\t\t\t\t\t\t uint32_t *nr_objects,\n+\t\t\t\t\t\t uint32_t preferred_pack)\n {\n \tuint32_t cur_fanout, cur_pack, cur_object;\n \tuint32_t alloc_fanout, alloc_objects, total_objects = 0;\n@@ -589,12 +619,17 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \t\t\t\tnth_midxed_pack_midx_entry(m,\n \t\t\t\t\t\t\t &entries_by_fanout[nr_fanout],\n \t\t\t\t\t\t\t cur_object);\n+\t\t\t\tif (nth_midxed_pack_int_id(m, cur_object) == preferred_pack)\n+\t\t\t\t\tentries_by_fanout[nr_fanout].preferred = 1;\n+\t\t\t\telse\n+\t\t\t\t\tentries_by_fanout[nr_fanout].preferred = 0;\n \t\t\t\tnr_fanout++;\n \t\t\t}\n \t\t}\n \n \t\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) {\n \t\t\tuint32_t start = 0, end;\n+\t\t\tint preferred = cur_pack == preferred_pack;\n \n \t\t\tif (cur_fanout)\n \t\t\t\tstart = get_pack_fanout(info[cur_pack].p, cur_fanout - 1);\n@@ -602,7 +637,11 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \n \t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n \t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n-\t\t\t\tfill_pack_entry(cur_pack, info[cur_pack].p, cur_object, &entries_by_fanout[nr_fanout]);\n+\t\t\t\tfill_pack_entry(cur_pack,\n+\t\t\t\t\t\tinfo[cur_pack].p,\n+\t\t\t\t\t\tcur_object,\n+\t\t\t\t\t\t&entries_by_fanout[nr_fanout],\n+\t\t\t\t\t\tpreferred);\n \t\t\t\tnr_fanout++;\n \t\t\t}\n \t\t}\n@@ -777,7 +816,9 @@ static int write_midx_large_offsets(struct hashfile *f,\n }\n \n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n-\t\t\t struct string_list *packs_to_drop, unsigned flags)\n+\t\t\t struct string_list *packs_to_drop,\n+\t\t\t const char *preferred_pack_name,\n+\t\t\t unsigned flags)\n {\n \tchar *midx_name;\n \tuint32_t i;\n@@ -828,7 +869,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n-\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);\n+\tif (preferred_pack_name) {\n+\t\tfor (i = 0; i < ctx.nr; i++) {\n+\t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n+\t\t\t\t\t\t ctx.info[i].pack_name)) {\n+\t\t\t\tctx.preferred_pack_idx = i;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t} else\n+\t\tctx.preferred_pack_idx = -1;\n+\n+\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,\n+\t\t\t\t\t ctx.preferred_pack_idx);\n \n \tctx.large_offsets_needed = 0;\n \tfor (i = 0; i < ctx.entries_nr; i++) {\n@@ -889,6 +942,31 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n \t}\n \n+\t/*\n+\t * Recompute the preferred_pack_idx (if applicable) according to the\n+\t * permuted pack order.\n+\t */\n+\tctx.preferred_pack_idx = -1;\n+\tif (preferred_pack_name) {\n+\t\tctx.preferred_pack_idx = lookup_idx_or_pack_name(ctx.info,\n+\t\t\t\t\t\t\t ctx.nr,\n+\t\t\t\t\t\t\t preferred_pack_name);\n+\t\tif (ctx.preferred_pack_idx < 0)\n+\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n+\t\t\t\tpreferred_pack_name);\n+\t\telse {\n+\t\t\tuint32_t orig = ctx.info[ctx.preferred_pack_idx].orig_pack_int_id;\n+\t\t\tuint32_t perm = ctx.pack_perm[orig];\n+\n+\t\t\tif (perm == PACK_EXPIRED) {\n+\t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n+\t\t\t\t\tpreferred_pack_name);\n+\t\t\t\tctx.preferred_pack_idx = -1;\n+\t\t\t} else\n+\t\t\t\tctx.preferred_pack_idx = perm;\n+\t\t}\n+\t}\n+\n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n \t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n@@ -947,9 +1025,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \treturn result;\n }\n \n-int write_midx_file(const char *object_dir, unsigned flags)\n+int write_midx_file(const char *object_dir,\n+\t\t const char *preferred_pack_name,\n+\t\t unsigned flags)\n {\n-\treturn write_midx_internal(object_dir, NULL, NULL, flags);\n+\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n+\t\t\t\t flags);\n }\n \n void clear_midx_file(struct repository *r)\n@@ -1184,7 +1265,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tfree(count);\n \n \tif (packs_to_drop.nr)\n-\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop, flags);\n+\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop, NULL, flags);\n \n \tstring_list_clear(&packs_to_drop, 0);\n \treturn result;\n@@ -1373,7 +1454,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t\tgoto cleanup;\n \t}\n \n-\tresult = write_midx_internal(object_dir, m, NULL, flags);\n+\tresult = write_midx_internal(object_dir, m, NULL, NULL, flags);\n \tm = NULL;\n \n cleanup:\ndiff --git a/midx.h b/midx.h\nindex b18cf53bc4..e7fea61109 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -47,7 +47,7 @@ int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pa\n int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n \n-int write_midx_file(const char *object_dir, unsigned flags);\n+int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);\n int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex b4afab1dfc..fd94ba9053 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -31,6 +31,14 @@ midx_read_expect () {\n \ttest_cmp expect actual\n }\n \n+midx_expect_object_offset () {\n+\tOID=\"$1\"\n+\tOFFSET=\"$2\"\n+\tOBJECT_DIR=\"$3\"\n+\ttest-tool read-midx --show-objects $OBJECT_DIR >actual &&\n+\tgrep \"^$OID $OFFSET\" actual\n+}\n+\n test_expect_success 'setup' '\n \ttest_oid_cache <<-EOF\n \tidxoff sha1:2999\n@@ -234,6 +242,37 @@ test_expect_success 'warn on improper hash version' '\n \t)\n '\n \n+test_expect_success 'midx picks objects from preferred pack' '\n+\ttest_when_finished rm -rf preferred.git &&\n+\tgit init --bare preferred.git &&\n+\t(\n+\t\tcd preferred.git &&\n+\n+\t\ta=$(echo \"a\" | git hash-object -w --stdin) &&\n+\t\tb=$(echo \"b\" | git hash-object -w --stdin) &&\n+\t\tc=$(echo \"c\" | git hash-object -w --stdin) &&\n+\n+\t\t# Set up two packs, duplicating the object \"B\" at different\n+\t\t# offsets.\n+\t\tgit pack-objects objects/pack/test-AB <<-EOF &&\n+\t\t$a\n+\t\t$b\n+\t\tEOF\n+\t\tbc=$(git pack-objects objects/pack/test-BC <<-EOF\n+\t\t$b\n+\t\t$c\n+\t\tEOF\n+\t\t) &&\n+\n+\t\tgit multi-pack-index --object-dir=objects \\\n+\t\t\twrite --preferred-pack=test-BC-$bc.idx 2>err &&\n+\t\ttest_must_be_empty err &&\n+\n+\t\tofs=$(git show-index \n---\n midx.c | 111 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n midx.h | 1 +\n 2 files changed, 112 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex 8d7a8927b8..820276cc45 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -12,6 +12,7 @@\n #include \"run-command.h\"\n #include \"repository.h\"\n #include \"chunk-format.h\"\n+#include \"pack.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -472,6 +473,7 @@ struct write_midx_context {\n \tuint32_t entries_nr;\n \n \tuint32_t *pack_perm;\n+\tuint32_t *pack_order;\n \tunsigned large_offsets_needed:1;\n \tuint32_t num_large_offsets;\n \n@@ -826,6 +828,66 @@ static int write_midx_large_offsets(struct hashfile *f,\n \treturn 0;\n }\n \n+static int midx_pack_order_cmp(const void *va, const void *vb, void *_ctx)\n+{\n+\tstruct write_midx_context *ctx = _ctx;\n+\n+\tstruct pack_midx_entry *a = &ctx->entries[*(const uint32_t *)va];\n+\tstruct pack_midx_entry *b = &ctx->entries[*(const uint32_t *)vb];\n+\n+\tuint32_t perm_a = ctx->pack_perm[a->pack_int_id];\n+\tuint32_t perm_b = ctx->pack_perm[b->pack_int_id];\n+\n+\t/* Sort objects in the preferred pack ahead of any others. */\n+\tif (a->preferred > b->preferred)\n+\t\treturn -1;\n+\tif (a->preferred < b->preferred)\n+\t\treturn 1;\n+\n+\t/* Then, order objects by which packs they appear in. */\n+\tif (perm_a < perm_b)\n+\t\treturn -1;\n+\tif (perm_a > perm_b)\n+\t\treturn 1;\n+\n+\t/* Then, disambiguate by their offset within each pack. */\n+\tif (a->offset < b->offset)\n+\t\treturn -1;\n+\tif (a->offset > b->offset)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n+{\n+\tuint32_t *pack_order;\n+\tuint32_t i;\n+\n+\tALLOC_ARRAY(pack_order, ctx->entries_nr);\n+\tfor (i = 0; i < ctx->entries_nr; i++)\n+\t\tpack_order[i] = i;\n+\tQSORT_S(pack_order, ctx->entries_nr, midx_pack_order_cmp, ctx);\n+\n+\treturn pack_order;\n+}\n+\n+static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n+\t\t\t\t struct write_midx_context *ctx)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex(midx_hash));\n+\n+\twrite_rev_file_order(buf.buf, ctx->pack_order, ctx->entries_nr,\n+\t\t\t midx_hash, WRITE_REV);\n+\n+\tstrbuf_release(&buf);\n+}\n+\n+static void clear_midx_files_ext(struct repository *r, const char *ext,\n+\t\t\t\t unsigned char *keep_hash);\n+\n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n \t\t\t struct string_list *packs_to_drop,\n \t\t\t const char *preferred_pack_name,\n@@ -1018,6 +1080,14 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n+\n+\tif (flags & MIDX_WRITE_REV_INDEX)\n+\t\tctx.pack_order = midx_pack_order(&ctx);\n+\n+\tif (flags & MIDX_WRITE_REV_INDEX)\n+\t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n+\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n+\n \tcommit_lock_file(&lk);\n \n cleanup:\n@@ -1032,6 +1102,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tfree(ctx.info);\n \tfree(ctx.entries);\n \tfree(ctx.pack_perm);\n+\tfree(ctx.pack_order);\n \tfree(midx_name);\n \treturn result;\n }\n@@ -1044,6 +1115,44 @@ int write_midx_file(const char *object_dir,\n \t\t\t\t flags);\n }\n \n+struct clear_midx_data {\n+\tchar *keep;\n+\tconst char *ext;\n+};\n+\n+static void clear_midx_file_ext(const char *full_path, size_t full_path_len,\n+\t\t\t\tconst char *file_name, void *_data)\n+{\n+\tstruct clear_midx_data *data = _data;\n+\n+\tif (!(starts_with(file_name, \"multi-pack-index-\") &&\n+\t ends_with(file_name, data->ext)))\n+\t\treturn;\n+\tif (data->keep && !strcmp(data->keep, file_name))\n+\t\treturn;\n+\n+\tif (unlink(full_path))\n+\t\tdie_errno(_(\"failed to remove %s\"), full_path);\n+}\n+\n+static void clear_midx_files_ext(struct repository *r, const char *ext,\n+\t\t\t\t unsigned char *keep_hash)\n+{\n+\tstruct clear_midx_data data;\n+\tmemset(&data, 0, sizeof(struct clear_midx_data));\n+\n+\tif (keep_hash)\n+\t\tdata.keep = xstrfmt(\"multi-pack-index-%s%s\",\n+\t\t\t\t hash_to_hex(keep_hash), ext);\n+\tdata.ext = ext;\n+\n+\tfor_each_file_in_pack_dir(r->objects->odb->path,\n+\t\t\t\t clear_midx_file_ext,\n+\t\t\t\t &data);\n+\n+\tfree(data.keep);\n+}\n+\n void clear_midx_file(struct repository *r)\n {\n \tchar *midx = get_midx_filename(r->objects->odb->path);\n@@ -1056,6 +1165,8 @@ void clear_midx_file(struct repository *r)\n \tif (remove_path(midx))\n \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n \n+\tclear_midx_files_ext(r, \".rev\", NULL);\n+\n \tfree(midx);\n }\n \ndiff --git a/midx.h b/midx.h\nindex 0a8294d2ee..8684cf0fef 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -40,6 +40,7 @@ struct multi_pack_index {\n };\n \n #define MIDX_PROGRESS (1 << 0)\n+#define MIDX_WRITE_REV_INDEX (1 << 1)\n \n char *get_midx_rev_filename(struct multi_pack_index *m);\n \n-- \n2.30.0.667.g81c0cbc6fd\n"},{"id":"417705","messageId":"976848bc4b40f1d1e7cdef5e5cf031ecd9f1ac0f.1614193703.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1614193703.git.me@ttaylorr.com","subject":"[PATCH v2 09/15] midx: don't free midx_name early","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-24T19:09:58Z","receivedAt":"2021-02-24T19:13:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A subsequent patch will need to refer back to 'midx_name' later on in\nthe function. In fact, this variable is already free()'d later on, so\nthis makes the later free() no longer redundant.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex d2c56c4bc6..db043d3e65 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -973,7 +973,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n-\tFREE_AND_NULL(midx_name);\n \n \tif (ctx.m)\n \t\tclose_midx(ctx.m);\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"417706","messageId":"0292508e12582462b799d3ba6b190eb29915661d.1614193703.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1614193703.git.me@ttaylorr.com","subject":"[PATCH v2 11/15] midx: make some functions non-static","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-24T19:10:06Z","receivedAt":"2021-02-24T19:13:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a subsequent commit, pack-revindex.c will become responsible for\nsorting a list of objects in the \"MIDX pack order\" (which will be\ndefined in the following patch). To do so, it will need to be know the\npack identifier and offset within that pack for each object in the MIDX.\n\nThe MIDX code already has functions for doing just that\n(nth_midxed_offset() and nth_midxed_pack_int_id()), but they are\nstatically declared.\n\nSince there is no reason that they couldn't be exposed publicly, and\nbecause they are already doing exactly what the caller in\npack-revindex.c will want, expose them publicly so that they can be\nreused there.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 4 ++--\n midx.h | 2 ++\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3ea795f416..27a8b76dfe 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -239,7 +239,7 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \treturn oid;\n }\n \n-static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n+off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n {\n \tconst unsigned char *offset_data;\n \tuint32_t offset32;\n@@ -258,7 +258,7 @@ static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n \treturn offset32;\n }\n \n-static uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n+uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n {\n \treturn get_be32(m->chunk_object_offsets +\n \t\t\t(off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);\ndiff --git a/midx.h b/midx.h\nindex e7fea61109..93bd68189e 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -40,6 +40,8 @@ struct multi_pack_index {\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n+off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);\n+uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);\n struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct multi_pack_index *m,\n \t\t\t\t\tuint32_t n);\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"417707","messageId":"5ed47f7e3a8c42bae051243061debd7b97f630da.1614193703.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1614193703.git.me@ttaylorr.com","subject":"[PATCH v2 10/15] midx: keep track of the checksum","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-24T19:10:02Z","receivedAt":"2021-02-24T19:13:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"write_midx_internal() uses a hashfile to write the multi-pack index, but\ndiscards its checksum. This makes sense, since nothing that takes place\nafter writing the MIDX cares about its checksum.\n\nThat is about to change in a subsequent patch, when the optional\nreverse index corresponding to the MIDX will want to include the MIDX's\nchecksum.\n\nStore the checksum of the MIDX in preparation for that.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex db043d3e65..3ea795f416 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -821,6 +821,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t unsigned flags)\n {\n \tchar *midx_name;\n+\tunsigned char midx_hash[GIT_MAX_RAWSZ];\n \tuint32_t i;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n@@ -1004,7 +1005,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n \twrite_chunkfile(cf, &ctx);\n \n-\tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n+\tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \tcommit_lock_file(&lk);\n \n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"417708","messageId":"404d730498938da034d860d894ddbb7d6dffc27d.1614193703.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1614193703.git.me@ttaylorr.com","subject":"[PATCH v2 12/15] Documentation/technical: describe multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-24T19:10:10Z","receivedAt":"2021-02-24T19:14:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"As a prerequisite to implementing multi-pack bitmaps, motivate and\ndescribe the format and ordering of the multi-pack reverse index.\n\nThe subsequent patch will implement reading this format, and the patch\nafter that will implement writing it while producing a multi-pack index.\n\nCo-authored-by: Jeff King \nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n Documentation/technical/pack-format.txt | 80 +++++++++++++++++++++++++\n 1 file changed, 80 insertions(+)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 1faa949bf6..77eb591057 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -379,3 +379,83 @@ CHUNK DATA:\n TRAILER:\n \n \tIndex checksum of the above contents.\n+\n+== multi-pack-index reverse indexes\n+\n+Similar to the pack-based reverse index, the multi-pack index can also\n+be used to generate a reverse index.\n+\n+Instead of mapping between offset, pack-, and index position, this\n+reverse index maps between an object's position within the MIDX, and\n+that object's position within a pseudo-pack that the MIDX describes.\n+\n+To clarify these three orderings, consider a multi-pack reachability\n+bitmap (which does not yet exist, but is what we are building towards\n+here). Each bit needs to correspond to an object in the MIDX, and so we\n+need an efficient mapping from bit position to MIDX position.\n+\n+One solution is to let bits occupy the same position in the oid-sorted\n+index stored by the MIDX. But because oids are effectively random, there\n+resulting reachability bitmaps would have no locality, and thus compress\n+poorly. (This is the reason that single-pack bitmaps use the pack\n+ordering, and not the .idx ordering, for the same purpose.)\n+\n+So we'd like to define an ordering for the whole MIDX based around\n+pack ordering, which has far better locality (and thus compresses more\n+efficiently). We can think of a pseudo-pack created by the concatenation\n+of all of the packs in the MIDX. E.g., if we had a MIDX with three packs\n+(a, b, c), with 10, 15, and 20 objects respectively, we can imagine an\n+ordering of the objects like:\n+\n+ |a,0|a,1|...|a,9|b,0|b,1|...|b,14|c,0|c,1|...|c,19|\n+\n+where the ordering of the packs is defined by the MIDX's pack list,\n+and then the ordering of objects within each pack is the same as the\n+order in the actual packfile.\n+\n+Given the list of packs and their counts of objects, you can\n+naïvely reconstruct that pseudo-pack ordering (e.g., the object at\n+position 27 must be (c,1) because packs \"a\" and \"b\" consumed 25 of the\n+slots). But there's a catch. Objects may be duplicated between packs, in\n+which case the MIDX only stores one pointer to the object (and thus we'd\n+want only one slot in the bitmap).\n+\n+Callers could handle duplicates themselves by reading objects in order\n+of their bit-position, but that's linear in the number of objects, and\n+much too expensive for ordinary bitmap lookups. Building a reverse index\n+solves this, since it is the logical inverse of the index, and that\n+index has already removed duplicates. But, building a reverse index on\n+the fly can be expensive. Since we already have an on-disk format for\n+pack-based reverse indexes, let's reuse it for the MIDX's pseudo-pack,\n+too.\n+\n+Objects from the MIDX are ordered as follows to string together the\n+pseudo-pack. Let _pack(o)_ return the pack from which _o_ was selected\n+by the MIDX, and define an ordering of packs based on their numeric ID\n+(as stored by the MIDX). Let _offset(o)_ return the object offset of _o_\n+within _pack(o)_. Then, compare _o~1~_ and _o~2~_ as follows:\n+\n+ - If one of _pack(o~1~)_ and _pack(o~2~)_ is preferred and the other\n+ is not, then the preferred one sorts first.\n++\n+(This is a detail that allows the MIDX bitmap to determine which\n+pack should be used by the pack-reuse mechanism, since it can ask\n+the MIDX for the pack containing the object at bit position 0).\n+\n+ - If _pack(o~1~) ≠ pack(o~2~)_, then sort the two objects in\n+ descending order based on the pack ID.\n+\n+ - Otherwise, _pack(o~1~) = pack(o~2~)_, and the objects are\n+ sorted in pack-order (i.e., _o~1~_ sorts ahead of _o~2~_ exactly\n+ when _offset(o~1~) < offset(o~2~)_).\n+\n+In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n+objects in packs stored by the MIDX, laid out in pack order, and the\n+packs arranged in MIDX order (with the preferred pack coming first).\n+\n+Finally, note that the MIDX's reverse index is not stored as a chunk in\n+the multi-pack-index itself. This is done because the reverse index\n+includes the checksum of the pack or MIDX to which it belongs, which\n+makes it impossible to write in the MIDX. To avoid races when rewriting\n+the MIDX, a MIDX reverse index includes the MIDX's checksum in its\n+filename (e.g., `multi-pack-index-xyz.rev`).\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"417709","messageId":"d4e01a44e72998f88dc253cd04845ac2e6bb165b.1614193703.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1614193703.git.me@ttaylorr.com","subject":"[PATCH v2 13/15] pack-revindex: read multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-24T19:10:14Z","receivedAt":"2021-02-24T19:14:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement reading for multi-pack reverse indexes, as described in the\nprevious patch.\n\nNote that these functions don't yet have any callers, and won't until\nmulti-pack reachability bitmaps are introduced in a later patch series.\nIn the meantime, this patch implements some of the infrastructure\nnecessary to support multi-pack bitmaps.\n\nThere are three new functions exposed by the revindex API:\n\n - load_midx_revindex(): loads the reverse index corresponding to the\n given multi-pack index.\n\n - midx_to_pack_pos() and pack_pos_to_midx(): these convert between the\n multi-pack index and pseudo-pack order.\n\nload_midx_revindex() and pack_pos_to_midx() are both relatively\nstraightforward.\n\nload_midx_revindex() needs a few functions to be exposed from the midx\nAPI. One to get the checksum of a midx, and another to get the .rev's\nfilename. Similar to recent changes in the packed_git struct, three new\nfields are added to the multi_pack_index struct: one to keep track of\nthe size, one to keep track of the mmap'd pointer, and another to point\npast the header and at the reverse index's data.\n\npack_pos_to_midx() simply reads the corresponding entry out of the\ntable.\n\nmidx_to_pack_pos() is the trickiest, since it needs to find an object's\nposition in the psuedo-pack order, but that order can only be recovered\nin the .rev file itself. This mapping can be implemented with a binary\nsearch, but note that the thing we're binary searching over isn't an\narray, but rather a _permutation_.\n\nSo, when comparing two items, it's helpful to keep in mind the\ndifference. Instead of a traditional binary search, where you are\ncomparing two things directly, here we're comparing a (pack, offset)\ntuple with an index into the multi-pack index. That index describes\nanother (pack, offset) tuple, and it is _those_ two tuples that are\ncompared.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 11 +++++\n midx.h | 6 +++\n pack-revindex.c | 127 ++++++++++++++++++++++++++++++++++++++++++++++++\n pack-revindex.h | 53 ++++++++++++++++++++\n packfile.c | 3 ++\n 5 files changed, 200 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex 27a8b76dfe..8d7a8927b8 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -47,11 +47,22 @@ static uint8_t oid_version(void)\n \t}\n }\n \n+static const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n+{\n+\treturn m->data + m->data_len - the_hash_algo->rawsz;\n+}\n+\n static char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\n \n+char *get_midx_rev_filename(struct multi_pack_index *m)\n+{\n+\treturn xstrfmt(\"%s/pack/multi-pack-index-%s.rev\",\n+\t\t m->object_dir, hash_to_hex(get_midx_checksum(m)));\n+}\n+\n static int midx_read_oid_fanout(const unsigned char *chunk_start,\n \t\t\t\tsize_t chunk_size, void *data)\n {\ndiff --git a/midx.h b/midx.h\nindex 93bd68189e..0a8294d2ee 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -15,6 +15,10 @@ struct multi_pack_index {\n \tconst unsigned char *data;\n \tsize_t data_len;\n \n+\tconst uint32_t *revindex_data;\n+\tconst uint32_t *revindex_map;\n+\tsize_t revindex_len;\n+\n \tuint32_t signature;\n \tunsigned char version;\n \tunsigned char hash_len;\n@@ -37,6 +41,8 @@ struct multi_pack_index {\n \n #define MIDX_PROGRESS (1 << 0)\n \n+char *get_midx_rev_filename(struct multi_pack_index *m);\n+\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 83fe4de773..2e15ba3a8f 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -3,6 +3,7 @@\n #include \"object-store.h\"\n #include \"packfile.h\"\n #include \"config.h\"\n+#include \"midx.h\"\n \n struct revindex_entry {\n \toff_t offset;\n@@ -292,6 +293,44 @@ int load_pack_revindex(struct packed_git *p)\n \treturn -1;\n }\n \n+int load_midx_revindex(struct multi_pack_index *m)\n+{\n+\tchar *revindex_name;\n+\tint ret;\n+\tif (m->revindex_data)\n+\t\treturn 0;\n+\n+\trevindex_name = get_midx_rev_filename(m);\n+\n+\tret = load_revindex_from_disk(revindex_name,\n+\t\t\t\t m->num_objects,\n+\t\t\t\t &m->revindex_map,\n+\t\t\t\t &m->revindex_len);\n+\tif (ret)\n+\t\tgoto cleanup;\n+\n+\tm->revindex_data = (const uint32_t *)((const char *)m->revindex_map + RIDX_HEADER_SIZE);\n+\n+cleanup:\n+\tfree(revindex_name);\n+\treturn ret;\n+}\n+\n+int close_midx_revindex(struct multi_pack_index *m)\n+{\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tif (munmap((void*)m->revindex_map, m->revindex_len))\n+\t\treturn -1;\n+\n+\tm->revindex_map = NULL;\n+\tm->revindex_data = NULL;\n+\tm->revindex_len = 0;\n+\n+\treturn 0;\n+}\n+\n int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n {\n \tunsigned lo, hi;\n@@ -346,3 +385,91 @@ off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n \telse\n \t\treturn nth_packed_object_offset(p, pack_pos_to_index(p, pos));\n }\n+\n+uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos)\n+{\n+\tif (!m->revindex_data)\n+\t\tBUG(\"pack_pos_to_midx: reverse index not yet loaded\");\n+\tif (m->num_objects <= pos)\n+\t\tBUG(\"pack_pos_to_midx: out-of-bounds object at %\"PRIu32, pos);\n+\treturn get_be32((const char *)m->revindex_data + (pos * sizeof(uint32_t)));\n+}\n+\n+struct midx_pack_key {\n+\tuint32_t pack;\n+\toff_t offset;\n+\n+\tuint32_t preferred_pack;\n+\tstruct multi_pack_index *midx;\n+};\n+\n+static int midx_pack_order_cmp(const void *va, const void *vb)\n+{\n+\tconst struct midx_pack_key *key = va;\n+\tstruct multi_pack_index *midx = key->midx;\n+\n+\tuint32_t versus = pack_pos_to_midx(midx, (uint32_t*)vb - (const uint32_t *)midx->revindex_data);\n+\tuint32_t versus_pack = nth_midxed_pack_int_id(midx, versus);\n+\toff_t versus_offset;\n+\n+\tuint32_t key_preferred = key->pack == key->preferred_pack;\n+\tuint32_t versus_preferred = versus_pack == key->preferred_pack;\n+\n+\t/*\n+\t * First, compare the preferred-ness, noting that the preferred pack\n+\t * comes first.\n+\t */\n+\tif (key_preferred && !versus_preferred)\n+\t\treturn -1;\n+\telse if (!key_preferred && versus_preferred)\n+\t\treturn 1;\n+\n+\t/* Then, break ties first by comparing the pack IDs. */\n+\tif (key->pack < versus_pack)\n+\t\treturn -1;\n+\telse if (key->pack > versus_pack)\n+\t\treturn 1;\n+\n+\t/* Finally, break ties by comparing offsets within a pack. */\n+\tversus_offset = nth_midxed_offset(midx, versus);\n+\tif (key->offset < versus_offset)\n+\t\treturn -1;\n+\telse if (key->offset > versus_offset)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)\n+{\n+\tstruct midx_pack_key key;\n+\tuint32_t *found;\n+\n+\tif (!m->revindex_data)\n+\t\tBUG(\"midx_to_pack_pos: reverse index not yet loaded\");\n+\tif (m->num_objects <= at)\n+\t\tBUG(\"midx_to_pack_pos: out-of-bounds object at %\"PRIu32, at);\n+\n+\tkey.pack = nth_midxed_pack_int_id(m, at);\n+\tkey.offset = nth_midxed_offset(m, at);\n+\tkey.midx = m;\n+\t/*\n+\t * The preferred pack sorts first, so determine its identifier by\n+\t * looking at the first object in pseudo-pack order.\n+\t *\n+\t * Note that if no --preferred-pack is explicitly given when writing a\n+\t * multi-pack index, then whichever pack has the lowest identifier\n+\t * implicitly is preferred (and includes all its objects, since ties are\n+\t * broken first by pack identifier).\n+\t */\n+\tkey.preferred_pack = nth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));\n+\n+\tfound = bsearch(&key, m->revindex_data, m->num_objects,\n+\t\t\tsizeof(uint32_t), midx_pack_order_cmp);\n+\n+\tif (!found)\n+\t\treturn error(\"bad offset for revindex\");\n+\n+\t*pos = found - m->revindex_data;\n+\treturn 0;\n+}\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex ba7c82c125..479b8f2f9c 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -14,6 +14,20 @@\n *\n * - offset: the byte offset within the .pack file at which the object contents\n * can be found\n+ *\n+ * The revindex can also be used with a multi-pack index (MIDX). In this\n+ * setting:\n+ *\n+ * - index position refers to an object's numeric position within the MIDX\n+ *\n+ * - pack position refers to an object's position within a non-existent pack\n+ * described by the MIDX. The pack structure is described in\n+ * Documentation/technical/pack-format.txt.\n+ *\n+ * It is effectively a concatanation of all packs in the MIDX (ordered by\n+ * their numeric ID within the MIDX) in their original order within each\n+ * pack), removing duplicates, and placing the preferred pack (if any)\n+ * first.\n */\n \n \n@@ -24,6 +38,7 @@\n #define GIT_TEST_REV_INDEX_DIE_IN_MEMORY \"GIT_TEST_REV_INDEX_DIE_IN_MEMORY\"\n \n struct packed_git;\n+struct multi_pack_index;\n \n /*\n * load_pack_revindex populates the revindex's internal data-structures for the\n@@ -34,6 +49,22 @@ struct packed_git;\n */\n int load_pack_revindex(struct packed_git *p);\n \n+/*\n+ * load_midx_revindex loads the '.rev' file corresponding to the given\n+ * multi-pack index by mmap-ing it and assigning pointers in the\n+ * multi_pack_index to point at it.\n+ *\n+ * A negative number is returned on error.\n+ */\n+int load_midx_revindex(struct multi_pack_index *m);\n+\n+/*\n+ * Frees resources associated with a multi-pack reverse index.\n+ *\n+ * A negative number is returned on error.\n+ */\n+int close_midx_revindex(struct multi_pack_index *m);\n+\n /*\n * offset_to_pack_pos converts an object offset to a pack position. This\n * function returns zero on success, and a negative number otherwise. The\n@@ -71,4 +102,26 @@ uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);\n */\n off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos);\n \n+/*\n+ * pack_pos_to_midx converts the object at position \"pos\" within the MIDX\n+ * pseudo-pack into a MIDX position.\n+ *\n+ * If the reverse index has not yet been loaded, or the position is out of\n+ * bounds, this function aborts.\n+ *\n+ * This function runs in time O(log N) with the number of objects in the MIDX.\n+ */\n+uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos);\n+\n+/*\n+ * midx_to_pack_pos converts from the MIDX-relative position at \"at\" to the\n+ * corresponding pack position.\n+ *\n+ * If the reverse index has not yet been loaded, or the position is out of\n+ * bounds, this function aborts.\n+ *\n+ * This function runs in constant time.\n+ */\n+int midx_to_pack_pos(struct multi_pack_index *midx, uint32_t at, uint32_t *pos);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 1fec12ac5f..82623e0cb4 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -862,6 +862,9 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \n \tif (!strcmp(file_name, \"multi-pack-index\"))\n \t\treturn;\n+\tif (starts_with(file_name, \"multi-pack-index\") &&\n+\t ends_with(file_name, \".rev\"))\n+\t\treturn;\n \tif (ends_with(file_name, \".idx\") ||\n \t ends_with(file_name, \".rev\") ||\n \t ends_with(file_name, \".pack\") ||\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"417710","messageId":"ab7012b283bd6a3d6bb16b5ffeec11acf6a4ab43.1614193703.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1614193703.git.me@ttaylorr.com","subject":"[PATCH v2 14/15] pack-write.c: extract 'write_rev_file_order'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-02-24T19:10:18Z","receivedAt":"2021-02-24T19:14:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Existing callers provide the reverse index code with an array of 'struct\npack_idx_entry *'s, which is then sorted by pack order (comparing the\noffsets of each object within the pack).\n\nPrepare for the multi-pack index to write a .rev file by providing a way\nto write the reverse index without an array of pack_idx_entry (which the\nMIDX code does not have).\n\nInstead, callers can invoke 'write_rev_index_positions()', which takes\nan array of uint32_t's. The ith entry in this array specifies the ith\nobject's (in index order) position within the pack (in pack order).\n\nExpose this new function for use in a later patch, and rewrite the\nexisting write_rev_file() in terms of this new function.\n\nSigned-off-by: Taylor Blau \n---\n pack-write.c | 39 ++++++++++++++++++++++++++++-----------\n pack.h | 1 +\n 2 files changed, 29 insertions(+), 11 deletions(-)\n\ndiff --git a/pack-write.c b/pack-write.c\nindex 680c36755d..75fcf70db1 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -201,21 +201,12 @@ static void write_rev_header(struct hashfile *f)\n }\n \n static void write_rev_index_positions(struct hashfile *f,\n-\t\t\t\t struct pack_idx_entry **objects,\n+\t\t\t\t uint32_t *pack_order,\n \t\t\t\t uint32_t nr_objects)\n {\n-\tuint32_t *pack_order;\n \tuint32_t i;\n-\n-\tALLOC_ARRAY(pack_order, nr_objects);\n-\tfor (i = 0; i < nr_objects; i++)\n-\t\tpack_order[i] = i;\n-\tQSORT_S(pack_order, nr_objects, pack_order_cmp, objects);\n-\n \tfor (i = 0; i < nr_objects; i++)\n \t\thashwrite_be32(f, pack_order[i]);\n-\n-\tfree(pack_order);\n }\n \n static void write_rev_trailer(struct hashfile *f, const unsigned char *hash)\n@@ -228,6 +219,32 @@ const char *write_rev_file(const char *rev_name,\n \t\t\t uint32_t nr_objects,\n \t\t\t const unsigned char *hash,\n \t\t\t unsigned flags)\n+{\n+\tuint32_t *pack_order;\n+\tuint32_t i;\n+\tconst char *ret;\n+\n+\tALLOC_ARRAY(pack_order, nr_objects);\n+\tfor (i = 0; i < nr_objects; i++)\n+\t\tpack_order[i] = i;\n+\tQSORT_S(pack_order, nr_objects, pack_order_cmp, objects);\n+\n+\tif (!(flags & (WRITE_REV | WRITE_REV_VERIFY)))\n+\t\treturn NULL;\n+\n+\tret = write_rev_file_order(rev_name, pack_order, nr_objects, hash,\n+\t\t\t\t flags);\n+\n+\tfree(pack_order);\n+\n+\treturn ret;\n+}\n+\n+const char *write_rev_file_order(const char *rev_name,\n+\t\t\t\t uint32_t *pack_order,\n+\t\t\t\t uint32_t nr_objects,\n+\t\t\t\t const unsigned char *hash,\n+\t\t\t\t unsigned flags)\n {\n \tstruct hashfile *f;\n \tint fd;\n@@ -262,7 +279,7 @@ const char *write_rev_file(const char *rev_name,\n \n \twrite_rev_header(f);\n \n-\twrite_rev_index_positions(f, objects, nr_objects);\n+\twrite_rev_index_positions(f, pack_order, nr_objects);\n \twrite_rev_trailer(f, hash);\n \n \tif (rev_name && adjust_shared_perm(rev_name) < 0)\ndiff --git a/pack.h b/pack.h\nindex afdcf8f5c7..09c2a7dd3a 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -94,6 +94,7 @@ struct ref;\n void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought);\n \n const char *write_rev_file(const char *rev_name, struct pack_idx_entry **objects, uint32_t nr_objects, const unsigned char *hash, unsigned flags);\n+const char *write_rev_file_order(const char *rev_name, uint32_t *pack_order, uint32_t nr_objects, const unsigned char *hash, unsigned flags);\n \n /*\n * The \"hdr\" output buffer should be at least this big, which will handle sizes\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"418095","messageId":"20210302040625.4035284-1-jonathantanmy@google.com","threadId":"55135","inReplyTo":"bc42b56ea2c3ac7ead1cad2a95ac01a7adfbb28f.1614193703.git.me@ttaylorr.com","subject":"Re: [PATCH v2 04/15] builtin/multi-pack-index.c: split sub-commands","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-03-02T04:06:25Z","receivedAt":"2021-03-02T15:28:34Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> index eea498e026..caf0248a98 100644\n> --- a/builtin/multi-pack-index.c\n> +++ b/builtin/multi-pack-index.c\n> @@ -5,17 +5,33 @@\n> #include \"midx.h\"\n> #include \"trace2.h\"\n> \n> +static char const * const builtin_multi_pack_index_write_usage[] = {\n> #define BUILTIN_MIDX_WRITE_USAGE \\\n> \tN_(\"git multi-pack-index [] write\")\n> +\tBUILTIN_MIDX_WRITE_USAGE,\n> +\tNULL\n> +};\n\nI think this way of writing is vulnerable to confusing errors if a\nmissing or extra backslash happens, so I would prefer the #define to be\noutside the variable declaration.\n\n> +static int cmd_multi_pack_index_repack(int argc, const char **argv)\n> +{\n> +\tstruct option *options;\n> +\tstatic struct option builtin_multi_pack_index_repack_options[] = {\n> \t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n> \t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n> \t\tOPT_END(),\n> \t};\n> \n> +\toptions = parse_options_dup(builtin_multi_pack_index_repack_options);\n> +\toptions = add_common_options(options);\n\nI looked for where this was freed, but I guess freeing this struct is\nnot really something we're worried about (which makes sense).\n\nThe other patches up to this one look good.\n"},{"id":"418097","messageId":"20210302041753.4037658-1-jonathantanmy@google.com","threadId":"55135","inReplyTo":"223b89909416ec7c5505f9cedaa80bf86ecc7b2e.1614193703.git.me@ttaylorr.com","subject":"Re: [PATCH v2 08/15] midx: allow marking a pack as preferred","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-03-02T04:17:53Z","receivedAt":"2021-03-02T15:28:34Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> @@ -589,12 +619,17 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n> \t\t\t\tnth_midxed_pack_midx_entry(m,\n> \t\t\t\t\t\t\t &entries_by_fanout[nr_fanout],\n> \t\t\t\t\t\t\t cur_object);\n> +\t\t\t\tif (nth_midxed_pack_int_id(m, cur_object) == preferred_pack)\n> +\t\t\t\t\tentries_by_fanout[nr_fanout].preferred = 1;\n> +\t\t\t\telse\n> +\t\t\t\t\tentries_by_fanout[nr_fanout].preferred = 0;\n> \t\t\t\tnr_fanout++;\n> \t\t\t}\n> \t\t}\n> \n> \t\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) {\n> \t\t\tuint32_t start = 0, end;\n> +\t\t\tint preferred = cur_pack == preferred_pack;\n> \n> \t\t\tif (cur_fanout)\n> \t\t\t\tstart = get_pack_fanout(info[cur_pack].p, cur_fanout - 1);\n> @@ -602,7 +637,11 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n> \n> \t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n> \t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n> -\t\t\t\tfill_pack_entry(cur_pack, info[cur_pack].p, cur_object, &entries_by_fanout[nr_fanout]);\n> +\t\t\t\tfill_pack_entry(cur_pack,\n> +\t\t\t\t\t\tinfo[cur_pack].p,\n> +\t\t\t\t\t\tcur_object,\n> +\t\t\t\t\t\t&entries_by_fanout[nr_fanout],\n> +\t\t\t\t\t\tpreferred);\n> \t\t\t\tnr_fanout++;\n> \t\t\t}\n> \t\t}\n\nI was initially confused that \"preferred\" was set twice, but this makes\nsense - the first one is when an existing midx is reused, and the second\none is for objects in packs that the midx (if it exists) does not cover.\n\n> @@ -828,7 +869,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n> \t\tgoto cleanup;\n> \n> -\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);\n> +\tif (preferred_pack_name) {\n> +\t\tfor (i = 0; i < ctx.nr; i++) {\n> +\t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n> +\t\t\t\t\t\t ctx.info[i].pack_name)) {\n> +\t\t\t\tctx.preferred_pack_idx = i;\n> +\t\t\t\tbreak;\n> +\t\t\t}\n> +\t\t}\n> +\t} else\n> +\t\tctx.preferred_pack_idx = -1;\n\nLooks safer to put \"ctx.preferred_pack_idx = -1\" before the \"if\", just\nin case the given pack name does not exist?\n\n> @@ -889,6 +942,31 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \t\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n> \t}\n> \n> +\t/*\n> +\t * Recompute the preferred_pack_idx (if applicable) according to the\n> +\t * permuted pack order.\n> +\t */\n> +\tctx.preferred_pack_idx = -1;\n> +\tif (preferred_pack_name) {\n> +\t\tctx.preferred_pack_idx = lookup_idx_or_pack_name(ctx.info,\n> +\t\t\t\t\t\t\t ctx.nr,\n> +\t\t\t\t\t\t\t preferred_pack_name);\n> +\t\tif (ctx.preferred_pack_idx < 0)\n> +\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n> +\t\t\t\tpreferred_pack_name);\n> +\t\telse {\n> +\t\t\tuint32_t orig = ctx.info[ctx.preferred_pack_idx].orig_pack_int_id;\n> +\t\t\tuint32_t perm = ctx.pack_perm[orig];\n> +\n> +\t\t\tif (perm == PACK_EXPIRED) {\n> +\t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n> +\t\t\t\t\tpreferred_pack_name);\n> +\t\t\t\tctx.preferred_pack_idx = -1;\n> +\t\t\t} else\n> +\t\t\t\tctx.preferred_pack_idx = perm;\n> +\t\t}\n> +\t}\n\nI couldn't figure out why the preferred pack index needs to be\nrecalculated here, since the pack entries would have already been\nsorted. Also, the tests still pass when I comment this part out. A\ncomment describing what's going on would be helpful.\n\nAll previous patches look good to me.\n"},{"id":"418096","messageId":"20210302042111.4038479-1-jonathantanmy@google.com","threadId":"55135","inReplyTo":"404d730498938da034d860d894ddbb7d6dffc27d.1614193703.git.me@ttaylorr.com","subject":"Re: [PATCH v2 12/15] Documentation/technical: describe multi-pack reverse indexes","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-03-02T04:21:11Z","receivedAt":"2021-03-02T15:28:35Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> +== multi-pack-index reverse indexes\n> +\n> +Similar to the pack-based reverse index, the multi-pack index can also\n> +be used to generate a reverse index.\n> +\n> +Instead of mapping between offset, pack-, and index position, this\n> +reverse index maps between an object's position within the MIDX, and\n> +that object's position within a pseudo-pack that the MIDX describes.\n> +\n> +To clarify these three orderings\n\nThe paragraph seems to only describe 2 orderings - object's position\nwithin the MIDX and object's position within the pseudo-pack. (Is the\nthird one the offset within the MIDX - which is, I believe, trivially\ncomputable from the position within the MIDX?)\n\nAlso, which are stored in the .rev file?\n\nThe previous patches look good to me, and I'll review the remaining\npatches hopefully tomorrow.\n"},{"id":"418099","messageId":"YD3AxOqA1wvUBkZO@nand.local","threadId":"55135","inReplyTo":"20210302042111.4038479-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 12/15] Documentation/technical: describe multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-02T04:36:20Z","receivedAt":"2021-03-02T15:28:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 01, 2021 at 08:21:11PM -0800, Jonathan Tan wrote:\n> The previous patches look good to me, and I'll review the remaining\n> patches hopefully tomorrow.\n\nThanks; I am sorely behind recent activity on the list. I had a\nlast-minute errand to run last weekend and I haven't managed to quite\ndig out of the hole I created for myself since then.\n\nIncidentally, I have had this code (and the tb/multi-pack-bitmaps)\nrunning on a couple of high-traffic repositories internal to GitHub, and\nso have a couple of improvements that I was hoping to squash in, too.\n\nThanks,\nTaylor\n"},{"id":"418110","messageId":"20210302183620.11040-1-jonathantanmy@google.com","threadId":"55135","inReplyTo":"d4e01a44e72998f88dc253cd04845ac2e6bb165b.1614193703.git.me@ttaylorr.com","subject":"Re: [PATCH v2 13/15] pack-revindex: read multi-pack reverse indexes","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-03-02T18:36:20Z","receivedAt":"2021-03-03T06:41:56Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> midx_to_pack_pos() is the trickiest, since it needs to find an object's\n> position in the psuedo-pack order, but that order can only be recovered\n> in the .rev file itself. This mapping can be implemented with a binary\n> search, but note that the thing we're binary searching over isn't an\n> array, but rather a _permutation_.\n> \n> So, when comparing two items, it's helpful to keep in mind the\n> difference. Instead of a traditional binary search, where you are\n> comparing two things directly, here we're comparing a (pack, offset)\n> tuple with an index into the multi-pack index. That index describes\n> another (pack, offset) tuple, and it is _those_ two tuples that are\n> compared.\n\nWell, the binary search is indeed over an array :-)\n\nI understood that the array we're searching over is an array of indexes\ninto the MIDX in pack-pos order, so I understood what's written here. It\nmight be easier for other readers if we just say that we're treating the\nelements of this array not as indexes into MIDX but as their\ncorresponding (is-preferred-pack, pack number, offset) tuples. But I'm\nfine with retaining the existing wording too.\n\nThe patch itself looks good.\n"},{"id":"418111","messageId":"YD6NVxDib8ccf/6Z@nand.local","threadId":"55135","inReplyTo":"20210302041753.4037658-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 08/15] midx: allow marking a pack as preferred","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-02T19:09:11Z","receivedAt":"2021-03-03T06:41:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 01, 2021 at 08:17:53PM -0800, Jonathan Tan wrote:\n> I was initially confused that \"preferred\" was set twice, but this makes\n> sense - the first one is when an existing midx is reused, and the second\n> one is for objects in packs that the midx (if it exists) does not cover.\n\nYep. Those two paths permeate a lot of the MIDX writer code, since it\nwants to reuse work from an existing MIDX if it can find one.\n\n> > @@ -828,7 +869,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> > \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n> > \t\tgoto cleanup;\n> >\n> > -\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);\n> > +\tif (preferred_pack_name) {\n> > +\t\tfor (i = 0; i < ctx.nr; i++) {\n> > +\t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n> > +\t\t\t\t\t\t ctx.info[i].pack_name)) {\n> > +\t\t\t\tctx.preferred_pack_idx = i;\n> > +\t\t\t\tbreak;\n> > +\t\t\t}\n> > +\t\t}\n> > +\t} else\n> > +\t\tctx.preferred_pack_idx = -1;\n>\n> Looks safer to put \"ctx.preferred_pack_idx = -1\" before the \"if\", just\n> in case the given pack name does not exist?\n\nAgreed.\n\n> > @@ -889,6 +942,31 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> > \t\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n> > \t}\n> >\n> > +\t/*\n> > +\t * Recompute the preferred_pack_idx (if applicable) according to the\n> > +\t * permuted pack order.\n> > +\t */\n> > +\tctx.preferred_pack_idx = -1;\n> > +\tif (preferred_pack_name) {\n> > +\t\tctx.preferred_pack_idx = lookup_idx_or_pack_name(ctx.info,\n> > +\t\t\t\t\t\t\t ctx.nr,\n> > +\t\t\t\t\t\t\t preferred_pack_name);\n> > +\t\tif (ctx.preferred_pack_idx < 0)\n> > +\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n> > +\t\t\t\tpreferred_pack_name);\n> > +\t\telse {\n> > +\t\t\tuint32_t orig = ctx.info[ctx.preferred_pack_idx].orig_pack_int_id;\n> > +\t\t\tuint32_t perm = ctx.pack_perm[orig];\n> > +\n> > +\t\t\tif (perm == PACK_EXPIRED) {\n> > +\t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n> > +\t\t\t\t\tpreferred_pack_name);\n> > +\t\t\t\tctx.preferred_pack_idx = -1;\n> > +\t\t\t} else\n> > +\t\t\t\tctx.preferred_pack_idx = perm;\n> > +\t\t}\n> > +\t}\n>\n> I couldn't figure out why the preferred pack index needs to be\n> recalculated here, since the pack entries would have already been\n> sorted. Also, the tests still pass when I comment this part out. A\n> comment describing what's going on would be helpful.\n\nFunny you mention that; I was wondering the same thing myself the other\nday when reading these patches again before deploying them to a couple\nof testing repositories at GitHub.\n\nIt is totally unnecessary: since we have already marked objects from the\npreferred pack in get_sorted_entries(), the rest of the code doesn't\ncare if the preferred pack was permuted or not.\n\nBut we *do* care if the pack which was preferred expired. The 'git\nrepack --geometric --write-midx' caller (which will appear in a later\nseries) should never do that, so emitting a warning() is worthwhile. I\nthink ultimately you want something like this squashed in:\n\n--- >8 ---\n\ndiff --git a/midx.c b/midx.c\nindex d2c56c4bc6..46f55ff6cf 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -582,7 +582,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \t\t\t\t\t\t struct pack_info *info,\n \t\t\t\t\t\t uint32_t nr_packs,\n \t\t\t\t\t\t uint32_t *nr_objects,\n-\t\t\t\t\t\t uint32_t preferred_pack)\n+\t\t\t\t\t\t int preferred_pack)\n {\n \tuint32_t cur_fanout, cur_pack, cur_object;\n \tuint32_t alloc_fanout, alloc_objects, total_objects = 0;\n@@ -869,6 +869,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n\n+\tctx.preferred_pack_idx = -1;\n \tif (preferred_pack_name) {\n \t\tfor (i = 0; i < ctx.nr; i++) {\n \t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n@@ -877,8 +878,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t\tbreak;\n \t\t\t}\n \t\t}\n-\t} else\n-\t\tctx.preferred_pack_idx = -1;\n+\t}\n\n \tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,\n \t\t\t\t\t ctx.preferred_pack_idx);\n@@ -942,28 +942,21 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n \t}\n\n-\t/*\n-\t * Recompute the preferred_pack_idx (if applicable) according to the\n-\t * permuted pack order.\n-\t */\n-\tctx.preferred_pack_idx = -1;\n+\t/* Check that the preferred pack wasn't expired (if given). */\n \tif (preferred_pack_name) {\n-\t\tctx.preferred_pack_idx = lookup_idx_or_pack_name(ctx.info,\n-\t\t\t\t\t\t\t ctx.nr,\n-\t\t\t\t\t\t\t preferred_pack_name);\n-\t\tif (ctx.preferred_pack_idx < 0)\n+\t\tint preferred_idx = lookup_idx_or_pack_name(ctx.info,\n+\t\t\t\t\t\t\t ctx.nr,\n+\t\t\t\t\t\t\t preferred_pack_name);\n+\t\tif (preferred_idx < 0)\n \t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n \t\t\t\tpreferred_pack_name);\n \t\telse {\n-\t\t\tuint32_t orig = ctx.info[ctx.preferred_pack_idx].orig_pack_int_id;\n+\t\t\tuint32_t orig = ctx.info[preferred_idx].orig_pack_int_id;\n \t\t\tuint32_t perm = ctx.pack_perm[orig];\n\n-\t\t\tif (perm == PACK_EXPIRED) {\n+\t\t\tif (perm == PACK_EXPIRED)\n \t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n \t\t\t\t\tpreferred_pack_name);\n-\t\t\t\tctx.preferred_pack_idx = -1;\n-\t\t\t} else\n-\t\t\t\tctx.preferred_pack_idx = perm;\n \t\t}\n \t}\n\n"},{"id":"418113","messageId":"20210302184033.12305-1-jonathantanmy@google.com","threadId":"55135","inReplyTo":"01bd6a35c6c441a30a22a4c2d17e9cf53de6b148.1614193703.git.me@ttaylorr.com","subject":"Re: [PATCH v2 15/15] pack-revindex: write multi-pack reverse indexes","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-03-02T18:40:33Z","receivedAt":"2021-03-03T06:41:56Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> @@ -1018,6 +1080,14 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \n> \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n> \tfree_chunkfile(cf);\n> +\n> +\tif (flags & MIDX_WRITE_REV_INDEX)\n> +\t\tctx.pack_order = midx_pack_order(&ctx);\n> +\n> +\tif (flags & MIDX_WRITE_REV_INDEX)\n> +\t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n> +\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n> +\n> \tcommit_lock_file(&lk);\n> \n> cleanup:\n\nAny reason why we're using 2 separate \"if\" statements?\n\nOther than that, this patch and patch 14 look good. Besides all my minor\ncomments, I think the overall patch set is in good shape and ready to be\nmerged. It's great that we could reuse some of the individual-pack reverse\nindex concepts and code too.\n"},{"id":"418114","messageId":"YD6OxwU936MRKa1I@nand.local","threadId":"55135","inReplyTo":"20210302042111.4038479-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 12/15] Documentation/technical: describe multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-02T19:15:19Z","receivedAt":"2021-03-03T06:41:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 01, 2021 at 08:21:11PM -0800, Jonathan Tan wrote:\n> > +== multi-pack-index reverse indexes\n> > +\n> > +Similar to the pack-based reverse index, the multi-pack index can also\n> > +be used to generate a reverse index.\n> > +\n> > +Instead of mapping between offset, pack-, and index position, this\n> > +reverse index maps between an object's position within the MIDX, and\n> > +that object's position within a pseudo-pack that the MIDX describes.\n> > +\n> > +To clarify these three orderings\n>\n> The paragraph seems to only describe 2 orderings - object's position\n> within the MIDX and object's position within the pseudo-pack. (Is the\n> third one the offset within the MIDX - which is, I believe, trivially\n> computable from the position within the MIDX?)\n\nSorry for the confusion. I was trying to distinguish between ordering\nbased on object offset, pack position, and index position.\n\nI guess you could count that as 2, 3, or 4 different orderings (if you\nclassify \"pack vs MIDX\", \"offset vs pack pos vs index pos\" or the last\nthree plus \"vs MIDX pos\").\n\nBut I think that all of that is needlessly confusing, so I'd much rather\njust say \"To clarify the difference between these orderings\".\n\n> Also, which are stored in the .rev file?\n\nThe paragraph above describes it a little bit \"this reverse index maps\nbetween ...\", but I think it could be made clearer. (I was intentionally\nbrief there since I wanted to not get too far into the details before\nexplaining the relevant concepts, but I think I went too far).\n\nHow does this sound?\n\n--- >8 ---\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 77eb591057..4bbbb188a4 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -387,12 +387,15 @@ be used to generate a reverse index.\n\n Instead of mapping between offset, pack-, and index position, this\n reverse index maps between an object's position within the MIDX, and\n-that object's position within a pseudo-pack that the MIDX describes.\n+that object's position within a pseudo-pack that the MIDX describes\n+(i.e., the ith entry of the multi-pack reverse index holds the MIDX\n+position of ith object in pseudo-pack order).\n\n-To clarify these three orderings, consider a multi-pack reachability\n-bitmap (which does not yet exist, but is what we are building towards\n-here). Each bit needs to correspond to an object in the MIDX, and so we\n-need an efficient mapping from bit position to MIDX position.\n+To clarify the difference between these orderings, consider a multi-pack\n+reachability bitmap (which does not yet exist, but is what we are\n+building towards here). Each bit needs to correspond to an object in the\n+MIDX, and so we need an efficient mapping from bit position to MIDX\n+position.\n\n One solution is to let bits occupy the same position in the oid-sorted\n index stored by the MIDX. But because oids are effectively random, there\n"},{"id":"418115","messageId":"YD6Ly6EIcz4PT6uB@nand.local","threadId":"55135","inReplyTo":"20210302040625.4035284-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 04/15] builtin/multi-pack-index.c: split sub-commands","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-02T19:02:35Z","receivedAt":"2021-03-03T06:41:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 01, 2021 at 08:06:25PM -0800, Jonathan Tan wrote:\n> > +static char const * const builtin_multi_pack_index_write_usage[] = {\n> > #define BUILTIN_MIDX_WRITE_USAGE \\\n> > \tN_(\"git multi-pack-index [] write\")\n> > +\tBUILTIN_MIDX_WRITE_USAGE,\n> > +\tNULL\n> > +};\n>\n> I think this way of writing is vulnerable to confusing errors if a\n> missing or extra backslash happens, so I would prefer the #define to be\n> outside the variable declaration.\n\nYeah, I can't say that I disagree with you. Of course, having the\n#define's outside of the declaration makes the whole thing a little more\nverbose, which isn't a huge deal.\n\nBut I was mirroring what Ævar was doing in the sub-thread he started at:\n\n https://public-inbox.org/git/20210215184118.11306-1-avarab@gmail.com/\n\nUnless you feel strongly, I think that what we have isn't so bad here.\n\n> > +static int cmd_multi_pack_index_repack(int argc, const char **argv)\n> > +{\n> > +\tstruct option *options;\n> > +\tstatic struct option builtin_multi_pack_index_repack_options[] = {\n> > \t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n> > \t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n> > \t\tOPT_END(),\n> > \t};\n> >\n> > +\toptions = parse_options_dup(builtin_multi_pack_index_repack_options);\n> > +\toptions = add_common_options(options);\n>\n> I looked for where this was freed, but I guess freeing this struct is\n> not really something we're worried about (which makes sense).\n\nRight.\n\nThanks,\nTaylor\n"},{"id":"418164","messageId":"YD+q1lmA8lFzDMT2@nand.local","threadId":"55135","inReplyTo":"20210302183620.11040-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 13/15] pack-revindex: read multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-03T15:27:18Z","receivedAt":"2021-03-04T00:23:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Mar 02, 2021 at 10:36:20AM -0800, Jonathan Tan wrote:\n> > midx_to_pack_pos() is the trickiest, since it needs to find an object's\n> > position in the psuedo-pack order, but that order can only be recovered\n> > in the .rev file itself. This mapping can be implemented with a binary\n> > search, but note that the thing we're binary searching over isn't an\n> > array, but rather a _permutation_.\n> >\n> > So, when comparing two items, it's helpful to keep in mind the\n> > difference. Instead of a traditional binary search, where you are\n> > comparing two things directly, here we're comparing a (pack, offset)\n> > tuple with an index into the multi-pack index. That index describes\n> > another (pack, offset) tuple, and it is _those_ two tuples that are\n> > compared.\n>\n> Well, the binary search is indeed over an array :-)\n\n:-). This might be more clearer as:\n\n ...isn't an array of values, but rather a permuted order of those values.\n\n> The patch itself looks good.\n\nThanks.\n\nTaylor\n"},{"id":"418165","messageId":"YD+rlaYGylVSkg1D@nand.local","threadId":"55135","inReplyTo":"20210302184033.12305-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 15/15] pack-revindex: write multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-03T15:30:29Z","receivedAt":"2021-03-04T00:23:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Mar 02, 2021 at 10:40:33AM -0800, Jonathan Tan wrote:\n> > @@ -1018,6 +1080,14 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> >\n> > \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n> > \tfree_chunkfile(cf);\n> > +\n> > +\tif (flags & MIDX_WRITE_REV_INDEX)\n> > +\t\tctx.pack_order = midx_pack_order(&ctx);\n> > +\n> > +\tif (flags & MIDX_WRITE_REV_INDEX)\n> > +\t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n> > +\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n> > +\n> > \tcommit_lock_file(&lk);\n> >\n> > cleanup:\n>\n> Any reason why we're using 2 separate \"if\" statements?\n\nYeah. This first if statement will turn into:\n\n if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))\n\nso that the pack order is computed in either case (since both the\nexisting write_midx_reverse_index() and the eventual write_midx_bitmap()\nwill be able to use the pack order).\n\nArguably there is never a practical reason to write one without the\nother (and writing a MIDX bitmap without a reverse index is a bug), so\nperhaps these options should be consolidated.\n\nBut that's cleanup that I'd rather do after all of this has settled\n(since it'd be weird to say: \"here's the option to write bitmaps, except\nwe can't write multi-pack bitmaps yet, but setting it actually writes\nthis other thing\").\n\n> Other than that, this patch and patch 14 look good. Besides all my minor\n> comments, I think the overall patch set is in good shape and ready to be\n> merged. It's great that we could reuse some of the individual-pack reverse\n> index concepts and code too.\n\nThanks, I am really glad that you had a chance to take a look at it. I\nalways find your review quite helpful.\n\nThanks,\nTaylor\n"},{"id":"418206","messageId":"20210304015444.1801419-1-jonathantanmy@google.com","threadId":"55135","inReplyTo":"YD6Ly6EIcz4PT6uB@nand.local","subject":"Re: [PATCH v2 04/15] builtin/multi-pack-index.c: split sub-commands","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-03-04T01:54:44Z","receivedAt":"2021-03-04T01:55:56Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> On Mon, Mar 01, 2021 at 08:06:25PM -0800, Jonathan Tan wrote:\n> > > +static char const * const builtin_multi_pack_index_write_usage[] = {\n> > > #define BUILTIN_MIDX_WRITE_USAGE \\\n> > > \tN_(\"git multi-pack-index [] write\")\n> > > +\tBUILTIN_MIDX_WRITE_USAGE,\n> > > +\tNULL\n> > > +};\n> >\n> > I think this way of writing is vulnerable to confusing errors if a\n> > missing or extra backslash happens, so I would prefer the #define to be\n> > outside the variable declaration.\n> \n> Yeah, I can't say that I disagree with you. Of course, having the\n> #define's outside of the declaration makes the whole thing a little more\n> verbose, which isn't a huge deal.\n\nI think it's the same verbosity - you just need to move the lines?\n\n> But I was mirroring what Ævar was doing in the sub-thread he started at:\n> \n> https://public-inbox.org/git/20210215184118.11306-1-avarab@gmail.com/\n> \n> Unless you feel strongly, I think that what we have isn't so bad here.\n\nYeah I don't feel that strongly about it.\n"},{"id":"418207","messageId":"20210304020017.1802260-1-jonathantanmy@google.com","threadId":"55135","inReplyTo":"YD6NVxDib8ccf/6Z@nand.local","subject":"Re: [PATCH v2 08/15] midx: allow marking a pack as preferred","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-03-04T02:00:17Z","receivedAt":"2021-03-04T02:02:06Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Funny you mention that; I was wondering the same thing myself the other\n> day when reading these patches again before deploying them to a couple\n> of testing repositories at GitHub.\n> \n> It is totally unnecessary: since we have already marked objects from the\n> preferred pack in get_sorted_entries(), the rest of the code doesn't\n> care if the preferred pack was permuted or not.\n> \n> But we *do* care if the pack which was preferred expired. The 'git\n> repack --geometric --write-midx' caller (which will appear in a later\n> series) should never do that, so emitting a warning() is worthwhile.\n\nAh, this makes sense.\n\n> I\n> think ultimately you want something like this squashed in:\n> \n> --- >8 ---\n> \n> diff --git a/midx.c b/midx.c\n> index d2c56c4bc6..46f55ff6cf 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -582,7 +582,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n> \t\t\t\t\t\t struct pack_info *info,\n> \t\t\t\t\t\t uint32_t nr_packs,\n> \t\t\t\t\t\t uint32_t *nr_objects,\n> -\t\t\t\t\t\t uint32_t preferred_pack)\n> +\t\t\t\t\t\t int preferred_pack)\n\nWhy this change?\n\n> {\n> \tuint32_t cur_fanout, cur_pack, cur_object;\n> \tuint32_t alloc_fanout, alloc_objects, total_objects = 0;\n> @@ -869,6 +869,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n> \t\tgoto cleanup;\n> \n> +\tctx.preferred_pack_idx = -1;\n> \tif (preferred_pack_name) {\n> \t\tfor (i = 0; i < ctx.nr; i++) {\n> \t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n> @@ -877,8 +878,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \t\t\t\tbreak;\n> \t\t\t}\n> \t\t}\n> -\t} else\n> -\t\tctx.preferred_pack_idx = -1;\n> +\t}\n> \n> \tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,\n> \t\t\t\t\t ctx.preferred_pack_idx);\n> @@ -942,28 +942,21 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> \t\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n> \t}\n> \n> -\t/*\n> -\t * Recompute the preferred_pack_idx (if applicable) according to the\n> -\t * permuted pack order.\n> -\t */\n> -\tctx.preferred_pack_idx = -1;\n> +\t/* Check that the preferred pack wasn't expired (if given). */\n> \tif (preferred_pack_name) {\n> -\t\tctx.preferred_pack_idx = lookup_idx_or_pack_name(ctx.info,\n> -\t\t\t\t\t\t\t ctx.nr,\n> -\t\t\t\t\t\t\t preferred_pack_name);\n> -\t\tif (ctx.preferred_pack_idx < 0)\n> +\t\tint preferred_idx = lookup_idx_or_pack_name(ctx.info,\n> +\t\t\t\t\t\t\t ctx.nr,\n> +\t\t\t\t\t\t\t preferred_pack_name);\n> +\t\tif (preferred_idx < 0)\n> \t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n> \t\t\t\tpreferred_pack_name);\n> \t\telse {\n> -\t\t\tuint32_t orig = ctx.info[ctx.preferred_pack_idx].orig_pack_int_id;\n> +\t\t\tuint32_t orig = ctx.info[preferred_idx].orig_pack_int_id;\n> \t\t\tuint32_t perm = ctx.pack_perm[orig];\n> \n> -\t\t\tif (perm == PACK_EXPIRED) {\n> +\t\t\tif (perm == PACK_EXPIRED)\n> \t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n> \t\t\t\t\tpreferred_pack_name);\n> -\t\t\t\tctx.preferred_pack_idx = -1;\n> -\t\t\t} else\n> -\t\t\t\tctx.preferred_pack_idx = perm;\n> \t\t}\n> \t}\n\nThe rest makes sense.\n> \n"},{"id":"418208","messageId":"20210304020444.1803406-1-jonathantanmy@google.com","threadId":"55135","inReplyTo":"YD+rlaYGylVSkg1D@nand.local","subject":"Re: [PATCH v2 15/15] pack-revindex: write multi-pack reverse indexes","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-03-04T02:04:44Z","receivedAt":"2021-03-04T02:05:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> On Tue, Mar 02, 2021 at 10:40:33AM -0800, Jonathan Tan wrote:\n> > > @@ -1018,6 +1080,14 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n> > >\n> > > \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n> > > \tfree_chunkfile(cf);\n> > > +\n> > > +\tif (flags & MIDX_WRITE_REV_INDEX)\n> > > +\t\tctx.pack_order = midx_pack_order(&ctx);\n> > > +\n> > > +\tif (flags & MIDX_WRITE_REV_INDEX)\n> > > +\t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n> > > +\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n> > > +\n> > > \tcommit_lock_file(&lk);\n> > >\n> > > cleanup:\n> >\n> > Any reason why we're using 2 separate \"if\" statements?\n> \n> Yeah. This first if statement will turn into:\n> \n> if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))\n> \n> so that the pack order is computed in either case (since both the\n> existing write_midx_reverse_index() and the eventual write_midx_bitmap()\n> will be able to use the pack order).\n\nAh, OK. That's what I was thinking of, but nice to have confirmation.\nMaybe write in the commit message that these are separated because in\nthe future, one of the conditions will change.\n"},{"id":"418210","messageId":"20210304020305.1802814-1-jonathantanmy@google.com","threadId":"55135","inReplyTo":"YD6OxwU936MRKa1I@nand.local","subject":"Re: [PATCH v2 12/15] Documentation/technical: describe multi-pack reverse indexes","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-03-04T02:03:05Z","receivedAt":"2021-03-04T02:35:55Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\n> index 77eb591057..4bbbb188a4 100644\n> --- a/Documentation/technical/pack-format.txt\n> +++ b/Documentation/technical/pack-format.txt\n> @@ -387,12 +387,15 @@ be used to generate a reverse index.\n> \n> Instead of mapping between offset, pack-, and index position, this\n> reverse index maps between an object's position within the MIDX, and\n> -that object's position within a pseudo-pack that the MIDX describes.\n> +that object's position within a pseudo-pack that the MIDX describes\n> +(i.e., the ith entry of the multi-pack reverse index holds the MIDX\n> +position of ith object in pseudo-pack order).\n> \n> -To clarify these three orderings, consider a multi-pack reachability\n> -bitmap (which does not yet exist, but is what we are building towards\n> -here). Each bit needs to correspond to an object in the MIDX, and so we\n> -need an efficient mapping from bit position to MIDX position.\n> +To clarify the difference between these orderings, consider a multi-pack\n> +reachability bitmap (which does not yet exist, but is what we are\n> +building towards here). Each bit needs to correspond to an object in the\n> +MIDX, and so we need an efficient mapping from bit position to MIDX\n> +position.\n> \n> One solution is to let bits occupy the same position in the oid-sorted\n> index stored by the MIDX. But because oids are effectively random, there\n\nThanks - this diff makes sense.\n"},{"id":"418211","messageId":"YEBN1ejeG3mBCnA+@nand.local","threadId":"55135","inReplyTo":"20210304015444.1801419-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 04/15] builtin/multi-pack-index.c: split sub-commands","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-04T03:02:45Z","receivedAt":"2021-03-04T03:04:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Mar 03, 2021 at 05:54:44PM -0800, Jonathan Tan wrote:\n> > On Mon, Mar 01, 2021 at 08:06:25PM -0800, Jonathan Tan wrote:\n> > > > +static char const * const builtin_multi_pack_index_write_usage[] = {\n> > > > #define BUILTIN_MIDX_WRITE_USAGE \\\n> > > > \tN_(\"git multi-pack-index [] write\")\n> > > > +\tBUILTIN_MIDX_WRITE_USAGE,\n> > > > +\tNULL\n> > > > +};\n> > >\n> > > I think this way of writing is vulnerable to confusing errors if a\n> > > missing or extra backslash happens, so I would prefer the #define to be\n> > > outside the variable declaration.\n> >\n> > Yeah, I can't say that I disagree with you. Of course, having the\n> > #define's outside of the declaration makes the whole thing a little more\n> > verbose, which isn't a huge deal.\n>\n> I think it's the same verbosity - you just need to move the lines?\n\nYeah, you're right. I'm being too subjective, and I don't really feel\nstrongly, either.\n\n>\n> > But I was mirroring what Ævar was doing in the sub-thread he started at:\n> >\n> > https://public-inbox.org/git/20210215184118.11306-1-avarab@gmail.com/\n> >\n> > Unless you feel strongly, I think that what we have isn't so bad here.\n>\n> Yeah I don't feel that strongly about it.\n\nI'll take your suggestion, thanks.\n\nThanks,\nTaylor\n"},{"id":"418212","messageId":"YEBOKpaD1gGXRTvp@nand.local","threadId":"55135","inReplyTo":"20210304020017.1802260-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 08/15] midx: allow marking a pack as preferred","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-04T03:04:10Z","receivedAt":"2021-03-04T03:05:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Mar 03, 2021 at 06:00:17PM -0800, Jonathan Tan wrote:\n> > I\n> > think ultimately you want something like this squashed in:\n> >\n> > --- >8 ---\n> >\n> > diff --git a/midx.c b/midx.c\n> > index d2c56c4bc6..46f55ff6cf 100644\n> > --- a/midx.c\n> > +++ b/midx.c\n> > @@ -582,7 +582,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n> > \t\t\t\t\t\t struct pack_info *info,\n> > \t\t\t\t\t\t uint32_t nr_packs,\n> > \t\t\t\t\t\t uint32_t *nr_objects,\n> > -\t\t\t\t\t\t uint32_t preferred_pack)\n> > +\t\t\t\t\t\t int preferred_pack)\n>\n> Why this change?\n\nThis was wrong in the original patch: ctx.preferred_pack is an integer,\nand is set to -1 when no preferred pack was specified.\n\nIt's certainly unlikely that we'd have 2^31 packs, but silently\nconverting a signed type to an unsigned one is misleading.\n\n> The rest makes sense.\n\nThanks for taking a look.\n\nTaylor\n"},{"id":"418213","messageId":"YEBOvgm8yG9RiL0P@nand.local","threadId":"55135","inReplyTo":"20210304020444.1803406-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 15/15] pack-revindex: write multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-04T03:06:38Z","receivedAt":"2021-03-04T03:08:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Mar 03, 2021 at 06:04:44PM -0800, Jonathan Tan wrote:\n> > > Any reason why we're using 2 separate \"if\" statements?\n> >\n> > Yeah. This first if statement will turn into:\n> >\n> > if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))\n> >\n> > so that the pack order is computed in either case (since both the\n> > existing write_midx_reverse_index() and the eventual write_midx_bitmap()\n> > will be able to use the pack order).\n>\n> Ah, OK. That's what I was thinking of, but nice to have confirmation.\n> Maybe write in the commit message that these are separated because in\n> the future, one of the conditions will change.\n\nThanks; that's a great idea.\n\nThanks,\nTaylor\n"},{"id":"418863","messageId":"181f11e4c55b364dc7f6a6530f397779171671a9.1615482270.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"[PATCH v3 02/16] builtin/multi-pack-index.c: don't handle 'progress' separately","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-11T17:04:40Z","receivedAt":"2021-03-11T17:05:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that there is a shared 'flags' member in the options structure,\nthere is no need to keep track of whether to force progress or not,\nsince ultimately the decision of whether or not to show a progress meter\nis controlled by a bit in the flags member.\n\nManipulate that bit directly, and drop the now-unnecessary 'progress'\nfield while we're at it.\n\nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 8 +++-----\n 1 file changed, 3 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 4a0ddb06c4..c70f020d8f 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -13,7 +13,6 @@ static char const * const builtin_multi_pack_index_usage[] = {\n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n \tunsigned long batch_size;\n-\tint progress;\n \tunsigned flags;\n } opts;\n \n@@ -23,7 +22,7 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tstatic struct option builtin_multi_pack_index_options[] = {\n \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n-\t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n+\t\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"), MIDX_PROGRESS),\n \t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n \t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n \t\tOPT_END(),\n@@ -31,15 +30,14 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \n \tgit_config(git_default_config, NULL);\n \n-\topts.progress = isatty(2);\n+\tif (isatty(2))\n+\t\topts.flags |= MIDX_PROGRESS;\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_multi_pack_index_options,\n \t\t\t builtin_multi_pack_index_usage, 0);\n \n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n-\tif (opts.progress)\n-\t\topts.flags |= MIDX_PROGRESS;\n \n \tif (argc == 0)\n \t\tusage_with_options(builtin_multi_pack_index_usage,\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"418864","messageId":"cover.1615482270.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1612998106.git.me@ttaylorr.com","subject":"[PATCH v3 00/16] midx: implement a multi-pack reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-11T17:04:31Z","receivedAt":"2021-03-11T17:05:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is another reroll of my series to implement a reverse index in\npreparation for multi-pack reachability bitmaps. The previous version\nwas based on 'ds/chunked-file-api', but that topic has since been merged\nto 'master'. This series is now built directly on top of 'master'.\n\nNot much has changed since last time. Jonathan Tan reviewed the previous\nversion, and I incorporated feedback from his review:\n\n - The usage macros in builtin/multi-pack-index.c were pulled out and\n defined separately.\n - Some sloppiness with converting a signed index referring to the\n preferred pack into an unsigned value was cleaned up.\n - Documentation clean-up, particularly in patches 12 and 13.\n\nThere are a couple of new things that we found while testing this out at\nGitHub.\n\n - We now call finalize_object_file() on the multi-pack reverse index\n to set the correct permissions.\n - Patch 14 removed a stray hunk that introduced a memory leak.\n - Patch 16 (courtesy of Peff) is new. It improves the cache locality\n of midx_pack_order_cmp(), which has a substantial impact on\n repositories with many objects.\n\nThanks in advance for your review.\n\nJeff King (1):\n midx.c: improve cache locality in midx_pack_order_cmp()\n\nTaylor Blau (15):\n builtin/multi-pack-index.c: inline 'flags' with options\n builtin/multi-pack-index.c: don't handle 'progress' separately\n builtin/multi-pack-index.c: define common usage with a macro\n builtin/multi-pack-index.c: split sub-commands\n builtin/multi-pack-index.c: don't enter bogus cmd_mode\n builtin/multi-pack-index.c: display usage on unrecognized command\n t/helper/test-read-midx.c: add '--show-objects'\n midx: allow marking a pack as preferred\n midx: don't free midx_name early\n midx: keep track of the checksum\n midx: make some functions non-static\n Documentation/technical: describe multi-pack reverse indexes\n pack-revindex: read multi-pack reverse indexes\n pack-write.c: extract 'write_rev_file_order'\n pack-revindex: write multi-pack reverse indexes\n\n Documentation/git-multi-pack-index.txt | 14 +-\n Documentation/technical/multi-pack-index.txt | 5 +-\n Documentation/technical/pack-format.txt | 83 +++++++\n builtin/multi-pack-index.c | 182 ++++++++++++---\n builtin/repack.c | 2 +-\n midx.c | 229 +++++++++++++++++--\n midx.h | 11 +-\n pack-revindex.c | 127 ++++++++++\n pack-revindex.h | 53 +++++\n pack-write.c | 36 ++-\n pack.h | 1 +\n packfile.c | 3 +\n t/helper/test-read-midx.c | 24 +-\n t/t5319-multi-pack-index.sh | 39 ++++\n 14 files changed, 740 insertions(+), 69 deletions(-)\n\nRange-diff against v2:\n 1: 0527fa89a9 = 1: 43fc0ad276 builtin/multi-pack-index.c: inline 'flags' with options\n 2: a4e107b1f8 = 2: 181f11e4c5 builtin/multi-pack-index.c: don't handle 'progress' separately\n 3: 8679dfd212 = 3: 94c498f0e2 builtin/multi-pack-index.c: define common usage with a macro\n 4: bc42b56ea2 ! 4: d084f90466 builtin/multi-pack-index.c: split sub-commands\n @@ Commit message\n \n ## builtin/multi-pack-index.c ##\n @@\n - #include \"midx.h\"\n - #include \"trace2.h\"\n + #define BUILTIN_MIDX_REPACK_USAGE \\\n + \tN_(\"git multi-pack-index [] repack [--batch-size=]\")\n \n +static char const * const builtin_multi_pack_index_write_usage[] = {\n - #define BUILTIN_MIDX_WRITE_USAGE \\\n - \tN_(\"git multi-pack-index [] write\")\n +\tBUILTIN_MIDX_WRITE_USAGE,\n +\tNULL\n +};\n - \n +static char const * const builtin_multi_pack_index_verify_usage[] = {\n - #define BUILTIN_MIDX_VERIFY_USAGE \\\n - \tN_(\"git multi-pack-index [] verify\")\n +\tBUILTIN_MIDX_VERIFY_USAGE,\n +\tNULL\n +};\n - \n +static char const * const builtin_multi_pack_index_expire_usage[] = {\n - #define BUILTIN_MIDX_EXPIRE_USAGE \\\n - \tN_(\"git multi-pack-index [] expire\")\n +\tBUILTIN_MIDX_EXPIRE_USAGE,\n +\tNULL\n +};\n - \n +static char const * const builtin_multi_pack_index_repack_usage[] = {\n - #define BUILTIN_MIDX_REPACK_USAGE \\\n - \tN_(\"git multi-pack-index [] repack [--batch-size=]\")\n +\tBUILTIN_MIDX_REPACK_USAGE,\n +\tNULL\n +};\n - \n static char const * const builtin_multi_pack_index_usage[] = {\n \tBUILTIN_MIDX_WRITE_USAGE,\n + \tBUILTIN_MIDX_VERIFY_USAGE,\n @@ builtin/multi-pack-index.c: static struct opts_multi_pack_index {\n \tunsigned flags;\n } opts;\n 5: 5daa2946d3 = 5: bc3b6837f2 builtin/multi-pack-index.c: don't enter bogus cmd_mode\n 6: 98d9ea0770 = 6: f117e442c3 builtin/multi-pack-index.c: display usage on unrecognized command\n 7: 2fd9f4debf = 7: ae85a68ef2 t/helper/test-read-midx.c: add '--show-objects'\n 8: 223b899094 ! 8: 30194a6786 midx: allow marking a pack as preferred\n @@ builtin/multi-pack-index.c\n #include \"trace2.h\"\n +#include \"object-store.h\"\n \n - static char const * const builtin_multi_pack_index_write_usage[] = {\n #define BUILTIN_MIDX_WRITE_USAGE \\\n -\tN_(\"git multi-pack-index [] write\")\n +\tN_(\"git multi-pack-index [] write [--preferred-pack=]\")\n - \tBUILTIN_MIDX_WRITE_USAGE,\n - \tNULL\n - };\n + \n + #define BUILTIN_MIDX_VERIFY_USAGE \\\n + \tN_(\"git multi-pack-index [] verify\")\n @@ builtin/multi-pack-index.c: static char const * const builtin_multi_pack_index_usage[] = {\n \n static struct opts_multi_pack_index {\n @@ midx.c: static void fill_pack_entry(uint32_t pack_int_id,\n \t\t\t\t\t\t uint32_t nr_packs,\n -\t\t\t\t\t\t uint32_t *nr_objects)\n +\t\t\t\t\t\t uint32_t *nr_objects,\n -+\t\t\t\t\t\t uint32_t preferred_pack)\n ++\t\t\t\t\t\t int preferred_pack)\n {\n \tuint32_t cur_fanout, cur_pack, cur_object;\n \tuint32_t alloc_fanout, alloc_objects, total_objects = 0;\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n \t\tgoto cleanup;\n \n -\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);\n ++\tctx.preferred_pack_idx = -1;\n +\tif (preferred_pack_name) {\n +\t\tfor (i = 0; i < ctx.nr; i++) {\n +\t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n +\t\t\t\tbreak;\n +\t\t\t}\n +\t\t}\n -+\t} else\n -+\t\tctx.preferred_pack_idx = -1;\n ++\t}\n +\n +\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,\n +\t\t\t\t\t ctx.preferred_pack_idx);\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n \t\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n \t}\n \n -+\t/*\n -+\t * Recompute the preferred_pack_idx (if applicable) according to the\n -+\t * permuted pack order.\n -+\t */\n -+\tctx.preferred_pack_idx = -1;\n ++\t/* Check that the preferred pack wasn't expired (if given). */\n +\tif (preferred_pack_name) {\n -+\t\tctx.preferred_pack_idx = lookup_idx_or_pack_name(ctx.info,\n -+\t\t\t\t\t\t\t ctx.nr,\n -+\t\t\t\t\t\t\t preferred_pack_name);\n -+\t\tif (ctx.preferred_pack_idx < 0)\n ++\t\tint preferred_idx = lookup_idx_or_pack_name(ctx.info,\n ++\t\t\t\t\t\t\t ctx.nr,\n ++\t\t\t\t\t\t\t preferred_pack_name);\n ++\t\tif (preferred_idx < 0)\n +\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n +\t\t\t\tpreferred_pack_name);\n +\t\telse {\n -+\t\t\tuint32_t orig = ctx.info[ctx.preferred_pack_idx].orig_pack_int_id;\n ++\t\t\tuint32_t orig = ctx.info[preferred_idx].orig_pack_int_id;\n +\t\t\tuint32_t perm = ctx.pack_perm[orig];\n +\n -+\t\t\tif (perm == PACK_EXPIRED) {\n ++\t\t\tif (perm == PACK_EXPIRED)\n +\t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n +\t\t\t\t\tpreferred_pack_name);\n -+\t\t\t\tctx.preferred_pack_idx = -1;\n -+\t\t\t} else\n -+\t\t\t\tctx.preferred_pack_idx = perm;\n +\t\t}\n +\t}\n +\n 9: 976848bc4b = 9: 5c5aca761a midx: don't free midx_name early\n10: 5ed47f7e3a = 10: a22a1463a5 midx: keep track of the checksum\n11: 0292508e12 = 11: efa54479b1 midx: make some functions non-static\n12: 404d730498 ! 12: 4745bb8590 Documentation/technical: describe multi-pack reverse indexes\n @@ Documentation/technical/pack-format.txt: CHUNK DATA:\n +\n +Instead of mapping between offset, pack-, and index position, this\n +reverse index maps between an object's position within the MIDX, and\n -+that object's position within a pseudo-pack that the MIDX describes.\n ++that object's position within a pseudo-pack that the MIDX describes\n ++(i.e., the ith entry of the multi-pack reverse index holds the MIDX\n ++position of ith object in pseudo-pack order).\n +\n -+To clarify these three orderings, consider a multi-pack reachability\n -+bitmap (which does not yet exist, but is what we are building towards\n -+here). Each bit needs to correspond to an object in the MIDX, and so we\n -+need an efficient mapping from bit position to MIDX position.\n ++To clarify the difference between these orderings, consider a multi-pack\n ++reachability bitmap (which does not yet exist, but is what we are\n ++building towards here). Each bit needs to correspond to an object in the\n ++MIDX, and so we need an efficient mapping from bit position to MIDX\n ++position.\n +\n +One solution is to let bits occupy the same position in the oid-sorted\n +index stored by the MIDX. But because oids are effectively random, there\n13: d4e01a44e7 ! 13: a6ebd4be91 pack-revindex: read multi-pack reverse indexes\n @@ Commit message\n position in the psuedo-pack order, but that order can only be recovered\n in the .rev file itself. This mapping can be implemented with a binary\n search, but note that the thing we're binary searching over isn't an\n - array, but rather a _permutation_.\n + array of values, but rather a permuted order of those values.\n \n So, when comparing two items, it's helpful to keep in mind the\n difference. Instead of a traditional binary search, where you are\n14: ab7012b283 ! 14: f5314f1822 pack-write.c: extract 'write_rev_file_order'\n @@ pack-write.c: const char *write_rev_file(const char *rev_name,\n +\t\tpack_order[i] = i;\n +\tQSORT_S(pack_order, nr_objects, pack_order_cmp, objects);\n +\n -+\tif (!(flags & (WRITE_REV | WRITE_REV_VERIFY)))\n -+\t\treturn NULL;\n -+\n +\tret = write_rev_file_order(rev_name, pack_order, nr_objects, hash,\n +\t\t\t\t flags);\n +\n15: 01bd6a35c6 ! 15: fa3acb5d5a pack-revindex: write multi-pack reverse indexes\n @@ Commit message\n for long, since subsequent patches will introduce the multi-pack bitmap,\n which will begin passing this field.\n \n + (In midx.c:write_midx_internal(), the two adjacent if statements share a\n + conditional, but are written separately since the first one will\n + eventually also handle the MIDX_WRITE_BITMAP flag, which does not yet\n + exist.)\n +\n Signed-off-by: Taylor Blau \n \n ## midx.c ##\n @@ midx.c: static int write_midx_large_offsets(struct hashfile *f,\n +\t\t\t\t struct write_midx_context *ctx)\n +{\n +\tstruct strbuf buf = STRBUF_INIT;\n ++\tconst char *tmp_file;\n +\n +\tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex(midx_hash));\n +\n -+\twrite_rev_file_order(buf.buf, ctx->pack_order, ctx->entries_nr,\n -+\t\t\t midx_hash, WRITE_REV);\n ++\ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n ++\t\t\t\t\tmidx_hash, WRITE_REV);\n ++\n ++\tif (finalize_object_file(tmp_file, buf.buf))\n ++\t\tdie(_(\"cannot store reverse index file\"));\n +\n +\tstrbuf_release(&buf);\n +}\n -: ---------- > 16: 550e785f10 midx.c: improve cache locality in midx_pack_order_cmp()\n-- \n2.30.0.667.g81c0cbc6fd\n"},{"id":"418865","messageId":"43fc0ad276406ff77283613c45188e102a6dc515.1615482270.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"[PATCH v3 01/16] builtin/multi-pack-index.c: inline 'flags' with options","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-11T17:04:36Z","receivedAt":"2021-03-11T17:05:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Subcommands of the 'git multi-pack-index' command (e.g., 'write',\n'verify', etc.) will want to optionally change a set of shared flags\nthat are eventually passed to the MIDX libraries.\n\nRight now, options and flags are handled separately. Inline them into\nthe same structure so that sub-commands can more easily share the\n'flags' data.\n\nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 13 ++++++-------\n 1 file changed, 6 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5bf88cd2a8..4a0ddb06c4 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -14,13 +14,12 @@ static struct opts_multi_pack_index {\n \tconst char *object_dir;\n \tunsigned long batch_size;\n \tint progress;\n+\tunsigned flags;\n } opts;\n \n int cmd_multi_pack_index(int argc, const char **argv,\n \t\t\t const char *prefix)\n {\n-\tunsigned flags = 0;\n-\n \tstatic struct option builtin_multi_pack_index_options[] = {\n \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n@@ -40,7 +39,7 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n \tif (opts.progress)\n-\t\tflags |= MIDX_PROGRESS;\n+\t\topts.flags |= MIDX_PROGRESS;\n \n \tif (argc == 0)\n \t\tusage_with_options(builtin_multi_pack_index_usage,\n@@ -55,16 +54,16 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \n \tif (!strcmp(argv[0], \"repack\"))\n \t\treturn midx_repack(the_repository, opts.object_dir,\n-\t\t\t(size_t)opts.batch_size, flags);\n+\t\t\t(size_t)opts.batch_size, opts.flags);\n \tif (opts.batch_size)\n \t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n \n \tif (!strcmp(argv[0], \"write\"))\n-\t\treturn write_midx_file(opts.object_dir, flags);\n+\t\treturn write_midx_file(opts.object_dir, opts.flags);\n \tif (!strcmp(argv[0], \"verify\"))\n-\t\treturn verify_midx_file(the_repository, opts.object_dir, flags);\n+\t\treturn verify_midx_file(the_repository, opts.object_dir, opts.flags);\n \tif (!strcmp(argv[0], \"expire\"))\n-\t\treturn expire_midx_packs(the_repository, opts.object_dir, flags);\n+\t\treturn expire_midx_packs(the_repository, opts.object_dir, opts.flags);\n \n \tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n }\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"418866","messageId":"94c498f0e25c8dde093b9a1ad8044c4ef37a6c5d.1615482270.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"[PATCH v3 03/16] builtin/multi-pack-index.c: define common usage with a macro","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-11T17:04:45Z","receivedAt":"2021-03-11T17:05:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Factor out the usage message into pieces corresponding to each mode.\nThis avoids options specific to one sub-command from being shared with\nanother in the usage.\n\nA subsequent commit will use these #define macros to have usage\nvariables for each sub-command without duplicating their contents.\n\nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 17 ++++++++++++++++-\n 1 file changed, 16 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex c70f020d8f..eea498e026 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,8 +5,23 @@\n #include \"midx.h\"\n #include \"trace2.h\"\n \n+#define BUILTIN_MIDX_WRITE_USAGE \\\n+\tN_(\"git multi-pack-index [] write\")\n+\n+#define BUILTIN_MIDX_VERIFY_USAGE \\\n+\tN_(\"git multi-pack-index [] verify\")\n+\n+#define BUILTIN_MIDX_EXPIRE_USAGE \\\n+\tN_(\"git multi-pack-index [] expire\")\n+\n+#define BUILTIN_MIDX_REPACK_USAGE \\\n+\tN_(\"git multi-pack-index [] repack [--batch-size=]\")\n+\n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [] (write|verify|expire|repack --batch-size=)\"),\n+\tBUILTIN_MIDX_WRITE_USAGE,\n+\tBUILTIN_MIDX_VERIFY_USAGE,\n+\tBUILTIN_MIDX_EXPIRE_USAGE,\n+\tBUILTIN_MIDX_REPACK_USAGE,\n \tNULL\n };\n \n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"418867","messageId":"d084f90466813597feb27975654e57de36eb62a4.1615482270.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"[PATCH v3 04/16] builtin/multi-pack-index.c: split sub-commands","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-11T17:04:49Z","receivedAt":"2021-03-11T17:05:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Handle sub-commands of the 'git multi-pack-index' builtin (e.g.,\n\"write\", \"repack\", etc.) separately from one another. This allows\nsub-commands with unique options, without forcing cmd_multi_pack_index()\nto reject invalid combinations itself.\n\nThis comes at the cost of some duplication and boilerplate. Luckily, the\nduplication is reduced to a minimum, since common options are shared\namong sub-commands due to a suggestion by Ævar. (Sub-commands do have to\nretain the common options, too, since this builtin accepts common\noptions on either side of the sub-command).\n\nRoughly speaking, cmd_multi_pack_index() parses options (including\ncommon ones), and stops at the first non-option, which is the\nsub-command. It then dispatches to the appropriate sub-command, which\nparses the remaining options (also including common options).\n\nUnknown options are kept by the sub-commands in order to detect their\npresence (and complain that too many arguments were given).\n\nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 131 ++++++++++++++++++++++++++++++-------\n 1 file changed, 106 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex eea498e026..23e51dfeb4 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -17,6 +17,22 @@\n #define BUILTIN_MIDX_REPACK_USAGE \\\n \tN_(\"git multi-pack-index [] repack [--batch-size=]\")\n \n+static char const * const builtin_multi_pack_index_write_usage[] = {\n+\tBUILTIN_MIDX_WRITE_USAGE,\n+\tNULL\n+};\n+static char const * const builtin_multi_pack_index_verify_usage[] = {\n+\tBUILTIN_MIDX_VERIFY_USAGE,\n+\tNULL\n+};\n+static char const * const builtin_multi_pack_index_expire_usage[] = {\n+\tBUILTIN_MIDX_EXPIRE_USAGE,\n+\tNULL\n+};\n+static char const * const builtin_multi_pack_index_repack_usage[] = {\n+\tBUILTIN_MIDX_REPACK_USAGE,\n+\tNULL\n+};\n static char const * const builtin_multi_pack_index_usage[] = {\n \tBUILTIN_MIDX_WRITE_USAGE,\n \tBUILTIN_MIDX_VERIFY_USAGE,\n@@ -31,25 +47,99 @@ static struct opts_multi_pack_index {\n \tunsigned flags;\n } opts;\n \n-int cmd_multi_pack_index(int argc, const char **argv,\n-\t\t\t const char *prefix)\n+static struct option common_opts[] = {\n+\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n+\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n+\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"), MIDX_PROGRESS),\n+\tOPT_END(),\n+};\n+\n+static struct option *add_common_options(struct option *prev)\n {\n-\tstatic struct option builtin_multi_pack_index_options[] = {\n-\t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n-\t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n-\t\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"), MIDX_PROGRESS),\n+\tstruct option *with_common = parse_options_concat(common_opts, prev);\n+\tfree(prev);\n+\treturn with_common;\n+}\n+\n+static int cmd_multi_pack_index_write(int argc, const char **argv)\n+{\n+\tstruct option *options = common_opts;\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options, builtin_multi_pack_index_write_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n+\t\t\t\t options);\n+\n+\treturn write_midx_file(opts.object_dir, opts.flags);\n+}\n+\n+static int cmd_multi_pack_index_verify(int argc, const char **argv)\n+{\n+\tstruct option *options = common_opts;\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options, builtin_multi_pack_index_verify_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_multi_pack_index_verify_usage,\n+\t\t\t\t options);\n+\n+\treturn verify_midx_file(the_repository, opts.object_dir, opts.flags);\n+}\n+\n+static int cmd_multi_pack_index_expire(int argc, const char **argv)\n+{\n+\tstruct option *options = common_opts;\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options, builtin_multi_pack_index_expire_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_multi_pack_index_expire_usage,\n+\t\t\t\t options);\n+\n+\treturn expire_midx_packs(the_repository, opts.object_dir, opts.flags);\n+}\n+\n+static int cmd_multi_pack_index_repack(int argc, const char **argv)\n+{\n+\tstruct option *options;\n+\tstatic struct option builtin_multi_pack_index_repack_options[] = {\n \t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n \t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n \t\tOPT_END(),\n \t};\n \n+\toptions = parse_options_dup(builtin_multi_pack_index_repack_options);\n+\toptions = add_common_options(options);\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options,\n+\t\t\t builtin_multi_pack_index_repack_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_multi_pack_index_repack_usage,\n+\t\t\t\t options);\n+\n+\treturn midx_repack(the_repository, opts.object_dir,\n+\t\t\t (size_t)opts.batch_size, opts.flags);\n+}\n+\n+int cmd_multi_pack_index(int argc, const char **argv,\n+\t\t\t const char *prefix)\n+{\n+\tstruct option *builtin_multi_pack_index_options = common_opts;\n+\n \tgit_config(git_default_config, NULL);\n \n \tif (isatty(2))\n \t\topts.flags |= MIDX_PROGRESS;\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_multi_pack_index_options,\n-\t\t\t builtin_multi_pack_index_usage, 0);\n+\t\t\t builtin_multi_pack_index_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n@@ -58,25 +148,16 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\tusage_with_options(builtin_multi_pack_index_usage,\n \t\t\t\t builtin_multi_pack_index_options);\n \n-\tif (argc > 1) {\n-\t\tdie(_(\"too many arguments\"));\n-\t\treturn 1;\n-\t}\n-\n \ttrace2_cmd_mode(argv[0]);\n \n \tif (!strcmp(argv[0], \"repack\"))\n-\t\treturn midx_repack(the_repository, opts.object_dir,\n-\t\t\t(size_t)opts.batch_size, opts.flags);\n-\tif (opts.batch_size)\n-\t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n-\n-\tif (!strcmp(argv[0], \"write\"))\n-\t\treturn write_midx_file(opts.object_dir, opts.flags);\n-\tif (!strcmp(argv[0], \"verify\"))\n-\t\treturn verify_midx_file(the_repository, opts.object_dir, opts.flags);\n-\tif (!strcmp(argv[0], \"expire\"))\n-\t\treturn expire_midx_packs(the_repository, opts.object_dir, opts.flags);\n-\n-\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n+\t\treturn cmd_multi_pack_index_repack(argc, argv);\n+\telse if (!strcmp(argv[0], \"write\"))\n+\t\treturn cmd_multi_pack_index_write(argc, argv);\n+\telse if (!strcmp(argv[0], \"verify\"))\n+\t\treturn cmd_multi_pack_index_verify(argc, argv);\n+\telse if (!strcmp(argv[0], \"expire\"))\n+\t\treturn cmd_multi_pack_index_expire(argc, argv);\n+\telse\n+\t\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n }\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"418871","messageId":"bc3b6837f274e3f648c3ce6e959c11d4ddf1bb11.1615482270.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"[PATCH v3 05/16] builtin/multi-pack-index.c: don't enter bogus cmd_mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-11T17:04:53Z","receivedAt":"2021-03-11T17:05:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Even before the recent refactoring, 'git multi-pack-index' calls\n'trace2_cmd_mode()' before verifying that the sub-command is recognized.\n\nPush this call down into the individual sub-commands so that we don't\nenter a bogus command mode.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 23e51dfeb4..b5678cc2bb 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -65,6 +65,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n {\n \tstruct option *options = common_opts;\n \n+\ttrace2_cmd_mode(argv[0]);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options, builtin_multi_pack_index_write_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n@@ -79,6 +81,8 @@ static int cmd_multi_pack_index_verify(int argc, const char **argv)\n {\n \tstruct option *options = common_opts;\n \n+\ttrace2_cmd_mode(argv[0]);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options, builtin_multi_pack_index_verify_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n@@ -93,6 +97,8 @@ static int cmd_multi_pack_index_expire(int argc, const char **argv)\n {\n \tstruct option *options = common_opts;\n \n+\ttrace2_cmd_mode(argv[0]);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options, builtin_multi_pack_index_expire_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n@@ -115,6 +121,8 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv)\n \toptions = parse_options_dup(builtin_multi_pack_index_repack_options);\n \toptions = add_common_options(options);\n \n+\ttrace2_cmd_mode(argv[0]);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options,\n \t\t\t builtin_multi_pack_index_repack_usage,\n@@ -148,8 +156,6 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\tusage_with_options(builtin_multi_pack_index_usage,\n \t\t\t\t builtin_multi_pack_index_options);\n \n-\ttrace2_cmd_mode(argv[0]);\n-\n \tif (!strcmp(argv[0], \"repack\"))\n \t\treturn cmd_multi_pack_index_repack(argc, argv);\n \telse if (!strcmp(argv[0], \"write\"))\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"418868","messageId":"5c5aca761a69e245eaece97703366c9f8d06a889.1615482270.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"[PATCH v3 09/16] midx: don't free midx_name early","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-11T17:05:12Z","receivedAt":"2021-03-11T17:05:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A subsequent patch will need to refer back to 'midx_name' later on in\nthe function. In fact, this variable is already free()'d later on, so\nthis makes the later free() no longer redundant.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 46f55ff6cf..e0009d3314 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -966,7 +966,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n-\tFREE_AND_NULL(midx_name);\n \n \tif (ctx.m)\n \t\tclose_midx(ctx.m);\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"418869","messageId":"ae85a68ef2791edd5661890d69e2399fe3f07b44.1615482270.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"[PATCH v3 07/16] t/helper/test-read-midx.c: add '--show-objects'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-11T17:05:03Z","receivedAt":"2021-03-11T17:05:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The 'read-midx' helper is used in places like t5319 to display basic\ninformation about a multi-pack-index.\n\nIn the next patch, the MIDX writing machinery will learn a new way to\nchoose from which pack an object is selected when multiple copies of\nthat object exist.\n\nTo disambiguate which pack introduces an object so that this feature can\nbe tested, add a '--show-objects' option which displays additional\ninformation about each object in the MIDX.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-midx.c | 24 ++++++++++++++++++++----\n 1 file changed, 20 insertions(+), 4 deletions(-)\n\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 2430880f78..7c2eb11a8e 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -4,7 +4,7 @@\n #include \"repository.h\"\n #include \"object-store.h\"\n \n-static int read_midx_file(const char *object_dir)\n+static int read_midx_file(const char *object_dir, int show_objects)\n {\n \tuint32_t i;\n \tstruct multi_pack_index *m;\n@@ -43,13 +43,29 @@ static int read_midx_file(const char *object_dir)\n \n \tprintf(\"object-dir: %s\\n\", m->object_dir);\n \n+\tif (show_objects) {\n+\t\tstruct object_id oid;\n+\t\tstruct pack_entry e;\n+\n+\t\tfor (i = 0; i < m->num_objects; i++) {\n+\t\t\tnth_midxed_object_oid(&oid, m, i);\n+\t\t\tfill_midx_entry(the_repository, &oid, &e, m);\n+\n+\t\t\tprintf(\"%s %\"PRIu64\"\\t%s\\n\",\n+\t\t\t oid_to_hex(&oid), e.offset, e.p->pack_name);\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\n \treturn 0;\n }\n \n int cmd__read_midx(int argc, const char **argv)\n {\n-\tif (argc != 2)\n-\t\tusage(\"read-midx \");\n+\tif (!(argc == 2 || argc == 3))\n+\t\tusage(\"read-midx [--show-objects] \");\n \n-\treturn read_midx_file(argv[1]);\n+\tif (!strcmp(argv[1], \"--show-objects\"))\n+\t\treturn read_midx_file(argv[2], 1);\n+\treturn read_midx_file(argv[1], 0);\n }\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"418870","messageId":"f117e442c3460661dc88beba6f1853d6c388b0fd.1615482270.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"[PATCH v3 06/16] builtin/multi-pack-index.c: display usage on unrecognized command","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-11T17:04:57Z","receivedAt":"2021-03-11T17:05:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When given a sub-command that it doesn't understand, 'git\nmulti-pack-index' dies with the following message:\n\n $ git multi-pack-index bogus\n fatal: unrecognized subcommand: bogus\n\nInstead of 'die()'-ing, we can display the usage text, which is much\nmore helpful:\n\n $ git.compile multi-pack-index bogus\n usage: git multi-pack-index [] write\n or: git multi-pack-index [] verify\n or: git multi-pack-index [] expire\n or: git multi-pack-index [] repack [--batch-size=]\n\n\t--object-dir object directory containing set of packfile and pack-index pairs\n\t--progress force progress reporting\n\nWhile we're at it, clean up some duplication between the \"no sub-command\"\nand \"unrecognized sub-command\" conditionals.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 7 ++++---\n 1 file changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex b5678cc2bb..243b6ccc7c 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -153,8 +153,7 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\topts.object_dir = get_object_directory();\n \n \tif (argc == 0)\n-\t\tusage_with_options(builtin_multi_pack_index_usage,\n-\t\t\t\t builtin_multi_pack_index_options);\n+\t\tgoto usage;\n \n \tif (!strcmp(argv[0], \"repack\"))\n \t\treturn cmd_multi_pack_index_repack(argc, argv);\n@@ -165,5 +164,7 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \telse if (!strcmp(argv[0], \"expire\"))\n \t\treturn cmd_multi_pack_index_expire(argc, argv);\n \telse\n-\t\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n+usage:\n+\t\tusage_with_options(builtin_multi_pack_index_usage,\n+\t\t\t\t builtin_multi_pack_index_options);\n }\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"418872","messageId":"30194a6786bec51e0f41de0e6c855dc2297806c6.1615482270.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"[PATCH v3 08/16] midx: allow marking a pack as preferred","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-11T17:05:07Z","receivedAt":"2021-03-11T17:05:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When multiple packs in the multi-pack index contain the same object, the\nMIDX machinery must make a choice about which pack it associates with\nthat object. Prior to this patch, the lowest-ordered[1] pack was always\nselected.\n\nPack selection for duplicate objects is relatively unimportant today,\nbut it will become important for multi-pack bitmaps. This is because we\ncan only invoke the pack-reuse mechanism when all of the bits for reused\nobjects come from the reuse pack (in order to ensure that all reused\ndeltas can find their base objects in the same pack).\n\nTo encourage the pack selection process to prefer one pack over another\n(the pack to be preferred is the one a caller would like to later use as\na reuse pack), introduce the concept of a \"preferred pack\". When\nprovided, the MIDX code will always prefer an object found in a\npreferred pack over any other.\n\nNo format changes are required to store the preferred pack, since it\nwill be able to be inferred with a corresponding MIDX bitmap, by looking\nup the pack associated with the object in the first bit position (this\nordering is described in detail in a subsequent commit).\n\n[1]: the ordering is specified by MIDX internals; for our purposes we\ncan consider the \"lowest ordered\" pack to be \"the one with the\nmost-recent mtime.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 14 ++-\n Documentation/technical/multi-pack-index.txt | 5 +-\n builtin/multi-pack-index.c | 18 +++-\n builtin/repack.c | 2 +-\n midx.c | 92 ++++++++++++++++++--\n midx.h | 2 +-\n t/t5319-multi-pack-index.sh | 39 +++++++++\n 7 files changed, 154 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex eb0caa0439..ffd601bc17 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -9,7 +9,8 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n SYNOPSIS\n --------\n [verse]\n-'git multi-pack-index' [--object-dir=] [--[no-]progress] \n+'git multi-pack-index' [--object-dir=] [--[no-]progress]\n+\t[--preferred-pack=] \n \n DESCRIPTION\n -----------\n@@ -30,7 +31,16 @@ OPTIONS\n The following subcommands are available:\n \n write::\n-\tWrite a new MIDX file.\n+\tWrite a new MIDX file. The following options are available for\n+\tthe `write` sub-command:\n++\n+--\n+\t--preferred-pack=::\n+\t\tOptionally specify the tie-breaking pack used when\n+\t\tmultiple packs contain the same object. If not given,\n+\t\tties are broken in favor of the pack with the lowest\n+\t\tmtime.\n+--\n \n verify::\n \tVerify the contents of the MIDX file.\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex e8e377a59f..fb688976c4 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -43,8 +43,9 @@ Design Details\n a change in format.\n \n - The MIDX keeps only one record per object ID. If an object appears\n- in multiple packfiles, then the MIDX selects the copy in the most-\n- recently modified packfile.\n+ in multiple packfiles, then the MIDX selects the copy in the\n+ preferred packfile, otherwise selecting from the most-recently\n+ modified packfile.\n \n - If there exist packfiles in the pack directory not registered in\n the MIDX, then those packfiles are loaded into the `packed_git`\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 243b6ccc7c..92f358f212 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -4,9 +4,10 @@\n #include \"parse-options.h\"\n #include \"midx.h\"\n #include \"trace2.h\"\n+#include \"object-store.h\"\n \n #define BUILTIN_MIDX_WRITE_USAGE \\\n-\tN_(\"git multi-pack-index [] write\")\n+\tN_(\"git multi-pack-index [] write [--preferred-pack=]\")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n@@ -43,6 +44,7 @@ static char const * const builtin_multi_pack_index_usage[] = {\n \n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n+\tconst char *preferred_pack;\n \tunsigned long batch_size;\n \tunsigned flags;\n } opts;\n@@ -63,7 +65,16 @@ static struct option *add_common_options(struct option *prev)\n \n static int cmd_multi_pack_index_write(int argc, const char **argv)\n {\n-\tstruct option *options = common_opts;\n+\tstruct option *options;\n+\tstatic struct option builtin_multi_pack_index_write_options[] = {\n+\t\tOPT_STRING(0, \"preferred-pack\", &opts.preferred_pack,\n+\t\t\t N_(\"preferred-pack\"),\n+\t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\toptions = parse_options_dup(builtin_multi_pack_index_write_options);\n+\toptions = add_common_options(options);\n \n \ttrace2_cmd_mode(argv[0]);\n \n@@ -74,7 +85,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\tusage_with_options(builtin_multi_pack_index_write_usage,\n \t\t\t\t options);\n \n-\treturn write_midx_file(opts.object_dir, opts.flags);\n+\treturn write_midx_file(opts.object_dir, opts.preferred_pack,\n+\t\t\t opts.flags);\n }\n \n static int cmd_multi_pack_index_verify(int argc, const char **argv)\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 01440de2d5..9f00806805 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -523,7 +523,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tremove_temporary_files();\n \n \tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n-\t\twrite_midx_file(get_object_directory(), 0);\n+\t\twrite_midx_file(get_object_directory(), NULL, 0);\n \n \tstring_list_clear(&names, 0);\n \tstring_list_clear(&rollback, 0);\ndiff --git a/midx.c b/midx.c\nindex 971faa8cfc..46f55ff6cf 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -431,6 +431,24 @@ static int pack_info_compare(const void *_a, const void *_b)\n \treturn strcmp(a->pack_name, b->pack_name);\n }\n \n+static int lookup_idx_or_pack_name(struct pack_info *info,\n+\t\t\t\t uint32_t nr,\n+\t\t\t\t const char *pack_name)\n+{\n+\tuint32_t lo = 0, hi = nr;\n+\twhile (lo < hi) {\n+\t\tuint32_t mi = lo + (hi - lo) / 2;\n+\t\tint cmp = cmp_idx_or_pack_name(pack_name, info[mi].pack_name);\n+\t\tif (cmp < 0)\n+\t\t\thi = mi;\n+\t\telse if (cmp > 0)\n+\t\t\tlo = mi + 1;\n+\t\telse\n+\t\t\treturn mi;\n+\t}\n+\treturn -1;\n+}\n+\n struct write_midx_context {\n \tstruct pack_info *info;\n \tuint32_t nr;\n@@ -445,6 +463,8 @@ struct write_midx_context {\n \tuint32_t *pack_perm;\n \tunsigned large_offsets_needed:1;\n \tuint32_t num_large_offsets;\n+\n+\tint preferred_pack_idx;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -489,6 +509,7 @@ struct pack_midx_entry {\n \tuint32_t pack_int_id;\n \ttime_t pack_mtime;\n \tuint64_t offset;\n+\tunsigned preferred : 1;\n };\n \n static int midx_oid_compare(const void *_a, const void *_b)\n@@ -500,6 +521,12 @@ static int midx_oid_compare(const void *_a, const void *_b)\n \tif (cmp)\n \t\treturn cmp;\n \n+\t/* Sort objects in a preferred pack first when multiple copies exist. */\n+\tif (a->preferred > b->preferred)\n+\t\treturn -1;\n+\tif (a->preferred < b->preferred)\n+\t\treturn 1;\n+\n \tif (a->pack_mtime > b->pack_mtime)\n \t\treturn -1;\n \telse if (a->pack_mtime < b->pack_mtime)\n@@ -527,7 +554,8 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n static void fill_pack_entry(uint32_t pack_int_id,\n \t\t\t struct packed_git *p,\n \t\t\t uint32_t cur_object,\n-\t\t\t struct pack_midx_entry *entry)\n+\t\t\t struct pack_midx_entry *entry,\n+\t\t\t int preferred)\n {\n \tif (nth_packed_object_id(&entry->oid, p, cur_object) < 0)\n \t\tdie(_(\"failed to locate object %d in packfile\"), cur_object);\n@@ -536,6 +564,7 @@ static void fill_pack_entry(uint32_t pack_int_id,\n \tentry->pack_mtime = p->mtime;\n \n \tentry->offset = nth_packed_object_offset(p, cur_object);\n+\tentry->preferred = !!preferred;\n }\n \n /*\n@@ -552,7 +581,8 @@ static void fill_pack_entry(uint32_t pack_int_id,\n static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \t\t\t\t\t\t struct pack_info *info,\n \t\t\t\t\t\t uint32_t nr_packs,\n-\t\t\t\t\t\t uint32_t *nr_objects)\n+\t\t\t\t\t\t uint32_t *nr_objects,\n+\t\t\t\t\t\t int preferred_pack)\n {\n \tuint32_t cur_fanout, cur_pack, cur_object;\n \tuint32_t alloc_fanout, alloc_objects, total_objects = 0;\n@@ -589,12 +619,17 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \t\t\t\tnth_midxed_pack_midx_entry(m,\n \t\t\t\t\t\t\t &entries_by_fanout[nr_fanout],\n \t\t\t\t\t\t\t cur_object);\n+\t\t\t\tif (nth_midxed_pack_int_id(m, cur_object) == preferred_pack)\n+\t\t\t\t\tentries_by_fanout[nr_fanout].preferred = 1;\n+\t\t\t\telse\n+\t\t\t\t\tentries_by_fanout[nr_fanout].preferred = 0;\n \t\t\t\tnr_fanout++;\n \t\t\t}\n \t\t}\n \n \t\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) {\n \t\t\tuint32_t start = 0, end;\n+\t\t\tint preferred = cur_pack == preferred_pack;\n \n \t\t\tif (cur_fanout)\n \t\t\t\tstart = get_pack_fanout(info[cur_pack].p, cur_fanout - 1);\n@@ -602,7 +637,11 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \n \t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n \t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n-\t\t\t\tfill_pack_entry(cur_pack, info[cur_pack].p, cur_object, &entries_by_fanout[nr_fanout]);\n+\t\t\t\tfill_pack_entry(cur_pack,\n+\t\t\t\t\t\tinfo[cur_pack].p,\n+\t\t\t\t\t\tcur_object,\n+\t\t\t\t\t\t&entries_by_fanout[nr_fanout],\n+\t\t\t\t\t\tpreferred);\n \t\t\t\tnr_fanout++;\n \t\t\t}\n \t\t}\n@@ -777,7 +816,9 @@ static int write_midx_large_offsets(struct hashfile *f,\n }\n \n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n-\t\t\t struct string_list *packs_to_drop, unsigned flags)\n+\t\t\t struct string_list *packs_to_drop,\n+\t\t\t const char *preferred_pack_name,\n+\t\t\t unsigned flags)\n {\n \tchar *midx_name;\n \tuint32_t i;\n@@ -828,7 +869,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n-\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);\n+\tctx.preferred_pack_idx = -1;\n+\tif (preferred_pack_name) {\n+\t\tfor (i = 0; i < ctx.nr; i++) {\n+\t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n+\t\t\t\t\t\t ctx.info[i].pack_name)) {\n+\t\t\t\tctx.preferred_pack_idx = i;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,\n+\t\t\t\t\t ctx.preferred_pack_idx);\n \n \tctx.large_offsets_needed = 0;\n \tfor (i = 0; i < ctx.entries_nr; i++) {\n@@ -889,6 +942,24 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n \t}\n \n+\t/* Check that the preferred pack wasn't expired (if given). */\n+\tif (preferred_pack_name) {\n+\t\tint preferred_idx = lookup_idx_or_pack_name(ctx.info,\n+\t\t\t\t\t\t\t ctx.nr,\n+\t\t\t\t\t\t\t preferred_pack_name);\n+\t\tif (preferred_idx < 0)\n+\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n+\t\t\t\tpreferred_pack_name);\n+\t\telse {\n+\t\t\tuint32_t orig = ctx.info[preferred_idx].orig_pack_int_id;\n+\t\t\tuint32_t perm = ctx.pack_perm[orig];\n+\n+\t\t\tif (perm == PACK_EXPIRED)\n+\t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n+\t\t\t\t\tpreferred_pack_name);\n+\t\t}\n+\t}\n+\n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n \t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n@@ -947,9 +1018,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \treturn result;\n }\n \n-int write_midx_file(const char *object_dir, unsigned flags)\n+int write_midx_file(const char *object_dir,\n+\t\t const char *preferred_pack_name,\n+\t\t unsigned flags)\n {\n-\treturn write_midx_internal(object_dir, NULL, NULL, flags);\n+\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n+\t\t\t\t flags);\n }\n \n void clear_midx_file(struct repository *r)\n@@ -1184,7 +1258,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tfree(count);\n \n \tif (packs_to_drop.nr)\n-\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop, flags);\n+\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop, NULL, flags);\n \n \tstring_list_clear(&packs_to_drop, 0);\n \treturn result;\n@@ -1373,7 +1447,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t\tgoto cleanup;\n \t}\n \n-\tresult = write_midx_internal(object_dir, m, NULL, flags);\n+\tresult = write_midx_internal(object_dir, m, NULL, NULL, flags);\n \tm = NULL;\n \n cleanup:\ndiff --git a/midx.h b/midx.h\nindex b18cf53bc4..e7fea61109 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -47,7 +47,7 @@ int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pa\n int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n \n-int write_midx_file(const char *object_dir, unsigned flags);\n+int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);\n int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex b4afab1dfc..fd94ba9053 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -31,6 +31,14 @@ midx_read_expect () {\n \ttest_cmp expect actual\n }\n \n+midx_expect_object_offset () {\n+\tOID=\"$1\"\n+\tOFFSET=\"$2\"\n+\tOBJECT_DIR=\"$3\"\n+\ttest-tool read-midx --show-objects $OBJECT_DIR >actual &&\n+\tgrep \"^$OID $OFFSET\" actual\n+}\n+\n test_expect_success 'setup' '\n \ttest_oid_cache <<-EOF\n \tidxoff sha1:2999\n@@ -234,6 +242,37 @@ test_expect_success 'warn on improper hash version' '\n \t)\n '\n \n+test_expect_success 'midx picks objects from preferred pack' '\n+\ttest_when_finished rm -rf preferred.git &&\n+\tgit init --bare preferred.git &&\n+\t(\n+\t\tcd preferred.git &&\n+\n+\t\ta=$(echo \"a\" | git hash-object -w --stdin) &&\n+\t\tb=$(echo \"b\" | git hash-object -w --stdin) &&\n+\t\tc=$(echo \"c\" | git hash-object -w --stdin) &&\n+\n+\t\t# Set up two packs, duplicating the object \"B\" at different\n+\t\t# offsets.\n+\t\tgit pack-objects objects/pack/test-AB <<-EOF &&\n+\t\t$a\n+\t\t$b\n+\t\tEOF\n+\t\tbc=$(git pack-objects objects/pack/test-BC <<-EOF\n+\t\t$b\n+\t\t$c\n+\t\tEOF\n+\t\t) &&\n+\n+\t\tgit multi-pack-index --object-dir=objects \\\n+\t\t\twrite --preferred-pack=test-BC-$bc.idx 2>err &&\n+\t\ttest_must_be_empty err &&\n+\n+\t\tofs=$(git show-index \n---\n midx.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex e0009d3314..31e6d3d2df 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -821,6 +821,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t unsigned flags)\n {\n \tchar *midx_name;\n+\tunsigned char midx_hash[GIT_MAX_RAWSZ];\n \tuint32_t i;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n@@ -997,7 +998,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n \twrite_chunkfile(cf, &ctx);\n \n-\tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n+\tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \tcommit_lock_file(&lk);\n \n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"418874","messageId":"efa54479b15f51b7d87aca2071f19ce04faf7d78.1615482270.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"[PATCH v3 11/16] midx: make some functions non-static","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-11T17:05:21Z","receivedAt":"2021-03-11T17:06:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a subsequent commit, pack-revindex.c will become responsible for\nsorting a list of objects in the \"MIDX pack order\" (which will be\ndefined in the following patch). To do so, it will need to be know the\npack identifier and offset within that pack for each object in the MIDX.\n\nThe MIDX code already has functions for doing just that\n(nth_midxed_offset() and nth_midxed_pack_int_id()), but they are\nstatically declared.\n\nSince there is no reason that they couldn't be exposed publicly, and\nbecause they are already doing exactly what the caller in\npack-revindex.c will want, expose them publicly so that they can be\nreused there.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 4 ++--\n midx.h | 2 ++\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 31e6d3d2df..0a5da49ed6 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -239,7 +239,7 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \treturn oid;\n }\n \n-static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n+off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n {\n \tconst unsigned char *offset_data;\n \tuint32_t offset32;\n@@ -258,7 +258,7 @@ static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n \treturn offset32;\n }\n \n-static uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n+uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n {\n \treturn get_be32(m->chunk_object_offsets +\n \t\t\t(off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);\ndiff --git a/midx.h b/midx.h\nindex e7fea61109..93bd68189e 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -40,6 +40,8 @@ struct multi_pack_index {\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n+off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);\n+uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);\n struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct multi_pack_index *m,\n \t\t\t\t\tuint32_t n);\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"418875","messageId":"4745bb8590f5cdc24445618dd63ba6bd541227b4.1615482270.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"[PATCH v3 12/16] Documentation/technical: describe multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-11T17:05:25Z","receivedAt":"2021-03-11T17:06:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"As a prerequisite to implementing multi-pack bitmaps, motivate and\ndescribe the format and ordering of the multi-pack reverse index.\n\nThe subsequent patch will implement reading this format, and the patch\nafter that will implement writing it while producing a multi-pack index.\n\nCo-authored-by: Jeff King \nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n Documentation/technical/pack-format.txt | 83 +++++++++++++++++++++++++\n 1 file changed, 83 insertions(+)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 1faa949bf6..4bbbb188a4 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -379,3 +379,86 @@ CHUNK DATA:\n TRAILER:\n \n \tIndex checksum of the above contents.\n+\n+== multi-pack-index reverse indexes\n+\n+Similar to the pack-based reverse index, the multi-pack index can also\n+be used to generate a reverse index.\n+\n+Instead of mapping between offset, pack-, and index position, this\n+reverse index maps between an object's position within the MIDX, and\n+that object's position within a pseudo-pack that the MIDX describes\n+(i.e., the ith entry of the multi-pack reverse index holds the MIDX\n+position of ith object in pseudo-pack order).\n+\n+To clarify the difference between these orderings, consider a multi-pack\n+reachability bitmap (which does not yet exist, but is what we are\n+building towards here). Each bit needs to correspond to an object in the\n+MIDX, and so we need an efficient mapping from bit position to MIDX\n+position.\n+\n+One solution is to let bits occupy the same position in the oid-sorted\n+index stored by the MIDX. But because oids are effectively random, there\n+resulting reachability bitmaps would have no locality, and thus compress\n+poorly. (This is the reason that single-pack bitmaps use the pack\n+ordering, and not the .idx ordering, for the same purpose.)\n+\n+So we'd like to define an ordering for the whole MIDX based around\n+pack ordering, which has far better locality (and thus compresses more\n+efficiently). We can think of a pseudo-pack created by the concatenation\n+of all of the packs in the MIDX. E.g., if we had a MIDX with three packs\n+(a, b, c), with 10, 15, and 20 objects respectively, we can imagine an\n+ordering of the objects like:\n+\n+ |a,0|a,1|...|a,9|b,0|b,1|...|b,14|c,0|c,1|...|c,19|\n+\n+where the ordering of the packs is defined by the MIDX's pack list,\n+and then the ordering of objects within each pack is the same as the\n+order in the actual packfile.\n+\n+Given the list of packs and their counts of objects, you can\n+naïvely reconstruct that pseudo-pack ordering (e.g., the object at\n+position 27 must be (c,1) because packs \"a\" and \"b\" consumed 25 of the\n+slots). But there's a catch. Objects may be duplicated between packs, in\n+which case the MIDX only stores one pointer to the object (and thus we'd\n+want only one slot in the bitmap).\n+\n+Callers could handle duplicates themselves by reading objects in order\n+of their bit-position, but that's linear in the number of objects, and\n+much too expensive for ordinary bitmap lookups. Building a reverse index\n+solves this, since it is the logical inverse of the index, and that\n+index has already removed duplicates. But, building a reverse index on\n+the fly can be expensive. Since we already have an on-disk format for\n+pack-based reverse indexes, let's reuse it for the MIDX's pseudo-pack,\n+too.\n+\n+Objects from the MIDX are ordered as follows to string together the\n+pseudo-pack. Let _pack(o)_ return the pack from which _o_ was selected\n+by the MIDX, and define an ordering of packs based on their numeric ID\n+(as stored by the MIDX). Let _offset(o)_ return the object offset of _o_\n+within _pack(o)_. Then, compare _o~1~_ and _o~2~_ as follows:\n+\n+ - If one of _pack(o~1~)_ and _pack(o~2~)_ is preferred and the other\n+ is not, then the preferred one sorts first.\n++\n+(This is a detail that allows the MIDX bitmap to determine which\n+pack should be used by the pack-reuse mechanism, since it can ask\n+the MIDX for the pack containing the object at bit position 0).\n+\n+ - If _pack(o~1~) ≠ pack(o~2~)_, then sort the two objects in\n+ descending order based on the pack ID.\n+\n+ - Otherwise, _pack(o~1~) = pack(o~2~)_, and the objects are\n+ sorted in pack-order (i.e., _o~1~_ sorts ahead of _o~2~_ exactly\n+ when _offset(o~1~) < offset(o~2~)_).\n+\n+In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n+objects in packs stored by the MIDX, laid out in pack order, and the\n+packs arranged in MIDX order (with the preferred pack coming first).\n+\n+Finally, note that the MIDX's reverse index is not stored as a chunk in\n+the multi-pack-index itself. This is done because the reverse index\n+includes the checksum of the pack or MIDX to which it belongs, which\n+makes it impossible to write in the MIDX. To avoid races when rewriting\n+the MIDX, a MIDX reverse index includes the MIDX's checksum in its\n+filename (e.g., `multi-pack-index-xyz.rev`).\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"418876","messageId":"a6ebd4be91e0ff0b31d1629c2e3a75c016d56397.1615482270.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"[PATCH v3 13/16] pack-revindex: read multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-11T17:05:29Z","receivedAt":"2021-03-11T17:06:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement reading for multi-pack reverse indexes, as described in the\nprevious patch.\n\nNote that these functions don't yet have any callers, and won't until\nmulti-pack reachability bitmaps are introduced in a later patch series.\nIn the meantime, this patch implements some of the infrastructure\nnecessary to support multi-pack bitmaps.\n\nThere are three new functions exposed by the revindex API:\n\n - load_midx_revindex(): loads the reverse index corresponding to the\n given multi-pack index.\n\n - midx_to_pack_pos() and pack_pos_to_midx(): these convert between the\n multi-pack index and pseudo-pack order.\n\nload_midx_revindex() and pack_pos_to_midx() are both relatively\nstraightforward.\n\nload_midx_revindex() needs a few functions to be exposed from the midx\nAPI. One to get the checksum of a midx, and another to get the .rev's\nfilename. Similar to recent changes in the packed_git struct, three new\nfields are added to the multi_pack_index struct: one to keep track of\nthe size, one to keep track of the mmap'd pointer, and another to point\npast the header and at the reverse index's data.\n\npack_pos_to_midx() simply reads the corresponding entry out of the\ntable.\n\nmidx_to_pack_pos() is the trickiest, since it needs to find an object's\nposition in the psuedo-pack order, but that order can only be recovered\nin the .rev file itself. This mapping can be implemented with a binary\nsearch, but note that the thing we're binary searching over isn't an\narray of values, but rather a permuted order of those values.\n\nSo, when comparing two items, it's helpful to keep in mind the\ndifference. Instead of a traditional binary search, where you are\ncomparing two things directly, here we're comparing a (pack, offset)\ntuple with an index into the multi-pack index. That index describes\nanother (pack, offset) tuple, and it is _those_ two tuples that are\ncompared.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 11 +++++\n midx.h | 6 +++\n pack-revindex.c | 127 ++++++++++++++++++++++++++++++++++++++++++++++++\n pack-revindex.h | 53 ++++++++++++++++++++\n packfile.c | 3 ++\n 5 files changed, 200 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex 0a5da49ed6..55f4567fca 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -47,11 +47,22 @@ static uint8_t oid_version(void)\n \t}\n }\n \n+static const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n+{\n+\treturn m->data + m->data_len - the_hash_algo->rawsz;\n+}\n+\n static char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\n \n+char *get_midx_rev_filename(struct multi_pack_index *m)\n+{\n+\treturn xstrfmt(\"%s/pack/multi-pack-index-%s.rev\",\n+\t\t m->object_dir, hash_to_hex(get_midx_checksum(m)));\n+}\n+\n static int midx_read_oid_fanout(const unsigned char *chunk_start,\n \t\t\t\tsize_t chunk_size, void *data)\n {\ndiff --git a/midx.h b/midx.h\nindex 93bd68189e..0a8294d2ee 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -15,6 +15,10 @@ struct multi_pack_index {\n \tconst unsigned char *data;\n \tsize_t data_len;\n \n+\tconst uint32_t *revindex_data;\n+\tconst uint32_t *revindex_map;\n+\tsize_t revindex_len;\n+\n \tuint32_t signature;\n \tunsigned char version;\n \tunsigned char hash_len;\n@@ -37,6 +41,8 @@ struct multi_pack_index {\n \n #define MIDX_PROGRESS (1 << 0)\n \n+char *get_midx_rev_filename(struct multi_pack_index *m);\n+\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 83fe4de773..2e15ba3a8f 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -3,6 +3,7 @@\n #include \"object-store.h\"\n #include \"packfile.h\"\n #include \"config.h\"\n+#include \"midx.h\"\n \n struct revindex_entry {\n \toff_t offset;\n@@ -292,6 +293,44 @@ int load_pack_revindex(struct packed_git *p)\n \treturn -1;\n }\n \n+int load_midx_revindex(struct multi_pack_index *m)\n+{\n+\tchar *revindex_name;\n+\tint ret;\n+\tif (m->revindex_data)\n+\t\treturn 0;\n+\n+\trevindex_name = get_midx_rev_filename(m);\n+\n+\tret = load_revindex_from_disk(revindex_name,\n+\t\t\t\t m->num_objects,\n+\t\t\t\t &m->revindex_map,\n+\t\t\t\t &m->revindex_len);\n+\tif (ret)\n+\t\tgoto cleanup;\n+\n+\tm->revindex_data = (const uint32_t *)((const char *)m->revindex_map + RIDX_HEADER_SIZE);\n+\n+cleanup:\n+\tfree(revindex_name);\n+\treturn ret;\n+}\n+\n+int close_midx_revindex(struct multi_pack_index *m)\n+{\n+\tif (!m)\n+\t\treturn 0;\n+\n+\tif (munmap((void*)m->revindex_map, m->revindex_len))\n+\t\treturn -1;\n+\n+\tm->revindex_map = NULL;\n+\tm->revindex_data = NULL;\n+\tm->revindex_len = 0;\n+\n+\treturn 0;\n+}\n+\n int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n {\n \tunsigned lo, hi;\n@@ -346,3 +385,91 @@ off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n \telse\n \t\treturn nth_packed_object_offset(p, pack_pos_to_index(p, pos));\n }\n+\n+uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos)\n+{\n+\tif (!m->revindex_data)\n+\t\tBUG(\"pack_pos_to_midx: reverse index not yet loaded\");\n+\tif (m->num_objects <= pos)\n+\t\tBUG(\"pack_pos_to_midx: out-of-bounds object at %\"PRIu32, pos);\n+\treturn get_be32((const char *)m->revindex_data + (pos * sizeof(uint32_t)));\n+}\n+\n+struct midx_pack_key {\n+\tuint32_t pack;\n+\toff_t offset;\n+\n+\tuint32_t preferred_pack;\n+\tstruct multi_pack_index *midx;\n+};\n+\n+static int midx_pack_order_cmp(const void *va, const void *vb)\n+{\n+\tconst struct midx_pack_key *key = va;\n+\tstruct multi_pack_index *midx = key->midx;\n+\n+\tuint32_t versus = pack_pos_to_midx(midx, (uint32_t*)vb - (const uint32_t *)midx->revindex_data);\n+\tuint32_t versus_pack = nth_midxed_pack_int_id(midx, versus);\n+\toff_t versus_offset;\n+\n+\tuint32_t key_preferred = key->pack == key->preferred_pack;\n+\tuint32_t versus_preferred = versus_pack == key->preferred_pack;\n+\n+\t/*\n+\t * First, compare the preferred-ness, noting that the preferred pack\n+\t * comes first.\n+\t */\n+\tif (key_preferred && !versus_preferred)\n+\t\treturn -1;\n+\telse if (!key_preferred && versus_preferred)\n+\t\treturn 1;\n+\n+\t/* Then, break ties first by comparing the pack IDs. */\n+\tif (key->pack < versus_pack)\n+\t\treturn -1;\n+\telse if (key->pack > versus_pack)\n+\t\treturn 1;\n+\n+\t/* Finally, break ties by comparing offsets within a pack. */\n+\tversus_offset = nth_midxed_offset(midx, versus);\n+\tif (key->offset < versus_offset)\n+\t\treturn -1;\n+\telse if (key->offset > versus_offset)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)\n+{\n+\tstruct midx_pack_key key;\n+\tuint32_t *found;\n+\n+\tif (!m->revindex_data)\n+\t\tBUG(\"midx_to_pack_pos: reverse index not yet loaded\");\n+\tif (m->num_objects <= at)\n+\t\tBUG(\"midx_to_pack_pos: out-of-bounds object at %\"PRIu32, at);\n+\n+\tkey.pack = nth_midxed_pack_int_id(m, at);\n+\tkey.offset = nth_midxed_offset(m, at);\n+\tkey.midx = m;\n+\t/*\n+\t * The preferred pack sorts first, so determine its identifier by\n+\t * looking at the first object in pseudo-pack order.\n+\t *\n+\t * Note that if no --preferred-pack is explicitly given when writing a\n+\t * multi-pack index, then whichever pack has the lowest identifier\n+\t * implicitly is preferred (and includes all its objects, since ties are\n+\t * broken first by pack identifier).\n+\t */\n+\tkey.preferred_pack = nth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));\n+\n+\tfound = bsearch(&key, m->revindex_data, m->num_objects,\n+\t\t\tsizeof(uint32_t), midx_pack_order_cmp);\n+\n+\tif (!found)\n+\t\treturn error(\"bad offset for revindex\");\n+\n+\t*pos = found - m->revindex_data;\n+\treturn 0;\n+}\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex ba7c82c125..479b8f2f9c 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -14,6 +14,20 @@\n *\n * - offset: the byte offset within the .pack file at which the object contents\n * can be found\n+ *\n+ * The revindex can also be used with a multi-pack index (MIDX). In this\n+ * setting:\n+ *\n+ * - index position refers to an object's numeric position within the MIDX\n+ *\n+ * - pack position refers to an object's position within a non-existent pack\n+ * described by the MIDX. The pack structure is described in\n+ * Documentation/technical/pack-format.txt.\n+ *\n+ * It is effectively a concatanation of all packs in the MIDX (ordered by\n+ * their numeric ID within the MIDX) in their original order within each\n+ * pack), removing duplicates, and placing the preferred pack (if any)\n+ * first.\n */\n \n \n@@ -24,6 +38,7 @@\n #define GIT_TEST_REV_INDEX_DIE_IN_MEMORY \"GIT_TEST_REV_INDEX_DIE_IN_MEMORY\"\n \n struct packed_git;\n+struct multi_pack_index;\n \n /*\n * load_pack_revindex populates the revindex's internal data-structures for the\n@@ -34,6 +49,22 @@ struct packed_git;\n */\n int load_pack_revindex(struct packed_git *p);\n \n+/*\n+ * load_midx_revindex loads the '.rev' file corresponding to the given\n+ * multi-pack index by mmap-ing it and assigning pointers in the\n+ * multi_pack_index to point at it.\n+ *\n+ * A negative number is returned on error.\n+ */\n+int load_midx_revindex(struct multi_pack_index *m);\n+\n+/*\n+ * Frees resources associated with a multi-pack reverse index.\n+ *\n+ * A negative number is returned on error.\n+ */\n+int close_midx_revindex(struct multi_pack_index *m);\n+\n /*\n * offset_to_pack_pos converts an object offset to a pack position. This\n * function returns zero on success, and a negative number otherwise. The\n@@ -71,4 +102,26 @@ uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);\n */\n off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos);\n \n+/*\n+ * pack_pos_to_midx converts the object at position \"pos\" within the MIDX\n+ * pseudo-pack into a MIDX position.\n+ *\n+ * If the reverse index has not yet been loaded, or the position is out of\n+ * bounds, this function aborts.\n+ *\n+ * This function runs in time O(log N) with the number of objects in the MIDX.\n+ */\n+uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos);\n+\n+/*\n+ * midx_to_pack_pos converts from the MIDX-relative position at \"at\" to the\n+ * corresponding pack position.\n+ *\n+ * If the reverse index has not yet been loaded, or the position is out of\n+ * bounds, this function aborts.\n+ *\n+ * This function runs in constant time.\n+ */\n+int midx_to_pack_pos(struct multi_pack_index *midx, uint32_t at, uint32_t *pos);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 1fec12ac5f..82623e0cb4 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -862,6 +862,9 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \n \tif (!strcmp(file_name, \"multi-pack-index\"))\n \t\treturn;\n+\tif (starts_with(file_name, \"multi-pack-index\") &&\n+\t ends_with(file_name, \".rev\"))\n+\t\treturn;\n \tif (ends_with(file_name, \".idx\") ||\n \t ends_with(file_name, \".rev\") ||\n \t ends_with(file_name, \".pack\") ||\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"418877","messageId":"f5314f1822ec9deb44bc78baa01b46a780581c53.1615482270.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"[PATCH v3 14/16] pack-write.c: extract 'write_rev_file_order'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-11T17:05:34Z","receivedAt":"2021-03-11T17:06:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Existing callers provide the reverse index code with an array of 'struct\npack_idx_entry *'s, which is then sorted by pack order (comparing the\noffsets of each object within the pack).\n\nPrepare for the multi-pack index to write a .rev file by providing a way\nto write the reverse index without an array of pack_idx_entry (which the\nMIDX code does not have).\n\nInstead, callers can invoke 'write_rev_index_positions()', which takes\nan array of uint32_t's. The ith entry in this array specifies the ith\nobject's (in index order) position within the pack (in pack order).\n\nExpose this new function for use in a later patch, and rewrite the\nexisting write_rev_file() in terms of this new function.\n\nSigned-off-by: Taylor Blau \n---\n pack-write.c | 36 +++++++++++++++++++++++++-----------\n pack.h | 1 +\n 2 files changed, 26 insertions(+), 11 deletions(-)\n\ndiff --git a/pack-write.c b/pack-write.c\nindex 2ca85a9d16..f1fc3ecafa 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -201,21 +201,12 @@ static void write_rev_header(struct hashfile *f)\n }\n \n static void write_rev_index_positions(struct hashfile *f,\n-\t\t\t\t struct pack_idx_entry **objects,\n+\t\t\t\t uint32_t *pack_order,\n \t\t\t\t uint32_t nr_objects)\n {\n-\tuint32_t *pack_order;\n \tuint32_t i;\n-\n-\tALLOC_ARRAY(pack_order, nr_objects);\n-\tfor (i = 0; i < nr_objects; i++)\n-\t\tpack_order[i] = i;\n-\tQSORT_S(pack_order, nr_objects, pack_order_cmp, objects);\n-\n \tfor (i = 0; i < nr_objects; i++)\n \t\thashwrite_be32(f, pack_order[i]);\n-\n-\tfree(pack_order);\n }\n \n static void write_rev_trailer(struct hashfile *f, const unsigned char *hash)\n@@ -228,6 +219,29 @@ const char *write_rev_file(const char *rev_name,\n \t\t\t uint32_t nr_objects,\n \t\t\t const unsigned char *hash,\n \t\t\t unsigned flags)\n+{\n+\tuint32_t *pack_order;\n+\tuint32_t i;\n+\tconst char *ret;\n+\n+\tALLOC_ARRAY(pack_order, nr_objects);\n+\tfor (i = 0; i < nr_objects; i++)\n+\t\tpack_order[i] = i;\n+\tQSORT_S(pack_order, nr_objects, pack_order_cmp, objects);\n+\n+\tret = write_rev_file_order(rev_name, pack_order, nr_objects, hash,\n+\t\t\t\t flags);\n+\n+\tfree(pack_order);\n+\n+\treturn ret;\n+}\n+\n+const char *write_rev_file_order(const char *rev_name,\n+\t\t\t\t uint32_t *pack_order,\n+\t\t\t\t uint32_t nr_objects,\n+\t\t\t\t const unsigned char *hash,\n+\t\t\t\t unsigned flags)\n {\n \tstruct hashfile *f;\n \tint fd;\n@@ -262,7 +276,7 @@ const char *write_rev_file(const char *rev_name,\n \n \twrite_rev_header(f);\n \n-\twrite_rev_index_positions(f, objects, nr_objects);\n+\twrite_rev_index_positions(f, pack_order, nr_objects);\n \twrite_rev_trailer(f, hash);\n \n \tif (rev_name && adjust_shared_perm(rev_name) < 0)\ndiff --git a/pack.h b/pack.h\nindex 857cbd5bd4..fa13954526 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -94,6 +94,7 @@ struct ref;\n void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought);\n \n const char *write_rev_file(const char *rev_name, struct pack_idx_entry **objects, uint32_t nr_objects, const unsigned char *hash, unsigned flags);\n+const char *write_rev_file_order(const char *rev_name, uint32_t *pack_order, uint32_t nr_objects, const unsigned char *hash, unsigned flags);\n \n /*\n * The \"hdr\" output buffer should be at least this big, which will handle sizes\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"418878","messageId":"fa3acb5d5af3aab46dd2a9703e4da03928625346.1615482270.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"[PATCH v3 15/16] pack-revindex: write multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-11T17:05:38Z","receivedAt":"2021-03-11T17:06:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement the writing half of multi-pack reverse indexes. This is\nnothing more than the format describe a few patches ago, with a new set\nof helper functions that will be used to clear out stale .rev files\ncorresponding to old MIDXs.\n\nUnfortunately, a very similar comparison function as the one implemented\nrecently in pack-revindex.c is reimplemented here, this time accepting a\nMIDX-internal type. An effort to DRY these up would create more\nindirection and overhead than is necessary, so it isn't pursued here.\n\nCurrently, there are no callers which pass the MIDX_WRITE_REV_INDEX\nflag, meaning that this is all dead code. But, that won't be the case\nfor long, since subsequent patches will introduce the multi-pack bitmap,\nwhich will begin passing this field.\n\n(In midx.c:write_midx_internal(), the two adjacent if statements share a\nconditional, but are written separately since the first one will\neventually also handle the MIDX_WRITE_BITMAP flag, which does not yet\nexist.)\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n midx.h | 1 +\n 2 files changed, 116 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex 55f4567fca..eea9574d92 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -12,6 +12,7 @@\n #include \"run-command.h\"\n #include \"repository.h\"\n #include \"chunk-format.h\"\n+#include \"pack.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -472,6 +473,7 @@ struct write_midx_context {\n \tuint32_t entries_nr;\n \n \tuint32_t *pack_perm;\n+\tuint32_t *pack_order;\n \tunsigned large_offsets_needed:1;\n \tuint32_t num_large_offsets;\n \n@@ -826,6 +828,70 @@ static int write_midx_large_offsets(struct hashfile *f,\n \treturn 0;\n }\n \n+static int midx_pack_order_cmp(const void *va, const void *vb, void *_ctx)\n+{\n+\tstruct write_midx_context *ctx = _ctx;\n+\n+\tstruct pack_midx_entry *a = &ctx->entries[*(const uint32_t *)va];\n+\tstruct pack_midx_entry *b = &ctx->entries[*(const uint32_t *)vb];\n+\n+\tuint32_t perm_a = ctx->pack_perm[a->pack_int_id];\n+\tuint32_t perm_b = ctx->pack_perm[b->pack_int_id];\n+\n+\t/* Sort objects in the preferred pack ahead of any others. */\n+\tif (a->preferred > b->preferred)\n+\t\treturn -1;\n+\tif (a->preferred < b->preferred)\n+\t\treturn 1;\n+\n+\t/* Then, order objects by which packs they appear in. */\n+\tif (perm_a < perm_b)\n+\t\treturn -1;\n+\tif (perm_a > perm_b)\n+\t\treturn 1;\n+\n+\t/* Then, disambiguate by their offset within each pack. */\n+\tif (a->offset < b->offset)\n+\t\treturn -1;\n+\tif (a->offset > b->offset)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n+{\n+\tuint32_t *pack_order;\n+\tuint32_t i;\n+\n+\tALLOC_ARRAY(pack_order, ctx->entries_nr);\n+\tfor (i = 0; i < ctx->entries_nr; i++)\n+\t\tpack_order[i] = i;\n+\tQSORT_S(pack_order, ctx->entries_nr, midx_pack_order_cmp, ctx);\n+\n+\treturn pack_order;\n+}\n+\n+static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n+\t\t\t\t struct write_midx_context *ctx)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *tmp_file;\n+\n+\tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex(midx_hash));\n+\n+\ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n+\t\t\t\t\tmidx_hash, WRITE_REV);\n+\n+\tif (finalize_object_file(tmp_file, buf.buf))\n+\t\tdie(_(\"cannot store reverse index file\"));\n+\n+\tstrbuf_release(&buf);\n+}\n+\n+static void clear_midx_files_ext(struct repository *r, const char *ext,\n+\t\t\t\t unsigned char *keep_hash);\n+\n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n \t\t\t struct string_list *packs_to_drop,\n \t\t\t const char *preferred_pack_name,\n@@ -1011,6 +1077,14 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n+\n+\tif (flags & MIDX_WRITE_REV_INDEX)\n+\t\tctx.pack_order = midx_pack_order(&ctx);\n+\n+\tif (flags & MIDX_WRITE_REV_INDEX)\n+\t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n+\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n+\n \tcommit_lock_file(&lk);\n \n cleanup:\n@@ -1025,6 +1099,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tfree(ctx.info);\n \tfree(ctx.entries);\n \tfree(ctx.pack_perm);\n+\tfree(ctx.pack_order);\n \tfree(midx_name);\n \treturn result;\n }\n@@ -1037,6 +1112,44 @@ int write_midx_file(const char *object_dir,\n \t\t\t\t flags);\n }\n \n+struct clear_midx_data {\n+\tchar *keep;\n+\tconst char *ext;\n+};\n+\n+static void clear_midx_file_ext(const char *full_path, size_t full_path_len,\n+\t\t\t\tconst char *file_name, void *_data)\n+{\n+\tstruct clear_midx_data *data = _data;\n+\n+\tif (!(starts_with(file_name, \"multi-pack-index-\") &&\n+\t ends_with(file_name, data->ext)))\n+\t\treturn;\n+\tif (data->keep && !strcmp(data->keep, file_name))\n+\t\treturn;\n+\n+\tif (unlink(full_path))\n+\t\tdie_errno(_(\"failed to remove %s\"), full_path);\n+}\n+\n+static void clear_midx_files_ext(struct repository *r, const char *ext,\n+\t\t\t\t unsigned char *keep_hash)\n+{\n+\tstruct clear_midx_data data;\n+\tmemset(&data, 0, sizeof(struct clear_midx_data));\n+\n+\tif (keep_hash)\n+\t\tdata.keep = xstrfmt(\"multi-pack-index-%s%s\",\n+\t\t\t\t hash_to_hex(keep_hash), ext);\n+\tdata.ext = ext;\n+\n+\tfor_each_file_in_pack_dir(r->objects->odb->path,\n+\t\t\t\t clear_midx_file_ext,\n+\t\t\t\t &data);\n+\n+\tfree(data.keep);\n+}\n+\n void clear_midx_file(struct repository *r)\n {\n \tchar *midx = get_midx_filename(r->objects->odb->path);\n@@ -1049,6 +1162,8 @@ void clear_midx_file(struct repository *r)\n \tif (remove_path(midx))\n \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n \n+\tclear_midx_files_ext(r, \".rev\", NULL);\n+\n \tfree(midx);\n }\n \ndiff --git a/midx.h b/midx.h\nindex 0a8294d2ee..8684cf0fef 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -40,6 +40,7 @@ struct multi_pack_index {\n };\n \n #define MIDX_PROGRESS (1 << 0)\n+#define MIDX_WRITE_REV_INDEX (1 << 1)\n \n char *get_midx_rev_filename(struct multi_pack_index *m);\n \n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"418879","messageId":"550e785f10ba14f166958501c007b75a04052a0d.1615482270.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"[PATCH v3 16/16] midx.c: improve cache locality in midx_pack_order_cmp()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-11T17:05:42Z","receivedAt":"2021-03-11T17:06:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nThere is a lot of pointer dereferencing in the pre-image version of\n'midx_pack_order_cmp()', which this patch gets rid of.\n\nInstead of comparing the pack preferred-ness and then the pack id, both\nof these checks are done at the same time by using the high-order bit of\nthe pack id to represent whether it's preferred. Then the pack id and\noffset are compared as usual.\n\nThis produces the same result so long as there are less than 2^31 packs,\nwhich seems like a likely assumption to make in practice.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n midx.c | 55 +++++++++++++++++++++++++++++--------------------------\n 1 file changed, 29 insertions(+), 26 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex eea9574d92..4835cc13d1 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -828,46 +828,49 @@ static int write_midx_large_offsets(struct hashfile *f,\n \treturn 0;\n }\n \n-static int midx_pack_order_cmp(const void *va, const void *vb, void *_ctx)\n+struct midx_pack_order_data {\n+\tuint32_t nr;\n+\tuint32_t pack;\n+\toff_t offset;\n+};\n+\n+static int midx_pack_order_cmp(const void *va, const void *vb)\n {\n-\tstruct write_midx_context *ctx = _ctx;\n-\n-\tstruct pack_midx_entry *a = &ctx->entries[*(const uint32_t *)va];\n-\tstruct pack_midx_entry *b = &ctx->entries[*(const uint32_t *)vb];\n-\n-\tuint32_t perm_a = ctx->pack_perm[a->pack_int_id];\n-\tuint32_t perm_b = ctx->pack_perm[b->pack_int_id];\n-\n-\t/* Sort objects in the preferred pack ahead of any others. */\n-\tif (a->preferred > b->preferred)\n+\tconst struct midx_pack_order_data *a = va, *b = vb;\n+\tif (a->pack < b->pack)\n \t\treturn -1;\n-\tif (a->preferred < b->preferred)\n+\telse if (a->pack > b->pack)\n \t\treturn 1;\n-\n-\t/* Then, order objects by which packs they appear in. */\n-\tif (perm_a < perm_b)\n+\telse if (a->offset < b->offset)\n \t\treturn -1;\n-\tif (perm_a > perm_b)\n+\telse if (a->offset > b->offset)\n \t\treturn 1;\n-\n-\t/* Then, disambiguate by their offset within each pack. */\n-\tif (a->offset < b->offset)\n-\t\treturn -1;\n-\tif (a->offset > b->offset)\n-\t\treturn 1;\n-\n-\treturn 0;\n+\telse\n+\t\treturn 0;\n }\n \n static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n {\n+\tstruct midx_pack_order_data *data;\n \tuint32_t *pack_order;\n \tuint32_t i;\n \n+\tALLOC_ARRAY(data, ctx->entries_nr);\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *e = &ctx->entries[i];\n+\t\tdata[i].nr = i;\n+\t\tdata[i].pack = ctx->pack_perm[e->pack_int_id];\n+\t\tif (!e->preferred)\n+\t\t\tdata[i].pack |= (1U << 31);\n+\t\tdata[i].offset = e->offset;\n+\t}\n+\n+\tQSORT(data, ctx->entries_nr, midx_pack_order_cmp);\n+\n \tALLOC_ARRAY(pack_order, ctx->entries_nr);\n \tfor (i = 0; i < ctx->entries_nr; i++)\n-\t\tpack_order[i] = i;\n-\tQSORT_S(pack_order, ctx->entries_nr, midx_pack_order_cmp, ctx);\n+\t\tpack_order[i] = data[i].nr;\n+\tfree(data);\n \n \treturn pack_order;\n }\n-- \n2.30.0.667.g81c0cbc6fd\n"},{"id":"418971","messageId":"c4e18157-3070-9ad1-00c9-1222340101ff@gmail.com","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"Re: [PATCH v3 00/16] midx: implement a multi-pack reverse index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-03-12T15:16:52Z","receivedAt":"2021-03-12T15:18:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/11/2021 12:04 PM, Taylor Blau wrote:\n> Here is another reroll of my series to implement a reverse index in\n> preparation for multi-pack reachability bitmaps. The previous version\n> was based on 'ds/chunked-file-api', but that topic has since been merged\n> to 'master'. This series is now built directly on top of 'master'.\n> \n> Not much has changed since last time. Jonathan Tan reviewed the previous\n> version, and I incorporated feedback from his review:\n> \n> - The usage macros in builtin/multi-pack-index.c were pulled out and\n> defined separately.\n> - Some sloppiness with converting a signed index referring to the\n> preferred pack into an unsigned value was cleaned up.\n> - Documentation clean-up, particularly in patches 12 and 13.\n> \n> There are a couple of new things that we found while testing this out at\n> GitHub.\n> \n> - We now call finalize_object_file() on the multi-pack reverse index\n> to set the correct permissions.\n> - Patch 14 removed a stray hunk that introduced a memory leak.\n> - Patch 16 (courtesy of Peff) is new. It improves the cache locality\n> of midx_pack_order_cmp(), which has a substantial impact on\n> repositories with many objects.\n> \n> Thanks in advance for your review.\n\nI've reviewed the changes since my last review and this one looks\ngood, including that new patch from Peff.\n\nThanks,\n-Stolee\n"},{"id":"420449","messageId":"YGG4F+TmtBC/WZmb@coredump.intra.peff.net","threadId":"55135","inReplyTo":"43fc0ad276406ff77283613c45188e102a6dc515.1615482270.git.me@ttaylorr.com","subject":"Re: [PATCH v3 01/16] builtin/multi-pack-index.c: inline 'flags' with options","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-29T11:20:55Z","receivedAt":"2021-03-29T11:21:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 11, 2021 at 12:04:36PM -0500, Taylor Blau wrote:\n\n> Subcommands of the 'git multi-pack-index' command (e.g., 'write',\n> 'verify', etc.) will want to optionally change a set of shared flags\n> that are eventually passed to the MIDX libraries.\n> \n> Right now, options and flags are handled separately. Inline them into\n> the same structure so that sub-commands can more easily share the\n> 'flags' data.\n\nThis \"opts\" struct is kind of funny. It is used to collect the options\nin cmd_multi_pack_index(), but nobody ever passes it anywhere! Instead,\nwe pass individual components of it around.\n\nSo I'm not sure I buy \"...so that sub-commands can more easily share the\nflags data\", since either way they are all receiving the individual\nflags field already. And your patch 2 could just as easily do the same\nsimplification by modifying the function-local \"flags\" variable.\n\nBut. I think things get more interesting when you later introduce\ncommon_opts, because now those options have to refer back to actuals\nstorage for each item. Which means that \"flags\" would have to become a\nglobal variable. And there it's nicer to have all of the options stuffed\ninto a struct, even if it is a single global struct.\n\nSo I think this is the right direction, but it took me a minute to\nrealize quite why.\n\n-Peff\n"},{"id":"420450","messageId":"YGG4k5XUKj073qS5@coredump.intra.peff.net","threadId":"55135","inReplyTo":"181f11e4c55b364dc7f6a6530f397779171671a9.1615482270.git.me@ttaylorr.com","subject":"Re: [PATCH v3 02/16] builtin/multi-pack-index.c: don't handle 'progress' separately","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-29T11:22:59Z","receivedAt":"2021-03-29T11:23:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 11, 2021 at 12:04:40PM -0500, Taylor Blau wrote:\n\n> Now that there is a shared 'flags' member in the options structure,\n> there is no need to keep track of whether to force progress or not,\n> since ultimately the decision of whether or not to show a progress meter\n> is controlled by a bit in the flags member.\n\nJust going back to what I wrote for patch 1, I think this \"now that\nthere is a shared flags...\" bit is what misled me.\n\nYou can easily have done this patch by just manipulating the local\n\"flags\" variable. And the rationale for the patch is \"we can get rid of\nopts.progress, because nobody ever reads it except to set a bit\nopts.flags\".\n\nDefinitely not worth re-rolling or anything; I'm just explaining my\nearlier comments. :)\n\n-Peff\n"},{"id":"420451","messageId":"YGG7tWBzo5NGl2+g@coredump.intra.peff.net","threadId":"55135","inReplyTo":"d084f90466813597feb27975654e57de36eb62a4.1615482270.git.me@ttaylorr.com","subject":"Re: [PATCH v3 04/16] builtin/multi-pack-index.c: split sub-commands","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-29T11:36:21Z","receivedAt":"2021-03-29T11:37:15Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 11, 2021 at 12:04:49PM -0500, Taylor Blau wrote:\n\n> Handle sub-commands of the 'git multi-pack-index' builtin (e.g.,\n> \"write\", \"repack\", etc.) separately from one another. This allows\n> sub-commands with unique options, without forcing cmd_multi_pack_index()\n> to reject invalid combinations itself.\n> \n> This comes at the cost of some duplication and boilerplate. Luckily, the\n> duplication is reduced to a minimum, since common options are shared\n> among sub-commands due to a suggestion by Ævar. (Sub-commands do have to\n> retain the common options, too, since this builtin accepts common\n> options on either side of the sub-command).\n> \n> Roughly speaking, cmd_multi_pack_index() parses options (including\n> common ones), and stops at the first non-option, which is the\n> sub-command. It then dispatches to the appropriate sub-command, which\n> parses the remaining options (also including common options).\n> \n> Unknown options are kept by the sub-commands in order to detect their\n> presence (and complain that too many arguments were given).\n\nMakes sense, and the implementation looks pretty clean.\n\nA few small nits:\n\n> +static struct option *add_common_options(struct option *prev)\n> {\n> -\tstatic struct option builtin_multi_pack_index_options[] = {\n> -\t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n> -\t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n> -\t\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"), MIDX_PROGRESS),\n> +\tstruct option *with_common = parse_options_concat(common_opts, prev);\n> +\tfree(prev);\n> +\treturn with_common;\n> +}\n\nThis free(prev) pattern is copied from builtin/checkout.c, where we have\nmultiple layers of options, each added by a function. So it requires\nthat callers duplicate the base set of options, and each subsequent\n\"add_foo_options()\" concatenates that and frees the old one.\n\nBut here, we only have one layer, so in the caller which uses it:\n\n> +static int cmd_multi_pack_index_repack(int argc, const char **argv)\n> +{\n> [..]\n> +\toptions = parse_options_dup(builtin_multi_pack_index_repack_options);\n> +\toptions = add_common_options(options);\n\nwe do a rather pointless dup() followed by free(). Perhaps not that big\na deal, and this would naturally extend to adding other option sets, so\nit may even be considered future-proofing. But it did confuse me for a\nmoment.\n\nHowever, we do end up leaking the return value from add_common_options()\nat the end of the function:\n\n> +\toptions = parse_options_dup(builtin_multi_pack_index_repack_options);\n> +\toptions = add_common_options(options);\n> +\n> +\targc = parse_options(argc, argv, NULL,\n> +\t\t\t options,\n> +\t\t\t builtin_multi_pack_index_repack_usage,\n> +\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n> +\tif (argc)\n> +\t\tusage_with_options(builtin_multi_pack_index_repack_usage,\n> +\t\t\t\t options);\n> +\n> +\treturn midx_repack(the_repository, opts.object_dir,\n> +\t\t\t (size_t)opts.batch_size, opts.flags);\n> +}\n\nThis is definitely a harmless leak in the sense that we are going to\nexit the program after midx_repack() returns anyway. But it might be\nworth keeping things tidy, as we've recently seen a renewed effort to do\nsome leak-checking of the test suite. I _think_ we can just free the\noptions struct (even though we are still using the values themselves, we\ndon't care about the \"struct options\" anymore). But even if not, an\nUNLEAK(options) annotation would do it.\n\n(This doesn't apply to the other functions, because they just use\ncommon_opts directly).\n\n-Peff\n"},{"id":"420452","messageId":"YGG9GogdjlijGxSF@coredump.intra.peff.net","threadId":"55135","inReplyTo":"f117e442c3460661dc88beba6f1853d6c388b0fd.1615482270.git.me@ttaylorr.com","subject":"Re: [PATCH v3 06/16] builtin/multi-pack-index.c: display usage on unrecognized command","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-29T11:42:18Z","receivedAt":"2021-03-29T11:43:10Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 11, 2021 at 12:04:57PM -0500, Taylor Blau wrote:\n\n> When given a sub-command that it doesn't understand, 'git\n> multi-pack-index' dies with the following message:\n> \n> $ git multi-pack-index bogus\n> fatal: unrecognized subcommand: bogus\n> \n> Instead of 'die()'-ing, we can display the usage text, which is much\n> more helpful:\n> \n> $ git.compile multi-pack-index bogus\n> usage: git multi-pack-index [] write\n> or: git multi-pack-index [] verify\n> or: git multi-pack-index [] expire\n> or: git multi-pack-index [] repack [--batch-size=]\n> \n> \t--object-dir object directory containing set of packfile and pack-index pairs\n> \t--progress force progress reporting\n> \n> While we're at it, clean up some duplication between the \"no sub-command\"\n> and \"unrecognized sub-command\" conditionals.\n\nI agree that it's much nicer to give the usage. But my preference in\ngeneral for cases like this is to _also_ explain what we found wrong\nwith the options we were given.\n\nE.g., with a bogus option, we say so:\n\n $ git multi-pack-index --foo\n error: unknown option `foo'\n usage: git multi-pack-index [] write [--preferred-pack=]\n [etc...]\n\nbut with a bogus sub-command, we get just the usage string:\n\n $ git multi-pack-index foo\n usage: git multi-pack-index [] write [--preferred-pack=]\n [etc...]\n\nSometimes it is quote obvious what is wrong, but sometimes typos can be\nhard to spot, especially because the usage message is so long.\n\nI.e., I'd suggest changing this:\n\n> \telse\n> -\t\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n> +usage:\n> +\t\tusage_with_options(builtin_multi_pack_index_usage,\n> +\t\t\t\t builtin_multi_pack_index_options);\n\nto:\n\n error(_(\"unrecognized subcommand: %s\"), argv[0]);\n usage_with_options(...);\n\n-Peff\n"},{"id":"420453","messageId":"YGHBe1ZB/iCRpqgD@coredump.intra.peff.net","threadId":"55135","inReplyTo":"30194a6786bec51e0f41de0e6c855dc2297806c6.1615482270.git.me@ttaylorr.com","subject":"Re: [PATCH v3 08/16] midx: allow marking a pack as preferred","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-29T12:00:59Z","receivedAt":"2021-03-29T12:01:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 11, 2021 at 12:05:07PM -0500, Taylor Blau wrote:\n\n> To encourage the pack selection process to prefer one pack over another\n> (the pack to be preferred is the one a caller would like to later use as\n> a reuse pack), introduce the concept of a \"preferred pack\". When\n> provided, the MIDX code will always prefer an object found in a\n> preferred pack over any other.\n> \n> No format changes are required to store the preferred pack, since it\n> will be able to be inferred with a corresponding MIDX bitmap, by looking\n> up the pack associated with the object in the first bit position (this\n> ordering is described in detail in a subsequent commit).\n\nI think in the long run we may want to add a midx chunk that gives the\norder of the packs (and likewise allow the caller of \"midx write\" to\nspecify the exact order), since that may allow correlating locality\nbetween history and object order within the .rev/.bitmap files.\n\nBut I think this is a nice stopping point for this series, since we're\nnot having to introduce any new on-disk formats to do it, and it seems\nto give pretty good results in practice. I guess we'll have to support\n--preferred-pack forever, but that's OK. Even if we do eventually\nsupport arbitrary orderings, it's just a simple subset of that\nfunctionality.\n\n> static int cmd_multi_pack_index_write(int argc, const char **argv)\n> {\n> -\tstruct option *options = common_opts;\n> +\tstruct option *options;\n> +\tstatic struct option builtin_multi_pack_index_write_options[] = {\n> +\t\tOPT_STRING(0, \"preferred-pack\", &opts.preferred_pack,\n> +\t\t\t N_(\"preferred-pack\"),\n> +\t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n> +\t\tOPT_END(),\n> +\t};\n> +\n> +\toptions = parse_options_dup(builtin_multi_pack_index_write_options);\n> +\toptions = add_common_options(options);\n> \n> \ttrace2_cmd_mode(argv[0]);\n> \n> @@ -74,7 +85,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n> \t\tusage_with_options(builtin_multi_pack_index_write_usage,\n> \t\t\t\t options);\n> \n> -\treturn write_midx_file(opts.object_dir, opts.flags);\n> +\treturn write_midx_file(opts.object_dir, opts.preferred_pack,\n> +\t\t\t opts.flags);\n> }\n\nThis has the same leak of \"options\" that I mentioned in the earlier\npatch.\n\n> diff --git a/midx.c b/midx.c\n> index 971faa8cfc..46f55ff6cf 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -431,6 +431,24 @@ static int pack_info_compare(const void *_a, const void *_b)\n> \treturn strcmp(a->pack_name, b->pack_name);\n> }\n> \n> +static int lookup_idx_or_pack_name(struct pack_info *info,\n> +\t\t\t\t uint32_t nr,\n> +\t\t\t\t const char *pack_name)\n> +{\n> +\tuint32_t lo = 0, hi = nr;\n> +\twhile (lo < hi) {\n> +\t\tuint32_t mi = lo + (hi - lo) / 2;\n> +\t\tint cmp = cmp_idx_or_pack_name(pack_name, info[mi].pack_name);\n> +\t\tif (cmp < 0)\n> +\t\t\thi = mi;\n> +\t\telse if (cmp > 0)\n> +\t\t\tlo = mi + 1;\n> +\t\telse\n> +\t\t\treturn mi;\n> +\t}\n> +\treturn -1;\n> +}\n\nCould this just be replaced with bsearch() in the caller?\n\n> +test_expect_success 'midx picks objects from preferred pack' '\n> +\ttest_when_finished rm -rf preferred.git &&\n> +\tgit init --bare preferred.git &&\n> +\t(\n> +\t\tcd preferred.git &&\n> +\n> +\t\ta=$(echo \"a\" | git hash-object -w --stdin) &&\n> +\t\tb=$(echo \"b\" | git hash-object -w --stdin) &&\n> +\t\tc=$(echo \"c\" | git hash-object -w --stdin) &&\n> +\n> +\t\t# Set up two packs, duplicating the object \"B\" at different\n> +\t\t# offsets.\n> +\t\tgit pack-objects objects/pack/test-AB <<-EOF &&\n> +\t\t$a\n> +\t\t$b\n> +\t\tEOF\n> +\t\tbc=$(git pack-objects objects/pack/test-BC <<-EOF\n> +\t\t$b\n> +\t\t$c\n> +\t\tEOF\n> +\t\t) &&\n\nI don't think pack-objects guarantees that the pack ordering matches the\ninput it received. compute_write_order() uses a variety of heuristics to\nreorder things. I think this will work in practice with the current\ncode, because the objects have the same type, there are no deltas, and\nthe fallback ordering is input-order (or traversal order, if --revs is\nused).\n\nSo it's probably OK in practice, though if we wanted to be paranoid we\ncould check that show-index produces different results for the $b entry\nof both packs. That said...\n\n> +\t\tgit multi-pack-index --object-dir=objects \\\n> +\t\t\twrite --preferred-pack=test-BC-$bc.idx 2>err &&\n> +\t\ttest_must_be_empty err &&\n> +\n> +\t\tofs=$(git show-index +\t\t\tcut -d\" \" -f1) &&\n> +\t\tmidx_expect_object_offset $b $ofs objects\n> +\t)\n\n...what we really care about is that the object came from BC. And we are\njust using the offset as a proxy for that. But doesn't \"test-tool\nread-midx\" give us the actual pack name? We could just be checking that.\n\nI also wondered if we should confirm that without the --preferred-pack\noption, we choose the other pack. I think it will always be true because\nthe default order is to sort them lexically. A comment to that effect\nmight be worth it (near the \"set up two packs\" comment).\n\n-Peff\n"},{"id":"420454","messageId":"YGHEN6rcao2tJ006@coredump.intra.peff.net","threadId":"55135","inReplyTo":"4745bb8590f5cdc24445618dd63ba6bd541227b4.1615482270.git.me@ttaylorr.com","subject":"Re: [PATCH v3 12/16] Documentation/technical: describe multi-pack reverse indexes","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-29T12:12:39Z","receivedAt":"2021-03-29T12:13:35Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 11, 2021 at 12:05:25PM -0500, Taylor Blau wrote:\n\n> As a prerequisite to implementing multi-pack bitmaps, motivate and\n> describe the format and ordering of the multi-pack reverse index.\n\nNicely written overall. I found a few typos / formatting issues.\n\n> +One solution is to let bits occupy the same position in the oid-sorted\n> +index stored by the MIDX. But because oids are effectively random, there\n\ns/there/their/\n\n> +Given the list of packs and their counts of objects, you can\n> +naïvely reconstruct that pseudo-pack ordering (e.g., the object at\n\nAn HTML entity seems to have snuck in. The source is utf8, so we can\njust say ï.\n\n> +position 27 must be (c,1) because packs \"a\" and \"b\" consumed 25 of the\n> +slots). But there's a catch. Objects may be duplicated between packs, in\n> +which case the MIDX only stores one pointer to the object (and thus we'd\n> +want only one slot in the bitmap).\n> +\n> +Callers could handle duplicates themselves by reading objects in order\n> +of their bit-position, but that's linear in the number of objects, and\n> +much too expensive for ordinary bitmap lookups. Building a reverse index\n> +solves this, since it is the logical inverse of the index, and that\n> +index has already removed duplicates. But, building a reverse index on\n> +the fly can be expensive. Since we already have an on-disk format for\n> +pack-based reverse indexes, let's reuse it for the MIDX's pseudo-pack,\n> +too.\n\nYep, I think this nicely builds up the logic explaining the need for the\nmidx .rev file.\n\n> +Objects from the MIDX are ordered as follows to string together the\n> +pseudo-pack. Let _pack(o)_ return the pack from which _o_ was selected\n> +by the MIDX, and define an ordering of packs based on their numeric ID\n> +(as stored by the MIDX). Let _offset(o)_ return the object offset of _o_\n> +within _pack(o)_. Then, compare _o~1~_ and _o~2~_ as follows:\n\nI guess the asciidoc-formatted version of this makes these nicely\nitalicized and subscripted. Personally I think pack(o) and o1 would be\nmore readable in the source (which is what I would tend to read). Or\nmaybe backticks if you want to be fancy.\n\n> + - If _pack(o~1~) ≠ pack(o~2~)_, then sort the two objects in\n> + descending order based on the pack ID.\n> +\n> + - Otherwise, _pack(o~1~) = pack(o~2~)_, and the objects are\n> + sorted in pack-order (i.e., _o~1~_ sorts ahead of _o~2~_ exactly\n> + when _offset(o~1~) < offset(o~2~)_).\n\nA few more HTML bits in the comparison operators.\n\n-Peff\n"},{"id":"420461","messageId":"YGHLe0nmh7R8hsVo@coredump.intra.peff.net","threadId":"55135","inReplyTo":"a6ebd4be91e0ff0b31d1629c2e3a75c016d56397.1615482270.git.me@ttaylorr.com","subject":"Re: [PATCH v3 13/16] pack-revindex: read multi-pack reverse indexes","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-29T12:43:39Z","receivedAt":"2021-03-29T12:44:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 11, 2021 at 12:05:29PM -0500, Taylor Blau wrote:\n\n> Implement reading for multi-pack reverse indexes, as described in the\n> previous patch.\n\nLooks good overall. I found a few tiny nits below.\n\n> +int load_midx_revindex(struct multi_pack_index *m)\n> +{\n> +\tchar *revindex_name;\n> +\tint ret;\n> +\tif (m->revindex_data)\n> +\t\treturn 0;\n> +\n> +\trevindex_name = get_midx_rev_filename(m);\n> +\n> +\tret = load_revindex_from_disk(revindex_name,\n> +\t\t\t\t m->num_objects,\n> +\t\t\t\t &m->revindex_map,\n> +\t\t\t\t &m->revindex_len);\n> +\tif (ret)\n> +\t\tgoto cleanup;\n\nOn error, I wondered if m->revindex_map, etc, would be modified. But it\nlooks like no, load_revindex_from_disk() is careful not to touch them\nunless it sees a valid revindex. Good.\n\n> +int close_midx_revindex(struct multi_pack_index *m)\n> +{\n> +\tif (!m)\n> +\t\treturn 0;\n> +\n> +\tif (munmap((void*)m->revindex_map, m->revindex_len))\n> +\t\treturn -1;\n> +\n> +\tm->revindex_map = NULL;\n> +\tm->revindex_data = NULL;\n> +\tm->revindex_len = 0;\n> +\n> +\treturn 0;\n> +}\n\nIt's hard to imagine why munmap() would fail. But if it does, we should\nprobably clear the struct fields anyway. I note that the matching code\nfor a \"struct packed_git\" does not bother even checking the return value\nof munmap. Perhaps we should just do the same here.\n\nThe packed_git version also returned early if revindex_map is NULL. Here\nthe burden is placed on the caller (it's hard to tell if that matters\nsince there aren't any callers yet, but it probably makes sense to push\nthe check down into this function).\n\n> +uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos)\n> +{\n> +\tif (!m->revindex_data)\n> +\t\tBUG(\"pack_pos_to_midx: reverse index not yet loaded\");\n> +\tif (m->num_objects <= pos)\n> +\t\tBUG(\"pack_pos_to_midx: out-of-bounds object at %\"PRIu32, pos);\n> +\treturn get_be32((const char *)m->revindex_data + (pos * sizeof(uint32_t)));\n> +}\n\nOK, this one is just a direct read of the .rev data, like\npack_pos_to_index() is. I think the final line can be simplified to:\n\n return get_be32(m->revindex_data + pos);\n\njust like pack_pos_to_index(). (I suspect this is a leftover from the\nearlier version of your .rev series where the pointer was still a \"void\n*\").\n\n> +int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)\n> +{\n> +\tstruct midx_pack_key key;\n> +\tuint32_t *found;\n> +\n> +\tif (!m->revindex_data)\n> +\t\tBUG(\"midx_to_pack_pos: reverse index not yet loaded\");\n> +\tif (m->num_objects <= at)\n> +\t\tBUG(\"midx_to_pack_pos: out-of-bounds object at %\"PRIu32, at);\n> +\n> +\tkey.pack = nth_midxed_pack_int_id(m, at);\n> +\tkey.offset = nth_midxed_offset(m, at);\n> +\tkey.midx = m;\n> +\t/*\n> +\t * The preferred pack sorts first, so determine its identifier by\n> +\t * looking at the first object in pseudo-pack order.\n> +\t *\n> +\t * Note that if no --preferred-pack is explicitly given when writing a\n> +\t * multi-pack index, then whichever pack has the lowest identifier\n> +\t * implicitly is preferred (and includes all its objects, since ties are\n> +\t * broken first by pack identifier).\n> +\t */\n> +\tkey.preferred_pack = nth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));\n> +\n> +\tfound = bsearch(&key, m->revindex_data, m->num_objects,\n> +\t\t\tsizeof(uint32_t), midx_pack_order_cmp);\n\nOK, this one is _roughly_ equivalent to offset_to_pack_pos(), in that we\nhave to binary search within the pack-ordered list to find the entry.\nMakes sense.\n\nProbably sizeof(*m->revindex_data) would be slightly nicer in the\nbsearch call (again, I suspect a holdover from when that was a void\npointer).\n\n-Peff\n"},{"id":"420462","messageId":"YGHNwin9zcrP00dI@coredump.intra.peff.net","threadId":"55135","inReplyTo":"fa3acb5d5af3aab46dd2a9703e4da03928625346.1615482270.git.me@ttaylorr.com","subject":"Re: [PATCH v3 15/16] pack-revindex: write multi-pack reverse indexes","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-29T12:53:22Z","receivedAt":"2021-03-29T12:54:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 11, 2021 at 12:05:38PM -0500, Taylor Blau wrote:\n\n> Implement the writing half of multi-pack reverse indexes. This is\n> nothing more than the format describe a few patches ago, with a new set\n> of helper functions that will be used to clear out stale .rev files\n> corresponding to old MIDXs.\n\nLooks good.\n\n> +struct clear_midx_data {\n> +\tchar *keep;\n> +\tconst char *ext;\n> +};\n> +\n> +static void clear_midx_file_ext(const char *full_path, size_t full_path_len,\n> +\t\t\t\tconst char *file_name, void *_data)\n\nThis will clean up _any_ stale midx .rev file. So even if we miss one\nwhen writing a new midx (due to a bug, race, power loss, etc), we'll\ncatch it later.\n\nWe _might_ want to also teach various tempfile-cleanup code run by gc to\nlikewise look for unattached midx .rev files, but I don't think we\nnecessarily have to do it now.\n\n> void clear_midx_file(struct repository *r)\n> {\n> \tchar *midx = get_midx_filename(r->objects->odb->path);\n> @@ -1049,6 +1162,8 @@ void clear_midx_file(struct repository *r)\n> \tif (remove_path(midx))\n> \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n> \n> +\tclear_midx_files_ext(r, \".rev\", NULL);\n> +\n> \tfree(midx);\n\nThe sole caller now doesn't pass the \"keep\" hash, so we'd always delete\nall of them. I guess we'll see that change once somebody starts actually\nwriting them.\n\n-Peff\n"},{"id":"420463","messageId":"YGHPINrwIZ1mKK+y@coredump.intra.peff.net","threadId":"55135","inReplyTo":"550e785f10ba14f166958501c007b75a04052a0d.1615482270.git.me@ttaylorr.com","subject":"Re: [PATCH v3 16/16] midx.c: improve cache locality in midx_pack_order_cmp()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-29T12:59:12Z","receivedAt":"2021-03-29T13:00:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 11, 2021 at 12:05:42PM -0500, Taylor Blau wrote:\n\n> From: Jeff King \n> \n> There is a lot of pointer dereferencing in the pre-image version of\n> 'midx_pack_order_cmp()', which this patch gets rid of.\n> \n> Instead of comparing the pack preferred-ness and then the pack id, both\n> of these checks are done at the same time by using the high-order bit of\n> the pack id to represent whether it's preferred. Then the pack id and\n> offset are compared as usual.\n> \n> This produces the same result so long as there are less than 2^31 packs,\n> which seems like a likely assumption to make in practice.\n\nObviously this patch is brilliant. ;)\n\nDid we record any numbers to show the improvement here? I don't think it\ncan be demonstrated with this series (since most of the code is dead),\nbut I recall that this was motivated by a noticeable slowdown.\n\nI briefly wondered whether the complicated midx_pack_order_cmp() in\npack-revindex.c, which is used for the bsearch() there, could benefit\nfrom the same speedup. It's only log(n), of course, instead of n*log(n),\nbut one might imagine making \"n\" calls to it. I don't think it makes\nsense, though. The pointer dereferencing there is into the midx mmap\nitself. Creating an auxiliary array would defeat the purpose.\n\n-Peff\n"},{"id":"420464","messageId":"YGHQnQ9/ulXd+jgu@coredump.intra.peff.net","threadId":"55135","inReplyTo":"cover.1615482270.git.me@ttaylorr.com","subject":"Re: [PATCH v3 00/16] midx: implement a multi-pack reverse index","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-29T13:05:33Z","receivedAt":"2021-03-29T13:06:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 11, 2021 at 12:04:31PM -0500, Taylor Blau wrote:\n\n> Here is another reroll of my series to implement a reverse index in\n> preparation for multi-pack reachability bitmaps. The previous version\n> was based on 'ds/chunked-file-api', but that topic has since been merged\n> to 'master'. This series is now built directly on top of 'master'.\n\nI gave the whole thing another careful read. Most of what I found were\nsmall nits, but enough that I think one more re-roll is worth it.\n\nThe biggest question is what we want to happen next. As you note, the\nconcept of a midx .rev file is useless until we have the matching\n.bitmap file. So we _could_ let this sit in next while the dependent\nbitmap topic is reviewed, and then merge them down together. But I'm\ninclined to treat this as an independent topic that can get merged to\nmaster on its own, since the early cleanups are valuable on their own,\nand the .rev parts at the end, even if dead, won't hurt anything.\n\nIf we did want to break it up, the useful line would be after \"allow\nmarking a pack as preferred\" (while it is mostly intended for the bitmap\nselection, it is theoretically useful on its own to make it more likely\nto find a copy of an object with a useful delta).\n\n-Peff\n"},{"id":"420507","messageId":"YGI6ySogGoYZi66A@nand.local","threadId":"55135","inReplyTo":"YGG7tWBzo5NGl2+g@coredump.intra.peff.net","subject":"Re: [PATCH v3 04/16] builtin/multi-pack-index.c: split sub-commands","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-29T20:38:33Z","receivedAt":"2021-03-29T20:39:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 29, 2021 at 07:36:21AM -0400, Jeff King wrote:\n> This is definitely a harmless leak in the sense that we are going to\n> exit the program after midx_repack() returns anyway. But it might be\n> worth keeping things tidy, as we've recently seen a renewed effort to do\n> some leak-checking of the test suite. I _think_ we can just free the\n> options struct (even though we are still using the values themselves, we\n> don't care about the \"struct options\" anymore). But even if not, an\n> UNLEAK(options) annotation would do it.\n\nI see what you're saying. Let me make sure that I got the right idea in\nmind after reading your email. I'm thinking of squashing the following\ndiff into this patch. For what it's worth, it causes 'valgrind\n--leak-check=full ./git-multi-pack-index repack' to exit cleanly (when\nit didn't before).\n\nDoes this match your expectations?\n\n--- >8 ---\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 23e51dfeb4..a78640c061 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -56,9 +56,7 @@ static struct option common_opts[] = {\n\n static struct option *add_common_options(struct option *prev)\n {\n-\tstruct option *with_common = parse_options_concat(common_opts, prev);\n-\tfree(prev);\n-\treturn with_common;\n+\treturn parse_options_concat(common_opts, prev);\n }\n\n static int cmd_multi_pack_index_write(int argc, const char **argv)\n@@ -112,8 +110,7 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv)\n \t\tOPT_END(),\n \t};\n\n-\toptions = parse_options_dup(builtin_multi_pack_index_repack_options);\n-\toptions = add_common_options(options);\n+\toptions = add_common_options(builtin_multi_pack_index_repack_options);\n\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options,\n@@ -123,6 +120,8 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv)\n \t\tusage_with_options(builtin_multi_pack_index_repack_usage,\n \t\t\t\t options);\n\n+\tFREE_AND_NULL(options);\n+\n \treturn midx_repack(the_repository, opts.object_dir,\n \t\t\t (size_t)opts.batch_size, opts.flags);\n }\n"},{"id":"420508","messageId":"YGI7b1xTYOQeyEy3@nand.local","threadId":"55135","inReplyTo":"YGG9GogdjlijGxSF@coredump.intra.peff.net","subject":"Re: [PATCH v3 06/16] builtin/multi-pack-index.c: display usage on unrecognized command","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-29T20:41:19Z","receivedAt":"2021-03-29T20:42:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 29, 2021 at 07:42:18AM -0400, Jeff King wrote:\n> I.e., I'd suggest changing this:\n>\n> > \telse\n> > -\t\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n> > +usage:\n> > +\t\tusage_with_options(builtin_multi_pack_index_usage,\n> > +\t\t\t\t builtin_multi_pack_index_options);\n>\n> to:\n>\n> error(_(\"unrecognized subcommand: %s\"), argv[0]);\n> usage_with_options(...);\n\nThanks, that's a helpful suggestion (and pretty easy to change, too).\n\nThanks,\nTaylor\n"},{"id":"420513","messageId":"YGJDYIxicaPDMdeZ@nand.local","threadId":"55135","inReplyTo":"YGHBe1ZB/iCRpqgD@coredump.intra.peff.net","subject":"Re: [PATCH v3 08/16] midx: allow marking a pack as preferred","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-29T21:15:12Z","receivedAt":"2021-03-29T21:16:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 29, 2021 at 08:00:59AM -0400, Jeff King wrote:\n> I think in the long run we may want to add a midx chunk that gives the\n> order of the packs (and likewise allow the caller of \"midx write\" to\n> specify the exact order), since that may allow correlating locality\n> between history and object order within the .rev/.bitmap files.\n>\n> But I think this is a nice stopping point for this series, since we're\n> not having to introduce any new on-disk formats to do it, and it seems\n> to give pretty good results in practice. I guess we'll have to support\n> --preferred-pack forever, but that's OK. Even if we do eventually\n> support arbitrary orderings, it's just a simple subset of that\n> functionality.\n\nTo add a little bit of extra detail, I think what you're getting at here\nis that it would be nice to let the order of the packs be dictated by\nmtime, not the order they appear in the MIDX (which is lexicographic by\ntheir hash, and thus effectively random).\n\nThe reason there being the same as you pointed out in\n\n https://lore.kernel.org/git/YDRdmh8oS5%2Fxq4rB@coredump.intra.peff.net/\n\nwhich is that it effectively would lay objects out from newest to\noldest.\n\nBut, there's a problem, which is that the MIDX doesn't store the packs'\nmtimes. That's fine for writing, since we can just look that information\nup ourselves. But the reading side can get broken. That's because the\nreader also has to know the pack order to go from MIDX- to bit-position.\n\nSo if a third party goes and touches some of the packs after the .rev\nfile was written, then the reader is going to think the packs ought to\nappear in a different order than they actually do. So relying on having\nto look up the mtimes again later on isn't good enough.\n\nThere are two solutions to the problem:\n\n - You could write the mtimes in the MIDX itself. This would give you a\n single point of reference, and resolve the TOCTOU race I just\n described.\n\n - Or, you could forget about mtimes entirely and let the MIDX dictate\n the pack ordering itself. That resolves the race in a\n similar-but-different way.\n\nOf the two, I prefer the latter, but I think it introduces functionality\nthat we don't necessarily need yet. That's because the objects within\nthe packs are still ordered as such, and so the compression we get in\nthe packs is just as good as it is for single-pack bitmaps. It's only at\nthe objects between pack boundaries that any runs of 1s or 0s might be\ninterrupted, but there are far fewer pack boundaries than objects, so it\ndoesn't seem to matter in practice.\n\nAnyway, I think that you know all of that already (mostly because we\nthought aloud together when I originally brought this up), but I figure\nthat this detail may be interesting for other readers, too.\n\n> > @@ -74,7 +85,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n> > \t\tusage_with_options(builtin_multi_pack_index_write_usage,\n> > \t\t\t\t options);\n> >\n> > -\treturn write_midx_file(opts.object_dir, opts.flags);\n> > +\treturn write_midx_file(opts.object_dir, opts.preferred_pack,\n> > +\t\t\t opts.flags);\n> > }\n>\n> This has the same leak of \"options\" that I mentioned in the earlier\n> patch.\n\nYup, thanks for pointing it out.\n\n> > diff --git a/midx.c b/midx.c\n> > index 971faa8cfc..46f55ff6cf 100644\n> > --- a/midx.c\n> > +++ b/midx.c\n> > @@ -431,6 +431,24 @@ static int pack_info_compare(const void *_a, const void *_b)\n> > \treturn strcmp(a->pack_name, b->pack_name);\n> > }\n> >\n> > +static int lookup_idx_or_pack_name(struct pack_info *info,\n> > +\t\t\t\t uint32_t nr,\n> > +\t\t\t\t const char *pack_name)\n> > +{\n> > +\tuint32_t lo = 0, hi = nr;\n> > +\twhile (lo < hi) {\n> > +\t\tuint32_t mi = lo + (hi - lo) / 2;\n> > +\t\tint cmp = cmp_idx_or_pack_name(pack_name, info[mi].pack_name);\n> > +\t\tif (cmp < 0)\n> > +\t\t\thi = mi;\n> > +\t\telse if (cmp > 0)\n> > +\t\t\tlo = mi + 1;\n> > +\t\telse\n> > +\t\t\treturn mi;\n> > +\t}\n> > +\treturn -1;\n> > +}\n>\n> Could this just be replaced with bsearch() in the caller?\n\nGreat suggestion. Yes, it can be. FWIW, I think that I may have\noriginally thought that it couldn't be since we were comparing a fixed\nstring to an array of structs (each having a field which holds the value\nwe actually want to compare). But bsearch() always passes the key as the\nfirst argument to the comparator, so this is possible to do.\n\n> > +\t\tgit multi-pack-index --object-dir=objects \\\n> > +\t\t\twrite --preferred-pack=test-BC-$bc.idx 2>err &&\n> > +\t\ttest_must_be_empty err &&\n> > +\n> > +\t\tofs=$(git show-index > +\t\t\tcut -d\" \" -f1) &&\n> > +\t\tmidx_expect_object_offset $b $ofs objects\n> > +\t)\n>\n> ...what we really care about is that the object came from BC. And we are\n> just using the offset as a proxy for that. But doesn't \"test-tool\n> read-midx\" give us the actual pack name? We could just be checking that.\n>\n> I also wondered if we should confirm that without the --preferred-pack\n> option, we choose the other pack. I think it will always be true because\n> the default order is to sort them lexically. A comment to that effect\n> might be worth it (near the \"set up two packs\" comment).\n\nBoth great points, thanks.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"420515","messageId":"YGJFDBds3imLWike@nand.local","threadId":"55135","inReplyTo":"YGHEN6rcao2tJ006@coredump.intra.peff.net","subject":"Re: [PATCH v3 12/16] Documentation/technical: describe multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-29T21:22:20Z","receivedAt":"2021-03-29T21:23:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 29, 2021 at 08:12:39AM -0400, Jeff King wrote:\n> On Thu, Mar 11, 2021 at 12:05:25PM -0500, Taylor Blau wrote:\n>\n> > As a prerequisite to implementing multi-pack bitmaps, motivate and\n> > describe the format and ordering of the multi-pack reverse index.\n>\n> Nicely written overall. I found a few typos / formatting issues.\n\nThanks for the attention to detail. Everything you wrote makes sense to\nme (including a quite-embarrassing mistake to switch \"their\" with\n\"there\").\n\nThanks,\nTaylor\n"},{"id":"420517","messageId":"YGJGXJ7Md0tWtnME@nand.local","threadId":"55135","inReplyTo":"YGHLe0nmh7R8hsVo@coredump.intra.peff.net","subject":"Re: [PATCH v3 13/16] pack-revindex: read multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-29T21:27:56Z","receivedAt":"2021-03-29T21:28:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 29, 2021 at 08:43:39AM -0400, Jeff King wrote:\n> On Thu, Mar 11, 2021 at 12:05:29PM -0500, Taylor Blau wrote:\n>\n> > Implement reading for multi-pack reverse indexes, as described in the\n> > previous patch.\n>\n> Looks good overall. I found a few tiny nits below.\n>\n> > +int load_midx_revindex(struct multi_pack_index *m)\n> > +{\n> > +\tchar *revindex_name;\n> > +\tint ret;\n> > +\tif (m->revindex_data)\n> > +\t\treturn 0;\n> > +\n> > +\trevindex_name = get_midx_rev_filename(m);\n> > +\n> > +\tret = load_revindex_from_disk(revindex_name,\n> > +\t\t\t\t m->num_objects,\n> > +\t\t\t\t &m->revindex_map,\n> > +\t\t\t\t &m->revindex_len);\n> > +\tif (ret)\n> > +\t\tgoto cleanup;\n>\n> On error, I wondered if m->revindex_map, etc, would be modified. But it\n> looks like no, load_revindex_from_disk() is careful not to touch them\n> unless it sees a valid revindex. Good.\n\nYep, that was intentional. Thanks.\n\n> > +int close_midx_revindex(struct multi_pack_index *m)\n> > +{\n> > +\tif (!m)\n> > +\t\treturn 0;\n> > +\n> > +\tif (munmap((void*)m->revindex_map, m->revindex_len))\n> > +\t\treturn -1;\n> > +\n> > +\tm->revindex_map = NULL;\n> > +\tm->revindex_data = NULL;\n> > +\tm->revindex_len = 0;\n> > +\n> > +\treturn 0;\n> > +}\n>\n> It's hard to imagine why munmap() would fail. But if it does, we should\n> probably clear the struct fields anyway. I note that the matching code\n> for a \"struct packed_git\" does not bother even checking the return value\n> of munmap. Perhaps we should just do the same here.\n\nI tend to agree that we should match the behavior of\n\"packfile.c:close_pack_revindex()\" and just not check the return value\nof munmap. Either the call to munmap() worked, and we shouldn't be\nreading revindex_map anymore, or it didn't, and something else is\nprobably wrong enough with the original mmap call that we probably also\nshouldn't be reading it.\n\n> The packed_git version also returned early if revindex_map is NULL. Here\n> the burden is placed on the caller (it's hard to tell if that matters\n> since there aren't any callers yet, but it probably makes sense to push\n> the check down into this function).\n\nYeah, I think that that function actually is doing the worst of both\nworlds (which is to check p->revindex_map, but not p itself).\n\nI modified the MIDX version to check both m and m->revindex_map (but I\nagree it's hard to tell with the caller coming in a later series).\n\n>\n> > +uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos)\n> > +{\n> > +\tif (!m->revindex_data)\n> > +\t\tBUG(\"pack_pos_to_midx: reverse index not yet loaded\");\n> > +\tif (m->num_objects <= pos)\n> > +\t\tBUG(\"pack_pos_to_midx: out-of-bounds object at %\"PRIu32, pos);\n> > +\treturn get_be32((const char *)m->revindex_data + (pos * sizeof(uint32_t)));\n> > +}\n>\n> OK, this one is just a direct read of the .rev data, like\n> pack_pos_to_index() is. I think the final line can be simplified to:\n>\n> return get_be32(m->revindex_data + pos);\n>\n> just like pack_pos_to_index(). (I suspect this is a leftover from the\n> earlier version of your .rev series where the pointer was still a \"void\n> *\").\n\nYes, definitely.\n\n> Probably sizeof(*m->revindex_data) would be slightly nicer in the\n> bsearch call (again, I suspect a holdover from when that was a void\n> pointer).\n\nYes, exactly.\n\nThanks,\nTaylor\n"},{"id":"420519","messageId":"xmqqeefxir6i.fsf@gitster.g","threadId":"55135","inReplyTo":"YGHQnQ9/ulXd+jgu@coredump.intra.peff.net","subject":"Re: [PATCH v3 00/16] midx: implement a multi-pack reverse index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-03-29T21:30:13Z","receivedAt":"2021-03-29T21:31:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Thu, Mar 11, 2021 at 12:04:31PM -0500, Taylor Blau wrote:\n>\n>> Here is another reroll of my series to implement a reverse index in\n>> preparation for multi-pack reachability bitmaps. The previous version\n>> was based on 'ds/chunked-file-api', but that topic has since been merged\n>> to 'master'. This series is now built directly on top of 'master'.\n>\n> I gave the whole thing another careful read. Most of what I found were\n> small nits, but enough that I think one more re-roll is worth it.\n\nThanks.\n\n> The biggest question is what we want to happen next. As you note, the\n> concept of a midx .rev file is useless until we have the matching\n> .bitmap file. So we _could_ let this sit in next while the dependent\n> bitmap topic is reviewed, and then merge them down together. But I'm\n> inclined to treat this as an independent topic that can get merged to\n> master on its own, since the early cleanups are valuable on their own,\n> and the .rev parts at the end, even if dead, won't hurt anything.\n\nIt was my impression as well that the early clean-ups are worth on\ntheir own.\n"},{"id":"420520","messageId":"YGJG9sALir6FuW44@nand.local","threadId":"55135","inReplyTo":"YGHNwin9zcrP00dI@coredump.intra.peff.net","subject":"Re: [PATCH v3 15/16] pack-revindex: write multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-29T21:30:30Z","receivedAt":"2021-03-29T21:31:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 29, 2021 at 08:53:22AM -0400, Jeff King wrote:\n> On Thu, Mar 11, 2021 at 12:05:38PM -0500, Taylor Blau wrote:\n>\n> > Implement the writing half of multi-pack reverse indexes. This is\n> > nothing more than the format describe a few patches ago, with a new set\n> > of helper functions that will be used to clear out stale .rev files\n> > corresponding to old MIDXs.\n>\n> Looks good.\n>\n> > +struct clear_midx_data {\n> > +\tchar *keep;\n> > +\tconst char *ext;\n> > +};\n> > +\n> > +static void clear_midx_file_ext(const char *full_path, size_t full_path_len,\n> > +\t\t\t\tconst char *file_name, void *_data)\n>\n> This will clean up _any_ stale midx .rev file. So even if we miss one\n> when writing a new midx (due to a bug, race, power loss, etc), we'll\n> catch it later.\n>\n> We _might_ want to also teach various tempfile-cleanup code run by gc to\n> likewise look for unattached midx .rev files, but I don't think we\n> necessarily have to do it now.\n\nAgreed there on both counts.\n\n> > void clear_midx_file(struct repository *r)\n> > {\n> > \tchar *midx = get_midx_filename(r->objects->odb->path);\n> > @@ -1049,6 +1162,8 @@ void clear_midx_file(struct repository *r)\n> > \tif (remove_path(midx))\n> > \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n> >\n> > +\tclear_midx_files_ext(r, \".rev\", NULL);\n> > +\n> > \tfree(midx);\n>\n> The sole caller now doesn't pass the \"keep\" hash, so we'd always delete\n> all of them. I guess we'll see that change once somebody starts actually\n> writing them.\n\nThat's right. I hope that the benefits of splitting the MIDX bitmaps\ntopic into two series has generally outweighed the drawbacks, but in\ninstances like these it can be kind of annoying.\n\nThanks,\nTaylor\n"},{"id":"420521","messageId":"YGJH3fzQXL6TvPK2@nand.local","threadId":"55135","inReplyTo":"YGHPINrwIZ1mKK+y@coredump.intra.peff.net","subject":"Re: [PATCH v3 16/16] midx.c: improve cache locality in midx_pack_order_cmp()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-29T21:34:21Z","receivedAt":"2021-03-29T21:35:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 29, 2021 at 08:59:12AM -0400, Jeff King wrote:\n> On Thu, Mar 11, 2021 at 12:05:42PM -0500, Taylor Blau wrote:\n>\n> > From: Jeff King \n> >\n> > There is a lot of pointer dereferencing in the pre-image version of\n> > 'midx_pack_order_cmp()', which this patch gets rid of.\n> >\n> > Instead of comparing the pack preferred-ness and then the pack id, both\n> > of these checks are done at the same time by using the high-order bit of\n> > the pack id to represent whether it's preferred. Then the pack id and\n> > offset are compared as usual.\n> >\n> > This produces the same result so long as there are less than 2^31 packs,\n> > which seems like a likely assumption to make in practice.\n>\n> Obviously this patch is brilliant. ;)\n\nObviously.\n\n> Did we record any numbers to show the improvement here? I don't think it\n> can be demonstrated with this series (since most of the code is dead),\n> but I recall that this was motivated by a noticeable slowdown.\n\nLooking through our messages, you wrote that this seemed to produce a\n.8 second speed-up on a large-ish repository that we were testing.\nThat's not significant overall, the fact that we were spending so long\nprobably caught our attention when looking at a profiler.\n\nI could go either way on mentioning it. It does feel a little like\ncheating to say, \"well, if you applied these other patches it would make\nit about this much faster\". So I'm mostly happy to just keep it vague\nand say that it makes things a little faster, unless you feel strongly\notherwise.\n\n> I briefly wondered whether the complicated midx_pack_order_cmp() in\n> pack-revindex.c, which is used for the bsearch() there, could benefit\n> from the same speedup. It's only log(n), of course, instead of n*log(n),\n> but one might imagine making \"n\" calls to it. I don't think it makes\n> sense, though. The pointer dereferencing there is into the midx mmap\n> itself. Creating an auxiliary array would defeat the purpose.\n\nRight.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"420523","messageId":"YGJIfdyghSUrq/0I@nand.local","threadId":"55135","inReplyTo":"YGHQnQ9/ulXd+jgu@coredump.intra.peff.net","subject":"Re: [PATCH v3 00/16] midx: implement a multi-pack reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-29T21:37:01Z","receivedAt":"2021-03-29T21:37:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 29, 2021 at 09:05:33AM -0400, Jeff King wrote:\n> On Thu, Mar 11, 2021 at 12:04:31PM -0500, Taylor Blau wrote:\n>\n> > Here is another reroll of my series to implement a reverse index in\n> > preparation for multi-pack reachability bitmaps. The previous version\n> > was based on 'ds/chunked-file-api', but that topic has since been merged\n> > to 'master'. This series is now built directly on top of 'master'.\n>\n> I gave the whole thing another careful read. Most of what I found were\n> small nits, but enough that I think one more re-roll is worth it.\n\nThanks. I agree that another re-roll is worth it. I have one prepared\nlocally, and I just had one outstanding question in:\n\n https://lore.kernel.org/git/YGI6ySogGoYZi66A@nand.local/\n\nthat I'll wait on your reply to before sending a reroll.\n\n> The biggest question is what we want to happen next. As you note, the\n> concept of a midx .rev file is useless until we have the matching\n> .bitmap file. So we _could_ let this sit in next while the dependent\n> bitmap topic is reviewed, and then merge them down together. But I'm\n> inclined to treat this as an independent topic that can get merged to\n> master on its own, since the early cleanups are valuable on their own,\n> and the .rev parts at the end, even if dead, won't hurt anything.\n\nThat matches what I was hoping for. I think the clean-ups are worth it\non their own, but I also think it's a good idea to take the whole\nseries, since it means there's one less long-running branch in flight\nwhile we review the MIDX bitmaps topic.\n\n(FWIW, I can also see an argument in the other direction along the lines\nof \"we may discover something later on that requires us to change the\nway multi-pack .rev files work\". I think that such an outcome is fairly\nunlikely, but worth considering anyway).\n\nThanks,\nTaylor\n"},{"id":"420559","messageId":"YGLNgBcWgCOYTxMv@coredump.intra.peff.net","threadId":"55135","inReplyTo":"YGI6ySogGoYZi66A@nand.local","subject":"Re: [PATCH v3 04/16] builtin/multi-pack-index.c: split sub-commands","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-30T07:04:32Z","receivedAt":"2021-03-30T07:05:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 29, 2021 at 04:38:33PM -0400, Taylor Blau wrote:\n\n> On Mon, Mar 29, 2021 at 07:36:21AM -0400, Jeff King wrote:\n> > This is definitely a harmless leak in the sense that we are going to\n> > exit the program after midx_repack() returns anyway. But it might be\n> > worth keeping things tidy, as we've recently seen a renewed effort to do\n> > some leak-checking of the test suite. I _think_ we can just free the\n> > options struct (even though we are still using the values themselves, we\n> > don't care about the \"struct options\" anymore). But even if not, an\n> > UNLEAK(options) annotation would do it.\n> \n> I see what you're saying. Let me make sure that I got the right idea in\n> mind after reading your email. I'm thinking of squashing the following\n> diff into this patch. For what it's worth, it causes 'valgrind\n> --leak-check=full ./git-multi-pack-index repack' to exit cleanly (when\n> it didn't before).\n> \n> Does this match your expectations?\n\nYes, though...\n\n> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\n> index 23e51dfeb4..a78640c061 100644\n> --- a/builtin/multi-pack-index.c\n> +++ b/builtin/multi-pack-index.c\n> @@ -56,9 +56,7 @@ static struct option common_opts[] = {\n> \n> static struct option *add_common_options(struct option *prev)\n> {\n> -\tstruct option *with_common = parse_options_concat(common_opts, prev);\n> -\tfree(prev);\n> -\treturn with_common;\n> +\treturn parse_options_concat(common_opts, prev);\n> }\n\nThis simplification is orthogonal to the leak, and I'd be OK if you\nwanted to retain it as it was before (because it future-proofs against\nadding more add_foo_options() later, though for now it is a useless\ndup/free pair).\n\n> @@ -123,6 +120,8 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv)\n> \t\tusage_with_options(builtin_multi_pack_index_repack_usage,\n> \t\t\t\t options);\n> \n> +\tFREE_AND_NULL(options);\n> +\n\nAnd this is the leak fix I care about. We'd want the same thing in the\nlater caller that adds another use of add_common_options(), of course.\n\n-Peff\n"},{"id":"420560","messageId":"YGLPND21b0o8JIfw@coredump.intra.peff.net","threadId":"55135","inReplyTo":"YGJDYIxicaPDMdeZ@nand.local","subject":"Re: [PATCH v3 08/16] midx: allow marking a pack as preferred","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-30T07:11:48Z","receivedAt":"2021-03-30T07:12:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 29, 2021 at 05:15:12PM -0400, Taylor Blau wrote:\n\n> There are two solutions to the problem:\n> \n> - You could write the mtimes in the MIDX itself. This would give you a\n> single point of reference, and resolve the TOCTOU race I just\n> described.\n> \n> - Or, you could forget about mtimes entirely and let the MIDX dictate\n> the pack ordering itself. That resolves the race in a\n> similar-but-different way.\n> \n> Of the two, I prefer the latter, but I think it introduces functionality\n> that we don't necessarily need yet.\n\nYeah, I'd strongly favor the latter over the former. The reason to go\nwith the solution you have in this series is that it doesn't require\nchanging anything in the on-disk midx format, and we think it is good\nenough. But once we are going to change the on-disk format, we might as\nwell give the writing side as much flexibility as possible.\n\nOf course the mtimes themselves are really just numbers, so in a sense\nthe two are really equivalent. ;)\n\n> That's because the objects within\n> the packs are still ordered as such, and so the compression we get in\n> the packs is just as good as it is for single-pack bitmaps. It's only at\n> the objects between pack boundaries that any runs of 1s or 0s might be\n> interrupted, but there are far fewer pack boundaries than objects, so it\n> doesn't seem to matter in practice.\n\nRight. The absolute worst case is a large number of single-object packs,\nin which case the bitmap order becomes essentially random with respect\nto history (because it would be sorted by sha1 of the packs).\n\nThe effect _might_ be measurable in more real-world cases, like say one\nbig pack and 100 pushes each with a handful of commits. The big pack\nwould be in good shape, but you have a lot of extra pack boundaries that\nhurt the bitmap compression.\n\nBut in practice, generating bitmaps is expensive enough that you'd\nprobably want to roll up some of the packs anyway (and that is certainly\nwhat we are doing at GitHub, using your \"repack --geometric\"). So you'd\nend usually with one big pack representing most of history, and then a\nhandful of roll-up packs.\n\nSo I'm a little curious whether one could even measure the impact of,\nsay, 100 little packs. But not enough to even run the experiment,\nbecause even that is not a case that is really that interesting.\n\n> Anyway, I think that you know all of that already (mostly because we\n> thought aloud together when I originally brought this up), but I figure\n> that this detail may be interesting for other readers, too.\n\nIndeed. And I know that you know everything I just wrote, but I agree\nit's nice to get a record of these discussions onto the list. :)\n\n-Peff\n"},{"id":"420561","messageId":"YGLP9oE4HtgLudg4@coredump.intra.peff.net","threadId":"55135","inReplyTo":"YGJIfdyghSUrq/0I@nand.local","subject":"Re: [PATCH v3 00/16] midx: implement a multi-pack reverse index","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-30T07:15:02Z","receivedAt":"2021-03-30T07:15:44Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 29, 2021 at 05:37:01PM -0400, Taylor Blau wrote:\n\n> > The biggest question is what we want to happen next. As you note, the\n> > concept of a midx .rev file is useless until we have the matching\n> > .bitmap file. So we _could_ let this sit in next while the dependent\n> > bitmap topic is reviewed, and then merge them down together. But I'm\n> > inclined to treat this as an independent topic that can get merged to\n> > master on its own, since the early cleanups are valuable on their own,\n> > and the .rev parts at the end, even if dead, won't hurt anything.\n> \n> That matches what I was hoping for. I think the clean-ups are worth it\n> on their own, but I also think it's a good idea to take the whole\n> series, since it means there's one less long-running branch in flight\n> while we review the MIDX bitmaps topic.\n> \n> (FWIW, I can also see an argument in the other direction along the lines\n> of \"we may discover something later on that requires us to change the\n> way multi-pack .rev files work\". I think that such an outcome is fairly\n> unlikely, but worth considering anyway).\n\nThat would be my general worry, too, but in this case I am not too\nconcerned because I know the code has received substantial exercise\nalready on real-world production servers. So while we may clean up some\ncosmetic bits or respond to review as it goes upstream, I'm much less\nworried about seeing some brown-paper-bag bug that would be sufficient\nto make us want to re-roll these .rev commits. And hopefully the\nexisting rounds have addressed the cosmetic/review bits.\n\n-Peff\n"},{"id":"420562","messageId":"YGLQLlllszQeV7zs@coredump.intra.peff.net","threadId":"55135","inReplyTo":"YGJH3fzQXL6TvPK2@nand.local","subject":"Re: [PATCH v3 16/16] midx.c: improve cache locality in midx_pack_order_cmp()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-30T07:15:58Z","receivedAt":"2021-03-30T07:16:50Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 29, 2021 at 05:34:21PM -0400, Taylor Blau wrote:\n\n> > Did we record any numbers to show the improvement here? I don't think it\n> > can be demonstrated with this series (since most of the code is dead),\n> > but I recall that this was motivated by a noticeable slowdown.\n> \n> Looking through our messages, you wrote that this seemed to produce a\n> .8 second speed-up on a large-ish repository that we were testing.\n> That's not significant overall, the fact that we were spending so long\n> probably caught our attention when looking at a profiler.\n\nThat sounds about right from my recollection.\n\n> I could go either way on mentioning it. It does feel a little like\n> cheating to say, \"well, if you applied these other patches it would make\n> it about this much faster\". So I'm mostly happy to just keep it vague\n> and say that it makes things a little faster, unless you feel strongly\n> otherwise.\n\nNo, I don't feel strongly. I just wanted to give people reading a sense\nof what to expect. Now we have.\n\n-Peff\n"},{"id":"420599","messageId":"YGMphbVWLp6X6TYS@nand.local","threadId":"55135","inReplyTo":"YGLP9oE4HtgLudg4@coredump.intra.peff.net","subject":"Re: [PATCH v3 00/16] midx: implement a multi-pack reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T13:37:09Z","receivedAt":"2021-03-30T13:38:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Mar 30, 2021 at 03:15:02AM -0400, Jeff King wrote:\n> On Mon, Mar 29, 2021 at 05:37:01PM -0400, Taylor Blau wrote:\n> > (FWIW, I can also see an argument in the other direction along the lines\n> > of \"we may discover something later on that requires us to change the\n> > way multi-pack .rev files work\". I think that such an outcome is fairly\n> > unlikely, but worth considering anyway).\n>\n> That would be my general worry, too, but in this case I am not too\n> concerned because I know the code has received substantial exercise\n> already on real-world production servers. So while we may clean up some\n> cosmetic bits or respond to review as it goes upstream, I'm much less\n> worried about seeing some brown-paper-bag bug that would be sufficient\n> to make us want to re-roll these .rev commits. And hopefully the\n> existing rounds have addressed the cosmetic/review bits.\n\nYes. Another benefit is that it should give us substantial confidence in\nthe correctness not just of this topic, but of the multi-pack bitmaps\nthat are built on top, too.\n\nThanks,\nTaylor\n"},{"id":"420603","messageId":"cover.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1612998106.git.me@ttaylorr.com","subject":"[PATCH v4 00/16] midx: implement a multi-pack reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:03:44Z","receivedAt":"2021-03-30T15:04:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is another reroll of my series to implement a reverse index in\npreparation for multi-pack reachability bitmaps.\n\nThis reroll differs only in the feedback I incorporated from Peff's review. They\nare mostly cosmetic; the most substantial change being that the --preferred-pack\ncode now uses bsearch() to locate the name of the preferred pack (instead of\nimplementing a binary search itself).\n\nI think that this version is ready to go. I would hope that it can head for\n'master' and avoid sitting in 'next' forever (since it has some worthwhile\ncleanups outside of preparing for MIDX bitmaps).\n\nBut in either case, this is the last prereq series before MIDX bitmaps, which\nI'll send shortly (based on this one).\n\nJeff King (1):\n midx.c: improve cache locality in midx_pack_order_cmp()\n\nTaylor Blau (15):\n builtin/multi-pack-index.c: inline 'flags' with options\n builtin/multi-pack-index.c: don't handle 'progress' separately\n builtin/multi-pack-index.c: define common usage with a macro\n builtin/multi-pack-index.c: split sub-commands\n builtin/multi-pack-index.c: don't enter bogus cmd_mode\n builtin/multi-pack-index.c: display usage on unrecognized command\n t/helper/test-read-midx.c: add '--show-objects'\n midx: allow marking a pack as preferred\n midx: don't free midx_name early\n midx: keep track of the checksum\n midx: make some functions non-static\n Documentation/technical: describe multi-pack reverse indexes\n pack-revindex: read multi-pack reverse indexes\n pack-write.c: extract 'write_rev_file_order'\n pack-revindex: write multi-pack reverse indexes\n\n Documentation/git-multi-pack-index.txt | 14 +-\n Documentation/technical/multi-pack-index.txt | 5 +-\n Documentation/technical/pack-format.txt | 83 +++++++\n builtin/multi-pack-index.c | 182 ++++++++++++---\n builtin/repack.c | 2 +-\n midx.c | 219 +++++++++++++++++--\n midx.h | 11 +-\n pack-revindex.c | 126 +++++++++++\n pack-revindex.h | 53 +++++\n pack-write.c | 36 ++-\n pack.h | 1 +\n packfile.c | 3 +\n t/helper/test-read-midx.c | 24 +-\n t/t5319-multi-pack-index.sh | 43 ++++\n 14 files changed, 734 insertions(+), 68 deletions(-)\n\nRange-diff against v3:\n 1: 43fc0ad276 ! 1: 90e021725f builtin/multi-pack-index.c: inline 'flags' with options\n @@ Commit message\n 'verify', etc.) will want to optionally change a set of shared flags\n that are eventually passed to the MIDX libraries.\n \n - Right now, options and flags are handled separately. Inline them into\n - the same structure so that sub-commands can more easily share the\n - 'flags' data.\n + Right now, options and flags are handled separately. That's fine, since\n + the options structure is never passed around. But a future patch will\n + make it so that common options shared by all sub-commands are defined in\n + a common location. That means that \"flags\" would have to become a global\n + variable.\n +\n + Group it with the options structure so that we reduce the number of\n + global variables we have overall.\n \n Signed-off-by: Taylor Blau \n \n 2: 181f11e4c5 = 2: 130c191b80 builtin/multi-pack-index.c: don't handle 'progress' separately\n 3: 94c498f0e2 = 3: 5a274b9096 builtin/multi-pack-index.c: define common usage with a macro\n 4: d084f90466 ! 4: b8c89cc239 builtin/multi-pack-index.c: split sub-commands\n @@ builtin/multi-pack-index.c: static struct opts_multi_pack_index {\n -\t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n -\t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n -\t\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"), MIDX_PROGRESS),\n -+\tstruct option *with_common = parse_options_concat(common_opts, prev);\n -+\tfree(prev);\n -+\treturn with_common;\n ++\treturn parse_options_concat(common_opts, prev);\n +}\n +\n +static int cmd_multi_pack_index_write(int argc, const char **argv)\n @@ builtin/multi-pack-index.c: static struct opts_multi_pack_index {\n \t\tOPT_END(),\n \t};\n \n -+\toptions = parse_options_dup(builtin_multi_pack_index_repack_options);\n -+\toptions = add_common_options(options);\n ++\toptions = add_common_options(builtin_multi_pack_index_repack_options);\n +\n +\targc = parse_options(argc, argv, NULL,\n +\t\t\t options,\n @@ builtin/multi-pack-index.c: static struct opts_multi_pack_index {\n +\t\tusage_with_options(builtin_multi_pack_index_repack_usage,\n +\t\t\t\t options);\n +\n ++\tFREE_AND_NULL(options);\n ++\n +\treturn midx_repack(the_repository, opts.object_dir,\n +\t\t\t (size_t)opts.batch_size, opts.flags);\n +}\n 5: bc3b6837f2 ! 5: d817920e2a builtin/multi-pack-index.c: don't enter bogus cmd_mode\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_expire(int argc, con\n \t\t\t options, builtin_multi_pack_index_expire_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_repack(int argc, const char **argv)\n - \toptions = parse_options_dup(builtin_multi_pack_index_repack_options);\n - \toptions = add_common_options(options);\n + \n + \toptions = add_common_options(builtin_multi_pack_index_repack_options);\n \n +\ttrace2_cmd_mode(argv[0]);\n +\n 6: f117e442c3 ! 6: 604a02ce85 builtin/multi-pack-index.c: display usage on unrecognized command\n @@ Commit message\n more helpful:\n \n $ git.compile multi-pack-index bogus\n + error: unrecognized subcommand: bogus\n usage: git multi-pack-index [] write\n or: git multi-pack-index [] verify\n or: git multi-pack-index [] expire\n @@ builtin/multi-pack-index.c: int cmd_multi_pack_index(int argc, const char **argv\n \tif (!strcmp(argv[0], \"repack\"))\n \t\treturn cmd_multi_pack_index_repack(argc, argv);\n @@ builtin/multi-pack-index.c: int cmd_multi_pack_index(int argc, const char **argv,\n + \t\treturn cmd_multi_pack_index_verify(argc, argv);\n \telse if (!strcmp(argv[0], \"expire\"))\n \t\treturn cmd_multi_pack_index_expire(argc, argv);\n - \telse\n +-\telse\n -\t\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n ++\telse {\n +usage:\n ++\t\terror(_(\"unrecognized subcommand: %s\"), argv[0]);\n +\t\tusage_with_options(builtin_multi_pack_index_usage,\n +\t\t\t\t builtin_multi_pack_index_options);\n ++\t}\n }\n 7: ae85a68ef2 = 7: 37e073ea27 t/helper/test-read-midx.c: add '--show-objects'\n 8: 30194a6786 ! 8: d061828e7e midx: allow marking a pack as preferred\n @@ builtin/multi-pack-index.c: static struct option *add_common_options(struct opti\n +\t\tOPT_END(),\n +\t};\n +\n -+\toptions = parse_options_dup(builtin_multi_pack_index_write_options);\n -+\toptions = add_common_options(options);\n ++\toptions = add_common_options(builtin_multi_pack_index_write_options);\n \n \ttrace2_cmd_mode(argv[0]);\n \n @@ builtin/multi-pack-index.c: static int cmd_multi_pack_index_write(int argc, cons\n \t\t\t\t options);\n \n -\treturn write_midx_file(opts.object_dir, opts.flags);\n ++\tFREE_AND_NULL(options);\n ++\n +\treturn write_midx_file(opts.object_dir, opts.preferred_pack,\n +\t\t\t opts.flags);\n }\n @@ midx.c: static int pack_info_compare(const void *_a, const void *_b)\n \treturn strcmp(a->pack_name, b->pack_name);\n }\n \n -+static int lookup_idx_or_pack_name(struct pack_info *info,\n -+\t\t\t\t uint32_t nr,\n -+\t\t\t\t const char *pack_name)\n ++static int idx_or_pack_name_cmp(const void *_va, const void *_vb)\n +{\n -+\tuint32_t lo = 0, hi = nr;\n -+\twhile (lo < hi) {\n -+\t\tuint32_t mi = lo + (hi - lo) / 2;\n -+\t\tint cmp = cmp_idx_or_pack_name(pack_name, info[mi].pack_name);\n -+\t\tif (cmp < 0)\n -+\t\t\thi = mi;\n -+\t\telse if (cmp > 0)\n -+\t\t\tlo = mi + 1;\n -+\t\telse\n -+\t\t\treturn mi;\n -+\t}\n -+\treturn -1;\n ++\tconst char *pack_name = _va;\n ++\tconst struct pack_info *compar = _vb;\n ++\n ++\treturn cmp_idx_or_pack_name(pack_name, compar->pack_name);\n +}\n +\n struct write_midx_context {\n @@ midx.c: static int write_midx_internal(const char *object_dir, struct multi_pack\n \n +\t/* Check that the preferred pack wasn't expired (if given). */\n +\tif (preferred_pack_name) {\n -+\t\tint preferred_idx = lookup_idx_or_pack_name(ctx.info,\n -+\t\t\t\t\t\t\t ctx.nr,\n -+\t\t\t\t\t\t\t preferred_pack_name);\n -+\t\tif (preferred_idx < 0)\n ++\t\tstruct pack_info *preferred = bsearch(preferred_pack_name,\n ++\t\t\t\t\t\t ctx.info, ctx.nr,\n ++\t\t\t\t\t\t sizeof(*ctx.info),\n ++\t\t\t\t\t\t idx_or_pack_name_cmp);\n ++\n ++\t\tif (!preferred)\n +\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n +\t\t\t\tpreferred_pack_name);\n +\t\telse {\n -+\t\t\tuint32_t orig = ctx.info[preferred_idx].orig_pack_int_id;\n -+\t\t\tuint32_t perm = ctx.pack_perm[orig];\n -+\n ++\t\t\tuint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];\n +\t\t\tif (perm == PACK_EXPIRED)\n +\t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n +\t\t\t\t\tpreferred_pack_name);\n @@ midx.h: int fill_midx_entry(struct repository *r, const struct object_id *oid, s\n int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);\n \n ## t/t5319-multi-pack-index.sh ##\n -@@ t/t5319-multi-pack-index.sh: midx_read_expect () {\n - \ttest_cmp expect actual\n - }\n - \n -+midx_expect_object_offset () {\n -+\tOID=\"$1\"\n -+\tOFFSET=\"$2\"\n -+\tOBJECT_DIR=\"$3\"\n -+\ttest-tool read-midx --show-objects $OBJECT_DIR >actual &&\n -+\tgrep \"^$OID $OFFSET\" actual\n -+}\n -+\n - test_expect_success 'setup' '\n - \ttest_oid_cache <<-EOF\n - \tidxoff sha1:2999\n @@ t/t5319-multi-pack-index.sh: test_expect_success 'warn on improper hash version' '\n \t)\n '\n @@ t/t5319-multi-pack-index.sh: test_expect_success 'warn on improper hash version'\n +\n +\t\t# Set up two packs, duplicating the object \"B\" at different\n +\t\t# offsets.\n ++\t\t#\n ++\t\t# Note that the \"BC\" pack (the one we choose as preferred) sorts\n ++\t\t# lexically after the \"AB\" pack, meaning that omitting the\n ++\t\t# --preferred-pack argument would cause this test to fail (since\n ++\t\t# the MIDX code would select the copy of \"b\" in the \"AB\" pack).\n +\t\tgit pack-objects objects/pack/test-AB <<-EOF &&\n +\t\t$a\n +\t\t$b\n @@ t/t5319-multi-pack-index.sh: test_expect_success 'warn on improper hash version'\n +\t\t\twrite --preferred-pack=test-BC-$bc.idx 2>err &&\n +\t\ttest_must_be_empty err &&\n +\n ++\t\techo hi &&\n ++\t\ttest-tool read-midx --show-objects objects >out &&\n ++\n +\t\tofs=$(git show-index expect &&\n ++\t\tgrep ^$b out >actual &&\n ++\n ++\t\ttest_cmp expect actual\n +\t)\n +'\n \n 9: 5c5aca761a = 9: 33b8af97e7 midx: don't free midx_name early\n10: a22a1463a5 = 10: 3fc9b83dc6 midx: keep track of the checksum\n11: efa54479b1 = 11: 2ada397320 midx: make some functions non-static\n12: 4745bb8590 ! 12: 8bb3dd24a7 Documentation/technical: describe multi-pack reverse indexes\n @@ Documentation/technical/pack-format.txt: CHUNK DATA:\n +position.\n +\n +One solution is to let bits occupy the same position in the oid-sorted\n -+index stored by the MIDX. But because oids are effectively random, there\n ++index stored by the MIDX. But because oids are effectively random, their\n +resulting reachability bitmaps would have no locality, and thus compress\n +poorly. (This is the reason that single-pack bitmaps use the pack\n +ordering, and not the .idx ordering, for the same purpose.)\n @@ Documentation/technical/pack-format.txt: CHUNK DATA:\n +order in the actual packfile.\n +\n +Given the list of packs and their counts of objects, you can\n -+naïvely reconstruct that pseudo-pack ordering (e.g., the object at\n ++naïvely reconstruct that pseudo-pack ordering (e.g., the object at\n +position 27 must be (c,1) because packs \"a\" and \"b\" consumed 25 of the\n +slots). But there's a catch. Objects may be duplicated between packs, in\n +which case the MIDX only stores one pointer to the object (and thus we'd\n @@ Documentation/technical/pack-format.txt: CHUNK DATA:\n +too.\n +\n +Objects from the MIDX are ordered as follows to string together the\n -+pseudo-pack. Let _pack(o)_ return the pack from which _o_ was selected\n ++pseudo-pack. Let `pack(o)` return the pack from which `o` was selected\n +by the MIDX, and define an ordering of packs based on their numeric ID\n -+(as stored by the MIDX). Let _offset(o)_ return the object offset of _o_\n -+within _pack(o)_. Then, compare _o~1~_ and _o~2~_ as follows:\n ++(as stored by the MIDX). Let `offset(o)` return the object offset of `o`\n ++within `pack(o)`. Then, compare `o1` and `o2` as follows:\n +\n -+ - If one of _pack(o~1~)_ and _pack(o~2~)_ is preferred and the other\n ++ - If one of `pack(o1)` and `pack(o2)` is preferred and the other\n + is not, then the preferred one sorts first.\n ++\n +(This is a detail that allows the MIDX bitmap to determine which\n +pack should be used by the pack-reuse mechanism, since it can ask\n +the MIDX for the pack containing the object at bit position 0).\n +\n -+ - If _pack(o~1~) ≠ pack(o~2~)_, then sort the two objects in\n -+ descending order based on the pack ID.\n ++ - If `pack(o1) ≠ pack(o2)`, then sort the two objects in descending\n ++ order based on the pack ID.\n +\n -+ - Otherwise, _pack(o~1~) = pack(o~2~)_, and the objects are\n -+ sorted in pack-order (i.e., _o~1~_ sorts ahead of _o~2~_ exactly\n -+ when _offset(o~1~) < offset(o~2~)_).\n ++ - Otherwise, `pack(o1) = pack(o2)`, and the objects are sorted in\n ++ pack-order (i.e., `o1` sorts ahead of `o2` exactly when `offset(o1)\n ++ < offset(o2)`).\n +\n +In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n +objects in packs stored by the MIDX, laid out in pack order, and the\n13: a6ebd4be91 ! 13: c070b9c99f pack-revindex: read multi-pack reverse indexes\n @@ pack-revindex.c: int load_pack_revindex(struct packed_git *p)\n +\n +int close_midx_revindex(struct multi_pack_index *m)\n +{\n -+\tif (!m)\n ++\tif (!m || !m->revindex_map)\n +\t\treturn 0;\n +\n -+\tif (munmap((void*)m->revindex_map, m->revindex_len))\n -+\t\treturn -1;\n ++\tmunmap((void*)m->revindex_map, m->revindex_len);\n +\n +\tm->revindex_map = NULL;\n +\tm->revindex_data = NULL;\n @@ pack-revindex.c: off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n +\t\tBUG(\"pack_pos_to_midx: reverse index not yet loaded\");\n +\tif (m->num_objects <= pos)\n +\t\tBUG(\"pack_pos_to_midx: out-of-bounds object at %\"PRIu32, pos);\n -+\treturn get_be32((const char *)m->revindex_data + (pos * sizeof(uint32_t)));\n ++\treturn get_be32(m->revindex_data + pos);\n +}\n +\n +struct midx_pack_key {\n @@ pack-revindex.c: off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n +\tkey.preferred_pack = nth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));\n +\n +\tfound = bsearch(&key, m->revindex_data, m->num_objects,\n -+\t\t\tsizeof(uint32_t), midx_pack_order_cmp);\n ++\t\t\tsizeof(*m->revindex_data), midx_pack_order_cmp);\n +\n +\tif (!found)\n +\t\treturn error(\"bad offset for revindex\");\n14: f5314f1822 = 14: 9f40019eb3 pack-write.c: extract 'write_rev_file_order'\n15: fa3acb5d5a = 15: 47409cc508 pack-revindex: write multi-pack reverse indexes\n16: 550e785f10 = 16: 7b793e7d09 midx.c: improve cache locality in midx_pack_order_cmp()\n-- \n2.30.0.667.g81c0cbc6fd\n"},{"id":"420604","messageId":"90e021725f3aa26355bb60c48da754af2e00f049.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"[PATCH v4 01/16] builtin/multi-pack-index.c: inline 'flags' with options","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:03:47Z","receivedAt":"2021-03-30T15:04:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Subcommands of the 'git multi-pack-index' command (e.g., 'write',\n'verify', etc.) will want to optionally change a set of shared flags\nthat are eventually passed to the MIDX libraries.\n\nRight now, options and flags are handled separately. That's fine, since\nthe options structure is never passed around. But a future patch will\nmake it so that common options shared by all sub-commands are defined in\na common location. That means that \"flags\" would have to become a global\nvariable.\n\nGroup it with the options structure so that we reduce the number of\nglobal variables we have overall.\n\nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 13 ++++++-------\n 1 file changed, 6 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 5bf88cd2a8..4a0ddb06c4 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -14,13 +14,12 @@ static struct opts_multi_pack_index {\n \tconst char *object_dir;\n \tunsigned long batch_size;\n \tint progress;\n+\tunsigned flags;\n } opts;\n \n int cmd_multi_pack_index(int argc, const char **argv,\n \t\t\t const char *prefix)\n {\n-\tunsigned flags = 0;\n-\n \tstatic struct option builtin_multi_pack_index_options[] = {\n \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n@@ -40,7 +39,7 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n \tif (opts.progress)\n-\t\tflags |= MIDX_PROGRESS;\n+\t\topts.flags |= MIDX_PROGRESS;\n \n \tif (argc == 0)\n \t\tusage_with_options(builtin_multi_pack_index_usage,\n@@ -55,16 +54,16 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \n \tif (!strcmp(argv[0], \"repack\"))\n \t\treturn midx_repack(the_repository, opts.object_dir,\n-\t\t\t(size_t)opts.batch_size, flags);\n+\t\t\t(size_t)opts.batch_size, opts.flags);\n \tif (opts.batch_size)\n \t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n \n \tif (!strcmp(argv[0], \"write\"))\n-\t\treturn write_midx_file(opts.object_dir, flags);\n+\t\treturn write_midx_file(opts.object_dir, opts.flags);\n \tif (!strcmp(argv[0], \"verify\"))\n-\t\treturn verify_midx_file(the_repository, opts.object_dir, flags);\n+\t\treturn verify_midx_file(the_repository, opts.object_dir, opts.flags);\n \tif (!strcmp(argv[0], \"expire\"))\n-\t\treturn expire_midx_packs(the_repository, opts.object_dir, flags);\n+\t\treturn expire_midx_packs(the_repository, opts.object_dir, opts.flags);\n \n \tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n }\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"420605","messageId":"130c191b805917bdd146c4122b8b8ceaae700580.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"[PATCH v4 02/16] builtin/multi-pack-index.c: don't handle 'progress' separately","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:03:51Z","receivedAt":"2021-03-30T15:04:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that there is a shared 'flags' member in the options structure,\nthere is no need to keep track of whether to force progress or not,\nsince ultimately the decision of whether or not to show a progress meter\nis controlled by a bit in the flags member.\n\nManipulate that bit directly, and drop the now-unnecessary 'progress'\nfield while we're at it.\n\nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 8 +++-----\n 1 file changed, 3 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 4a0ddb06c4..c70f020d8f 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -13,7 +13,6 @@ static char const * const builtin_multi_pack_index_usage[] = {\n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n \tunsigned long batch_size;\n-\tint progress;\n \tunsigned flags;\n } opts;\n \n@@ -23,7 +22,7 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \tstatic struct option builtin_multi_pack_index_options[] = {\n \t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n \t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n-\t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n+\t\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"), MIDX_PROGRESS),\n \t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n \t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n \t\tOPT_END(),\n@@ -31,15 +30,14 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \n \tgit_config(git_default_config, NULL);\n \n-\topts.progress = isatty(2);\n+\tif (isatty(2))\n+\t\topts.flags |= MIDX_PROGRESS;\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_multi_pack_index_options,\n \t\t\t builtin_multi_pack_index_usage, 0);\n \n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n-\tif (opts.progress)\n-\t\topts.flags |= MIDX_PROGRESS;\n \n \tif (argc == 0)\n \t\tusage_with_options(builtin_multi_pack_index_usage,\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"420606","messageId":"b8c89cc239f75bd4e0d81058801684ec8ea7a270.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"[PATCH v4 04/16] builtin/multi-pack-index.c: split sub-commands","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:03:57Z","receivedAt":"2021-03-30T15:04:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Handle sub-commands of the 'git multi-pack-index' builtin (e.g.,\n\"write\", \"repack\", etc.) separately from one another. This allows\nsub-commands with unique options, without forcing cmd_multi_pack_index()\nto reject invalid combinations itself.\n\nThis comes at the cost of some duplication and boilerplate. Luckily, the\nduplication is reduced to a minimum, since common options are shared\namong sub-commands due to a suggestion by Ævar. (Sub-commands do have to\nretain the common options, too, since this builtin accepts common\noptions on either side of the sub-command).\n\nRoughly speaking, cmd_multi_pack_index() parses options (including\ncommon ones), and stops at the first non-option, which is the\nsub-command. It then dispatches to the appropriate sub-command, which\nparses the remaining options (also including common options).\n\nUnknown options are kept by the sub-commands in order to detect their\npresence (and complain that too many arguments were given).\n\nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 130 ++++++++++++++++++++++++++++++-------\n 1 file changed, 105 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex eea498e026..a78640c061 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -17,6 +17,22 @@\n #define BUILTIN_MIDX_REPACK_USAGE \\\n \tN_(\"git multi-pack-index [] repack [--batch-size=]\")\n \n+static char const * const builtin_multi_pack_index_write_usage[] = {\n+\tBUILTIN_MIDX_WRITE_USAGE,\n+\tNULL\n+};\n+static char const * const builtin_multi_pack_index_verify_usage[] = {\n+\tBUILTIN_MIDX_VERIFY_USAGE,\n+\tNULL\n+};\n+static char const * const builtin_multi_pack_index_expire_usage[] = {\n+\tBUILTIN_MIDX_EXPIRE_USAGE,\n+\tNULL\n+};\n+static char const * const builtin_multi_pack_index_repack_usage[] = {\n+\tBUILTIN_MIDX_REPACK_USAGE,\n+\tNULL\n+};\n static char const * const builtin_multi_pack_index_usage[] = {\n \tBUILTIN_MIDX_WRITE_USAGE,\n \tBUILTIN_MIDX_VERIFY_USAGE,\n@@ -31,25 +47,98 @@ static struct opts_multi_pack_index {\n \tunsigned flags;\n } opts;\n \n-int cmd_multi_pack_index(int argc, const char **argv,\n-\t\t\t const char *prefix)\n+static struct option common_opts[] = {\n+\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n+\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n+\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"), MIDX_PROGRESS),\n+\tOPT_END(),\n+};\n+\n+static struct option *add_common_options(struct option *prev)\n {\n-\tstatic struct option builtin_multi_pack_index_options[] = {\n-\t\tOPT_FILENAME(0, \"object-dir\", &opts.object_dir,\n-\t\t N_(\"object directory containing set of packfile and pack-index pairs\")),\n-\t\tOPT_BIT(0, \"progress\", &opts.flags, N_(\"force progress reporting\"), MIDX_PROGRESS),\n+\treturn parse_options_concat(common_opts, prev);\n+}\n+\n+static int cmd_multi_pack_index_write(int argc, const char **argv)\n+{\n+\tstruct option *options = common_opts;\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options, builtin_multi_pack_index_write_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_multi_pack_index_write_usage,\n+\t\t\t\t options);\n+\n+\treturn write_midx_file(opts.object_dir, opts.flags);\n+}\n+\n+static int cmd_multi_pack_index_verify(int argc, const char **argv)\n+{\n+\tstruct option *options = common_opts;\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options, builtin_multi_pack_index_verify_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_multi_pack_index_verify_usage,\n+\t\t\t\t options);\n+\n+\treturn verify_midx_file(the_repository, opts.object_dir, opts.flags);\n+}\n+\n+static int cmd_multi_pack_index_expire(int argc, const char **argv)\n+{\n+\tstruct option *options = common_opts;\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options, builtin_multi_pack_index_expire_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_multi_pack_index_expire_usage,\n+\t\t\t\t options);\n+\n+\treturn expire_midx_packs(the_repository, opts.object_dir, opts.flags);\n+}\n+\n+static int cmd_multi_pack_index_repack(int argc, const char **argv)\n+{\n+\tstruct option *options;\n+\tstatic struct option builtin_multi_pack_index_repack_options[] = {\n \t\tOPT_MAGNITUDE(0, \"batch-size\", &opts.batch_size,\n \t\t N_(\"during repack, collect pack-files of smaller size into a batch that is larger than this size\")),\n \t\tOPT_END(),\n \t};\n \n+\toptions = add_common_options(builtin_multi_pack_index_repack_options);\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t options,\n+\t\t\t builtin_multi_pack_index_repack_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tif (argc)\n+\t\tusage_with_options(builtin_multi_pack_index_repack_usage,\n+\t\t\t\t options);\n+\n+\tFREE_AND_NULL(options);\n+\n+\treturn midx_repack(the_repository, opts.object_dir,\n+\t\t\t (size_t)opts.batch_size, opts.flags);\n+}\n+\n+int cmd_multi_pack_index(int argc, const char **argv,\n+\t\t\t const char *prefix)\n+{\n+\tstruct option *builtin_multi_pack_index_options = common_opts;\n+\n \tgit_config(git_default_config, NULL);\n \n \tif (isatty(2))\n \t\topts.flags |= MIDX_PROGRESS;\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_multi_pack_index_options,\n-\t\t\t builtin_multi_pack_index_usage, 0);\n+\t\t\t builtin_multi_pack_index_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tif (!opts.object_dir)\n \t\topts.object_dir = get_object_directory();\n@@ -58,25 +147,16 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\tusage_with_options(builtin_multi_pack_index_usage,\n \t\t\t\t builtin_multi_pack_index_options);\n \n-\tif (argc > 1) {\n-\t\tdie(_(\"too many arguments\"));\n-\t\treturn 1;\n-\t}\n-\n \ttrace2_cmd_mode(argv[0]);\n \n \tif (!strcmp(argv[0], \"repack\"))\n-\t\treturn midx_repack(the_repository, opts.object_dir,\n-\t\t\t(size_t)opts.batch_size, opts.flags);\n-\tif (opts.batch_size)\n-\t\tdie(_(\"--batch-size option is only for 'repack' subcommand\"));\n-\n-\tif (!strcmp(argv[0], \"write\"))\n-\t\treturn write_midx_file(opts.object_dir, opts.flags);\n-\tif (!strcmp(argv[0], \"verify\"))\n-\t\treturn verify_midx_file(the_repository, opts.object_dir, opts.flags);\n-\tif (!strcmp(argv[0], \"expire\"))\n-\t\treturn expire_midx_packs(the_repository, opts.object_dir, opts.flags);\n-\n-\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n+\t\treturn cmd_multi_pack_index_repack(argc, argv);\n+\telse if (!strcmp(argv[0], \"write\"))\n+\t\treturn cmd_multi_pack_index_write(argc, argv);\n+\telse if (!strcmp(argv[0], \"verify\"))\n+\t\treturn cmd_multi_pack_index_verify(argc, argv);\n+\telse if (!strcmp(argv[0], \"expire\"))\n+\t\treturn cmd_multi_pack_index_expire(argc, argv);\n+\telse\n+\t\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n }\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"420607","messageId":"5a274b909674401fa682b08ce08ad5b6acb47784.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"[PATCH v4 03/16] builtin/multi-pack-index.c: define common usage with a macro","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:03:54Z","receivedAt":"2021-03-30T15:04:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Factor out the usage message into pieces corresponding to each mode.\nThis avoids options specific to one sub-command from being shared with\nanother in the usage.\n\nA subsequent commit will use these #define macros to have usage\nvariables for each sub-command without duplicating their contents.\n\nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 17 ++++++++++++++++-\n 1 file changed, 16 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex c70f020d8f..eea498e026 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -5,8 +5,23 @@\n #include \"midx.h\"\n #include \"trace2.h\"\n \n+#define BUILTIN_MIDX_WRITE_USAGE \\\n+\tN_(\"git multi-pack-index [] write\")\n+\n+#define BUILTIN_MIDX_VERIFY_USAGE \\\n+\tN_(\"git multi-pack-index [] verify\")\n+\n+#define BUILTIN_MIDX_EXPIRE_USAGE \\\n+\tN_(\"git multi-pack-index [] expire\")\n+\n+#define BUILTIN_MIDX_REPACK_USAGE \\\n+\tN_(\"git multi-pack-index [] repack [--batch-size=]\")\n+\n static char const * const builtin_multi_pack_index_usage[] = {\n-\tN_(\"git multi-pack-index [] (write|verify|expire|repack --batch-size=)\"),\n+\tBUILTIN_MIDX_WRITE_USAGE,\n+\tBUILTIN_MIDX_VERIFY_USAGE,\n+\tBUILTIN_MIDX_EXPIRE_USAGE,\n+\tBUILTIN_MIDX_REPACK_USAGE,\n \tNULL\n };\n \n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"420608","messageId":"604a02ce8549be870dc7af3109289fa0a4b1e60f.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"[PATCH v4 06/16] builtin/multi-pack-index.c: display usage on unrecognized command","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:04:04Z","receivedAt":"2021-03-30T15:04:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When given a sub-command that it doesn't understand, 'git\nmulti-pack-index' dies with the following message:\n\n $ git multi-pack-index bogus\n fatal: unrecognized subcommand: bogus\n\nInstead of 'die()'-ing, we can display the usage text, which is much\nmore helpful:\n\n $ git.compile multi-pack-index bogus\n error: unrecognized subcommand: bogus\n usage: git multi-pack-index [] write\n or: git multi-pack-index [] verify\n or: git multi-pack-index [] expire\n or: git multi-pack-index [] repack [--batch-size=]\n\n --object-dir object directory containing set of packfile and pack-index pairs\n --progress force progress reporting\n\nWhile we're at it, clean up some duplication between the \"no sub-command\"\nand \"unrecognized sub-command\" conditionals.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 11 +++++++----\n 1 file changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex b590c4fc88..8711174fae 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -152,8 +152,7 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\topts.object_dir = get_object_directory();\n \n \tif (argc == 0)\n-\t\tusage_with_options(builtin_multi_pack_index_usage,\n-\t\t\t\t builtin_multi_pack_index_options);\n+\t\tgoto usage;\n \n \tif (!strcmp(argv[0], \"repack\"))\n \t\treturn cmd_multi_pack_index_repack(argc, argv);\n@@ -163,6 +162,10 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\treturn cmd_multi_pack_index_verify(argc, argv);\n \telse if (!strcmp(argv[0], \"expire\"))\n \t\treturn cmd_multi_pack_index_expire(argc, argv);\n-\telse\n-\t\tdie(_(\"unrecognized subcommand: %s\"), argv[0]);\n+\telse {\n+usage:\n+\t\terror(_(\"unrecognized subcommand: %s\"), argv[0]);\n+\t\tusage_with_options(builtin_multi_pack_index_usage,\n+\t\t\t\t builtin_multi_pack_index_options);\n+\t}\n }\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"420609","messageId":"d817920e2a66d6509d39e645a7c1975af090f05e.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"[PATCH v4 05/16] builtin/multi-pack-index.c: don't enter bogus cmd_mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:04:01Z","receivedAt":"2021-03-30T15:04:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Even before the recent refactoring, 'git multi-pack-index' calls\n'trace2_cmd_mode()' before verifying that the sub-command is recognized.\n\nPush this call down into the individual sub-commands so that we don't\nenter a bogus command mode.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/multi-pack-index.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex a78640c061..b590c4fc88 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -63,6 +63,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n {\n \tstruct option *options = common_opts;\n \n+\ttrace2_cmd_mode(argv[0]);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options, builtin_multi_pack_index_write_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n@@ -77,6 +79,8 @@ static int cmd_multi_pack_index_verify(int argc, const char **argv)\n {\n \tstruct option *options = common_opts;\n \n+\ttrace2_cmd_mode(argv[0]);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options, builtin_multi_pack_index_verify_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n@@ -91,6 +95,8 @@ static int cmd_multi_pack_index_expire(int argc, const char **argv)\n {\n \tstruct option *options = common_opts;\n \n+\ttrace2_cmd_mode(argv[0]);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options, builtin_multi_pack_index_expire_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n@@ -112,6 +118,8 @@ static int cmd_multi_pack_index_repack(int argc, const char **argv)\n \n \toptions = add_common_options(builtin_multi_pack_index_repack_options);\n \n+\ttrace2_cmd_mode(argv[0]);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options,\n \t\t\t builtin_multi_pack_index_repack_usage,\n@@ -147,8 +155,6 @@ int cmd_multi_pack_index(int argc, const char **argv,\n \t\tusage_with_options(builtin_multi_pack_index_usage,\n \t\t\t\t builtin_multi_pack_index_options);\n \n-\ttrace2_cmd_mode(argv[0]);\n-\n \tif (!strcmp(argv[0], \"repack\"))\n \t\treturn cmd_multi_pack_index_repack(argc, argv);\n \telse if (!strcmp(argv[0], \"write\"))\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"420610","messageId":"37e073ea277045f00cdb3dfa2827f8c8d67ed9cb.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"[PATCH v4 07/16] t/helper/test-read-midx.c: add '--show-objects'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:04:07Z","receivedAt":"2021-03-30T15:04:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The 'read-midx' helper is used in places like t5319 to display basic\ninformation about a multi-pack-index.\n\nIn the next patch, the MIDX writing machinery will learn a new way to\nchoose from which pack an object is selected when multiple copies of\nthat object exist.\n\nTo disambiguate which pack introduces an object so that this feature can\nbe tested, add a '--show-objects' option which displays additional\ninformation about each object in the MIDX.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-midx.c | 24 ++++++++++++++++++++----\n 1 file changed, 20 insertions(+), 4 deletions(-)\n\ndiff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c\nindex 2430880f78..7c2eb11a8e 100644\n--- a/t/helper/test-read-midx.c\n+++ b/t/helper/test-read-midx.c\n@@ -4,7 +4,7 @@\n #include \"repository.h\"\n #include \"object-store.h\"\n \n-static int read_midx_file(const char *object_dir)\n+static int read_midx_file(const char *object_dir, int show_objects)\n {\n \tuint32_t i;\n \tstruct multi_pack_index *m;\n@@ -43,13 +43,29 @@ static int read_midx_file(const char *object_dir)\n \n \tprintf(\"object-dir: %s\\n\", m->object_dir);\n \n+\tif (show_objects) {\n+\t\tstruct object_id oid;\n+\t\tstruct pack_entry e;\n+\n+\t\tfor (i = 0; i < m->num_objects; i++) {\n+\t\t\tnth_midxed_object_oid(&oid, m, i);\n+\t\t\tfill_midx_entry(the_repository, &oid, &e, m);\n+\n+\t\t\tprintf(\"%s %\"PRIu64\"\\t%s\\n\",\n+\t\t\t oid_to_hex(&oid), e.offset, e.p->pack_name);\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\n \treturn 0;\n }\n \n int cmd__read_midx(int argc, const char **argv)\n {\n-\tif (argc != 2)\n-\t\tusage(\"read-midx \");\n+\tif (!(argc == 2 || argc == 3))\n+\t\tusage(\"read-midx [--show-objects] \");\n \n-\treturn read_midx_file(argv[1]);\n+\tif (!strcmp(argv[1], \"--show-objects\"))\n+\t\treturn read_midx_file(argv[2], 1);\n+\treturn read_midx_file(argv[1], 0);\n }\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"420611","messageId":"d061828e7e52eb97c72dbc0853b02180feea0d8b.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"[PATCH v4 08/16] midx: allow marking a pack as preferred","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:04:11Z","receivedAt":"2021-03-30T15:04:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When multiple packs in the multi-pack index contain the same object, the\nMIDX machinery must make a choice about which pack it associates with\nthat object. Prior to this patch, the lowest-ordered[1] pack was always\nselected.\n\nPack selection for duplicate objects is relatively unimportant today,\nbut it will become important for multi-pack bitmaps. This is because we\ncan only invoke the pack-reuse mechanism when all of the bits for reused\nobjects come from the reuse pack (in order to ensure that all reused\ndeltas can find their base objects in the same pack).\n\nTo encourage the pack selection process to prefer one pack over another\n(the pack to be preferred is the one a caller would like to later use as\na reuse pack), introduce the concept of a \"preferred pack\". When\nprovided, the MIDX code will always prefer an object found in a\npreferred pack over any other.\n\nNo format changes are required to store the preferred pack, since it\nwill be able to be inferred with a corresponding MIDX bitmap, by looking\nup the pack associated with the object in the first bit position (this\nordering is described in detail in a subsequent commit).\n\n[1]: the ordering is specified by MIDX internals; for our purposes we\ncan consider the \"lowest ordered\" pack to be \"the one with the\nmost-recent mtime.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-multi-pack-index.txt | 14 +++-\n Documentation/technical/multi-pack-index.txt | 5 +-\n builtin/multi-pack-index.c | 19 ++++-\n builtin/repack.c | 2 +-\n midx.c | 82 +++++++++++++++++---\n midx.h | 2 +-\n t/t5319-multi-pack-index.sh | 43 ++++++++++\n 7 files changed, 149 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt\nindex eb0caa0439..ffd601bc17 100644\n--- a/Documentation/git-multi-pack-index.txt\n+++ b/Documentation/git-multi-pack-index.txt\n@@ -9,7 +9,8 @@ git-multi-pack-index - Write and verify multi-pack-indexes\n SYNOPSIS\n --------\n [verse]\n-'git multi-pack-index' [--object-dir=] [--[no-]progress] \n+'git multi-pack-index' [--object-dir=] [--[no-]progress]\n+\t[--preferred-pack=] \n \n DESCRIPTION\n -----------\n@@ -30,7 +31,16 @@ OPTIONS\n The following subcommands are available:\n \n write::\n-\tWrite a new MIDX file.\n+\tWrite a new MIDX file. The following options are available for\n+\tthe `write` sub-command:\n++\n+--\n+\t--preferred-pack=::\n+\t\tOptionally specify the tie-breaking pack used when\n+\t\tmultiple packs contain the same object. If not given,\n+\t\tties are broken in favor of the pack with the lowest\n+\t\tmtime.\n+--\n \n verify::\n \tVerify the contents of the MIDX file.\ndiff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt\nindex e8e377a59f..fb688976c4 100644\n--- a/Documentation/technical/multi-pack-index.txt\n+++ b/Documentation/technical/multi-pack-index.txt\n@@ -43,8 +43,9 @@ Design Details\n a change in format.\n \n - The MIDX keeps only one record per object ID. If an object appears\n- in multiple packfiles, then the MIDX selects the copy in the most-\n- recently modified packfile.\n+ in multiple packfiles, then the MIDX selects the copy in the\n+ preferred packfile, otherwise selecting from the most-recently\n+ modified packfile.\n \n - If there exist packfiles in the pack directory not registered in\n the MIDX, then those packfiles are loaded into the `packed_git`\ndiff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c\nindex 8711174fae..5d3ea445fd 100644\n--- a/builtin/multi-pack-index.c\n+++ b/builtin/multi-pack-index.c\n@@ -4,9 +4,10 @@\n #include \"parse-options.h\"\n #include \"midx.h\"\n #include \"trace2.h\"\n+#include \"object-store.h\"\n \n #define BUILTIN_MIDX_WRITE_USAGE \\\n-\tN_(\"git multi-pack-index [] write\")\n+\tN_(\"git multi-pack-index [] write [--preferred-pack=]\")\n \n #define BUILTIN_MIDX_VERIFY_USAGE \\\n \tN_(\"git multi-pack-index [] verify\")\n@@ -43,6 +44,7 @@ static char const * const builtin_multi_pack_index_usage[] = {\n \n static struct opts_multi_pack_index {\n \tconst char *object_dir;\n+\tconst char *preferred_pack;\n \tunsigned long batch_size;\n \tunsigned flags;\n } opts;\n@@ -61,7 +63,15 @@ static struct option *add_common_options(struct option *prev)\n \n static int cmd_multi_pack_index_write(int argc, const char **argv)\n {\n-\tstruct option *options = common_opts;\n+\tstruct option *options;\n+\tstatic struct option builtin_multi_pack_index_write_options[] = {\n+\t\tOPT_STRING(0, \"preferred-pack\", &opts.preferred_pack,\n+\t\t\t N_(\"preferred-pack\"),\n+\t\t\t N_(\"pack for reuse when computing a multi-pack bitmap\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\toptions = add_common_options(builtin_multi_pack_index_write_options);\n \n \ttrace2_cmd_mode(argv[0]);\n \n@@ -72,7 +82,10 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)\n \t\tusage_with_options(builtin_multi_pack_index_write_usage,\n \t\t\t\t options);\n \n-\treturn write_midx_file(opts.object_dir, opts.flags);\n+\tFREE_AND_NULL(options);\n+\n+\treturn write_midx_file(opts.object_dir, opts.preferred_pack,\n+\t\t\t opts.flags);\n }\n \n static int cmd_multi_pack_index_verify(int argc, const char **argv)\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 6ce2556c9e..2847fdfbab 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -721,7 +721,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tremove_temporary_files();\n \n \tif (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0))\n-\t\twrite_midx_file(get_object_directory(), 0);\n+\t\twrite_midx_file(get_object_directory(), NULL, 0);\n \n \tstring_list_clear(&names, 0);\n \tstring_list_clear(&rollback, 0);\ndiff --git a/midx.c b/midx.c\nindex becfafe65e..4a9d84352c 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -431,6 +431,14 @@ static int pack_info_compare(const void *_a, const void *_b)\n \treturn strcmp(a->pack_name, b->pack_name);\n }\n \n+static int idx_or_pack_name_cmp(const void *_va, const void *_vb)\n+{\n+\tconst char *pack_name = _va;\n+\tconst struct pack_info *compar = _vb;\n+\n+\treturn cmp_idx_or_pack_name(pack_name, compar->pack_name);\n+}\n+\n struct write_midx_context {\n \tstruct pack_info *info;\n \tuint32_t nr;\n@@ -445,6 +453,8 @@ struct write_midx_context {\n \tuint32_t *pack_perm;\n \tunsigned large_offsets_needed:1;\n \tuint32_t num_large_offsets;\n+\n+\tint preferred_pack_idx;\n };\n \n static void add_pack_to_midx(const char *full_path, size_t full_path_len,\n@@ -489,6 +499,7 @@ struct pack_midx_entry {\n \tuint32_t pack_int_id;\n \ttime_t pack_mtime;\n \tuint64_t offset;\n+\tunsigned preferred : 1;\n };\n \n static int midx_oid_compare(const void *_a, const void *_b)\n@@ -500,6 +511,12 @@ static int midx_oid_compare(const void *_a, const void *_b)\n \tif (cmp)\n \t\treturn cmp;\n \n+\t/* Sort objects in a preferred pack first when multiple copies exist. */\n+\tif (a->preferred > b->preferred)\n+\t\treturn -1;\n+\tif (a->preferred < b->preferred)\n+\t\treturn 1;\n+\n \tif (a->pack_mtime > b->pack_mtime)\n \t\treturn -1;\n \telse if (a->pack_mtime < b->pack_mtime)\n@@ -527,7 +544,8 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,\n static void fill_pack_entry(uint32_t pack_int_id,\n \t\t\t struct packed_git *p,\n \t\t\t uint32_t cur_object,\n-\t\t\t struct pack_midx_entry *entry)\n+\t\t\t struct pack_midx_entry *entry,\n+\t\t\t int preferred)\n {\n \tif (nth_packed_object_id(&entry->oid, p, cur_object) < 0)\n \t\tdie(_(\"failed to locate object %d in packfile\"), cur_object);\n@@ -536,6 +554,7 @@ static void fill_pack_entry(uint32_t pack_int_id,\n \tentry->pack_mtime = p->mtime;\n \n \tentry->offset = nth_packed_object_offset(p, cur_object);\n+\tentry->preferred = !!preferred;\n }\n \n /*\n@@ -552,7 +571,8 @@ static void fill_pack_entry(uint32_t pack_int_id,\n static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \t\t\t\t\t\t struct pack_info *info,\n \t\t\t\t\t\t uint32_t nr_packs,\n-\t\t\t\t\t\t uint32_t *nr_objects)\n+\t\t\t\t\t\t uint32_t *nr_objects,\n+\t\t\t\t\t\t int preferred_pack)\n {\n \tuint32_t cur_fanout, cur_pack, cur_object;\n \tuint32_t alloc_fanout, alloc_objects, total_objects = 0;\n@@ -589,12 +609,17 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \t\t\t\tnth_midxed_pack_midx_entry(m,\n \t\t\t\t\t\t\t &entries_by_fanout[nr_fanout],\n \t\t\t\t\t\t\t cur_object);\n+\t\t\t\tif (nth_midxed_pack_int_id(m, cur_object) == preferred_pack)\n+\t\t\t\t\tentries_by_fanout[nr_fanout].preferred = 1;\n+\t\t\t\telse\n+\t\t\t\t\tentries_by_fanout[nr_fanout].preferred = 0;\n \t\t\t\tnr_fanout++;\n \t\t\t}\n \t\t}\n \n \t\tfor (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) {\n \t\t\tuint32_t start = 0, end;\n+\t\t\tint preferred = cur_pack == preferred_pack;\n \n \t\t\tif (cur_fanout)\n \t\t\t\tstart = get_pack_fanout(info[cur_pack].p, cur_fanout - 1);\n@@ -602,7 +627,11 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,\n \n \t\t\tfor (cur_object = start; cur_object < end; cur_object++) {\n \t\t\t\tALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);\n-\t\t\t\tfill_pack_entry(cur_pack, info[cur_pack].p, cur_object, &entries_by_fanout[nr_fanout]);\n+\t\t\t\tfill_pack_entry(cur_pack,\n+\t\t\t\t\t\tinfo[cur_pack].p,\n+\t\t\t\t\t\tcur_object,\n+\t\t\t\t\t\t&entries_by_fanout[nr_fanout],\n+\t\t\t\t\t\tpreferred);\n \t\t\t\tnr_fanout++;\n \t\t\t}\n \t\t}\n@@ -777,7 +806,9 @@ static int write_midx_large_offsets(struct hashfile *f,\n }\n \n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n-\t\t\t struct string_list *packs_to_drop, unsigned flags)\n+\t\t\t struct string_list *packs_to_drop,\n+\t\t\t const char *preferred_pack_name,\n+\t\t\t unsigned flags)\n {\n \tchar *midx_name;\n \tuint32_t i;\n@@ -828,7 +859,19 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tif (ctx.m && ctx.nr == ctx.m->num_packs && !packs_to_drop)\n \t\tgoto cleanup;\n \n-\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr);\n+\tctx.preferred_pack_idx = -1;\n+\tif (preferred_pack_name) {\n+\t\tfor (i = 0; i < ctx.nr; i++) {\n+\t\t\tif (!cmp_idx_or_pack_name(preferred_pack_name,\n+\t\t\t\t\t\t ctx.info[i].pack_name)) {\n+\t\t\t\tctx.preferred_pack_idx = i;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr,\n+\t\t\t\t\t ctx.preferred_pack_idx);\n \n \tctx.large_offsets_needed = 0;\n \tfor (i = 0; i < ctx.entries_nr; i++) {\n@@ -889,6 +932,24 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\tpack_name_concat_len += strlen(ctx.info[i].pack_name) + 1;\n \t}\n \n+\t/* Check that the preferred pack wasn't expired (if given). */\n+\tif (preferred_pack_name) {\n+\t\tstruct pack_info *preferred = bsearch(preferred_pack_name,\n+\t\t\t\t\t\t ctx.info, ctx.nr,\n+\t\t\t\t\t\t sizeof(*ctx.info),\n+\t\t\t\t\t\t idx_or_pack_name_cmp);\n+\n+\t\tif (!preferred)\n+\t\t\twarning(_(\"unknown preferred pack: '%s'\"),\n+\t\t\t\tpreferred_pack_name);\n+\t\telse {\n+\t\t\tuint32_t perm = ctx.pack_perm[preferred->orig_pack_int_id];\n+\t\t\tif (perm == PACK_EXPIRED)\n+\t\t\t\twarning(_(\"preferred pack '%s' is expired\"),\n+\t\t\t\t\tpreferred_pack_name);\n+\t\t}\n+\t}\n+\n \tif (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)\n \t\tpack_name_concat_len += MIDX_CHUNK_ALIGNMENT -\n \t\t\t\t\t(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);\n@@ -947,9 +1008,12 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \treturn result;\n }\n \n-int write_midx_file(const char *object_dir, unsigned flags)\n+int write_midx_file(const char *object_dir,\n+\t\t const char *preferred_pack_name,\n+\t\t unsigned flags)\n {\n-\treturn write_midx_internal(object_dir, NULL, NULL, flags);\n+\treturn write_midx_internal(object_dir, NULL, NULL, preferred_pack_name,\n+\t\t\t\t flags);\n }\n \n void clear_midx_file(struct repository *r)\n@@ -1184,7 +1248,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tfree(count);\n \n \tif (packs_to_drop.nr)\n-\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop, flags);\n+\t\tresult = write_midx_internal(object_dir, m, &packs_to_drop, NULL, flags);\n \n \tstring_list_clear(&packs_to_drop, 0);\n \treturn result;\n@@ -1373,7 +1437,7 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,\n \t\tgoto cleanup;\n \t}\n \n-\tresult = write_midx_internal(object_dir, m, NULL, flags);\n+\tresult = write_midx_internal(object_dir, m, NULL, NULL, flags);\n \tm = NULL;\n \n cleanup:\ndiff --git a/midx.h b/midx.h\nindex b18cf53bc4..e7fea61109 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -47,7 +47,7 @@ int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pa\n int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name);\n int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);\n \n-int write_midx_file(const char *object_dir, unsigned flags);\n+int write_midx_file(const char *object_dir, const char *preferred_pack_name, unsigned flags);\n void clear_midx_file(struct repository *r);\n int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags);\n int expire_midx_packs(struct repository *r, const char *object_dir, unsigned flags);\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex b4afab1dfc..031a5570c0 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -234,6 +234,49 @@ test_expect_success 'warn on improper hash version' '\n \t)\n '\n \n+test_expect_success 'midx picks objects from preferred pack' '\n+\ttest_when_finished rm -rf preferred.git &&\n+\tgit init --bare preferred.git &&\n+\t(\n+\t\tcd preferred.git &&\n+\n+\t\ta=$(echo \"a\" | git hash-object -w --stdin) &&\n+\t\tb=$(echo \"b\" | git hash-object -w --stdin) &&\n+\t\tc=$(echo \"c\" | git hash-object -w --stdin) &&\n+\n+\t\t# Set up two packs, duplicating the object \"B\" at different\n+\t\t# offsets.\n+\t\t#\n+\t\t# Note that the \"BC\" pack (the one we choose as preferred) sorts\n+\t\t# lexically after the \"AB\" pack, meaning that omitting the\n+\t\t# --preferred-pack argument would cause this test to fail (since\n+\t\t# the MIDX code would select the copy of \"b\" in the \"AB\" pack).\n+\t\tgit pack-objects objects/pack/test-AB <<-EOF &&\n+\t\t$a\n+\t\t$b\n+\t\tEOF\n+\t\tbc=$(git pack-objects objects/pack/test-BC <<-EOF\n+\t\t$b\n+\t\t$c\n+\t\tEOF\n+\t\t) &&\n+\n+\t\tgit multi-pack-index --object-dir=objects \\\n+\t\t\twrite --preferred-pack=test-BC-$bc.idx 2>err &&\n+\t\ttest_must_be_empty err &&\n+\n+\t\techo hi &&\n+\t\ttest-tool read-midx --show-objects objects >out &&\n+\n+\t\tofs=$(git show-index expect &&\n+\t\tgrep ^$b out >actual &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n \n test_expect_success 'verify multi-pack-index success' '\n \tgit multi-pack-index verify --object-dir=$objdir\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"420612","messageId":"3fc9b83dc61837409b3d68f4676fc7a982f10593.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"[PATCH v4 10/16] midx: keep track of the checksum","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:04:17Z","receivedAt":"2021-03-30T15:05:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"write_midx_internal() uses a hashfile to write the multi-pack index, but\ndiscards its checksum. This makes sense, since nothing that takes place\nafter writing the MIDX cares about its checksum.\n\nThat is about to change in a subsequent patch, when the optional\nreverse index corresponding to the MIDX will want to include the MIDX's\nchecksum.\n\nStore the checksum of the MIDX in preparation for that.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 3edde2b68d..526795ff0e 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -811,6 +811,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t\t\t unsigned flags)\n {\n \tchar *midx_name;\n+\tunsigned char midx_hash[GIT_MAX_RAWSZ];\n \tuint32_t i;\n \tstruct hashfile *f = NULL;\n \tstruct lock_file lk;\n@@ -987,7 +988,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \twrite_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);\n \twrite_chunkfile(cf, &ctx);\n \n-\tfinalize_hashfile(f, NULL, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n+\tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n \tcommit_lock_file(&lk);\n \n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"420613","messageId":"33b8af97e7c6fb2d3016422d084eae7aca7cb70c.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"[PATCH v4 09/16] midx: don't free midx_name early","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:04:14Z","receivedAt":"2021-03-30T15:05:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A subsequent patch will need to refer back to 'midx_name' later on in\nthe function. In fact, this variable is already free()'d later on, so\nthis makes the later free() no longer redundant.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 4a9d84352c..3edde2b68d 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -956,7 +956,6 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \thold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));\n-\tFREE_AND_NULL(midx_name);\n \n \tif (ctx.m)\n \t\tclose_midx(ctx.m);\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"420614","messageId":"2ada3973202feb737af6c8cbeafae0241afdff90.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"[PATCH v4 11/16] midx: make some functions non-static","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:04:20Z","receivedAt":"2021-03-30T15:05:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a subsequent commit, pack-revindex.c will become responsible for\nsorting a list of objects in the \"MIDX pack order\" (which will be\ndefined in the following patch). To do so, it will need to be know the\npack identifier and offset within that pack for each object in the MIDX.\n\nThe MIDX code already has functions for doing just that\n(nth_midxed_offset() and nth_midxed_pack_int_id()), but they are\nstatically declared.\n\nSince there is no reason that they couldn't be exposed publicly, and\nbecause they are already doing exactly what the caller in\npack-revindex.c will want, expose them publicly so that they can be\nreused there.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 4 ++--\n midx.h | 2 ++\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 526795ff0e..2173a9b45c 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -239,7 +239,7 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \treturn oid;\n }\n \n-static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n+off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n {\n \tconst unsigned char *offset_data;\n \tuint32_t offset32;\n@@ -258,7 +258,7 @@ static off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)\n \treturn offset32;\n }\n \n-static uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n+uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)\n {\n \treturn get_be32(m->chunk_object_offsets +\n \t\t\t(off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);\ndiff --git a/midx.h b/midx.h\nindex e7fea61109..93bd68189e 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -40,6 +40,8 @@ struct multi_pack_index {\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n+off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);\n+uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);\n struct object_id *nth_midxed_object_oid(struct object_id *oid,\n \t\t\t\t\tstruct multi_pack_index *m,\n \t\t\t\t\tuint32_t n);\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"420615","messageId":"9f40019eb3d3315cb03440e6237bced4feb6cf67.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"[PATCH v4 14/16] pack-write.c: extract 'write_rev_file_order'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:04:29Z","receivedAt":"2021-03-30T15:05:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Existing callers provide the reverse index code with an array of 'struct\npack_idx_entry *'s, which is then sorted by pack order (comparing the\noffsets of each object within the pack).\n\nPrepare for the multi-pack index to write a .rev file by providing a way\nto write the reverse index without an array of pack_idx_entry (which the\nMIDX code does not have).\n\nInstead, callers can invoke 'write_rev_index_positions()', which takes\nan array of uint32_t's. The ith entry in this array specifies the ith\nobject's (in index order) position within the pack (in pack order).\n\nExpose this new function for use in a later patch, and rewrite the\nexisting write_rev_file() in terms of this new function.\n\nSigned-off-by: Taylor Blau \n---\n pack-write.c | 36 +++++++++++++++++++++++++-----------\n pack.h | 1 +\n 2 files changed, 26 insertions(+), 11 deletions(-)\n\ndiff --git a/pack-write.c b/pack-write.c\nindex 2ca85a9d16..f1fc3ecafa 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -201,21 +201,12 @@ static void write_rev_header(struct hashfile *f)\n }\n \n static void write_rev_index_positions(struct hashfile *f,\n-\t\t\t\t struct pack_idx_entry **objects,\n+\t\t\t\t uint32_t *pack_order,\n \t\t\t\t uint32_t nr_objects)\n {\n-\tuint32_t *pack_order;\n \tuint32_t i;\n-\n-\tALLOC_ARRAY(pack_order, nr_objects);\n-\tfor (i = 0; i < nr_objects; i++)\n-\t\tpack_order[i] = i;\n-\tQSORT_S(pack_order, nr_objects, pack_order_cmp, objects);\n-\n \tfor (i = 0; i < nr_objects; i++)\n \t\thashwrite_be32(f, pack_order[i]);\n-\n-\tfree(pack_order);\n }\n \n static void write_rev_trailer(struct hashfile *f, const unsigned char *hash)\n@@ -228,6 +219,29 @@ const char *write_rev_file(const char *rev_name,\n \t\t\t uint32_t nr_objects,\n \t\t\t const unsigned char *hash,\n \t\t\t unsigned flags)\n+{\n+\tuint32_t *pack_order;\n+\tuint32_t i;\n+\tconst char *ret;\n+\n+\tALLOC_ARRAY(pack_order, nr_objects);\n+\tfor (i = 0; i < nr_objects; i++)\n+\t\tpack_order[i] = i;\n+\tQSORT_S(pack_order, nr_objects, pack_order_cmp, objects);\n+\n+\tret = write_rev_file_order(rev_name, pack_order, nr_objects, hash,\n+\t\t\t\t flags);\n+\n+\tfree(pack_order);\n+\n+\treturn ret;\n+}\n+\n+const char *write_rev_file_order(const char *rev_name,\n+\t\t\t\t uint32_t *pack_order,\n+\t\t\t\t uint32_t nr_objects,\n+\t\t\t\t const unsigned char *hash,\n+\t\t\t\t unsigned flags)\n {\n \tstruct hashfile *f;\n \tint fd;\n@@ -262,7 +276,7 @@ const char *write_rev_file(const char *rev_name,\n \n \twrite_rev_header(f);\n \n-\twrite_rev_index_positions(f, objects, nr_objects);\n+\twrite_rev_index_positions(f, pack_order, nr_objects);\n \twrite_rev_trailer(f, hash);\n \n \tif (rev_name && adjust_shared_perm(rev_name) < 0)\ndiff --git a/pack.h b/pack.h\nindex 857cbd5bd4..fa13954526 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -94,6 +94,7 @@ struct ref;\n void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought);\n \n const char *write_rev_file(const char *rev_name, struct pack_idx_entry **objects, uint32_t nr_objects, const unsigned char *hash, unsigned flags);\n+const char *write_rev_file_order(const char *rev_name, uint32_t *pack_order, uint32_t nr_objects, const unsigned char *hash, unsigned flags);\n \n /*\n * The \"hdr\" output buffer should be at least this big, which will handle sizes\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"420616","messageId":"8bb3dd24a7caaef36d041108c5a0503f6f84e410.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"[PATCH v4 12/16] Documentation/technical: describe multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:04:23Z","receivedAt":"2021-03-30T15:05:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"As a prerequisite to implementing multi-pack bitmaps, motivate and\ndescribe the format and ordering of the multi-pack reverse index.\n\nThe subsequent patch will implement reading this format, and the patch\nafter that will implement writing it while producing a multi-pack index.\n\nCo-authored-by: Jeff King \nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n Documentation/technical/pack-format.txt | 83 +++++++++++++++++++++++++\n 1 file changed, 83 insertions(+)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 1faa949bf6..8d2f42f29e 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -379,3 +379,86 @@ CHUNK DATA:\n TRAILER:\n \n \tIndex checksum of the above contents.\n+\n+== multi-pack-index reverse indexes\n+\n+Similar to the pack-based reverse index, the multi-pack index can also\n+be used to generate a reverse index.\n+\n+Instead of mapping between offset, pack-, and index position, this\n+reverse index maps between an object's position within the MIDX, and\n+that object's position within a pseudo-pack that the MIDX describes\n+(i.e., the ith entry of the multi-pack reverse index holds the MIDX\n+position of ith object in pseudo-pack order).\n+\n+To clarify the difference between these orderings, consider a multi-pack\n+reachability bitmap (which does not yet exist, but is what we are\n+building towards here). Each bit needs to correspond to an object in the\n+MIDX, and so we need an efficient mapping from bit position to MIDX\n+position.\n+\n+One solution is to let bits occupy the same position in the oid-sorted\n+index stored by the MIDX. But because oids are effectively random, their\n+resulting reachability bitmaps would have no locality, and thus compress\n+poorly. (This is the reason that single-pack bitmaps use the pack\n+ordering, and not the .idx ordering, for the same purpose.)\n+\n+So we'd like to define an ordering for the whole MIDX based around\n+pack ordering, which has far better locality (and thus compresses more\n+efficiently). We can think of a pseudo-pack created by the concatenation\n+of all of the packs in the MIDX. E.g., if we had a MIDX with three packs\n+(a, b, c), with 10, 15, and 20 objects respectively, we can imagine an\n+ordering of the objects like:\n+\n+ |a,0|a,1|...|a,9|b,0|b,1|...|b,14|c,0|c,1|...|c,19|\n+\n+where the ordering of the packs is defined by the MIDX's pack list,\n+and then the ordering of objects within each pack is the same as the\n+order in the actual packfile.\n+\n+Given the list of packs and their counts of objects, you can\n+naïvely reconstruct that pseudo-pack ordering (e.g., the object at\n+position 27 must be (c,1) because packs \"a\" and \"b\" consumed 25 of the\n+slots). But there's a catch. Objects may be duplicated between packs, in\n+which case the MIDX only stores one pointer to the object (and thus we'd\n+want only one slot in the bitmap).\n+\n+Callers could handle duplicates themselves by reading objects in order\n+of their bit-position, but that's linear in the number of objects, and\n+much too expensive for ordinary bitmap lookups. Building a reverse index\n+solves this, since it is the logical inverse of the index, and that\n+index has already removed duplicates. But, building a reverse index on\n+the fly can be expensive. Since we already have an on-disk format for\n+pack-based reverse indexes, let's reuse it for the MIDX's pseudo-pack,\n+too.\n+\n+Objects from the MIDX are ordered as follows to string together the\n+pseudo-pack. Let `pack(o)` return the pack from which `o` was selected\n+by the MIDX, and define an ordering of packs based on their numeric ID\n+(as stored by the MIDX). Let `offset(o)` return the object offset of `o`\n+within `pack(o)`. Then, compare `o1` and `o2` as follows:\n+\n+ - If one of `pack(o1)` and `pack(o2)` is preferred and the other\n+ is not, then the preferred one sorts first.\n++\n+(This is a detail that allows the MIDX bitmap to determine which\n+pack should be used by the pack-reuse mechanism, since it can ask\n+the MIDX for the pack containing the object at bit position 0).\n+\n+ - If `pack(o1) ≠ pack(o2)`, then sort the two objects in descending\n+ order based on the pack ID.\n+\n+ - Otherwise, `pack(o1) = pack(o2)`, and the objects are sorted in\n+ pack-order (i.e., `o1` sorts ahead of `o2` exactly when `offset(o1)\n+ < offset(o2)`).\n+\n+In short, a MIDX's pseudo-pack is the de-duplicated concatenation of\n+objects in packs stored by the MIDX, laid out in pack order, and the\n+packs arranged in MIDX order (with the preferred pack coming first).\n+\n+Finally, note that the MIDX's reverse index is not stored as a chunk in\n+the multi-pack-index itself. This is done because the reverse index\n+includes the checksum of the pack or MIDX to which it belongs, which\n+makes it impossible to write in the MIDX. To avoid races when rewriting\n+the MIDX, a MIDX reverse index includes the MIDX's checksum in its\n+filename (e.g., `multi-pack-index-xyz.rev`).\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"420617","messageId":"c070b9c99f50dffe10d627e3506d55bc7fd500d1.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"[PATCH v4 13/16] pack-revindex: read multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:04:26Z","receivedAt":"2021-03-30T15:05:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement reading for multi-pack reverse indexes, as described in the\nprevious patch.\n\nNote that these functions don't yet have any callers, and won't until\nmulti-pack reachability bitmaps are introduced in a later patch series.\nIn the meantime, this patch implements some of the infrastructure\nnecessary to support multi-pack bitmaps.\n\nThere are three new functions exposed by the revindex API:\n\n - load_midx_revindex(): loads the reverse index corresponding to the\n given multi-pack index.\n\n - midx_to_pack_pos() and pack_pos_to_midx(): these convert between the\n multi-pack index and pseudo-pack order.\n\nload_midx_revindex() and pack_pos_to_midx() are both relatively\nstraightforward.\n\nload_midx_revindex() needs a few functions to be exposed from the midx\nAPI. One to get the checksum of a midx, and another to get the .rev's\nfilename. Similar to recent changes in the packed_git struct, three new\nfields are added to the multi_pack_index struct: one to keep track of\nthe size, one to keep track of the mmap'd pointer, and another to point\npast the header and at the reverse index's data.\n\npack_pos_to_midx() simply reads the corresponding entry out of the\ntable.\n\nmidx_to_pack_pos() is the trickiest, since it needs to find an object's\nposition in the psuedo-pack order, but that order can only be recovered\nin the .rev file itself. This mapping can be implemented with a binary\nsearch, but note that the thing we're binary searching over isn't an\narray of values, but rather a permuted order of those values.\n\nSo, when comparing two items, it's helpful to keep in mind the\ndifference. Instead of a traditional binary search, where you are\ncomparing two things directly, here we're comparing a (pack, offset)\ntuple with an index into the multi-pack index. That index describes\nanother (pack, offset) tuple, and it is _those_ two tuples that are\ncompared.\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 11 +++++\n midx.h | 6 +++\n pack-revindex.c | 126 ++++++++++++++++++++++++++++++++++++++++++++++++\n pack-revindex.h | 53 ++++++++++++++++++++\n packfile.c | 3 ++\n 5 files changed, 199 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex 2173a9b45c..c04e794888 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -47,11 +47,22 @@ static uint8_t oid_version(void)\n \t}\n }\n \n+static const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n+{\n+\treturn m->data + m->data_len - the_hash_algo->rawsz;\n+}\n+\n static char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\n \n+char *get_midx_rev_filename(struct multi_pack_index *m)\n+{\n+\treturn xstrfmt(\"%s/pack/multi-pack-index-%s.rev\",\n+\t\t m->object_dir, hash_to_hex(get_midx_checksum(m)));\n+}\n+\n static int midx_read_oid_fanout(const unsigned char *chunk_start,\n \t\t\t\tsize_t chunk_size, void *data)\n {\ndiff --git a/midx.h b/midx.h\nindex 93bd68189e..0a8294d2ee 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -15,6 +15,10 @@ struct multi_pack_index {\n \tconst unsigned char *data;\n \tsize_t data_len;\n \n+\tconst uint32_t *revindex_data;\n+\tconst uint32_t *revindex_map;\n+\tsize_t revindex_len;\n+\n \tuint32_t signature;\n \tunsigned char version;\n \tunsigned char hash_len;\n@@ -37,6 +41,8 @@ struct multi_pack_index {\n \n #define MIDX_PROGRESS (1 << 0)\n \n+char *get_midx_rev_filename(struct multi_pack_index *m);\n+\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 4262530449..0e4a31d9db 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -3,6 +3,7 @@\n #include \"object-store.h\"\n #include \"packfile.h\"\n #include \"config.h\"\n+#include \"midx.h\"\n \n struct revindex_entry {\n \toff_t offset;\n@@ -293,6 +294,43 @@ int load_pack_revindex(struct packed_git *p)\n \treturn -1;\n }\n \n+int load_midx_revindex(struct multi_pack_index *m)\n+{\n+\tchar *revindex_name;\n+\tint ret;\n+\tif (m->revindex_data)\n+\t\treturn 0;\n+\n+\trevindex_name = get_midx_rev_filename(m);\n+\n+\tret = load_revindex_from_disk(revindex_name,\n+\t\t\t\t m->num_objects,\n+\t\t\t\t &m->revindex_map,\n+\t\t\t\t &m->revindex_len);\n+\tif (ret)\n+\t\tgoto cleanup;\n+\n+\tm->revindex_data = (const uint32_t *)((const char *)m->revindex_map + RIDX_HEADER_SIZE);\n+\n+cleanup:\n+\tfree(revindex_name);\n+\treturn ret;\n+}\n+\n+int close_midx_revindex(struct multi_pack_index *m)\n+{\n+\tif (!m || !m->revindex_map)\n+\t\treturn 0;\n+\n+\tmunmap((void*)m->revindex_map, m->revindex_len);\n+\n+\tm->revindex_map = NULL;\n+\tm->revindex_data = NULL;\n+\tm->revindex_len = 0;\n+\n+\treturn 0;\n+}\n+\n int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n {\n \tunsigned lo, hi;\n@@ -347,3 +385,91 @@ off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n \telse\n \t\treturn nth_packed_object_offset(p, pack_pos_to_index(p, pos));\n }\n+\n+uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos)\n+{\n+\tif (!m->revindex_data)\n+\t\tBUG(\"pack_pos_to_midx: reverse index not yet loaded\");\n+\tif (m->num_objects <= pos)\n+\t\tBUG(\"pack_pos_to_midx: out-of-bounds object at %\"PRIu32, pos);\n+\treturn get_be32(m->revindex_data + pos);\n+}\n+\n+struct midx_pack_key {\n+\tuint32_t pack;\n+\toff_t offset;\n+\n+\tuint32_t preferred_pack;\n+\tstruct multi_pack_index *midx;\n+};\n+\n+static int midx_pack_order_cmp(const void *va, const void *vb)\n+{\n+\tconst struct midx_pack_key *key = va;\n+\tstruct multi_pack_index *midx = key->midx;\n+\n+\tuint32_t versus = pack_pos_to_midx(midx, (uint32_t*)vb - (const uint32_t *)midx->revindex_data);\n+\tuint32_t versus_pack = nth_midxed_pack_int_id(midx, versus);\n+\toff_t versus_offset;\n+\n+\tuint32_t key_preferred = key->pack == key->preferred_pack;\n+\tuint32_t versus_preferred = versus_pack == key->preferred_pack;\n+\n+\t/*\n+\t * First, compare the preferred-ness, noting that the preferred pack\n+\t * comes first.\n+\t */\n+\tif (key_preferred && !versus_preferred)\n+\t\treturn -1;\n+\telse if (!key_preferred && versus_preferred)\n+\t\treturn 1;\n+\n+\t/* Then, break ties first by comparing the pack IDs. */\n+\tif (key->pack < versus_pack)\n+\t\treturn -1;\n+\telse if (key->pack > versus_pack)\n+\t\treturn 1;\n+\n+\t/* Finally, break ties by comparing offsets within a pack. */\n+\tversus_offset = nth_midxed_offset(midx, versus);\n+\tif (key->offset < versus_offset)\n+\t\treturn -1;\n+\telse if (key->offset > versus_offset)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+int midx_to_pack_pos(struct multi_pack_index *m, uint32_t at, uint32_t *pos)\n+{\n+\tstruct midx_pack_key key;\n+\tuint32_t *found;\n+\n+\tif (!m->revindex_data)\n+\t\tBUG(\"midx_to_pack_pos: reverse index not yet loaded\");\n+\tif (m->num_objects <= at)\n+\t\tBUG(\"midx_to_pack_pos: out-of-bounds object at %\"PRIu32, at);\n+\n+\tkey.pack = nth_midxed_pack_int_id(m, at);\n+\tkey.offset = nth_midxed_offset(m, at);\n+\tkey.midx = m;\n+\t/*\n+\t * The preferred pack sorts first, so determine its identifier by\n+\t * looking at the first object in pseudo-pack order.\n+\t *\n+\t * Note that if no --preferred-pack is explicitly given when writing a\n+\t * multi-pack index, then whichever pack has the lowest identifier\n+\t * implicitly is preferred (and includes all its objects, since ties are\n+\t * broken first by pack identifier).\n+\t */\n+\tkey.preferred_pack = nth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));\n+\n+\tfound = bsearch(&key, m->revindex_data, m->num_objects,\n+\t\t\tsizeof(*m->revindex_data), midx_pack_order_cmp);\n+\n+\tif (!found)\n+\t\treturn error(\"bad offset for revindex\");\n+\n+\t*pos = found - m->revindex_data;\n+\treturn 0;\n+}\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex ba7c82c125..479b8f2f9c 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -14,6 +14,20 @@\n *\n * - offset: the byte offset within the .pack file at which the object contents\n * can be found\n+ *\n+ * The revindex can also be used with a multi-pack index (MIDX). In this\n+ * setting:\n+ *\n+ * - index position refers to an object's numeric position within the MIDX\n+ *\n+ * - pack position refers to an object's position within a non-existent pack\n+ * described by the MIDX. The pack structure is described in\n+ * Documentation/technical/pack-format.txt.\n+ *\n+ * It is effectively a concatanation of all packs in the MIDX (ordered by\n+ * their numeric ID within the MIDX) in their original order within each\n+ * pack), removing duplicates, and placing the preferred pack (if any)\n+ * first.\n */\n \n \n@@ -24,6 +38,7 @@\n #define GIT_TEST_REV_INDEX_DIE_IN_MEMORY \"GIT_TEST_REV_INDEX_DIE_IN_MEMORY\"\n \n struct packed_git;\n+struct multi_pack_index;\n \n /*\n * load_pack_revindex populates the revindex's internal data-structures for the\n@@ -34,6 +49,22 @@ struct packed_git;\n */\n int load_pack_revindex(struct packed_git *p);\n \n+/*\n+ * load_midx_revindex loads the '.rev' file corresponding to the given\n+ * multi-pack index by mmap-ing it and assigning pointers in the\n+ * multi_pack_index to point at it.\n+ *\n+ * A negative number is returned on error.\n+ */\n+int load_midx_revindex(struct multi_pack_index *m);\n+\n+/*\n+ * Frees resources associated with a multi-pack reverse index.\n+ *\n+ * A negative number is returned on error.\n+ */\n+int close_midx_revindex(struct multi_pack_index *m);\n+\n /*\n * offset_to_pack_pos converts an object offset to a pack position. This\n * function returns zero on success, and a negative number otherwise. The\n@@ -71,4 +102,26 @@ uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);\n */\n off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos);\n \n+/*\n+ * pack_pos_to_midx converts the object at position \"pos\" within the MIDX\n+ * pseudo-pack into a MIDX position.\n+ *\n+ * If the reverse index has not yet been loaded, or the position is out of\n+ * bounds, this function aborts.\n+ *\n+ * This function runs in time O(log N) with the number of objects in the MIDX.\n+ */\n+uint32_t pack_pos_to_midx(struct multi_pack_index *m, uint32_t pos);\n+\n+/*\n+ * midx_to_pack_pos converts from the MIDX-relative position at \"at\" to the\n+ * corresponding pack position.\n+ *\n+ * If the reverse index has not yet been loaded, or the position is out of\n+ * bounds, this function aborts.\n+ *\n+ * This function runs in constant time.\n+ */\n+int midx_to_pack_pos(struct multi_pack_index *midx, uint32_t at, uint32_t *pos);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 6661f3325a..8668345d93 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -862,6 +862,9 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \n \tif (!strcmp(file_name, \"multi-pack-index\"))\n \t\treturn;\n+\tif (starts_with(file_name, \"multi-pack-index\") &&\n+\t ends_with(file_name, \".rev\"))\n+\t\treturn;\n \tif (ends_with(file_name, \".idx\") ||\n \t ends_with(file_name, \".rev\") ||\n \t ends_with(file_name, \".pack\") ||\n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"420618","messageId":"7b793e7d0918bc5869f9c2aef818fdd7353ea77b.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"[PATCH v4 16/16] midx.c: improve cache locality in midx_pack_order_cmp()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:04:36Z","receivedAt":"2021-03-30T15:05:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jeff King \n\nThere is a lot of pointer dereferencing in the pre-image version of\n'midx_pack_order_cmp()', which this patch gets rid of.\n\nInstead of comparing the pack preferred-ness and then the pack id, both\nof these checks are done at the same time by using the high-order bit of\nthe pack id to represent whether it's preferred. Then the pack id and\noffset are compared as usual.\n\nThis produces the same result so long as there are less than 2^31 packs,\nwhich seems like a likely assumption to make in practice.\n\nSigned-off-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n midx.c | 55 +++++++++++++++++++++++++++++--------------------------\n 1 file changed, 29 insertions(+), 26 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex b96eaa12fb..9e86583172 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -818,46 +818,49 @@ static int write_midx_large_offsets(struct hashfile *f,\n \treturn 0;\n }\n \n-static int midx_pack_order_cmp(const void *va, const void *vb, void *_ctx)\n+struct midx_pack_order_data {\n+\tuint32_t nr;\n+\tuint32_t pack;\n+\toff_t offset;\n+};\n+\n+static int midx_pack_order_cmp(const void *va, const void *vb)\n {\n-\tstruct write_midx_context *ctx = _ctx;\n-\n-\tstruct pack_midx_entry *a = &ctx->entries[*(const uint32_t *)va];\n-\tstruct pack_midx_entry *b = &ctx->entries[*(const uint32_t *)vb];\n-\n-\tuint32_t perm_a = ctx->pack_perm[a->pack_int_id];\n-\tuint32_t perm_b = ctx->pack_perm[b->pack_int_id];\n-\n-\t/* Sort objects in the preferred pack ahead of any others. */\n-\tif (a->preferred > b->preferred)\n+\tconst struct midx_pack_order_data *a = va, *b = vb;\n+\tif (a->pack < b->pack)\n \t\treturn -1;\n-\tif (a->preferred < b->preferred)\n+\telse if (a->pack > b->pack)\n \t\treturn 1;\n-\n-\t/* Then, order objects by which packs they appear in. */\n-\tif (perm_a < perm_b)\n+\telse if (a->offset < b->offset)\n \t\treturn -1;\n-\tif (perm_a > perm_b)\n+\telse if (a->offset > b->offset)\n \t\treturn 1;\n-\n-\t/* Then, disambiguate by their offset within each pack. */\n-\tif (a->offset < b->offset)\n-\t\treturn -1;\n-\tif (a->offset > b->offset)\n-\t\treturn 1;\n-\n-\treturn 0;\n+\telse\n+\t\treturn 0;\n }\n \n static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n {\n+\tstruct midx_pack_order_data *data;\n \tuint32_t *pack_order;\n \tuint32_t i;\n \n+\tALLOC_ARRAY(data, ctx->entries_nr);\n+\tfor (i = 0; i < ctx->entries_nr; i++) {\n+\t\tstruct pack_midx_entry *e = &ctx->entries[i];\n+\t\tdata[i].nr = i;\n+\t\tdata[i].pack = ctx->pack_perm[e->pack_int_id];\n+\t\tif (!e->preferred)\n+\t\t\tdata[i].pack |= (1U << 31);\n+\t\tdata[i].offset = e->offset;\n+\t}\n+\n+\tQSORT(data, ctx->entries_nr, midx_pack_order_cmp);\n+\n \tALLOC_ARRAY(pack_order, ctx->entries_nr);\n \tfor (i = 0; i < ctx->entries_nr; i++)\n-\t\tpack_order[i] = i;\n-\tQSORT_S(pack_order, ctx->entries_nr, midx_pack_order_cmp, ctx);\n+\t\tpack_order[i] = data[i].nr;\n+\tfree(data);\n \n \treturn pack_order;\n }\n-- \n2.30.0.667.g81c0cbc6fd\n"},{"id":"420619","messageId":"47409cc50835fb03d535d3817f01d62a68512144.1617116623.git.me@ttaylorr.com","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"[PATCH v4 15/16] pack-revindex: write multi-pack reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:04:32Z","receivedAt":"2021-03-30T15:05:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement the writing half of multi-pack reverse indexes. This is\nnothing more than the format describe a few patches ago, with a new set\nof helper functions that will be used to clear out stale .rev files\ncorresponding to old MIDXs.\n\nUnfortunately, a very similar comparison function as the one implemented\nrecently in pack-revindex.c is reimplemented here, this time accepting a\nMIDX-internal type. An effort to DRY these up would create more\nindirection and overhead than is necessary, so it isn't pursued here.\n\nCurrently, there are no callers which pass the MIDX_WRITE_REV_INDEX\nflag, meaning that this is all dead code. But, that won't be the case\nfor long, since subsequent patches will introduce the multi-pack bitmap,\nwhich will begin passing this field.\n\n(In midx.c:write_midx_internal(), the two adjacent if statements share a\nconditional, but are written separately since the first one will\neventually also handle the MIDX_WRITE_BITMAP flag, which does not yet\nexist.)\n\nSigned-off-by: Taylor Blau \n---\n midx.c | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n midx.h | 1 +\n 2 files changed, 116 insertions(+)\n\ndiff --git a/midx.c b/midx.c\nindex c04e794888..b96eaa12fb 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -12,6 +12,7 @@\n #include \"run-command.h\"\n #include \"repository.h\"\n #include \"chunk-format.h\"\n+#include \"pack.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -462,6 +463,7 @@ struct write_midx_context {\n \tuint32_t entries_nr;\n \n \tuint32_t *pack_perm;\n+\tuint32_t *pack_order;\n \tunsigned large_offsets_needed:1;\n \tuint32_t num_large_offsets;\n \n@@ -816,6 +818,70 @@ static int write_midx_large_offsets(struct hashfile *f,\n \treturn 0;\n }\n \n+static int midx_pack_order_cmp(const void *va, const void *vb, void *_ctx)\n+{\n+\tstruct write_midx_context *ctx = _ctx;\n+\n+\tstruct pack_midx_entry *a = &ctx->entries[*(const uint32_t *)va];\n+\tstruct pack_midx_entry *b = &ctx->entries[*(const uint32_t *)vb];\n+\n+\tuint32_t perm_a = ctx->pack_perm[a->pack_int_id];\n+\tuint32_t perm_b = ctx->pack_perm[b->pack_int_id];\n+\n+\t/* Sort objects in the preferred pack ahead of any others. */\n+\tif (a->preferred > b->preferred)\n+\t\treturn -1;\n+\tif (a->preferred < b->preferred)\n+\t\treturn 1;\n+\n+\t/* Then, order objects by which packs they appear in. */\n+\tif (perm_a < perm_b)\n+\t\treturn -1;\n+\tif (perm_a > perm_b)\n+\t\treturn 1;\n+\n+\t/* Then, disambiguate by their offset within each pack. */\n+\tif (a->offset < b->offset)\n+\t\treturn -1;\n+\tif (a->offset > b->offset)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t *midx_pack_order(struct write_midx_context *ctx)\n+{\n+\tuint32_t *pack_order;\n+\tuint32_t i;\n+\n+\tALLOC_ARRAY(pack_order, ctx->entries_nr);\n+\tfor (i = 0; i < ctx->entries_nr; i++)\n+\t\tpack_order[i] = i;\n+\tQSORT_S(pack_order, ctx->entries_nr, midx_pack_order_cmp, ctx);\n+\n+\treturn pack_order;\n+}\n+\n+static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n+\t\t\t\t struct write_midx_context *ctx)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *tmp_file;\n+\n+\tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex(midx_hash));\n+\n+\ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n+\t\t\t\t\tmidx_hash, WRITE_REV);\n+\n+\tif (finalize_object_file(tmp_file, buf.buf))\n+\t\tdie(_(\"cannot store reverse index file\"));\n+\n+\tstrbuf_release(&buf);\n+}\n+\n+static void clear_midx_files_ext(struct repository *r, const char *ext,\n+\t\t\t\t unsigned char *keep_hash);\n+\n static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n \t\t\t struct string_list *packs_to_drop,\n \t\t\t const char *preferred_pack_name,\n@@ -1001,6 +1067,14 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tfinalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);\n \tfree_chunkfile(cf);\n+\n+\tif (flags & MIDX_WRITE_REV_INDEX)\n+\t\tctx.pack_order = midx_pack_order(&ctx);\n+\n+\tif (flags & MIDX_WRITE_REV_INDEX)\n+\t\twrite_midx_reverse_index(midx_name, midx_hash, &ctx);\n+\tclear_midx_files_ext(the_repository, \".rev\", midx_hash);\n+\n \tcommit_lock_file(&lk);\n \n cleanup:\n@@ -1015,6 +1089,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \tfree(ctx.info);\n \tfree(ctx.entries);\n \tfree(ctx.pack_perm);\n+\tfree(ctx.pack_order);\n \tfree(midx_name);\n \treturn result;\n }\n@@ -1027,6 +1102,44 @@ int write_midx_file(const char *object_dir,\n \t\t\t\t flags);\n }\n \n+struct clear_midx_data {\n+\tchar *keep;\n+\tconst char *ext;\n+};\n+\n+static void clear_midx_file_ext(const char *full_path, size_t full_path_len,\n+\t\t\t\tconst char *file_name, void *_data)\n+{\n+\tstruct clear_midx_data *data = _data;\n+\n+\tif (!(starts_with(file_name, \"multi-pack-index-\") &&\n+\t ends_with(file_name, data->ext)))\n+\t\treturn;\n+\tif (data->keep && !strcmp(data->keep, file_name))\n+\t\treturn;\n+\n+\tif (unlink(full_path))\n+\t\tdie_errno(_(\"failed to remove %s\"), full_path);\n+}\n+\n+static void clear_midx_files_ext(struct repository *r, const char *ext,\n+\t\t\t\t unsigned char *keep_hash)\n+{\n+\tstruct clear_midx_data data;\n+\tmemset(&data, 0, sizeof(struct clear_midx_data));\n+\n+\tif (keep_hash)\n+\t\tdata.keep = xstrfmt(\"multi-pack-index-%s%s\",\n+\t\t\t\t hash_to_hex(keep_hash), ext);\n+\tdata.ext = ext;\n+\n+\tfor_each_file_in_pack_dir(r->objects->odb->path,\n+\t\t\t\t clear_midx_file_ext,\n+\t\t\t\t &data);\n+\n+\tfree(data.keep);\n+}\n+\n void clear_midx_file(struct repository *r)\n {\n \tchar *midx = get_midx_filename(r->objects->odb->path);\n@@ -1039,6 +1152,8 @@ void clear_midx_file(struct repository *r)\n \tif (remove_path(midx))\n \t\tdie(_(\"failed to clear multi-pack-index at %s\"), midx);\n \n+\tclear_midx_files_ext(r, \".rev\", NULL);\n+\n \tfree(midx);\n }\n \ndiff --git a/midx.h b/midx.h\nindex 0a8294d2ee..8684cf0fef 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -40,6 +40,7 @@ struct multi_pack_index {\n };\n \n #define MIDX_PROGRESS (1 << 0)\n+#define MIDX_WRITE_REV_INDEX (1 << 1)\n \n char *get_midx_rev_filename(struct multi_pack_index *m);\n \n-- \n2.30.0.667.g81c0cbc6fd\n\n"},{"id":"420624","messageId":"YGNHj6iRlCOjjCL8@coredump.intra.peff.net","threadId":"55135","inReplyTo":"cover.1617116623.git.me@ttaylorr.com","subject":"Re: [PATCH v4 00/16] midx: implement a multi-pack reverse index","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-30T15:45:19Z","receivedAt":"2021-03-30T15:46:19Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Mar 30, 2021 at 11:03:44AM -0400, Taylor Blau wrote:\n\n> Here is another reroll of my series to implement a reverse index in\n> preparation for multi-pack reachability bitmaps.\n\nThanks, this addresses all of my comments from the last round.\n\n> This reroll differs only in the feedback I incorporated from Peff's review. They\n> are mostly cosmetic; the most substantial change being that the --preferred-pack\n> code now uses bsearch() to locate the name of the preferred pack (instead of\n> implementing a binary search itself).\n\nYeah, I read over this part carefully, since it's actual new code (that\nisn't run yet!), but I think it is correct.\n\nOne minor observation:\n\n> @@ t/t5319-multi-pack-index.sh: test_expect_success 'warn on improper hash version'\n> +\t\t\twrite --preferred-pack=test-BC-$bc.idx 2>err &&\n> +\t\ttest_must_be_empty err &&\n> +\n> ++\t\techo hi &&\n> ++\t\ttest-tool read-midx --show-objects objects >out &&\n> ++\n> +\t\tofs=$(git show-index +\t\t\tcut -d\" \" -f1) &&\n> -+\t\tmidx_expect_object_offset $b $ofs objects\n> ++\t\tprintf \"%s %s\\tobjects/pack/test-BC-%s.pack\\n\" \\\n> ++\t\t\t\"$b\" \"$ofs\" \"$bc\" >expect &&\n> ++\t\tgrep ^$b out >actual &&\n> ++\n> ++\t\ttest_cmp expect actual\n> +\t)\n> +'\n\nI'd probably have just skipped show-index entirely, and done:\n\n grep \"^$b .* objects/pack/test-BC\" actual\n\nwhich expresses the intent ($b came from that pack). But I don't mind\nthe more exacting version (and certainly it is not worth a re-roll even\nif you prefer mine).\n\n-Peff\n"},{"id":"420625","messageId":"YGNIp4AHcdaHHHUc@nand.local","threadId":"55135","inReplyTo":"YGNHj6iRlCOjjCL8@coredump.intra.peff.net","subject":"Re: [PATCH v4 00/16] midx: implement a multi-pack reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-30T15:49:59Z","receivedAt":"2021-03-30T15:50:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Mar 30, 2021 at 11:45:19AM -0400, Jeff King wrote:\n> > This reroll differs only in the feedback I incorporated from Peff's review. They\n> > are mostly cosmetic; the most substantial change being that the --preferred-pack\n> > code now uses bsearch() to locate the name of the preferred pack (instead of\n> > implementing a binary search itself).\n>\n> Yeah, I read over this part carefully, since it's actual new code (that\n> isn't run yet!), but I think it is correct.\n\nThankfully this does have coverage via any test that passes\n`--preferred-pack` (like the one below).\n\n> One minor observation:\n>\n> > @@ t/t5319-multi-pack-index.sh: test_expect_success 'warn on improper hash version'\n> > +\t\t\twrite --preferred-pack=test-BC-$bc.idx 2>err &&\n> > +\t\ttest_must_be_empty err &&\n> > +\n> > ++\t\techo hi &&\n> > ++\t\ttest-tool read-midx --show-objects objects >out &&\n> > ++\n> > +\t\tofs=$(git show-index > +\t\t\tcut -d\" \" -f1) &&\n> > -+\t\tmidx_expect_object_offset $b $ofs objects\n> > ++\t\tprintf \"%s %s\\tobjects/pack/test-BC-%s.pack\\n\" \\\n> > ++\t\t\t\"$b\" \"$ofs\" \"$bc\" >expect &&\n> > ++\t\tgrep ^$b out >actual &&\n> > ++\n> > ++\t\ttest_cmp expect actual\n> > +\t)\n> > +'\n>\n> I'd probably have just skipped show-index entirely, and done:\n>\n> grep \"^$b .* objects/pack/test-BC\" actual\n>\n> which expresses the intent ($b came from that pack). But I don't mind\n> the more exacting version (and certainly it is not worth a re-roll even\n> if you prefer mine).\n\nI originally wrote it that way, but decided to write both expect and\nactual to make debugging easier if this ever regresses. Not like it's\nthat hard to run the test-tool yourself in the trash directory, but\nhaving a snapshot of that object from the MIDX's perspective might make\nthings a little easier.\n\nAnyway, I agree with you that it doesn't probably matter a ton either\nway.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"420626","messageId":"YGNLcBH5rldiSAWC@coredump.intra.peff.net","threadId":"55135","inReplyTo":"YGNIp4AHcdaHHHUc@nand.local","subject":"Re: [PATCH v4 00/16] midx: implement a multi-pack reverse index","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-30T16:01:52Z","receivedAt":"2021-03-30T16:02:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Mar 30, 2021 at 11:49:59AM -0400, Taylor Blau wrote:\n\n> On Tue, Mar 30, 2021 at 11:45:19AM -0400, Jeff King wrote:\n> > > This reroll differs only in the feedback I incorporated from Peff's review. They\n> > > are mostly cosmetic; the most substantial change being that the --preferred-pack\n> > > code now uses bsearch() to locate the name of the preferred pack (instead of\n> > > implementing a binary search itself).\n> >\n> > Yeah, I read over this part carefully, since it's actual new code (that\n> > isn't run yet!), but I think it is correct.\n> \n> Thankfully this does have coverage via any test that passes\n> `--preferred-pack` (like the one below).\n\nOh right, I forgot this was touching that early part. So now I'm doubly\nconfident in it.\n\n-Peff\n"},{"id":"420763","messageId":"YGUUkoaFOdfs7nRs@nand.local","threadId":"55135","inReplyTo":"d061828e7e52eb97c72dbc0853b02180feea0d8b.1617116623.git.me@ttaylorr.com","subject":"Re: [PATCH v4 08/16] midx: allow marking a pack as preferred","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-04-01T00:32:18Z","receivedAt":"2021-04-01T00:33:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Junio,\n\nOn Tue, Mar 30, 2021 at 11:04:11AM -0400, Taylor Blau wrote:\n\nI accidentally left a stray debugging line in here, and managed to skip\nover it when reading the range-diff. It's right...\n\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> index b4afab1dfc..031a5570c0 100755\n> --- a/t/t5319-multi-pack-index.sh\n> +++ b/t/t5319-multi-pack-index.sh\n> @@ -234,6 +234,49 @@ test_expect_success 'warn on improper hash version' '\n> \t)\n> '\n>\n> +test_expect_success 'midx picks objects from preferred pack' '\n> +\ttest_when_finished rm -rf preferred.git &&\n> +\tgit init --bare preferred.git &&\n> +\t(\n> +\t\tcd preferred.git &&\n> +\n> +\t\ta=$(echo \"a\" | git hash-object -w --stdin) &&\n> +\t\tb=$(echo \"b\" | git hash-object -w --stdin) &&\n> +\t\tc=$(echo \"c\" | git hash-object -w --stdin) &&\n> +\n> +\t\t# Set up two packs, duplicating the object \"B\" at different\n> +\t\t# offsets.\n> +\t\t#\n> +\t\t# Note that the \"BC\" pack (the one we choose as preferred) sorts\n> +\t\t# lexically after the \"AB\" pack, meaning that omitting the\n> +\t\t# --preferred-pack argument would cause this test to fail (since\n> +\t\t# the MIDX code would select the copy of \"b\" in the \"AB\" pack).\n> +\t\tgit pack-objects objects/pack/test-AB <<-EOF &&\n> +\t\t$a\n> +\t\t$b\n> +\t\tEOF\n> +\t\tbc=$(git pack-objects objects/pack/test-BC <<-EOF\n> +\t\t$b\n> +\t\t$c\n> +\t\tEOF\n> +\t\t) &&\n> +\n> +\t\tgit multi-pack-index --object-dir=objects \\\n> +\t\t\twrite --preferred-pack=test-BC-$bc.idx 2>err &&\n> +\t\ttest_must_be_empty err &&\n> +\n> +\t\techo hi &&\n\n...here. Would you mind fixing it up locally before applying this to\nnext?\n\nSorry for the trouble.\n\nThanks,\nTaylor\n"},{"id":"434984","messageId":"patch-1.1-366ba928bd-20210908T010743Z-avarab@gmail.com","threadId":"55135","inReplyTo":"9f40019eb3d3315cb03440e6237bced4feb6cf67.1617116623.git.me@ttaylorr.com","subject":"[PATCH] pack-write: skip *.rev work when not writing *.rev","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-08T01:08:03Z","receivedAt":"2021-09-08T01:08:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a performance regression introduced in a587b5a786 (pack-write.c:\nextract 'write_rev_file_order', 2021-03-30) and stop needlessly\nallocating the \"pack_order\" array and sorting it with\n\"pack_order_cmp()\", only to throw that work away when we discover that\nwe're not writing *.rev files after all.\n\nThis redundant work was not present in the original version of this\ncode added in 8ef50d9958 (pack-write.c: prepare to write 'pack-*.rev'\nfiles, 2021-01-25). There we'd call write_rev_file() from\ne.g. finish_tmp_packfile(), but we'd \"return NULL\" early in\nwrite_rev_file() if not doing a \"WRITE_REV\" or \"WRITE_REV_VERIFY\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n pack-write.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/pack-write.c b/pack-write.c\nindex f1fc3ecafa..1883848e7c 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -224,6 +224,9 @@ const char *write_rev_file(const char *rev_name,\n \tuint32_t i;\n \tconst char *ret;\n \n+\tif (!(flags & WRITE_REV) && !(flags & WRITE_REV_VERIFY))\n+\t\treturn NULL;\n+\n \tALLOC_ARRAY(pack_order, nr_objects);\n \tfor (i = 0; i < nr_objects; i++)\n \t\tpack_order[i] = i;\n-- \n2.33.0.819.gea1b153a43c\n\n"},{"id":"434988","messageId":"CAPUEspgh0kbm-ro13Xt2-g-hPoBMK2-34iP2GvK_gaa2bGC+GQ@mail.gmail.com","threadId":"55135","inReplyTo":"patch-1.1-366ba928bd-20210908T010743Z-avarab@gmail.com","subject":"Re: [PATCH] pack-write: skip *.rev work when not writing *.rev","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2021-09-08T01:35:10Z","receivedAt":"2021-09-08T01:35:26Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Tue, Sep 7, 2021 at 6:10 PM Ævar Arnfjörð Bjarmason wrote:\n> + if (!(flags & WRITE_REV) && !(flags & WRITE_REV_VERIFY))\n> + return NULL;\n\nI see this expression matches exactly the logic from 8ef50d9958 which\nis why I presume\nyou used it, but the simpler (and logically equivalent[1]) :\n\n if !((flags & WRITE_REV) || (flags & WRITE_REV_VERIFY))\n\nis easier to read IMHO\n\nCarlo\n\n[1] https://en.wikipedia.org/wiki/De_Morgan%27s_laws\n"},{"id":"435013","messageId":"YTgjA/mwH7wY4oyZ@nand.local","threadId":"55135","inReplyTo":"CAPUEspgh0kbm-ro13Xt2-g-hPoBMK2-34iP2GvK_gaa2bGC+GQ@mail.gmail.com","subject":"Re: [PATCH] pack-write: skip *.rev work when not writing *.rev","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-08T02:42:11Z","receivedAt":"2021-09-08T02:42:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 06:35:10PM -0700, Carlo Arenas wrote:\n> On Tue, Sep 7, 2021 at 6:10 PM Ævar Arnfjörð Bjarmason wrote:\n> > + if (!(flags & WRITE_REV) && !(flags & WRITE_REV_VERIFY))\n> > + return NULL;\n>\n> I see this expression matches exactly the logic from 8ef50d9958 which\n> is why I presume\n> you used it, but the simpler (and logically equivalent[1]) :\n>\n> if !((flags & WRITE_REV) || (flags & WRITE_REV_VERIFY))\n\nEven simpler would be:\n\n if (!(flags & (WRITE_REV | WRITE_REV_VERIFY)))\n\nalthough with optimization flags other than -O0, it seems that each of\nthese three produce the same result [1], so I don't think that it\nmatters much either way ;-).\n\nThanks,\nTaylor\n\n[1]: https://godbolt.org/z/fxxhzEz79\n"},{"id":"435014","messageId":"YTglEreI7nFCII/b@nand.local","threadId":"55135","inReplyTo":"patch-1.1-366ba928bd-20210908T010743Z-avarab@gmail.com","subject":"Re: [PATCH] pack-write: skip *.rev work when not writing *.rev","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-08T02:50:58Z","receivedAt":"2021-09-08T02:51:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 08, 2021 at 03:08:03AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Fix a performance regression introduced in a587b5a786 (pack-write.c:\n> extract 'write_rev_file_order', 2021-03-30) and stop needlessly\n> allocating the \"pack_order\" array and sorting it with\n> \"pack_order_cmp()\", only to throw that work away when we discover that\n> we're not writing *.rev files after all.\n>\n> This redundant work was not present in the original version of this\n> code added in 8ef50d9958 (pack-write.c: prepare to write 'pack-*.rev'\n> files, 2021-01-25). There we'd call write_rev_file() from\n> e.g. finish_tmp_packfile(), but we'd \"return NULL\" early in\n> write_rev_file() if not doing a \"WRITE_REV\" or \"WRITE_REV_VERIFY\".\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> pack-write.c | 3 +++\n> 1 file changed, 3 insertions(+)\n>\n> diff --git a/pack-write.c b/pack-write.c\n> index f1fc3ecafa..1883848e7c 100644\n> --- a/pack-write.c\n> +++ b/pack-write.c\n> @@ -224,6 +224,9 @@ const char *write_rev_file(const char *rev_name,\n> \tuint32_t i;\n> \tconst char *ret;\n>\n> +\tif (!(flags & WRITE_REV) && !(flags & WRITE_REV_VERIFY))\n> +\t\treturn NULL;\n> +\n\nGreat catch! This fix as-is is obviously correct, but it does make the\nsame checks in write_rev_file_order redundant as a result.\n\nIf we call write_rev_file() without WRITE_REV, then we'll never call\nwrite_rev_file_order(). The other caller of write_rev_file_order() is\nfrom midx.c:write_midx_reverse_index(), which is only called by\nwrite_midx_internal() where it is guarded by a similar conditional.\n\nSo I think we could probably either:\n\n - remove the check from write_rev_file_order() altogether, moving it\n to write_rev_file() like you did here, or\n\n - remove the check from write_rev_file_order() and elevate it to the\n caller which is missing the check in finish_tmp_packfile()\n\nOf the two, I think the former is more appealing (since no other\nfunctions called by finish_tmp_packfile() are guarded like that; they\nconditionally behave as noops depending on `flags`).\n\nBut what you wrote here is just fine as-is, so the above are just some\noptional ideas for potential improvements.\n\nThanks,\nTaylor\n"},{"id":"435015","messageId":"YTgy85aiCL1bJjKE@nand.local","threadId":"55135","inReplyTo":"YTglEreI7nFCII/b@nand.local","subject":"Re: [PATCH] pack-write: skip *.rev work when not writing *.rev","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-08T03:50:11Z","receivedAt":"2021-09-08T03:50:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 10:50:58PM -0400, Taylor Blau wrote:\n> Of the two, I think the former is more appealing (since no other\n> functions called by finish_tmp_packfile() are guarded like that; they\n> conditionally behave as noops depending on `flags`).\n\nSorry; this is nonsensical. The only other function we call is\nwrite_idx_file() which merely changes its behavior based on flags, but\nit never behaves as a noop.\n\nThat doesn't change my thinking about preferring the former of my two\nsuggestions, but just wanted to correct my error.\n\nThanks,\nTaylor\n"},{"id":"435060","messageId":"87r1dziczs.fsf@evledraar.gmail.com","threadId":"55135","inReplyTo":"YTgy85aiCL1bJjKE@nand.local","subject":"Re: [PATCH] pack-write: skip *.rev work when not writing *.rev","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-08T10:18:38Z","receivedAt":"2021-09-08T10:21:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Sep 07 2021, Taylor Blau wrote:\n\n> On Tue, Sep 07, 2021 at 10:50:58PM -0400, Taylor Blau wrote:\n>> Of the two, I think the former is more appealing (since no other\n>> functions called by finish_tmp_packfile() are guarded like that; they\n>> conditionally behave as noops depending on `flags`).\n>\n> Sorry; this is nonsensical. The only other function we call is\n> write_idx_file() which merely changes its behavior based on flags, but\n> it never behaves as a noop.\n>\n> That doesn't change my thinking about preferring the former of my two\n> suggestions, but just wanted to correct my error.\n\nI agree that this code is very confusing overall, but would prefer to\nwait on refactoring further until the two topics in flight (this and the\nother pack-write topic) settle.\n\nAs shown in\nhttps://lore.kernel.org/git/87v93bidhn.fsf@evledraar.gmail.com/ I think\nthe best thing to do is neither of the narrow fixes you suggest, but to\nmore deeply untangle the whole mess around how we choose to write these\nfiles & with what options. A lot of it is bit-twiddling back and forth\nfor no real reason.\n\nOnce we do that it becomes impossible to land in a mode where these\nfunctions need to in principle deal with writing a \"real\" file and the\n\"verify\" mode, which as noted in the linked E-Mail is the case now, we\njust need/want these \"is more than one set?\" checks & assertions because\nwe've made the interface overly confusing/general.\n"},{"id":"435111","messageId":"xmqqo8935ast.fsf@gitster.g","threadId":"55135","inReplyTo":"YTgjA/mwH7wY4oyZ@nand.local","subject":"Re: [PATCH] pack-write: skip *.rev work when not writing *.rev","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-08T15:47:30Z","receivedAt":"2021-09-08T15:47:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> On Tue, Sep 07, 2021 at 06:35:10PM -0700, Carlo Arenas wrote:\n>> On Tue, Sep 7, 2021 at 6:10 PM Ævar Arnfjörð Bjarmason wrote:\n>> > + if (!(flags & WRITE_REV) && !(flags & WRITE_REV_VERIFY))\n>> > + return NULL;\n>>\n>> I see this expression matches exactly the logic from 8ef50d9958 which\n>> is why I presume\n>> you used it, but the simpler (and logically equivalent[1]) :\n>>\n>> if !((flags & WRITE_REV) || (flags & WRITE_REV_VERIFY))\n>\n> Even simpler would be:\n>\n> if (!(flags & (WRITE_REV | WRITE_REV_VERIFY)))\n>\n> although with optimization flags other than -O0, it seems that each of\n> these three produce the same result [1], so I don't think that it\n> matters much either way ;-).\n\nIf all result in the same binary, the only deciding factor would be\nhow readable the code is to human readers.\n\nI too find that your version, \"we care about these two bits---if\nflags does not have either of them, then...\", the easiest to follow.\nBut the original is not unreadable and is good enough once it has\nalready been written.\n\nThanks.\n"},{"id":"435116","messageId":"YTjljc0kU2WhmTzU@nand.local","threadId":"55135","inReplyTo":"87r1dziczs.fsf@evledraar.gmail.com","subject":"Re: [PATCH] pack-write: skip *.rev work when not writing *.rev","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-08T16:32:13Z","receivedAt":"2021-09-08T16:32:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 08, 2021 at 12:18:38PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Tue, Sep 07 2021, Taylor Blau wrote:\n>\n> > On Tue, Sep 07, 2021 at 10:50:58PM -0400, Taylor Blau wrote:\n> >> Of the two, I think the former is more appealing (since no other\n> >> functions called by finish_tmp_packfile() are guarded like that; they\n> >> conditionally behave as noops depending on `flags`).\n> >\n> > Sorry; this is nonsensical. The only other function we call is\n> > write_idx_file() which merely changes its behavior based on flags, but\n> > it never behaves as a noop.\n> >\n> > That doesn't change my thinking about preferring the former of my two\n> > suggestions, but just wanted to correct my error.\n>\n> I agree that this code is very confusing overall, but would prefer to\n> wait on refactoring further until the two topics in flight (this and the\n> other pack-write topic) settle.\n\nI'm fine to wait on any further refactorings. And I agree that this code\nis confusing, since when I read it last night I thought that the check\nin write_rev_file_order() was a duplicate of the one you introduced, but\nit is not:\n\n if ((flags & WRITE_REV) && (flags & WRITE_REV_VERIFY))\n die(_(\"cannot both write and verify reverse index\"));\n\nand that check is different than the one you added, which I think is\nappropriate.\n\nSo this patch looks good to me, and sorry for the confusion.\n\nThanks,\nTaylor\n"},{"id":"436259","messageId":"20210917211337.GC2118053@szeder.dev","threadId":"55135","inReplyTo":"87czx1awwg.fsf@evledraar.gmail.com","subject":"Re: [PATCH 3/5] commit-graph: use parse_options_concat()","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-09-17T21:13:37Z","receivedAt":"2021-09-17T21:13:43Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Feb 15, 2021 at 09:39:11PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Feb 15 2021, Taylor Blau wrote:\n> \n> > On Mon, Feb 15, 2021 at 07:41:16PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> >> Make use of the parse_options_concat() so we don't need to copy/paste\n> >> common options like --object-dir. This is inspired by a similar change\n> >> to \"checkout\" in 2087182272\n> >> (checkout: split options[] array in three pieces, 2019-03-29).\n> >>\n> >> A minor behavior change here is that now we're going to list both\n> >> --object-dir and --progress first, before we'd list --progress along\n> >> with other options.\n\nThe final version of this patch that was picked up is at \n\n https://public-inbox.org/git/patch-v4-3.7-32cc0d1c7bc-20210823T122854Z-avarab@gmail.com/\n\nI reply to this old version because of the following pieces of the\ndiscussion:\n\n> > \"Behavior change\" referring only to the output of `git commit-graph -h`,\n> > no?\n> >\n> > Looking at the code (and understanding this whole situation a little bit\n> > better), I'd think that this wouldn't cause us to parse anything\n> > differently before or after this change, right?\n> \n> Indeed, I just mean the \"-h\" or \"--invalid-opt\" output changed in the\n> order we show the options in.\n\n[...]\n\n> but I wanted to just focus on\n> refactoring existing behavior & get rid of the copy/pasted options\n\nNo, there is more behavior change: since 84e4484f12 (commit-graph: use\nparse_options_concat(), 2021-08-23) the 'git commit-graph' command\ndoes accept the '--[no-]progress' options as well, but before that\nonly its subcommands did, and 'git commit-graph --progress ...'\nerrored out with \"unknown option\".\n\nWorse, sometimes 'git commit-graph --progress ...' doesn't work as\nit's supposed to. The patch below descibes the problem and fixes it,\nbut on second thought I don't think that it is the right approach.\n\nIn general, even when all subcommands of a git command understand a\nparticular --option, that does not mean that it's a good idea to teach\nthat option to that git command. E.g. what if we later add another\nsubcommand for which that --option doesn't make any sense? And from\nthe quoted discussion above it seems that teaching 'git commit-graph'\nthe '--progress' option was not intentional at all.\n\nI'm inclined to think that '--progress' should rather be removed from\nthe common 'git commit-graph' options; luckily it's not too late,\nbecause it hasn't been released yet.\n\n\n --- >8 ---\n\nSubject: [PATCH] commit-graph: fix 'git commit-graph --[no-]progress ...'\n\nUntil recenly 'git commit-graph' didn't have a '--progress' option,\nonly its subcommands did, but this changed with 84e4484f12\n(commit-graph: use parse_options_concat(), 2021-08-23), and now the\n'git commit-graph' command accepts the '--[no-]progress' options as\nwell. Alas, they don't always works as they are supposed to, because\nthe isatty(2) check is only performed in the subcommands, i.e. after\nthe \"main\" 'git commit-graph' command has parsed its options, and it\nunconditionally overwrites whatever '--[no-]progress' option might\nhave been given:\n\n $ GIT_PROGRESS_DELAY=0 git commit-graph --no-progress write --reachable\n Collecting referenced commits: 1617, done.\n Loading known commits in commit graph: 100% (1617/1617), done.\n [...]\n $ GIT_PROGRESS_DELAY=0 git commit-graph --progress write 2>out\n $ wc -c out\n 0 out\n\nMove the isatty(2) check to cmd_commit_graph(), before it calls\nparse_options(), so 'git commit-graph --[no-]progress' will be able to\noverride it as well.\n\nSigned-off-by: SZEDER Gábor \n---\n builtin/commit-graph.c | 3 +--\n 1 file changed, 1 insertion(+), 2 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 21fc6e934b..3a873ceaf6 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -101,7 +101,6 @@ static int graph_verify(int argc, const char **argv)\n \n \ttrace2_cmd_mode(\"verify\");\n \n-\topts.progress = isatty(2);\n \targc = parse_options(argc, argv, NULL,\n \t\t\t options,\n \t\t\t builtin_commit_graph_verify_usage, 0);\n@@ -250,7 +249,6 @@ static int graph_write(int argc, const char **argv)\n \t};\n \tstruct option *options = add_common_options(builtin_commit_graph_write_options);\n \n-\topts.progress = isatty(2);\n \topts.enable_changed_paths = -1;\n \twrite_opts.size_multiple = 2;\n \twrite_opts.max_commits = 0;\n@@ -331,6 +329,7 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \tstruct option *builtin_commit_graph_options = common_opts;\n \n \tgit_config(git_default_config, NULL);\n+\topts.progress = isatty(2);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_commit_graph_options,\n \t\t\t builtin_commit_graph_usage,\n-- \n2.33.0.517.ga8dcee0d0a\n\n --- 8< ---\n\n"},{"id":"436272","messageId":"YUUQzswYL5x74Tps@coredump.intra.peff.net","threadId":"55135","inReplyTo":"20210917211337.GC2118053@szeder.dev","subject":"Re: [PATCH 3/5] commit-graph: use parse_options_concat()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-17T22:03:58Z","receivedAt":"2021-09-17T22:04:01Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 17, 2021 at 11:13:37PM +0200, SZEDER Gábor wrote:\n\n> Worse, sometimes 'git commit-graph --progress ...' doesn't work as\n> it's supposed to. The patch below descibes the problem and fixes it,\n> but on second thought I don't think that it is the right approach.\n> \n> In general, even when all subcommands of a git command understand a\n> particular --option, that does not mean that it's a good idea to teach\n> that option to that git command. E.g. what if we later add another\n> subcommand for which that --option doesn't make any sense? And from\n> the quoted discussion above it seems that teaching 'git commit-graph'\n> the '--progress' option was not intentional at all.\n> \n> I'm inclined to think that '--progress' should rather be removed from\n> the common 'git commit-graph' options; luckily it's not too late,\n> because it hasn't been released yet.\n\nI wasn't following this series closely, but having seen your fix below,\nI'm inclined to agree with you. Just because we _can_ allow options\nbefore or after sub-commands does not necessarily make it a good idea.\n\nThere is a distinct meaning to options before/after the command for the\nbase \"git\" command (e.g., \"git -C foo branch\" versus \"git branch -C\nfoo\"), and I think that has been useful overall.\n\n> --- >8 ---\n> \n> Subject: [PATCH] commit-graph: fix 'git commit-graph --[no-]progress ...'\n\nThis patch looks like a sensible fix if we don't simply remove the \"git\ncommit-graph --progress write\" version.\n\n-Peff\n"},{"id":"436290","messageId":"87zgsad6mn.fsf@evledraar.gmail.com","threadId":"55135","inReplyTo":"20210917211337.GC2118053@szeder.dev","subject":"Re: [PATCH 3/5] commit-graph: use parse_options_concat()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-18T00:58:27Z","receivedAt":"2021-09-18T01:17:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Sep 17 2021, SZEDER Gábor wrote:\n\n> On Mon, Feb 15, 2021 at 09:39:11PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Mon, Feb 15 2021, Taylor Blau wrote:\n>> \n>> > On Mon, Feb 15, 2021 at 07:41:16PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> >> Make use of the parse_options_concat() so we don't need to copy/paste\n>> >> common options like --object-dir. This is inspired by a similar change\n>> >> to \"checkout\" in 2087182272\n>> >> (checkout: split options[] array in three pieces, 2019-03-29).\n>> >>\n>> >> A minor behavior change here is that now we're going to list both\n>> >> --object-dir and --progress first, before we'd list --progress along\n>> >> with other options.\n>\n> The final version of this patch that was picked up is at \n>\n> https://public-inbox.org/git/patch-v4-3.7-32cc0d1c7bc-20210823T122854Z-avarab@gmail.com/\n>\n> I reply to this old version because of the following pieces of the\n> discussion:\n>\n>> > \"Behavior change\" referring only to the output of `git commit-graph -h`,\n>> > no?\n>> >\n>> > Looking at the code (and understanding this whole situation a little bit\n>> > better), I'd think that this wouldn't cause us to parse anything\n>> > differently before or after this change, right?\n>> \n>> Indeed, I just mean the \"-h\" or \"--invalid-opt\" output changed in the\n>> order we show the options in.\n>\n> [...]\n>\n>> but I wanted to just focus on\n>> refactoring existing behavior & get rid of the copy/pasted options\n>\n> No, there is more behavior change: since 84e4484f12 (commit-graph: use\n> parse_options_concat(), 2021-08-23) the 'git commit-graph' command\n> does accept the '--[no-]progress' options as well, but before that\n> only its subcommands did, and 'git commit-graph --progress ...'\n> errored out with \"unknown option\".\n>\n> Worse, sometimes 'git commit-graph --progress ...' doesn't work as\n> it's supposed to. The patch below descibes the problem and fixes it,\n> but on second thought I don't think that it is the right approach.\n>\n> In general, even when all subcommands of a git command understand a\n> particular --option, that does not mean that it's a good idea to teach\n> that option to that git command. E.g. what if we later add another\n> subcommand for which that --option doesn't make any sense? And from\n> the quoted discussion above it seems that teaching 'git commit-graph'\n> the '--progress' option was not intentional at all.\n>\n> I'm inclined to think that '--progress' should rather be removed from\n> the common 'git commit-graph' options; luckily it's not too late,\n> because it hasn't been released yet.\n>\n>\n> --- >8 ---\n>\n> Subject: [PATCH] commit-graph: fix 'git commit-graph --[no-]progress ...'\n>\n> Until recenly 'git commit-graph' didn't have a '--progress' option,\n> only its subcommands did, but this changed with 84e4484f12\n> (commit-graph: use parse_options_concat(), 2021-08-23), and now the\n> 'git commit-graph' command accepts the '--[no-]progress' options as\n> well. Alas, they don't always works as they are supposed to, because\n> the isatty(2) check is only performed in the subcommands, i.e. after\n> the \"main\" 'git commit-graph' command has parsed its options, and it\n> unconditionally overwrites whatever '--[no-]progress' option might\n> have been given:\n>\n> $ GIT_PROGRESS_DELAY=0 git commit-graph --no-progress write --reachable\n> Collecting referenced commits: 1617, done.\n> Loading known commits in commit graph: 100% (1617/1617), done.\n> [...]\n> $ GIT_PROGRESS_DELAY=0 git commit-graph --progress write 2>out\n> $ wc -c out\n> 0 out\n>\n> Move the isatty(2) check to cmd_commit_graph(), before it calls\n> parse_options(), so 'git commit-graph --[no-]progress' will be able to\n> override it as well.\n>\n> Signed-off-by: SZEDER Gábor \n> ---\n> builtin/commit-graph.c | 3 +--\n> 1 file changed, 1 insertion(+), 2 deletions(-)\n>\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index 21fc6e934b..3a873ceaf6 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -101,7 +101,6 @@ static int graph_verify(int argc, const char **argv)\n> \n> \ttrace2_cmd_mode(\"verify\");\n> \n> -\topts.progress = isatty(2);\n> \targc = parse_options(argc, argv, NULL,\n> \t\t\t options,\n> \t\t\t builtin_commit_graph_verify_usage, 0);\n> @@ -250,7 +249,6 @@ static int graph_write(int argc, const char **argv)\n> \t};\n> \tstruct option *options = add_common_options(builtin_commit_graph_write_options);\n> \n> -\topts.progress = isatty(2);\n> \topts.enable_changed_paths = -1;\n> \twrite_opts.size_multiple = 2;\n> \twrite_opts.max_commits = 0;\n> @@ -331,6 +329,7 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> \tstruct option *builtin_commit_graph_options = common_opts;\n> \n> \tgit_config(git_default_config, NULL);\n> +\topts.progress = isatty(2);\n> \targc = parse_options(argc, argv, prefix,\n> \t\t\t builtin_commit_graph_options,\n> \t\t\t builtin_commit_graph_usage,\n\nYes, this was unintentional on my part, sorry, and thanks for cleaning\nup my mess.\n\nHowever, I have wondered how we should be dealing with these\nsub-commands in general.\n\nIn the case of commit-graph we've always documented it at the top-level\nas OPTIONS, so even though the usage shows:\n\n git commit-graph write \n\nWe've always accepted \"--object-dir\" after \"git commit-graph\", and all\nthe other options are documented in their per-subcommand sections.\n\nSo just from reading the documentation you might think that this (with\nyour fix here) is intentional behavior, and we should just fix the\nsynopsis.\n\nThen we have the more recent multi-pack-index which *is* documented as:\n\n 'git multi-pack-index' [--object-dir=] [--[no-]progress]\n [--preferred-pack=] \n\nSo actually, the reason this crept in is probably because I was copying\nthe pattern we've had there since 60ca94769ce\n(builtin/multi-pack-index.c: split sub-commands, 2021-03-30), my commit\nmessage says as much.\n\nGiven that and multi-pack-index's documented behavior I think that it\nprobably makes sense to keep and document this, and as a follow-up\n(which I or Taylor could do) change the synopsis accordingly.\n\nAside from whatever bugs have crept or existing behavior, I think it\nmakes sense as UI to do things like:\n\n git commit-graph --object-dir= write --reachable\n git commit-graph --progress write\n git commit-graph --progress verify\n\netc., as --progress is a not-subcommand-specific option, not really. We\nmight have a subcommand that doesn't have progress output, but I still\nthink it makes sense to have it in that position, maybe we'll end up\nadding it later.\n\nBrian and I also had a discussion back in April[1] about\n--object-format, i.e. should we be making every single command support:\n\n git hash-object --object-format=sha256\n\nOr (as I suggested) doesn't it make more sense to do:\n\n git --object-format=sha256 hash-object\n\nLike the --progress option it does mean that you'll end up with commands\nfor whom that'll just be ignored:\n\n git --object-format=sha256 version\n\nBut that's conceptually similar to repo settings, and I don't think it's\nconfusing, the same can be said about e.g.:\n\n git -c this.doesNotUse=thisConfig version\n\nHaving said that for --progress it probably makes sense to eventually\nhave:\n\n git --progress commit-graph write\n\nI.e. maybe we'd want a top-level option for it, given how many commands\nhave that option and us needing to pass a \"do_progress\" flag all over\nthe place.\n\nOf course we'd need to (silently or not) support it also as:\n\n git commit-graph --progress write\n git commit-graph write --progress\n\nWhich is the case here.\n\n1. https://lore.kernel.org/git/8735vq2l8a.fsf@evledraar.gmail.com/\n"},{"id":"436292","messageId":"YUVreesWdRmBYl1C@nand.local","threadId":"55135","inReplyTo":"YUUQzswYL5x74Tps@coredump.intra.peff.net","subject":"Re: [PATCH 3/5] commit-graph: use parse_options_concat()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-18T04:30:49Z","receivedAt":"2021-09-18T04:30:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Sep 17, 2021 at 06:03:58PM -0400, Jeff King wrote:\n> > I'm inclined to think that '--progress' should rather be removed from\n> > the common 'git commit-graph' options; luckily it's not too late,\n> > because it hasn't been released yet.\n>\n> I wasn't following this series closely, but having seen your fix below,\n> I'm inclined to agree with you. Just because we _can_ allow options\n> before or after sub-commands does not necessarily make it a good idea.\n>\nI agree. Suppose we had a \"git commit-graph remove\" sub-command that\nremoved the commit-graph file (ignoring that there are probably better\nhypothetical examples than this ;)). It's not obvious what --progress\nmeans in the context of that mode.\n\nHere's a patch that does what you and Gábor are suggesting as an\nalternative. Unfortunately, we can't do the same for the\nmulti-pack-index command, since the analogous change there is 60ca94769c\n(builtin/multi-pack-index.c: split sub-commands, 2021-03-30), which was\nreleased in 2.32.\n\nAnyway, as promised:\n\n--- 8< ---\n\nSubject: [PATCH] builtin/commit-graph.c: don't accept common --[no-]progress\n\nIn 84e4484f12 (commit-graph: use parse_options_concat(), 2021-08-23) we\nunified common options of commit-graph's subcommands into a single\n\"common_opts\" array.\n\nBut 84e4484f12 introduced a behavior change which is to accept the\n\"--[no-]progress\" option before any sub-commands, e.g.,\n\n git commit-graph --progress write ...\n\nPrior to that commit, the above would error out with \"unknown option\".\n\nThere are two issues with this behavior change. First is that the\ntop-level --[no-]progress is not always respected. This is because\nisatty(2) is performed in the sub-commands, which unconditionally\noverwrites any --[no-]progress that was given at the top-level.\n\nBut the second issue is that the existing sub-commands of commit-graph\nonly happen to both have a sensible interpretation of what `--progress`\nor `--no-progress` means. If we ever added a sub-command which didn't\nhave a notion of progress, we would be forced to ignore the top-level\n`--[no-]progress` altogether.\n\nSince we haven't released a version of Git that supports --[no-]progress\nas a top-level option for `git commit-graph`, let's remove it.\n\nSuggested-by: SZEDER Gábor \nSigned-off-by: Taylor Blau \n---\n builtin/commit-graph.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 21fc6e934b..067587a0fd 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -50,8 +50,6 @@ static struct option common_opts[] = {\n \tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n \t\t N_(\"dir\"),\n \t\t N_(\"the object directory to store the graph\")),\n-\tOPT_BOOL(0, \"progress\", &opts.progress,\n-\t\t N_(\"force progress reporting\")),\n \tOPT_END()\n };\n\n@@ -95,6 +93,8 @@ static int graph_verify(int argc, const char **argv)\n \tstatic struct option builtin_commit_graph_verify_options[] = {\n \t\tOPT_BOOL(0, \"shallow\", &opts.shallow,\n \t\t\t N_(\"if the commit-graph is split, only verify the tip file\")),\n+\t\tOPT_BOOL(0, \"progress\", &opts.progress,\n+\t\t\t N_(\"force progress reporting\")),\n \t\tOPT_END(),\n \t};\n \tstruct option *options = add_common_options(builtin_commit_graph_verify_options);\n@@ -246,6 +246,8 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_CALLBACK_F(0, \"max-new-filters\", &write_opts.max_new_filters,\n \t\t\tNULL, N_(\"maximum number of changed-path Bloom filters to compute\"),\n \t\t\t0, write_option_max_new_filters),\n+\t\tOPT_BOOL(0, \"progress\", &opts.progress,\n+\t\t\t N_(\"force progress reporting\")),\n \t\tOPT_END(),\n \t};\n \tstruct option *options = add_common_options(builtin_commit_graph_write_options);\n--\n2.33.0.96.g73915697e6\n\n"},{"id":"436296","messageId":"87v92ycpil.fsf@evledraar.gmail.com","threadId":"55135","inReplyTo":"YUVreesWdRmBYl1C@nand.local","subject":"Re: [PATCH 3/5] commit-graph: use parse_options_concat()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-18T07:20:38Z","receivedAt":"2021-09-18T07:27:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Sep 18 2021, Taylor Blau wrote:\n\n> On Fri, Sep 17, 2021 at 06:03:58PM -0400, Jeff King wrote:\n>> > I'm inclined to think that '--progress' should rather be removed from\n>> > the common 'git commit-graph' options; luckily it's not too late,\n>> > because it hasn't been released yet.\n>>\n>> I wasn't following this series closely, but having seen your fix below,\n>> I'm inclined to agree with you. Just because we _can_ allow options\n>> before or after sub-commands does not necessarily make it a good idea.\n>>\n> I agree. Suppose we had a \"git commit-graph remove\" sub-command that\n> removed the commit-graph file (ignoring that there are probably better\n> hypothetical examples than this ;)). It's not obvious what --progress\n> means in the context of that mode.\n\nWell, you might have as many as tens of commit-graph files, and could be\nrunning this on AIX where I/O is apparently implemented in terms of\ncarrier pigeon messaging judging by how slow it is :)\n\nBut as argued in\nhttps://lore.kernel.org/git/87zgsad6mn.fsf@evledraar.gmail.com/ I don't\nsee how it's going to be any more confusing to user than \"git -c foo=bar\nversion\" (the -c does nothing there)>\n\n> Here's a patch that does what you and Gábor are suggesting as an\n> alternative. Unfortunately, we can't do the same for the\n> multi-pack-index command, since the analogous change there is 60ca94769c\n> (builtin/multi-pack-index.c: split sub-commands, 2021-03-30), which was\n> released in 2.32.\n\nIf we came up with some call about what we want subcommands in general\nto look like I'd think it would be fine to convert multi-pack-index to\nit, perhaps with some deprecation period where it would issue a\nwarning() while it understood both forms.\n\n> Anyway, as promised:\n>\n> --- 8< ---\n>\n> Subject: [PATCH] builtin/commit-graph.c: don't accept common --[no-]progress\n>\n> In 84e4484f12 (commit-graph: use parse_options_concat(), 2021-08-23) we\n> unified common options of commit-graph's subcommands into a single\n> \"common_opts\" array.\n>\n> But 84e4484f12 introduced a behavior change which is to accept the\n> \"--[no-]progress\" option before any sub-commands, e.g.,\n>\n> git commit-graph --progress write ...\n>\n> Prior to that commit, the above would error out with \"unknown option\".\n>\n> There are two issues with this behavior change. First is that the\n> top-level --[no-]progress is not always respected. This is because\n> isatty(2) is performed in the sub-commands, which unconditionally\n> overwrites any --[no-]progress that was given at the top-level.\n>\n> But the second issue is that the existing sub-commands of commit-graph\n> only happen to both have a sensible interpretation of what `--progress`\n> or `--no-progress` means. If we ever added a sub-command which didn't\n> have a notion of progress, we would be forced to ignore the top-level\n> `--[no-]progress` altogether.\n>\n> Since we haven't released a version of Git that supports --[no-]progress\n> as a top-level option for `git commit-graph`, let's remove it.\n>\n> Suggested-by: SZEDER Gábor \n> Signed-off-by: Taylor Blau \n> ---\n> builtin/commit-graph.c | 6 ++++--\n> 1 file changed, 4 insertions(+), 2 deletions(-)\n>\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index 21fc6e934b..067587a0fd 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -50,8 +50,6 @@ static struct option common_opts[] = {\n> \tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n> \t\t N_(\"dir\"),\n> \t\t N_(\"the object directory to store the graph\")),\n> -\tOPT_BOOL(0, \"progress\", &opts.progress,\n> -\t\t N_(\"force progress reporting\")),\n> \tOPT_END()\n> };\n>\n> @@ -95,6 +93,8 @@ static int graph_verify(int argc, const char **argv)\n> \tstatic struct option builtin_commit_graph_verify_options[] = {\n> \t\tOPT_BOOL(0, \"shallow\", &opts.shallow,\n> \t\t\t N_(\"if the commit-graph is split, only verify the tip file\")),\n> +\t\tOPT_BOOL(0, \"progress\", &opts.progress,\n> +\t\t\t N_(\"force progress reporting\")),\n> \t\tOPT_END(),\n> \t};\n> \tstruct option *options = add_common_options(builtin_commit_graph_verify_options);\n> @@ -246,6 +246,8 @@ static int graph_write(int argc, const char **argv)\n> \t\tOPT_CALLBACK_F(0, \"max-new-filters\", &write_opts.max_new_filters,\n> \t\t\tNULL, N_(\"maximum number of changed-path Bloom filters to compute\"),\n> \t\t\t0, write_option_max_new_filters),\n> +\t\tOPT_BOOL(0, \"progress\", &opts.progress,\n> +\t\t\t N_(\"force progress reporting\")),\n> \t\tOPT_END(),\n> \t};\n> \tstruct option *options = add_common_options(builtin_commit_graph_write_options);\n\nThis is a good change, but if you're up for bonus points leaves the docs\nin an odd where we (as noted in [1]) document the --object-dir and\n--progress options under OPTIONS, but now only take the former before\nthe sub-command.\n\n1. https://lore.kernel.org/git/87zgsad6mn.fsf@evledraar.gmail.com/\n"},{"id":"436310","messageId":"YUYMINltnUlM8Ziv@nand.local","threadId":"55135","inReplyTo":"87v92ycpil.fsf@evledraar.gmail.com","subject":"Re: [PATCH 3/5] commit-graph: use parse_options_concat()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-18T15:56:16Z","receivedAt":"2021-09-18T15:56:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Sep 18, 2021 at 09:20:38AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> If we came up with some call about what we want subcommands in general\n> to look like I'd think it would be fine to convert multi-pack-index to\n> it, perhaps with some deprecation period where it would issue a\n> warning() while it understood both forms.\n\nI'm not sure about what to do with the multi-pack-index command.\nProbably going through a deprecation process makes the most sense if we\ndo want to get rid of the top-level `--[no-]progress` there, too. But\nlet's have the discussion elsewhere and not buried in the MIDX\nbitmaps thread ;).\n\n> This is a good change, but if you're up for bonus points leaves the docs\n> in an odd where we (as noted in [1]) document the --object-dir and\n> --progress options under OPTIONS, but now only take the former before\n> the sub-command.\n\nThanks for noticing. I got up and did something in between writing and\nsending this patch, and had a nagging feeling of forgetting something\nbefore I sent. But I couldn't figure out what ;).\n\nI'll resubmit this patch (with the doc changes squashed in) as a new\nthread on the list so we can have the discussion not buried in another\nthread.\n\nThanks,\nTaylor\n"},{"id":"436311","messageId":"YUYMknppMEg9Px/S@nand.local","threadId":"55135","inReplyTo":"YUYMINltnUlM8Ziv@nand.local","subject":"Re: [PATCH 3/5] commit-graph: use parse_options_concat()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-18T15:58:10Z","receivedAt":"2021-09-18T15:58:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Sep 18, 2021 at 11:56:16AM -0400, Taylor Blau wrote:\n> > This is a good change, but if you're up for bonus points leaves the docs\n> > in an odd where we (as noted in [1]) document the --object-dir and\n> > --progress options under OPTIONS, but now only take the former before\n> > the sub-command.\n>\n> Thanks for noticing. I got up and did something in between writing and\n> sending this patch, and had a nagging feeling of forgetting something\n> before I sent. But I couldn't figure out what ;).\n\nActually, I stand by the original patch. Yes, the top-level OPTIONS of\ngit-commit-graph(1) mentions `--[no-]progress`, but the synopsis makes\nclear that those are only accepted after the sub-commands.\n\nSo I think it's fine as-is.\n\nThanks,\nTaylor\n"}]}