{"thread":{"id":"54962","subject":"[PATCH 0/8] pack-revindex: introduce on-disk '.rev' format","startedAt":"2021-01-08T18:20:42Z","lastAt":"2021-01-30T09:08:30Z","messageCount":82,"participants":["Taylor Blau","Derrick Stolee","Ævar Arnfjörð Bjarmason","Junio C Hamano","Jeff King","Eric Sunshine"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"413853","messageId":"cover.1610129989.git.me@ttaylorr.com","threadId":"54962","inReplyTo":null,"subject":"[PATCH 0/8] pack-revindex: introduce on-disk '.rev' format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:19:52Z","receivedAt":"2021-01-08T18:20:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi,\n\nThis is the second of two series to implement support for an on-disk format for\nstoring the reverse index. (It depends on the patches in the previous series\n[1]).\n\nThe format is described in the first patch, but it is roughly as follows:\n\n - It begins with a 12-byte header, containing a magic string, a version\n identifier, and a hash function identifier.\n\n - It then contains a 4 * N (where 'N' is the number of objects) table of index\n positions, sorted by each object's offset within the corresponding packfile.\n\n - Finally, a trailer contains a checksum of the corresponding packfile, and a\n checksum of the above contents.\n\nSince this is a large change, a new 'pack.writeReverseIndex' option is\nintroduced, which defaults to 'false'. When false, `*.rev` files are not\nwritten, and Git gracefully falls back to generate each reverse index in\nmemory. This could optionally be tied to the \"feature.experimental\" option, and\neventually the defalt changed to 'true' in a couple of releases.\n\nTo test these new changes, the test suite now understands\n'GIT_TEST_WRITE_REV_INDEX' to mean that 'pack.writeReverseIndex' should be\n'true' everywhere. Some minor test fall-out is addressed in the sixth patch\nbefore enabling this new mode in the seventh patch.\n\nOne option that is _not_ persued in this series is to store the (pack) offset of\neach object in the `.rev` file. This would at worst triple the size of the file\n(by having to store an additional eight bytes per entry), and add complexity\n(like storing an extended offset table as in the `*.idx` format). An extensive\ndiscussion about why this option was not persued can be found in the first\npatch.\n\nThanks in advance for your review.\n\n[1]: https://lore.kernel.org/git/cover.1610129796.git.me@ttaylorr.com/\n\nTaylor Blau (8):\n packfile: prepare for the existence of '*.rev' files\n pack-write.c: prepare to write 'pack-*.rev' files\n builtin/index-pack.c: write reverse indexes\n builtin/pack-objects.c: respect 'pack.writeReverseIndex'\n Documentation/config/pack.txt: advertise 'pack.writeReverseIndex'\n t: prepare for GIT_TEST_WRITE_REV_INDEX\n t: support GIT_TEST_WRITE_REV_INDEX\n pack-revindex: ensure that on-disk reverse indexes are given\n precedence\n\n Documentation/config/pack.txt | 7 ++\n Documentation/git-index-pack.txt | 20 ++--\n Documentation/technical/pack-format.txt | 17 ++++\n builtin/index-pack.c | 67 +++++++++++--\n builtin/pack-objects.c | 9 ++\n builtin/repack.c | 1 +\n object-store.h | 3 +\n pack-revindex.c | 116 ++++++++++++++++++++--\n pack-revindex.h | 3 +\n pack-write.c | 123 +++++++++++++++++++++++-\n pack.h | 4 +\n packfile.c | 13 ++-\n packfile.h | 1 +\n t/README | 3 +\n t/t5319-multi-pack-index.sh | 2 +-\n t/t5325-reverse-index.sh | 94 ++++++++++++++++++\n t/t5604-clone-reference.sh | 2 +-\n t/t5702-protocol-v2.sh | 4 +-\n t/t6500-gc.sh | 4 +-\n t/t9300-fast-import.sh | 2 +-\n tmp-objdir.c | 4 +-\n 21 files changed, 463 insertions(+), 36 deletions(-)\n create mode 100755 t/t5325-reverse-index.sh\n\n-- \n2.30.0.138.g6d7191ea01\n"},{"id":"413854","messageId":"88393e266292b022fba3ff41f6623486cb41f66d.1610129989.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1610129989.git.me@ttaylorr.com","subject":"[PATCH 2/8] pack-write.c: prepare to write 'pack-*.rev' files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:20:02Z","receivedAt":"2021-01-08T18:20:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This patch prepares for callers to be able to write reverse index files\nto disk.\n\nIt adds the necessary machinery to write a format-compliant .rev file\nfrom within 'write_rev_file()', which is called from\n'finish_tmp_packfile()'.\n\nSigned-off-by: Taylor Blau \n---\n pack-write.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++-\n pack.h | 4 ++\n 2 files changed, 126 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-write.c b/pack-write.c\nindex 3513665e1e..68db5a9edf 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -166,6 +166,116 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec\n \treturn index_name;\n }\n \n+static int pack_order_cmp(const void *va, const void *vb, void *ctx)\n+{\n+\tstruct pack_idx_entry **objects = ctx;\n+\n+\toff_t oa = objects[*(uint32_t*)va]->offset;\n+\toff_t ob = objects[*(uint32_t*)vb]->offset;\n+\n+\tif (oa < ob)\n+\t\treturn -1;\n+\tif (oa > ob)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+#define RIDX_SIGNATURE 0x52494458 /* \"RIDX\" */\n+#define RIDX_VERSION 1\n+\n+static void write_rev_header(struct hashfile *f)\n+{\n+\tuint32_t oid_version;\n+\tswitch (hash_algo_by_ptr(the_hash_algo)) {\n+\tcase GIT_HASH_SHA1:\n+\t\toid_version = 1;\n+\t\tbreak;\n+\tcase GIT_HASH_SHA256:\n+\t\toid_version = 2;\n+\t\tbreak;\n+\tdefault:\n+\t\tdie(\"write_rev_header: unknown hash version\");\n+\t}\n+\n+\thashwrite_be32(f, RIDX_SIGNATURE);\n+\thashwrite_be32(f, RIDX_VERSION);\n+\thashwrite_be32(f, oid_version);\n+}\n+\n+static void write_rev_index_positions(struct hashfile *f,\n+\t\t\t\t struct pack_idx_entry **objects,\n+\t\t\t\t uint32_t nr_objects)\n+{\n+\tuint32_t *pack_order;\n+\tuint32_t i;\n+\n+\tALLOC_ARRAY(pack_order, nr_objects);\n+\tfor (i = 0; i < nr_objects; i++)\n+\t\tpack_order[i] = i;\n+\tQSORT_S(pack_order, nr_objects, pack_order_cmp, objects);\n+\n+\tfor (i = 0; i < nr_objects; i++)\n+\t\thashwrite_be32(f, pack_order[i]);\n+\n+\tfree(pack_order);\n+}\n+\n+static void write_rev_trailer(struct hashfile *f, const unsigned char *hash)\n+{\n+\thashwrite(f, hash, the_hash_algo->rawsz);\n+}\n+\n+const char *write_rev_file(const char *rev_name,\n+\t\t\t struct pack_idx_entry **objects,\n+\t\t\t uint32_t nr_objects,\n+\t\t\t const unsigned char *hash,\n+\t\t\t unsigned flags)\n+{\n+\tstruct hashfile *f;\n+\tint fd;\n+\n+\tif ((flags & WRITE_REV) && (flags & WRITE_REV_VERIFY))\n+\t\tdie(_(\"cannot both write and verify reverse index\"));\n+\n+\tif (flags & WRITE_REV) {\n+\t\tif (!rev_name) {\n+\t\t\tstruct strbuf tmp_file = STRBUF_INIT;\n+\t\t\tfd = odb_mkstemp(&tmp_file, \"pack/tmp_rev_XXXXXX\");\n+\t\t\trev_name = strbuf_detach(&tmp_file, NULL);\n+\t\t} else {\n+\t\t\tunlink(rev_name);\n+\t\t\tfd = open(rev_name, O_CREAT|O_EXCL|O_WRONLY, 0600);\n+\t\t\tif (fd < 0)\n+\t\t\t\tdie_errno(\"unable to create '%s'\", rev_name);\n+\t\t}\n+\t\tf = hashfd(fd, rev_name);\n+\t} else if (flags & WRITE_REV_VERIFY) {\n+\t\tstruct stat statbuf;\n+\t\tif (stat(rev_name, &statbuf)) {\n+\t\t\tif (errno == ENOENT) {\n+\t\t\t\t/* .rev files are optional */\n+\t\t\t\treturn NULL;\n+\t\t\t} else\n+\t\t\t\tdie_errno(_(\"could not stat: %s\"), rev_name);\n+\t\t}\n+\t\tf = hashfd_check(rev_name);\n+\t} else\n+\t\treturn NULL;\n+\n+\twrite_rev_header(f);\n+\n+\twrite_rev_index_positions(f, objects, nr_objects);\n+\twrite_rev_trailer(f, hash);\n+\n+\tif (rev_name && adjust_shared_perm(rev_name) < 0)\n+\t\tdie(_(\"failed to make %s readable\"), rev_name);\n+\n+\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |\n+\t\t\t\t ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));\n+\n+\treturn rev_name;\n+}\n+\n off_t write_pack_header(struct hashfile *f, uint32_t nr_entries)\n {\n \tstruct pack_header hdr;\n@@ -341,7 +451,7 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \t\t\t struct pack_idx_option *pack_idx_opts,\n \t\t\t unsigned char hash[])\n {\n-\tconst char *idx_tmp_name;\n+\tconst char *idx_tmp_name, *rev_tmp_name = NULL;\n \tint basename_len = name_buffer->len;\n \n \tif (adjust_shared_perm(pack_tmp_name))\n@@ -352,6 +462,9 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \tif (adjust_shared_perm(idx_tmp_name))\n \t\tdie_errno(\"unable to make temporary index file readable\");\n \n+\trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n+\t\t\t\t pack_idx_opts->flags);\n+\n \tstrbuf_addf(name_buffer, \"%s.pack\", hash_to_hex(hash));\n \n \tif (rename(pack_tmp_name, name_buffer->buf))\n@@ -365,5 +478,13 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \n \tstrbuf_setlen(name_buffer, basename_len);\n \n+\tif (rev_tmp_name) {\n+\t\tstrbuf_addf(name_buffer, \"%s.rev\", hash_to_hex(hash));\n+\t\tif (rename(rev_tmp_name, name_buffer->buf))\n+\t\t\tdie_errno(\"unable to rename temporary reverse-index file\");\n+\t}\n+\n+\tstrbuf_setlen(name_buffer, basename_len);\n+\n \tfree((void *)idx_tmp_name);\n }\ndiff --git a/pack.h b/pack.h\nindex 9fc0945ac9..30439e0784 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -42,6 +42,8 @@ struct pack_idx_option {\n \t/* flag bits */\n #define WRITE_IDX_VERIFY 01 /* verify only, do not write the idx file */\n #define WRITE_IDX_STRICT 02\n+#define WRITE_REV 04\n+#define WRITE_REV_VERIFY 010\n \n \tuint32_t version;\n \tuint32_t off32_limit;\n@@ -87,6 +89,8 @@ off_t write_pack_header(struct hashfile *f, uint32_t);\n void fixup_pack_header_footer(int, unsigned char *, const char *, uint32_t, unsigned char *, off_t);\n char *index_pack_lockfile(int fd);\n \n+const char *write_rev_file(const char *rev_name, struct pack_idx_entry **objects, uint32_t nr_objects, const unsigned char *hash, unsigned flags);\n+\n /*\n * The \"hdr\" output buffer should be at least this big, which will handle sizes\n * up to 2^67.\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413855","messageId":"6e580d43d1d1e45bfe58606b98c5034a342d8241.1610129989.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1610129989.git.me@ttaylorr.com","subject":"[PATCH 8/8] pack-revindex: ensure that on-disk reverse indexes are given precedence","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:20:29Z","receivedAt":"2021-01-08T18:20:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When an on-disk reverse index exists, there is no need to generate one\nin memory. In fact, doing so can be slow, and require large amounts of\nthe heap.\n\nLet's make sure that we treat the on-disk reverse index with precedence\n(i.e., that when it exists, we don't bother trying to generate an\nequivalent one in memory) by teaching Git how to conditionally die()\nwhen generating a reverse index in memory.\n\nThen, add a test to ensure that when (a) an on-disk reverse index\nexists, and (b) when setting GIT_TEST_REV_INDEX_DIE_IN_MEMORY, that we\ndo not die, implying that we read from the on-disk one.\n\nSigned-off-by: Taylor Blau \n---\n pack-revindex.c | 4 ++++\n pack-revindex.h | 1 +\n t/t5325-reverse-index.sh | 8 ++++++++\n 3 files changed, 13 insertions(+)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 1baaf2c42a..683d22e898 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -2,6 +2,7 @@\n #include \"pack-revindex.h\"\n #include \"object-store.h\"\n #include \"packfile.h\"\n+#include \"config.h\"\n \n struct revindex_entry {\n \toff_t offset;\n@@ -166,6 +167,9 @@ static void create_pack_revindex(struct packed_git *p)\n \n static int load_pack_revindex_from_memory(struct packed_git *p)\n {\n+\tif (git_env_bool(GIT_TEST_REV_INDEX_DIE_IN_MEMORY, 0))\n+\t\tdie(\"dying as requested by '%s'\",\n+\t\t GIT_TEST_REV_INDEX_DIE_IN_MEMORY);\n \tif (open_pack_index(p))\n \t\treturn -1;\n \tcreate_pack_revindex(p);\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex d2d466e298..e271da871a 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -2,6 +2,7 @@\n #define PACK_REVINDEX_H\n \n #define GIT_TEST_WRITE_REV_INDEX \"GIT_TEST_WRITE_REV_INDEX\"\n+#define GIT_TEST_REV_INDEX_DIE_IN_MEMORY \"GIT_TEST_REV_INDEX_DIE_IN_MEMORY\"\n \n struct packed_git;\n \ndiff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh\nindex 5a59cc71e4..9d4eecccc9 100755\n--- a/t/t5325-reverse-index.sh\n+++ b/t/t5325-reverse-index.sh\n@@ -83,4 +83,12 @@ test_expect_success 'pack-objects respects pack.writeReverseIndex' '\n \ttest_path_is_file pack-1-*.rev\n '\n \n+test_expect_success 'reverse index is not generated when available on disk' '\n+\tgit index-pack --rev-index $packdir/pack-$pack.pack &&\n+\n+\tgit rev-parse HEAD >tip &&\n+\tGIT_TEST_REV_INDEX_DIE_IN_MEMORY=1 git cat-file \\\n+\t\t--batch-check=\"%(objectsize:disk)\" \n---\n Documentation/git-index-pack.txt | 20 ++++++---\n builtin/index-pack.c | 64 ++++++++++++++++++++++++-----\n t/t5325-reverse-index.sh | 69 ++++++++++++++++++++++++++++++++\n 3 files changed, 137 insertions(+), 16 deletions(-)\n create mode 100755 t/t5325-reverse-index.sh\n\ndiff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt\nindex af0c26232c..b65f380269 100644\n--- a/Documentation/git-index-pack.txt\n+++ b/Documentation/git-index-pack.txt\n@@ -9,17 +9,18 @@ git-index-pack - Build pack index file for an existing packed archive\n SYNOPSIS\n --------\n [verse]\n-'git index-pack' [-v] [-o ] \n+'git index-pack' [-v] [-o ] [--[no-]rev-index] \n 'git index-pack' --stdin [--fix-thin] [--keep] [-v] [-o ]\n- []\n+\t\t [--[no-]rev-index] []\n \n \n DESCRIPTION\n -----------\n Reads a packed archive (.pack) from the specified file, and\n-builds a pack index file (.idx) for it. The packed archive\n-together with the pack index can then be placed in the\n-objects/pack/ directory of a Git repository.\n+builds a pack index file (.idx) for it. Optionally writes a\n+reverse-index (.rev) for the specified pack. The packed\n+archive together with the pack index can then be placed in\n+the objects/pack/ directory of a Git repository.\n \n \n OPTIONS\n@@ -33,7 +34,14 @@ OPTIONS\n \tfile is constructed from the name of packed archive\n \tfile by replacing .pack with .idx (and the program\n \tfails if the name of packed archive does not end\n-\twith .pack).\n+\twith .pack). Incompatible with `--rev-index`.\n+\n+--[no-]rev-index::\n+\tWhen this flag is provided, generate a reverse index\n+\t(a `.rev` file) corresponding to the given pack. If\n+\t`--verify` is given, ensure that the existing\n+\treverse index is correct. Takes precedence over\n+\t`pack.writeReverseIndex`.\n \n --stdin::\n \tWhen this flag is provided, the pack is read from stdin\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 4b8d86e0ad..03408250b1 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -17,7 +17,7 @@\n #include \"promisor-remote.h\"\n \n static const char index_pack_usage[] =\n-\"git index-pack [-v] [-o ] [--keep | --keep=] [--verify] [--strict] ( | --stdin [--fix-thin] [])\";\n+\"git index-pack [-v] [-o ] [--keep | --keep=] [--[no-]rev-index] [--verify] [--strict] ( | --stdin [--fix-thin] [])\";\n \n struct object_entry {\n \tstruct pack_idx_entry idx;\n@@ -1436,13 +1436,13 @@ static void fix_unresolved_deltas(struct hashfile *f)\n \tfree(sorted_by_pos);\n }\n \n-static const char *derive_filename(const char *pack_name, const char *suffix,\n-\t\t\t\t struct strbuf *buf)\n+static const char *derive_filename(const char *pack_name, const char *strip,\n+\t\t\t\t const char *suffix, struct strbuf *buf)\n {\n \tsize_t len;\n-\tif (!strip_suffix(pack_name, \".pack\", &len))\n-\t\tdie(_(\"packfile name '%s' does not end with '.pack'\"),\n-\t\t pack_name);\n+\tif (!strip_suffix(pack_name, strip, &len))\n+\t\tdie(_(\"packfile name '%s' does not end with '%s'\"),\n+\t\t pack_name, strip);\n \tstrbuf_add(buf, pack_name, len);\n \tstrbuf_addch(buf, '.');\n \tstrbuf_addstr(buf, suffix);\n@@ -1459,7 +1459,7 @@ static void write_special_file(const char *suffix, const char *msg,\n \tint msg_len = strlen(msg);\n \n \tif (pack_name)\n-\t\tfilename = derive_filename(pack_name, suffix, &name_buf);\n+\t\tfilename = derive_filename(pack_name, \".pack\", suffix, &name_buf);\n \telse\n \t\tfilename = odb_pack_name(&name_buf, hash, suffix);\n \n@@ -1484,12 +1484,14 @@ static void write_special_file(const char *suffix, const char *msg,\n \n static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t const char *final_index_name, const char *curr_index_name,\n+\t\t const char *final_rev_index_name, const char *curr_rev_index_name,\n \t\t const char *keep_msg, const char *promisor_msg,\n \t\t unsigned char *hash)\n {\n \tconst char *report = \"pack\";\n \tstruct strbuf pack_name = STRBUF_INIT;\n \tstruct strbuf index_name = STRBUF_INIT;\n+\tstruct strbuf rev_index_name = STRBUF_INIT;\n \tint err;\n \n \tif (!from_stdin) {\n@@ -1524,6 +1526,16 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t} else\n \t\tchmod(final_index_name, 0444);\n \n+\tif (curr_rev_index_name) {\n+\t\tif (final_rev_index_name != curr_rev_index_name) {\n+\t\t\tif (!final_rev_index_name)\n+\t\t\t\tfinal_rev_index_name = odb_pack_name(&rev_index_name, hash, \"rev\");\n+\t\t\tif (finalize_object_file(curr_rev_index_name, final_rev_index_name))\n+\t\t\t\tdie(_(\"cannot store reverse index file\"));\n+\t\t} else\n+\t\t\tchmod(final_rev_index_name, 0444);\n+\t}\n+\n \tif (do_fsck_object) {\n \t\tstruct packed_git *p;\n \t\tp = add_packed_git(final_index_name, strlen(final_index_name), 0);\n@@ -1553,6 +1565,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t}\n \t}\n \n+\tstrbuf_release(&rev_index_name);\n \tstrbuf_release(&index_name);\n \tstrbuf_release(&pack_name);\n }\n@@ -1578,6 +1591,12 @@ static int git_index_pack_config(const char *k, const char *v, void *cb)\n \t\t}\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"pack.writereverseindex\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\topts->flags |= WRITE_REV;\n+\t\telse\n+\t\t\topts->flags &= ~WRITE_REV;\n+\t}\n \treturn git_default_config(k, v, cb);\n }\n \n@@ -1695,12 +1714,14 @@ static void show_pack_info(int stat_only)\n \n int cmd_index_pack(int argc, const char **argv, const char *prefix)\n {\n-\tint i, fix_thin_pack = 0, verify = 0, stat_only = 0;\n+\tint i, fix_thin_pack = 0, verify = 0, stat_only = 0, rev_index;\n \tconst char *curr_index;\n-\tconst char *index_name = NULL, *pack_name = NULL;\n+\tconst char *curr_rev_index = NULL;\n+\tconst char *index_name = NULL, *pack_name = NULL, *rev_index_name = NULL;\n \tconst char *keep_msg = NULL;\n \tconst char *promisor_msg = NULL;\n \tstruct strbuf index_name_buf = STRBUF_INIT;\n+\tstruct strbuf rev_index_name_buf = STRBUF_INIT;\n \tstruct pack_idx_entry **idx_objects;\n \tstruct pack_idx_option opts;\n \tunsigned char pack_hash[GIT_MAX_RAWSZ];\n@@ -1727,6 +1748,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (prefix && chdir(prefix))\n \t\tdie(_(\"Cannot come back to cwd\"));\n \n+\trev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV));\n+\n \tfor (i = 1; i < argc; i++) {\n \t\tconst char *arg = argv[i];\n \n@@ -1805,6 +1828,10 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\tif (hash_algo == GIT_HASH_UNKNOWN)\n \t\t\t\t\tdie(_(\"unknown hash algorithm '%s'\"), arg);\n \t\t\t\trepo_set_hash_algo(the_repository, hash_algo);\n+\t\t\t} else if (!strcmp(arg, \"--rev-index\")) {\n+\t\t\t\trev_index = 1;\n+\t\t\t} else if (!strcmp(arg, \"--no-rev-index\")) {\n+\t\t\t\trev_index = 0;\n \t\t\t} else\n \t\t\t\tusage(index_pack_usage);\n \t\t\tcontinue;\n@@ -1824,7 +1851,16 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (from_stdin && hash_algo)\n \t\tdie(_(\"--object-format cannot be used with --stdin\"));\n \tif (!index_name && pack_name)\n-\t\tindex_name = derive_filename(pack_name, \"idx\", &index_name_buf);\n+\t\tindex_name = derive_filename(pack_name, \".pack\", \"idx\", &index_name_buf);\n+\n+\topts.flags &= ~(WRITE_REV | WRITE_REV_VERIFY);\n+\tif (rev_index) {\n+\t\topts.flags |= verify ? WRITE_REV_VERIFY : WRITE_REV;\n+\t\tif (index_name)\n+\t\t\trev_index_name = derive_filename(index_name,\n+\t\t\t\t\t\t\t \".idx\", \"rev\",\n+\t\t\t\t\t\t\t &rev_index_name_buf);\n+\t}\n \n \tif (verify) {\n \t\tif (!index_name)\n@@ -1878,11 +1914,16 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tfor (i = 0; i < nr_objects; i++)\n \t\tidx_objects[i] = &objects[i].idx;\n \tcurr_index = write_idx_file(index_name, idx_objects, nr_objects, &opts, pack_hash);\n+\tif (rev_index)\n+\t\tcurr_rev_index = write_rev_file(rev_index_name, idx_objects,\n+\t\t\t\t\t\tnr_objects, pack_hash,\n+\t\t\t\t\t\topts.flags);\n \tfree(idx_objects);\n \n \tif (!verify)\n \t\tfinal(pack_name, curr_pack,\n \t\t index_name, curr_index,\n+\t\t rev_index_name, curr_rev_index,\n \t\t keep_msg, promisor_msg,\n \t\t pack_hash);\n \telse\n@@ -1893,10 +1934,13 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \n \tfree(objects);\n \tstrbuf_release(&index_name_buf);\n+\tstrbuf_release(&rev_index_name_buf);\n \tif (pack_name == NULL)\n \t\tfree((void *) curr_pack);\n \tif (index_name == NULL)\n \t\tfree((void *) curr_index);\n+\tif (rev_index_name == NULL)\n+\t\tfree((void *) curr_rev_index);\n \n \t/*\n \t * Let the caller know this pack is not self contained\ndiff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh\nnew file mode 100755\nindex 0000000000..f9afa698dc\n--- /dev/null\n+++ b/t/t5325-reverse-index.sh\n@@ -0,0 +1,69 @@\n+#!/bin/sh\n+\n+test_description='on-disk reverse index'\n+. ./test-lib.sh\n+\n+packdir=.git/objects/pack\n+\n+test_expect_success 'setup' '\n+\ttest_commit base &&\n+\n+\tpack=$(git pack-objects --all $packdir/pack) &&\n+\trev=$packdir/pack-$pack.rev &&\n+\n+\ttest_path_is_missing $rev\n+'\n+\n+test_index_pack () {\n+\trm -f $rev &&\n+\tconf=$1 &&\n+\tshift &&\n+\tgit -c pack.writeReverseIndex=$conf index-pack \"$@\" \\\n+\t\t$packdir/pack-$pack.pack\n+}\n+\n+test_expect_success 'index-pack with pack.writeReverseIndex' '\n+\ttest_index_pack \"\" &&\n+\ttest_path_is_missing $rev &&\n+\n+\ttest_index_pack false &&\n+\ttest_path_is_missing $rev &&\n+\n+\ttest_index_pack true &&\n+\ttest_path_is_file $rev\n+'\n+\n+test_expect_success 'index-pack with --[no-]rev-index' '\n+\tfor conf in \"\" true false\n+\tdo\n+\t\ttest_index_pack \"$conf\" --rev-index &&\n+\t\ttest_path_exists $rev &&\n+\n+\t\ttest_index_pack \"$conf\" --no-rev-index &&\n+\t\ttest_path_is_missing $rev\n+\tdone\n+'\n+\n+test_expect_success 'index-pack can verify reverse indexes' '\n+\ttest_when_finished \"rm -f $rev\" &&\n+\ttest_index_pack true &&\n+\n+\ttest_path_is_file $rev &&\n+\tgit index-pack --rev-index --verify $packdir/pack-$pack.pack &&\n+\n+\t# Intentionally corrupt the reverse index.\n+\tchmod u+w $rev &&\n+\tprintf \"xxxx\" | dd of=$rev bs=1 count=4 conv=notrunc &&\n+\n+\ttest_must_fail git index-pack --rev-index --verify \\\n+\t\t$packdir/pack-$pack.pack 2>err &&\n+\tgrep \"validation error\" err\n+'\n+\n+test_expect_success 'index-pack infers reverse index name with -o' '\n+\tgit index-pack --rev-index -o other.idx $packdir/pack-$pack.pack &&\n+\ttest_path_is_file other.idx &&\n+\ttest_path_is_file other.rev\n+'\n+\n+test_done\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413857","messageId":"ddf47a0a50e0a0a5b5a2d6665a13c245dc77f426.1610129989.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1610129989.git.me@ttaylorr.com","subject":"[PATCH 1/8] packfile: prepare for the existence of '*.rev' files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:19:57Z","receivedAt":"2021-01-08T18:20:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Specify the format of the on-disk reverse index 'pack-*.rev' file, as\nwell as prepare the code for the existence of such files.\n\nThe reverse index maps from pack relative positions (i.e., an index into\nthe array of object which is sorted by their offsets within the\npackfile) to their position within the 'pack-*.idx' file. Today, this is\ndone by building up a list of (off_t, uint32_t) tuples for each object\n(the off_t corresponding to that object's offset, and the uint32_t\ncorresponding to its position in the index). To convert between pack and\nindex position quickly, this array of tuples is radix sorted based on\nits offset.\n\nThis has two major drawbacks:\n\nFirst, the in-memory cost scales linearly with the number of objects in\na pack. Each 'struct revindex_entry' is sizeof(off_t) +\nsizeof(uint32_t) + padding bytes for a total of 16.\n\nTo observe this, force Git to load the reverse index by, for e.g.,\nrunning 'git cat-file --batch-check=\"%(objectsize:disk)\"'. When asking\nfor a single object in a fresh clone of the kernel, Git needs to\nallocate 120+ MB of memory in order to hold the reverse index in memory.\n\nSecond, the cost to sort also scales with the size of the pack.\nLuckily, this is a linear function since 'load_pack_revindex()' uses a\nradix sort, but this cost still must be paid once per pack per process.\n\nAs an example, it takes ~60x longer to print the _size_ of an object as\nit does to print that entire object's _contents_:\n\n Benchmark #1: git.compile cat-file --batch /dev/null\n Time (mean ± σ): 22.6 ms ± 0.5 ms [User: 2.4 ms, System: 7.9 ms]\n Range (min … max): 21.4 ms … 23.5 ms 41 runs\n\n Benchmark #2: git.compile cat-file --batch /dev/null\n Time (mean ± σ): 17.2 ms ± 0.7 ms [User: 2.8 ms, System: 5.5 ms]\n Range (min … max): 15.6 ms … 18.2 ms 45 runs\n\n(Numbers taken in the kernel after cheating and using the next patch to\ngenerate a reverse index). There are a couple of approaches to improve\ncold cache performance not pursued here:\n\n - We could include the object offsets in the reverse index format.\n Predictably, this does result in fewer page faults, but it triples\n the size of the file, while simultaneously duplicating a ton of data\n already available in the .idx file. (This was the original way I\n implemented the format, and it did show\n `--batch-check='%(objectsize:disk)'` winning out against `--batch`.)\n\n On the other hand, this increase in size also results in a large\n block-cache footprint, which could potentially hurt other workloads.\n\n - We could store the mapping from pack to index position in more\n cache-friendly way, like constructing a binary search tree from the\n table and writing the values in breadth-first order. This would\n result in much better locality, but the price you pay is trading\n O(1) lookup in 'pack_pos_to_index()' for an O(log n) one (since you\n can no longer directly index the table).\n\nSo, neither of these approaches are taken here. (Thankfully, the format\nis versioned, so we are free to pursue these in the future.) But, cold\ncache performance likely isn't interesting outside of one-off cases like\nasking for the size of an object directly. In real-world usage, Git is\noften performing many operations in the revindex,\n\nThe trade-off is worth it, since we will avoid the vast majority of the\ncost of generating the revindex that the extra pointer chase will look\nlike noise in the following patch's benchmarks.\n\nThis patch describes the format and prepares callers (like in\npack-revindex.c) to be able to read *.rev files once they exist. An\nimplementation of the writer will appear in the next patch, and callers\nwill gradually begin to start using the writer in the patches that\nfollow after that.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/technical/pack-format.txt | 17 ++++\n builtin/repack.c | 1 +\n object-store.h | 3 +\n pack-revindex.c | 112 +++++++++++++++++++++---\n packfile.c | 13 ++-\n packfile.h | 1 +\n tmp-objdir.c | 4 +-\n 7 files changed, 139 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex f96b2e605f..9593f8bc68 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -259,6 +259,23 @@ Pack file entry: <+\n \n Index checksum of all of the above.\n \n+== pack-*.rev files have the format:\n+\n+ - A 4-byte magic number '0x52494458' ('RIDX').\n+\n+ - A 4-byte version identifier (= 1)\n+\n+ - A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256)\n+\n+ - A table of index positions, sorted by their corresponding offsets in the\n+ packfile.\n+\n+ - A trailer, containing a:\n+\n+ checksum of the corresponding packfile, and\n+\n+ a checksum of all of the above.\n+\n == multi-pack-index (MIDX) files have the following format:\n \n The multi-pack-index files refer to multiple pack-files and loose objects.\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 279be11a16..8d643ddcb9 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -208,6 +208,7 @@ static struct {\n } exts[] = {\n \t{\".pack\"},\n \t{\".idx\"},\n+\t{\".rev\", 1},\n \t{\".bitmap\", 1},\n \t{\".promisor\", 1},\n };\ndiff --git a/object-store.h b/object-store.h\nindex c4fc9dd74e..3fbf11280f 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -85,6 +85,9 @@ struct packed_git {\n \t\t multi_pack_index:1;\n \tunsigned char hash[GIT_MAX_RAWSZ];\n \tstruct revindex_entry *revindex;\n+\tconst void *revindex_data;\n+\tconst void *revindex_map;\n+\tsize_t revindex_size;\n \t/* something like \".git/objects/pack/xxxxx.pack\" */\n \tchar pack_name[FLEX_ARRAY]; /* more */\n };\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 2cd9d632f1..1baaf2c42a 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -164,16 +164,98 @@ static void create_pack_revindex(struct packed_git *p)\n \tsort_revindex(p->revindex, num_ent, p->pack_size);\n }\n \n-int load_pack_revindex(struct packed_git *p)\n+static int load_pack_revindex_from_memory(struct packed_git *p)\n {\n-\tif (!p->revindex) {\n-\t\tif (open_pack_index(p))\n-\t\t\treturn -1;\n-\t\tcreate_pack_revindex(p);\n-\t}\n+\tif (open_pack_index(p))\n+\t\treturn -1;\n+\tcreate_pack_revindex(p);\n \treturn 0;\n }\n \n+static char *pack_revindex_filename(struct packed_git *p)\n+{\n+\tsize_t len;\n+\tif (!strip_suffix(p->pack_name, \".pack\", &len))\n+\t\tBUG(\"pack_name does not end in .pack\");\n+\treturn xstrfmt(\"%.*s.rev\", (int)len, p->pack_name);\n+}\n+\n+#define RIDX_MIN_SIZE (12 + (2 * the_hash_algo->rawsz))\n+\n+static int load_revindex_from_disk(char *revindex_name,\n+\t\t\t\t uint32_t num_objects,\n+\t\t\t\t const void **data, size_t *len)\n+{\n+\tint fd, ret = 0;\n+\tstruct stat st;\n+\tsize_t revindex_size;\n+\n+\tfd = git_open(revindex_name);\n+\n+\tif (fd < 0) {\n+\t\tret = -1;\n+\t\tgoto cleanup;\n+\t}\n+\tif (fstat(fd, &st)) {\n+\t\tret = error_errno(_(\"failed to read %s\"), revindex_name);\n+\t\tgoto cleanup;\n+\t}\n+\n+\trevindex_size = xsize_t(st.st_size);\n+\n+\tif (revindex_size < RIDX_MIN_SIZE) {\n+\t\tret = error(_(\"reverse-index file %s is too small\"), revindex_name);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (revindex_size - RIDX_MIN_SIZE != st_mult(sizeof(uint32_t), num_objects)) {\n+\t\tret = error(_(\"reverse-index file %s is corrupt\"), revindex_name);\n+\t\tgoto cleanup;\n+\t}\n+\n+\t*len = revindex_size;\n+\t*data = xmmap(NULL, revindex_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\n+cleanup:\n+\tclose(fd);\n+\treturn ret;\n+}\n+\n+static int load_pack_revindex_from_disk(struct packed_git *p)\n+{\n+\tchar *revindex_name;\n+\tint ret;\n+\tif (open_pack_index(p))\n+\t\treturn -1;\n+\n+\trevindex_name = pack_revindex_filename(p);\n+\n+\tret = load_revindex_from_disk(revindex_name,\n+\t\t\t\t p->num_objects,\n+\t\t\t\t &p->revindex_map,\n+\t\t\t\t &p->revindex_size);\n+\tif (ret)\n+\t\tgoto cleanup;\n+\n+\tp->revindex_data = (char *)p->revindex_map + 12;\n+\n+cleanup:\n+\tfree(revindex_name);\n+\treturn ret;\n+}\n+\n+int load_pack_revindex(struct packed_git *p)\n+{\n+\tif (p->revindex || p->revindex_data)\n+\t\treturn 0;\n+\n+\tif (!load_pack_revindex_from_disk(p))\n+\t\treturn 0;\n+\telse if (!load_pack_revindex_from_memory(p))\n+\t\treturn 0;\n+\treturn -1;\n+}\n+\n int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n {\n \tint lo = 0;\n@@ -203,18 +285,28 @@ int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n \n uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos)\n {\n-\tif (!p->revindex)\n+\tif (!(p->revindex || p->revindex_data))\n \t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n \tif (pos >= p->num_objects)\n \t\tBUG(\"pack_pos_to_index: out-of-bounds object at %\"PRIu32, pos);\n-\treturn p->revindex[pos].nr;\n+\n+\tif (p->revindex)\n+\t\treturn p->revindex[pos].nr;\n+\telse\n+\t\treturn get_be32((char *)p->revindex_data + (pos * sizeof(uint32_t)));\n }\n \n off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n {\n-\tif (!p->revindex)\n+\tif (!(p->revindex || p->revindex_data))\n \t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n \tif (pos > p->num_objects)\n \t\tBUG(\"pack_pos_to_offset: out-of-bounds object at %\"PRIu32, pos);\n-\treturn p->revindex[pos].offset;\n+\n+\tif (p->revindex)\n+\t\treturn p->revindex[pos].offset;\n+\telse if (pos == p->num_objects)\n+\t\treturn p->pack_size - the_hash_algo->rawsz;\n+\telse\n+\t\treturn nth_packed_object_offset(p, pack_pos_to_index(p, pos));\n }\ndiff --git a/packfile.c b/packfile.c\nindex 46c9c7ea3c..e636e5ca17 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -324,11 +324,21 @@ void close_pack_index(struct packed_git *p)\n \t}\n }\n \n+void close_pack_revindex(struct packed_git *p) {\n+\tif (!p->revindex_map)\n+\t\treturn;\n+\n+\tmunmap((void *)p->revindex_map, p->revindex_size);\n+\tp->revindex_map = NULL;\n+\tp->revindex_data = NULL;\n+}\n+\n void close_pack(struct packed_git *p)\n {\n \tclose_pack_windows(p);\n \tclose_pack_fd(p);\n \tclose_pack_index(p);\n+\tclose_pack_revindex(p);\n }\n \n void close_object_store(struct raw_object_store *o)\n@@ -351,7 +361,7 @@ void close_object_store(struct raw_object_store *o)\n \n void unlink_pack_path(const char *pack_name, int force_delete)\n {\n-\tstatic const char *exts[] = {\".pack\", \".idx\", \".keep\", \".bitmap\", \".promisor\"};\n+\tstatic const char *exts[] = {\".pack\", \".idx\", \".rev\", \".keep\", \".bitmap\", \".promisor\"};\n \tint i;\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t plen;\n@@ -853,6 +863,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tif (!strcmp(file_name, \"multi-pack-index\"))\n \t\treturn;\n \tif (ends_with(file_name, \".idx\") ||\n+\t ends_with(file_name, \".rev\") ||\n \t ends_with(file_name, \".pack\") ||\n \t ends_with(file_name, \".bitmap\") ||\n \t ends_with(file_name, \".keep\") ||\ndiff --git a/packfile.h b/packfile.h\nindex a58fc738e0..4cfec9e8d3 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -90,6 +90,7 @@ uint32_t get_pack_fanout(struct packed_git *p, uint32_t value);\n \n unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n void close_pack_windows(struct packed_git *);\n+void close_pack_revindex(struct packed_git *);\n void close_pack(struct packed_git *);\n void close_object_store(struct raw_object_store *o);\n void unuse_pack(struct pack_window **);\ndiff --git a/tmp-objdir.c b/tmp-objdir.c\nindex 42ed4db5d3..da414df14f 100644\n--- a/tmp-objdir.c\n+++ b/tmp-objdir.c\n@@ -187,7 +187,9 @@ static int pack_copy_priority(const char *name)\n \t\treturn 2;\n \tif (ends_with(name, \".idx\"))\n \t\treturn 3;\n-\treturn 4;\n+\tif (ends_with(name, \".rev\"))\n+\t\treturn 4;\n+\treturn 5;\n }\n \n static int pack_copy_cmp(const char *a, const char *b)\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413858","messageId":"e297a31875a1135ba17743aa57c24e2991bf8148.1610129989.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1610129989.git.me@ttaylorr.com","subject":"[PATCH 4/8] builtin/pack-objects.c: respect 'pack.writeReverseIndex'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:20:11Z","receivedAt":"2021-01-08T18:21:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that we have an implementation that can write the new reverse index\nformat, enable writing a .rev file in 'git pack-objects' by consulting\nthe pack.writeReverseIndex configuration variable.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 7 +++++++\n t/t5325-reverse-index.sh | 13 +++++++++++++\n 2 files changed, 20 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a193cdaf2f..80adce154a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2951,6 +2951,13 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\t pack_idx_opts.version);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"pack.writereverseindex\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\tpack_idx_opts.flags |= WRITE_REV;\n+\t\telse\n+\t\t\tpack_idx_opts.flags &= ~WRITE_REV;\n+\t\treturn 0;\n+\t}\n \tif (!strcmp(k, \"uploadpack.blobpackfileuri\")) {\n \t\tstruct configured_exclusion *ex = xmalloc(sizeof(*ex));\n \t\tconst char *oid_end, *pack_end;\ndiff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh\nindex f9afa698dc..431699ade2 100755\n--- a/t/t5325-reverse-index.sh\n+++ b/t/t5325-reverse-index.sh\n@@ -66,4 +66,17 @@ test_expect_success 'index-pack infers reverse index name with -o' '\n \ttest_path_is_file other.rev\n '\n \n+test_expect_success 'pack-objects respects pack.writeReverseIndex' '\n+\ttest_when_finished \"rm -fr pack-1-*\" &&\n+\n+\tgit -c pack.writeReverseIndex= pack-objects --all pack-1 &&\n+\ttest_path_is_missing pack-1-*.rev &&\n+\n+\tgit -c pack.writeReverseIndex=false pack-objects --all pack-1 &&\n+\ttest_path_is_missing pack-1-*.rev &&\n+\n+\tgit -c pack.writeReverseIndex=true pack-objects --all pack-1 &&\n+\ttest_path_is_file pack-1-*.rev\n+'\n+\n test_done\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413859","messageId":"3525c4d114c8655953aca44d8effa1652ccc93d3.1610129989.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1610129989.git.me@ttaylorr.com","subject":"[PATCH 7/8] t: support GIT_TEST_WRITE_REV_INDEX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:20:25Z","receivedAt":"2021-01-08T18:21:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Add a new option that unconditionally enables the pack.writeReverseIndex\nsetting in order to run the whole test suite in a mode that generates\non-disk reverse indexes.\n\nOnce on-disk reverse indexes are proven out over several releases, we\ncan change the default value of that configuration to 'true', and drop\nthis patch.\n\nSigned-off-by: Taylor Blau \n---\n builtin/index-pack.c | 5 ++++-\n builtin/pack-objects.c | 2 ++\n pack-revindex.h | 2 ++\n t/README | 3 +++\n 4 files changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 03408250b1..0bde325a8b 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1748,7 +1748,10 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (prefix && chdir(prefix))\n \t\tdie(_(\"Cannot come back to cwd\"));\n \n-\trev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV));\n+\tif (git_env_bool(GIT_TEST_WRITE_REV_INDEX, 0))\n+\t\trev_index = 1;\n+\telse\n+\t\trev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV));\n \n \tfor (i = 1; i < argc; i++) {\n \t\tconst char *arg = argv[i];\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 80adce154a..5b3395bd7a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3597,6 +3597,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \n \treset_pack_idx_option(&pack_idx_opts);\n \tgit_config(git_pack_config, NULL);\n+\tif (git_env_bool(GIT_TEST_WRITE_REV_INDEX, 0))\n+\t\tpack_idx_opts.flags |= WRITE_REV;\n \n \tprogress = isatty(2);\n \targc = parse_options(argc, argv, prefix, pack_objects_options,\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex b501a7cd62..d2d466e298 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -1,6 +1,8 @@\n #ifndef PACK_REVINDEX_H\n #define PACK_REVINDEX_H\n \n+#define GIT_TEST_WRITE_REV_INDEX \"GIT_TEST_WRITE_REV_INDEX\"\n+\n struct packed_git;\n \n int load_pack_revindex(struct packed_git *p);\ndiff --git a/t/README b/t/README\nindex c730a70770..0f97a51640 100644\n--- a/t/README\n+++ b/t/README\n@@ -439,6 +439,9 @@ GIT_TEST_DEFAULT_HASH= specifies which hash algorithm to\n use in the test scripts. Recognized values for are \"sha1\"\n and \"sha256\".\n \n+GIT_TEST_WRITE_REV_INDEX=, when true enables the\n+'pack.writeReverseIndex' setting.\n+\n Naming Tests\n ------------\n \n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413860","messageId":"2288571fbeb83d6bd98917f4788c3221aa973c81.1610129989.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1610129989.git.me@ttaylorr.com","subject":"[PATCH 6/8] t: prepare for GIT_TEST_WRITE_REV_INDEX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:20:20Z","receivedAt":"2021-01-08T18:21:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In the next patch, we'll add support for unconditionally enabling the\n'pack.writeReverseIndex' setting with a new GIT_TEST_WRITE_REV_INDEX\nenvironment variable.\n\nThis causes a little bit of fallout with tests that, for example,\ncompare the list of files in the pack directory being unprepared to see\n.rev files in its output.\n\nFor now, sprinkle these locations with a 'grep -v \"\\.rev$\"' to ignore\nthem. Once the pack.writeReverseIndex option has been thoroughly\ntested, we will default it to 'true', removing GIT_TEST_WRITE_REV_INDEX,\nand making it possible to revert this patch.\n\nAt that time, we'll have to adjust the expected output to contain the\nrelevant .rev files, but for now this will do just fine.\n\nSigned-off-by: Taylor Blau \n---\n t/t5319-multi-pack-index.sh | 2 +-\n t/t5325-reverse-index.sh | 4 ++++\n t/t5604-clone-reference.sh | 2 +-\n t/t5702-protocol-v2.sh | 4 ++--\n t/t6500-gc.sh | 4 ++--\n t/t9300-fast-import.sh | 2 +-\n 6 files changed, 11 insertions(+), 7 deletions(-)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 297de502a9..9696f88c2f 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -710,7 +710,7 @@ test_expect_success 'expire respects .keep files' '\n \t\tPACKA=$(ls .git/objects/pack/a-pack*\\.pack | sed s/\\.pack\\$//) &&\n \t\ttouch $PACKA.keep &&\n \t\tgit multi-pack-index expire &&\n-\t\tls -S .git/objects/pack/a-pack* | grep $PACKA >a-pack-files &&\n+\t\tls -S .git/objects/pack/a-pack* | grep $PACKA | grep -v \"\\.rev$\" >a-pack-files &&\n \t\ttest_line_count = 3 a-pack-files &&\n \t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n \t\ttest_line_count = 2 midx-list\ndiff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh\nindex 431699ade2..5a59cc71e4 100755\n--- a/t/t5325-reverse-index.sh\n+++ b/t/t5325-reverse-index.sh\n@@ -3,6 +3,10 @@\n test_description='on-disk reverse index'\n . ./test-lib.sh\n \n+# The below tests want control over the 'pack.writeReverseIndex' setting\n+# themselves to assert various combinations of it with other options.\n+sane_unset GIT_TEST_WRITE_REV_INDEX\n+\n packdir=.git/objects/pack\n \n test_expect_success 'setup' '\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 2f7be23044..d064426d03 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -318,7 +318,7 @@ test_expect_success SYMLINKS 'clone repo with symlinked or unknown files at obje\n \t\ttest_cmp T.objects T$option.objects &&\n \t\t(\n \t\t\tcd T$option/.git/objects &&\n-\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw &&\n+\t\t\tfind . -type f | grep -v \\.rev$ | sort >../../../T$option.objects-files.raw &&\n \t\t\tfind . -type l | sort >../../../T$option.objects-symlinks.raw\n \t\t)\n \tdone &&\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 7d5b17909b..9ebf045739 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -848,7 +848,7 @@ test_expect_success 'part of packfile response provided as URI' '\n \ttest -f h2found &&\n \n \t# Ensure that there are exactly 6 files (3 .pack and 3 .idx).\n-\tls http_child/.git/objects/pack/* >filelist &&\n+\tls http_child/.git/objects/pack/* | grep -v \\.rev$ >filelist &&\n \ttest_line_count = 6 filelist\n '\n \n@@ -902,7 +902,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' '\n \t\tclone \"$HTTPD_URL/smart/http_parent\" http_child &&\n \n \t# Ensure that there are exactly 4 files (2 .pack and 2 .idx).\n-\tls http_child/.git/objects/pack/* >filelist &&\n+\tls http_child/.git/objects/pack/* | grep -v \\.rev$ >filelist &&\n \ttest_line_count = 4 filelist\n '\n \ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 4a3b8f48ac..d52f92f5a1 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -106,13 +106,13 @@ test_expect_success 'auto gc with too many loose objects does not attempt to cre\n \ttest_commit \"$(test_oid obj2)\" &&\n \t# Our first gc will create a pack; our second will create a second pack\n \tgit gc --auto &&\n-\tls .git/objects/pack | sort >existing_packs &&\n+\tls .git/objects/pack | grep -v \\.rev$ | sort >existing_packs &&\n \ttest_commit \"$(test_oid obj3)\" &&\n \ttest_commit \"$(test_oid obj4)\" &&\n \n \tgit gc --auto 2>err &&\n \ttest_i18ngrep ! \"^warning:\" err &&\n-\tls .git/objects/pack/ | sort >post_packs &&\n+\tls .git/objects/pack/ | grep -v \\.rev$ | sort >post_packs &&\n \tcomm -1 -3 existing_packs post_packs >new &&\n \tcomm -2 -3 existing_packs post_packs >del &&\n \ttest_line_count = 0 del && # No packs are deleted\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 308c1ef42c..100df52a71 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -1629,7 +1629,7 @@ test_expect_success 'O: blank lines not necessary after other commands' '\n \tINPUT_END\n \n \tgit fast-import actual &&\n \ttest_cmp expect actual\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413861","messageId":"5d3e96a498527959ae2f77476cd00474e36fbc97.1610129989.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1610129989.git.me@ttaylorr.com","subject":"[PATCH 5/8] Documentation/config/pack.txt: advertise 'pack.writeReverseIndex'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:20:16Z","receivedAt":"2021-01-08T18:21:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the pack.writeReverseIndex configuration is respected in both\n'git index-pack' and 'git pack-objects' (and therefore, all of their\ncallers), we can safely advertise it for use in the git-config manual.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/pack.txt | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\nindex 837f1b1679..3da4ea98e2 100644\n--- a/Documentation/config/pack.txt\n+++ b/Documentation/config/pack.txt\n@@ -133,3 +133,10 @@ pack.writeBitmapHashCache::\n \tbetween an older, bitmapped pack and objects that have been\n \tpushed since the last gc). The downside is that it consumes 4\n \tbytes per object of disk space. Defaults to true.\n+\n+pack.writeReverseIndex::\n+\tWhen true, git will write a corresponding .rev file (see:\n+\tlink:../technical/pack-format.html[Documentation/technical/pack-format.txt])\n+\tfor each new packfile that it writes in all places except for\n+\tlinkgit:git-fast-import[1] and in the bulk checkin mechanism.\n+\tDefaults to false.\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414175","messageId":"9879225b-216a-574b-7bf8-a5f772732182@gmail.com","threadId":"54962","inReplyTo":"3525c4d114c8655953aca44d8effa1652ccc93d3.1610129989.git.me@ttaylorr.com","subject":"Re: [PATCH 7/8] t: support GIT_TEST_WRITE_REV_INDEX","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-01-12T16:49:40Z","receivedAt":"2021-01-12T16:50:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/8/2021 1:20 PM, Taylor Blau wrote:\n> Add a new option that unconditionally enables the pack.writeReverseIndex\n> setting in order to run the whole test suite in a mode that generates\n> on-disk reverse indexes.\n> \n> Once on-disk reverse indexes are proven out over several releases, we\n> can change the default value of that configuration to 'true', and drop\n> this patch.\n\n...\n\n> +GIT_TEST_WRITE_REV_INDEX=, when true enables the\n> +'pack.writeReverseIndex' setting.\n> +\n\nShould this also be added to the second run of the test\nsuite with optional variables?\n\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 6c27b886b8f..d1cbf330a14 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -22,6 +22,7 @@ linux-gcc)\n \texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n \texport GIT_TEST_ADD_I_USE_BUILTIN=1\n+\texport GIT_TEST_WRITE_REV_INDEX=1\n \tmake test\n \t;;\n linux-clang)\n\nThanks,\n-Stolee\n"},{"id":"414183","messageId":"878s8y5bdn.fsf@evledraar.gmail.com","threadId":"54962","inReplyTo":"2288571fbeb83d6bd98917f4788c3221aa973c81.1610129989.git.me@ttaylorr.com","subject":"Re: [PATCH 6/8] t: prepare for GIT_TEST_WRITE_REV_INDEX","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-01-12T17:11:00Z","receivedAt":"2021-01-12T17:11:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jan 08 2021, Taylor Blau wrote:\n\n> For now, sprinkle these locations with a 'grep -v \"\\.rev$\"' to ignore\n> them. Once the pack.writeReverseIndex option has been thoroughly\n> tested, we will default it to 'true', removing GIT_TEST_WRITE_REV_INDEX,\n> and making it possible to revert this patch.\n\nMaybe some of it we can change/revert, but some of it just seems to be\ntest warts we can fix:\n\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> index 297de502a9..9696f88c2f 100755\n> --- a/t/t5319-multi-pack-index.sh\n> +++ b/t/t5319-multi-pack-index.sh\n> @@ -710,7 +710,7 @@ test_expect_success 'expire respects .keep files' '\n> \t\tPACKA=$(ls .git/objects/pack/a-pack*\\.pack | sed s/\\.pack\\$//) &&\n> \t\ttouch $PACKA.keep &&\n> \t\tgit multi-pack-index expire &&\n> -\t\tls -S .git/objects/pack/a-pack* | grep $PACKA >a-pack-files &&\n> +\t\tls -S .git/objects/pack/a-pack* | grep $PACKA | grep -v \"\\.rev$\" >a-pack-files &&\n\nThis seems to be testing \"a *.keep file made my pack not expire\". Can't\nit just check for *.{pack,idx,keep} or even just *.pack?\n\n> \t\ttest_line_count = 3 a-pack-files &&\n> \t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n> \t\ttest_line_count = 2 midx-list\n> diff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh\n> index 431699ade2..5a59cc71e4 100755\n> --- a/t/t5325-reverse-index.sh\n> +++ b/t/t5325-reverse-index.sh\n> @@ -3,6 +3,10 @@\n> test_description='on-disk reverse index'\n> . ./test-lib.sh\n> \n> +# The below tests want control over the 'pack.writeReverseIndex' setting\n> +# themselves to assert various combinations of it with other options.\n> +sane_unset GIT_TEST_WRITE_REV_INDEX\n> +\n> packdir=.git/objects/pack\n> \n> test_expect_success 'setup' '\n> diff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\n> index 2f7be23044..d064426d03 100755\n> --- a/t/t5604-clone-reference.sh\n> +++ b/t/t5604-clone-reference.sh\n> @@ -318,7 +318,7 @@ test_expect_success SYMLINKS 'clone repo with symlinked or unknown files at obje\n> \t\ttest_cmp T.objects T$option.objects &&\n> \t\t(\n> \t\t\tcd T$option/.git/objects &&\n> -\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw &&\n> +\t\t\tfind . -type f | grep -v \\.rev$ | sort >../../../T$option.objects-files.raw &&\n> \t\t\tfind . -type l | sort >../../../T$option.objects-symlinks.raw\n\nThere's an existing loop just below that where we grep out\n/commit-graph/, /multi-pack-index/ etc. whith other test modes add to\nthe objects directory with sed. Seems like this belongs there, not in\nthe find above it.\n\n> \t\t)\n> \tdone &&\n> diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\n> index 7d5b17909b..9ebf045739 100755\n> --- a/t/t5702-protocol-v2.sh\n> +++ b/t/t5702-protocol-v2.sh\n> @@ -848,7 +848,7 @@ test_expect_success 'part of packfile response provided as URI' '\n> \ttest -f h2found &&\n> \n> \t# Ensure that there are exactly 6 files (3 .pack and 3 .idx).\n> -\tls http_child/.git/objects/pack/* >filelist &&\n> +\tls http_child/.git/objects/pack/* | grep -v \\.rev$ >filelist &&\n> \ttest_line_count = 6 filelist\n> '\n\nMaybe just check *.{pack,idx,keep}. I was looking at that code the other\nday and it's really just being overly specific. It really just cares\nabout the *.pack files.\n\n> @@ -902,7 +902,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' '\n> \t\tclone \"$HTTPD_URL/smart/http_parent\" http_child &&\n> \n> \t# Ensure that there are exactly 4 files (2 .pack and 2 .idx).\n> -\tls http_child/.git/objects/pack/* >filelist &&\n> +\tls http_child/.git/objects/pack/* | grep -v \\.rev$ >filelist &&\n\nditto.\n\n> \ttest_line_count = 4 filelist\n> '\n> \n> diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\n> index 4a3b8f48ac..d52f92f5a1 100755\n> --- a/t/t6500-gc.sh\n> +++ b/t/t6500-gc.sh\n> @@ -106,13 +106,13 @@ test_expect_success 'auto gc with too many loose objects does not attempt to cre\n> \ttest_commit \"$(test_oid obj2)\" &&\n> \t# Our first gc will create a pack; our second will create a second pack\n> \tgit gc --auto &&\n> -\tls .git/objects/pack | sort >existing_packs &&\n> +\tls .git/objects/pack | grep -v \\.rev$ | sort >existing_packs &&\n> \ttest_commit \"$(test_oid obj3)\" &&\n> \ttest_commit \"$(test_oid obj4)\" &&\n> \n> \tgit gc --auto 2>err &&\n> \ttest_i18ngrep ! \"^warning:\" err &&\n> -\tls .git/objects/pack/ | sort >post_packs &&\n> +\tls .git/objects/pack/ | grep -v \\.rev$ | sort >post_packs &&\n> \tcomm -1 -3 existing_packs post_packs >new &&\n> \tcomm -2 -3 existing_packs post_packs >del &&\n> \ttest_line_count = 0 del && # No packs are deleted\n\nThis is all part of account where we later use comm/wc -l to check how\nmany new packs we have,so just check *.pack?\n\n> diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\n> index 308c1ef42c..100df52a71 100755\n> --- a/t/t9300-fast-import.sh\n> +++ b/t/t9300-fast-import.sh\n> @@ -1629,7 +1629,7 @@ test_expect_success 'O: blank lines not necessary after other commands' '\n> \tINPUT_END\n> \n> \tgit fast-import -\ttest 8 = $(find .git/objects/pack -type f | grep -v multi-pack-index | wc -l) &&\n> +\ttest 8 = $(find .git/objects/pack -type f \\( -name \"*.idx\" -o -name \"*.pack\" \\) | wc -l) &&\n\nYay, there the existing multi-pack-index case is amended in a\nfuture-proof way :)\n"},{"id":"414184","messageId":"875z425b17.fsf@evledraar.gmail.com","threadId":"54962","inReplyTo":"3525c4d114c8655953aca44d8effa1652ccc93d3.1610129989.git.me@ttaylorr.com","subject":"Re: [PATCH 7/8] t: support GIT_TEST_WRITE_REV_INDEX","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-01-12T17:18:28Z","receivedAt":"2021-01-12T17:19:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jan 08 2021, Taylor Blau wrote:\n\n> Signed-off-by: Taylor Blau \n> ---\n> builtin/index-pack.c | 5 ++++-\n> builtin/pack-objects.c | 2 ++\n> pack-revindex.h | 2 ++\n> t/README | 3 +++\n> 4 files changed, 11 insertions(+), 1 deletion(-)\n>\n> diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n> index 03408250b1..0bde325a8b 100644\n> --- a/builtin/index-pack.c\n> +++ b/builtin/index-pack.c\n> @@ -1748,7 +1748,10 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n> \tif (prefix && chdir(prefix))\n> \t\tdie(_(\"Cannot come back to cwd\"));\n> \n> -\trev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV));\n> +\tif (git_env_bool(GIT_TEST_WRITE_REV_INDEX, 0))\n> +\t\trev_index = 1;\n> +\telse\n> +\t\trev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV));\n\nWhy not make an explicit GIT_TEST_WRITE_REV_INDEX=false meaningful? It's\nalso sometimes handy to turn these off in the tests.\n\n rev_index = git_env_bool(\"GIT_TEST_WRITE_REV_INDEX\",\n \t!!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV)));\n\n> +#define GIT_TEST_WRITE_REV_INDEX \"GIT_TEST_WRITE_REV_INDEX\"\n\nMicro style nit: FWIW I'm not a fan of this macro->string indirection a\nfew GIT_TEST_* names have had since 859fdc0c3c (commit-graph: define\nGIT_TEST_COMMIT_GRAPH, 2018-08-29).\n\nMost of them just use git_env_bool(\"GIT_TEST_[...]\") which IMO makes it\neasier to eyeball a \"git grep\"\n\n> +GIT_TEST_WRITE_REV_INDEX=, when true enables the\n> +'pack.writeReverseIndex' setting.\n> +\n\nRe the git_env_bool() default value comment above: I see our other\nboolean GIT_TEST_* docs say \"when true\", but mostly they mean things\nlike:\n\n GIT_TEST_WRITE_REV_INDEX=, when set, configures the\n 'pack.writeReverseIndex' setting. Defaults to 'false'.\n"},{"id":"414185","messageId":"X/3dr8oGCX7XR2Wc@nand.local","threadId":"54962","inReplyTo":"9879225b-216a-574b-7bf8-a5f772732182@gmail.com","subject":"Re: [PATCH 7/8] t: support GIT_TEST_WRITE_REV_INDEX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-12T17:34:39Z","receivedAt":"2021-01-12T17:35:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 12, 2021 at 11:49:40AM -0500, Derrick Stolee wrote:\n> Should this also be added to the second run of the test\n> suite with optional variables?\n\nAh, I wasn't aware of this script myself. Yes, it should be there,\nthanks.\n\nThanks,\nTaylor\n"},{"id":"414186","messageId":"077f1a42-f37a-c7b4-b05f-fba424ac7f17@gmail.com","threadId":"54962","inReplyTo":"875z425b17.fsf@evledraar.gmail.com","subject":"Re: [PATCH 7/8] t: support GIT_TEST_WRITE_REV_INDEX","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-01-12T17:39:37Z","receivedAt":"2021-01-12T17:40:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/12/2021 12:18 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Jan 08 2021, Taylor Blau wrote:\n>> -\trev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV));\n>> +\tif (git_env_bool(GIT_TEST_WRITE_REV_INDEX, 0))\n>> +\t\trev_index = 1;\n>> +\telse\n>> +\t\trev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV));\n> \n> Why not make an explicit GIT_TEST_WRITE_REV_INDEX=false meaningful? It's\n> also sometimes handy to turn these off in the tests.\n> \n> rev_index = git_env_bool(\"GIT_TEST_WRITE_REV_INDEX\",\n> \t!!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV)));\n \nThis will cause tests that explicitly request a rev-index to fail when\nGIT_TEST_WRITE_REF_INDEX=false. I'm not sure that's a good pattern to\nfollow.\n\n>> +#define GIT_TEST_WRITE_REV_INDEX \"GIT_TEST_WRITE_REV_INDEX\"\n> \n> Micro style nit: FWIW I'm not a fan of this macro->string indirection a\n> few GIT_TEST_* names have had since 859fdc0c3c (commit-graph: define\n> GIT_TEST_COMMIT_GRAPH, 2018-08-29).\n> \n> Most of them just use git_env_bool(\"GIT_TEST_[...]\") which IMO makes it\n> easier to eyeball a \"git grep\"\n\nIn the case of GIT_TEST_COMMIT_GRAPH, there are multiple places that\ncheck the environment variable, and it is probably best to have the\nstrings consistent through a macro.\n\nFor something like GIT_TEST_WRITE_REV_INDEX, this macro is less\nimportant because it is checked only once.\n\nThanks,\n-Stolee\n"},{"id":"414188","messageId":"X/3n1pbZuNOIfmSr@nand.local","threadId":"54962","inReplyTo":"077f1a42-f37a-c7b4-b05f-fba424ac7f17@gmail.com","subject":"Re: [PATCH 7/8] t: support GIT_TEST_WRITE_REV_INDEX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-12T18:17:58Z","receivedAt":"2021-01-12T18:19:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 12, 2021 at 12:39:37PM -0500, Derrick Stolee wrote:\n> On 1/12/2021 12:18 PM, Ævar Arnfjörð Bjarmason wrote:\n> >\n> > On Fri, Jan 08 2021, Taylor Blau wrote:\n> >> -\trev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV));\n> >> +\tif (git_env_bool(GIT_TEST_WRITE_REV_INDEX, 0))\n> >> +\t\trev_index = 1;\n> >> +\telse\n> >> +\t\trev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV));\n> >\n> > Why not make an explicit GIT_TEST_WRITE_REV_INDEX=false meaningful? It's\n> > also sometimes handy to turn these off in the tests.\n> >\n> > rev_index = git_env_bool(\"GIT_TEST_WRITE_REV_INDEX\",\n> > \t!!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV)));\n>\n> This will cause tests that explicitly request a rev-index to fail when\n> GIT_TEST_WRITE_REF_INDEX=false. I'm not sure that's a good pattern to\n> follow.\n\nI agree that this wouldn't work; the second parameter is the default\nvalue, not a substitute for \"false\".\n\nIt _would_ work currently, since there aren't any tests in this series\nthat explicitly\n\n export GIT_TEST_WRITE_REF_INDEX=false\n\nt5325 does 'sane_unset GIT_TEST_WRITE_REV_INDEX', which accomplishes\nthe same thing without setting a value. If we instead used the export\nform, this would break, so I think the implementation is more robust\nas-is.\n\n> >> +#define GIT_TEST_WRITE_REV_INDEX \"GIT_TEST_WRITE_REV_INDEX\"\n> >\n> > Micro style nit: FWIW I'm not a fan of this macro->string indirection a\n> > few GIT_TEST_* names have had since 859fdc0c3c (commit-graph: define\n> > GIT_TEST_COMMIT_GRAPH, 2018-08-29).\n> >\n> > Most of them just use git_env_bool(\"GIT_TEST_[...]\") which IMO makes it\n> > easier to eyeball a \"git grep\"\n>\n> In the case of GIT_TEST_COMMIT_GRAPH, there are multiple places that\n> check the environment variable, and it is probably best to have the\n> strings consistent through a macro.\n>\n> For something like GIT_TEST_WRITE_REV_INDEX, this macro is less\n> important because it is checked only once.\n\nFWIW, I'm not a huge fan of the indirection either, but I think that its\nuse here is warranted, since it is checked twice (in index-pack, and\npack-objects).\n\nIt _could_ be checked lower in the call stack, but I tried this when\ndeveloping this series and it ended up being far messier than what is\npresented here. The trickiness is with the extra verification mode\n(which ensures that an existing '.rev' file was written correctly), and\nhaving to indicate that at each caller.\n\n> Thanks,\n> -Stolee\n\nThanks,\nTaylor\n"},{"id":"414190","messageId":"X/3tMHdgS7tmMduS@nand.local","threadId":"54962","inReplyTo":"878s8y5bdn.fsf@evledraar.gmail.com","subject":"Re: [PATCH 6/8] t: prepare for GIT_TEST_WRITE_REV_INDEX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-12T18:40:48Z","receivedAt":"2021-01-12T18:41:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Ævar,\n\nYour suggestions are quite helpful, and I'm glad to apply them,\nespecially if it means that this \"clean up\" patch can actually harden us\nfrom similar changes in the future.\n\nOn Tue, Jan 12, 2021 at 06:11:00PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Fri, Jan 08 2021, Taylor Blau wrote:\n>\n> > For now, sprinkle these locations with a 'grep -v \"\\.rev$\"' to ignore\n> > them. Once the pack.writeReverseIndex option has been thoroughly\n> > tested, we will default it to 'true', removing GIT_TEST_WRITE_REV_INDEX,\n> > and making it possible to revert this patch.\n>\n> Maybe some of it we can change/revert, but some of it just seems to be\n> test warts we can fix:\n>\n> > diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> > index 297de502a9..9696f88c2f 100755\n> > --- a/t/t5319-multi-pack-index.sh\n> > +++ b/t/t5319-multi-pack-index.sh\n> > @@ -710,7 +710,7 @@ test_expect_success 'expire respects .keep files' '\n> > \t\tPACKA=$(ls .git/objects/pack/a-pack*\\.pack | sed s/\\.pack\\$//) &&\n> > \t\ttouch $PACKA.keep &&\n> > \t\tgit multi-pack-index expire &&\n> > -\t\tls -S .git/objects/pack/a-pack* | grep $PACKA >a-pack-files &&\n> > +\t\tls -S .git/objects/pack/a-pack* | grep $PACKA | grep -v \"\\.rev$\" >a-pack-files &&\n>\n> This seems to be testing \"a *.keep file made my pack not expire\". Can't\n> it just check for *.{pack,idx,keep} or even just *.pack?\n\nYeah, and I think the simplest thing to do here is just check that these\nfiles exist at all. Something like:\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 9696f88c2f..f5e50508c9 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -710,7 +710,9 @@ test_expect_success 'expire respects .keep files' '\n \t\tPACKA=$(ls .git/objects/pack/a-pack*\\.pack | sed s/\\.pack\\$//) &&\n \t\ttouch $PACKA.keep &&\n \t\tgit multi-pack-index expire &&\n-\t\tls -S .git/objects/pack/a-pack* | grep $PACKA | grep -v \"\\.rev$\" >a-pack-files &&\n+\t\ttest_is_file .git/objects/pack/pack-a-$PACKA.idx &&\n+\t\ttest_is_file .git/objects/pack/pack-a-$PACKA.keep &&\n+\t\ttest_is_file .git/objects/pack/pack-a-$PACKA.pack &&\n \t\ttest_line_count = 3 a-pack-files &&\n \t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n \t\ttest_line_count = 2 midx-list\n\n> > diff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\n> > index 2f7be23044..d064426d03 100755\n> > --- a/t/t5604-clone-reference.sh\n> > +++ b/t/t5604-clone-reference.sh\n> > @@ -318,7 +318,7 @@ test_expect_success SYMLINKS 'clone repo with symlinked or unknown files at obje\n> > \t\ttest_cmp T.objects T$option.objects &&\n> > \t\t(\n> > \t\t\tcd T$option/.git/objects &&\n> > -\t\t\tfind . -type f | sort >../../../T$option.objects-files.raw &&\n> > +\t\t\tfind . -type f | grep -v \\.rev$ | sort >../../../T$option.objects-files.raw &&\n> > \t\t\tfind . -type l | sort >../../../T$option.objects-symlinks.raw\n>\n> There's an existing loop just below that where we grep out\n> /commit-graph/, /multi-pack-index/ etc. whith other test modes add to\n> the objects directory with sed. Seems like this belongs there, not in\n> the find above it.\n\nMuch cleaner! Thank you.\n\n> > diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\n> > index 7d5b17909b..9ebf045739 100755\n> > --- a/t/t5702-protocol-v2.sh\n> > +++ b/t/t5702-protocol-v2.sh\n> > @@ -848,7 +848,7 @@ test_expect_success 'part of packfile response provided as URI' '\n> > \ttest -f h2found &&\n> >\n> > \t# Ensure that there are exactly 6 files (3 .pack and 3 .idx).\n> > -\tls http_child/.git/objects/pack/* >filelist &&\n> > +\tls http_child/.git/objects/pack/* | grep -v \\.rev$ >filelist &&\n> > \ttest_line_count = 6 filelist\n> > '\n>\n> Maybe just check *.{pack,idx,keep}. I was looking at that code the other\n> day and it's really just being overly specific. It really just cares\n> about the *.pack files.\n\nI think here and in t6500 as well as t9300 the easiest thing to do is\njust\n\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 9ebf045739..73cd9e3ff6 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -848,8 +848,10 @@ test_expect_success 'part of packfile response provided as URI' '\n \ttest -f h2found &&\n\n \t# Ensure that there are exactly 6 files (3 .pack and 3 .idx).\n-\tls http_child/.git/objects/pack/* | grep -v \\.rev$ >filelist &&\n-\ttest_line_count = 6 filelist\n+\tls http_child/.git/objects/pack/*.pack >packlist &&\n+\tls http_child/.git/objects/pack/*.idx >idxlist &&\n+\ttest_line_count = 3 idxlist &&\n+\ttest_line_count = 3 packlist\n '\n\n> > diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\n> > index 308c1ef42c..100df52a71 100755\n> > --- a/t/t9300-fast-import.sh\n> > +++ b/t/t9300-fast-import.sh\n> > @@ -1629,7 +1629,7 @@ test_expect_success 'O: blank lines not necessary after other commands' '\n> > \tINPUT_END\n> >\n> > \tgit fast-import > -\ttest 8 = $(find .git/objects/pack -type f | grep -v multi-pack-index | wc -l) &&\n> > +\ttest 8 = $(find .git/objects/pack -type f \\( -name \"*.idx\" -o -name \"*.pack\" \\) | wc -l) &&\n>\n> Yay, there the existing multi-pack-index case is amended in a\n> future-proof way :)\n\nHere's actually a spot where I'm unhappy with the resulting complexity,\nand I think that it would be much cleaner if we did the same\npacklist+idxlist thing and then checked the line count of each.\n\nThanks,\nTaylor\n"},{"id":"414293","messageId":"48926ae1821f026c00c6237b771e0f9150b8b267.1610576805.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1610576805.git.me@ttaylorr.com","subject":"[PATCH v2 8/8] pack-revindex: ensure that on-disk reverse indexes are given precedence","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:28:38Z","receivedAt":"2021-01-14T02:04:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When an on-disk reverse index exists, there is no need to generate one\nin memory. In fact, doing so can be slow, and require large amounts of\nthe heap.\n\nLet's make sure that we treat the on-disk reverse index with precedence\n(i.e., that when it exists, we don't bother trying to generate an\nequivalent one in memory) by teaching Git how to conditionally die()\nwhen generating a reverse index in memory.\n\nThen, add a test to ensure that when (a) an on-disk reverse index\nexists, and (b) when setting GIT_TEST_REV_INDEX_DIE_IN_MEMORY, that we\ndo not die, implying that we read from the on-disk one.\n\nSigned-off-by: Taylor Blau \n---\n pack-revindex.c | 4 ++++\n pack-revindex.h | 1 +\n t/t5325-reverse-index.sh | 9 +++++++++\n 3 files changed, 14 insertions(+)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 369812dd21..f264319f34 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -2,6 +2,7 @@\n #include \"pack-revindex.h\"\n #include \"object-store.h\"\n #include \"packfile.h\"\n+#include \"config.h\"\n \n struct revindex_entry {\n \toff_t offset;\n@@ -166,6 +167,9 @@ static void create_pack_revindex(struct packed_git *p)\n \n static int load_pack_revindex_from_memory(struct packed_git *p)\n {\n+\tif (git_env_bool(GIT_TEST_REV_INDEX_DIE_IN_MEMORY, 0))\n+\t\tdie(\"dying as requested by '%s'\",\n+\t\t GIT_TEST_REV_INDEX_DIE_IN_MEMORY);\n \tif (open_pack_index(p))\n \t\treturn -1;\n \tcreate_pack_revindex(p);\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex 7237b2b6f8..97f5893d3a 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -17,6 +17,7 @@\n */\n \n #define GIT_TEST_WRITE_REV_INDEX \"GIT_TEST_WRITE_REV_INDEX\"\n+#define GIT_TEST_REV_INDEX_DIE_IN_MEMORY \"GIT_TEST_REV_INDEX_DIE_IN_MEMORY\"\n \n struct packed_git;\n \ndiff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh\nindex be452bb343..a344b18d7e 100755\n--- a/t/t5325-reverse-index.sh\n+++ b/t/t5325-reverse-index.sh\n@@ -85,4 +85,13 @@ test_expect_success 'pack-objects respects pack.writeReverseIndex' '\n \ttest_path_is_file pack-1-*.rev\n '\n \n+test_expect_success 'reverse index is not generated when available on disk' '\n+\ttest_index_pack true &&\n+\ttest_path_is_file $rev &&\n+\n+\tgit rev-parse HEAD >tip &&\n+\tGIT_TEST_REV_INDEX_DIE_IN_MEMORY=1 git cat-file \\\n+\t\t--batch-check=\"%(objectsize:disk)\" \n---\n t/t5319-multi-pack-index.sh | 5 +++--\n t/t5325-reverse-index.sh | 4 ++++\n t/t5604-clone-reference.sh | 2 +-\n t/t5702-protocol-v2.sh | 12 ++++++++----\n t/t6500-gc.sh | 6 +++---\n t/t9300-fast-import.sh | 5 ++++-\n 6 files changed, 23 insertions(+), 11 deletions(-)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 297de502a9..2fc3aadbd1 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -710,8 +710,9 @@ test_expect_success 'expire respects .keep files' '\n \t\tPACKA=$(ls .git/objects/pack/a-pack*\\.pack | sed s/\\.pack\\$//) &&\n \t\ttouch $PACKA.keep &&\n \t\tgit multi-pack-index expire &&\n-\t\tls -S .git/objects/pack/a-pack* | grep $PACKA >a-pack-files &&\n-\t\ttest_line_count = 3 a-pack-files &&\n+\t\ttest_path_is_file $PACKA.idx &&\n+\t\ttest_path_is_file $PACKA.keep &&\n+\t\ttest_path_is_file $PACKA.pack &&\n \t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n \t\ttest_line_count = 2 midx-list\n \t)\ndiff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh\nindex 87040263b7..be452bb343 100755\n--- a/t/t5325-reverse-index.sh\n+++ b/t/t5325-reverse-index.sh\n@@ -3,6 +3,10 @@\n test_description='on-disk reverse index'\n . ./test-lib.sh\n \n+# The below tests want control over the 'pack.writeReverseIndex' setting\n+# themselves to assert various combinations of it with other options.\n+sane_unset GIT_TEST_WRITE_REV_INDEX\n+\n packdir=.git/objects/pack\n \n test_expect_success 'setup' '\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 2f7be23044..7d93588982 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -326,7 +326,7 @@ test_expect_success SYMLINKS 'clone repo with symlinked or unknown files at obje\n \tfor raw in $(ls T*.raw)\n \tdo\n \t\tsed -e \"s!/../!/Y/!; s![0-9a-f]\\{38,\\}!Z!\" -e \"/commit-graph/d\" \\\n-\t\t -e \"/multi-pack-index/d\" <$raw >$raw.de-sha-1 &&\n+\t\t -e \"/multi-pack-index/d\" -e \"/rev/d\" <$raw >$raw.de-sha-1 &&\n \t\tsort $raw.de-sha-1 >$raw.de-sha || return 1\n \tdone &&\n \ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 7d5b17909b..73cd9e3ff6 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -848,8 +848,10 @@ test_expect_success 'part of packfile response provided as URI' '\n \ttest -f h2found &&\n \n \t# Ensure that there are exactly 6 files (3 .pack and 3 .idx).\n-\tls http_child/.git/objects/pack/* >filelist &&\n-\ttest_line_count = 6 filelist\n+\tls http_child/.git/objects/pack/*.pack >packlist &&\n+\tls http_child/.git/objects/pack/*.idx >idxlist &&\n+\ttest_line_count = 3 idxlist &&\n+\ttest_line_count = 3 packlist\n '\n \n test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n@@ -902,8 +904,10 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' '\n \t\tclone \"$HTTPD_URL/smart/http_parent\" http_child &&\n \n \t# Ensure that there are exactly 4 files (2 .pack and 2 .idx).\n-\tls http_child/.git/objects/pack/* >filelist &&\n-\ttest_line_count = 4 filelist\n+\tls http_child/.git/objects/pack/*.pack >packlist &&\n+\tls http_child/.git/objects/pack/*.idx >idxlist &&\n+\ttest_line_count = 2 idxlist &&\n+\ttest_line_count = 2 packlist\n '\n \n test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' '\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 4a3b8f48ac..f76586f808 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -106,17 +106,17 @@ test_expect_success 'auto gc with too many loose objects does not attempt to cre\n \ttest_commit \"$(test_oid obj2)\" &&\n \t# Our first gc will create a pack; our second will create a second pack\n \tgit gc --auto &&\n-\tls .git/objects/pack | sort >existing_packs &&\n+\tls .git/objects/pack/pack-*.pack | sort >existing_packs &&\n \ttest_commit \"$(test_oid obj3)\" &&\n \ttest_commit \"$(test_oid obj4)\" &&\n \n \tgit gc --auto 2>err &&\n \ttest_i18ngrep ! \"^warning:\" err &&\n-\tls .git/objects/pack/ | sort >post_packs &&\n+\tls .git/objects/pack/pack-*.pack | sort >post_packs &&\n \tcomm -1 -3 existing_packs post_packs >new &&\n \tcomm -2 -3 existing_packs post_packs >del &&\n \ttest_line_count = 0 del && # No packs are deleted\n-\ttest_line_count = 2 new # There is one new pack and its .idx\n+\ttest_line_count = 1 new # There is one new pack\n '\n \n test_expect_success 'gc --no-quiet' '\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 308c1ef42c..2cc1f43c1b 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -1629,7 +1629,10 @@ test_expect_success 'O: blank lines not necessary after other commands' '\n \tINPUT_END\n \n \tgit fast-import packlist &&\n+\tls -la .git/objects/pack/pack-*.pack >idxlist &&\n+\ttest_line_count = 4 idxlist &&\n+\ttest_line_count = 4 packlist &&\n \ttest $(git rev-parse refs/tags/O3-2nd) = $(git rev-parse O3^) &&\n \tgit log --reverse --pretty=oneline O3 | sed s/^.*z// >actual &&\n \ttest_cmp expect actual\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414309","messageId":"6742c15c84bafbcc1c06e2633de51dcda63e3314.1610576805.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1610576805.git.me@ttaylorr.com","subject":"[PATCH v2 1/8] packfile: prepare for the existence of '*.rev' files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:28:06Z","receivedAt":"2021-01-14T02:06:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Specify the format of the on-disk reverse index 'pack-*.rev' file, as\nwell as prepare the code for the existence of such files.\n\nThe reverse index maps from pack relative positions (i.e., an index into\nthe array of object which is sorted by their offsets within the\npackfile) to their position within the 'pack-*.idx' file. Today, this is\ndone by building up a list of (off_t, uint32_t) tuples for each object\n(the off_t corresponding to that object's offset, and the uint32_t\ncorresponding to its position in the index). To convert between pack and\nindex position quickly, this array of tuples is radix sorted based on\nits offset.\n\nThis has two major drawbacks:\n\nFirst, the in-memory cost scales linearly with the number of objects in\na pack. Each 'struct revindex_entry' is sizeof(off_t) +\nsizeof(uint32_t) + padding bytes for a total of 16.\n\nTo observe this, force Git to load the reverse index by, for e.g.,\nrunning 'git cat-file --batch-check=\"%(objectsize:disk)\"'. When asking\nfor a single object in a fresh clone of the kernel, Git needs to\nallocate 120+ MB of memory in order to hold the reverse index in memory.\n\nSecond, the cost to sort also scales with the size of the pack.\nLuckily, this is a linear function since 'load_pack_revindex()' uses a\nradix sort, but this cost still must be paid once per pack per process.\n\nAs an example, it takes ~60x longer to print the _size_ of an object as\nit does to print that entire object's _contents_:\n\n Benchmark #1: git.compile cat-file --batch /dev/null\n Time (mean ± σ): 22.6 ms ± 0.5 ms [User: 2.4 ms, System: 7.9 ms]\n Range (min … max): 21.4 ms … 23.5 ms 41 runs\n\n Benchmark #2: git.compile cat-file --batch /dev/null\n Time (mean ± σ): 17.2 ms ± 0.7 ms [User: 2.8 ms, System: 5.5 ms]\n Range (min … max): 15.6 ms … 18.2 ms 45 runs\n\n(Numbers taken in the kernel after cheating and using the next patch to\ngenerate a reverse index). There are a couple of approaches to improve\ncold cache performance not pursued here:\n\n - We could include the object offsets in the reverse index format.\n Predictably, this does result in fewer page faults, but it triples\n the size of the file, while simultaneously duplicating a ton of data\n already available in the .idx file. (This was the original way I\n implemented the format, and it did show\n `--batch-check='%(objectsize:disk)'` winning out against `--batch`.)\n\n On the other hand, this increase in size also results in a large\n block-cache footprint, which could potentially hurt other workloads.\n\n - We could store the mapping from pack to index position in more\n cache-friendly way, like constructing a binary search tree from the\n table and writing the values in breadth-first order. This would\n result in much better locality, but the price you pay is trading\n O(1) lookup in 'pack_pos_to_index()' for an O(log n) one (since you\n can no longer directly index the table).\n\nSo, neither of these approaches are taken here. (Thankfully, the format\nis versioned, so we are free to pursue these in the future.) But, cold\ncache performance likely isn't interesting outside of one-off cases like\nasking for the size of an object directly. In real-world usage, Git is\noften performing many operations in the revindex,\n\nThe trade-off is worth it, since we will avoid the vast majority of the\ncost of generating the revindex that the extra pointer chase will look\nlike noise in the following patch's benchmarks.\n\nThis patch describes the format and prepares callers (like in\npack-revindex.c) to be able to read *.rev files once they exist. An\nimplementation of the writer will appear in the next patch, and callers\nwill gradually begin to start using the writer in the patches that\nfollow after that.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/technical/pack-format.txt | 17 ++++\n builtin/repack.c | 1 +\n object-store.h | 3 +\n pack-revindex.c | 112 +++++++++++++++++++++---\n pack-revindex.h | 7 +-\n packfile.c | 13 ++-\n packfile.h | 1 +\n tmp-objdir.c | 4 +-\n 8 files changed, 145 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex f96b2e605f..9593f8bc68 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -259,6 +259,23 @@ Pack file entry: <+\n \n Index checksum of all of the above.\n \n+== pack-*.rev files have the format:\n+\n+ - A 4-byte magic number '0x52494458' ('RIDX').\n+\n+ - A 4-byte version identifier (= 1)\n+\n+ - A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256)\n+\n+ - A table of index positions, sorted by their corresponding offsets in the\n+ packfile.\n+\n+ - A trailer, containing a:\n+\n+ checksum of the corresponding packfile, and\n+\n+ a checksum of all of the above.\n+\n == multi-pack-index (MIDX) files have the following format:\n \n The multi-pack-index files refer to multiple pack-files and loose objects.\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 279be11a16..8d643ddcb9 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -208,6 +208,7 @@ static struct {\n } exts[] = {\n \t{\".pack\"},\n \t{\".idx\"},\n+\t{\".rev\", 1},\n \t{\".bitmap\", 1},\n \t{\".promisor\", 1},\n };\ndiff --git a/object-store.h b/object-store.h\nindex c4fc9dd74e..3fbf11280f 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -85,6 +85,9 @@ struct packed_git {\n \t\t multi_pack_index:1;\n \tunsigned char hash[GIT_MAX_RAWSZ];\n \tstruct revindex_entry *revindex;\n+\tconst void *revindex_data;\n+\tconst void *revindex_map;\n+\tsize_t revindex_size;\n \t/* something like \".git/objects/pack/xxxxx.pack\" */\n \tchar pack_name[FLEX_ARRAY]; /* more */\n };\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 5e69bc7372..369812dd21 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -164,16 +164,98 @@ static void create_pack_revindex(struct packed_git *p)\n \tsort_revindex(p->revindex, num_ent, p->pack_size);\n }\n \n-int load_pack_revindex(struct packed_git *p)\n+static int load_pack_revindex_from_memory(struct packed_git *p)\n {\n-\tif (!p->revindex) {\n-\t\tif (open_pack_index(p))\n-\t\t\treturn -1;\n-\t\tcreate_pack_revindex(p);\n-\t}\n+\tif (open_pack_index(p))\n+\t\treturn -1;\n+\tcreate_pack_revindex(p);\n \treturn 0;\n }\n \n+static char *pack_revindex_filename(struct packed_git *p)\n+{\n+\tsize_t len;\n+\tif (!strip_suffix(p->pack_name, \".pack\", &len))\n+\t\tBUG(\"pack_name does not end in .pack\");\n+\treturn xstrfmt(\"%.*s.rev\", (int)len, p->pack_name);\n+}\n+\n+#define RIDX_MIN_SIZE (12 + (2 * the_hash_algo->rawsz))\n+\n+static int load_revindex_from_disk(char *revindex_name,\n+\t\t\t\t uint32_t num_objects,\n+\t\t\t\t const void **data, size_t *len)\n+{\n+\tint fd, ret = 0;\n+\tstruct stat st;\n+\tsize_t revindex_size;\n+\n+\tfd = git_open(revindex_name);\n+\n+\tif (fd < 0) {\n+\t\tret = -1;\n+\t\tgoto cleanup;\n+\t}\n+\tif (fstat(fd, &st)) {\n+\t\tret = error_errno(_(\"failed to read %s\"), revindex_name);\n+\t\tgoto cleanup;\n+\t}\n+\n+\trevindex_size = xsize_t(st.st_size);\n+\n+\tif (revindex_size < RIDX_MIN_SIZE) {\n+\t\tret = error(_(\"reverse-index file %s is too small\"), revindex_name);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (revindex_size - RIDX_MIN_SIZE != st_mult(sizeof(uint32_t), num_objects)) {\n+\t\tret = error(_(\"reverse-index file %s is corrupt\"), revindex_name);\n+\t\tgoto cleanup;\n+\t}\n+\n+\t*len = revindex_size;\n+\t*data = xmmap(NULL, revindex_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\n+cleanup:\n+\tclose(fd);\n+\treturn ret;\n+}\n+\n+static int load_pack_revindex_from_disk(struct packed_git *p)\n+{\n+\tchar *revindex_name;\n+\tint ret;\n+\tif (open_pack_index(p))\n+\t\treturn -1;\n+\n+\trevindex_name = pack_revindex_filename(p);\n+\n+\tret = load_revindex_from_disk(revindex_name,\n+\t\t\t\t p->num_objects,\n+\t\t\t\t &p->revindex_map,\n+\t\t\t\t &p->revindex_size);\n+\tif (ret)\n+\t\tgoto cleanup;\n+\n+\tp->revindex_data = (char *)p->revindex_map + 12;\n+\n+cleanup:\n+\tfree(revindex_name);\n+\treturn ret;\n+}\n+\n+int load_pack_revindex(struct packed_git *p)\n+{\n+\tif (p->revindex || p->revindex_data)\n+\t\treturn 0;\n+\n+\tif (!load_pack_revindex_from_disk(p))\n+\t\treturn 0;\n+\telse if (!load_pack_revindex_from_memory(p))\n+\t\treturn 0;\n+\treturn -1;\n+}\n+\n int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n {\n \tunsigned lo, hi;\n@@ -203,18 +285,28 @@ int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n \n uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos)\n {\n-\tif (!p->revindex)\n+\tif (!(p->revindex || p->revindex_data))\n \t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n \tif (p->num_objects <= pos)\n \t\tBUG(\"pack_pos_to_index: out-of-bounds object at %\"PRIu32, pos);\n-\treturn p->revindex[pos].nr;\n+\n+\tif (p->revindex)\n+\t\treturn p->revindex[pos].nr;\n+\telse\n+\t\treturn get_be32((char *)p->revindex_data + (pos * sizeof(uint32_t)));\n }\n \n off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n {\n-\tif (!p->revindex)\n+\tif (!(p->revindex || p->revindex_data))\n \t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n \tif (p->num_objects < pos)\n \t\tBUG(\"pack_pos_to_offset: out-of-bounds object at %\"PRIu32, pos);\n-\treturn p->revindex[pos].offset;\n+\n+\tif (p->revindex)\n+\t\treturn p->revindex[pos].offset;\n+\telse if (pos == p->num_objects)\n+\t\treturn p->pack_size - the_hash_algo->rawsz;\n+\telse\n+\t\treturn nth_packed_object_offset(p, pack_pos_to_index(p, pos));\n }\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex 6e0320b08b..01622cf21a 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -21,6 +21,9 @@ struct packed_git;\n /*\n * load_pack_revindex populates the revindex's internal data-structures for the\n * given pack, returning zero on success and a negative value otherwise.\n+ *\n+ * If a '.rev' file is present, it is checked for consistency, mmap'd, and\n+ * pointers are assigned into it (instead of using the in-memory variant).\n */\n int load_pack_revindex(struct packed_git *p);\n \n@@ -55,7 +58,9 @@ uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);\n * If the reverse index has not yet been loaded, or the position is out of\n * bounds, this function aborts.\n *\n- * This function runs in constant time.\n+ * This function runs in constant time under both in-memory and on-disk reverse\n+ * indexes, but an additional step is taken to consult the corresponding .idx\n+ * file when using the on-disk format.\n */\n off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos);\n \ndiff --git a/packfile.c b/packfile.c\nindex 7bb1750934..b04eac9286 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -324,11 +324,21 @@ void close_pack_index(struct packed_git *p)\n \t}\n }\n \n+void close_pack_revindex(struct packed_git *p) {\n+\tif (!p->revindex_map)\n+\t\treturn;\n+\n+\tmunmap((void *)p->revindex_map, p->revindex_size);\n+\tp->revindex_map = NULL;\n+\tp->revindex_data = NULL;\n+}\n+\n void close_pack(struct packed_git *p)\n {\n \tclose_pack_windows(p);\n \tclose_pack_fd(p);\n \tclose_pack_index(p);\n+\tclose_pack_revindex(p);\n }\n \n void close_object_store(struct raw_object_store *o)\n@@ -351,7 +361,7 @@ void close_object_store(struct raw_object_store *o)\n \n void unlink_pack_path(const char *pack_name, int force_delete)\n {\n-\tstatic const char *exts[] = {\".pack\", \".idx\", \".keep\", \".bitmap\", \".promisor\"};\n+\tstatic const char *exts[] = {\".pack\", \".idx\", \".rev\", \".keep\", \".bitmap\", \".promisor\"};\n \tint i;\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t plen;\n@@ -853,6 +863,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tif (!strcmp(file_name, \"multi-pack-index\"))\n \t\treturn;\n \tif (ends_with(file_name, \".idx\") ||\n+\t ends_with(file_name, \".rev\") ||\n \t ends_with(file_name, \".pack\") ||\n \t ends_with(file_name, \".bitmap\") ||\n \t ends_with(file_name, \".keep\") ||\ndiff --git a/packfile.h b/packfile.h\nindex a58fc738e0..4cfec9e8d3 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -90,6 +90,7 @@ uint32_t get_pack_fanout(struct packed_git *p, uint32_t value);\n \n unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n void close_pack_windows(struct packed_git *);\n+void close_pack_revindex(struct packed_git *);\n void close_pack(struct packed_git *);\n void close_object_store(struct raw_object_store *o);\n void unuse_pack(struct pack_window **);\ndiff --git a/tmp-objdir.c b/tmp-objdir.c\nindex 42ed4db5d3..da414df14f 100644\n--- a/tmp-objdir.c\n+++ b/tmp-objdir.c\n@@ -187,7 +187,9 @@ static int pack_copy_priority(const char *name)\n \t\treturn 2;\n \tif (ends_with(name, \".idx\"))\n \t\treturn 3;\n-\treturn 4;\n+\tif (ends_with(name, \".rev\"))\n+\t\treturn 4;\n+\treturn 5;\n }\n \n static int pack_copy_cmp(const char *a, const char *b)\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414311","messageId":"cover.1610576805.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1610129989.git.me@ttaylorr.com","subject":"[PATCH v2 0/8] pack-revindex: introduce on-disk '.rev' format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:28:01Z","receivedAt":"2021-01-14T02:07:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi,\n\nThis is the second of two series to implement support for an on-disk format for\nstoring the reverse index. Note that this depends on the patches in the previous\nseries [1], which was recently updated).\n\nThis version is largely unchanged from the original, with the following\nexceptions:\n\n - It has been rebased onto the patches in the first series.\n - The operands of two comparisons in 'offset_to_pack_pos()' were swapped so\n that the smaller of the two appears on the left-hand side of the comparison.\n - A brown-paper-bag bug was fixed in tests so that they pass on Windows (last\n night's integration broke 'seen' on Windows).\n - The GIT_TEST_WRITE_REV_INDEX mode was enabled in the \"all-features\" test.\n\nThanks in advance for your review.\n\n[1]: https://lore.kernel.org/git/cover.1610129796.git.me@ttaylorr.com/\n\nTaylor Blau (8):\n packfile: prepare for the existence of '*.rev' files\n pack-write.c: prepare to write 'pack-*.rev' files\n builtin/index-pack.c: write reverse indexes\n builtin/pack-objects.c: respect 'pack.writeReverseIndex'\n Documentation/config/pack.txt: advertise 'pack.writeReverseIndex'\n t: prepare for GIT_TEST_WRITE_REV_INDEX\n t: support GIT_TEST_WRITE_REV_INDEX\n pack-revindex: ensure that on-disk reverse indexes are given\n precedence\n\n Documentation/config/pack.txt | 7 ++\n Documentation/git-index-pack.txt | 20 ++--\n Documentation/technical/pack-format.txt | 17 ++++\n builtin/index-pack.c | 67 +++++++++++--\n builtin/pack-objects.c | 9 ++\n builtin/repack.c | 1 +\n ci/run-build-and-tests.sh | 1 +\n object-store.h | 3 +\n pack-revindex.c | 116 ++++++++++++++++++++--\n pack-revindex.h | 10 +-\n pack-write.c | 123 +++++++++++++++++++++++-\n pack.h | 4 +\n packfile.c | 13 ++-\n packfile.h | 1 +\n t/README | 3 +\n t/t5319-multi-pack-index.sh | 5 +-\n t/t5325-reverse-index.sh | 97 +++++++++++++++++++\n t/t5604-clone-reference.sh | 2 +-\n t/t5702-protocol-v2.sh | 12 ++-\n t/t6500-gc.sh | 6 +-\n t/t9300-fast-import.sh | 5 +-\n tmp-objdir.c | 4 +-\n 22 files changed, 485 insertions(+), 41 deletions(-)\n create mode 100755 t/t5325-reverse-index.sh\n\n -: ---------- > 1: e1aa89244a pack-revindex: introduce a new API\n -: ---------- > 2: 0fca7d5812 write_reuse_object(): convert to new revindex API\n -: ---------- > 3: 7676822a54 write_reused_pack_one(): convert to new revindex API\n -: ---------- > 4: dd7133fdb7 write_reused_pack_verbatim(): convert to new revindex API\n -: ---------- > 5: 8e93ca3886 check_object(): convert to new revindex API\n -: ---------- > 6: 084bbf2145 bitmap_position_packfile(): convert to new revindex API\n -: ---------- > 7: 68794e9484 show_objects_for_type(): convert to new revindex API\n -: ---------- > 8: 31ac6f5703 get_size_by_pos(): convert to new revindex API\n -: ---------- > 9: acd80069a2 try_partial_reuse(): convert to new revindex API\n -: ---------- > 10: 569acdca7f rebuild_existing_bitmaps(): convert to new revindex API\n -: ---------- > 11: 9881637724 get_delta_base_oid(): convert to new revindex API\n -: ---------- > 12: df8bb571a5 retry_bad_packed_offset(): convert to new revindex API\n -: ---------- > 13: 41b2e00947 packed_object_info(): convert to new revindex API\n -: ---------- > 14: 8ad49d231f unpack_entry(): convert to new revindex API\n -: ---------- > 15: e757476351 for_each_object_in_pack(): convert to new revindex API\n -: ---------- > 16: a500311e33 builtin/gc.c: guess the size of the revindex\n -: ---------- > 17: 67d14da04a pack-revindex: remove unused 'find_pack_revindex()'\n -: ---------- > 18: 3b5c92be68 pack-revindex: remove unused 'find_revindex_position()'\n -: ---------- > 19: cabafce4a1 pack-revindex: hide the definition of 'revindex_entry'\n -: ---------- > 20: 8400ff6c96 pack-revindex.c: avoid direct revindex access in 'offset_to_pack_pos()'\n 1: ddf47a0a50 ! 21: 6742c15c84 packfile: prepare for the existence of '*.rev' files\n @@ pack-revindex.c: static void create_pack_revindex(struct packed_git *p)\n +\n int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n {\n - \tint lo = 0;\n + \tunsigned lo, hi;\n @@ pack-revindex.c: int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n\n uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos)\n @@ pack-revindex.c: int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_\n -\tif (!p->revindex)\n +\tif (!(p->revindex || p->revindex_data))\n \t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n - \tif (pos >= p->num_objects)\n + \tif (p->num_objects <= pos)\n \t\tBUG(\"pack_pos_to_index: out-of-bounds object at %\"PRIu32, pos);\n -\treturn p->revindex[pos].nr;\n +\n @@ pack-revindex.c: int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_\n -\tif (!p->revindex)\n +\tif (!(p->revindex || p->revindex_data))\n \t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n - \tif (pos > p->num_objects)\n + \tif (p->num_objects < pos)\n \t\tBUG(\"pack_pos_to_offset: out-of-bounds object at %\"PRIu32, pos);\n -\treturn p->revindex[pos].offset;\n +\n @@ pack-revindex.c: int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_\n +\t\treturn nth_packed_object_offset(p, pack_pos_to_index(p, pos));\n }\n\n + ## pack-revindex.h ##\n +@@ pack-revindex.h: struct packed_git;\n + /*\n + * load_pack_revindex populates the revindex's internal data-structures for the\n + * given pack, returning zero on success and a negative value otherwise.\n ++ *\n ++ * If a '.rev' file is present, it is checked for consistency, mmap'd, and\n ++ * pointers are assigned into it (instead of using the in-memory variant).\n + */\n + int load_pack_revindex(struct packed_git *p);\n +\n +@@ pack-revindex.h: uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);\n + * If the reverse index has not yet been loaded, or the position is out of\n + * bounds, this function aborts.\n + *\n +- * This function runs in constant time.\n ++ * This function runs in constant time under both in-memory and on-disk reverse\n ++ * indexes, but an additional step is taken to consult the corresponding .idx\n ++ * file when using the on-disk format.\n + */\n + off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos);\n +\n +\n ## packfile.c ##\n @@ packfile.c: void close_pack_index(struct packed_git *p)\n \t}\n 2: 88393e2662 = 22: 8648c87fa7 pack-write.c: prepare to write 'pack-*.rev' files\n 3: b0a7329824 ! 23: 5b18ada611 builtin/index-pack.c: write reverse indexes\n @@ t/t5325-reverse-index.sh (new)\n +\trm -f $rev &&\n +\tconf=$1 &&\n +\tshift &&\n ++\t# remove the index since Windows won't overwrite an existing file\n ++\trm $packdir/pack-$pack.idx &&\n +\tgit -c pack.writeReverseIndex=$conf index-pack \"$@\" \\\n +\t\t$packdir/pack-$pack.pack\n +}\n 4: e297a31875 = 24: 68bde3ea97 builtin/pack-objects.c: respect 'pack.writeReverseIndex'\n 5: 5d3e96a498 = 25: 38a253d0ce Documentation/config/pack.txt: advertise 'pack.writeReverseIndex'\n 6: 2288571fbe < -: ---------- t: prepare for GIT_TEST_WRITE_REV_INDEX\n -: ---------- > 26: 12cdf2d67a t: prepare for GIT_TEST_WRITE_REV_INDEX\n 7: 3525c4d114 ! 27: 6b647d9775 t: support GIT_TEST_WRITE_REV_INDEX\n @@ Commit message\n\n Add a new option that unconditionally enables the pack.writeReverseIndex\n setting in order to run the whole test suite in a mode that generates\n - on-disk reverse indexes.\n + on-disk reverse indexes. Additionally, enable this mode in the second\n + run of tests under linux-gcc in 'ci/run-build-and-tests.sh'.\n\n Once on-disk reverse indexes are proven out over several releases, we\n can change the default value of that configuration to 'true', and drop\n @@ builtin/pack-objects.c: int cmd_pack_objects(int argc, const char **argv, const\n \tprogress = isatty(2);\n \targc = parse_options(argc, argv, prefix, pack_objects_options,\n\n + ## ci/run-build-and-tests.sh ##\n +@@ ci/run-build-and-tests.sh: linux-gcc)\n + \texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n + \texport GIT_TEST_MULTI_PACK_INDEX=1\n + \texport GIT_TEST_ADD_I_USE_BUILTIN=1\n ++\texport GIT_TEST_WRITE_REV_INDEX=1\n + \tmake test\n + \t;;\n + linux-clang)\n +\n ## pack-revindex.h ##\n @@\n - #ifndef PACK_REVINDEX_H\n - #define PACK_REVINDEX_H\n + * can be found\n + */\n\n +#define GIT_TEST_WRITE_REV_INDEX \"GIT_TEST_WRITE_REV_INDEX\"\n +\n struct packed_git;\n\n - int load_pack_revindex(struct packed_git *p);\n + /*\n\n ## t/README ##\n @@ t/README: GIT_TEST_DEFAULT_HASH= specifies which hash algorithm to\n 8: 6e580d43d1 ! 28: 48926ae182 pack-revindex: ensure that on-disk reverse indexes are given precedence\n @@ pack-revindex.c: static void create_pack_revindex(struct packed_git *p)\n\n ## pack-revindex.h ##\n @@\n - #define PACK_REVINDEX_H\n + */\n\n #define GIT_TEST_WRITE_REV_INDEX \"GIT_TEST_WRITE_REV_INDEX\"\n +#define GIT_TEST_REV_INDEX_DIE_IN_MEMORY \"GIT_TEST_REV_INDEX_DIE_IN_MEMORY\"\n @@ t/t5325-reverse-index.sh: test_expect_success 'pack-objects respects pack.writeR\n '\n\n +test_expect_success 'reverse index is not generated when available on disk' '\n -+\tgit index-pack --rev-index $packdir/pack-$pack.pack &&\n ++\ttest_index_pack true &&\n ++\ttest_path_is_file $rev &&\n +\n +\tgit rev-parse HEAD >tip &&\n +\tGIT_TEST_REV_INDEX_DIE_IN_MEMORY=1 git cat-file \\\n--\n2.30.0.138.g6d7191ea01\n"},{"id":"414314","messageId":"38a253d0ce29868000d7659158b1a1d5b841cdc7.1610576805.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1610576805.git.me@ttaylorr.com","subject":"[PATCH v2 5/8] Documentation/config/pack.txt: advertise 'pack.writeReverseIndex'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:28:24Z","receivedAt":"2021-01-14T02:09:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the pack.writeReverseIndex configuration is respected in both\n'git index-pack' and 'git pack-objects' (and therefore, all of their\ncallers), we can safely advertise it for use in the git-config manual.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/pack.txt | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\nindex 837f1b1679..3da4ea98e2 100644\n--- a/Documentation/config/pack.txt\n+++ b/Documentation/config/pack.txt\n@@ -133,3 +133,10 @@ pack.writeBitmapHashCache::\n \tbetween an older, bitmapped pack and objects that have been\n \tpushed since the last gc). The downside is that it consumes 4\n \tbytes per object of disk space. Defaults to true.\n+\n+pack.writeReverseIndex::\n+\tWhen true, git will write a corresponding .rev file (see:\n+\tlink:../technical/pack-format.html[Documentation/technical/pack-format.txt])\n+\tfor each new packfile that it writes in all places except for\n+\tlinkgit:git-fast-import[1] and in the bulk checkin mechanism.\n+\tDefaults to false.\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414313","messageId":"68bde3ea972f5b3753d7e9063d0490c67c74709b.1610576805.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1610576805.git.me@ttaylorr.com","subject":"[PATCH v2 4/8] builtin/pack-objects.c: respect 'pack.writeReverseIndex'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:28:19Z","receivedAt":"2021-01-14T02:09:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that we have an implementation that can write the new reverse index\nformat, enable writing a .rev file in 'git pack-objects' by consulting\nthe pack.writeReverseIndex configuration variable.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 7 +++++++\n t/t5325-reverse-index.sh | 13 +++++++++++++\n 2 files changed, 20 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5b0c4489e2..d784569200 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2955,6 +2955,13 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\t pack_idx_opts.version);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"pack.writereverseindex\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\tpack_idx_opts.flags |= WRITE_REV;\n+\t\telse\n+\t\t\tpack_idx_opts.flags &= ~WRITE_REV;\n+\t\treturn 0;\n+\t}\n \tif (!strcmp(k, \"uploadpack.blobpackfileuri\")) {\n \t\tstruct configured_exclusion *ex = xmalloc(sizeof(*ex));\n \t\tconst char *oid_end, *pack_end;\ndiff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh\nindex 2dae213126..87040263b7 100755\n--- a/t/t5325-reverse-index.sh\n+++ b/t/t5325-reverse-index.sh\n@@ -68,4 +68,17 @@ test_expect_success 'index-pack infers reverse index name with -o' '\n \ttest_path_is_file other.rev\n '\n \n+test_expect_success 'pack-objects respects pack.writeReverseIndex' '\n+\ttest_when_finished \"rm -fr pack-1-*\" &&\n+\n+\tgit -c pack.writeReverseIndex= pack-objects --all pack-1 &&\n+\ttest_path_is_missing pack-1-*.rev &&\n+\n+\tgit -c pack.writeReverseIndex=false pack-objects --all pack-1 &&\n+\ttest_path_is_missing pack-1-*.rev &&\n+\n+\tgit -c pack.writeReverseIndex=true pack-objects --all pack-1 &&\n+\ttest_path_is_file pack-1-*.rev\n+'\n+\n test_done\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414315","messageId":"8648c87fa71aec427dc11afcba6548fb66a1413b.1610576805.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1610576805.git.me@ttaylorr.com","subject":"[PATCH v2 2/8] pack-write.c: prepare to write 'pack-*.rev' files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:28:11Z","receivedAt":"2021-01-14T02:09:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This patch prepares for callers to be able to write reverse index files\nto disk.\n\nIt adds the necessary machinery to write a format-compliant .rev file\nfrom within 'write_rev_file()', which is called from\n'finish_tmp_packfile()'.\n\nSigned-off-by: Taylor Blau \n---\n pack-write.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++-\n pack.h | 4 ++\n 2 files changed, 126 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-write.c b/pack-write.c\nindex 3513665e1e..68db5a9edf 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -166,6 +166,116 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec\n \treturn index_name;\n }\n \n+static int pack_order_cmp(const void *va, const void *vb, void *ctx)\n+{\n+\tstruct pack_idx_entry **objects = ctx;\n+\n+\toff_t oa = objects[*(uint32_t*)va]->offset;\n+\toff_t ob = objects[*(uint32_t*)vb]->offset;\n+\n+\tif (oa < ob)\n+\t\treturn -1;\n+\tif (oa > ob)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+#define RIDX_SIGNATURE 0x52494458 /* \"RIDX\" */\n+#define RIDX_VERSION 1\n+\n+static void write_rev_header(struct hashfile *f)\n+{\n+\tuint32_t oid_version;\n+\tswitch (hash_algo_by_ptr(the_hash_algo)) {\n+\tcase GIT_HASH_SHA1:\n+\t\toid_version = 1;\n+\t\tbreak;\n+\tcase GIT_HASH_SHA256:\n+\t\toid_version = 2;\n+\t\tbreak;\n+\tdefault:\n+\t\tdie(\"write_rev_header: unknown hash version\");\n+\t}\n+\n+\thashwrite_be32(f, RIDX_SIGNATURE);\n+\thashwrite_be32(f, RIDX_VERSION);\n+\thashwrite_be32(f, oid_version);\n+}\n+\n+static void write_rev_index_positions(struct hashfile *f,\n+\t\t\t\t struct pack_idx_entry **objects,\n+\t\t\t\t uint32_t nr_objects)\n+{\n+\tuint32_t *pack_order;\n+\tuint32_t i;\n+\n+\tALLOC_ARRAY(pack_order, nr_objects);\n+\tfor (i = 0; i < nr_objects; i++)\n+\t\tpack_order[i] = i;\n+\tQSORT_S(pack_order, nr_objects, pack_order_cmp, objects);\n+\n+\tfor (i = 0; i < nr_objects; i++)\n+\t\thashwrite_be32(f, pack_order[i]);\n+\n+\tfree(pack_order);\n+}\n+\n+static void write_rev_trailer(struct hashfile *f, const unsigned char *hash)\n+{\n+\thashwrite(f, hash, the_hash_algo->rawsz);\n+}\n+\n+const char *write_rev_file(const char *rev_name,\n+\t\t\t struct pack_idx_entry **objects,\n+\t\t\t uint32_t nr_objects,\n+\t\t\t const unsigned char *hash,\n+\t\t\t unsigned flags)\n+{\n+\tstruct hashfile *f;\n+\tint fd;\n+\n+\tif ((flags & WRITE_REV) && (flags & WRITE_REV_VERIFY))\n+\t\tdie(_(\"cannot both write and verify reverse index\"));\n+\n+\tif (flags & WRITE_REV) {\n+\t\tif (!rev_name) {\n+\t\t\tstruct strbuf tmp_file = STRBUF_INIT;\n+\t\t\tfd = odb_mkstemp(&tmp_file, \"pack/tmp_rev_XXXXXX\");\n+\t\t\trev_name = strbuf_detach(&tmp_file, NULL);\n+\t\t} else {\n+\t\t\tunlink(rev_name);\n+\t\t\tfd = open(rev_name, O_CREAT|O_EXCL|O_WRONLY, 0600);\n+\t\t\tif (fd < 0)\n+\t\t\t\tdie_errno(\"unable to create '%s'\", rev_name);\n+\t\t}\n+\t\tf = hashfd(fd, rev_name);\n+\t} else if (flags & WRITE_REV_VERIFY) {\n+\t\tstruct stat statbuf;\n+\t\tif (stat(rev_name, &statbuf)) {\n+\t\t\tif (errno == ENOENT) {\n+\t\t\t\t/* .rev files are optional */\n+\t\t\t\treturn NULL;\n+\t\t\t} else\n+\t\t\t\tdie_errno(_(\"could not stat: %s\"), rev_name);\n+\t\t}\n+\t\tf = hashfd_check(rev_name);\n+\t} else\n+\t\treturn NULL;\n+\n+\twrite_rev_header(f);\n+\n+\twrite_rev_index_positions(f, objects, nr_objects);\n+\twrite_rev_trailer(f, hash);\n+\n+\tif (rev_name && adjust_shared_perm(rev_name) < 0)\n+\t\tdie(_(\"failed to make %s readable\"), rev_name);\n+\n+\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |\n+\t\t\t\t ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));\n+\n+\treturn rev_name;\n+}\n+\n off_t write_pack_header(struct hashfile *f, uint32_t nr_entries)\n {\n \tstruct pack_header hdr;\n@@ -341,7 +451,7 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \t\t\t struct pack_idx_option *pack_idx_opts,\n \t\t\t unsigned char hash[])\n {\n-\tconst char *idx_tmp_name;\n+\tconst char *idx_tmp_name, *rev_tmp_name = NULL;\n \tint basename_len = name_buffer->len;\n \n \tif (adjust_shared_perm(pack_tmp_name))\n@@ -352,6 +462,9 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \tif (adjust_shared_perm(idx_tmp_name))\n \t\tdie_errno(\"unable to make temporary index file readable\");\n \n+\trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n+\t\t\t\t pack_idx_opts->flags);\n+\n \tstrbuf_addf(name_buffer, \"%s.pack\", hash_to_hex(hash));\n \n \tif (rename(pack_tmp_name, name_buffer->buf))\n@@ -365,5 +478,13 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \n \tstrbuf_setlen(name_buffer, basename_len);\n \n+\tif (rev_tmp_name) {\n+\t\tstrbuf_addf(name_buffer, \"%s.rev\", hash_to_hex(hash));\n+\t\tif (rename(rev_tmp_name, name_buffer->buf))\n+\t\t\tdie_errno(\"unable to rename temporary reverse-index file\");\n+\t}\n+\n+\tstrbuf_setlen(name_buffer, basename_len);\n+\n \tfree((void *)idx_tmp_name);\n }\ndiff --git a/pack.h b/pack.h\nindex 9fc0945ac9..30439e0784 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -42,6 +42,8 @@ struct pack_idx_option {\n \t/* flag bits */\n #define WRITE_IDX_VERIFY 01 /* verify only, do not write the idx file */\n #define WRITE_IDX_STRICT 02\n+#define WRITE_REV 04\n+#define WRITE_REV_VERIFY 010\n \n \tuint32_t version;\n \tuint32_t off32_limit;\n@@ -87,6 +89,8 @@ off_t write_pack_header(struct hashfile *f, uint32_t);\n void fixup_pack_header_footer(int, unsigned char *, const char *, uint32_t, unsigned char *, off_t);\n char *index_pack_lockfile(int fd);\n \n+const char *write_rev_file(const char *rev_name, struct pack_idx_entry **objects, uint32_t nr_objects, const unsigned char *hash, unsigned flags);\n+\n /*\n * The \"hdr\" output buffer should be at least this big, which will handle sizes\n * up to 2^67.\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414316","messageId":"6b647d9775fd97238f1170dc998770c196a92cc4.1610576805.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1610576805.git.me@ttaylorr.com","subject":"[PATCH v2 7/8] t: support GIT_TEST_WRITE_REV_INDEX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:28:33Z","receivedAt":"2021-01-14T02:09:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Add a new option that unconditionally enables the pack.writeReverseIndex\nsetting in order to run the whole test suite in a mode that generates\non-disk reverse indexes. Additionally, enable this mode in the second\nrun of tests under linux-gcc in 'ci/run-build-and-tests.sh'.\n\nOnce on-disk reverse indexes are proven out over several releases, we\ncan change the default value of that configuration to 'true', and drop\nthis patch.\n\nSigned-off-by: Taylor Blau \n---\n builtin/index-pack.c | 5 ++++-\n builtin/pack-objects.c | 2 ++\n ci/run-build-and-tests.sh | 1 +\n pack-revindex.h | 2 ++\n t/README | 3 +++\n 5 files changed, 12 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 03408250b1..0bde325a8b 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1748,7 +1748,10 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (prefix && chdir(prefix))\n \t\tdie(_(\"Cannot come back to cwd\"));\n \n-\trev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV));\n+\tif (git_env_bool(GIT_TEST_WRITE_REV_INDEX, 0))\n+\t\trev_index = 1;\n+\telse\n+\t\trev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV));\n \n \tfor (i = 1; i < argc; i++) {\n \t\tconst char *arg = argv[i];\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex d784569200..24df0c98f7 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3601,6 +3601,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \n \treset_pack_idx_option(&pack_idx_opts);\n \tgit_config(git_pack_config, NULL);\n+\tif (git_env_bool(GIT_TEST_WRITE_REV_INDEX, 0))\n+\t\tpack_idx_opts.flags |= WRITE_REV;\n \n \tprogress = isatty(2);\n \targc = parse_options(argc, argv, prefix, pack_objects_options,\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 6c27b886b8..d1cbf330a1 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -22,6 +22,7 @@ linux-gcc)\n \texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n \texport GIT_TEST_ADD_I_USE_BUILTIN=1\n+\texport GIT_TEST_WRITE_REV_INDEX=1\n \tmake test\n \t;;\n linux-clang)\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex 01622cf21a..7237b2b6f8 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -16,6 +16,8 @@\n * can be found\n */\n \n+#define GIT_TEST_WRITE_REV_INDEX \"GIT_TEST_WRITE_REV_INDEX\"\n+\n struct packed_git;\n \n /*\ndiff --git a/t/README b/t/README\nindex c730a70770..0f97a51640 100644\n--- a/t/README\n+++ b/t/README\n@@ -439,6 +439,9 @@ GIT_TEST_DEFAULT_HASH= specifies which hash algorithm to\n use in the test scripts. Recognized values for are \"sha1\"\n and \"sha256\".\n \n+GIT_TEST_WRITE_REV_INDEX=, when true enables the\n+'pack.writeReverseIndex' setting.\n+\n Naming Tests\n ------------\n \n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414317","messageId":"5b18ada61113faa9dc1de584366cb39b6a449ec6.1610576805.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1610576805.git.me@ttaylorr.com","subject":"[PATCH v2 3/8] builtin/index-pack.c: write reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:28:15Z","receivedAt":"2021-01-14T02:09:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Teach 'git index-pack' to optionally write and verify reverse index with\n'--[no-]rev-index', as well as respecting the 'pack.writeReverseIndex'\nconfiguration option.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-index-pack.txt | 20 ++++++---\n builtin/index-pack.c | 64 +++++++++++++++++++++++-----\n t/t5325-reverse-index.sh | 71 ++++++++++++++++++++++++++++++++\n 3 files changed, 139 insertions(+), 16 deletions(-)\n create mode 100755 t/t5325-reverse-index.sh\n\ndiff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt\nindex af0c26232c..b65f380269 100644\n--- a/Documentation/git-index-pack.txt\n+++ b/Documentation/git-index-pack.txt\n@@ -9,17 +9,18 @@ git-index-pack - Build pack index file for an existing packed archive\n SYNOPSIS\n --------\n [verse]\n-'git index-pack' [-v] [-o ] \n+'git index-pack' [-v] [-o ] [--[no-]rev-index] \n 'git index-pack' --stdin [--fix-thin] [--keep] [-v] [-o ]\n- []\n+\t\t [--[no-]rev-index] []\n \n \n DESCRIPTION\n -----------\n Reads a packed archive (.pack) from the specified file, and\n-builds a pack index file (.idx) for it. The packed archive\n-together with the pack index can then be placed in the\n-objects/pack/ directory of a Git repository.\n+builds a pack index file (.idx) for it. Optionally writes a\n+reverse-index (.rev) for the specified pack. The packed\n+archive together with the pack index can then be placed in\n+the objects/pack/ directory of a Git repository.\n \n \n OPTIONS\n@@ -33,7 +34,14 @@ OPTIONS\n \tfile is constructed from the name of packed archive\n \tfile by replacing .pack with .idx (and the program\n \tfails if the name of packed archive does not end\n-\twith .pack).\n+\twith .pack). Incompatible with `--rev-index`.\n+\n+--[no-]rev-index::\n+\tWhen this flag is provided, generate a reverse index\n+\t(a `.rev` file) corresponding to the given pack. If\n+\t`--verify` is given, ensure that the existing\n+\treverse index is correct. Takes precedence over\n+\t`pack.writeReverseIndex`.\n \n --stdin::\n \tWhen this flag is provided, the pack is read from stdin\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 4b8d86e0ad..03408250b1 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -17,7 +17,7 @@\n #include \"promisor-remote.h\"\n \n static const char index_pack_usage[] =\n-\"git index-pack [-v] [-o ] [--keep | --keep=] [--verify] [--strict] ( | --stdin [--fix-thin] [])\";\n+\"git index-pack [-v] [-o ] [--keep | --keep=] [--[no-]rev-index] [--verify] [--strict] ( | --stdin [--fix-thin] [])\";\n \n struct object_entry {\n \tstruct pack_idx_entry idx;\n@@ -1436,13 +1436,13 @@ static void fix_unresolved_deltas(struct hashfile *f)\n \tfree(sorted_by_pos);\n }\n \n-static const char *derive_filename(const char *pack_name, const char *suffix,\n-\t\t\t\t struct strbuf *buf)\n+static const char *derive_filename(const char *pack_name, const char *strip,\n+\t\t\t\t const char *suffix, struct strbuf *buf)\n {\n \tsize_t len;\n-\tif (!strip_suffix(pack_name, \".pack\", &len))\n-\t\tdie(_(\"packfile name '%s' does not end with '.pack'\"),\n-\t\t pack_name);\n+\tif (!strip_suffix(pack_name, strip, &len))\n+\t\tdie(_(\"packfile name '%s' does not end with '%s'\"),\n+\t\t pack_name, strip);\n \tstrbuf_add(buf, pack_name, len);\n \tstrbuf_addch(buf, '.');\n \tstrbuf_addstr(buf, suffix);\n@@ -1459,7 +1459,7 @@ static void write_special_file(const char *suffix, const char *msg,\n \tint msg_len = strlen(msg);\n \n \tif (pack_name)\n-\t\tfilename = derive_filename(pack_name, suffix, &name_buf);\n+\t\tfilename = derive_filename(pack_name, \".pack\", suffix, &name_buf);\n \telse\n \t\tfilename = odb_pack_name(&name_buf, hash, suffix);\n \n@@ -1484,12 +1484,14 @@ static void write_special_file(const char *suffix, const char *msg,\n \n static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t const char *final_index_name, const char *curr_index_name,\n+\t\t const char *final_rev_index_name, const char *curr_rev_index_name,\n \t\t const char *keep_msg, const char *promisor_msg,\n \t\t unsigned char *hash)\n {\n \tconst char *report = \"pack\";\n \tstruct strbuf pack_name = STRBUF_INIT;\n \tstruct strbuf index_name = STRBUF_INIT;\n+\tstruct strbuf rev_index_name = STRBUF_INIT;\n \tint err;\n \n \tif (!from_stdin) {\n@@ -1524,6 +1526,16 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t} else\n \t\tchmod(final_index_name, 0444);\n \n+\tif (curr_rev_index_name) {\n+\t\tif (final_rev_index_name != curr_rev_index_name) {\n+\t\t\tif (!final_rev_index_name)\n+\t\t\t\tfinal_rev_index_name = odb_pack_name(&rev_index_name, hash, \"rev\");\n+\t\t\tif (finalize_object_file(curr_rev_index_name, final_rev_index_name))\n+\t\t\t\tdie(_(\"cannot store reverse index file\"));\n+\t\t} else\n+\t\t\tchmod(final_rev_index_name, 0444);\n+\t}\n+\n \tif (do_fsck_object) {\n \t\tstruct packed_git *p;\n \t\tp = add_packed_git(final_index_name, strlen(final_index_name), 0);\n@@ -1553,6 +1565,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t}\n \t}\n \n+\tstrbuf_release(&rev_index_name);\n \tstrbuf_release(&index_name);\n \tstrbuf_release(&pack_name);\n }\n@@ -1578,6 +1591,12 @@ static int git_index_pack_config(const char *k, const char *v, void *cb)\n \t\t}\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"pack.writereverseindex\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\topts->flags |= WRITE_REV;\n+\t\telse\n+\t\t\topts->flags &= ~WRITE_REV;\n+\t}\n \treturn git_default_config(k, v, cb);\n }\n \n@@ -1695,12 +1714,14 @@ static void show_pack_info(int stat_only)\n \n int cmd_index_pack(int argc, const char **argv, const char *prefix)\n {\n-\tint i, fix_thin_pack = 0, verify = 0, stat_only = 0;\n+\tint i, fix_thin_pack = 0, verify = 0, stat_only = 0, rev_index;\n \tconst char *curr_index;\n-\tconst char *index_name = NULL, *pack_name = NULL;\n+\tconst char *curr_rev_index = NULL;\n+\tconst char *index_name = NULL, *pack_name = NULL, *rev_index_name = NULL;\n \tconst char *keep_msg = NULL;\n \tconst char *promisor_msg = NULL;\n \tstruct strbuf index_name_buf = STRBUF_INIT;\n+\tstruct strbuf rev_index_name_buf = STRBUF_INIT;\n \tstruct pack_idx_entry **idx_objects;\n \tstruct pack_idx_option opts;\n \tunsigned char pack_hash[GIT_MAX_RAWSZ];\n@@ -1727,6 +1748,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (prefix && chdir(prefix))\n \t\tdie(_(\"Cannot come back to cwd\"));\n \n+\trev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV));\n+\n \tfor (i = 1; i < argc; i++) {\n \t\tconst char *arg = argv[i];\n \n@@ -1805,6 +1828,10 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\tif (hash_algo == GIT_HASH_UNKNOWN)\n \t\t\t\t\tdie(_(\"unknown hash algorithm '%s'\"), arg);\n \t\t\t\trepo_set_hash_algo(the_repository, hash_algo);\n+\t\t\t} else if (!strcmp(arg, \"--rev-index\")) {\n+\t\t\t\trev_index = 1;\n+\t\t\t} else if (!strcmp(arg, \"--no-rev-index\")) {\n+\t\t\t\trev_index = 0;\n \t\t\t} else\n \t\t\t\tusage(index_pack_usage);\n \t\t\tcontinue;\n@@ -1824,7 +1851,16 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (from_stdin && hash_algo)\n \t\tdie(_(\"--object-format cannot be used with --stdin\"));\n \tif (!index_name && pack_name)\n-\t\tindex_name = derive_filename(pack_name, \"idx\", &index_name_buf);\n+\t\tindex_name = derive_filename(pack_name, \".pack\", \"idx\", &index_name_buf);\n+\n+\topts.flags &= ~(WRITE_REV | WRITE_REV_VERIFY);\n+\tif (rev_index) {\n+\t\topts.flags |= verify ? WRITE_REV_VERIFY : WRITE_REV;\n+\t\tif (index_name)\n+\t\t\trev_index_name = derive_filename(index_name,\n+\t\t\t\t\t\t\t \".idx\", \"rev\",\n+\t\t\t\t\t\t\t &rev_index_name_buf);\n+\t}\n \n \tif (verify) {\n \t\tif (!index_name)\n@@ -1878,11 +1914,16 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tfor (i = 0; i < nr_objects; i++)\n \t\tidx_objects[i] = &objects[i].idx;\n \tcurr_index = write_idx_file(index_name, idx_objects, nr_objects, &opts, pack_hash);\n+\tif (rev_index)\n+\t\tcurr_rev_index = write_rev_file(rev_index_name, idx_objects,\n+\t\t\t\t\t\tnr_objects, pack_hash,\n+\t\t\t\t\t\topts.flags);\n \tfree(idx_objects);\n \n \tif (!verify)\n \t\tfinal(pack_name, curr_pack,\n \t\t index_name, curr_index,\n+\t\t rev_index_name, curr_rev_index,\n \t\t keep_msg, promisor_msg,\n \t\t pack_hash);\n \telse\n@@ -1893,10 +1934,13 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \n \tfree(objects);\n \tstrbuf_release(&index_name_buf);\n+\tstrbuf_release(&rev_index_name_buf);\n \tif (pack_name == NULL)\n \t\tfree((void *) curr_pack);\n \tif (index_name == NULL)\n \t\tfree((void *) curr_index);\n+\tif (rev_index_name == NULL)\n+\t\tfree((void *) curr_rev_index);\n \n \t/*\n \t * Let the caller know this pack is not self contained\ndiff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh\nnew file mode 100755\nindex 0000000000..2dae213126\n--- /dev/null\n+++ b/t/t5325-reverse-index.sh\n@@ -0,0 +1,71 @@\n+#!/bin/sh\n+\n+test_description='on-disk reverse index'\n+. ./test-lib.sh\n+\n+packdir=.git/objects/pack\n+\n+test_expect_success 'setup' '\n+\ttest_commit base &&\n+\n+\tpack=$(git pack-objects --all $packdir/pack) &&\n+\trev=$packdir/pack-$pack.rev &&\n+\n+\ttest_path_is_missing $rev\n+'\n+\n+test_index_pack () {\n+\trm -f $rev &&\n+\tconf=$1 &&\n+\tshift &&\n+\t# remove the index since Windows won't overwrite an existing file\n+\trm $packdir/pack-$pack.idx &&\n+\tgit -c pack.writeReverseIndex=$conf index-pack \"$@\" \\\n+\t\t$packdir/pack-$pack.pack\n+}\n+\n+test_expect_success 'index-pack with pack.writeReverseIndex' '\n+\ttest_index_pack \"\" &&\n+\ttest_path_is_missing $rev &&\n+\n+\ttest_index_pack false &&\n+\ttest_path_is_missing $rev &&\n+\n+\ttest_index_pack true &&\n+\ttest_path_is_file $rev\n+'\n+\n+test_expect_success 'index-pack with --[no-]rev-index' '\n+\tfor conf in \"\" true false\n+\tdo\n+\t\ttest_index_pack \"$conf\" --rev-index &&\n+\t\ttest_path_exists $rev &&\n+\n+\t\ttest_index_pack \"$conf\" --no-rev-index &&\n+\t\ttest_path_is_missing $rev\n+\tdone\n+'\n+\n+test_expect_success 'index-pack can verify reverse indexes' '\n+\ttest_when_finished \"rm -f $rev\" &&\n+\ttest_index_pack true &&\n+\n+\ttest_path_is_file $rev &&\n+\tgit index-pack --rev-index --verify $packdir/pack-$pack.pack &&\n+\n+\t# Intentionally corrupt the reverse index.\n+\tchmod u+w $rev &&\n+\tprintf \"xxxx\" | dd of=$rev bs=1 count=4 conv=notrunc &&\n+\n+\ttest_must_fail git index-pack --rev-index --verify \\\n+\t\t$packdir/pack-$pack.pack 2>err &&\n+\tgrep \"validation error\" err\n+'\n+\n+test_expect_success 'index-pack infers reverse index name with -o' '\n+\tgit index-pack --rev-index -o other.idx $packdir/pack-$pack.pack &&\n+\ttest_path_is_file other.idx &&\n+\ttest_path_is_file other.rev\n+'\n+\n+test_done\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414333","messageId":"xmqqwnwgx9rm.fsf@gitster.c.googlers.com","threadId":"54962","inReplyTo":"6742c15c84bafbcc1c06e2633de51dcda63e3314.1610576805.git.me@ttaylorr.com","subject":"Re: [PATCH v2 1/8] packfile: prepare for the existence of '*.rev' files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-14T07:22:53Z","receivedAt":"2021-01-14T07:23:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> +== pack-*.rev files have the format:\n> +\n> + - A 4-byte magic number '0x52494458' ('RIDX').\n> +\n> + - A 4-byte version identifier (= 1)\n> +\n> + - A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256)\n\nThese two are presumably 4-byte-wide network byte order integers.\nWe should spell it out.\n\n> + - A table of index positions, sorted by their corresponding offsets in the\n> + packfile.\n\nLikewise, how wide is each entry and in what byte order, and how\nmany entries are there in the table?\n\n\t... oh, what about the \"one beyond the last\"? We cannot\n\tgo back to the forward index to learn the offset of such\n\tan non-existent object, can we?\n\nAgain, I expect it to be 4-byte-wide network byte order integer.\n\n> -int load_pack_revindex(struct packed_git *p)\n> +static int load_pack_revindex_from_memory(struct packed_git *p)\n\nI said it when I saw the beginning of v1 API patches, but it is a\nbit unreasonable to call the act of \"computing from the forward\nindex\" \"to load from memory\". Loading from disk perfectly works as\na phrase, though.\n\n> +#define RIDX_MIN_SIZE (12 + (2 * the_hash_algo->rawsz))\n> +\n> +static int load_revindex_from_disk(char *revindex_name,\n> +\t\t\t\t uint32_t num_objects,\n> +\t\t\t\t const void **data, size_t *len)\n> +{\n> +\tint fd, ret = 0;\n> +\tstruct stat st;\n> +\tsize_t revindex_size;\n> +\n> +\tfd = git_open(revindex_name);\n> +\n> +\tif (fd < 0) {\n> +\t\tret = -1;\n> +\t\tgoto cleanup;\n> +\t}\n> +\tif (fstat(fd, &st)) {\n> +\t\tret = error_errno(_(\"failed to read %s\"), revindex_name);\n> +\t\tgoto cleanup;\n> +\t}\n> +\n> +\trevindex_size = xsize_t(st.st_size);\n> +\n> +\tif (revindex_size < RIDX_MIN_SIZE) {\n> +\t\tret = error(_(\"reverse-index file %s is too small\"), revindex_name);\n> +\t\tgoto cleanup;\n> +\t}\n> +\n> +\tif (revindex_size - RIDX_MIN_SIZE != st_mult(sizeof(uint32_t), num_objects)) {\n> +\t\tret = error(_(\"reverse-index file %s is corrupt\"), revindex_name);\n> +\t\tgoto cleanup;\n> +\t}\n> +\n> +\t*len = revindex_size;\n> +\t*data = xmmap(NULL, revindex_size, PROT_READ, MAP_PRIVATE, fd, 0);\n> +\n> +cleanup:\n> +\tclose(fd);\n> +\treturn ret;\n> +}\n> +\n> +static int load_pack_revindex_from_disk(struct packed_git *p)\n> +{\n> +\tchar *revindex_name;\n> +\tint ret;\n> +\tif (open_pack_index(p))\n> +\t\treturn -1;\n> +\n> +\trevindex_name = pack_revindex_filename(p);\n> +\n> +\tret = load_revindex_from_disk(revindex_name,\n> +\t\t\t\t p->num_objects,\n> +\t\t\t\t &p->revindex_map,\n> +\t\t\t\t &p->revindex_size);\n> +\tif (ret)\n> +\t\tgoto cleanup;\n> +\n> +\tp->revindex_data = (char *)p->revindex_map + 12;\n\nWe've seen hardcoded constant \"12\" twice so far in this patch.\n\nWe need a C proprocessor macro \"#define RIDX_FILE_HEADER_SIZE 12\" or\nsomething, perhaps?\n\n> +cleanup:\n> +\tfree(revindex_name);\n> +\treturn ret;\n> +}\n> +\n> +int load_pack_revindex(struct packed_git *p)\n> +{\n> +\tif (p->revindex || p->revindex_data)\n> +\t\treturn 0;\n> +\n> +\tif (!load_pack_revindex_from_disk(p))\n> +\t\treturn 0;\n> +\telse if (!load_pack_revindex_from_memory(p))\n> +\t\treturn 0;\n> +\treturn -1;\n> +}\n> +\n> int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n> {\n> \tunsigned lo, hi;\n> @@ -203,18 +285,28 @@ int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n> \n> uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos)\n> {\n> -\tif (!p->revindex)\n> +\tif (!(p->revindex || p->revindex_data))\n> \t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n> \tif (p->num_objects <= pos)\n> \t\tBUG(\"pack_pos_to_index: out-of-bounds object at %\"PRIu32, pos);\n> -\treturn p->revindex[pos].nr;\n> +\n> +\tif (p->revindex)\n> +\t\treturn p->revindex[pos].nr;\n> +\telse\n> +\t\treturn get_be32((char *)p->revindex_data + (pos * sizeof(uint32_t)));\n\nGood. We are using 32-bit uint in network byte order. We should\ndocument it as such.\n\nLet's not strip const away while casting, though. get_be32()\nensures that it only reads and never writes thru the pointer, and\np->revindex_data is a \"const void *\".\n\n> }\n> \n> off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n> {\n> -\tif (!p->revindex)\n> +\tif (!(p->revindex || p->revindex_data))\n> \t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n> \tif (p->num_objects < pos)\n> \t\tBUG(\"pack_pos_to_offset: out-of-bounds object at %\"PRIu32, pos);\n> -\treturn p->revindex[pos].offset;\n> +\n> +\tif (p->revindex)\n> +\t\treturn p->revindex[pos].offset;\n> +\telse if (pos == p->num_objects)\n> +\t\treturn p->pack_size - the_hash_algo->rawsz;\n\nOK, here is the answer to my previous question. We should document\nthat the table has num_objects entries in the on-disk file (we do\nnot need to say that there is no sentinel entry in the table at the\nend).\n\n> +\telse\n> +\t\treturn nth_packed_object_offset(p, pack_pos_to_index(p, pos));\n> }\n> diff --git a/pack-revindex.h b/pack-revindex.h\n> index 6e0320b08b..01622cf21a 100644\n> --- a/pack-revindex.h\n> +++ b/pack-revindex.h\n> @@ -21,6 +21,9 @@ struct packed_git;\n> /*\n> * load_pack_revindex populates the revindex's internal data-structures for the\n> * given pack, returning zero on success and a negative value otherwise.\n> + *\n> + * If a '.rev' file is present, it is checked for consistency, mmap'd, and\n> + * pointers are assigned into it (instead of using the in-memory variant).\n\nHmph, I missed where it got checked for consistency, though. If the\nfile is corrupt and has say duplicated entries, we'd happily grab\nthe data via get_be32(), for example.\n\n> @@ -55,7 +58,9 @@ uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);\n> * If the reverse index has not yet been loaded, or the position is out of\n> * bounds, this function aborts.\n> *\n> - * This function runs in constant time.\n> + * This function runs in constant time under both in-memory and on-disk reverse\n> + * indexes, but an additional step is taken to consult the corresponding .idx\n> + * file when using the on-disk format.\n\nAgain, I know this is a kind of detail that is interesting to those\nwho implemented the function, but I wonder how it would help those\nwho wonder if they should call it or use some other method to\nachieve what they want.\n\n"},{"id":"414334","messageId":"xmqqsg74x9ks.fsf@gitster.c.googlers.com","threadId":"54962","inReplyTo":"6742c15c84bafbcc1c06e2633de51dcda63e3314.1610576805.git.me@ttaylorr.com","subject":"Re: [PATCH v2 1/8] packfile: prepare for the existence of '*.rev' files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-14T07:26:59Z","receivedAt":"2021-01-14T07:27:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Specify the format of the on-disk reverse index 'pack-*.rev' file, as\n> well as prepare the code for the existence of such files.\n\nWe've changed the pack .idx file format once as the file format is\nversioned. I wonder if you considered placing the reverse index\ninformation in the same file, with version bump, in the .idx?\n"},{"id":"414353","messageId":"10326ece-f433-4793-336e-a30786bf1f2f@gmail.com","threadId":"54962","inReplyTo":"xmqqwnwgx9rm.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 1/8] packfile: prepare for the existence of '*.rev' files","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-01-14T12:07:08Z","receivedAt":"2021-01-14T12:07:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/14/2021 2:22 AM, Junio C Hamano wrote:\n> Taylor Blau writes:\n> \n>> +== pack-*.rev files have the format:\n>> +\n>> + - A 4-byte magic number '0x52494458' ('RIDX').\n>> +\n>> + - A 4-byte version identifier (= 1)\n>> +\n>> + - A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256)\n> \n> These two are presumably 4-byte-wide network byte order integers.\n> We should spell it out.\n\nIn the past, we've included the sentence \"All multi-byte numbers are\nin network byte order.\" to clarify this. These two entries need to\nspecify that the \"identifier\" is actually an integer. Perhaps these\nthree values could be provided as:\n\n+== pack-*.rev files have the format:\n+\n+ - A 4-byte identifier '0x52494458' ('RIDX').\n+\n+ - A 4-byte integer version (= 1)\n+\n+ - A 4-byte integer hash function version (= 1 for SHA-1, 2 for SHA-256)\n\n>> /*\n>> * load_pack_revindex populates the revindex's internal data-structures for the\n>> * given pack, returning zero on success and a negative value otherwise.\n>> + *\n>> + * If a '.rev' file is present, it is checked for consistency, mmap'd, and\n>> + * pointers are assigned into it (instead of using the in-memory variant).\n> \n> Hmph, I missed where it got checked for consistency, though. If the\n> file is corrupt and has say duplicated entries, we'd happily grab\n> the data via get_be32(), for example.\n\nEven if the consistency check is just verifying the trailing hash, that\nseems like something that requires O(N) before performing a lookup. Perhaps\nthis was copied from somewhere else, or means something different?\n\nThanks,\n-Stolee\n\n"},{"id":"414369","messageId":"YACJsdDToLQ24p/Y@nand.local","threadId":"54962","inReplyTo":"xmqqsg74x9ks.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 1/8] packfile: prepare for the existence of '*.rev' files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-14T18:13:05Z","receivedAt":"2021-01-14T18:13:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 13, 2021 at 11:26:59PM -0800, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > Specify the format of the on-disk reverse index 'pack-*.rev' file, as\n> > well as prepare the code for the existence of such files.\n>\n> We've changed the pack .idx file format once as the file format is\n> versioned. I wonder if you considered placing the reverse index\n> information in the same file, with version bump, in the .idx?\n\nFunny enough, I couldn't remember whether I considered this or not.\nPeff reminded me off-list that he and I *had* considered this, but\ndecided against it.\n\nThe main benefit to introducing a new '.rev' format is that we can have\npacks that can be upgraded to write a reverse index without having to\nrewrite their forward index. It would also allow us to avoid teaching\nother implementations about a new version of the index file (since they\ncan ignore it and continue to build their equivalent of the reverse\nindex file in memory by reading the forward index).\n\n(Peff reminds me that dumb-http does look at remote .idx files, so this\nnew format would leak across to clients, whether or not that's something\nto be concerned about...).\n\nOf course, having the contents of the .rev file be included in the .idx\nfile nets us one fewer file to manage, but I'm not sure that's a reason\nto do things one way or another.\n\nYour response did pique my interest, since I was wondering if we could\nimprove the cold cache performance if the .rev file's contents were\nincluded in the .idx, but after giving it some thought I don't think we\ncan. Reasons are:\n\n - If the reverse index's contents appears at the end of the .idx file,\n then in any .idx file large enough to matter, we'll almost certainly\n still be evicting cache lines back and forth when swapping between\n reading the forward- and reverse-indexes. So, no gains to be had\n there.\n\n - If, on the other hand, we included the reverse index's contents by\n interleaving it with the forward index's offsets, then we'd be\n worsening the cache performance of the forward index.\n\nSo, I'm more in favor of a new .rev file rather than a v3 .idx version.\nApologies for not including more of a rationale \"why\" in the cover\nletter (had I not forgotten that I'd even considered it, I would have).\n\nThanks,\nTaylor\n"},{"id":"414371","messageId":"YACNX+ExF1yAn4g3@nand.local","threadId":"54962","inReplyTo":"xmqqwnwgx9rm.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 1/8] packfile: prepare for the existence of '*.rev' files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-14T18:28:47Z","receivedAt":"2021-01-14T18:29:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 13, 2021 at 11:22:53PM -0800, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > +== pack-*.rev files have the format:\n> > +\n> > + - A 4-byte magic number '0x52494458' ('RIDX').\n> > +\n> > + - A 4-byte version identifier (= 1)\n> > +\n> > + - A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256)\n>\n> These two are presumably 4-byte-wide network byte order integers.\n> We should spell it out.\n\nYep, all entries are network order. I added a sentence at the bottom of\nthis sub-section to say as much.\n\n> We've seen hardcoded constant \"12\" twice so far in this patch.\n>\n> We need a C proprocessor macro \"#define RIDX_FILE_HEADER_SIZE 12\" or\n> something, perhaps?\n\nGood idea, thanks.\n\n> > -\treturn p->revindex[pos].nr;\n> > +\n> > +\tif (p->revindex)\n> > +\t\treturn p->revindex[pos].nr;\n> > +\telse\n> > +\t\treturn get_be32((char *)p->revindex_data + (pos * sizeof(uint32_t)));\n>\n> Good. We are using 32-bit uint in network byte order. We should\n> document it as such.\n>\n> Let's not strip const away while casting, though. get_be32()\n> ensures that it only reads and never writes thru the pointer, and\n> p->revindex_data is a \"const void *\".\n\nAgreed, and thanks for the suggestion. I take it that what you mean is:\n\n-\t\treturn get_be32((char *)p->revindex_data + (pos * sizeof(uint32_t)));\n+\t\treturn get_be32((const char *)p->revindex_data + (pos * sizeof(uint32_t)));\n\n...yes?\n\n> > diff --git a/pack-revindex.h b/pack-revindex.h\n> > index 6e0320b08b..01622cf21a 100644\n> > --- a/pack-revindex.h\n> > +++ b/pack-revindex.h\n> > @@ -21,6 +21,9 @@ struct packed_git;\n> > /*\n> > * load_pack_revindex populates the revindex's internal data-structures for the\n> > * given pack, returning zero on success and a negative value otherwise.\n> > + *\n> > + * If a '.rev' file is present, it is checked for consistency, mmap'd, and\n> > + * pointers are assigned into it (instead of using the in-memory variant).\n>\n> Hmph, I missed where it got checked for consistency, though. If the\n> file is corrupt and has say duplicated entries, we'd happily grab\n> the data via get_be32(), for example.\n\nIt doesn't, I'm mistaken. I removed that incorrect detail from this\ncomment. Thanks for catching it.\n\nThanks,\nTaylor\n"},{"id":"414379","messageId":"YACiGJO+kqz2yleZ@coredump.intra.peff.net","threadId":"54962","inReplyTo":"10326ece-f433-4793-336e-a30786bf1f2f@gmail.com","subject":"Re: [PATCH v2 1/8] packfile: prepare for the existence of '*.rev' files","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-14T19:57:12Z","receivedAt":"2021-01-14T19:58:10Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 14, 2021 at 07:07:08AM -0500, Derrick Stolee wrote:\n\n> >> + * If a '.rev' file is present, it is checked for consistency, mmap'd, and\n> >> + * pointers are assigned into it (instead of using the in-memory variant).\n> > \n> > Hmph, I missed where it got checked for consistency, though. If the\n> > file is corrupt and has say duplicated entries, we'd happily grab\n> > the data via get_be32(), for example.\n> \n> Even if the consistency check is just verifying the trailing hash, that\n> seems like something that requires O(N) before performing a lookup. Perhaps\n> this was copied from somewhere else, or means something different?\n\nFor the .idx file, we check that the size is what we expect. This is\nimportant because it lets us access the mapped bytes in normal use\nwithout having to do a bounds check.\n\nIt looks like we do the same for the .rev file here, which is good. If\ncalling that \"checked for consistency\" is too strong, I don't think it's\na big deal to drop the wording (we do not make any such claim for\nopen_pack_index()).\n\n-Peff\n"},{"id":"414393","messageId":"xmqqmtxbw814.fsf@gitster.c.googlers.com","threadId":"54962","inReplyTo":"YACJsdDToLQ24p/Y@nand.local","subject":"Re: [PATCH v2 1/8] packfile: prepare for the existence of '*.rev' files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-14T20:57:59Z","receivedAt":"2021-01-14T20:58:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Your response did pique my interest, since I was wondering if we could\n> improve the cold cache performance if the .rev file's contents were\n> included in the .idx,...\n\nYes, that was the primary thing I was wondering. As long as it was\nconsidered and rejected for good reason, that is good.\n\nThanks.\n\n\n"},{"id":"415021","messageId":"YAtXmie2kHNrcBwY@coredump.intra.peff.net","threadId":"54962","inReplyTo":"6742c15c84bafbcc1c06e2633de51dcda63e3314.1610576805.git.me@ttaylorr.com","subject":"Re: [PATCH v2 1/8] packfile: prepare for the existence of '*.rev' files","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-22T22:54:18Z","receivedAt":"2021-01-22T22:55:29Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 13, 2021 at 05:28:06PM -0500, Taylor Blau wrote:\n\n> (Numbers taken in the kernel after cheating and using the next patch to\n> generate a reverse index). There are a couple of approaches to improve\n> cold cache performance not pursued here:\n> \n> - We could include the object offsets in the reverse index format.\n> Predictably, this does result in fewer page faults, but it triples\n> the size of the file, while simultaneously duplicating a ton of data\n> already available in the .idx file. (This was the original way I\n> implemented the format, and it did show\n> `--batch-check='%(objectsize:disk)'` winning out against `--batch`.)\n> \n> On the other hand, this increase in size also results in a large\n> block-cache footprint, which could potentially hurt other workloads.\n> \n> - We could store the mapping from pack to index position in more\n> cache-friendly way, like constructing a binary search tree from the\n> table and writing the values in breadth-first order. This would\n> result in much better locality, but the price you pay is trading\n> O(1) lookup in 'pack_pos_to_index()' for an O(log n) one (since you\n> can no longer directly index the table).\n> \n> So, neither of these approaches are taken here. (Thankfully, the format\n> is versioned, so we are free to pursue these in the future.) But, cold\n> cache performance likely isn't interesting outside of one-off cases like\n> asking for the size of an object directly. In real-world usage, Git is\n> often performing many operations in the revindex,\n\nI think you've nicely covered the arguments for and against the extra\noffset here. This final paragraph ends in a comma, which makes me wonder\nif you wanted to say something more. I'd guess it is along the lines\nthat most commands will be looking up more than one object, so that\ncold-cache effort is amortized.\n\nOr another way of thinking about it: 17ms versus 25ms in the cold-cache\nfor a _single_ object is not that big a deal, because the extra 8ms does\nnot scale as we ask about more objects. Here's an actual argument in\nnumbers (test repo is linux.git after building a .rev file using your\nseries):\n\nFor a single object, the extra cold-cache costs give --batch a slight\nedge:\n\n $ git rev-parse HEAD >obj\n $ hyperfine -p 'echo 3 | sudo tee /proc/sys/vm/drop_caches' \\\n 'git cat-file --buffer --batch-check=\"%(objectsize:disk)\" obj-1000\n $ hyperfine -p 'echo 3 | sudo tee /proc/sys/vm/drop_caches' \\\n 'git cat-file --buffer --batch-check=\"%(objectsize:disk)\" Documentation/technical/pack-format.txt | 17 ++++\n> builtin/repack.c | 1 +\n> object-store.h | 3 +\n> pack-revindex.c | 112 +++++++++++++++++++++---\n> pack-revindex.h | 7 +-\n> packfile.c | 13 ++-\n> packfile.h | 1 +\n> tmp-objdir.c | 4 +-\n> 8 files changed, 145 insertions(+), 13 deletions(-)\n\nOh, there's a patch here, too. :)\n\nIt mostly looks good to me. I agree with Junio that \"compute\" is a\nbetter verb than \"load\" for generating the in-memory revindex.\n\n> +static int load_pack_revindex_from_disk(struct packed_git *p)\n> +{\n> +\tchar *revindex_name;\n> +\tint ret;\n> +\tif (open_pack_index(p))\n> +\t\treturn -1;\n> +\n> +\trevindex_name = pack_revindex_filename(p);\n> +\n> +\tret = load_revindex_from_disk(revindex_name,\n> +\t\t\t\t p->num_objects,\n> +\t\t\t\t &p->revindex_map,\n> +\t\t\t\t &p->revindex_size);\n> +\tif (ret)\n> +\t\tgoto cleanup;\n> +\n> +\tp->revindex_data = (char *)p->revindex_map + 12;\n\nJunio mentioned once spot where we lose constness through a cast. This\nis another. I wonder if revindex_map should just be a \"char *\" to make\npointer arithmetic easier without having to cast.\n\nBut also...\n\n> +\tif (p->revindex)\n> +\t\treturn p->revindex[pos].nr;\n> +\telse\n> +\t\treturn get_be32((char *)p->revindex_data + (pos * sizeof(uint32_t)));\n\nIf p->revindex_data were \"const uint32_t *\", then this line would just\nbe:\n\n return get_be32(p->revindex_data + pos);\n\nNot a huge deal either way since the whole point is to abstract this\nbehind a function where it only has to be written once. I don't think\nthere is any downside from the compiler's view (and we already use this\ntrick for the bitmap name-hash cache).\n\n> diff --git a/packfile.c b/packfile.c\n> index 7bb1750934..b04eac9286 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -324,11 +324,21 @@ void close_pack_index(struct packed_git *p)\n> \t}\n> }\n> \n> +void close_pack_revindex(struct packed_git *p) {\n> +\tif (!p->revindex_map)\n> +\t\treturn;\n> +\n> +\tmunmap((void *)p->revindex_map, p->revindex_size);\n> +\tp->revindex_map = NULL;\n> +\tp->revindex_data = NULL;\n> +}\n> +\n> void close_pack(struct packed_git *p)\n> {\n> \tclose_pack_windows(p);\n> \tclose_pack_fd(p);\n> \tclose_pack_index(p);\n> +\tclose_pack_revindex(p);\n> }\n\nThinking out loud a bit: a .rev file means we're spending an extra map\nper pack (but not a descriptor, since we close after mmap). And like the\n.idx files (but unlike .pack file maps), we don't keep track of these\nand try to close them when under memory pressure. I think that's\nprobably OK in terms of bytes. It may mean running up against operating\nsystem number-of-mmap limits more quickly when you have a very large\nnumber of packs, as mentioned in:\n\n https://lore.kernel.org/git/20200601044511.GA2529317@coredump.intra.peff.net/\n\nBut this is probably bumping the number of problematic packs from 30k to\n20k. Both are sufficiently ridiculous that I don't think it matters in\npractice.\n\n> diff --git a/tmp-objdir.c b/tmp-objdir.c\n> index 42ed4db5d3..da414df14f 100644\n> --- a/tmp-objdir.c\n> +++ b/tmp-objdir.c\n> @@ -187,7 +187,9 @@ static int pack_copy_priority(const char *name)\n> \t\treturn 2;\n> \tif (ends_with(name, \".idx\"))\n> \t\treturn 3;\n> -\treturn 4;\n> +\tif (ends_with(name, \".rev\"))\n> +\t\treturn 4;\n> +\treturn 5;\n> }\n\nProbably not super important, but: should the .idx file still come last\nhere? Simultaneous readers won't start using the pack until the .idx\nfile is present. We'd probably prefer they see the whole thing\natomically, than see a .idx missing its .rev (they won't ever produce a\nwrong answer, but they'll generate the in-core revindex on the fly when\nthey don't need to).\n\nI guess one could argue that .bitmap files should get similar treatment,\nbut we'd not generally see those in the quarantine objdir anyway, so\nnobody ever gave it much thought.\n\n-Peff\n"},{"id":"415022","messageId":"YAtey9krU32mgEBV@coredump.intra.peff.net","threadId":"54962","inReplyTo":"8648c87fa71aec427dc11afcba6548fb66a1413b.1610576805.git.me@ttaylorr.com","subject":"Re: [PATCH v2 2/8] pack-write.c: prepare to write 'pack-*.rev' files","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-22T23:24:59Z","receivedAt":"2021-01-22T23:26:09Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 13, 2021 at 05:28:11PM -0500, Taylor Blau wrote:\n\n> +static void write_rev_index_positions(struct hashfile *f,\n> +\t\t\t\t struct pack_idx_entry **objects,\n> +\t\t\t\t uint32_t nr_objects)\n> +{\n> +\tuint32_t *pack_order;\n> +\tuint32_t i;\n> +\n> +\tALLOC_ARRAY(pack_order, nr_objects);\n> +\tfor (i = 0; i < nr_objects; i++)\n> +\t\tpack_order[i] = i;\n> +\tQSORT_S(pack_order, nr_objects, pack_order_cmp, objects);\n\nqsort? Don't we have a perfectly good radix sort for this exact purpose? :)\n\nI guess it is awkward to use because it hard-codes the assumption that\nwe are sorting revindex_entry structs, with their offsets directly\navailable.\n\nWe _could_ actually just generate an in-memory revindex array, and then\nuse that to write out the .rev file. That has the nice property that\nwe'd continue exercising the fallback code in the tests.\n\nBut a revindex_entry is much larger than the uint32_t's we need here. So\nwe'd be incurring extra memory costs during the generation, which is\nprobably not worth it. If we want better test coverage, we probably\nshould explicitly run a series of tests both with and without a .rev\nfile.\n\nIt's possible we'd still benefit from using a more generalized radix\nsort, but it probably isn't worth the trouble. The timings from\n8b8dfd5132 (pack-revindex: radix-sort the revindex, 2013-07-11) claim a\n4x speedup on sorting 3M entries (the size matters because we are\ncomparing an O(n) sort versus an O(n log n) one). We can imagine that at\n10M entries for the current kernel, it might even be 8x. But the\nabsolute numbers are pretty small. The radix sort takes ~150ms for\nlinux.git on my machine. At 8x, that's 1.2s. For a repack of the\nkernel, that is mostly a drop in the bucket. It mattered a lot more when\nmany processes were doing it on the fly.\n\n> +static int pack_order_cmp(const void *va, const void *vb, void *ctx)\n> +{\n> +\tstruct pack_idx_entry **objects = ctx;\n> +\n> +\toff_t oa = objects[*(uint32_t*)va]->offset;\n> +\toff_t ob = objects[*(uint32_t*)vb]->offset;\n\nDereferencing a pointer to index another array always makes me nervous\nthat we may have a bounds problem with bogus data.\n\nIn this case we know it is OK because we filled the array ourselves with\nin-bound numbers in write_rev_index_positions.\n\n> +#define RIDX_SIGNATURE 0x52494458 /* \"RIDX\" */\n> +#define RIDX_VERSION 1\n\nI was surprised we didn't define these already on the reading side, but\nit looks like we didn't. In patch 1, we probably should be checking\nRIDX_SIGNATURE in load_revindex_from_disk(). But much more importantly,\nwe should be checking that we find version 1, since that's what will\nmake it safe to later invent a version 2.\n\n> +static void write_rev_header(struct hashfile *f)\n> +{\n> +\tuint32_t oid_version;\n> +\tswitch (hash_algo_by_ptr(the_hash_algo)) {\n> +\tcase GIT_HASH_SHA1:\n> +\t\toid_version = 1;\n> +\t\tbreak;\n> +\tcase GIT_HASH_SHA256:\n> +\t\toid_version = 2;\n> +\t\tbreak;\n> +\tdefault:\n> +\t\tdie(\"write_rev_header: unknown hash version\");\n> +\t}\n\nI forgot to comment on this in patch 1, but: I think the format is\nreally independent of the hash size. The contents are identical for a\nsha-1 versus sha-256 file.\n\nThat said, I don't overly mind having a hash identifier if it might help\ndebug things (OTOH, how the heck do you end up with one that matches the\ntrailer's packfile but _doesn't_ match the trailer's contents?).\n\nIf we do have it, should we also be checking it in the loading function?\n\n> +const char *write_rev_file(const char *rev_name,\n> +\t\t\t struct pack_idx_entry **objects,\n> +\t\t\t uint32_t nr_objects,\n> +\t\t\t const unsigned char *hash,\n> +\t\t\t unsigned flags)\n> +{\n> +\tstruct hashfile *f;\n> +\tint fd;\n> +\n> +\tif ((flags & WRITE_REV) && (flags & WRITE_REV_VERIFY))\n> +\t\tdie(_(\"cannot both write and verify reverse index\"));\n> +\n> +\tif (flags & WRITE_REV) {\n> +\t\tif (!rev_name) {\n> +\t\t\tstruct strbuf tmp_file = STRBUF_INIT;\n> +\t\t\tfd = odb_mkstemp(&tmp_file, \"pack/tmp_rev_XXXXXX\");\n> +\t\t\trev_name = strbuf_detach(&tmp_file, NULL);\n> +\t\t} else {\n> +\t\t\tunlink(rev_name);\n> +\t\t\tfd = open(rev_name, O_CREAT|O_EXCL|O_WRONLY, 0600);\n> +\t\t\tif (fd < 0)\n> +\t\t\t\tdie_errno(\"unable to create '%s'\", rev_name);\n> +\t\t}\n\nSo if the caller gave us a name, we force-overwrite it. That seemed\nweird to me at first, but it makes sense; the atomic rename-into-place\nwriters will not be passing in the name. And this is exactly how\nwrite_idx_file() works.\n\nI wonder if we could factor out some of this repeated logic, but I\nsuspect it is mostly diminishing returns. Maybe this \"open a pack file\nfor writing\" could become a helper function, though.\n\n> diff --git a/pack.h b/pack.h\n> index 9fc0945ac9..30439e0784 100644\n> --- a/pack.h\n> +++ b/pack.h\n> @@ -42,6 +42,8 @@ struct pack_idx_option {\n> \t/* flag bits */\n> #define WRITE_IDX_VERIFY 01 /* verify only, do not write the idx file */\n> #define WRITE_IDX_STRICT 02\n> +#define WRITE_REV 04\n> +#define WRITE_REV_VERIFY 010\n\nIt is a little funny that the write-rev function has both WRITE_REV and\nWRITE_REV_VERIFY (which we must be sure are not both provided), but we\ndo not need both for the IDX.\n\nI thought maybe the reason is that we'd pass these flags to\nwrite_idx_file() or similar, and it would need to know whether to write\njust an idx, or both. That doesn't seem to happen in this patch, but\nmaybe it does in a future one...\n\n-Peff\n"},{"id":"415024","messageId":"YAtlZ49mTfTGg11/@coredump.intra.peff.net","threadId":"54962","inReplyTo":"5b18ada61113faa9dc1de584366cb39b6a449ec6.1610576805.git.me@ttaylorr.com","subject":"Re: [PATCH v2 3/8] builtin/index-pack.c: write reverse indexes","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-22T23:53:11Z","receivedAt":"2021-01-22T23:58:02Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 13, 2021 at 05:28:15PM -0500, Taylor Blau wrote:\n\n> OPTIONS\n> @@ -33,7 +34,14 @@ OPTIONS\n> \tfile is constructed from the name of packed archive\n> \tfile by replacing .pack with .idx (and the program\n> \tfails if the name of packed archive does not end\n> -\twith .pack).\n> +\twith .pack). Incompatible with `--rev-index`.\n\nI wondered which option was incompatible, but couldn't see from the\ncontext. It is \"index-pack -o\", which kind of makes sense. We can derive\n\"foo.rev\" from \"foo.idx\", but normally \"-o\" does not do any deriving.\n\nSo I was all set to say \"OK, we can live without it\", but...\n\n> @@ -1824,7 +1851,16 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n> \tif (from_stdin && hash_algo)\n> \t\tdie(_(\"--object-format cannot be used with --stdin\"));\n> \tif (!index_name && pack_name)\n> -\t\tindex_name = derive_filename(pack_name, \"idx\", &index_name_buf);\n> +\t\tindex_name = derive_filename(pack_name, \".pack\", \"idx\", &index_name_buf);\n> +\n> +\topts.flags &= ~(WRITE_REV | WRITE_REV_VERIFY);\n> +\tif (rev_index) {\n> +\t\topts.flags |= verify ? WRITE_REV_VERIFY : WRITE_REV;\n> +\t\tif (index_name)\n> +\t\t\trev_index_name = derive_filename(index_name,\n> +\t\t\t\t\t\t\t \".idx\", \"rev\",\n> +\t\t\t\t\t\t\t &rev_index_name_buf);\n> +\t}\n\n...here we do end up deriving \".rev\" from \".idx\" anyway. So I guess we\nprobably could support \"-o\". I also wonder what happens with \"git\nindex-pack -o foo.idx\" when pack.writeReverseIndex is set. It looks like\nit would just work because of this block. But then shouldn't\n\"--rev-index\" work, too? And indeed, there is a test for that at the end\nof the patch! So is the documentation just wrong?\n\nI admit to finding the use of opts.flags versus the rev_index option a\nbit confusing. It seems like they are doing roughly the same thing, but\ninfluenced by different sources. It seems like we should be able to have\na single local variable (then that goes on to set opts.flags for any\nsub-functions we call). Or maybe two, if we need to distinguish config\nversus command-line, but then they should have clear names\n(rev_index_config and rev_index_cmdline or something).\n\nAs an aside, looking at derive_filename(), it seems a bit weird that one\nargument has a dot in the suffix and the other does not. I guess you are\nfollowing the convention from write_special_file(), which omits it in\nthe newly-added suffix. But it is slightly awkward to omit it for the\nold suffix in derive_filename(), because we want to strip_suffix() it\nall at once.\n\nProbably not that big a deal, but if anybody feels strongly, then\nderive_filename() could do:\n\n if (!strip_suffix(pack_name, strip, &len) ||\n !len || pack_name[len] != '.')\n\tdie(\"does not end in .%s\", strip);\n\n> @@ -1578,6 +1591,12 @@ static int git_index_pack_config(const char *k, const char *v, void *cb)\n> \t\t}\n> \t\treturn 0;\n> \t}\n> +\tif (!strcmp(k, \"pack.writereverseindex\")) {\n> +\t\tif (git_config_bool(k, v))\n> +\t\t\topts->flags |= WRITE_REV;\n> +\t\telse\n> +\t\t\topts->flags &= ~WRITE_REV;\n> +\t}\n> \treturn git_default_config(k, v, cb);\n> }\n\nIMHO we'll eventually want to turn this feature on by default. In which\ncase we'll have to update every caller which is checking the config\nmanually. Should we hide this in a function that looks up the config,\nand sets the default? Or alternatively, I guess, they could all use some\nshared initializer for \"flags\".\n\n> +\t# Intentionally corrupt the reverse index.\n> +\tchmod u+w $rev &&\n> +\tprintf \"xxxx\" | dd of=$rev bs=1 count=4 conv=notrunc &&\n> +\n> +\ttest_must_fail git index-pack --rev-index --verify \\\n> +\t\t$packdir/pack-$pack.pack 2>err &&\n> +\tgrep \"validation error\" err\n> +'\n\nThis isn't that subtle of a corruption, because we are corrupting the\nfirst 4 bytes, which is the magic signature. Maybe something further in\nthe actual data would be interesting instead of or in addition?\n\nI dunno. There are a lot of edge cases around corruption (likewise, we\nmight care how the normal reading code-path perceives a signature\ncorruption like this). I'm not sure it's all that interesting to test\nall of them.\n\n> +test_expect_success 'index-pack infers reverse index name with -o' '\n> +\tgit index-pack --rev-index -o other.idx $packdir/pack-$pack.pack &&\n> +\ttest_path_is_file other.idx &&\n> +\ttest_path_is_file other.rev\n> +'\n\nHey, we _do_ support \"--rev-index -o\". Is it just the documentation that\nis wrong?\n\n-Peff\n"},{"id":"415025","messageId":"YAtmb7wXQse76kJi@coredump.intra.peff.net","threadId":"54962","inReplyTo":"68bde3ea972f5b3753d7e9063d0490c67c74709b.1610576805.git.me@ttaylorr.com","subject":"Re: [PATCH v2 4/8] builtin/pack-objects.c: respect 'pack.writeReverseIndex'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-22T23:57:35Z","receivedAt":"2021-01-22T23:59:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 13, 2021 at 05:28:19PM -0500, Taylor Blau wrote:\n\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 5b0c4489e2..d784569200 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -2955,6 +2955,13 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n> \t\t\t pack_idx_opts.version);\n> \t\treturn 0;\n> \t}\n> +\tif (!strcmp(k, \"pack.writereverseindex\")) {\n> +\t\tif (git_config_bool(k, v))\n> +\t\t\tpack_idx_opts.flags |= WRITE_REV;\n> +\t\telse\n> +\t\t\tpack_idx_opts.flags &= ~WRITE_REV;\n> +\t\treturn 0;\n> +\t}\n\nThis turned out delightfully simple. And I guess this is the \"why is\nWRITE_REV\" caller I asked about from patch 2. It is\nfinish_tmp_packfile() where the magic happens. That unconditionally\ncalls write_rev_file(), but it's a noop if WRITE_REV isn't specified.\n\nMakes sense.\n\n-Peff\n"},{"id":"415027","messageId":"YAto/ndOVfKSkizS@coredump.intra.peff.net","threadId":"54962","inReplyTo":"YAtmb7wXQse76kJi@coredump.intra.peff.net","subject":"Re: [PATCH v2 4/8] builtin/pack-objects.c: respect 'pack.writeReverseIndex'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-23T00:08:30Z","receivedAt":"2021-01-23T00:09:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 22, 2021 at 06:57:35PM -0500, Jeff King wrote:\n\n> > +\tif (!strcmp(k, \"pack.writereverseindex\")) {\n> > +\t\tif (git_config_bool(k, v))\n> > +\t\t\tpack_idx_opts.flags |= WRITE_REV;\n> > +\t\telse\n> > +\t\t\tpack_idx_opts.flags &= ~WRITE_REV;\n> > +\t\treturn 0;\n> > +\t}\n> \n> This turned out delightfully simple. And I guess this is the \"why is\n> WRITE_REV\" caller I asked about from patch 2. It is\n> finish_tmp_packfile() where the magic happens. That unconditionally\n> calls write_rev_file(), but it's a noop if WRITE_REV isn't specified.\n> \n> Makes sense.\n\nOh, one subtlety here: this is in pack-objects itself, _not_ in\ngit-repack. This has bit us before with options like\nrepack.writebitmaps, which was originally pack.writebitmaps and\nintroduced all sorts of awkwardness (because pack-objects serves many\nother purposes besides repacks).\n\nI think this _might_ be OK, because we wouldn't even hit the code-paths\nthat handle this unless we are also writing a .idx (and really, the\npoint is that the two should always be written together or not at all).\n\nSo probably it's fine, but I wonder if we should err on the side of\nconservatism by saying that pack-objects will not change behavior\nwithout receiving a new command-line option, and that repack should\ntrigger that option. I dunno. I guess that it makes it weird with\nrespect to index-pack, which wants to see a very low-level option, too.\nSo maybe this is the best way forward.\n\n(Sorry for being non-committal; I'm mostly thinking out loud).\n\n-Peff\n"},{"id":"415205","messageId":"YA8DlQwSzNZXq+AU@nand.local","threadId":"54962","inReplyTo":"YAtXmie2kHNrcBwY@coredump.intra.peff.net","subject":"Re: [PATCH v2 1/8] packfile: prepare for the existence of '*.rev' files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-25T17:44:53Z","receivedAt":"2021-01-25T17:56:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jan 22, 2021 at 05:54:18PM -0500, Jeff King wrote:\n> All of which is a really verbose way of saying: you might want to add a\n> few words after the comma:\n>\n> In real-world usage, Git is often performing many operations in the\n> revindex (i.e., rather than asking about a single object, we'd\n> generally ask about a range of history).\n>\n> :) But hopefully it shows that including the offsets is not really\n> making things better for the cold cache anyway.\n\nThanks for including a compelling argument in favor of the approach that\nI took in this patch.\n\nI added something along the lines of what you suggested to the final\nparagraph, so now it concludes nicely instead of ending in a comma. I\nbriefly considered whether I should add something about how these\noperations scale and how the warming efforts are really amortized across\nall of the objects, but I decided against it.\n\nI think that this argument is already documented here, and that there's\nno way to concisely state it in an already long patch. Interested\nreaders will easily be able to find our discussion here, which is good.\n\n> > Documentation/technical/pack-format.txt | 17 ++++\n> > builtin/repack.c | 1 +\n> > object-store.h | 3 +\n> > pack-revindex.c | 112 +++++++++++++++++++++---\n> > pack-revindex.h | 7 +-\n> > packfile.c | 13 ++-\n> > packfile.h | 1 +\n> > tmp-objdir.c | 4 +-\n> > 8 files changed, 145 insertions(+), 13 deletions(-)\n>\n> Oh, there's a patch here, too. :)\n\n:-).\n\n> It mostly looks good to me. I agree with Junio that \"compute\" is a\n> better verb than \"load\" for generating the in-memory revindex.\n\nYeah, I settled on load_pack_revindex() either calling\n\"create_pack_revindex_in_memory()\" or \"load_pack_revindex_from_disk()\".\n\n> > +static int load_pack_revindex_from_disk(struct packed_git *p)\n> > +{\n> > +\tchar *revindex_name;\n> > +\tint ret;\n> > +\tif (open_pack_index(p))\n> > +\t\treturn -1;\n> > +\n> > +\trevindex_name = pack_revindex_filename(p);\n> > +\n> > +\tret = load_revindex_from_disk(revindex_name,\n> > +\t\t\t\t p->num_objects,\n> > +\t\t\t\t &p->revindex_map,\n> > +\t\t\t\t &p->revindex_size);\n> > +\tif (ret)\n> > +\t\tgoto cleanup;\n> > +\n> > +\tp->revindex_data = (char *)p->revindex_map + 12;\n>\n> Junio mentioned once spot where we lose constness through a cast. This\n> is another. I wonder if revindex_map should just be a \"char *\" to make\n> pointer arithmetic easier without having to cast.\n>\n> But also...\n>\n> > +\tif (p->revindex)\n> > +\t\treturn p->revindex[pos].nr;\n> > +\telse\n> > +\t\treturn get_be32((char *)p->revindex_data + (pos * sizeof(uint32_t)));\n>\n> If p->revindex_data were \"const uint32_t *\", then this line would just\n> be:\n>\n> return get_be32(p->revindex_data + pos);\n>\n> Not a huge deal either way since the whole point is to abstract this\n> behind a function where it only has to be written once. I don't think\n> there is any downside from the compiler's view (and we already use this\n> trick for the bitmap name-hash cache).\n\nHonestly, I'm not a huge fan of implicitly scaling pos by\nsizeof(*p->revindex_data), but I can understand why it reads more\nclearly here. I don't really feel strongly either way, so I'm happy to\nchange it in favor of your suggestion.\n\nOf course, since RIDX_HEADER_SIZE is in bytes, not uint32_t's (and it\nhas to be, since it's also used in the RIDX_MIN_SIZE macro, which is\ncompared against the st_size of stating the .rev file), you have to do\ngross stuff like:\n\n p->revindex_data = (const uint32_t *)((const char *)p->revindex_map + RIDX_HEADER_SIZE);\n\nBut I guess the tradeoff is worth it, since the readers are easier to\nparse.\n\n> Thinking out loud a bit: a .rev file means we're spending an extra map\n> per pack (but not a descriptor, since we close after mmap). And like the\n> .idx files (but unlike .pack file maps), we don't keep track of these\n> and try to close them when under memory pressure. I think that's\n> probably OK in terms of bytes. It may mean running up against operating\n> system number-of-mmap limits more quickly when you have a very large\n> number of packs, as mentioned in:\n>\n> https://lore.kernel.org/git/20200601044511.GA2529317@coredump.intra.peff.net/\n>\n> But this is probably bumping the number of problematic packs from 30k to\n> 20k. Both are sufficiently ridiculous that I don't think it matters in\n> practice.\n\nAgreed.\n\n> > diff --git a/tmp-objdir.c b/tmp-objdir.c\n> > index 42ed4db5d3..da414df14f 100644\n> > --- a/tmp-objdir.c\n> > +++ b/tmp-objdir.c\n> > @@ -187,7 +187,9 @@ static int pack_copy_priority(const char *name)\n> > \t\treturn 2;\n> > \tif (ends_with(name, \".idx\"))\n> > \t\treturn 3;\n> > -\treturn 4;\n> > +\tif (ends_with(name, \".rev\"))\n> > +\t\treturn 4;\n> > +\treturn 5;\n> > }\n>\n> Probably not super important, but: should the .idx file still come last\n> here? Simultaneous readers won't start using the pack until the .idx\n> file is present. We'd probably prefer they see the whole thing\n> atomically, than see a .idx missing its .rev (they won't ever produce a\n> wrong answer, but they'll generate the in-core revindex on the fly when\n> they don't need to).\n>\n> I guess one could argue that .bitmap files should get similar treatment,\n> but we'd not generally see those in the quarantine objdir anyway, so\n> nobody ever gave it much thought.\n\nYeah, you're right (.idx files should come last, and probably an\nargument to include .bitmap files here, too, exists. I'll leave the\nlatter as #leftoverbits).\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"415227","messageId":"YA8NdHJloTf15qXf@coredump.intra.peff.net","threadId":"54962","inReplyTo":"YA8DlQwSzNZXq+AU@nand.local","subject":"Re: [PATCH v2 1/8] packfile: prepare for the existence of '*.rev' files","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-25T18:27:00Z","receivedAt":"2021-01-25T18:28:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 25, 2021 at 12:44:53PM -0500, Taylor Blau wrote:\n\n> Thanks for including a compelling argument in favor of the approach that\n> I took in this patch.\n> \n> I added something along the lines of what you suggested to the final\n> paragraph, so now it concludes nicely instead of ending in a comma. I\n> briefly considered whether I should add something about how these\n> operations scale and how the warming efforts are really amortized across\n> all of the objects, but I decided against it.\n> \n> I think that this argument is already documented here, and that there's\n> no way to concisely state it in an already long patch. Interested\n> readers will easily be able to find our discussion here, which is good.\n\nThat sounds good. It is sort of arguing against a strawman anyway.\n\n> > It mostly looks good to me. I agree with Junio that \"compute\" is a\n> > better verb than \"load\" for generating the in-memory revindex.\n> \n> Yeah, I settled on load_pack_revindex() either calling\n> \"create_pack_revindex_in_memory()\" or \"load_pack_revindex_from_disk()\".\n\nPerfect.\n\n> > If p->revindex_data were \"const uint32_t *\", then this line would just\n> > be:\n> >\n> > return get_be32(p->revindex_data + pos);\n> >\n> > Not a huge deal either way since the whole point is to abstract this\n> > behind a function where it only has to be written once. I don't think\n> > there is any downside from the compiler's view (and we already use this\n> > trick for the bitmap name-hash cache).\n> \n> Honestly, I'm not a huge fan of implicitly scaling pos by\n> sizeof(*p->revindex_data), but I can understand why it reads more\n> clearly here. I don't really feel strongly either way, so I'm happy to\n> change it in favor of your suggestion.\n> \n> Of course, since RIDX_HEADER_SIZE is in bytes, not uint32_t's (and it\n> has to be, since it's also used in the RIDX_MIN_SIZE macro, which is\n> compared against the st_size of stating the .rev file), you have to do\n> gross stuff like:\n> \n> p->revindex_data = (const uint32_t *)((const char *)p->revindex_map + RIDX_HEADER_SIZE);\n> \n> But I guess the tradeoff is worth it, since the readers are easier to\n> parse.\n\nYeah, that is definitely a downside. Perhaps keeping everything in bytes\nmakes things a bit more obvious. In which case I might suggest that\nrevindex_data just be a \"const char *\". You'd have to scale any pointer\ncomputations at the point of use then, but you'd avoid needing to do any\nextra casting.\n\n-Peff\n"},{"id":"415228","messageId":"xmqqy2ggyh1a.fsf@gitster.c.googlers.com","threadId":"54962","inReplyTo":"YA8DlQwSzNZXq+AU@nand.local","subject":"Re: [PATCH v2 1/8] packfile: prepare for the existence of '*.rev' files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-25T19:04:33Z","receivedAt":"2021-01-25T19:06:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n>> Thinking out loud a bit: a .rev file means we're spending an extra map\n>> per pack (but not a descriptor, since we close after mmap). And like the\n>> .idx files (but unlike .pack file maps), we don't keep track of these\n>> and try to close them when under memory pressure. I think that's\n>> probably OK in terms of bytes. It may mean running up against operating\n>> system number-of-mmap limits more quickly ...\n>> ...\n>> > \tif (ends_with(name, \".idx\"))\n>> > \t\treturn 3;\n>> > -\treturn 4;\n>> > +\tif (ends_with(name, \".rev\"))\n>> > +\t\treturn 4;\n>> > +\treturn 5;\n>> > }\n>>\n>> Probably not super important, but: should the .idx file still come last\n>> here? Simultaneous readers won't start using the pack until the .idx\n>> file is present. We'd probably prefer they see the whole thing\n>> atomically, than see a .idx missing its .rev (they won't ever produce a\n>> wrong answer, but they'll generate the in-core revindex on the fly when\n>> they don't need to).\n\nAt some point, we may want to \n\n - introduce .idx version 3 that is more extensible, so that the\n reverse info is included in one of its chunks;\n\n - make the .rev data for all packs stored as a chunk in .midx, so\n we can first check with .midx and not open any .rev files.\n\neither of which would reduce the numberfrom 30k down to 10k ;-)\n"},{"id":"415232","messageId":"YA8j+3nq8nI/hcIQ@nand.local","threadId":"54962","inReplyTo":"YAtlZ49mTfTGg11/@coredump.intra.peff.net","subject":"Re: [PATCH v2 3/8] builtin/index-pack.c: write reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-25T20:03:07Z","receivedAt":"2021-01-25T20:04:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jan 22, 2021 at 06:53:11PM -0500, Jeff King wrote:\n> On Wed, Jan 13, 2021 at 05:28:15PM -0500, Taylor Blau wrote:\n>\n> > OPTIONS\n> > @@ -33,7 +34,14 @@ OPTIONS\n> > \tfile is constructed from the name of packed archive\n> > \tfile by replacing .pack with .idx (and the program\n> > \tfails if the name of packed archive does not end\n> > -\twith .pack).\n> > +\twith .pack). Incompatible with `--rev-index`.\n>\n> I wondered which option was incompatible, but couldn't see from the\n> context. It is \"index-pack -o\", which kind of makes sense. We can derive\n> \"foo.rev\" from \"foo.idx\", but normally \"-o\" does not do any deriving.\n>\n> So I was all set to say \"OK, we can live without it\", but...\n>\n> > @@ -1824,7 +1851,16 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n> > \tif (from_stdin && hash_algo)\n> > \t\tdie(_(\"--object-format cannot be used with --stdin\"));\n> > \tif (!index_name && pack_name)\n> > -\t\tindex_name = derive_filename(pack_name, \"idx\", &index_name_buf);\n> > +\t\tindex_name = derive_filename(pack_name, \".pack\", \"idx\", &index_name_buf);\n> > +\n> > +\topts.flags &= ~(WRITE_REV | WRITE_REV_VERIFY);\n> > +\tif (rev_index) {\n> > +\t\topts.flags |= verify ? WRITE_REV_VERIFY : WRITE_REV;\n> > +\t\tif (index_name)\n> > +\t\t\trev_index_name = derive_filename(index_name,\n> > +\t\t\t\t\t\t\t \".idx\", \"rev\",\n> > +\t\t\t\t\t\t\t &rev_index_name_buf);\n> > +\t}\n>\n> ...here we do end up deriving \".rev\" from \".idx\" anyway. So I guess we\n> probably could support \"-o\". I also wonder what happens with \"git\n> index-pack -o foo.idx\" when pack.writeReverseIndex is set. It looks like\n> it would just work because of this block. But then shouldn't\n> \"--rev-index\" work, too? And indeed, there is a test for that at the end\n> of the patch! So is the documentation just wrong?\n\nHah! The documentation is just plain wrong. It's been a while, but I\nhave a vague recollection of writing this documentation before changing\nthe implementation of index-pack to allow this. Clearly, I forgot to go\nback to update the broken documentation.\n\nHilariously, there is even a test in t5325 that demonstrates this\nworking! 'git index-pack --rev-index -o other.idx' writes both\n'other.idx' and 'other.rev'. That was easy :-).\n\n> I admit to finding the use of opts.flags versus the rev_index option a\n> bit confusing. It seems like they are doing roughly the same thing, but\n> influenced by different sources. It seems like we should be able to have\n> a single local variable (then that goes on to set opts.flags for any\n> sub-functions we call). Or maybe two, if we need to distinguish config\n> versus command-line, but then they should have clear names\n> (rev_index_config and rev_index_cmdline or something).\n\nYeah, I know. It's because we already pass a pointer to a struct\npack_idx_option to git_index_pack_config(), so in effect the 'flags' on\nthat struct *is* rev_index_config.\n\nIt's a little ugly, I agree, but I'm skeptical that the effort to clean\nit up is worth it, mostly because the pack_idx_option struct probably\nshouldn't be part of the index-pack builtin in the first place.\n\n> As an aside, looking at derive_filename(), it seems a bit weird that one\n> argument has a dot in the suffix and the other does not. I guess you are\n> following the convention from write_special_file(), which omits it in\n> the newly-added suffix. But it is slightly awkward to omit it for the\n> old suffix in derive_filename(), because we want to strip_suffix() it\n> all at once.\n\n> Probably not that big a deal, but if anybody feels strongly, then\n> derive_filename() could do:\n>\n> if (!strip_suffix(pack_name, strip, &len) ||\n> !len || pack_name[len] != '.')\n> \tdie(\"does not end in .%s\", strip);\n\nThat does make the callers look nicer, but it needs an extra two things:\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex ef2874a8e6..c758f3b8e9 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1441,11 +1441,10 @@ static const char *derive_filename(const char *pack_name, const char *strip,\n {\n size_t len;\n if (!strip_suffix(pack_name, strip, &len) || !len ||\n- pack_name[len] != '.')\n+ pack_name[len - 1] != '.')\n die(_(\"packfile name '%s' does not end with '.%s'\"),\n pack_name, strip);\n strbuf_add(buf, pack_name, len);\n- strbuf_addch(buf, '.');\n strbuf_addstr(buf, suffix);\n return buf->buf;\n }\n\nAnd then it does what you are looking for. I'll pull that change out\nwith your Suggested-by as a preparatory commit right before this one.\n\n> > @@ -1578,6 +1591,12 @@ static int git_index_pack_config(const char *k, const char *v, void *cb)\n> > \t\t}\n> > \t\treturn 0;\n> > \t}\n> > +\tif (!strcmp(k, \"pack.writereverseindex\")) {\n> > +\t\tif (git_config_bool(k, v))\n> > +\t\t\topts->flags |= WRITE_REV;\n> > +\t\telse\n> > +\t\t\topts->flags &= ~WRITE_REV;\n> > +\t}\n> > \treturn git_default_config(k, v, cb);\n> > }\n>\n> IMHO we'll eventually want to turn this feature on by default. In which\n> case we'll have to update every caller which is checking the config\n> manually. Should we hide this in a function that looks up the config,\n> and sets the default? Or alternatively, I guess, they could all use some\n> shared initializer for \"flags\".\n\nNote that there are only two such callers, so I'm not sure the effort to\nextract this would be worth it.\n\n> > +\t# Intentionally corrupt the reverse index.\n> > +\tchmod u+w $rev &&\n> > +\tprintf \"xxxx\" | dd of=$rev bs=1 count=4 conv=notrunc &&\n> > +\n> > +\ttest_must_fail git index-pack --rev-index --verify \\\n> > +\t\t$packdir/pack-$pack.pack 2>err &&\n> > +\tgrep \"validation error\" err\n> > +'\n>\n> This isn't that subtle of a corruption, because we are corrupting the\n> first 4 bytes, which is the magic signature. Maybe something further in\n> the actual data would be interesting instead of or in addition?\n>\n> I dunno. There are a lot of edge cases around corruption (likewise, we\n> might care how the normal reading code-path perceives a signature\n> corruption like this). I'm not sure it's all that interesting to test\n> all of them.\n\nAgreed, I think what is here (even though it's not a severe corruption)\nwould be sufficient to make me feel good about our error handling in\ngeneral.\n\nThanks,\nTaylor\n"},{"id":"415234","messageId":"YA8oPcvS0n8pUmcB@nand.local","threadId":"54962","inReplyTo":"YAto/ndOVfKSkizS@coredump.intra.peff.net","subject":"Re: [PATCH v2 4/8] builtin/pack-objects.c: respect 'pack.writeReverseIndex'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-25T20:21:17Z","receivedAt":"2021-01-25T20:22:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jan 22, 2021 at 07:08:30PM -0500, Jeff King wrote:\n> On Fri, Jan 22, 2021 at 06:57:35PM -0500, Jeff King wrote:\n>\n> > > +\tif (!strcmp(k, \"pack.writereverseindex\")) {\n> > > +\t\tif (git_config_bool(k, v))\n> > > +\t\t\tpack_idx_opts.flags |= WRITE_REV;\n> > > +\t\telse\n> > > +\t\t\tpack_idx_opts.flags &= ~WRITE_REV;\n> > > +\t\treturn 0;\n> > > +\t}\n> >\n> > This turned out delightfully simple. And I guess this is the \"why is\n> > WRITE_REV\" caller I asked about from patch 2. It is\n> > finish_tmp_packfile() where the magic happens. That unconditionally\n> > calls write_rev_file(), but it's a noop if WRITE_REV isn't specified.\n> >\n> > Makes sense.\n>\n> Oh, one subtlety here: this is in pack-objects itself, _not_ in\n> git-repack. This has bit us before with options like\n> repack.writebitmaps, which was originally pack.writebitmaps and\n> introduced all sorts of awkwardness (because pack-objects serves many\n> other purposes besides repacks).\n\nI'd think that we'd want a single option to control whether or not\nreverse indexes are written to disk. I briefly considered (and I believe\nthat you and I even discussed) having options to control this behavior\nper command, but it got out of hand quickly.\n\nAnd that might have been OK, but I don't think the complexity was even\nwarranted, because really on-disk reverse indexes are an all-or-nothing\nthing. That is: either you want to have revindexes accompanying .packs,\nor you don't. IOW, it doesn't matter whether those packs were pushed to\nus, or generated during repack, or from another pack-objects or what\nhave you.\n\nThanks,\nTaylor\n"},{"id":"415236","messageId":"YA8vJcb7YbLwmF/V@coredump.intra.peff.net","threadId":"54962","inReplyTo":"YA8oPcvS0n8pUmcB@nand.local","subject":"Re: [PATCH v2 4/8] builtin/pack-objects.c: respect 'pack.writeReverseIndex'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-25T20:50:45Z","receivedAt":"2021-01-25T20:51:56Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 25, 2021 at 03:21:17PM -0500, Taylor Blau wrote:\n\n> > Oh, one subtlety here: this is in pack-objects itself, _not_ in\n> > git-repack. This has bit us before with options like\n> > repack.writebitmaps, which was originally pack.writebitmaps and\n> > introduced all sorts of awkwardness (because pack-objects serves many\n> > other purposes besides repacks).\n> \n> I'd think that we'd want a single option to control whether or not\n> reverse indexes are written to disk. I briefly considered (and I believe\n> that you and I even discussed) having options to control this behavior\n> per command, but it got out of hand quickly.\n\nYeah, I think per-command is something we should avoid.\n\nMy concern is mostly that pack-objects in a non-repack setting would\naccidentally try to write a .rev file (or complain that it cannot). But\nagain, we already have the same potential issue for .idx files, so by\nfollowing those code paths as you did, we should be OK.\n\nAnd certainly we test that code path a lot in the normal test suite\n(i.e., every fetch or push), so your run-the-test-suite-with-rev-files\npatches from later would presumably catch it otherwise.\n\n> And that might have been OK, but I don't think the complexity was even\n> warranted, because really on-disk reverse indexes are an all-or-nothing\n> thing. That is: either you want to have revindexes accompanying .packs,\n> or you don't. IOW, it doesn't matter whether those packs were pushed to\n> us, or generated during repack, or from another pack-objects or what\n> have you.\n\nRight, I agree with all of that.\n\n-Peff\n"},{"id":"415247","messageId":"cover.1611617819.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1610129989.git.me@ttaylorr.com","subject":"[PATCH v3 00/10] pack-revindex: introduce on-disk '.rev' format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-25T23:37:09Z","receivedAt":"2021-01-25T23:38:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi,\n\nHere is a third reroll of my series to introduce an on-disk format for the\nreverse index. Since the first series (to introduce a new API) has been merged\nto 'next', this series has been rebased onto 'next', too.\n\nThis version is largely unchanged from the previous, with the following\nexceptions:\n\n - Some commit messages and documentation have been clarified to address\n reviewer questions (including why we don't want to store offsets in the .rev\n file, and that all four-byte numbers are in network order, etc.).\n\n - A few more things are done on opening the revindex file, namely checking\n that the version/hash ID are known.\n\n - The idiom \"load in memory\" has been removed.\n\n - Other minor changes.\n\nA range diff is below.\n\nThanks in advance for your review.\n\nTaylor Blau (10):\n packfile: prepare for the existence of '*.rev' files\n pack-write.c: prepare to write 'pack-*.rev' files\n builtin/index-pack.c: allow stripping arbitrary extensions\n builtin/index-pack.c: write reverse indexes\n builtin/pack-objects.c: respect 'pack.writeReverseIndex'\n Documentation/config/pack.txt: advertise 'pack.writeReverseIndex'\n t: prepare for GIT_TEST_WRITE_REV_INDEX\n t: support GIT_TEST_WRITE_REV_INDEX\n pack-revindex: ensure that on-disk reverse indexes are given\n precedence\n t5325: check both on-disk and in-memory reverse index\n\n Documentation/config/pack.txt | 7 ++\n Documentation/git-index-pack.txt | 18 ++-\n Documentation/technical/pack-format.txt | 20 ++++\n builtin/index-pack.c | 69 +++++++++--\n builtin/pack-objects.c | 9 ++\n builtin/repack.c | 1 +\n ci/run-build-and-tests.sh | 1 +\n object-store.h | 3 +\n pack-revindex.c | 148 ++++++++++++++++++++++--\n pack-revindex.h | 14 ++-\n pack-write.c | 120 ++++++++++++++++++-\n pack.h | 4 +\n packfile.c | 13 ++-\n packfile.h | 1 +\n t/README | 3 +\n t/t5319-multi-pack-index.sh | 5 +-\n t/t5325-reverse-index.sh | 142 +++++++++++++++++++++++\n t/t5604-clone-reference.sh | 2 +-\n t/t5702-protocol-v2.sh | 12 +-\n t/t6500-gc.sh | 6 +-\n t/t9300-fast-import.sh | 5 +-\n tmp-objdir.c | 6 +-\n 22 files changed, 567 insertions(+), 42 deletions(-)\n create mode 100755 t/t5325-reverse-index.sh\n\nRange-diff against v2:\n21: 6742c15c84 ! 1: 6f8b70ab27 packfile: prepare for the existence of '*.rev' files\n @@ Commit message\n is versioned, so we are free to pursue these in the future.) But, cold\n cache performance likely isn't interesting outside of one-off cases like\n asking for the size of an object directly. In real-world usage, Git is\n - often performing many operations in the revindex,\n + often performing many operations in the revindex (i.e., asking about\n + many objects rather than a single one).\n\n The trade-off is worth it, since we will avoid the vast majority of the\n cost of generating the revindex that the extra pointer chase will look\n @@ Documentation/technical/pack-format.txt: Pack file entry: <+\n +\n + - A 4-byte magic number '0x52494458' ('RIDX').\n +\n -+ - A 4-byte version identifier (= 1)\n ++ - A 4-byte version identifier (= 1).\n +\n -+ - A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256)\n ++ - A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256).\n +\n -+ - A table of index positions, sorted by their corresponding offsets in the\n -+ packfile.\n ++ - A table of index positions (one per packed object, num_objects in\n ++ total, each a 4-byte unsigned integer in network order), sorted by\n ++ their corresponding offsets in the packfile.\n +\n + - A trailer, containing a:\n +\n + checksum of the corresponding packfile, and\n +\n + a checksum of all of the above.\n ++\n ++All 4-byte numbers are in network order.\n +\n == multi-pack-index (MIDX) files have the following format:\n\n @@ object-store.h: struct packed_git {\n \t\t multi_pack_index:1;\n \tunsigned char hash[GIT_MAX_RAWSZ];\n \tstruct revindex_entry *revindex;\n -+\tconst void *revindex_data;\n -+\tconst void *revindex_map;\n ++\tconst uint32_t *revindex_data;\n ++\tconst uint32_t *revindex_map;\n +\tsize_t revindex_size;\n \t/* something like \".git/objects/pack/xxxxx.pack\" */\n \tchar pack_name[FLEX_ARRAY]; /* more */\n @@ pack-revindex.c: static void create_pack_revindex(struct packed_git *p)\n }\n\n -int load_pack_revindex(struct packed_git *p)\n -+static int load_pack_revindex_from_memory(struct packed_git *p)\n ++static int create_pack_revindex_in_memory(struct packed_git *p)\n {\n -\tif (!p->revindex) {\n -\t\tif (open_pack_index(p))\n @@ pack-revindex.c: static void create_pack_revindex(struct packed_git *p)\n +\treturn xstrfmt(\"%.*s.rev\", (int)len, p->pack_name);\n +}\n +\n -+#define RIDX_MIN_SIZE (12 + (2 * the_hash_algo->rawsz))\n ++#define RIDX_HEADER_SIZE (12)\n ++#define RIDX_MIN_SIZE (RIDX_HEADER_SIZE + (2 * the_hash_algo->rawsz))\n ++\n ++struct revindex_header {\n ++\tuint32_t signature;\n ++\tuint32_t version;\n ++\tuint32_t hash_id;\n ++};\n +\n +static int load_revindex_from_disk(char *revindex_name,\n +\t\t\t\t uint32_t num_objects,\n -+\t\t\t\t const void **data, size_t *len)\n ++\t\t\t\t const uint32_t **data_p, size_t *len_p)\n +{\n +\tint fd, ret = 0;\n +\tstruct stat st;\n ++\tvoid *data = NULL;\n +\tsize_t revindex_size;\n ++\tstruct revindex_header *hdr;\n +\n +\tfd = git_open(revindex_name);\n +\n @@ pack-revindex.c: static void create_pack_revindex(struct packed_git *p)\n +\t\tgoto cleanup;\n +\t}\n +\n -+\t*len = revindex_size;\n -+\t*data = xmmap(NULL, revindex_size, PROT_READ, MAP_PRIVATE, fd, 0);\n ++\tdata = xmmap(NULL, revindex_size, PROT_READ, MAP_PRIVATE, fd, 0);\n ++\thdr = data;\n ++\n ++\tif (ntohl(hdr->signature) != RIDX_SIGNATURE) {\n ++\t\tret = error(_(\"reverse-index file %s has unknown signature\"), revindex_name);\n ++\t\tgoto cleanup;\n ++\t}\n ++\tif (ntohl(hdr->version) != 1) {\n ++\t\tret = error(_(\"reverse-index file %s has unsupported version %\"PRIu32),\n ++\t\t\t revindex_name, ntohl(hdr->version));\n ++\t\tgoto cleanup;\n ++\t}\n ++\tif (!(ntohl(hdr->hash_id) == 1 || ntohl(hdr->hash_id) == 2)) {\n ++\t\tret = error(_(\"reverse-index file %s has unsupported hash id %\"PRIu32),\n ++\t\t\t revindex_name, ntohl(hdr->hash_id));\n ++\t\tgoto cleanup;\n ++\t}\n +\n +cleanup:\n ++\tif (ret) {\n ++\t\tif (data)\n ++\t\t\tmunmap(data, revindex_size);\n ++\t} else {\n ++\t\t*len_p = revindex_size;\n ++\t\t*data_p = (const uint32_t *)data;\n ++\t}\n ++\n +\tclose(fd);\n +\treturn ret;\n +}\n @@ pack-revindex.c: static void create_pack_revindex(struct packed_git *p)\n +\tif (ret)\n +\t\tgoto cleanup;\n +\n -+\tp->revindex_data = (char *)p->revindex_map + 12;\n ++\tp->revindex_data = (const uint32_t *)((const char *)p->revindex_map + RIDX_HEADER_SIZE);\n +\n +cleanup:\n +\tfree(revindex_name);\n @@ pack-revindex.c: static void create_pack_revindex(struct packed_git *p)\n +\n +\tif (!load_pack_revindex_from_disk(p))\n +\t\treturn 0;\n -+\telse if (!load_pack_revindex_from_memory(p))\n ++\telse if (!create_pack_revindex_in_memory(p))\n +\t\treturn 0;\n +\treturn -1;\n +}\n @@ pack-revindex.c: int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_\n +\tif (p->revindex)\n +\t\treturn p->revindex[pos].nr;\n +\telse\n -+\t\treturn get_be32((char *)p->revindex_data + (pos * sizeof(uint32_t)));\n ++\t\treturn get_be32(p->revindex_data + pos);\n }\n\n off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n @@ pack-revindex.c: int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_\n }\n\n ## pack-revindex.h ##\n -@@ pack-revindex.h: struct packed_git;\n +@@\n + * can be found\n + */\n +\n ++#define RIDX_SIGNATURE 0x52494458 /* \"RIDX\" */\n ++#define RIDX_VERSION 1\n ++\n + struct packed_git;\n +\n /*\n * load_pack_revindex populates the revindex's internal data-structures for the\n * given pack, returning zero on success and a negative value otherwise.\n + *\n -+ * If a '.rev' file is present, it is checked for consistency, mmap'd, and\n -+ * pointers are assigned into it (instead of using the in-memory variant).\n ++ * If a '.rev' file is present it is mmap'd, and pointers are assigned into it\n ++ * (instead of using the in-memory variant).\n */\n int load_pack_revindex(struct packed_git *p);\n\n @@ packfile.h: uint32_t get_pack_fanout(struct packed_git *p, uint32_t value);\n\n ## tmp-objdir.c ##\n @@ tmp-objdir.c: static int pack_copy_priority(const char *name)\n + \t\treturn 1;\n + \tif (ends_with(name, \".pack\"))\n \t\treturn 2;\n - \tif (ends_with(name, \".idx\"))\n +-\tif (ends_with(name, \".idx\"))\n ++\tif (ends_with(name, \".rev\"))\n \t\treturn 3;\n -\treturn 4;\n -+\tif (ends_with(name, \".rev\"))\n ++\tif (ends_with(name, \".idx\"))\n +\t\treturn 4;\n +\treturn 5;\n }\n22: 8648c87fa7 ! 2: 5efc870742 pack-write.c: prepare to write 'pack-*.rev' files\n @@ Commit message\n from within 'write_rev_file()', which is called from\n 'finish_tmp_packfile()'.\n\n + Similar to the process by which the reverse index is computed in memory,\n + these new paths also have to sort a list of objects by their offsets\n + within a packfile. These new paths use a qsort() (as opposed to a radix\n + sort), since our specialized radix sort requires a full revindex_entry\n + struct per object, which is more memory than we need to allocate.\n +\n + The qsort is obviously slower, but the theoretical slowdown would\n + require a repository with a large amount of objects, likely implying\n + that the time spent in, say, pack-objects during a repack would dominate\n + the overall runtime.\n +\n Signed-off-by: Taylor Blau \n\n ## pack-write.c ##\n @@ pack-write.c: const char *write_idx_file(const char *index_name, struct pack_idx\n +\treturn 0;\n +}\n +\n -+#define RIDX_SIGNATURE 0x52494458 /* \"RIDX\" */\n -+#define RIDX_VERSION 1\n -+\n +static void write_rev_header(struct hashfile *f)\n +{\n +\tuint32_t oid_version;\n @@ pack-write.c: void finish_tmp_packfile(struct strbuf *name_buffer,\n +\n \tfree((void *)idx_tmp_name);\n }\n +\n\n ## pack.h ##\n @@ pack.h: struct pack_idx_option {\n @@ pack.h: struct pack_idx_option {\n\n \tuint32_t version;\n \tuint32_t off32_limit;\n -@@ pack.h: off_t write_pack_header(struct hashfile *f, uint32_t);\n - void fixup_pack_header_footer(int, unsigned char *, const char *, uint32_t, unsigned char *, off_t);\n - char *index_pack_lockfile(int fd);\n +@@ pack.h: struct ref;\n +\n + void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought);\n\n +const char *write_rev_file(const char *rev_name, struct pack_idx_entry **objects, uint32_t nr_objects, const unsigned char *hash, unsigned flags);\n +\n -: ---------- > 3: 8a3e70454b builtin/index-pack.c: allow stripping arbitrary extensions\n23: 5b18ada611 ! 4: a8ee59fccf builtin/index-pack.c: write reverse indexes\n @@ Documentation/git-index-pack.txt: git-index-pack - Build pack index file for an\n\n OPTIONS\n @@ Documentation/git-index-pack.txt: OPTIONS\n - \tfile is constructed from the name of packed archive\n - \tfile by replacing .pack with .idx (and the program\n \tfails if the name of packed archive does not end\n --\twith .pack).\n -+\twith .pack). Incompatible with `--rev-index`.\n -+\n + \twith .pack).\n +\n +--[no-]rev-index::\n +\tWhen this flag is provided, generate a reverse index\n +\t(a `.rev` file) corresponding to the given pack. If\n +\t`--verify` is given, ensure that the existing\n +\treverse index is correct. Takes precedence over\n +\t`pack.writeReverseIndex`.\n -\n ++\n --stdin::\n \tWhen this flag is provided, the pack is read from stdin\n + \tinstead and a copy is then written to . If\n\n ## builtin/index-pack.c ##\n @@\n @@ builtin/index-pack.c\n\n struct object_entry {\n \tstruct pack_idx_entry idx;\n -@@ builtin/index-pack.c: static void fix_unresolved_deltas(struct hashfile *f)\n - \tfree(sorted_by_pos);\n - }\n -\n --static const char *derive_filename(const char *pack_name, const char *suffix,\n --\t\t\t\t struct strbuf *buf)\n -+static const char *derive_filename(const char *pack_name, const char *strip,\n -+\t\t\t\t const char *suffix, struct strbuf *buf)\n - {\n - \tsize_t len;\n --\tif (!strip_suffix(pack_name, \".pack\", &len))\n --\t\tdie(_(\"packfile name '%s' does not end with '.pack'\"),\n --\t\t pack_name);\n -+\tif (!strip_suffix(pack_name, strip, &len))\n -+\t\tdie(_(\"packfile name '%s' does not end with '%s'\"),\n -+\t\t pack_name, strip);\n - \tstrbuf_add(buf, pack_name, len);\n - \tstrbuf_addch(buf, '.');\n - \tstrbuf_addstr(buf, suffix);\n -@@ builtin/index-pack.c: static void write_special_file(const char *suffix, const char *msg,\n - \tint msg_len = strlen(msg);\n -\n - \tif (pack_name)\n --\t\tfilename = derive_filename(pack_name, suffix, &name_buf);\n -+\t\tfilename = derive_filename(pack_name, \".pack\", suffix, &name_buf);\n - \telse\n - \t\tfilename = odb_pack_name(&name_buf, hash, suffix);\n -\n @@ builtin/index-pack.c: static void write_special_file(const char *suffix, const char *msg,\n\n static void final(const char *final_pack_name, const char *curr_pack_name,\n @@ builtin/index-pack.c: int cmd_index_pack(int argc, const char **argv, const char\n \t\t\t\tusage(index_pack_usage);\n \t\t\tcontinue;\n @@ builtin/index-pack.c: int cmd_index_pack(int argc, const char **argv, const char *prefix)\n - \tif (from_stdin && hash_algo)\n - \t\tdie(_(\"--object-format cannot be used with --stdin\"));\n \tif (!index_name && pack_name)\n --\t\tindex_name = derive_filename(pack_name, \"idx\", &index_name_buf);\n -+\t\tindex_name = derive_filename(pack_name, \".pack\", \"idx\", &index_name_buf);\n -+\n + \t\tindex_name = derive_filename(pack_name, \"pack\", \"idx\", &index_name_buf);\n +\n +\topts.flags &= ~(WRITE_REV | WRITE_REV_VERIFY);\n +\tif (rev_index) {\n +\t\topts.flags |= verify ? WRITE_REV_VERIFY : WRITE_REV;\n +\t\tif (index_name)\n +\t\t\trev_index_name = derive_filename(index_name,\n -+\t\t\t\t\t\t\t \".idx\", \"rev\",\n ++\t\t\t\t\t\t\t \"idx\", \"rev\",\n +\t\t\t\t\t\t\t &rev_index_name_buf);\n +\t}\n -\n ++\n \tif (verify) {\n \t\tif (!index_name)\n + \t\t\tdie(_(\"--verify with no packfile name given\"));\n @@ builtin/index-pack.c: int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tfor (i = 0; i < nr_objects; i++)\n \t\tidx_objects[i] = &objects[i].idx;\n24: 68bde3ea97 = 5: 5bebe05a16 builtin/pack-objects.c: respect 'pack.writeReverseIndex'\n25: 38a253d0ce = 6: 7e29f2d3a0 Documentation/config/pack.txt: advertise 'pack.writeReverseIndex'\n26: 12cdf2d67a = 7: 7cf16485cc t: prepare for GIT_TEST_WRITE_REV_INDEX\n27: 6b647d9775 ! 8: 02550a251d t: support GIT_TEST_WRITE_REV_INDEX\n @@ builtin/pack-objects.c: int cmd_pack_objects(int argc, const char **argv, const\n\n ## ci/run-build-and-tests.sh ##\n @@ ci/run-build-and-tests.sh: linux-gcc)\n - \texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n \texport GIT_TEST_ADD_I_USE_BUILTIN=1\n + \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n +\texport GIT_TEST_WRITE_REV_INDEX=1\n \tmake test\n \t;;\n @@ pack-revindex.h\n * can be found\n */\n\n ++\n + #define RIDX_SIGNATURE 0x52494458 /* \"RIDX\" */\n + #define RIDX_VERSION 1\n +\n +#define GIT_TEST_WRITE_REV_INDEX \"GIT_TEST_WRITE_REV_INDEX\"\n +\n struct packed_git;\n28: 48926ae182 ! 9: a66d2f9f7c pack-revindex: ensure that on-disk reverse indexes are given precedence\n @@ pack-revindex.c\n \toff_t offset;\n @@ pack-revindex.c: static void create_pack_revindex(struct packed_git *p)\n\n - static int load_pack_revindex_from_memory(struct packed_git *p)\n + static int create_pack_revindex_in_memory(struct packed_git *p)\n {\n +\tif (git_env_bool(GIT_TEST_REV_INDEX_DIE_IN_MEMORY, 0))\n +\t\tdie(\"dying as requested by '%s'\",\n @@ pack-revindex.c: static void create_pack_revindex(struct packed_git *p)\n\n ## pack-revindex.h ##\n @@\n - */\n + #define RIDX_VERSION 1\n\n #define GIT_TEST_WRITE_REV_INDEX \"GIT_TEST_WRITE_REV_INDEX\"\n +#define GIT_TEST_REV_INDEX_DIE_IN_MEMORY \"GIT_TEST_REV_INDEX_DIE_IN_MEMORY\"\n -: ---------- > 10: 38c8afabf2 t5325: check both on-disk and in-memory reverse index\n--\n2.30.0.138.g6d7191ea01\n"},{"id":"415248","messageId":"6f8b70ab276c0579c957c315743fdab63462a605.1611617820.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1611617819.git.me@ttaylorr.com","subject":"[PATCH v3 01/10] packfile: prepare for the existence of '*.rev' files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-25T23:37:14Z","receivedAt":"2021-01-25T23:38:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Specify the format of the on-disk reverse index 'pack-*.rev' file, as\nwell as prepare the code for the existence of such files.\n\nThe reverse index maps from pack relative positions (i.e., an index into\nthe array of object which is sorted by their offsets within the\npackfile) to their position within the 'pack-*.idx' file. Today, this is\ndone by building up a list of (off_t, uint32_t) tuples for each object\n(the off_t corresponding to that object's offset, and the uint32_t\ncorresponding to its position in the index). To convert between pack and\nindex position quickly, this array of tuples is radix sorted based on\nits offset.\n\nThis has two major drawbacks:\n\nFirst, the in-memory cost scales linearly with the number of objects in\na pack. Each 'struct revindex_entry' is sizeof(off_t) +\nsizeof(uint32_t) + padding bytes for a total of 16.\n\nTo observe this, force Git to load the reverse index by, for e.g.,\nrunning 'git cat-file --batch-check=\"%(objectsize:disk)\"'. When asking\nfor a single object in a fresh clone of the kernel, Git needs to\nallocate 120+ MB of memory in order to hold the reverse index in memory.\n\nSecond, the cost to sort also scales with the size of the pack.\nLuckily, this is a linear function since 'load_pack_revindex()' uses a\nradix sort, but this cost still must be paid once per pack per process.\n\nAs an example, it takes ~60x longer to print the _size_ of an object as\nit does to print that entire object's _contents_:\n\n Benchmark #1: git.compile cat-file --batch /dev/null\n Time (mean ± σ): 22.6 ms ± 0.5 ms [User: 2.4 ms, System: 7.9 ms]\n Range (min … max): 21.4 ms … 23.5 ms 41 runs\n\n Benchmark #2: git.compile cat-file --batch /dev/null\n Time (mean ± σ): 17.2 ms ± 0.7 ms [User: 2.8 ms, System: 5.5 ms]\n Range (min … max): 15.6 ms … 18.2 ms 45 runs\n\n(Numbers taken in the kernel after cheating and using the next patch to\ngenerate a reverse index). There are a couple of approaches to improve\ncold cache performance not pursued here:\n\n - We could include the object offsets in the reverse index format.\n Predictably, this does result in fewer page faults, but it triples\n the size of the file, while simultaneously duplicating a ton of data\n already available in the .idx file. (This was the original way I\n implemented the format, and it did show\n `--batch-check='%(objectsize:disk)'` winning out against `--batch`.)\n\n On the other hand, this increase in size also results in a large\n block-cache footprint, which could potentially hurt other workloads.\n\n - We could store the mapping from pack to index position in more\n cache-friendly way, like constructing a binary search tree from the\n table and writing the values in breadth-first order. This would\n result in much better locality, but the price you pay is trading\n O(1) lookup in 'pack_pos_to_index()' for an O(log n) one (since you\n can no longer directly index the table).\n\nSo, neither of these approaches are taken here. (Thankfully, the format\nis versioned, so we are free to pursue these in the future.) But, cold\ncache performance likely isn't interesting outside of one-off cases like\nasking for the size of an object directly. In real-world usage, Git is\noften performing many operations in the revindex (i.e., asking about\nmany objects rather than a single one).\n\nThe trade-off is worth it, since we will avoid the vast majority of the\ncost of generating the revindex that the extra pointer chase will look\nlike noise in the following patch's benchmarks.\n\nThis patch describes the format and prepares callers (like in\npack-revindex.c) to be able to read *.rev files once they exist. An\nimplementation of the writer will appear in the next patch, and callers\nwill gradually begin to start using the writer in the patches that\nfollow after that.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/technical/pack-format.txt | 20 ++++\n builtin/repack.c | 1 +\n object-store.h | 3 +\n pack-revindex.c | 144 ++++++++++++++++++++++--\n pack-revindex.h | 10 +-\n packfile.c | 13 ++-\n packfile.h | 1 +\n tmp-objdir.c | 6 +-\n 8 files changed, 184 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 96d2fc589f..8833b71c8b 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -274,6 +274,26 @@ Pack file entry: <+\n \n Index checksum of all of the above.\n \n+== pack-*.rev files have the format:\n+\n+ - A 4-byte magic number '0x52494458' ('RIDX').\n+\n+ - A 4-byte version identifier (= 1).\n+\n+ - A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256).\n+\n+ - A table of index positions (one per packed object, num_objects in\n+ total, each a 4-byte unsigned integer in network order), sorted by\n+ their corresponding offsets in the packfile.\n+\n+ - A trailer, containing a:\n+\n+ checksum of the corresponding packfile, and\n+\n+ a checksum of all of the above.\n+\n+All 4-byte numbers are in network order.\n+\n == multi-pack-index (MIDX) files have the following format:\n \n The multi-pack-index files refer to multiple pack-files and loose objects.\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 2158b48f4c..01440de2d5 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -209,6 +209,7 @@ static struct {\n } exts[] = {\n \t{\".pack\"},\n \t{\".idx\"},\n+\t{\".rev\", 1},\n \t{\".bitmap\", 1},\n \t{\".promisor\", 1},\n };\ndiff --git a/object-store.h b/object-store.h\nindex c4fc9dd74e..541dab0858 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -85,6 +85,9 @@ struct packed_git {\n \t\t multi_pack_index:1;\n \tunsigned char hash[GIT_MAX_RAWSZ];\n \tstruct revindex_entry *revindex;\n+\tconst uint32_t *revindex_data;\n+\tconst uint32_t *revindex_map;\n+\tsize_t revindex_size;\n \t/* something like \".git/objects/pack/xxxxx.pack\" */\n \tchar pack_name[FLEX_ARRAY]; /* more */\n };\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 5e69bc7372..a174fa5388 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -164,16 +164,130 @@ static void create_pack_revindex(struct packed_git *p)\n \tsort_revindex(p->revindex, num_ent, p->pack_size);\n }\n \n-int load_pack_revindex(struct packed_git *p)\n+static int create_pack_revindex_in_memory(struct packed_git *p)\n {\n-\tif (!p->revindex) {\n-\t\tif (open_pack_index(p))\n-\t\t\treturn -1;\n-\t\tcreate_pack_revindex(p);\n-\t}\n+\tif (open_pack_index(p))\n+\t\treturn -1;\n+\tcreate_pack_revindex(p);\n \treturn 0;\n }\n \n+static char *pack_revindex_filename(struct packed_git *p)\n+{\n+\tsize_t len;\n+\tif (!strip_suffix(p->pack_name, \".pack\", &len))\n+\t\tBUG(\"pack_name does not end in .pack\");\n+\treturn xstrfmt(\"%.*s.rev\", (int)len, p->pack_name);\n+}\n+\n+#define RIDX_HEADER_SIZE (12)\n+#define RIDX_MIN_SIZE (RIDX_HEADER_SIZE + (2 * the_hash_algo->rawsz))\n+\n+struct revindex_header {\n+\tuint32_t signature;\n+\tuint32_t version;\n+\tuint32_t hash_id;\n+};\n+\n+static int load_revindex_from_disk(char *revindex_name,\n+\t\t\t\t uint32_t num_objects,\n+\t\t\t\t const uint32_t **data_p, size_t *len_p)\n+{\n+\tint fd, ret = 0;\n+\tstruct stat st;\n+\tvoid *data = NULL;\n+\tsize_t revindex_size;\n+\tstruct revindex_header *hdr;\n+\n+\tfd = git_open(revindex_name);\n+\n+\tif (fd < 0) {\n+\t\tret = -1;\n+\t\tgoto cleanup;\n+\t}\n+\tif (fstat(fd, &st)) {\n+\t\tret = error_errno(_(\"failed to read %s\"), revindex_name);\n+\t\tgoto cleanup;\n+\t}\n+\n+\trevindex_size = xsize_t(st.st_size);\n+\n+\tif (revindex_size < RIDX_MIN_SIZE) {\n+\t\tret = error(_(\"reverse-index file %s is too small\"), revindex_name);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (revindex_size - RIDX_MIN_SIZE != st_mult(sizeof(uint32_t), num_objects)) {\n+\t\tret = error(_(\"reverse-index file %s is corrupt\"), revindex_name);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tdata = xmmap(NULL, revindex_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\thdr = data;\n+\n+\tif (ntohl(hdr->signature) != RIDX_SIGNATURE) {\n+\t\tret = error(_(\"reverse-index file %s has unknown signature\"), revindex_name);\n+\t\tgoto cleanup;\n+\t}\n+\tif (ntohl(hdr->version) != 1) {\n+\t\tret = error(_(\"reverse-index file %s has unsupported version %\"PRIu32),\n+\t\t\t revindex_name, ntohl(hdr->version));\n+\t\tgoto cleanup;\n+\t}\n+\tif (!(ntohl(hdr->hash_id) == 1 || ntohl(hdr->hash_id) == 2)) {\n+\t\tret = error(_(\"reverse-index file %s has unsupported hash id %\"PRIu32),\n+\t\t\t revindex_name, ntohl(hdr->hash_id));\n+\t\tgoto cleanup;\n+\t}\n+\n+cleanup:\n+\tif (ret) {\n+\t\tif (data)\n+\t\t\tmunmap(data, revindex_size);\n+\t} else {\n+\t\t*len_p = revindex_size;\n+\t\t*data_p = (const uint32_t *)data;\n+\t}\n+\n+\tclose(fd);\n+\treturn ret;\n+}\n+\n+static int load_pack_revindex_from_disk(struct packed_git *p)\n+{\n+\tchar *revindex_name;\n+\tint ret;\n+\tif (open_pack_index(p))\n+\t\treturn -1;\n+\n+\trevindex_name = pack_revindex_filename(p);\n+\n+\tret = load_revindex_from_disk(revindex_name,\n+\t\t\t\t p->num_objects,\n+\t\t\t\t &p->revindex_map,\n+\t\t\t\t &p->revindex_size);\n+\tif (ret)\n+\t\tgoto cleanup;\n+\n+\tp->revindex_data = (const uint32_t *)((const char *)p->revindex_map + RIDX_HEADER_SIZE);\n+\n+cleanup:\n+\tfree(revindex_name);\n+\treturn ret;\n+}\n+\n+int load_pack_revindex(struct packed_git *p)\n+{\n+\tif (p->revindex || p->revindex_data)\n+\t\treturn 0;\n+\n+\tif (!load_pack_revindex_from_disk(p))\n+\t\treturn 0;\n+\telse if (!create_pack_revindex_in_memory(p))\n+\t\treturn 0;\n+\treturn -1;\n+}\n+\n int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n {\n \tunsigned lo, hi;\n@@ -203,18 +317,28 @@ int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n \n uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos)\n {\n-\tif (!p->revindex)\n+\tif (!(p->revindex || p->revindex_data))\n \t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n \tif (p->num_objects <= pos)\n \t\tBUG(\"pack_pos_to_index: out-of-bounds object at %\"PRIu32, pos);\n-\treturn p->revindex[pos].nr;\n+\n+\tif (p->revindex)\n+\t\treturn p->revindex[pos].nr;\n+\telse\n+\t\treturn get_be32(p->revindex_data + pos);\n }\n \n off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n {\n-\tif (!p->revindex)\n+\tif (!(p->revindex || p->revindex_data))\n \t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n \tif (p->num_objects < pos)\n \t\tBUG(\"pack_pos_to_offset: out-of-bounds object at %\"PRIu32, pos);\n-\treturn p->revindex[pos].offset;\n+\n+\tif (p->revindex)\n+\t\treturn p->revindex[pos].offset;\n+\telse if (pos == p->num_objects)\n+\t\treturn p->pack_size - the_hash_algo->rawsz;\n+\telse\n+\t\treturn nth_packed_object_offset(p, pack_pos_to_index(p, pos));\n }\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex 6e0320b08b..61b2f3ab75 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -16,11 +16,17 @@\n * can be found\n */\n \n+#define RIDX_SIGNATURE 0x52494458 /* \"RIDX\" */\n+#define RIDX_VERSION 1\n+\n struct packed_git;\n \n /*\n * load_pack_revindex populates the revindex's internal data-structures for the\n * given pack, returning zero on success and a negative value otherwise.\n+ *\n+ * If a '.rev' file is present it is mmap'd, and pointers are assigned into it\n+ * (instead of using the in-memory variant).\n */\n int load_pack_revindex(struct packed_git *p);\n \n@@ -55,7 +61,9 @@ uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);\n * If the reverse index has not yet been loaded, or the position is out of\n * bounds, this function aborts.\n *\n- * This function runs in constant time.\n+ * This function runs in constant time under both in-memory and on-disk reverse\n+ * indexes, but an additional step is taken to consult the corresponding .idx\n+ * file when using the on-disk format.\n */\n off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos);\n \ndiff --git a/packfile.c b/packfile.c\nindex 4b938b4372..1fec12ac5f 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -324,11 +324,21 @@ void close_pack_index(struct packed_git *p)\n \t}\n }\n \n+void close_pack_revindex(struct packed_git *p) {\n+\tif (!p->revindex_map)\n+\t\treturn;\n+\n+\tmunmap((void *)p->revindex_map, p->revindex_size);\n+\tp->revindex_map = NULL;\n+\tp->revindex_data = NULL;\n+}\n+\n void close_pack(struct packed_git *p)\n {\n \tclose_pack_windows(p);\n \tclose_pack_fd(p);\n \tclose_pack_index(p);\n+\tclose_pack_revindex(p);\n }\n \n void close_object_store(struct raw_object_store *o)\n@@ -351,7 +361,7 @@ void close_object_store(struct raw_object_store *o)\n \n void unlink_pack_path(const char *pack_name, int force_delete)\n {\n-\tstatic const char *exts[] = {\".pack\", \".idx\", \".keep\", \".bitmap\", \".promisor\"};\n+\tstatic const char *exts[] = {\".pack\", \".idx\", \".rev\", \".keep\", \".bitmap\", \".promisor\"};\n \tint i;\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t plen;\n@@ -853,6 +863,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \tif (!strcmp(file_name, \"multi-pack-index\"))\n \t\treturn;\n \tif (ends_with(file_name, \".idx\") ||\n+\t ends_with(file_name, \".rev\") ||\n \t ends_with(file_name, \".pack\") ||\n \t ends_with(file_name, \".bitmap\") ||\n \t ends_with(file_name, \".keep\") ||\ndiff --git a/packfile.h b/packfile.h\nindex a58fc738e0..4cfec9e8d3 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -90,6 +90,7 @@ uint32_t get_pack_fanout(struct packed_git *p, uint32_t value);\n \n unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n void close_pack_windows(struct packed_git *);\n+void close_pack_revindex(struct packed_git *);\n void close_pack(struct packed_git *);\n void close_object_store(struct raw_object_store *o);\n void unuse_pack(struct pack_window **);\ndiff --git a/tmp-objdir.c b/tmp-objdir.c\nindex 42ed4db5d3..b8d880e362 100644\n--- a/tmp-objdir.c\n+++ b/tmp-objdir.c\n@@ -185,9 +185,11 @@ static int pack_copy_priority(const char *name)\n \t\treturn 1;\n \tif (ends_with(name, \".pack\"))\n \t\treturn 2;\n-\tif (ends_with(name, \".idx\"))\n+\tif (ends_with(name, \".rev\"))\n \t\treturn 3;\n-\treturn 4;\n+\tif (ends_with(name, \".idx\"))\n+\t\treturn 4;\n+\treturn 5;\n }\n \n static int pack_copy_cmp(const char *a, const char *b)\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"415250","messageId":"02550a251de4f477a537968fda921b52586794dc.1611617820.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1611617819.git.me@ttaylorr.com","subject":"[PATCH v3 08/10] t: support GIT_TEST_WRITE_REV_INDEX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-25T23:37:42Z","receivedAt":"2021-01-25T23:39:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Add a new option that unconditionally enables the pack.writeReverseIndex\nsetting in order to run the whole test suite in a mode that generates\non-disk reverse indexes. Additionally, enable this mode in the second\nrun of tests under linux-gcc in 'ci/run-build-and-tests.sh'.\n\nOnce on-disk reverse indexes are proven out over several releases, we\ncan change the default value of that configuration to 'true', and drop\nthis patch.\n\nSigned-off-by: Taylor Blau \n---\n builtin/index-pack.c | 5 ++++-\n builtin/pack-objects.c | 2 ++\n ci/run-build-and-tests.sh | 1 +\n pack-revindex.h | 3 +++\n t/README | 3 +++\n 5 files changed, 13 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex d5cd665b98..54f74c4874 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1748,7 +1748,10 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (prefix && chdir(prefix))\n \t\tdie(_(\"Cannot come back to cwd\"));\n \n-\trev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV));\n+\tif (git_env_bool(GIT_TEST_WRITE_REV_INDEX, 0))\n+\t\trev_index = 1;\n+\telse\n+\t\trev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV));\n \n \tfor (i = 1; i < argc; i++) {\n \t\tconst char *arg = argv[i];\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex d784569200..24df0c98f7 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3601,6 +3601,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \n \treset_pack_idx_option(&pack_idx_opts);\n \tgit_config(git_pack_config, NULL);\n+\tif (git_env_bool(GIT_TEST_WRITE_REV_INDEX, 0))\n+\t\tpack_idx_opts.flags |= WRITE_REV;\n \n \tprogress = isatty(2);\n \targc = parse_options(argc, argv, prefix, pack_objects_options,\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 50e0b90073..a66b5e8c75 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -24,6 +24,7 @@ linux-gcc)\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n \texport GIT_TEST_ADD_I_USE_BUILTIN=1\n \texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n+\texport GIT_TEST_WRITE_REV_INDEX=1\n \tmake test\n \t;;\n linux-clang)\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex 61b2f3ab75..d1a0595e89 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -16,9 +16,12 @@\n * can be found\n */\n \n+\n #define RIDX_SIGNATURE 0x52494458 /* \"RIDX\" */\n #define RIDX_VERSION 1\n \n+#define GIT_TEST_WRITE_REV_INDEX \"GIT_TEST_WRITE_REV_INDEX\"\n+\n struct packed_git;\n \n /*\ndiff --git a/t/README b/t/README\nindex c730a70770..0f97a51640 100644\n--- a/t/README\n+++ b/t/README\n@@ -439,6 +439,9 @@ GIT_TEST_DEFAULT_HASH= specifies which hash algorithm to\n use in the test scripts. Recognized values for are \"sha1\"\n and \"sha256\".\n \n+GIT_TEST_WRITE_REV_INDEX=, when true enables the\n+'pack.writeReverseIndex' setting.\n+\n Naming Tests\n ------------\n \n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"415251","messageId":"7cf16485cccccf365101d30138d9ee8b00d705d0.1611617820.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1611617819.git.me@ttaylorr.com","subject":"[PATCH v3 07/10] t: prepare for GIT_TEST_WRITE_REV_INDEX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-25T23:37:38Z","receivedAt":"2021-01-25T23:39:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In the next patch, we'll add support for unconditionally enabling the\n'pack.writeReverseIndex' setting with a new GIT_TEST_WRITE_REV_INDEX\nenvironment variable.\n\nThis causes a little bit of fallout with tests that, for example,\ncompare the list of files in the pack directory being unprepared to see\n.rev files in its output.\n\nThose locations can be cleaned up to look for specific file extensions,\nrather than take everything in the pack directory (for instance) and\nthen grep out unwanted items.\n\nOnce the pack.writeReverseIndex option has been thoroughly\ntested, we will default it to 'true', removing GIT_TEST_WRITE_REV_INDEX,\nand making it possible to revert this patch.\n\nSigned-off-by: Taylor Blau \n---\n t/t5319-multi-pack-index.sh | 5 +++--\n t/t5325-reverse-index.sh | 4 ++++\n t/t5604-clone-reference.sh | 2 +-\n t/t5702-protocol-v2.sh | 12 ++++++++----\n t/t6500-gc.sh | 6 +++---\n t/t9300-fast-import.sh | 5 ++++-\n 6 files changed, 23 insertions(+), 11 deletions(-)\n\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 297de502a9..2fc3aadbd1 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -710,8 +710,9 @@ test_expect_success 'expire respects .keep files' '\n \t\tPACKA=$(ls .git/objects/pack/a-pack*\\.pack | sed s/\\.pack\\$//) &&\n \t\ttouch $PACKA.keep &&\n \t\tgit multi-pack-index expire &&\n-\t\tls -S .git/objects/pack/a-pack* | grep $PACKA >a-pack-files &&\n-\t\ttest_line_count = 3 a-pack-files &&\n+\t\ttest_path_is_file $PACKA.idx &&\n+\t\ttest_path_is_file $PACKA.keep &&\n+\t\ttest_path_is_file $PACKA.pack &&\n \t\ttest-tool read-midx .git/objects | grep idx >midx-list &&\n \t\ttest_line_count = 2 midx-list\n \t)\ndiff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh\nindex 87040263b7..be452bb343 100755\n--- a/t/t5325-reverse-index.sh\n+++ b/t/t5325-reverse-index.sh\n@@ -3,6 +3,10 @@\n test_description='on-disk reverse index'\n . ./test-lib.sh\n \n+# The below tests want control over the 'pack.writeReverseIndex' setting\n+# themselves to assert various combinations of it with other options.\n+sane_unset GIT_TEST_WRITE_REV_INDEX\n+\n packdir=.git/objects/pack\n \n test_expect_success 'setup' '\ndiff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh\nindex 5d682706ae..e845d621f6 100755\n--- a/t/t5604-clone-reference.sh\n+++ b/t/t5604-clone-reference.sh\n@@ -329,7 +329,7 @@ test_expect_success SYMLINKS 'clone repo with symlinked or unknown files at obje\n \tfor raw in $(ls T*.raw)\n \tdo\n \t\tsed -e \"s!/../!/Y/!; s![0-9a-f]\\{38,\\}!Z!\" -e \"/commit-graph/d\" \\\n-\t\t -e \"/multi-pack-index/d\" <$raw >$raw.de-sha-1 &&\n+\t\t -e \"/multi-pack-index/d\" -e \"/rev/d\" <$raw >$raw.de-sha-1 &&\n \t\tsort $raw.de-sha-1 >$raw.de-sha || return 1\n \tdone &&\n \ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 3d994e0b1b..e8f0b4a299 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -851,8 +851,10 @@ test_expect_success 'part of packfile response provided as URI' '\n \ttest -f h2found &&\n \n \t# Ensure that there are exactly 6 files (3 .pack and 3 .idx).\n-\tls http_child/.git/objects/pack/* >filelist &&\n-\ttest_line_count = 6 filelist\n+\tls http_child/.git/objects/pack/*.pack >packlist &&\n+\tls http_child/.git/objects/pack/*.idx >idxlist &&\n+\ttest_line_count = 3 idxlist &&\n+\ttest_line_count = 3 packlist\n '\n \n test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n@@ -905,8 +907,10 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' '\n \t\tclone \"$HTTPD_URL/smart/http_parent\" http_child &&\n \n \t# Ensure that there are exactly 4 files (2 .pack and 2 .idx).\n-\tls http_child/.git/objects/pack/* >filelist &&\n-\ttest_line_count = 4 filelist\n+\tls http_child/.git/objects/pack/*.pack >packlist &&\n+\tls http_child/.git/objects/pack/*.idx >idxlist &&\n+\ttest_line_count = 2 idxlist &&\n+\ttest_line_count = 2 packlist\n '\n \n test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' '\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 4a3b8f48ac..f76586f808 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -106,17 +106,17 @@ test_expect_success 'auto gc with too many loose objects does not attempt to cre\n \ttest_commit \"$(test_oid obj2)\" &&\n \t# Our first gc will create a pack; our second will create a second pack\n \tgit gc --auto &&\n-\tls .git/objects/pack | sort >existing_packs &&\n+\tls .git/objects/pack/pack-*.pack | sort >existing_packs &&\n \ttest_commit \"$(test_oid obj3)\" &&\n \ttest_commit \"$(test_oid obj4)\" &&\n \n \tgit gc --auto 2>err &&\n \ttest_i18ngrep ! \"^warning:\" err &&\n-\tls .git/objects/pack/ | sort >post_packs &&\n+\tls .git/objects/pack/pack-*.pack | sort >post_packs &&\n \tcomm -1 -3 existing_packs post_packs >new &&\n \tcomm -2 -3 existing_packs post_packs >del &&\n \ttest_line_count = 0 del && # No packs are deleted\n-\ttest_line_count = 2 new # There is one new pack and its .idx\n+\ttest_line_count = 1 new # There is one new pack\n '\n \n test_expect_success 'gc --no-quiet' '\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 3d17e932a0..8f1caf8025 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -1632,7 +1632,10 @@ test_expect_success 'O: blank lines not necessary after other commands' '\n \tINPUT_END\n \n \tgit fast-import packlist &&\n+\tls -la .git/objects/pack/pack-*.pack >idxlist &&\n+\ttest_line_count = 4 idxlist &&\n+\ttest_line_count = 4 packlist &&\n \ttest $(git rev-parse refs/tags/O3-2nd) = $(git rev-parse O3^) &&\n \tgit log --reverse --pretty=oneline O3 | sed s/^.*z// >actual &&\n \ttest_cmp expect actual\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"415252","messageId":"5bebe05a166bbebbf29347dea377ce0c8d15c344.1611617820.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1611617819.git.me@ttaylorr.com","subject":"[PATCH v3 05/10] builtin/pack-objects.c: respect 'pack.writeReverseIndex'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-25T23:37:30Z","receivedAt":"2021-01-25T23:39:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that we have an implementation that can write the new reverse index\nformat, enable writing a .rev file in 'git pack-objects' by consulting\nthe pack.writeReverseIndex configuration variable.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 7 +++++++\n t/t5325-reverse-index.sh | 13 +++++++++++++\n 2 files changed, 20 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5b0c4489e2..d784569200 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2955,6 +2955,13 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\t pack_idx_opts.version);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"pack.writereverseindex\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\tpack_idx_opts.flags |= WRITE_REV;\n+\t\telse\n+\t\t\tpack_idx_opts.flags &= ~WRITE_REV;\n+\t\treturn 0;\n+\t}\n \tif (!strcmp(k, \"uploadpack.blobpackfileuri\")) {\n \t\tstruct configured_exclusion *ex = xmalloc(sizeof(*ex));\n \t\tconst char *oid_end, *pack_end;\ndiff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh\nindex 2dae213126..87040263b7 100755\n--- a/t/t5325-reverse-index.sh\n+++ b/t/t5325-reverse-index.sh\n@@ -68,4 +68,17 @@ test_expect_success 'index-pack infers reverse index name with -o' '\n \ttest_path_is_file other.rev\n '\n \n+test_expect_success 'pack-objects respects pack.writeReverseIndex' '\n+\ttest_when_finished \"rm -fr pack-1-*\" &&\n+\n+\tgit -c pack.writeReverseIndex= pack-objects --all pack-1 &&\n+\ttest_path_is_missing pack-1-*.rev &&\n+\n+\tgit -c pack.writeReverseIndex=false pack-objects --all pack-1 &&\n+\ttest_path_is_missing pack-1-*.rev &&\n+\n+\tgit -c pack.writeReverseIndex=true pack-objects --all pack-1 &&\n+\ttest_path_is_file pack-1-*.rev\n+'\n+\n test_done\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"415253","messageId":"a66d2f9f7c20eeb813656e66b3ad9d42f2eecf34.1611617820.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1611617819.git.me@ttaylorr.com","subject":"[PATCH v3 09/10] pack-revindex: ensure that on-disk reverse indexes are given precedence","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-25T23:37:46Z","receivedAt":"2021-01-25T23:42:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When an on-disk reverse index exists, there is no need to generate one\nin memory. In fact, doing so can be slow, and require large amounts of\nthe heap.\n\nLet's make sure that we treat the on-disk reverse index with precedence\n(i.e., that when it exists, we don't bother trying to generate an\nequivalent one in memory) by teaching Git how to conditionally die()\nwhen generating a reverse index in memory.\n\nThen, add a test to ensure that when (a) an on-disk reverse index\nexists, and (b) when setting GIT_TEST_REV_INDEX_DIE_IN_MEMORY, that we\ndo not die, implying that we read from the on-disk one.\n\nSigned-off-by: Taylor Blau \n---\n pack-revindex.c | 4 ++++\n pack-revindex.h | 1 +\n t/t5325-reverse-index.sh | 9 +++++++++\n 3 files changed, 14 insertions(+)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex a174fa5388..83fe4de773 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -2,6 +2,7 @@\n #include \"pack-revindex.h\"\n #include \"object-store.h\"\n #include \"packfile.h\"\n+#include \"config.h\"\n \n struct revindex_entry {\n \toff_t offset;\n@@ -166,6 +167,9 @@ static void create_pack_revindex(struct packed_git *p)\n \n static int create_pack_revindex_in_memory(struct packed_git *p)\n {\n+\tif (git_env_bool(GIT_TEST_REV_INDEX_DIE_IN_MEMORY, 0))\n+\t\tdie(\"dying as requested by '%s'\",\n+\t\t GIT_TEST_REV_INDEX_DIE_IN_MEMORY);\n \tif (open_pack_index(p))\n \t\treturn -1;\n \tcreate_pack_revindex(p);\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex d1a0595e89..ba7c82c125 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -21,6 +21,7 @@\n #define RIDX_VERSION 1\n \n #define GIT_TEST_WRITE_REV_INDEX \"GIT_TEST_WRITE_REV_INDEX\"\n+#define GIT_TEST_REV_INDEX_DIE_IN_MEMORY \"GIT_TEST_REV_INDEX_DIE_IN_MEMORY\"\n \n struct packed_git;\n \ndiff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh\nindex be452bb343..a344b18d7e 100755\n--- a/t/t5325-reverse-index.sh\n+++ b/t/t5325-reverse-index.sh\n@@ -85,4 +85,13 @@ test_expect_success 'pack-objects respects pack.writeReverseIndex' '\n \ttest_path_is_file pack-1-*.rev\n '\n \n+test_expect_success 'reverse index is not generated when available on disk' '\n+\ttest_index_pack true &&\n+\ttest_path_is_file $rev &&\n+\n+\tgit rev-parse HEAD >tip &&\n+\tGIT_TEST_REV_INDEX_DIE_IN_MEMORY=1 git cat-file \\\n+\t\t--batch-check=\"%(objectsize:disk)\" \nSigned-off-by: Taylor Blau \n---\n t/t5325-reverse-index.sh | 45 ++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 45 insertions(+)\n\ndiff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh\nindex a344b18d7e..b1dd726d0e 100755\n--- a/t/t5325-reverse-index.sh\n+++ b/t/t5325-reverse-index.sh\n@@ -94,4 +94,49 @@ test_expect_success 'reverse index is not generated when available on disk' '\n \t\t--batch-check=\"%(objectsize:disk)\" tree &&\n+\t\t\tgit cat-file --batch-check=\"%(objectsize:disk)\" actual &&\n+\n+\t\t\tgit cat-file -p HEAD^{tree} &&\n+\n+\t\t\tprintf \"%s\\n\" \"$(test_oid disklen)\" >expect &&\n+\t\t\ttest_cmp expect actual\n+\t\t)\n+\t'\n+\n+\ttest_expect_success \"cleanup revindex tests (on_disk=$on_disk)\" '\n+\t\trm -fr repo\n+\t'\n+}\n+\n+revindex_tests \"true\"\n+revindex_tests \"false\"\n+\n test_done\n-- \n2.30.0.138.g6d7191ea01\n"},{"id":"415255","messageId":"7e29f2d3a08c42b1e8368c4a2f52a11cc47ee959.1611617820.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1611617819.git.me@ttaylorr.com","subject":"[PATCH v3 06/10] Documentation/config/pack.txt: advertise 'pack.writeReverseIndex'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-25T23:37:34Z","receivedAt":"2021-01-25T23:43:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the pack.writeReverseIndex configuration is respected in both\n'git index-pack' and 'git pack-objects' (and therefore, all of their\ncallers), we can safely advertise it for use in the git-config manual.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/pack.txt | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\nindex 837f1b1679..3da4ea98e2 100644\n--- a/Documentation/config/pack.txt\n+++ b/Documentation/config/pack.txt\n@@ -133,3 +133,10 @@ pack.writeBitmapHashCache::\n \tbetween an older, bitmapped pack and objects that have been\n \tpushed since the last gc). The downside is that it consumes 4\n \tbytes per object of disk space. Defaults to true.\n+\n+pack.writeReverseIndex::\n+\tWhen true, git will write a corresponding .rev file (see:\n+\tlink:../technical/pack-format.html[Documentation/technical/pack-format.txt])\n+\tfor each new packfile that it writes in all places except for\n+\tlinkgit:git-fast-import[1] and in the bulk checkin mechanism.\n+\tDefaults to false.\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"415256","messageId":"5efc87074246b8d2038569b13045dbe506cc24fa.1611617820.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1611617819.git.me@ttaylorr.com","subject":"[PATCH v3 02/10] pack-write.c: prepare to write 'pack-*.rev' files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-25T23:37:18Z","receivedAt":"2021-01-25T23:45:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This patch prepares for callers to be able to write reverse index files\nto disk.\n\nIt adds the necessary machinery to write a format-compliant .rev file\nfrom within 'write_rev_file()', which is called from\n'finish_tmp_packfile()'.\n\nSimilar to the process by which the reverse index is computed in memory,\nthese new paths also have to sort a list of objects by their offsets\nwithin a packfile. These new paths use a qsort() (as opposed to a radix\nsort), since our specialized radix sort requires a full revindex_entry\nstruct per object, which is more memory than we need to allocate.\n\nThe qsort is obviously slower, but the theoretical slowdown would\nrequire a repository with a large amount of objects, likely implying\nthat the time spent in, say, pack-objects during a repack would dominate\nthe overall runtime.\n\nSigned-off-by: Taylor Blau \n---\n pack-write.c | 120 ++++++++++++++++++++++++++++++++++++++++++++++++++-\n pack.h | 4 ++\n 2 files changed, 123 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-write.c b/pack-write.c\nindex e9bb3fd949..680c36755d 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -167,6 +167,113 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec\n \treturn index_name;\n }\n \n+static int pack_order_cmp(const void *va, const void *vb, void *ctx)\n+{\n+\tstruct pack_idx_entry **objects = ctx;\n+\n+\toff_t oa = objects[*(uint32_t*)va]->offset;\n+\toff_t ob = objects[*(uint32_t*)vb]->offset;\n+\n+\tif (oa < ob)\n+\t\treturn -1;\n+\tif (oa > ob)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static void write_rev_header(struct hashfile *f)\n+{\n+\tuint32_t oid_version;\n+\tswitch (hash_algo_by_ptr(the_hash_algo)) {\n+\tcase GIT_HASH_SHA1:\n+\t\toid_version = 1;\n+\t\tbreak;\n+\tcase GIT_HASH_SHA256:\n+\t\toid_version = 2;\n+\t\tbreak;\n+\tdefault:\n+\t\tdie(\"write_rev_header: unknown hash version\");\n+\t}\n+\n+\thashwrite_be32(f, RIDX_SIGNATURE);\n+\thashwrite_be32(f, RIDX_VERSION);\n+\thashwrite_be32(f, oid_version);\n+}\n+\n+static void write_rev_index_positions(struct hashfile *f,\n+\t\t\t\t struct pack_idx_entry **objects,\n+\t\t\t\t uint32_t nr_objects)\n+{\n+\tuint32_t *pack_order;\n+\tuint32_t i;\n+\n+\tALLOC_ARRAY(pack_order, nr_objects);\n+\tfor (i = 0; i < nr_objects; i++)\n+\t\tpack_order[i] = i;\n+\tQSORT_S(pack_order, nr_objects, pack_order_cmp, objects);\n+\n+\tfor (i = 0; i < nr_objects; i++)\n+\t\thashwrite_be32(f, pack_order[i]);\n+\n+\tfree(pack_order);\n+}\n+\n+static void write_rev_trailer(struct hashfile *f, const unsigned char *hash)\n+{\n+\thashwrite(f, hash, the_hash_algo->rawsz);\n+}\n+\n+const char *write_rev_file(const char *rev_name,\n+\t\t\t struct pack_idx_entry **objects,\n+\t\t\t uint32_t nr_objects,\n+\t\t\t const unsigned char *hash,\n+\t\t\t unsigned flags)\n+{\n+\tstruct hashfile *f;\n+\tint fd;\n+\n+\tif ((flags & WRITE_REV) && (flags & WRITE_REV_VERIFY))\n+\t\tdie(_(\"cannot both write and verify reverse index\"));\n+\n+\tif (flags & WRITE_REV) {\n+\t\tif (!rev_name) {\n+\t\t\tstruct strbuf tmp_file = STRBUF_INIT;\n+\t\t\tfd = odb_mkstemp(&tmp_file, \"pack/tmp_rev_XXXXXX\");\n+\t\t\trev_name = strbuf_detach(&tmp_file, NULL);\n+\t\t} else {\n+\t\t\tunlink(rev_name);\n+\t\t\tfd = open(rev_name, O_CREAT|O_EXCL|O_WRONLY, 0600);\n+\t\t\tif (fd < 0)\n+\t\t\t\tdie_errno(\"unable to create '%s'\", rev_name);\n+\t\t}\n+\t\tf = hashfd(fd, rev_name);\n+\t} else if (flags & WRITE_REV_VERIFY) {\n+\t\tstruct stat statbuf;\n+\t\tif (stat(rev_name, &statbuf)) {\n+\t\t\tif (errno == ENOENT) {\n+\t\t\t\t/* .rev files are optional */\n+\t\t\t\treturn NULL;\n+\t\t\t} else\n+\t\t\t\tdie_errno(_(\"could not stat: %s\"), rev_name);\n+\t\t}\n+\t\tf = hashfd_check(rev_name);\n+\t} else\n+\t\treturn NULL;\n+\n+\twrite_rev_header(f);\n+\n+\twrite_rev_index_positions(f, objects, nr_objects);\n+\twrite_rev_trailer(f, hash);\n+\n+\tif (rev_name && adjust_shared_perm(rev_name) < 0)\n+\t\tdie(_(\"failed to make %s readable\"), rev_name);\n+\n+\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |\n+\t\t\t\t ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));\n+\n+\treturn rev_name;\n+}\n+\n off_t write_pack_header(struct hashfile *f, uint32_t nr_entries)\n {\n \tstruct pack_header hdr;\n@@ -342,7 +449,7 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \t\t\t struct pack_idx_option *pack_idx_opts,\n \t\t\t unsigned char hash[])\n {\n-\tconst char *idx_tmp_name;\n+\tconst char *idx_tmp_name, *rev_tmp_name = NULL;\n \tint basename_len = name_buffer->len;\n \n \tif (adjust_shared_perm(pack_tmp_name))\n@@ -353,6 +460,9 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \tif (adjust_shared_perm(idx_tmp_name))\n \t\tdie_errno(\"unable to make temporary index file readable\");\n \n+\trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n+\t\t\t\t pack_idx_opts->flags);\n+\n \tstrbuf_addf(name_buffer, \"%s.pack\", hash_to_hex(hash));\n \n \tif (rename(pack_tmp_name, name_buffer->buf))\n@@ -366,6 +476,14 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \n \tstrbuf_setlen(name_buffer, basename_len);\n \n+\tif (rev_tmp_name) {\n+\t\tstrbuf_addf(name_buffer, \"%s.rev\", hash_to_hex(hash));\n+\t\tif (rename(rev_tmp_name, name_buffer->buf))\n+\t\t\tdie_errno(\"unable to rename temporary reverse-index file\");\n+\t}\n+\n+\tstrbuf_setlen(name_buffer, basename_len);\n+\n \tfree((void *)idx_tmp_name);\n }\n \ndiff --git a/pack.h b/pack.h\nindex 9ae640f417..afdcf8f5c7 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -42,6 +42,8 @@ struct pack_idx_option {\n \t/* flag bits */\n #define WRITE_IDX_VERIFY 01 /* verify only, do not write the idx file */\n #define WRITE_IDX_STRICT 02\n+#define WRITE_REV 04\n+#define WRITE_REV_VERIFY 010\n \n \tuint32_t version;\n \tuint32_t off32_limit;\n@@ -91,6 +93,8 @@ struct ref;\n \n void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought);\n \n+const char *write_rev_file(const char *rev_name, struct pack_idx_entry **objects, uint32_t nr_objects, const unsigned char *hash, unsigned flags);\n+\n /*\n * The \"hdr\" output buffer should be at least this big, which will handle sizes\n * up to 2^67.\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"415257","messageId":"a8ee59fccf678bd1bf57930cd6395f36273f74d6.1611617820.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1611617819.git.me@ttaylorr.com","subject":"[PATCH v3 04/10] builtin/index-pack.c: write reverse indexes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-25T23:37:26Z","receivedAt":"2021-01-25T23:45:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Teach 'git index-pack' to optionally write and verify reverse index with\n'--[no-]rev-index', as well as respecting the 'pack.writeReverseIndex'\nconfiguration option.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-index-pack.txt | 18 +++++---\n builtin/index-pack.c | 50 ++++++++++++++++++++--\n t/t5325-reverse-index.sh | 71 ++++++++++++++++++++++++++++++++\n 3 files changed, 131 insertions(+), 8 deletions(-)\n create mode 100755 t/t5325-reverse-index.sh\n\ndiff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt\nindex af0c26232c..69ba904d44 100644\n--- a/Documentation/git-index-pack.txt\n+++ b/Documentation/git-index-pack.txt\n@@ -9,17 +9,18 @@ git-index-pack - Build pack index file for an existing packed archive\n SYNOPSIS\n --------\n [verse]\n-'git index-pack' [-v] [-o ] \n+'git index-pack' [-v] [-o ] [--[no-]rev-index] \n 'git index-pack' --stdin [--fix-thin] [--keep] [-v] [-o ]\n- []\n+\t\t [--[no-]rev-index] []\n \n \n DESCRIPTION\n -----------\n Reads a packed archive (.pack) from the specified file, and\n-builds a pack index file (.idx) for it. The packed archive\n-together with the pack index can then be placed in the\n-objects/pack/ directory of a Git repository.\n+builds a pack index file (.idx) for it. Optionally writes a\n+reverse-index (.rev) for the specified pack. The packed\n+archive together with the pack index can then be placed in\n+the objects/pack/ directory of a Git repository.\n \n \n OPTIONS\n@@ -35,6 +36,13 @@ OPTIONS\n \tfails if the name of packed archive does not end\n \twith .pack).\n \n+--[no-]rev-index::\n+\tWhen this flag is provided, generate a reverse index\n+\t(a `.rev` file) corresponding to the given pack. If\n+\t`--verify` is given, ensure that the existing\n+\treverse index is correct. Takes precedence over\n+\t`pack.writeReverseIndex`.\n+\n --stdin::\n \tWhen this flag is provided, the pack is read from stdin\n \tinstead and a copy is then written to . If\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex c758f3b8e9..d5cd665b98 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -17,7 +17,7 @@\n #include \"promisor-remote.h\"\n \n static const char index_pack_usage[] =\n-\"git index-pack [-v] [-o ] [--keep | --keep=] [--verify] [--strict] ( | --stdin [--fix-thin] [])\";\n+\"git index-pack [-v] [-o ] [--keep | --keep=] [--[no-]rev-index] [--verify] [--strict] ( | --stdin [--fix-thin] [])\";\n \n struct object_entry {\n \tstruct pack_idx_entry idx;\n@@ -1484,12 +1484,14 @@ static void write_special_file(const char *suffix, const char *msg,\n \n static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t const char *final_index_name, const char *curr_index_name,\n+\t\t const char *final_rev_index_name, const char *curr_rev_index_name,\n \t\t const char *keep_msg, const char *promisor_msg,\n \t\t unsigned char *hash)\n {\n \tconst char *report = \"pack\";\n \tstruct strbuf pack_name = STRBUF_INIT;\n \tstruct strbuf index_name = STRBUF_INIT;\n+\tstruct strbuf rev_index_name = STRBUF_INIT;\n \tint err;\n \n \tif (!from_stdin) {\n@@ -1524,6 +1526,16 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t} else\n \t\tchmod(final_index_name, 0444);\n \n+\tif (curr_rev_index_name) {\n+\t\tif (final_rev_index_name != curr_rev_index_name) {\n+\t\t\tif (!final_rev_index_name)\n+\t\t\t\tfinal_rev_index_name = odb_pack_name(&rev_index_name, hash, \"rev\");\n+\t\t\tif (finalize_object_file(curr_rev_index_name, final_rev_index_name))\n+\t\t\t\tdie(_(\"cannot store reverse index file\"));\n+\t\t} else\n+\t\t\tchmod(final_rev_index_name, 0444);\n+\t}\n+\n \tif (do_fsck_object) {\n \t\tstruct packed_git *p;\n \t\tp = add_packed_git(final_index_name, strlen(final_index_name), 0);\n@@ -1553,6 +1565,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t}\n \t}\n \n+\tstrbuf_release(&rev_index_name);\n \tstrbuf_release(&index_name);\n \tstrbuf_release(&pack_name);\n }\n@@ -1578,6 +1591,12 @@ static int git_index_pack_config(const char *k, const char *v, void *cb)\n \t\t}\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"pack.writereverseindex\")) {\n+\t\tif (git_config_bool(k, v))\n+\t\t\topts->flags |= WRITE_REV;\n+\t\telse\n+\t\t\topts->flags &= ~WRITE_REV;\n+\t}\n \treturn git_default_config(k, v, cb);\n }\n \n@@ -1695,12 +1714,14 @@ static void show_pack_info(int stat_only)\n \n int cmd_index_pack(int argc, const char **argv, const char *prefix)\n {\n-\tint i, fix_thin_pack = 0, verify = 0, stat_only = 0;\n+\tint i, fix_thin_pack = 0, verify = 0, stat_only = 0, rev_index;\n \tconst char *curr_index;\n-\tconst char *index_name = NULL, *pack_name = NULL;\n+\tconst char *curr_rev_index = NULL;\n+\tconst char *index_name = NULL, *pack_name = NULL, *rev_index_name = NULL;\n \tconst char *keep_msg = NULL;\n \tconst char *promisor_msg = NULL;\n \tstruct strbuf index_name_buf = STRBUF_INIT;\n+\tstruct strbuf rev_index_name_buf = STRBUF_INIT;\n \tstruct pack_idx_entry **idx_objects;\n \tstruct pack_idx_option opts;\n \tunsigned char pack_hash[GIT_MAX_RAWSZ];\n@@ -1727,6 +1748,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (prefix && chdir(prefix))\n \t\tdie(_(\"Cannot come back to cwd\"));\n \n+\trev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV));\n+\n \tfor (i = 1; i < argc; i++) {\n \t\tconst char *arg = argv[i];\n \n@@ -1805,6 +1828,10 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\tif (hash_algo == GIT_HASH_UNKNOWN)\n \t\t\t\t\tdie(_(\"unknown hash algorithm '%s'\"), arg);\n \t\t\t\trepo_set_hash_algo(the_repository, hash_algo);\n+\t\t\t} else if (!strcmp(arg, \"--rev-index\")) {\n+\t\t\t\trev_index = 1;\n+\t\t\t} else if (!strcmp(arg, \"--no-rev-index\")) {\n+\t\t\t\trev_index = 0;\n \t\t\t} else\n \t\t\t\tusage(index_pack_usage);\n \t\t\tcontinue;\n@@ -1826,6 +1853,15 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (!index_name && pack_name)\n \t\tindex_name = derive_filename(pack_name, \"pack\", \"idx\", &index_name_buf);\n \n+\topts.flags &= ~(WRITE_REV | WRITE_REV_VERIFY);\n+\tif (rev_index) {\n+\t\topts.flags |= verify ? WRITE_REV_VERIFY : WRITE_REV;\n+\t\tif (index_name)\n+\t\t\trev_index_name = derive_filename(index_name,\n+\t\t\t\t\t\t\t \"idx\", \"rev\",\n+\t\t\t\t\t\t\t &rev_index_name_buf);\n+\t}\n+\n \tif (verify) {\n \t\tif (!index_name)\n \t\t\tdie(_(\"--verify with no packfile name given\"));\n@@ -1878,11 +1914,16 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tfor (i = 0; i < nr_objects; i++)\n \t\tidx_objects[i] = &objects[i].idx;\n \tcurr_index = write_idx_file(index_name, idx_objects, nr_objects, &opts, pack_hash);\n+\tif (rev_index)\n+\t\tcurr_rev_index = write_rev_file(rev_index_name, idx_objects,\n+\t\t\t\t\t\tnr_objects, pack_hash,\n+\t\t\t\t\t\topts.flags);\n \tfree(idx_objects);\n \n \tif (!verify)\n \t\tfinal(pack_name, curr_pack,\n \t\t index_name, curr_index,\n+\t\t rev_index_name, curr_rev_index,\n \t\t keep_msg, promisor_msg,\n \t\t pack_hash);\n \telse\n@@ -1893,10 +1934,13 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \n \tfree(objects);\n \tstrbuf_release(&index_name_buf);\n+\tstrbuf_release(&rev_index_name_buf);\n \tif (pack_name == NULL)\n \t\tfree((void *) curr_pack);\n \tif (index_name == NULL)\n \t\tfree((void *) curr_index);\n+\tif (rev_index_name == NULL)\n+\t\tfree((void *) curr_rev_index);\n \n \t/*\n \t * Let the caller know this pack is not self contained\ndiff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh\nnew file mode 100755\nindex 0000000000..2dae213126\n--- /dev/null\n+++ b/t/t5325-reverse-index.sh\n@@ -0,0 +1,71 @@\n+#!/bin/sh\n+\n+test_description='on-disk reverse index'\n+. ./test-lib.sh\n+\n+packdir=.git/objects/pack\n+\n+test_expect_success 'setup' '\n+\ttest_commit base &&\n+\n+\tpack=$(git pack-objects --all $packdir/pack) &&\n+\trev=$packdir/pack-$pack.rev &&\n+\n+\ttest_path_is_missing $rev\n+'\n+\n+test_index_pack () {\n+\trm -f $rev &&\n+\tconf=$1 &&\n+\tshift &&\n+\t# remove the index since Windows won't overwrite an existing file\n+\trm $packdir/pack-$pack.idx &&\n+\tgit -c pack.writeReverseIndex=$conf index-pack \"$@\" \\\n+\t\t$packdir/pack-$pack.pack\n+}\n+\n+test_expect_success 'index-pack with pack.writeReverseIndex' '\n+\ttest_index_pack \"\" &&\n+\ttest_path_is_missing $rev &&\n+\n+\ttest_index_pack false &&\n+\ttest_path_is_missing $rev &&\n+\n+\ttest_index_pack true &&\n+\ttest_path_is_file $rev\n+'\n+\n+test_expect_success 'index-pack with --[no-]rev-index' '\n+\tfor conf in \"\" true false\n+\tdo\n+\t\ttest_index_pack \"$conf\" --rev-index &&\n+\t\ttest_path_exists $rev &&\n+\n+\t\ttest_index_pack \"$conf\" --no-rev-index &&\n+\t\ttest_path_is_missing $rev\n+\tdone\n+'\n+\n+test_expect_success 'index-pack can verify reverse indexes' '\n+\ttest_when_finished \"rm -f $rev\" &&\n+\ttest_index_pack true &&\n+\n+\ttest_path_is_file $rev &&\n+\tgit index-pack --rev-index --verify $packdir/pack-$pack.pack &&\n+\n+\t# Intentionally corrupt the reverse index.\n+\tchmod u+w $rev &&\n+\tprintf \"xxxx\" | dd of=$rev bs=1 count=4 conv=notrunc &&\n+\n+\ttest_must_fail git index-pack --rev-index --verify \\\n+\t\t$packdir/pack-$pack.pack 2>err &&\n+\tgrep \"validation error\" err\n+'\n+\n+test_expect_success 'index-pack infers reverse index name with -o' '\n+\tgit index-pack --rev-index -o other.idx $packdir/pack-$pack.pack &&\n+\ttest_path_is_file other.idx &&\n+\ttest_path_is_file other.rev\n+'\n+\n+test_done\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"415258","messageId":"8a3e70454b9bc64fc7a5ff07d47f7fde018e6a3d.1611617820.git.me@ttaylorr.com","threadId":"54962","inReplyTo":"cover.1611617819.git.me@ttaylorr.com","subject":"[PATCH v3 03/10] builtin/index-pack.c: allow stripping arbitrary extensions","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-25T23:37:22Z","receivedAt":"2021-01-25T23:45:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To derive the filename for a .idx file, 'git index-pack' uses\nderive_filename() to strip the '.pack' suffix and add the new suffix.\n\nPrepare for stripping off suffixes other than '.pack' by making the\nsuffix to strip a parameter of derive_filename(). In order to make this\nconsistent with the \"suffix\" parameter which does not begin with a \".\",\nan additional check in derive_filename.\n\nSuggested-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n builtin/index-pack.c | 16 ++++++++--------\n 1 file changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 557bd2f348..c758f3b8e9 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1436,15 +1436,15 @@ static void fix_unresolved_deltas(struct hashfile *f)\n \tfree(sorted_by_pos);\n }\n \n-static const char *derive_filename(const char *pack_name, const char *suffix,\n-\t\t\t\t struct strbuf *buf)\n+static const char *derive_filename(const char *pack_name, const char *strip,\n+\t\t\t\t const char *suffix, struct strbuf *buf)\n {\n \tsize_t len;\n-\tif (!strip_suffix(pack_name, \".pack\", &len))\n-\t\tdie(_(\"packfile name '%s' does not end with '.pack'\"),\n-\t\t pack_name);\n+\tif (!strip_suffix(pack_name, strip, &len) || !len ||\n+\t pack_name[len - 1] != '.')\n+\t\tdie(_(\"packfile name '%s' does not end with '.%s'\"),\n+\t\t pack_name, strip);\n \tstrbuf_add(buf, pack_name, len);\n-\tstrbuf_addch(buf, '.');\n \tstrbuf_addstr(buf, suffix);\n \treturn buf->buf;\n }\n@@ -1459,7 +1459,7 @@ static void write_special_file(const char *suffix, const char *msg,\n \tint msg_len = strlen(msg);\n \n \tif (pack_name)\n-\t\tfilename = derive_filename(pack_name, suffix, &name_buf);\n+\t\tfilename = derive_filename(pack_name, \"pack\", suffix, &name_buf);\n \telse\n \t\tfilename = odb_pack_name(&name_buf, hash, suffix);\n \n@@ -1824,7 +1824,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (from_stdin && hash_algo)\n \t\tdie(_(\"--object-format cannot be used with --stdin\"));\n \tif (!index_name && pack_name)\n-\t\tindex_name = derive_filename(pack_name, \"idx\", &index_name_buf);\n+\t\tindex_name = derive_filename(pack_name, \"pack\", \"idx\", &index_name_buf);\n \n \tif (verify) {\n \t\tif (!index_name)\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"415274","messageId":"YA8aoQK2ABiA64ME@nand.local","threadId":"54962","inReplyTo":"xmqqy2ggyh1a.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 1/8] packfile: prepare for the existence of '*.rev' files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-25T19:23:13Z","receivedAt":"2021-01-26T05:04:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jan 25, 2021 at 11:04:33AM -0800, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> >> Thinking out loud a bit: a .rev file means we're spending an extra map\n> >> per pack (but not a descriptor, since we close after mmap). And like the\n> >> .idx files (but unlike .pack file maps), we don't keep track of these\n> >> and try to close them when under memory pressure. I think that's\n> >> probably OK in terms of bytes. It may mean running up against operating\n> >> system number-of-mmap limits more quickly ...\n> >> ...\n> >> > \tif (ends_with(name, \".idx\"))\n> >> > \t\treturn 3;\n> >> > -\treturn 4;\n> >> > +\tif (ends_with(name, \".rev\"))\n> >> > +\t\treturn 4;\n> >> > +\treturn 5;\n> >> > }\n> >>\n> >> Probably not super important, but: should the .idx file still come last\n> >> here? Simultaneous readers won't start using the pack until the .idx\n> >> file is present. We'd probably prefer they see the whole thing\n> >> atomically, than see a .idx missing its .rev (they won't ever produce a\n> >> wrong answer, but they'll generate the in-core revindex on the fly when\n> >> they don't need to).\n>\n> At some point, we may want to\n>\n> - introduce .idx version 3 that is more extensible, so that the\n> reverse info is included in one of its chunks;\n\nI'm not opposed to doing so outside of this series. I'd be fine with\nresurrecting the series that Stolee posted a while ago to extract a\n\"chunk writer\" API (for using in the commit-graph and MIDX code) to also\nbe used here.\n\nThat got stalled out because of the conflicts that it would have\nproduced with Abhishek's work, but now that that's getting picked up it\ncould move forward.\n\nBut brian is also working on an index v3 for some hash transition stuff,\nso I'd rather let that settle first.\n\nOf course, you could combine those efforts, but I don't think that what\nwe have here is such a bad interim state.\n\n> - make the .rev data for all packs stored as a chunk in .midx, so\n> we can first check with .midx and not open any .rev files.\n\nThis is trickier than you might think because of how the MIDX selects\na pack when more than one pack contains a given object. There is also\nthe concept of a \"multi-pack reverse index\" which you can think of as\nthe reverse index for a pack that is more-or-less concatenating all of\nthe objects in the MIDX together (in pack order). That is the same order\nwe'll use to lay out the bits of a multi-pack bitmap, eventually.\n\n> either of which would reduce the numberfrom 30k down to 10k ;-)\n\nYour \";-)\" is a good reminder that there are probably many other\nproblems with having 30k (or even 10k!) packs that would make solving\nthis (index v3 + MIDX chunk) not worthwhile.\n\nThanks,\nTaylor\n"},{"id":"415313","messageId":"xmqq7do04e6c.fsf@gitster.c.googlers.com","threadId":"54962","inReplyTo":"cover.1611617819.git.me@ttaylorr.com","subject":"Re: [PATCH v3 00/10] pack-revindex: introduce on-disk '.rev' format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-26T02:36:43Z","receivedAt":"2021-01-26T20:44:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Here is a third reroll of my series to introduce an on-disk format for the\n> reverse index. Since the first series (to introduce a new API) has been merged\n> to 'next', this series has been rebased onto 'next', too.\n\nEhh, does that mean you are OK to see the remainder of 'next' to\ntake this topic hostage?\n\nUnless you use some new features that came from other topics in\n'next', I'd discourage such a rebasing. If the API topic gained\nsome fix-up patches on top since it was merged to 'next', it is\nperfectly sensible to rebase this series on top of the updated API\ntopic---it does not change the fact that this topic is dependent on\nthe API topic.\n\nAs it happens that the API topic is now in 'master', none of the\nabove complaint should actually apply, even if this new round of\npatches do not cleanly apply to the tip of the API topic, as long as\nthey apply cleanly to tonight's 'master'. It will make the topic\nineligible to be merged later to 'maint', but this is a new feature,\nso nothing is lost.\n\nSo, I'll try to apply them first on top of the tip of the API topic,\nwhich is at 779412b9 (for_each_object_in_pack(): clarify pack vs\nindex ordering, 2021-01-14), and if I do not feel like spending time\nto resolve conflicts, I'll then try to apply them on top of tonight's\nmaster. We'll see what happens.\n\nThanks.\n"},{"id":"415314","messageId":"YA+DS87avb2Klc4x@nand.local","threadId":"54962","inReplyTo":"xmqq7do04e6c.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v3 00/10] pack-revindex: introduce on-disk '.rev' format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-26T02:49:47Z","receivedAt":"2021-01-26T20:46:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jan 25, 2021 at 06:36:43PM -0800, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > Here is a third reroll of my series to introduce an on-disk format for the\n> > reverse index. Since the first series (to introduce a new API) has been merged\n> > to 'next', this series has been rebased onto 'next', too.\n>\n> Ehh, does that mean you are OK to see the remainder of 'next' to\n> take this topic hostage?\n>\n> Unless you use some new features that came from other topics in\n> 'next', I'd discourage such a rebasing. If the API topic gained\n> some fix-up patches on top since it was merged to 'next', it is\n> perfectly sensible to rebase this series on top of the updated API\n> topic---it does not change the fact that this topic is dependent on\n> the API topic.\n\nAh; my apologies. I thought that rebasing onto next would make it easier\nfor you to merge this topic, but after reading what you wrote I can see\nthat's not the case.\n\nThe only dependency that this topic should have is on the API one, which\nI am glad to see in 'master'.\n\n> As it happens that the API topic is now in 'master', none of the\n> above complaint should actually apply, even if this new round of\n> patches do not cleanly apply to the tip of the API topic, as long as\n> they apply cleanly to tonight's 'master'. It will make the topic\n> ineligible to be merged later to 'maint', but this is a new feature,\n> so nothing is lost.\n>\n> So, I'll try to apply them first on top of the tip of the API topic,\n> which is at 779412b9 (for_each_object_in_pack(): clarify pack vs\n> index ordering, 2021-01-14), and if I do not feel like spending time\n> to resolve conflicts, I'll then try to apply them on top of tonight's\n> master. We'll see what happens.\n\nI'm happy to send another version based on 'master', but I expect that\nthis should apply cleanly either way.\n\n> Thanks.\n\nThanks,\nTaylor\n"},{"id":"415320","messageId":"YA8YxtUUwE0lLxQt@nand.local","threadId":"54962","inReplyTo":"YAtey9krU32mgEBV@coredump.intra.peff.net","subject":"Re: [PATCH v2 2/8] pack-write.c: prepare to write 'pack-*.rev' files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-25T19:15:18Z","receivedAt":"2021-01-26T20:49:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jan 22, 2021 at 06:24:59PM -0500, Jeff King wrote:\n> On Wed, Jan 13, 2021 at 05:28:11PM -0500, Taylor Blau wrote:\n>\n> [...]\n>\n> qsort? Don't we have a perfectly good radix sort for this exact purpose? :)\n\nYeah, I think your argument against it (which I agree with is\nbasically):\n\n We could, and it would be faster, but it would (1) require allocating\n twice as much data, and (2) the speed increase is a drop in the bucket\n on any repository with enough objects to even notice it.\n\nI'll add something to that effect to the commit message.\n\n> Dereferencing a pointer to index another array always makes me nervous\n> that we may have a bounds problem with bogus data.\n>\n> In this case we know it is OK because we filled the array ourselves with\n> in-bound numbers in write_rev_index_positions.\n\nRight.\n\n> I was surprised we didn't define these already on the reading side, but\n> it looks like we didn't. In patch 1, we probably should be checking\n> RIDX_SIGNATURE in load_revindex_from_disk(). But much more importantly,\n> we should be checking that we find version 1, since that's what will\n> make it safe to later invent a version 2.\n\n:-). Thanks for noticing, fixed.\n\n> I forgot to comment on this in patch 1, but: I think the format is\n> really independent of the hash size. The contents are identical for a\n> sha-1 versus sha-256 file.\n\nRight; the format is identical. That is, the fields are the same, but\nthe hashes are longer.\n\n> That said, I don't overly mind having a hash identifier if it might help\n> debug things (OTOH, how the heck do you end up with one that matches the\n> trailer's packfile but _doesn't_ match the trailer's contents?).\n>\n> If we do have it, should we also be checking it in the loading function?\n\nI'm not opposed to it, but I imagined that this field would be primarily\nfor debugging purposes. I was surprised to learn that we _don't_ verify\nthe checksum for .idx files. So, I'm reluctant to start doing so here,\nhonestly.\n\n> So if the caller gave us a name, we force-overwrite it. That seemed\n> weird to me at first, but it makes sense; the atomic rename-into-place\n> writers will not be passing in the name. And this is exactly how\n> write_idx_file() works.\n\nYep.\n\n> I wonder if we could factor out some of this repeated logic, but I\n> suspect it is mostly diminishing returns. Maybe this \"open a pack file\n> for writing\" could become a helper function, though.\n\nYeah. I tried factoring it out before replying, and it's a little gross.\nMost of my discomfort with it lies in the complexity of the parameters.\n\nConsider extracting the code in write_idx_file():\n\n - There would need to be a double pointer for 'index_name' (which is\n sometimes read, and sometimes written to).\n\n - There would be an unsigned bit for \"verify\" (i.e., \"open with\n hashfd_check() or not\").\n\n - There would be a \"pattern\" variable if we were creating a new\n temporary file with odb_mkstemp().\n\nHaving the caller be forced to juggle the combinations of passing\nNULL/0 or not for each of those three makes me leery that this is worth\ndoing, so I tend to agree with your judgement that this provides a\ndiminishing return.\n\nThanks,\nTaylor\n"},{"id":"415359","messageId":"YBCM/fdZZwoPVsrI@coredump.intra.peff.net","threadId":"54962","inReplyTo":"YA8YxtUUwE0lLxQt@nand.local","subject":"Re: [PATCH v2 2/8] pack-write.c: prepare to write 'pack-*.rev' files","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-26T21:43:25Z","receivedAt":"2021-01-26T22:20:01Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 25, 2021 at 02:15:18PM -0500, Taylor Blau wrote:\n\n> On Fri, Jan 22, 2021 at 06:24:59PM -0500, Jeff King wrote:\n> > On Wed, Jan 13, 2021 at 05:28:11PM -0500, Taylor Blau wrote:\n> >\n> > [...]\n> >\n> > qsort? Don't we have a perfectly good radix sort for this exact purpose? :)\n> \n> Yeah, I think your argument against it (which I agree with is\n> basically):\n> \n> We could, and it would be faster, but it would (1) require allocating\n> twice as much data, and (2) the speed increase is a drop in the bucket\n> on any repository with enough objects to even notice it.\n> \n> I'll add something to that effect to the commit message.\n\nYep, that's accurate.\n\nIt might make more of a difference when we eventually have a revindex\ncache for a midx (because there we're mostly doing things linear in the\nnumber of objects, and not actually traversing). But I'm quite happy to\npunt it until then (or later; even there I'd guess it might shave one\nsecond off a 10-15 second operation. That's not nothing, but if nobody\never cares enough to implement it, I won't be sad).\n\n> > I forgot to comment on this in patch 1, but: I think the format is\n> > really independent of the hash size. The contents are identical for a\n> > sha-1 versus sha-256 file.\n> \n> Right; the format is identical. That is, the fields are the same, but\n> the hashes are longer.\n\nHmm. Actually, I wasn't thinking it through completely. I thought the\nfile would literally have the exact same bytes in it, but the packfile\nhash as well as the hashfd() trailing hash would both be different\nsizes. So even if it doesn't match the packfile (which obviously is a\nbug), it would hard to even be able to figure that out without knowing\nthe hash identifier.\n\n> > That said, I don't overly mind having a hash identifier if it might help\n> > debug things (OTOH, how the heck do you end up with one that matches the\n> > trailer's packfile but _doesn't_ match the trailer's contents?).\n> >\n> > If we do have it, should we also be checking it in the loading function?\n> \n> I'm not opposed to it, but I imagined that this field would be primarily\n> for debugging purposes. I was surprised to learn that we _don't_ verify\n> the checksum for .idx files. So, I'm reluctant to start doing so here,\n> honestly.\n\nI think we're talking about two different things. I just meant that if\nwe see a .rev file that says \"I'm sha256\", we should make sure we are in\na sha256 repo (since that is what we'd be assuming for the .idx and\n.pack files!).\n\nWe can't do that same check for .idx files, because they don't have a\nhash identifier field in them.\n\nChecking that the pack hash matches the trailer from the packfile is a\nseparate matter. I also think that's worth doing, but am likewise\nsurprised that we don't check it for the .idx. I'm OK leaving it out\nsince we don't do it there, either.\n\n> > I wonder if we could factor out some of this repeated logic, but I\n> > suspect it is mostly diminishing returns. Maybe this \"open a pack file\n> > for writing\" could become a helper function, though.\n> \n> Yeah. I tried factoring it out before replying, and it's a little gross.\n> Most of my discomfort with it lies in the complexity of the parameters.\n> \n> Consider extracting the code in write_idx_file():\n> \n> - There would need to be a double pointer for 'index_name' (which is\n> sometimes read, and sometimes written to).\n> \n> - There would be an unsigned bit for \"verify\" (i.e., \"open with\n> hashfd_check() or not\").\n> \n> - There would be a \"pattern\" variable if we were creating a new\n> temporary file with odb_mkstemp().\n> \n> Having the caller be forced to juggle the combinations of passing\n> NULL/0 or not for each of those three makes me leery that this is worth\n> doing, so I tend to agree with your judgement that this provides a\n> diminishing return.\n\nThanks for taking a look. I'm not too surprised it turned out ugly.\n\n-Peff\n"},{"id":"415535","messageId":"YBNWXU8xzw0087DC@coredump.intra.peff.net","threadId":"54962","inReplyTo":"6f8b70ab276c0579c957c315743fdab63462a605.1611617820.git.me@ttaylorr.com","subject":"Re: [PATCH v3 01/10] packfile: prepare for the existence of '*.rev' files","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-29T00:27:09Z","receivedAt":"2021-01-29T00:28:08Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 25, 2021 at 06:37:14PM -0500, Taylor Blau wrote:\n\n> +struct revindex_header {\n> +\tuint32_t signature;\n> +\tuint32_t version;\n> +\tuint32_t hash_id;\n> +};\n> [...]\n> +\tstruct revindex_header *hdr;\n> [...]\n> +\tdata = xmmap(NULL, revindex_size, PROT_READ, MAP_PRIVATE, fd, 0);\n> +\thdr = data;\n> +\n> +\tif (ntohl(hdr->signature) != RIDX_SIGNATURE) {\n\nI guess this technique was pulled from the .idx code paths, but IMHO we\nshould avoid this kind of struct-casting. It relies on the compiler's\nstruct packing, and I'm not 100% sure it doesn't violate some weird\npointer-aliasing rules.\n\nOTOH, as long as we do not ever care about sizeof(revindex_header), this\nis likely to work in practice, since every field is at least 4-byte\naligned (but there is probably an extra 4 bytes of padding at the end).\n\nThe \"right\" way is probably something like:\n\n const char *header = data;\n if (get_be32(header) != RIDX_SIGNATURE)\n error...;\n header += 4;\n if (get_be32(header) != 1)\n error...;\n header += 4;\n ...etc...\n\nI thought we had helpers to read and advance the pointer, but it looks\nlike those are specific to the bitmap code (e.g., read_be32(), though it\nuses a separate offset variable).\n\nI dunno. Maybe I am being overly picky. The .idx code already does it\nlike this, and I believe the index (as in .git/index) does, too. We have\nrun into problems (as in b5007211b6 (pack-bitmap: do not use gcc packed\nattribute, 2014-11-27)), but that was due to a more odd-sized struct, as\nwell as using sizeof().\n\nThe rest of the patch looks good to me.\n\n-Peff\n"},{"id":"415536","messageId":"YBNWsJBDUpNiRXP4@coredump.intra.peff.net","threadId":"54962","inReplyTo":"8a3e70454b9bc64fc7a5ff07d47f7fde018e6a3d.1611617820.git.me@ttaylorr.com","subject":"Re: [PATCH v3 03/10] builtin/index-pack.c: allow stripping arbitrary extensions","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-29T00:28:32Z","receivedAt":"2021-01-29T00:29:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 25, 2021 at 06:37:22PM -0500, Taylor Blau wrote:\n\n> To derive the filename for a .idx file, 'git index-pack' uses\n> derive_filename() to strip the '.pack' suffix and add the new suffix.\n> \n> Prepare for stripping off suffixes other than '.pack' by making the\n> suffix to strip a parameter of derive_filename(). In order to make this\n> consistent with the \"suffix\" parameter which does not begin with a \".\",\n> an additional check in derive_filename.\n\nMaybe \"add\" missing from the final line?\n\n> +static const char *derive_filename(const char *pack_name, const char *strip,\n> +\t\t\t\t const char *suffix, struct strbuf *buf)\n> {\n> \tsize_t len;\n> -\tif (!strip_suffix(pack_name, \".pack\", &len))\n> -\t\tdie(_(\"packfile name '%s' does not end with '.pack'\"),\n> -\t\t pack_name);\n> +\tif (!strip_suffix(pack_name, strip, &len) || !len ||\n> +\t pack_name[len - 1] != '.')\n> +\t\tdie(_(\"packfile name '%s' does not end with '.%s'\"),\n> +\t\t pack_name, strip);\n> \tstrbuf_add(buf, pack_name, len);\n> -\tstrbuf_addch(buf, '.');\n\nLooks good to me.\n\n-Peff\n"},{"id":"415537","messageId":"YBNXQDlmdZVeMsA8@coredump.intra.peff.net","threadId":"54962","inReplyTo":"7e29f2d3a08c42b1e8368c4a2f52a11cc47ee959.1611617820.git.me@ttaylorr.com","subject":"Re: [PATCH v3 06/10] Documentation/config/pack.txt: advertise 'pack.writeReverseIndex'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-29T00:30:56Z","receivedAt":"2021-01-29T00:31:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 25, 2021 at 06:37:34PM -0500, Taylor Blau wrote:\n\n> +pack.writeReverseIndex::\n> +\tWhen true, git will write a corresponding .rev file (see:\n> +\tlink:../technical/pack-format.html[Documentation/technical/pack-format.txt])\n> +\tfor each new packfile that it writes in all places except for\n> +\tlinkgit:git-fast-import[1] and in the bulk checkin mechanism.\n> +\tDefaults to false.\n\nWe may want to teach fast-import about this, too, but I think it's OK to\ndraw the line here for now (it's already reasonably well known that the\ngenerated packs aren't amazing and benefit from repacking).\n\n-Peff\n"},{"id":"415539","messageId":"YBNasXXZc1BEkMxH@coredump.intra.peff.net","threadId":"54962","inReplyTo":"7cf16485cccccf365101d30138d9ee8b00d705d0.1611617820.git.me@ttaylorr.com","subject":"Re: [PATCH v3 07/10] t: prepare for GIT_TEST_WRITE_REV_INDEX","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-29T00:45:37Z","receivedAt":"2021-01-29T00:48:43Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 25, 2021 at 06:37:38PM -0500, Taylor Blau wrote:\n\n> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> index 297de502a9..2fc3aadbd1 100755\n> --- a/t/t5319-multi-pack-index.sh\n> +++ b/t/t5319-multi-pack-index.sh\n> @@ -710,8 +710,9 @@ test_expect_success 'expire respects .keep files' '\n> \t\tPACKA=$(ls .git/objects/pack/a-pack*\\.pack | sed s/\\.pack\\$//) &&\n> \t\ttouch $PACKA.keep &&\n> \t\tgit multi-pack-index expire &&\n> -\t\tls -S .git/objects/pack/a-pack* | grep $PACKA >a-pack-files &&\n> -\t\ttest_line_count = 3 a-pack-files &&\n> +\t\ttest_path_is_file $PACKA.idx &&\n> +\t\ttest_path_is_file $PACKA.keep &&\n> +\t\ttest_path_is_file $PACKA.pack &&\n\nI find the post-image here more readable than the original. It probably\nruns faster, too (zero processes instead of three).\n\n> --- a/t/t5325-reverse-index.sh\n> +++ b/t/t5325-reverse-index.sh\n> @@ -3,6 +3,10 @@\n> test_description='on-disk reverse index'\n> . ./test-lib.sh\n> \n> +# The below tests want control over the 'pack.writeReverseIndex' setting\n> +# themselves to assert various combinations of it with other options.\n> +sane_unset GIT_TEST_WRITE_REV_INDEX\n\nI think we had a discussion a while ago about sane_unset outside of an\n&&-chain, where it does nothing that regular \"unset\" would not. But I\ndon't know if we reached any kind of conclusion. I actually argued that\nit was fine in:\n\n https://lore.kernel.org/git/20200630185536.GA1888406@coredump.intra.peff.net/\n\nSo I guess I should probably stand by that. ;)\n\n> --- a/t/t5604-clone-reference.sh\n> +++ b/t/t5604-clone-reference.sh\n> @@ -329,7 +329,7 @@ test_expect_success SYMLINKS 'clone repo with symlinked or unknown files at obje\n> \tfor raw in $(ls T*.raw)\n> \tdo\n> \t\tsed -e \"s!/../!/Y/!; s![0-9a-f]\\{38,\\}!Z!\" -e \"/commit-graph/d\" \\\n> -\t\t -e \"/multi-pack-index/d\" <$raw >$raw.de-sha-1 &&\n> +\t\t -e \"/multi-pack-index/d\" -e \"/rev/d\" <$raw >$raw.de-sha-1 &&\n\nThis one is less readable than before. :) I'm not sure how to really\nimprove that, though.\n\n> --- a/t/t5702-protocol-v2.sh\n> +++ b/t/t5702-protocol-v2.sh\n> @@ -851,8 +851,10 @@ test_expect_success 'part of packfile response provided as URI' '\n> \ttest -f h2found &&\n> \n> \t# Ensure that there are exactly 6 files (3 .pack and 3 .idx).\n> -\tls http_child/.git/objects/pack/* >filelist &&\n> -\ttest_line_count = 6 filelist\n> +\tls http_child/.git/objects/pack/*.pack >packlist &&\n> +\tls http_child/.git/objects/pack/*.idx >idxlist &&\n> +\ttest_line_count = 3 idxlist &&\n> +\ttest_line_count = 3 packlist\n> '\n\nHmm. Too bad we can't rely on shell brace expansion, as:\n\n ls http_child/.git/objects/pack/*.{pack,idx}\n\nwould be more readable. You could still do it in a single \"ls\" by\nwriting out both arguments manually, but it's probably not that\nimportant.\n\n> test_expect_success 'fetching with valid packfile URI but invalid hash fails' '\n> @@ -905,8 +907,10 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' '\n> \t\tclone \"$HTTPD_URL/smart/http_parent\" http_child &&\n> \n> \t# Ensure that there are exactly 4 files (2 .pack and 2 .idx).\n> -\tls http_child/.git/objects/pack/* >filelist &&\n> -\ttest_line_count = 4 filelist\n> +\tls http_child/.git/objects/pack/*.pack >packlist &&\n> +\tls http_child/.git/objects/pack/*.idx >idxlist &&\n> +\ttest_line_count = 2 idxlist &&\n> +\ttest_line_count = 2 packlist\n> '\n\nLikewise. But...\n\n> diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\n> index 4a3b8f48ac..f76586f808 100755\n> --- a/t/t6500-gc.sh\n> +++ b/t/t6500-gc.sh\n> @@ -106,17 +106,17 @@ test_expect_success 'auto gc with too many loose objects does not attempt to cre\n> \ttest_commit \"$(test_oid obj2)\" &&\n> \t# Our first gc will create a pack; our second will create a second pack\n> \tgit gc --auto &&\n> -\tls .git/objects/pack | sort >existing_packs &&\n> +\tls .git/objects/pack/pack-*.pack | sort >existing_packs &&\n> \ttest_commit \"$(test_oid obj3)\" &&\n> \ttest_commit \"$(test_oid obj4)\" &&\n> \n> \tgit gc --auto 2>err &&\n> \ttest_i18ngrep ! \"^warning:\" err &&\n> -\tls .git/objects/pack/ | sort >post_packs &&\n> +\tls .git/objects/pack/pack-*.pack | sort >post_packs &&\n> \tcomm -1 -3 existing_packs post_packs >new &&\n> \tcomm -2 -3 existing_packs post_packs >del &&\n> \ttest_line_count = 0 del && # No packs are deleted\n> -\ttest_line_count = 2 new # There is one new pack and its .idx\n> +\ttest_line_count = 1 new # There is one new pack\n> '\n\nThis one is making the test a bit looser (it would miss a case where we\nsomehow failed to generate the .idx). That seems like an unlikely bug,\nbut I wonder if we can keep the original behavior. I guess:\n\n ls .git/objects/pack/*.pack \\\n .git/objects/pack/*.idx |\n sort >post_packs\n\nwould work?\n\n> test_expect_success 'gc --no-quiet' '\n> diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\n> index 3d17e932a0..8f1caf8025 100755\n> --- a/t/t9300-fast-import.sh\n> +++ b/t/t9300-fast-import.sh\n> @@ -1632,7 +1632,10 @@ test_expect_success 'O: blank lines not necessary after other commands' '\n> \tINPUT_END\n> \n> \tgit fast-import -\ttest 8 = $(find .git/objects/pack -type f | grep -v multi-pack-index | wc -l) &&\n> +\tls -la .git/objects/pack/pack-*.pack >packlist &&\n> +\tls -la .git/objects/pack/pack-*.pack >idxlist &&\n> +\ttest_line_count = 4 idxlist &&\n> +\ttest_line_count = 4 packlist &&\n\nAnother case where I don't think we're losing anything (actually, we are\ngaining just slightly; if a bug somehow created 6 packs and 2 idx files,\nwe'd now notice!).\n\n-Peff\n"},{"id":"415540","messageId":"YBNbM1STgzXxfe0u@coredump.intra.peff.net","threadId":"54962","inReplyTo":"02550a251de4f477a537968fda921b52586794dc.1611617820.git.me@ttaylorr.com","subject":"Re: [PATCH v3 08/10] t: support GIT_TEST_WRITE_REV_INDEX","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-29T00:47:47Z","receivedAt":"2021-01-29T00:50:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 25, 2021 at 06:37:42PM -0500, Taylor Blau wrote:\n\n> Add a new option that unconditionally enables the pack.writeReverseIndex\n> setting in order to run the whole test suite in a mode that generates\n> on-disk reverse indexes. Additionally, enable this mode in the second\n> run of tests under linux-gcc in 'ci/run-build-and-tests.sh'.\n> \n> Once on-disk reverse indexes are proven out over several releases, we\n> can change the default value of that configuration to 'true', and drop\n> this patch.\n\nSounds like a good plan.\n\n> diff --git a/pack-revindex.h b/pack-revindex.h\n> index 61b2f3ab75..d1a0595e89 100644\n> --- a/pack-revindex.h\n> +++ b/pack-revindex.h\n> @@ -16,9 +16,12 @@\n> * can be found\n> */\n> \n> +\n> #define RIDX_SIGNATURE 0x52494458 /* \"RIDX\" */\n> #define RIDX_VERSION 1\n\nFunny stray whitespace.\n\n-Peff\n"},{"id":"415542","messageId":"YBNcjmn5Jm/EmHwS@coredump.intra.peff.net","threadId":"54962","inReplyTo":"a66d2f9f7c20eeb813656e66b3ad9d42f2eecf34.1611617820.git.me@ttaylorr.com","subject":"Re: [PATCH v3 09/10] pack-revindex: ensure that on-disk reverse indexes are given precedence","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-29T00:53:34Z","receivedAt":"2021-01-29T00:54:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 25, 2021 at 06:37:46PM -0500, Taylor Blau wrote:\n\n> When an on-disk reverse index exists, there is no need to generate one\n> in memory. In fact, doing so can be slow, and require large amounts of\n> the heap.\n> \n> Let's make sure that we treat the on-disk reverse index with precedence\n> (i.e., that when it exists, we don't bother trying to generate an\n> equivalent one in memory) by teaching Git how to conditionally die()\n> when generating a reverse index in memory.\n> \n> Then, add a test to ensure that when (a) an on-disk reverse index\n> exists, and (b) when setting GIT_TEST_REV_INDEX_DIE_IN_MEMORY, that we\n> do not die, implying that we read from the on-disk one.\n\nI don't love this kind of hackery, as it will have to live forever if we\nwant to keep testing this feature. But I also don't know how else to\ntell in the regular test suite that we are avoiding the slow path.\n\nWould it be enough to instead add a t/perf test which checks the speed\nof:\n\n echo HEAD | git cat-file --batch-check='%(objectsize:disk)'\n\n? I hate relying on the perf suite, because it gets run way less often\n(and requires a lot of squinting to interpret the results). But it\nwouldn't require any extra code the binary, as it's observing the actual\nuser-visible thing we care about: speed.\n\n(I guess we care as much or more about peak heap usage, but measuring\nthat is a whole other can of worms).\n\n-Peff\n"},{"id":"415543","messageId":"YBNfAEb+LYAIiPXB@coredump.intra.peff.net","threadId":"54962","inReplyTo":"38c8afabf25a7f5e144850938cf00b53e9cf25fd.1611617820.git.me@ttaylorr.com","subject":"Re: [PATCH v3 10/10] t5325: check both on-disk and in-memory reverse index","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-29T01:04:00Z","receivedAt":"2021-01-29T01:05:59Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 25, 2021 at 06:37:51PM -0500, Taylor Blau wrote:\n\n> Right now, the test suite can be run with 'GIT_TEST_WRITE_REV_INDEX=1'\n> in the environment, which causes all operations which write a pack to\n> also write a .rev file.\n> \n> To prepare for when that eventually becomes the default, we should\n> continue to test the in-memory reverse index, too, in order to avoid\n> losing existing coverage. Unfortuantely, explicit existing coverage is\n> rather sparse, so only a basic test is added.\n> \n> The new test is parameterized over whether or not the .rev file should\n> be written, and is run in both modes by t5325 (without having to touch\n> GIT_TEST_WRITE_REV_INDEX).\n\nThanks, I think this is worth having.\n\n> +revindex_tests () {\n> +\ton_disk=\"$1\"\n> +\n> +\ttest_expect_success \"setup revindex tests (on_disk=$on_disk)\" \"\n> +\t\ttest_oid_cache <<-EOF &&\n> +\t\tdisklen sha1:47\n> +\t\tdisklen sha256:59\n> +\t\tEOF\n\nThese sizes aren't really guaranteed to be stable. I think you've\neliminated most variables by not having the opportunity for deltas,\nthough we're still subject to the whims of zlib, for example.\n\nI think what we care about most is just that the on-disk and fallback\ncases generate the same results. If they do, we can assume those results\nare probably sane. If they're not, then a lot of _other_ stuff is going\nto be broken (like, say, pack-objects, which is writing out N bytes\nbased on the sizes claimed by the revindex).\n\nWhat do you think about just doing this:\n\n ...setup some commits... &&\n git rev-list --objects --no-object-names --all >objects &&\n\n git -c pack.writeReverseIndex=false repack -ad &&\n test_path_is_missing .git/objects/pack/*.rev &&\n git cat-file --batch-check=\"%(objectsize:disk) %(objectname)\" \\\n in-core &&\n\n git -c pack.writeReverseIndex=true repack -ad &&\n test_path_is_file .git/objects/pack/*.rev &&\n git cat-file --batch-check=\"%(objectsize:disk) %(objectname)\" \\\n on-disk &&\n\n test_cmp on-disk in-core\n\n\n> +\t\t\tif test ztrue = \\\"z$on_disk\\\"\n> +\t\t\tthen\n> +\t\t\t\tgit config pack.writeReverseIndex true\n> +\t\t\tfi &&\n\nAre we autotools now? :) I think we can assume that on_disk contains\nsomething sensible. Perhaps even:\n\n git config pack.writeReverseIndex $on_disk\n\nthough if you take my advice above, all of this goes away.\n\n-Peff\n"},{"id":"415544","messageId":"YBNfZeRiHgQwSqGq@coredump.intra.peff.net","threadId":"54962","inReplyTo":"38c8afabf25a7f5e144850938cf00b53e9cf25fd.1611617820.git.me@ttaylorr.com","subject":"Re: [PATCH v3 10/10] t5325: check both on-disk and in-memory reverse index","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-29T01:05:41Z","receivedAt":"2021-01-29T01:08:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 25, 2021 at 06:37:51PM -0500, Taylor Blau wrote:\n\n> Right now, the test suite can be run with 'GIT_TEST_WRITE_REV_INDEX=1'\n> in the environment, which causes all operations which write a pack to\n> also write a .rev file.\n> \n> To prepare for when that eventually becomes the default, we should\n> continue to test the in-memory reverse index, too, in order to avoid\n> losing existing coverage. Unfortuantely, explicit existing coverage is\n> rather sparse, so only a basic test is added.\n\nOh, I forgot to mention: if re-rolling, s/fortuan/fortuna/.\n\n-Peff\n"},{"id":"415545","messageId":"YBNfgywXe5nr+96y@coredump.intra.peff.net","threadId":"54962","inReplyTo":"cover.1611617819.git.me@ttaylorr.com","subject":"Re: [PATCH v3 00/10] pack-revindex: introduce on-disk '.rev' format","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-29T01:06:11Z","receivedAt":"2021-01-29T01:08:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 25, 2021 at 06:37:09PM -0500, Taylor Blau wrote:\n\n> Here is a third reroll of my series to introduce an on-disk format for the\n> reverse index. Since the first series (to introduce a new API) has been merged\n> to 'next', this series has been rebased onto 'next', too.\n> \n> This version is largely unchanged from the previous, with the following\n> exceptions:\n\nThanks, these all seemed like improvements to me. I hadn't read\ncarefully through the test patches yet, so I did so now (in addition to\nchecking out the changes in the earlier patches).\n\nMy comments range between \"musings\" and \"suggestions\". I wouldn't be too\ntorn up if they are all discarded and this applies as-is, but in\naggregate there may be enough to merit one more re-roll.\n\n-Peff\n"},{"id":"415546","messageId":"CAPig+cQZkzFiAyCj6W7-7WC+Dx7aojcj4a+Xvmqvo10RLCoSZw@mail.gmail.com","threadId":"54962","inReplyTo":"YBNasXXZc1BEkMxH@coredump.intra.peff.net","subject":"Re: [PATCH v3 07/10] t: prepare for GIT_TEST_WRITE_REV_INDEX","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-01-29T01:09:03Z","receivedAt":"2021-01-29T01:10:30Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jan 28, 2021 at 7:49 PM Jeff King wrote:\n> On Mon, Jan 25, 2021 at 06:37:38PM -0500, Taylor Blau wrote:\n> > +sane_unset GIT_TEST_WRITE_REV_INDEX\n>\n> I think we had a discussion a while ago about sane_unset outside of an\n> &&-chain, where it does nothing that regular \"unset\" would not. But I\n> don't know if we reached any kind of conclusion. I actually argued that\n> it was fine in:\n> https://lore.kernel.org/git/20200630185536.GA1888406@coredump.intra.peff.net/\n> So I guess I should probably stand by that. ;)\n\nI recall that discussion. The fact that sane_unset() outside of a test\ncaused a hiccup in your reading of the patch which led you to dig\nthrough the mail archive and spend time writing about it in your\nreview could be interpreted as yet another argument against its use\noutside of tests (specifically, it wastes reviewer time). Of course,\nit's such a minor thing, though, not at all worth a re-roll.\n"},{"id":"415548","messageId":"YBNhgX1urWjnziEZ@nand.local","threadId":"54962","inReplyTo":"YBNWXU8xzw0087DC@coredump.intra.peff.net","subject":"Re: [PATCH v3 01/10] packfile: prepare for the existence of '*.rev' files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-29T01:14:41Z","receivedAt":"2021-01-29T01:15:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jan 28, 2021 at 07:27:09PM -0500, Jeff King wrote:\n> I guess this technique was pulled from the .idx code paths, but IMHO we\n> should avoid this kind of struct-casting. It relies on the compiler's\n> struct packing, and I'm not 100% sure it doesn't violate some weird\n> pointer-aliasing rules.\n\nYeah, that's exactly where it came from. I briefly wondered about this\nmyself, but decided not to worry since (a) it closely follows an\nexisting pattern and (b) no compiler should ever pad anywhere but the\nend of this struct since all other fields are aligned without padding.\n\n> OTOH, as long as we do not ever care about sizeof(revindex_header), this\n> is likely to work in practice, since every field is at least 4-byte\n> aligned (but there is probably an extra 4 bytes of padding at the end).\n>\n> The \"right\" way is probably something like:\n>\n> const char *header = data;\n> if (get_be32(header) != RIDX_SIGNATURE)\n> error...;\n> header += 4;\n> if (get_be32(header) != 1)\n> error...;\n> header += 4;\n> ...etc...\n\nMakes sense.\n\n> I thought we had helpers to read and advance the pointer, but it looks\n> like those are specific to the bitmap code (e.g., read_be32(), though it\n> uses a separate offset variable).\n>\n> I dunno. Maybe I am being overly picky. The .idx code already does it\n> like this, and I believe the index (as in .git/index) does, too. We have\n> run into problems (as in b5007211b6 (pack-bitmap: do not use gcc packed\n> attribute, 2014-11-27)), but that was due to a more odd-sized struct, as\n> well as using sizeof().\n\nHow about a set of follow-up patches to address all of these spots at\nthe same time? That would allow us to move forward here (which is safe\nto do, as you note), and address all of these instances together\nuniformly.\n\nSound good?\n\nThanks,\nTaylor\n"},{"id":"415549","messageId":"YBNhqDTG5dEcIbWP@nand.local","threadId":"54962","inReplyTo":"YBNWsJBDUpNiRXP4@coredump.intra.peff.net","subject":"Re: [PATCH v3 03/10] builtin/index-pack.c: allow stripping arbitrary extensions","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-29T01:15:20Z","receivedAt":"2021-01-29T01:16:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jan 28, 2021 at 07:28:32PM -0500, Jeff King wrote:\n> On Mon, Jan 25, 2021 at 06:37:22PM -0500, Taylor Blau wrote:\n> > Prepare for stripping off suffixes other than '.pack' by making the\n> > suffix to strip a parameter of derive_filename(). In order to make this\n> > consistent with the \"suffix\" parameter which does not begin with a \".\",\n> > an additional check in derive_filename.\n>\n> Maybe \"add\" missing from the final line?\n\nOops, yeah. I'm happy to send a replacement to help with queuing, if the\nmaintainer thinks that would be helpful.\n\nThanks,\nTaylor\n"},{"id":"415550","messageId":"YBNiNAVhlRpNKzr+@nand.local","threadId":"54962","inReplyTo":"YBNXQDlmdZVeMsA8@coredump.intra.peff.net","subject":"Re: [PATCH v3 06/10] Documentation/config/pack.txt: advertise 'pack.writeReverseIndex'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-29T01:17:40Z","receivedAt":"2021-01-29T01:18:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jan 28, 2021 at 07:30:56PM -0500, Jeff King wrote:\n> On Mon, Jan 25, 2021 at 06:37:34PM -0500, Taylor Blau wrote:\n>\n> > +pack.writeReverseIndex::\n> > +\tWhen true, git will write a corresponding .rev file (see:\n> > +\tlink:../technical/pack-format.html[Documentation/technical/pack-format.txt])\n> > +\tfor each new packfile that it writes in all places except for\n> > +\tlinkgit:git-fast-import[1] and in the bulk checkin mechanism.\n> > +\tDefaults to false.\n>\n> We may want to teach fast-import about this, too, but I think it's OK to\n> draw the line here for now (it's already reasonably well known that the\n> generated packs aren't amazing and benefit from repacking).\n\nMaybe this was worth explaining in the documentation, but this choice\nwas intentional. I'm not aware of any use-case that keeps the packs\ngenerated by fast-import around for very long (i.e., I'd expect any 'git\nfast-import' to be pretty quickly followed by a 'git repack -ad'), so I\ndon't think we'd want to spend time generating a reverse index for a\npack structure that we're about to immediately discard.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"415552","messageId":"YBNjLqOe1NtpziV7@nand.local","threadId":"54962","inReplyTo":"YBNasXXZc1BEkMxH@coredump.intra.peff.net","subject":"Re: [PATCH v3 07/10] t: prepare for GIT_TEST_WRITE_REV_INDEX","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-29T01:21:50Z","receivedAt":"2021-01-29T01:23:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jan 28, 2021 at 07:45:37PM -0500, Jeff King wrote:\n> On Mon, Jan 25, 2021 at 06:37:38PM -0500, Taylor Blau wrote:\n>\n> > diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\n> > index 297de502a9..2fc3aadbd1 100755\n> > --- a/t/t5319-multi-pack-index.sh\n> > +++ b/t/t5319-multi-pack-index.sh\n> > @@ -710,8 +710,9 @@ test_expect_success 'expire respects .keep files' '\n> > \t\tPACKA=$(ls .git/objects/pack/a-pack*\\.pack | sed s/\\.pack\\$//) &&\n> > \t\ttouch $PACKA.keep &&\n> > \t\tgit multi-pack-index expire &&\n> > -\t\tls -S .git/objects/pack/a-pack* | grep $PACKA >a-pack-files &&\n> > -\t\ttest_line_count = 3 a-pack-files &&\n> > +\t\ttest_path_is_file $PACKA.idx &&\n> > +\t\ttest_path_is_file $PACKA.keep &&\n> > +\t\ttest_path_is_file $PACKA.pack &&\n>\n> I find the post-image here more readable than the original. It probably\n> runs faster, too (zero processes instead of three).\n>\n> > --- a/t/t5325-reverse-index.sh\n> > +++ b/t/t5325-reverse-index.sh\n> > @@ -3,6 +3,10 @@\n> > test_description='on-disk reverse index'\n> > . ./test-lib.sh\n> >\n> > +# The below tests want control over the 'pack.writeReverseIndex' setting\n> > +# themselves to assert various combinations of it with other options.\n> > +sane_unset GIT_TEST_WRITE_REV_INDEX\n>\n> I think we had a discussion a while ago about sane_unset outside of an\n> &&-chain, where it does nothing that regular \"unset\" would not. But I\n> don't know if we reached any kind of conclusion. I actually argued that\n> it was fine in:\n>\n> https://lore.kernel.org/git/20200630185536.GA1888406@coredump.intra.peff.net/\n>\n> So I guess I should probably stand by that. ;)\n\nI think I probably took this from the trace2 tests? Not sure. I'm glad\nthat it's not wrong, strictly speaking.\n\nThis is another instance that I'd be happy to send a follow-up patch to\nget rid of all of these at once, unless you feel strongly that it should\nbe changed in this series before applying.\n\n> > --- a/t/t5604-clone-reference.sh\n> > +++ b/t/t5604-clone-reference.sh\n> > @@ -329,7 +329,7 @@ test_expect_success SYMLINKS 'clone repo with symlinked or unknown files at obje\n> > \tfor raw in $(ls T*.raw)\n> > \tdo\n> > \t\tsed -e \"s!/../!/Y/!; s![0-9a-f]\\{38,\\}!Z!\" -e \"/commit-graph/d\" \\\n> > -\t\t -e \"/multi-pack-index/d\" <$raw >$raw.de-sha-1 &&\n> > +\t\t -e \"/multi-pack-index/d\" -e \"/rev/d\" <$raw >$raw.de-sha-1 &&\n>\n> This one is less readable than before. :) I'm not sure how to really\n> improve that, though.\n\nYeah, this follows from a suggestion that Ævar gave in:\n\n https://lore.kernel.org/git/878s8y5bdn.fsf@evledraar.gmail.com/\n\nFWIW, I found that his suggestion was *clearer* than what was in v2. But\nI get that others may feel differently, too.\n\n> > --- a/t/t5702-protocol-v2.sh\n> > +++ b/t/t5702-protocol-v2.sh\n> > @@ -851,8 +851,10 @@ test_expect_success 'part of packfile response provided as URI' '\n> > \ttest -f h2found &&\n> >\n> > \t# Ensure that there are exactly 6 files (3 .pack and 3 .idx).\n> > -\tls http_child/.git/objects/pack/* >filelist &&\n> > -\ttest_line_count = 6 filelist\n> > +\tls http_child/.git/objects/pack/*.pack >packlist &&\n> > +\tls http_child/.git/objects/pack/*.idx >idxlist &&\n> > +\ttest_line_count = 3 idxlist &&\n> > +\ttest_line_count = 3 packlist\n> > '\n>\n> Hmm. Too bad we can't rely on shell brace expansion, as:\n>\n> ls http_child/.git/objects/pack/*.{pack,idx}\n>\n> would be more readable. You could still do it in a single \"ls\" by\n> writing out both arguments manually, but it's probably not that\n> important.\n\nAgreed on all of that.\n\n> This one is making the test a bit looser (it would miss a case where we\n> somehow failed to generate the .idx). That seems like an unlikely bug,\n> but I wonder if we can keep the original behavior. I guess:\n>\n> ls .git/objects/pack/*.pack \\\n> .git/objects/pack/*.idx |\n> sort >post_packs\n>\n> would work?\n\nSure, I see what you're saying. To be honest, I'm skeptical that we'd\nhave a bug which failed only this one test, so I'm hesitant to send a\nreplacement/reroll for this alone.\n\nIf you feel strongly, though, I'm happy to change it. (But, I'll err on\nthe side of leaving it as-is, since you indicated in your response to\nv3's cover letter that you'd be OK if I discarded some or all of your\nsuggestions).\n\n> > test_expect_success 'gc --no-quiet' '\n> > diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\n> > index 3d17e932a0..8f1caf8025 100755\n> > --- a/t/t9300-fast-import.sh\n> > +++ b/t/t9300-fast-import.sh\n> > @@ -1632,7 +1632,10 @@ test_expect_success 'O: blank lines not necessary after other commands' '\n> > \tINPUT_END\n> >\n> > \tgit fast-import > -\ttest 8 = $(find .git/objects/pack -type f | grep -v multi-pack-index | wc -l) &&\n> > +\tls -la .git/objects/pack/pack-*.pack >packlist &&\n> > +\tls -la .git/objects/pack/pack-*.pack >idxlist &&\n> > +\ttest_line_count = 4 idxlist &&\n> > +\ttest_line_count = 4 packlist &&\n>\n> Another case where I don't think we're losing anything (actually, we are\n> gaining just slightly; if a bug somehow created 6 packs and 2 idx files,\n> we'd now notice!).\n\nYay!\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"415553","messageId":"YBNkCsshbCkyC0w4@nand.local","threadId":"54962","inReplyTo":"YBNcjmn5Jm/EmHwS@coredump.intra.peff.net","subject":"Re: [PATCH v3 09/10] pack-revindex: ensure that on-disk reverse indexes are given precedence","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-29T01:25:30Z","receivedAt":"2021-01-29T01:26:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jan 28, 2021 at 07:53:34PM -0500, Jeff King wrote:\n> On Mon, Jan 25, 2021 at 06:37:46PM -0500, Taylor Blau wrote:\n>\n> > When an on-disk reverse index exists, there is no need to generate one\n> > in memory. In fact, doing so can be slow, and require large amounts of\n> > the heap.\n> >\n> > Let's make sure that we treat the on-disk reverse index with precedence\n> > (i.e., that when it exists, we don't bother trying to generate an\n> > equivalent one in memory) by teaching Git how to conditionally die()\n> > when generating a reverse index in memory.\n> >\n> > Then, add a test to ensure that when (a) an on-disk reverse index\n> > exists, and (b) when setting GIT_TEST_REV_INDEX_DIE_IN_MEMORY, that we\n> > do not die, implying that we read from the on-disk one.\n>\n> I don't love this kind of hackery, as it will have to live forever if we\n> want to keep testing this feature. But I also don't know how else to\n> tell in the regular test suite that we are avoiding the slow path.\n>\n> Would it be enough to instead add a t/perf test which checks the speed\n> of:\n>\n> echo HEAD | git cat-file --batch-check='%(objectsize:disk)'\n>\n> ? I hate relying on the perf suite, because it gets run way less often\n> (and requires a lot of squinting to interpret the results). But it\n> wouldn't require any extra code the binary, as it's observing the actual\n> user-visible thing we care about: speed.\n>\n> (I guess we care as much or more about peak heap usage, but measuring\n> that is a whole other can of worms).\n\nYeah, I think that the thing we care most about is peak RSS, but I agree\nthat I don't really want to wade into figuring out how to measure that\nreliably during test time (I think there is a separate and less relevant\nargument about whether that is something that we should be testing or\nnot).\n\nBut, getting back to your original comment, I think that I actually\nprefer to have the GIT_TEST_XYZ_DIE stuff in the binary than I do\nrelying on the perf suite to catch stuff like this.\n\nI understand your concern about the binary size. I guess you could\n#ifdef this out and only build it in during the test suite, but then\nyou're testing a different binary, and so that calls into question the\nintegrity of the test suite itself, etc. etc.\n\nSo, I guess that's all to say that I while I do find this to be hack-y\nand gross, I don't think that it's all that bad when you compare it to\nthe alternatives.\n\nAs usual, I'm happy to change it if you feel strongly that it should be\nchanged.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"415554","messageId":"YBNlkvrxvAYrLeMc@nand.local","threadId":"54962","inReplyTo":"YBNfZeRiHgQwSqGq@coredump.intra.peff.net","subject":"Re: [PATCH v3 10/10] t5325: check both on-disk and in-memory reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-29T01:32:02Z","receivedAt":"2021-01-29T01:32:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jan 28, 2021 at 08:05:41PM -0500, Jeff King wrote:\n> Oh, I forgot to mention: if re-rolling, s/fortuan/fortuna/.\n\nI do like your suggestion quite a lot: it gets rid of some ugliness,\nwhile making the overall test structure simpler. Here's a replacement\nfor the final series.\n\nJunio: when queuing, please apply this one instead of the original v3\n10/10.\n\nThanks,\nTaylor\n\n--- >8 ---\n\nSubject: [PATCH] t5325: check both on-disk and in-memory reverse index\n\nRight now, the test suite can be run with 'GIT_TEST_WRITE_REV_INDEX=1'\nin the environment, which causes all operations which write a pack to\nalso write a .rev file.\n\nTo prepare for when that eventually becomes the default, we should\ncontinue to test the in-memory reverse index, too, in order to avoid\nlosing existing coverage. Unfortunately, explicit existing coverage is\nrather sparse, so only a basic test is added that compares the result of\n\n git rev-list --objects --no-object-names --all |\n git cat-file --batch-check='%(objectsize:disk) %(objectname)'\n\nwith and without an on-disk reverse index.\n\nSuggested-by: Jeff King \nHelped-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n t/t5325-reverse-index.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh\nindex a344b18d7e..da453f68d6 100755\n--- a/t/t5325-reverse-index.sh\n+++ b/t/t5325-reverse-index.sh\n@@ -94,4 +94,27 @@ test_expect_success 'reverse index is not generated when available on disk' '\n \t\t--batch-check=\"%(objectsize:disk)\" objects &&\n+\n+\t\tgit -c pack.writeReverseIndex=false repack -ad &&\n+\t\ttest_path_is_missing $packdir/pack-*.rev &&\n+\t\tgit cat-file --batch-check=\"%(objectsize:disk) %(objectname)\" \\\n+\t\t\tin-core &&\n+\n+\t\tgit -c pack.writeReverseIndex=true repack -ad &&\n+\t\ttest_path_is_file $packdir/pack-*.rev &&\n+\t\tgit cat-file --batch-check=\"%(objectsize:disk) %(objectname)\" \\\n+\t\t\ton-disk &&\n+\n+\t\ttest_cmp on-disk in-core\n+\t)\n+'\n test_done\n--\n2.30.0.138.g6d7191ea01\n\n"},{"id":"415555","messageId":"YBNmKXy9qxvbjltG@nand.local","threadId":"54962","inReplyTo":"YBNfgywXe5nr+96y@coredump.intra.peff.net","subject":"Re: [PATCH v3 00/10] pack-revindex: introduce on-disk '.rev' format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-29T01:34:33Z","receivedAt":"2021-01-29T01:35:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jan 28, 2021 at 08:06:11PM -0500, Jeff King wrote:\n> On Mon, Jan 25, 2021 at 06:37:09PM -0500, Taylor Blau wrote:\n>\n> > Here is a third reroll of my series to introduce an on-disk format for the\n> > reverse index. Since the first series (to introduce a new API) has been merged\n> > to 'next', this series has been rebased onto 'next', too.\n> >\n> > This version is largely unchanged from the previous, with the following\n> > exceptions:\n>\n> Thanks, these all seemed like improvements to me. I hadn't read\n> carefully through the test patches yet, so I did so now (in addition to\n> checking out the changes in the earlier patches).\n\nThank you; I appreciate your close review :-).\n\n> My comments range between \"musings\" and \"suggestions\". I wouldn't be too\n> torn up if they are all discarded and this applies as-is, but in\n> aggregate there may be enough to merit one more re-roll.\n\nThanks again. I discarded a fair amount of them, which I hope you don't\nmind. I think a few of them could be reasonably dealt with outside of\nthis series. Mostly because they have implications on other parts of the\ncodebase. So I'd like to deal with them all at once, but without adding\nto the scope of this series.\n\nI did especially like your final suggestion about the new test in t5325,\nso I sent a replacement patch that takes that suggestion (as well as the\ntypofix that you pointed out).\n\nUnless you have any objections, I think that this is ready to get picked\nup.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"415558","messageId":"xmqqsg6ktqe1.fsf@gitster.c.googlers.com","threadId":"54962","inReplyTo":"YBNasXXZc1BEkMxH@coredump.intra.peff.net","subject":"Re: [PATCH v3 07/10] t: prepare for GIT_TEST_WRITE_REV_INDEX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-29T02:42:46Z","receivedAt":"2021-01-29T02:43:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> \t# Ensure that there are exactly 6 files (3 .pack and 3 .idx).\n>> -\tls http_child/.git/objects/pack/* >filelist &&\n>> -\ttest_line_count = 6 filelist\n>> +\tls http_child/.git/objects/pack/*.pack >packlist &&\n>> +\tls http_child/.git/objects/pack/*.idx >idxlist &&\n>> +\ttest_line_count = 3 idxlist &&\n>> +\ttest_line_count = 3 packlist\n>> '\n>\n> Hmm. Too bad we can't rely on shell brace expansion, as:\n>\n> ls http_child/.git/objects/pack/*.{pack,idx}\n>\n> would be more readable. You could still do it in a single \"ls\" by\n> writing out both arguments manually, but it's probably not that\n> important.\n\nThis part looks a bit familiar as I had to fix the interaction with\nJonathan's topic, IIRC. We need to update the comment. It is not\nensuring \"exact 6\"---it merely is interested in having three\npack/idx pair, and carefully expressing that by preparing for the\npresence of other cruft in objects/pack/ directory other people may\ncreate (like \".rev\", but we may gain more).\n\nI wonder if we even _care_ about .idx. Why not just count .pack, to\nprepare for a possible distant future where we do not even write .idx\nbut append to existing multi-pack-index as we download a new pack\nstream and store it in a .pack, or something?\n\n>> -\tls .git/objects/pack | sort >existing_packs &&\n>> +\tls .git/objects/pack/pack-*.pack | sort >existing_packs &&\n>> \ttest_commit \"$(test_oid obj3)\" &&\n>> \ttest_commit \"$(test_oid obj4)\" &&\n>> \n>> \tgit gc --auto 2>err &&\n>> \ttest_i18ngrep ! \"^warning:\" err &&\n>> -\tls .git/objects/pack/ | sort >post_packs &&\n>> +\tls .git/objects/pack/pack-*.pack | sort >post_packs &&\n>> \tcomm -1 -3 existing_packs post_packs >new &&\n>> \tcomm -2 -3 existing_packs post_packs >del &&\n>> \ttest_line_count = 0 del && # No packs are deleted\n>> -\ttest_line_count = 2 new # There is one new pack and its .idx\n>> +\ttest_line_count = 1 new # There is one new pack\n>> '\n>\n> This one is making the test a bit looser (it would miss a case where we\n> somehow failed to generate the .idx). That seems like an unlikely bug,\n> but I wonder if we can keep the original behavior. I guess:\n>\n> ls .git/objects/pack/*.pack \\\n> .git/objects/pack/*.idx |\n> sort >post_packs\n>\n> would work?\n\nI guess we are looking at the same issue from opposite angle. I\nsuspect that it might even be a good thing to only care about .pack\nand ignore everything else in the longer run.\n"},{"id":"415637","messageId":"YBUbOpsyl2BVqRJQ@coredump.intra.peff.net","threadId":"54962","inReplyTo":"YBNhgX1urWjnziEZ@nand.local","subject":"Re: [PATCH v3 01/10] packfile: prepare for the existence of '*.rev' files","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-30T08:39:22Z","receivedAt":"2021-01-30T09:07:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 28, 2021 at 08:14:41PM -0500, Taylor Blau wrote:\n\n> > I dunno. Maybe I am being overly picky. The .idx code already does it\n> > like this, and I believe the index (as in .git/index) does, too. We have\n> > run into problems (as in b5007211b6 (pack-bitmap: do not use gcc packed\n> > attribute, 2014-11-27)), but that was due to a more odd-sized struct, as\n> > well as using sizeof().\n> \n> How about a set of follow-up patches to address all of these spots at\n> the same time? That would allow us to move forward here (which is safe\n> to do, as you note), and address all of these instances together\n> uniformly.\n> \n> Sound good?\n\nThat sounds fine to me. TBH, I am probably OK if that cleanup never\nhappens, as long as nobody finds that it is a problem on there platform.\n\nI do wonder if we should provide more guidance about what is OK and what\nis not in CodingGuidelines, but I have a feeling it would end up to\nesoteric to be generally useful.\n\n-Peff\n"},{"id":"415638","messageId":"YBUbu67CdryFdTXK@coredump.intra.peff.net","threadId":"54962","inReplyTo":"YBNiNAVhlRpNKzr+@nand.local","subject":"Re: [PATCH v3 06/10] Documentation/config/pack.txt: advertise 'pack.writeReverseIndex'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-30T08:41:31Z","receivedAt":"2021-01-30T09:07:57Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 28, 2021 at 08:17:40PM -0500, Taylor Blau wrote:\n\n> On Thu, Jan 28, 2021 at 07:30:56PM -0500, Jeff King wrote:\n> > On Mon, Jan 25, 2021 at 06:37:34PM -0500, Taylor Blau wrote:\n> >\n> > > +pack.writeReverseIndex::\n> > > +\tWhen true, git will write a corresponding .rev file (see:\n> > > +\tlink:../technical/pack-format.html[Documentation/technical/pack-format.txt])\n> > > +\tfor each new packfile that it writes in all places except for\n> > > +\tlinkgit:git-fast-import[1] and in the bulk checkin mechanism.\n> > > +\tDefaults to false.\n> >\n> > We may want to teach fast-import about this, too, but I think it's OK to\n> > draw the line here for now (it's already reasonably well known that the\n> > generated packs aren't amazing and benefit from repacking).\n> \n> Maybe this was worth explaining in the documentation, but this choice\n> was intentional. I'm not aware of any use-case that keeps the packs\n> generated by fast-import around for very long (i.e., I'd expect any 'git\n> fast-import' to be pretty quickly followed by a 'git repack -ad'), so I\n> don't think we'd want to spend time generating a reverse index for a\n> pack structure that we're about to immediately discard.\n\nOne case where I imagine they are kept is in remote-helpers that use the\nimport/export mechanism (e.g., something interacting with remote-hg). I\nsuspect people there just live with the somewhat lousy packs, and then\nperhaps occasionally \"git repack -adf\". That would give them .rev files\nthen, too.\n\nBut as I said, I think it's OK to ignore this for now. If somebody with\nan interest in fast-import wants to add support later, they can.\n\n-Peff\n"},{"id":"415639","messageId":"YBUcz3CY29DjhW/U@coredump.intra.peff.net","threadId":"54962","inReplyTo":"YBNkCsshbCkyC0w4@nand.local","subject":"Re: [PATCH v3 09/10] pack-revindex: ensure that on-disk reverse indexes are given precedence","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-30T08:46:07Z","receivedAt":"2021-01-30T09:08:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 28, 2021 at 08:25:30PM -0500, Taylor Blau wrote:\n\n> But, getting back to your original comment, I think that I actually\n> prefer to have the GIT_TEST_XYZ_DIE stuff in the binary than I do\n> relying on the perf suite to catch stuff like this.\n> \n> I understand your concern about the binary size. I guess you could\n> #ifdef this out and only build it in during the test suite, but then\n> you're testing a different binary, and so that calls into question the\n> integrity of the test suite itself, etc. etc.\n\nI care less about binary size, and more just that the production binary\npeople use day-to-day has a bunch of weird test-only behaviors baked\ninto it. I guess nobody is likely to trigger this by accident, though,\nand anybody who can maliciously impact the environment can do a lot\nworse. So it's not the end of the world to keep. It just offends my\nsense of cleanliness and simplicity. ;)\n\n> So, I guess that's all to say that I while I do find this to be hack-y\n> and gross, I don't think that it's all that bad when you compare it to\n> the alternatives.\n> \n> As usual, I'm happy to change it if you feel strongly that it should be\n> changed.\n\nI agree the alternatives are pretty lousy. I don't feel strongly enough\nto complain further. :)\n\n-Peff\n"},{"id":"415640","messageId":"YBUcFpzhuV+7kRwi@coredump.intra.peff.net","threadId":"54962","inReplyTo":"YBNjLqOe1NtpziV7@nand.local","subject":"Re: [PATCH v3 07/10] t: prepare for GIT_TEST_WRITE_REV_INDEX","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-30T08:43:02Z","receivedAt":"2021-01-30T09:08:29Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 28, 2021 at 08:21:50PM -0500, Taylor Blau wrote:\n\n> [sane_unset outside of a test]\n> \n> I think I probably took this from the trace2 tests? Not sure. I'm glad\n> that it's not wrong, strictly speaking.\n> \n> This is another instance that I'd be happy to send a follow-up patch to\n> get rid of all of these at once, unless you feel strongly that it should\n> be changed in this series before applying.\n\nNope, I don't feel strongly.\n\n> > This one is making the test a bit looser (it would miss a case where we\n> > somehow failed to generate the .idx). That seems like an unlikely bug,\n> > but I wonder if we can keep the original behavior. I guess:\n> >\n> > ls .git/objects/pack/*.pack \\\n> > .git/objects/pack/*.idx |\n> > sort >post_packs\n> >\n> > would work?\n> \n> Sure, I see what you're saying. To be honest, I'm skeptical that we'd\n> have a bug which failed only this one test, so I'm hesitant to send a\n> replacement/reroll for this alone.\n> \n> If you feel strongly, though, I'm happy to change it. (But, I'll err on\n> the side of leaving it as-is, since you indicated in your response to\n> v3's cover letter that you'd be OK if I discarded some or all of your\n> suggestions).\n\nNope, I don't feel strongly. Junio even argued elsewhere that these\ntests may be better off just looking for pack files (which is looser\nthan what they do now, but as we've said, unlikely to matter for any\nrealistic bug). So I'm happy enough with what you have.\n\n-Peff\n"},{"id":"415641","messageId":"YBUdKgmk2X4wd++h@coredump.intra.peff.net","threadId":"54962","inReplyTo":"YBNlkvrxvAYrLeMc@nand.local","subject":"Re: [PATCH v3 10/10] t5325: check both on-disk and in-memory reverse index","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-30T08:47:38Z","receivedAt":"2021-01-30T09:08:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 28, 2021 at 08:32:02PM -0500, Taylor Blau wrote:\n\n> On Thu, Jan 28, 2021 at 08:05:41PM -0500, Jeff King wrote:\n> > Oh, I forgot to mention: if re-rolling, s/fortuan/fortuna/.\n> \n> I do like your suggestion quite a lot: it gets rid of some ugliness,\n> while making the overall test structure simpler. Here's a replacement\n> for the final series.\n> \n> Junio: when queuing, please apply this one instead of the original v3\n> 10/10.\n\nThanks, this looks good to me.\n\n-Peff\n"}]}