{"thread":{"id":"54961","subject":"[PATCH 00/20] pack-revindex: prepare for on-disk reverse index","startedAt":"2021-01-08T18:17:24Z","lastAt":"2021-01-15T09:34:17Z","messageCount":121,"participants":["Taylor Blau","Derrick Stolee","Junio C Hamano","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":20},"messages":[{"id":"413832","messageId":"cover.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":null,"subject":"[PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:16:39Z","receivedAt":"2021-01-08T18:17:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi,\n\nThis is the first of two series to introduce an on-disk alternative to the\nreverse index which is currently generated once per-process and stored in\nmemory.\n\nAs a reminder, the reverse index is used to translate an object's position\nrelative to other objects within the same packfile to that object's position\nrelative to the pack's index.\n\nGenerating the reverse index in memory for repositories with large packs has two\nsignificant drawbacks:\n\n  - It requires allocating sizeof(struct revindex_entry) per packed object.\n\n  - It requires us to sort the entries by their pack offset. This is implemented\n    in sort_revindex() using a radix sort, but still takes considerable time (as\n    benchmarks found in the second series demonstrate).\n\nBoth of these can be addressed by storing the reverse index in a new '.rev' file\nalongside the packs. This file is written once (during pack creation), and does\nnot require sorting when accessed, since it is stored in a sorted order.\n\nThis series lays the groundwork necessary to introduce the file format (which is\nimplemented in the second series). I'm splitting these two up since I think the\nchanges in this series can be queued independently from those in the latter\nseries.\n\nThe goal of this series is to remove direct access of the `struct\nrevindex_entry` type, as well as `struct packed_git`'s `revindex` field. The\non-disk format will be mmap'd and accessed directly, but the format is\nsufficiently different that the whole `revindex` array can't be written as-is.\n\nSeparately from our main goal, the new API that is introduced in this series is\nIMHO easier to read, and more clearly describes what order the given and\nreturned values are relative to.\n\nThe changes are structured as follows:\n\n  - First, a new API is proposed.\n\n  - Then, uses of the old API are removed one by one and replaced with their\n    new counterparts.\n\n  - Finally, without any callers remaining, the old API is removed.\n\nThanks in advance for your review.\n\nTaylor Blau (20):\n  pack-revindex: introduce a new API\n  write_reuse_object(): convert to new revindex API\n  write_reused_pack_one(): convert to new revindex API\n  write_reused_pack_verbatim(): convert to new revindex API\n  check_object(): convert to new revindex API\n  bitmap_position_packfile(): convert to new revindex API\n  show_objects_for_type(): convert to new revindex API\n  get_size_by_pos(): convert to new revindex API\n  try_partial_reuse(): convert to new revindex API\n  rebuild_existing_bitmaps(): convert to new revindex API\n  get_delta_base_oid(): convert to new revindex API\n  retry_bad_packed_offset(): convert to new revindex API\n  packed_object_info(): convert to new revindex API\n  unpack_entry(): convert to new revindex API\n  for_each_object_in_pack(): convert to new revindex API\n  builtin/gc.c: guess the size of the revindex\n  pack-revindex: remove unused 'find_pack_revindex()'\n  pack-revindex: remove unused 'find_revindex_position()'\n  pack-revindex: hide the definition of 'revindex_entry'\n  pack-revindex.c: avoid direct revindex access in\n    'offset_to_pack_pos()'\n\n builtin/gc.c           |  2 +-\n builtin/pack-objects.c | 33 +++++++++++++++++-----------\n pack-bitmap.c          | 44 ++++++++++++++++++-------------------\n pack-revindex.c        | 49 +++++++++++++++++++++++++++--------------\n pack-revindex.h        | 10 +++------\n packfile.c             | 50 ++++++++++++++++++++++++++----------------\n 6 files changed, 109 insertions(+), 79 deletions(-)\n\n-- \n2.30.0.138.g6d7191ea01\n"},{"id":"413833","messageId":"fa6b8309088fd04410ca7276c5cf14db0fb82fb2.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 01/20] pack-revindex: introduce a new API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:16:43Z","receivedAt":"2021-01-08T18:17:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In the next several patches, we will prepare for loading a reverse index\neither in memory, or from a yet-to-be-introduced on-disk format. To do\nthat, we'll introduce an API that avoids the caller explicitly indexing\nthe revindex pointer in the packed_git structure.\n\nThere are four ways to interact with the reverse index. Accordingly,\nfour functions will be exported from 'pack-revindex.h' by the time that\nthe existing API is removed. A caller may:\n\n 1. Load the pack's reverse index. This involves opening up the index,\n    generating an array, and then sorting it. Since opening the index\n    can fail, this function ('load_pack_revindex()') returns an int.\n    Accordingly, it takes only a single argument: the 'struct\n    packed_git' the caller wants to build a reverse index for.\n\n    This function is well-suited for both the current and new API.\n    Callers will have to continue to open the reverse index explicitly,\n    but this function will eventually learn how to detect and load a\n    reverse index from the on-disk format, if one exists. Otherwise, it\n    will fallback to generating one in memory from scratch.\n\n 2. Convert a pack position into an offset. This operation is now\n    called `pack_pos_to_offset()`. It takes a pack and a position, and\n    returns the corresponding off_t.\n\n 3. Convert a pack position into an index position. Same as above; this\n    takes a pack and a position, and returns a uint32_t. This operation\n    is known as `pack_pos_to_index()`.\n\n 4. Find the pack position for a given offset. This operation is now\n    known as `offset_to_pack_pos()`. It takes a pack, an offset, and a\n    pointer to a uint32_t where the position is written, if an object\n    exists at that offset. Otherwise, -1 is returned to indicate\n    failure.\n\n    Unlike some of the callers that used to access '->offset' and '->nr'\n    directly, the error checking around this call is somewhat more\n    robust. This is important since callers can pass an offset which\n    does not contain an object.\n\n    This will become important in a subsequent patch where a caller\n    which does not but could check the return value treats the signed\n    `-1` from `find_revindex_position()` as an index into the 'revindex'\n    array.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-revindex.c | 32 ++++++++++++++++++++++++++++++++\n pack-revindex.h |  4 ++++\n 2 files changed, 36 insertions(+)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex ecdde39cf4..6d86a85208 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -203,3 +203,35 @@ struct revindex_entry *find_pack_revindex(struct packed_git *p, off_t ofs)\n \n \treturn p->revindex + pos;\n }\n+\n+int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n+{\n+\tint ret;\n+\n+\tif (load_pack_revindex(p) < 0)\n+\t\treturn -1;\n+\n+\tret = find_revindex_position(p, ofs);\n+\tif (ret < 0)\n+\t\treturn -1;\n+\t*pos = ret;\n+\treturn 0;\n+}\n+\n+uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos)\n+{\n+\tif (!p->revindex)\n+\t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n+\tif (pos >= p->num_objects)\n+\t\tBUG(\"pack_pos_to_index: out-of-bounds object at %\"PRIu32, pos);\n+\treturn p->revindex[pos].nr;\n+}\n+\n+off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n+{\n+\tif (!p->revindex)\n+\t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n+\tif (pos > p->num_objects)\n+\t\tBUG(\"pack_pos_to_offset: out-of-bounds object at %\"PRIu32, pos);\n+\treturn p->revindex[pos].offset;\n+}\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex 848331d5d6..256c0a9106 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -13,4 +13,8 @@ int find_revindex_position(struct packed_git *p, off_t ofs);\n \n struct revindex_entry *find_pack_revindex(struct packed_git *p, off_t ofs);\n \n+int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos);\n+uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);\n+off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos);\n+\n #endif\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413834","messageId":"00668523e1cd860f6de08dd7c5a2a54edc08b7b6.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 02/20] write_reuse_object(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:16:48Z","receivedAt":"2021-01-08T18:17:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"First replace 'find_pack_revindex()' with its replacement\n'offset_to_pack_pos()'. This prevents any bogus OFS_DELTA that may make\nits way through until 'write_reuse_object()' from causing a bad memory\nread (if 'revidx' is 'NULL')\n\nNext, replace a direct access of '->nr' with the wrapper function\n'pack_pos_to_index()'.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/pack-objects.c | 11 +++++++----\n 1 file changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 2a00358f34..03d25db442 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -419,7 +419,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n {\n \tstruct packed_git *p = IN_PACK(entry);\n \tstruct pack_window *w_curs = NULL;\n-\tstruct revindex_entry *revidx;\n+\tuint32_t pos;\n \toff_t offset;\n \tenum object_type type = oe_type(entry);\n \toff_t datalen;\n@@ -436,10 +436,13 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\t\t\t      type, entry_size);\n \n \toffset = entry->in_pack_offset;\n-\trevidx = find_pack_revindex(p, offset);\n-\tdatalen = revidx[1].offset - offset;\n+\tif (offset_to_pack_pos(p, offset, &pos) < 0)\n+\t\tdie(_(\"write_reuse_object: could not locate %s\"),\n+\t\t    oid_to_hex(&entry->idx.oid));\n+\tdatalen = pack_pos_to_offset(p, pos + 1) - offset;\n \tif (!pack_to_stdout && p->index_version > 1 &&\n-\t    check_pack_crc(p, &w_curs, offset, datalen, revidx->nr)) {\n+\t    check_pack_crc(p, &w_curs, offset, datalen,\n+\t\t\t   pack_pos_to_index(p, pos))) {\n \t\terror(_(\"bad packed object CRC for %s\"),\n \t\t      oid_to_hex(&entry->idx.oid));\n \t\tunuse_pack(&w_curs);\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413836","messageId":"81ab11e18c0b00030019f9f521216f3469fdd744.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 03/20] write_reused_pack_one(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:16:53Z","receivedAt":"2021-01-08T18:17:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Replace direct revindex accesses with calls to 'pack_pos_to_offset()'\nand 'pack_pos_to_index()'.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/pack-objects.c | 12 ++++++++----\n 1 file changed, 8 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 03d25db442..ea7df9270f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -866,8 +866,8 @@ static void write_reused_pack_one(size_t pos, struct hashfile *out,\n \tenum object_type type;\n \tunsigned long size;\n \n-\toffset = reuse_packfile->revindex[pos].offset;\n-\tnext = reuse_packfile->revindex[pos + 1].offset;\n+\toffset = pack_pos_to_offset(reuse_packfile, pos);\n+\tnext = pack_pos_to_offset(reuse_packfile, pos + 1);\n \n \trecord_reused_object(offset, offset - hashfile_total(out));\n \n@@ -887,11 +887,15 @@ static void write_reused_pack_one(size_t pos, struct hashfile *out,\n \n \t\t/* Convert to REF_DELTA if we must... */\n \t\tif (!allow_ofs_delta) {\n-\t\t\tint base_pos = find_revindex_position(reuse_packfile, base_offset);\n+\t\t\tuint32_t base_pos;\n \t\t\tstruct object_id base_oid;\n \n+\t\t\tif (offset_to_pack_pos(reuse_packfile, base_offset, &base_pos) < 0)\n+\t\t\t\tdie(_(\"expected object at offset %\"PRIuMAX),\n+\t\t\t\t    (uintmax_t)base_offset);\n+\n \t\t\tnth_packed_object_id(&base_oid, reuse_packfile,\n-\t\t\t\t\t     reuse_packfile->revindex[base_pos].nr);\n+\t\t\t\t\t     pack_pos_to_index(reuse_packfile, base_pos));\n \n \t\t\tlen = encode_in_pack_object_header(header, sizeof(header),\n \t\t\t\t\t\t\t   OBJ_REF_DELTA, size);\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413835","messageId":"14b35d01a062f2dfdd710718b659064042dc21d6.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 04/20] write_reused_pack_verbatim(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:16:56Z","receivedAt":"2021-01-08T18:17:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Replace a direct access to the revindex array with\n'pack_pos_to_offset()'.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/pack-objects.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ea7df9270f..4341bc27b4 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -948,7 +948,7 @@ static size_t write_reused_pack_verbatim(struct hashfile *out,\n \t\toff_t to_write;\n \n \t\twritten = (pos * BITS_IN_EWORD);\n-\t\tto_write = reuse_packfile->revindex[written].offset\n+\t\tto_write = pack_pos_to_offset(reuse_packfile, written)\n \t\t\t- sizeof(struct pack_header);\n \n \t\t/* We're recording one chunk, not one object. */\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413837","messageId":"3b170663dd57f33678cd24270d65153772043670.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 06/20] bitmap_position_packfile(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:17:05Z","receivedAt":"2021-01-08T18:18:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Replace find_revindex_position() with its counterpart in the new API,\noffset_to_pack_pos().\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d88745fb02..d6861ddd4d 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -407,11 +407,14 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n static inline int bitmap_position_packfile(struct bitmap_index *bitmap_git,\n \t\t\t\t\t   const struct object_id *oid)\n {\n+\tuint32_t pos;\n \toff_t offset = find_pack_entry_one(oid->hash, bitmap_git->pack);\n \tif (!offset)\n \t\treturn -1;\n \n-\treturn find_revindex_position(bitmap_git->pack, offset);\n+\tif (offset_to_pack_pos(bitmap_git->pack, offset, &pos) < 0)\n+\t\treturn -1;\n+\treturn pos;\n }\n \n static int bitmap_position(struct bitmap_index *bitmap_git,\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413838","messageId":"541fe679f3e8e02617c4c9461b85fd310f55ccb6.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 08/20] get_size_by_pos(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:17:13Z","receivedAt":"2021-01-08T18:18:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Remove another caller that holds onto a 'struct revindex_entry' by\nreplacing the direct indexing with calls to 'pack_pos_to_offset()' and\n'pack_pos_to_index()'.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 80c57bde73..422505d7af 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -835,11 +835,11 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \toi.sizep = &size;\n \n \tif (pos < pack->num_objects) {\n-\t\tstruct revindex_entry *entry = &pack->revindex[pos];\n-\t\tif (packed_object_info(the_repository, pack,\n-\t\t\t\t       entry->offset, &oi) < 0) {\n+\t\toff_t ofs = pack_pos_to_offset(pack, pos);\n+\t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n-\t\t\tnth_packed_object_id(&oid, pack, entry->nr);\n+\t\t\tnth_packed_object_id(&oid, pack,\n+\t\t\t\t\t     pack_pos_to_index(pack, pos));\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&oid));\n \t\t}\n \t} else {\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413839","messageId":"bc67bb462ae0c87b34e46568d54b170a8aec870b.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 07/20] show_objects_for_type(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:17:09Z","receivedAt":"2021-01-08T18:18:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Avoid storing the revindex entry directly, since this structure will\nsoon be removed from the public interface. Instead, store the offset and\nindex position by calling 'pack_pos_to_offset()' and\n'pack_pos_to_index()', respectively.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 13 +++++++------\n 1 file changed, 7 insertions(+), 6 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d6861ddd4d..80c57bde73 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -711,21 +711,22 @@ static void show_objects_for_type(\n \n \t\tfor (offset = 0; offset < BITS_IN_EWORD; ++offset) {\n \t\t\tstruct object_id oid;\n-\t\t\tstruct revindex_entry *entry;\n-\t\t\tuint32_t hash = 0;\n+\t\t\tuint32_t hash = 0, n;\n+\t\t\toff_t ofs;\n \n \t\t\tif ((word >> offset) == 0)\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n \n-\t\t\tentry = &bitmap_git->pack->revindex[pos + offset];\n-\t\t\tnth_packed_object_id(&oid, bitmap_git->pack, entry->nr);\n+\t\t\tn = pack_pos_to_index(bitmap_git->pack, pos + offset);\n+\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n+\t\t\tnth_packed_object_id(&oid, bitmap_git->pack, n);\n \n \t\t\tif (bitmap_git->hashes)\n-\t\t\t\thash = get_be32(bitmap_git->hashes + entry->nr);\n+\t\t\t\thash = get_be32(bitmap_git->hashes + n);\n \n-\t\t\tshow_reach(&oid, object_type, 0, hash, bitmap_git->pack, entry->offset);\n+\t\t\tshow_reach(&oid, object_type, 0, hash, bitmap_git->pack, ofs);\n \t\t}\n \t}\n }\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413840","messageId":"c47e77a30eb40d9841a60a28b620671860dc2461.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 05/20] check_object(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:17:00Z","receivedAt":"2021-01-08T18:18:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Replace direct accesses to the revindex with calls to\n'offset_to_pack_pos()' and 'pack_pos_to_index()'.\n\nSince this caller already had some error checking (it can jump to the\n'give_up' label if it encounters an error), we can easily check whether\nor not the provided offset points to an object in the given pack. This\nerror checking existed prior to this patch, too, since the caller checks\nwhether the return value from 'find_pack_revindex()' was NULL or not.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/pack-objects.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 4341bc27b4..a193cdaf2f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1813,11 +1813,11 @@ static void check_object(struct object_entry *entry, uint32_t object_index)\n \t\t\t\tgoto give_up;\n \t\t\t}\n \t\t\tif (reuse_delta && !entry->preferred_base) {\n-\t\t\t\tstruct revindex_entry *revidx;\n-\t\t\t\trevidx = find_pack_revindex(p, ofs);\n-\t\t\t\tif (!revidx)\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (offset_to_pack_pos(p, ofs, &pos) < 0)\n \t\t\t\t\tgoto give_up;\n-\t\t\t\tif (!nth_packed_object_id(&base_ref, p, revidx->nr))\n+\t\t\t\tif (!nth_packed_object_id(&base_ref, p,\n+\t\t\t\t\t\t\t  pack_pos_to_index(p, pos)))\n \t\t\t\t\thave_base = 1;\n \t\t\t}\n \t\t\tentry->in_pack_header_size = used + used_0;\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413841","messageId":"e00c434ab283698be638022bacab4c0ee248a0a8.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 11/20] get_delta_base_oid(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:17:25Z","receivedAt":"2021-01-08T18:18:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Replace direct accesses to the 'struct revindex' type with a call to\n'pack_pos_to_index()'.\n\nLikewise drop the old-style 'find_pack_revindex()' with its replacement\n'offset_to_pack_pos()' (while continuing to perform the same error\nchecking).\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n packfile.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 86f5c8dbf6..3e3f391949 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1235,18 +1235,18 @@ static int get_delta_base_oid(struct packed_git *p,\n \t\toidread(oid, base);\n \t\treturn 0;\n \t} else if (type == OBJ_OFS_DELTA) {\n-\t\tstruct revindex_entry *revidx;\n+\t\tuint32_t base_pos;\n \t\toff_t base_offset = get_delta_base(p, w_curs, &curpos,\n \t\t\t\t\t\t   type, delta_obj_offset);\n \n \t\tif (!base_offset)\n \t\t\treturn -1;\n \n-\t\trevidx = find_pack_revindex(p, base_offset);\n-\t\tif (!revidx)\n+\t\tif (offset_to_pack_pos(p, base_offset, &base_pos) < 0)\n \t\t\treturn -1;\n \n-\t\treturn nth_packed_object_id(oid, p, revidx->nr);\n+\t\treturn nth_packed_object_id(oid, p,\n+\t\t\t\t\t    pack_pos_to_index(p, base_pos));\n \t} else\n \t\treturn -1;\n }\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413842","messageId":"aae01d70293545e32c317454748ef72480331d49.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 12/20] retry_bad_packed_offset(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:17:30Z","receivedAt":"2021-01-08T18:18:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Perform exactly the same conversion as in the previous commit to another\ncaller within 'packfile.c'.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n packfile.c | 7 +++----\n 1 file changed, 3 insertions(+), 4 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 3e3f391949..7c37f9ec5c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1256,12 +1256,11 @@ static int retry_bad_packed_offset(struct repository *r,\n \t\t\t\t   off_t obj_offset)\n {\n \tint type;\n-\tstruct revindex_entry *revidx;\n+\tuint32_t pos;\n \tstruct object_id oid;\n-\trevidx = find_pack_revindex(p, obj_offset);\n-\tif (!revidx)\n+\tif (offset_to_pack_pos(p, obj_offset, &pos) < 0)\n \t\treturn OBJ_BAD;\n-\tnth_packed_object_id(&oid, p, revidx->nr);\n+\tnth_packed_object_id(&oid, p, pack_pos_to_index(p, pos));\n \tmark_bad_packed_object(p, oid.hash);\n \ttype = oid_object_info(r, &oid, NULL);\n \tif (type <= OBJ_NONE)\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413843","messageId":"7c17db7a7df8b524f13969efd1cb5e6e95de5a2d.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 16/20] builtin/gc.c: guess the size of the revindex","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:17:47Z","receivedAt":"2021-01-08T18:18:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"'estimate_repack_memory()' takes into account the amount of memory\nrequired to load the reverse index in memory by multiplying the assumed\nnumber of objects by the size of the 'revindex_entry' struct.\n\nPrepare for hiding the definition of 'struct revindex_entry' by removing\na 'sizeof()' of that type from outside of pack-revindex.c. Instead,\nguess that one off_t and one uint32_t are required per object. Strictly\nspeaking, this is a worse guess than asking for 'sizeof(struct\nrevindex_entry)' directly, since the true size of this struct is 16\nbytes with padding on the end of the struct in order to align the offset\nfield.\n\nBut, this is an approximation anyway, and it does remove a use of the\n'struct revindex_entry' from outside of pack-revindex internals.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/gc.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 4c24f41852..c60811f212 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -301,7 +301,7 @@ static uint64_t estimate_repack_memory(struct packed_git *pack)\n \t/* and then obj_hash[], underestimated in fact */\n \theap += sizeof(struct object *) * nr_objects;\n \t/* revindex is used also */\n-\theap += sizeof(struct revindex_entry) * nr_objects;\n+\theap += (sizeof(off_t) + sizeof(uint32_t)) * nr_objects;\n \t/*\n \t * read_sha1_file() (either at delta calculation phase, or\n \t * writing phase) also fills up the delta base cache\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413850","messageId":"a3249986f9ab935825bc37e1bf980e44532700ae.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 15/20] for_each_object_in_pack(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:17:43Z","receivedAt":"2021-01-08T18:18:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Avoid looking at the 'revindex' pointer directly and instead call\n'pack_pos_to_index()'.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n packfile.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 34467ea4a3..46c9c7ea3c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2082,7 +2082,7 @@ int for_each_object_in_pack(struct packed_git *p,\n \t\tstruct object_id oid;\n \n \t\tif (flags & FOR_EACH_OBJECT_PACK_ORDER)\n-\t\t\tpos = p->revindex[i].nr;\n+\t\t\tpos = pack_pos_to_index(p, i);\n \t\telse\n \t\t\tpos = i;\n \n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413844","messageId":"97eaa7b2d6d923da0dca6e96b890d4b85bf9d7d8.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 10/20] rebuild_existing_bitmaps(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:17:21Z","receivedAt":"2021-01-08T18:18:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Remove another instance of looking at the revindex directly by instead\ncalling 'pack_pos_to_index()'. Unlike other patches, this caller only\ncares about the index position of each object in the loop.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 5 ++---\n 1 file changed, 2 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 3f5cd4e77d..24b9628dca 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1392,11 +1392,10 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \n \tfor (i = 0; i < num_objects; ++i) {\n \t\tstruct object_id oid;\n-\t\tstruct revindex_entry *entry;\n \t\tstruct object_entry *oe;\n \n-\t\tentry = &bitmap_git->pack->revindex[i];\n-\t\tnth_packed_object_id(&oid, bitmap_git->pack, entry->nr);\n+\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n+\t\t\t\t     pack_pos_to_index(bitmap_git->pack, i));\n \t\toe = packlist_find(mapping, &oid);\n \n \t\tif (oe)\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413845","messageId":"54f4ad329f56808432549aa885f2847d5c9a8ac6.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 09/20] try_partial_reuse(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:17:17Z","receivedAt":"2021-01-08T18:18:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Remove another instance of direct revindex manipulation by calling\n'pack_pos_to_offset()' instead (the caller here does not care about the\nindex position of the object at position 'pos').\n\nSomewhat confusingly, the subsequent call to unpack_object_header()\ntakes a pointer to &offset and then updates it with a new value. But,\ntry_partial_reuse() cares about the offset of both the base's header and\ncontents. The existing code made a copy of the offset field, and only\naddresses and manipulates one of them.\n\nInstead, store the return of pack_pos_to_offset twice: once in header\nand another in offset. Header will be left untouched, but offset will be\naddressed and modified by unpack_object_header().\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 13 +++++--------\n 1 file changed, 5 insertions(+), 8 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 422505d7af..3f5cd4e77d 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1069,23 +1069,21 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t\t      struct bitmap *reuse,\n \t\t\t      struct pack_window **w_curs)\n {\n-\tstruct revindex_entry *revidx;\n-\toff_t offset;\n+\toff_t offset, header;\n \tenum object_type type;\n \tunsigned long size;\n \n \tif (pos >= bitmap_git->pack->num_objects)\n \t\treturn; /* not actually in the pack */\n \n-\trevidx = &bitmap_git->pack->revindex[pos];\n-\toffset = revidx->offset;\n+\toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n \ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n \tif (type < 0)\n \t\treturn; /* broken packfile, punt */\n \n \tif (type == OBJ_REF_DELTA || type == OBJ_OFS_DELTA) {\n \t\toff_t base_offset;\n-\t\tint base_pos;\n+\t\tuint32_t base_pos;\n \n \t\t/*\n \t\t * Find the position of the base object so we can look it up\n@@ -1096,11 +1094,10 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * more detail.\n \t\t */\n \t\tbase_offset = get_delta_base(bitmap_git->pack, w_curs,\n-\t\t\t\t\t     &offset, type, revidx->offset);\n+\t\t\t\t\t     &offset, type, header);\n \t\tif (!base_offset)\n \t\t\treturn;\n-\t\tbase_pos = find_revindex_position(bitmap_git->pack, base_offset);\n-\t\tif (base_pos < 0)\n+\t\tif (offset_to_pack_pos(bitmap_git->pack, base_offset, &base_pos) < 0)\n \t\t\treturn;\n \n \t\t/*\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413846","messageId":"eab7ab1f35fa9703f56a99fa539839869fe4e54c.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 13/20] packed_object_info(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:17:34Z","receivedAt":"2021-01-08T18:18:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Convert another call of 'find_pack_revindex()' to its replacement\n'pack_pos_to_offset()'. Likewise:\n\n  - Avoid manipulating `struct packed_git`'s `revindex` pointer directly\n    by removing the pointer-as-array indexing.\n\n  - Add an additional guard to check that the offset 'obj_offset()'\n    points to a real object. This should be the case with well-behaved\n    callers to 'packed_object_info()', but isn't guarenteed.\n\n    Other blocks that fill in various other values from the 'struct\n    object_info' request handle bad inputs by setting the type to\n    'OBJ_BAD' and jumping to 'out'. Do the same when given a bad offset\n    here.\n\n    The previous code would have segfaulted when given a bad\n    'obj_offset' value, since 'find_pack_revindex()' would return\n    'NULL', and then the line that fills 'oi->disk_sizep' would try to\n    access 'NULL[1]' with a stride of 16 bytes (the width of 'struct\n    revindex_entry)'.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n packfile.c | 9 +++++++--\n 1 file changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 7c37f9ec5c..469c8d4f57 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1537,8 +1537,13 @@ int packed_object_info(struct repository *r, struct packed_git *p,\n \t}\n \n \tif (oi->disk_sizep) {\n-\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n-\t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n+\t\tuint32_t pos;\n+\t\tif (offset_to_pack_pos(p, obj_offset, &pos) < 0) {\n+\t\t\ttype = OBJ_BAD;\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\t*oi->disk_sizep = pack_pos_to_offset(p, pos + 1) - obj_offset;\n \t}\n \n \tif (oi->typep || oi->type_name) {\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413847","messageId":"13c49ed40ca72b7ab50939244616f0a90b5bf7f6.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 14/20] unpack_entry(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:17:39Z","receivedAt":"2021-01-08T18:18:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Remove direct manipulation of the 'struct revindex_entry' type as well\nas calls to the deprecated API in 'packfile.c:unpack_entry()'. Usual\nclean-up is performed (replacing '->nr' with calls to\n'pack_pos_to_index()' and so on). Add an additional check to make\nsure that 'obj_offset()' points at a valid object.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n packfile.c | 24 ++++++++++++++++--------\n 1 file changed, 16 insertions(+), 8 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 469c8d4f57..34467ea4a3 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1692,11 +1692,19 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,\n \t\t}\n \n \t\tif (do_check_packed_object_crc && p->index_version > 1) {\n-\t\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n-\t\t\toff_t len = revidx[1].offset - obj_offset;\n-\t\t\tif (check_pack_crc(p, &w_curs, obj_offset, len, revidx->nr)) {\n+\t\t\tuint32_t pos, nr;\n+\t\t\toff_t len;\n+\n+\t\t\tif (offset_to_pack_pos(p, obj_offset, &pos) < 0) {\n+\t\t\t\tdata = NULL;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\tlen = pack_pos_to_offset(p, pos + 1) - obj_offset;\n+\t\t\tnr = pack_pos_to_index(p, pos);\n+\t\t\tif (check_pack_crc(p, &w_curs, obj_offset, len, nr)) {\n \t\t\t\tstruct object_id oid;\n-\t\t\t\tnth_packed_object_id(&oid, p, revidx->nr);\n+\t\t\t\tnth_packed_object_id(&oid, p, nr);\n \t\t\t\terror(\"bad packed object CRC for %s\",\n \t\t\t\t      oid_to_hex(&oid));\n \t\t\t\tmark_bad_packed_object(p, oid.hash);\n@@ -1779,11 +1787,11 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,\n \t\t\t * This is costly but should happen only in the presence\n \t\t\t * of a corrupted pack, and is better than failing outright.\n \t\t\t */\n-\t\t\tstruct revindex_entry *revidx;\n+\t\t\tuint32_t pos;\n \t\t\tstruct object_id base_oid;\n-\t\t\trevidx = find_pack_revindex(p, obj_offset);\n-\t\t\tif (revidx) {\n-\t\t\t\tnth_packed_object_id(&base_oid, p, revidx->nr);\n+\t\t\tif (!(offset_to_pack_pos(p, obj_offset, &pos))) {\n+\t\t\t\tnth_packed_object_id(&base_oid, p,\n+\t\t\t\t\t\t     pack_pos_to_index(p, pos));\n \t\t\t\terror(\"failed to read delta base object %s\"\n \t\t\t\t      \" at offset %\"PRIuMAX\" from %s\",\n \t\t\t\t      oid_to_hex(&base_oid), (uintmax_t)obj_offset,\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413848","messageId":"d60411d524656f4680ac578765b2a8704325a060.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 18/20] pack-revindex: remove unused 'find_revindex_position()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:17:57Z","receivedAt":"2021-01-08T18:18:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that all 'find_revindex_position()' callers have been removed (and\nconverted to the more descriptive 'offset_to_pack_pos()'), it is almost\nsafe to get rid of 'find_revindex_position()' entirely. Almost, except\nfor the fact that 'offset_to_pack_pos()' calls\n'find_revindex_position()'.\n\nInline 'find_revindex_position()' into 'offset_to_pack_pos()', and\nthen remove 'find_revindex_position()' entirely.\n\nThis is a straightforward refactoring with one minor snag.\n'offset_to_pack_pos()' used to load the index before calling\n'find_revindex_position()'. That means that by the time\n'find_revindex_position()' starts executing, 'p->num_objects' can be\nsafely read. After inlining, be careful to not read 'p->num_objects'\nuntil _after_ 'load_pack_revindex()' (which loads the index as a\nside-effect) has been called.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-revindex.c | 29 +++++++++++------------------\n pack-revindex.h |  1 -\n 2 files changed, 11 insertions(+), 19 deletions(-)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 4e42238906..9392c4be73 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -169,16 +169,23 @@ int load_pack_revindex(struct packed_git *p)\n \treturn 0;\n }\n \n-int find_revindex_position(struct packed_git *p, off_t ofs)\n+int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n {\n \tint lo = 0;\n-\tint hi = p->num_objects + 1;\n-\tconst struct revindex_entry *revindex = p->revindex;\n+\tint hi;\n+\tconst struct revindex_entry *revindex;\n+\n+\tif (load_pack_revindex(p) < 0)\n+\t\treturn -1;\n+\n+\thi = p->num_objects + 1;\n+\trevindex = p->revindex;\n \n \tdo {\n \t\tconst unsigned mi = lo + (hi - lo) / 2;\n \t\tif (revindex[mi].offset == ofs) {\n-\t\t\treturn mi;\n+\t\t\t*pos = mi;\n+\t\t\treturn 0;\n \t\t} else if (ofs < revindex[mi].offset)\n \t\t\thi = mi;\n \t\telse\n@@ -189,20 +196,6 @@ int find_revindex_position(struct packed_git *p, off_t ofs)\n \treturn -1;\n }\n \n-int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n-{\n-\tint ret;\n-\n-\tif (load_pack_revindex(p) < 0)\n-\t\treturn -1;\n-\n-\tret = find_revindex_position(p, ofs);\n-\tif (ret < 0)\n-\t\treturn -1;\n-\t*pos = ret;\n-\treturn 0;\n-}\n-\n uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos)\n {\n \tif (!p->revindex)\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex 07c1a7a3c8..b5dd114fd5 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -9,7 +9,6 @@ struct revindex_entry {\n };\n \n int load_pack_revindex(struct packed_git *p);\n-int find_revindex_position(struct packed_git *p, off_t ofs);\n \n int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos);\n uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413849","messageId":"7c0e4acc845d1135e684188b2ccc61cf358994dc.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 19/20] pack-revindex: hide the definition of 'revindex_entry'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:18:01Z","receivedAt":"2021-01-08T18:18:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that all spots outside of pack-revindex.c that reference 'struct\nrevindex_entry' directly have been removed, it is safe to hide the\nimplementation by moving it from pack-revindex.h to pack-revindex.c.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-revindex.c | 5 +++++\n pack-revindex.h | 5 -----\n 2 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 9392c4be73..36ef276378 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -3,6 +3,11 @@\n #include \"object-store.h\"\n #include \"packfile.h\"\n \n+struct revindex_entry {\n+\toff_t offset;\n+\tunsigned int nr;\n+};\n+\n /*\n  * Pack index for existing packs give us easy access to the offsets into\n  * corresponding pack file where each object's data starts, but the entries\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex b5dd114fd5..b501a7cd62 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -3,11 +3,6 @@\n \n struct packed_git;\n \n-struct revindex_entry {\n-\toff_t offset;\n-\tunsigned int nr;\n-};\n-\n int load_pack_revindex(struct packed_git *p);\n \n int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos);\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413851","messageId":"c4c88bcc3da6c0a7e28b82e0558a44195403fe25.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 17/20] pack-revindex: remove unused 'find_pack_revindex()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:17:52Z","receivedAt":"2021-01-08T18:19:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that no callers of 'find_pack_revindex()' remain, remove the\nfunction's declaration and implementation entirely.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-revindex.c | 15 ---------------\n pack-revindex.h |  2 --\n 2 files changed, 17 deletions(-)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 6d86a85208..4e42238906 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -189,21 +189,6 @@ int find_revindex_position(struct packed_git *p, off_t ofs)\n \treturn -1;\n }\n \n-struct revindex_entry *find_pack_revindex(struct packed_git *p, off_t ofs)\n-{\n-\tint pos;\n-\n-\tif (load_pack_revindex(p))\n-\t\treturn NULL;\n-\n-\tpos = find_revindex_position(p, ofs);\n-\n-\tif (pos < 0)\n-\t\treturn NULL;\n-\n-\treturn p->revindex + pos;\n-}\n-\n int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n {\n \tint ret;\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex 256c0a9106..07c1a7a3c8 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -11,8 +11,6 @@ struct revindex_entry {\n int load_pack_revindex(struct packed_git *p);\n int find_revindex_position(struct packed_git *p, off_t ofs);\n \n-struct revindex_entry *find_pack_revindex(struct packed_git *p, off_t ofs);\n-\n int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos);\n uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);\n off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos);\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"413852","messageId":"eada1ffcfafc3fb57de80626e368672cb8b22318.1610129796.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH 20/20] pack-revindex.c: avoid direct revindex access in 'offset_to_pack_pos()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-08T18:18:06Z","receivedAt":"2021-01-08T18:19:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To prepare for on-disk reverse indexes, remove a spot in\n'offset_to_pack_pos()' that looks at the 'revindex' array in 'struct\npacked_git'.\n\nEven though this use of the revindex pointer is within pack-revindex.c,\nthis clean up is still worth doing. Since the 'revindex' pointer will be\nNULL when reading from an on-disk reverse index (instead the\n'revindex_data' pointer will be mmaped to the 'pack-*.rev' file), this\ncall-site would have to include a conditional to lookup the offset for\nposition 'mi' each iteration through the search.\n\nSo instead of open-coding 'pack_pos_to_offset()', call it directly from\nwithin 'offset_to_pack_pos()'.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-revindex.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 36ef276378..2cd9d632f1 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -178,20 +178,20 @@ int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n {\n \tint lo = 0;\n \tint hi;\n-\tconst struct revindex_entry *revindex;\n \n \tif (load_pack_revindex(p) < 0)\n \t\treturn -1;\n \n \thi = p->num_objects + 1;\n-\trevindex = p->revindex;\n \n \tdo {\n \t\tconst unsigned mi = lo + (hi - lo) / 2;\n-\t\tif (revindex[mi].offset == ofs) {\n+\t\toff_t got = pack_pos_to_offset(p, mi);\n+\n+\t\tif (got == ofs) {\n \t\t\t*pos = mi;\n \t\t\treturn 0;\n-\t\t} else if (ofs < revindex[mi].offset)\n+\t\t} else if (ofs < got)\n \t\t\thi = mi;\n \t\telse\n \t\t\tlo = mi + 1;\n-- \n2.30.0.138.g6d7191ea01\n"},{"id":"414036","messageId":"b1a6110a-a097-931f-5710-92a1f59a842b@gmail.com","threadId":"54961","inReplyTo":"c47e77a30eb40d9841a60a28b620671860dc2461.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 05/20] check_object(): convert to new revindex API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-01-11T11:43:23Z","receivedAt":"2021-01-11T11:44:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/8/2021 1:17 PM, Taylor Blau wrote:\n> Replace direct accesses to the revindex with calls to\n> 'offset_to_pack_pos()' and 'pack_pos_to_index()'.\n> \n> Since this caller already had some error checking (it can jump to the\n> 'give_up' label if it encounters an error), we can easily check whether\n> or not the provided offset points to an object in the given pack. This\n> error checking existed prior to this patch, too, since the caller checks\n> whether the return value from 'find_pack_revindex()' was NULL or not.\n> \n> Signed-off-by: Taylor Blau <me@ttaylorr.com>\n> ---\n>  builtin/pack-objects.c | 8 ++++----\n>  1 file changed, 4 insertions(+), 4 deletions(-)\n> \n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 4341bc27b4..a193cdaf2f 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -1813,11 +1813,11 @@ static void check_object(struct object_entry *entry, uint32_t object_index)\n>  \t\t\t\tgoto give_up;\n>  \t\t\t}\n>  \t\t\tif (reuse_delta && !entry->preferred_base) {\n> -\t\t\t\tstruct revindex_entry *revidx;\n> -\t\t\t\trevidx = find_pack_revindex(p, ofs);\n> -\t\t\t\tif (!revidx)\n> +\t\t\t\tuint32_t pos;\n> +\t\t\t\tif (offset_to_pack_pos(p, ofs, &pos) < 0)\n\nThe current implementation does not return a positive value. Only\n-1 on error and 0 on success. Is this \"< 0\" doing anything important?\nSeems like it would be easiest to do\n\n\tif (offset_to_pack_pos(p, ofs, &pos))\n\n>  \t\t\t\t\tgoto give_up;\n> -\t\t\t\tif (!nth_packed_object_id(&base_ref, p, revidx->nr))\n> +\t\t\t\tif (!nth_packed_object_id(&base_ref, p,\n> +\t\t\t\t\t\t\t  pack_pos_to_index(p, pos)))\n>  \t\t\t\t\thave_base = 1;\n>  \t\t\t}\n\nThanks,\n-Stolee\n"},{"id":"414037","messageId":"87cd1b2c-7a28-da77-4ae4-99ffbbdfda72@gmail.com","threadId":"54961","inReplyTo":"7c17db7a7df8b524f13969efd1cb5e6e95de5a2d.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 16/20] builtin/gc.c: guess the size of the revindex","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-01-11T11:52:24Z","receivedAt":"2021-01-11T11:54:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/8/2021 1:17 PM, Taylor Blau wrote:\n> 'estimate_repack_memory()' takes into account the amount of memory\n> required to load the reverse index in memory by multiplying the assumed\n> number of objects by the size of the 'revindex_entry' struct.\n> \n> Prepare for hiding the definition of 'struct revindex_entry' by removing\n> a 'sizeof()' of that type from outside of pack-revindex.c. Instead,\n> guess that one off_t and one uint32_t are required per object. Strictly\n> speaking, this is a worse guess than asking for 'sizeof(struct\n> revindex_entry)' directly, since the true size of this struct is 16\n> bytes with padding on the end of the struct in order to align the offset\n> field.\n\nThis is so far the only not-completely-obvious change.\n \n> But, this is an approximation anyway, and it does remove a use of the\n> 'struct revindex_entry' from outside of pack-revindex internals.\n\nAnd this might be enough justification for it, but...\n\n> -\theap += sizeof(struct revindex_entry) * nr_objects;\n> +\theap += (sizeof(off_t) + sizeof(uint32_t)) * nr_objects;\n...outside of the estimation change, will this need another change\nwhen the rev-index is mmap'd? Should this instead be an API call,\nsuch as estimate_rev_index_memory(nr_objects)? That would\ncentralize the estimate to be next to the code that currently\ninteracts with 'struct revindex_entry' and will later interact with\nthe mmap region.\n\nThanks,\n-Stolee\n"},{"id":"414038","messageId":"624d0642-b6c9-7c76-aeb6-d7e18b0aad1f@gmail.com","threadId":"54961","inReplyTo":"d60411d524656f4680ac578765b2a8704325a060.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 18/20] pack-revindex: remove unused 'find_revindex_position()'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-01-11T11:57:00Z","receivedAt":"2021-01-11T11:57:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/8/2021 1:17 PM, Taylor Blau wrote:\n> -int find_revindex_position(struct packed_git *p, off_t ofs)\n> +int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n>  {\n>  \tint lo = 0;\n> -\tint hi = p->num_objects + 1;\n> -\tconst struct revindex_entry *revindex = p->revindex;\n> +\tint hi;\n> +\tconst struct revindex_entry *revindex;\n> +\n> +\tif (load_pack_revindex(p) < 0)\n> +\t\treturn -1;\n> +\n> +\thi = p->num_objects + 1;\n> +\trevindex = p->revindex;\n>  \n>  \tdo {\n>  \t\tconst unsigned mi = lo + (hi - lo) / 2;\n>  \t\tif (revindex[mi].offset == ofs) {\n> -\t\t\treturn mi;\n> +\t\t\t*pos = mi;\n> +\t\t\treturn 0;\n>  \t\t} else if (ofs < revindex[mi].offset)\n>  \t\t\thi = mi;\n>  \t\telse\n> @@ -189,20 +196,6 @@ int find_revindex_position(struct packed_git *p, off_t ofs)\n>  \treturn -1;\n>  }\n>  \n> -int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n> -{\n> -\tint ret;\n> -\n> -\tif (load_pack_revindex(p) < 0)\n> -\t\treturn -1;\n> -\n> -\tret = find_revindex_position(p, ofs);\n> -\tif (ret < 0)\n> -\t\treturn -1;\n> -\t*pos = ret;\n> -\treturn 0;\n> -}\n> -\n\nNot that this is new to the current patch, but this patch made me\nwonder if we should initialize *pos = -1 in the case of a failure\nto find the position? A correct caller should not use the value\nif they are checking for the fail-to-find case properly. But, I\ncould see someone making a mistake and having trouble diagnosing\nthe problem because their position variable was initialized to\nzero or a previous successful case.\n\nThanks,\n-Stolee\n"},{"id":"414039","messageId":"8b3a59c5-5cb7-cd0d-2658-b1e7cac236c3@gmail.com","threadId":"54961","inReplyTo":"7c0e4acc845d1135e684188b2ccc61cf358994dc.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 19/20] pack-revindex: hide the definition of 'revindex_entry'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-01-11T11:57:47Z","receivedAt":"2021-01-11T11:58:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/8/2021 1:18 PM, Taylor Blau wrote:\n> diff --git a/pack-revindex.h b/pack-revindex.h\n> index b5dd114fd5..b501a7cd62 100644\n> --- a/pack-revindex.h\n> +++ b/pack-revindex.h\n> @@ -3,11 +3,6 @@\n>  \n>  struct packed_git;\n>  \n> -struct revindex_entry {\n> -\toff_t offset;\n> -\tunsigned int nr;\n> -};\n> -\n\nVery nice. A successful anonymization of a struct.\n\n-Stolee\n"},{"id":"414040","messageId":"75ba9979-1a1f-de9f-c2cc-1433d30ed09d@gmail.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-01-11T12:07:17Z","receivedAt":"2021-01-11T12:08:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/8/2021 1:16 PM, Taylor Blau wrote:\n...\n>   - First, a new API is proposed.\n> \n>   - Then, uses of the old API are removed one by one and replaced with their\n>     new counterparts.\n> \n>   - Finally, without any callers remaining, the old API is removed.\n\nThis patch series has a clear layout that was easy to follow.\n\nIn a vacuum, the conversions from immediate struct member lookups\nto API calls seems like adding overhead to something that is done\nfrequently in a loop. However, you do justify it:\n\n> Generating the reverse index in memory for repositories with large packs has two\n> significant drawbacks:\n> \n>   - It requires allocating sizeof(struct revindex_entry) per packed object.\n> \n>   - It requires us to sort the entries by their pack offset. This is implemented\n>     in sort_revindex() using a radix sort, but still takes considerable time (as\n>     benchmarks found in the second series demonstrate).\n> \n> Both of these can be addressed by storing the reverse index in a new '.rev' file\n> alongside the packs. This file is written once (during pack creation), and does\n> not require sorting when accessed, since it is stored in a sorted order.\n\nEven if these method calls do add a bit of overhead to each\naccess, it helps to not compute the table from scratch before\nany access is possible.\n\nThis will be particularly valuable for operations that use only\na few position lookups, such as \"is object A reachable from\ncommit C?\"\n\nOperations that iterate through every object in a bitmap are\nmore likely to notice a difference, but that will probably be\nvisible in the next series.\n\nMy comments on this series are very minor.\n\nI made only one comment about \"if (method() < 0)\" versus\n\"if (method())\" but that pattern appears in multiple patches.\n_If_ you decide to change that pattern, then I'm sure you can\nfind all uses.\n\nReviewed-by: Derrick Stolee <dstolee@microsoft.com>\n\nThanks,\n-Stolee\n"},{"id":"414056","messageId":"X/x5tdbM3PzkqbFQ@nand.local","threadId":"54961","inReplyTo":"b1a6110a-a097-931f-5710-92a1f59a842b@gmail.com","subject":"Re: [PATCH 05/20] check_object(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-11T16:15:49Z","receivedAt":"2021-01-11T16:16:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jan 11, 2021 at 06:43:23AM -0500, Derrick Stolee wrote:\n> > @@ -1813,11 +1813,11 @@ static void check_object(struct object_entry *entry, uint32_t object_index)\n> >  \t\t\t\tgoto give_up;\n> >  \t\t\t}\n> >  \t\t\tif (reuse_delta && !entry->preferred_base) {\n> > -\t\t\t\tstruct revindex_entry *revidx;\n> > -\t\t\t\trevidx = find_pack_revindex(p, ofs);\n> > -\t\t\t\tif (!revidx)\n> > +\t\t\t\tuint32_t pos;\n> > +\t\t\t\tif (offset_to_pack_pos(p, ofs, &pos) < 0)\n>\n> The current implementation does not return a positive value. Only\n> -1 on error and 0 on success. Is this \"< 0\" doing anything important?\n> Seems like it would be easiest to do\n>\n> \tif (offset_to_pack_pos(p, ofs, &pos))\n>\n> [snip]\n\nEither would work, of course. I tend to find the '< 0' form easier to\nread, but I may be in the minority there. For me, the negative return\nvalue makes clear that the function encountered an error.\n\nA secondary benefit is that if the function ever were to return a\npositive value that _didn't_ indicate an error, we would already be\nprotected against it. That is probably a pretty weak argument, though,\nsince any such refactoring would probably require the callers to change,\ntoo.\n\nAnyway, that's all to say that I'm happy to leave it as-is, but I'm\nequally happy to change it, too.\n\nThanks,\nTaylor\n"},{"id":"414057","messageId":"X/x7mrcwfxGO8xH7@nand.local","threadId":"54961","inReplyTo":"87cd1b2c-7a28-da77-4ae4-99ffbbdfda72@gmail.com","subject":"Re: [PATCH 16/20] builtin/gc.c: guess the size of the revindex","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-11T16:23:54Z","receivedAt":"2021-01-11T16:24:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jan 11, 2021 at 06:52:24AM -0500, Derrick Stolee wrote:\n> This is so far the only not-completely-obvious change.\n>\n> > But, this is an approximation anyway, and it does remove a use of the\n> > 'struct revindex_entry' from outside of pack-revindex internals.\n>\n> And this might be enough justification for it, but...\n>\n> > -\theap += sizeof(struct revindex_entry) * nr_objects;\n> > +\theap += (sizeof(off_t) + sizeof(uint32_t)) * nr_objects;\n>\n> ...outside of the estimation change, will this need another change\n> when the rev-index is mmap'd? Should this instead be an API call,\n> such as estimate_rev_index_memory(nr_objects)? That would\n> centralize the estimate to be next to the code that currently\n> interacts with 'struct revindex_entry' and will later interact with\n> the mmap region.\n\nI definitely did consider this, and it seems that I made a mistake in\nnot documenting my consideration (since I assumed that it was so benign\nnobody would notice / care ;-)).\n\nThe reason I didn't pursue it here was that we haven't yet loaded the\nreverse index by this point. So, you'd want a function that at least\nstats the '*.rev' file (and either does or doesn't parse it [1]), or\naborts early to indicate otherwise.\n\nOne would hope that 'load_pack_revindex()' would do just that, but it\nfalls back to load a reverse index in memory, which involves exactly the\nslow sort that we're trying to avoid. (Of course, we're going to have to\ndo it later anyway, but allocating many GB of heap just to provide an\nestimation seems ill-advised to me ;-).)\n\nSo, we'd have to expand the API in some way or another, and to me it\ndidn't seem worth it. As I mentioned in the commit message, I'm\nskeptical of the value of being accurate here, since this is (after all)\nan estimation.\n\nPerhaps a longer response than you were bargaining for, but... :-).\n\nThanks,\nTaylor\n\n[1]: Likely negligible, since all \"parsing\" really does is verify the\ninternal checksum, and then assign a pointer into it.\n"},{"id":"414058","messageId":"X/x8dXFfQUdpKeVn@nand.local","threadId":"54961","inReplyTo":"624d0642-b6c9-7c76-aeb6-d7e18b0aad1f@gmail.com","subject":"Re: [PATCH 18/20] pack-revindex: remove unused 'find_revindex_position()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-11T16:27:33Z","receivedAt":"2021-01-11T16:28:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jan 11, 2021 at 06:57:00AM -0500, Derrick Stolee wrote:\n> On 1/8/2021 1:17 PM, Taylor Blau wrote:\n> > -int find_revindex_position(struct packed_git *p, off_t ofs)\n> > +int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n> >  {\n> >  \tint lo = 0;\n> > -\tint hi = p->num_objects + 1;\n> > -\tconst struct revindex_entry *revindex = p->revindex;\n> > +\tint hi;\n> > +\tconst struct revindex_entry *revindex;\n> > +\n> > +\tif (load_pack_revindex(p) < 0)\n> > +\t\treturn -1;\n> > +\n> > +\thi = p->num_objects + 1;\n> > +\trevindex = p->revindex;\n> >\n> >  \tdo {\n> >  \t\tconst unsigned mi = lo + (hi - lo) / 2;\n> >  \t\tif (revindex[mi].offset == ofs) {\n> > -\t\t\treturn mi;\n> > +\t\t\t*pos = mi;\n> > +\t\t\treturn 0;\n> >  \t\t} else if (ofs < revindex[mi].offset)\n> >  \t\t\thi = mi;\n> >  \t\telse\n> > @@ -189,20 +196,6 @@ int find_revindex_position(struct packed_git *p, off_t ofs)\n> >  \treturn -1;\n> >  }\n> >\n> > -int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n> > -{\n> > -\tint ret;\n> > -\n> > -\tif (load_pack_revindex(p) < 0)\n> > -\t\treturn -1;\n> > -\n> > -\tret = find_revindex_position(p, ofs);\n> > -\tif (ret < 0)\n> > -\t\treturn -1;\n> > -\t*pos = ret;\n> > -\treturn 0;\n> > -}\n> > -\n>\n> Not that this is new to the current patch, but this patch made me\n> wonder if we should initialize *pos = -1 in the case of a failure\n> to find the position? A correct caller should not use the value\n> if they are checking for the fail-to-find case properly. But, I\n> could see someone making a mistake and having trouble diagnosing\n> the problem because their position variable was initialized to\n> zero or a previous successful case.\n\n*pos = -1 may be more confusing than clarifying since pos is unsigned.\n\nIt would be nice if there was a clear signal beyond returning a negative\nvalue. I guess you could take a double pointer here which would allow\nyou to assign NULL, but that feels rather cumbersome as a means to catch\ncallers who failed to check the return value.\n\nIt does raise the argument of whether or not we should allow the\nprogram to continue at all if 'ret < 0' (i.e., 'offset_to_pack_pos()'\neither 'die()'s or returns a usable uint32_t), but I'm OK with the\ncurrent behavior.\n\n> Thanks,\n> -Stolee\n\nThanks,\nTaylor\n"},{"id":"414060","messageId":"X/x9MMv4hBZMGKBT@nand.local","threadId":"54961","inReplyTo":"75ba9979-1a1f-de9f-c2cc-1433d30ed09d@gmail.com","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-11T16:30:40Z","receivedAt":"2021-01-11T16:31:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jan 11, 2021 at 07:07:17AM -0500, Derrick Stolee wrote:\n> My comments on this series are very minor.\n>\n> I made only one comment about \"if (method() < 0)\" versus\n> \"if (method())\" but that pattern appears in multiple patches.\n> _If_ you decide to change that pattern, then I'm sure you can\n> find all uses.\n\nI have no strong opinion here, so I'm happy to defer to your or others'\njudgement. My very weak opinion is that I'd just as soon leave it as-is,\nbut that if I'm rerolling and others would like to see it changed, then\nI'm happy to do it.\n\n> Reviewed-by: Derrick Stolee <dstolee@microsoft.com>\n\nThank you for your review. I think that I owe you some as well,\nsomewhere in the near-2,000 emails that I still have :-/.\n\nThanks,\nTaylor\n"},{"id":"414061","messageId":"13c28eca-81d5-10c9-c92c-162547416014@gmail.com","threadId":"54961","inReplyTo":"X/x7mrcwfxGO8xH7@nand.local","subject":"Re: [PATCH 16/20] builtin/gc.c: guess the size of the revindex","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-01-11T17:09:27Z","receivedAt":"2021-01-11T17:10:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/11/2021 11:23 AM, Taylor Blau wrote:\n> On Mon, Jan 11, 2021 at 06:52:24AM -0500, Derrick Stolee wrote:\n>> This is so far the only not-completely-obvious change.\n>>\n>>> But, this is an approximation anyway, and it does remove a use of the\n>>> 'struct revindex_entry' from outside of pack-revindex internals.\n>>\n>> And this might be enough justification for it, but...\n>>\n>>> -\theap += sizeof(struct revindex_entry) * nr_objects;\n>>> +\theap += (sizeof(off_t) + sizeof(uint32_t)) * nr_objects;\n>>\n>> ...outside of the estimation change, will this need another change\n>> when the rev-index is mmap'd? Should this instead be an API call,\n>> such as estimate_rev_index_memory(nr_objects)? That would\n>> centralize the estimate to be next to the code that currently\n>> interacts with 'struct revindex_entry' and will later interact with\n>> the mmap region.\n> \n> I definitely did consider this, and it seems that I made a mistake in\n> not documenting my consideration (since I assumed that it was so benign\n> nobody would notice / care ;-)).\n> \n> The reason I didn't pursue it here was that we haven't yet loaded the\n> reverse index by this point. So, you'd want a function that at least\n> stats the '*.rev' file (and either does or doesn't parse it [1]), or\n> aborts early to indicate otherwise.\n\nIn this patch, I would expect it to use sizeof(struct revindex_entry).\nLater, the method would know if a .rev file exists and do the right\nthing instead. (Also, should mmap'd data count towards this estimate?)\n\n> One would hope that 'load_pack_revindex()' would do just that, but it\n> falls back to load a reverse index in memory, which involves exactly the\n> slow sort that we're trying to avoid. (Of course, we're going to have to\n> do it later anyway, but allocating many GB of heap just to provide an\n> estimation seems ill-advised to me ;-).)\n> \n> So, we'd have to expand the API in some way or another, and to me it\n> didn't seem worth it. As I mentioned in the commit message, I'm\n> skeptical of the value of being accurate here, since this is (after all)\n> an estimation.\n\nYes, I'm probably just poking somewhere it was easy to poke. This is\nprobably not worth the time I'm spending asking about it.\n\nFeel free to disregard.\n\n-Stolee\n"},{"id":"414062","messageId":"61f6acde-3788-03ee-8dce-f621984a3402@gmail.com","threadId":"54961","inReplyTo":"X/x8dXFfQUdpKeVn@nand.local","subject":"Re: [PATCH 18/20] pack-revindex: remove unused 'find_revindex_position()'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-01-11T17:11:12Z","receivedAt":"2021-01-11T17:12:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/11/2021 11:27 AM, Taylor Blau wrote:\n> On Mon, Jan 11, 2021 at 06:57:00AM -0500, Derrick Stolee wrote:\n>> Not that this is new to the current patch, but this patch made me\n>> wonder if we should initialize *pos = -1 in the case of a failure\n>> to find the position? A correct caller should not use the value\n>> if they are checking for the fail-to-find case properly. But, I\n>> could see someone making a mistake and having trouble diagnosing\n>> the problem because their position variable was initialized to\n>> zero or a previous successful case.\n> \n> *pos = -1 may be more confusing than clarifying since pos is unsigned.\n\nRIGHT. My bad.\n\n> It would be nice if there was a clear signal beyond returning a negative\n> value. I guess you could take a double pointer here which would allow\n> you to assign NULL, but that feels rather cumbersome as a means to catch\n> callers who failed to check the return value.\n> \n> It does raise the argument of whether or not we should allow the\n> program to continue at all if 'ret < 0' (i.e., 'offset_to_pack_pos()'\n> either 'die()'s or returns a usable uint32_t), but I'm OK with the\n> current behavior.\n\nI was thinking \"*pos = -1\" was a free way to \"help\" a developer\nwho uses the API incorrectly, but it's _not_ free. Ignore me.\n\nThanks,\n-Stolee\n"},{"id":"414063","messageId":"7e56e831-c6d6-3454-94c5-b8e888497568@gmail.com","threadId":"54961","inReplyTo":"X/x9MMv4hBZMGKBT@nand.local","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-01-11T17:15:58Z","receivedAt":"2021-01-11T17:16:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/11/2021 11:30 AM, Taylor Blau wrote:\n> On Mon, Jan 11, 2021 at 07:07:17AM -0500, Derrick Stolee wrote:\n>> My comments on this series are very minor.\n>>\n>> I made only one comment about \"if (method() < 0)\" versus\n>> \"if (method())\" but that pattern appears in multiple patches.\n>> _If_ you decide to change that pattern, then I'm sure you can\n>> find all uses.\n> \n> I have no strong opinion here, so I'm happy to defer to your or others'\n> judgement. My very weak opinion is that I'd just as soon leave it as-is,\n> but that if I'm rerolling and others would like to see it changed, then\n> I'm happy to do it.\n\nWell, I found 782 instances of \") < 0)\" in the codebase, and my initial\nscan of these shows they are doing exactly what you are asking. So as\nfar as code style goes, there is plenty of precedent.\n\nThe thing that makes me react to this is that it _looks_ like an extra\ncomparison. However, I'm sure the assembly instructions have the same\nperformance characteristics between \"!= 0\" and \"< 0\".\n\nThanks,\n-Stolee\n\n"},{"id":"414064","messageId":"X/yLBm0SSR82Tob8@nand.local","threadId":"54961","inReplyTo":"7e56e831-c6d6-3454-94c5-b8e888497568@gmail.com","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-11T17:29:42Z","receivedAt":"2021-01-11T17:30:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jan 11, 2021 at 12:15:58PM -0500, Derrick Stolee wrote:\n> On 1/11/2021 11:30 AM, Taylor Blau wrote:\n> > On Mon, Jan 11, 2021 at 07:07:17AM -0500, Derrick Stolee wrote:\n> >> My comments on this series are very minor.\n> >>\n> >> I made only one comment about \"if (method() < 0)\" versus\n> >> \"if (method())\" but that pattern appears in multiple patches.\n> >> _If_ you decide to change that pattern, then I'm sure you can\n> >> find all uses.\n> >\n> > I have no strong opinion here, so I'm happy to defer to your or others'\n> > judgement. My very weak opinion is that I'd just as soon leave it as-is,\n> > but that if I'm rerolling and others would like to see it changed, then\n> > I'm happy to do it.\n>\n> Well, I found 782 instances of \") < 0)\" in the codebase, and my initial\n> scan of these shows they are doing exactly what you are asking. So as\n> far as code style goes, there is plenty of precedent.\n\nThanks for looking, I was curious about that myself after our thread,\nbut I hadn't yet bothered to look.\n\n> The thing that makes me react to this is that it _looks_ like an extra\n> comparison. However, I'm sure the assembly instructions have the same\n> performance characteristics between \"!= 0\" and \"< 0\".\n\nIt should make no difference. Both comparisons will do a 'cmp $0 ...'\nwhere '...' is probably an indirect into the current frame. The '!= 0'\nwill use je, and the '< 0' comparison will use 'jns'. Both conditional\njumps should be implemented by checking a CPU flag only (ZF and SF,\nrespectively).\n\nNot that any of this matters, it's just fun to look.\n\n> Thanks,\n> -Stolee\n>\nThanks,\nTaylor\n"},{"id":"414066","messageId":"xmqqlfcz8ggj.fsf@gitster.c.googlers.com","threadId":"54961","inReplyTo":"7e56e831-c6d6-3454-94c5-b8e888497568@gmail.com","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-11T18:40:44Z","receivedAt":"2021-01-11T18:41:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee <stolee@gmail.com> writes:\n\n> Well, I found 782 instances of \") < 0)\" in the codebase, and my initial\n> scan of these shows they are doing exactly what you are asking. So as\n> far as code style goes, there is plenty of precedent.\n>\n> The thing that makes me react to this is that it _looks_ like an extra\n> comparison. However, I'm sure the assembly instructions have the same\n> performance characteristics between \"!= 0\" and \"< 0\".\n\nI'd prefer to keep it that way for the human cost's point of view.\n\nPerhaps it could be subjective but\n\n\tif (func_that_signals_error_with_return_value() < 0)\n\nis immediately recognizable as checking for an error to folks who\nwere trained to write C in POSIX environment, as \"on error, return\nnegative\" is a convention that they are familiar with.  At least to\nme, your \"if (func_that_signals_error_with_return_value())\" looks\nunnatural and makes me look at the function to see what its return\nvalue means.\n\nIf there are helper functions that use \"non-zero is an error and\nzero is success\" convention, we should look at them to see why they\ndo not do the usual \"a negative is an error and a non-negative is\nsuccess\".  And if the *only* reason they do so is because their\nnormal return do not have to give more than one kind of \"success\",\nwe should see if we can fix them to follow the usual \"a negative is\nan error\" convention, I would think.\n\nThanks.\n"},{"id":"414100","messageId":"X/1guCOGWybOzIS7@coredump.intra.peff.net","threadId":"54961","inReplyTo":"fa6b8309088fd04410ca7276c5cf14db0fb82fb2.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 01/20] pack-revindex: introduce a new API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-12T08:41:28Z","receivedAt":"2021-01-12T08:42:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 08, 2021 at 01:16:43PM -0500, Taylor Blau wrote:\n\n> In the next several patches, we will prepare for loading a reverse index\n> either in memory, or from a yet-to-be-introduced on-disk format. To do\n> that, we'll introduce an API that avoids the caller explicitly indexing\n> the revindex pointer in the packed_git structure.\n\nThis API looks good to me. Here are a few extra thoughts:\n\n> There are four ways to interact with the reverse index. Accordingly,\n> four functions will be exported from 'pack-revindex.h' by the time that\n> the existing API is removed. A caller may:\n\nThis tells us what the new API functions do. That's useful, but should\nit be in the header file itself, documenting each function?\n\nLikewise, I think we'd want to define the concepts in that\ndocumentation. Something like:\n\n\n  /*\n   * A revindex allows converting efficiently between three properties\n   * of an object within a pack:\n   *\n   *  - index position: the numeric position within the list of\n   *    sorted object ids found in the .idx file\n   *\n   *  - pack position: the numeric position within the list of objects\n   *    in their order within the actual .pack file (i.e., 0 is the\n   *    first object in the .pack, 1 is the second, and so on)\n   *\n   *  - offset: the byte offset within the .pack file at which the\n   *    object contents can be found\n   */\n\nAnd then above each function we can just say that it converts X to Y\n(like you have in the commit message). It may also be worth indicating\nthe run-time of each (some of them are constant-time once you have a\nrevindex, and some are log(n)).\n\n> +int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n\nThe types here make sense. off_t is clearly needed for a pack offset,\nand uint32_t is correct for the position fields, because packs have a\n4-byte object count.\n\nSeparating the error return from the out-parameter makes the interface\nslightly more awkward, but is needed to use the properly-sized types.\nMakes sense.\n\n> +int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n> +{\n> +\tint ret;\n> +\n> +\tif (load_pack_revindex(p) < 0)\n> +\t\treturn -1;\n\nThis one lazy-loads the revindex for us, which seems handy...\n\n> +uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos)\n> +{\n> +\tif (!p->revindex)\n> +\t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n> +\tif (pos >= p->num_objects)\n> +\t\tBUG(\"pack_pos_to_index: out-of-bounds object at %\"PRIu32, pos);\n> +\treturn p->revindex[pos].nr;\n> +}\n\nBut these ones don't. I'm glad we at least catch it with a BUG(), but it\nmakes the API a little funny. Returning an error here would require a\nsimilarly awkward out-parameter, I guess.\n\n-Peff\n"},{"id":"414102","messageId":"X/1iM0p5d8Zj8ucS@coredump.intra.peff.net","threadId":"54961","inReplyTo":"00668523e1cd860f6de08dd7c5a2a54edc08b7b6.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 02/20] write_reuse_object(): convert to new revindex API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-12T08:47:47Z","receivedAt":"2021-01-12T08:48:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 08, 2021 at 01:16:48PM -0500, Taylor Blau wrote:\n\n> First replace 'find_pack_revindex()' with its replacement\n> 'offset_to_pack_pos()'. This prevents any bogus OFS_DELTA that may make\n> its way through until 'write_reuse_object()' from causing a bad memory\n> read (if 'revidx' is 'NULL')\n\nNice. Better abstraction, and we're catching more errors.\n\n> @@ -436,10 +436,13 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n>  \t\t\t\t\t      type, entry_size);\n>  \n>  \toffset = entry->in_pack_offset;\n> -\trevidx = find_pack_revindex(p, offset);\n> -\tdatalen = revidx[1].offset - offset;\n> +\tif (offset_to_pack_pos(p, offset, &pos) < 0)\n> +\t\tdie(_(\"write_reuse_object: could not locate %s\"),\n> +\t\t    oid_to_hex(&entry->idx.oid));\n\nIf we believe the offset is bogus, should we print that in the error\nmessage, too? Something like:\n\n  die(\"could not locate %s, expected at offset %\"PRIuMAX\" in pack %s\",\n      oid_to_hex(&entry->idx.oid), (uintmax_t)offset, p->pack_name);\n\n> +\tdatalen = pack_pos_to_offset(p, pos + 1) - offset;\n\nThis \"pos + 1\" means we may be looking one past the end of the array.\nThat's OK (at least for now), because our revindex always puts in an\nextra dummy value exactly for computing these kinds of byte-distances.\nThat might be worth documenting in the API header.\n\n-Peff\n"},{"id":"414114","messageId":"X/1ivewkRCD5BpcZ@coredump.intra.peff.net","threadId":"54961","inReplyTo":"81ab11e18c0b00030019f9f521216f3469fdd744.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 03/20] write_reused_pack_one(): convert to new revindex API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-12T08:50:05Z","receivedAt":"2021-01-12T08:50:48Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 08, 2021 at 01:16:53PM -0500, Taylor Blau wrote:\n\n> -\toffset = reuse_packfile->revindex[pos].offset;\n> -\tnext = reuse_packfile->revindex[pos + 1].offset;\n> +\toffset = pack_pos_to_offset(reuse_packfile, pos);\n> +\tnext = pack_pos_to_offset(reuse_packfile, pos + 1);\n\nMakes sense.\n\n> @@ -887,11 +887,15 @@ static void write_reused_pack_one(size_t pos, struct hashfile *out,\n>  \n>  \t\t/* Convert to REF_DELTA if we must... */\n>  \t\tif (!allow_ofs_delta) {\n> -\t\t\tint base_pos = find_revindex_position(reuse_packfile, base_offset);\n> +\t\t\tuint32_t base_pos;\n>  \t\t\tstruct object_id base_oid;\n>  \n> +\t\t\tif (offset_to_pack_pos(reuse_packfile, base_offset, &base_pos) < 0)\n> +\t\t\t\tdie(_(\"expected object at offset %\"PRIuMAX),\n> +\t\t\t\t    (uintmax_t)base_offset);\n\nThis error does mention the offset, which is good. But not the pack name\n(nor the object name, but we don't have it!).\n\n-Peff\n"},{"id":"414115","messageId":"X/1i0pLQ5fcrlIoj@coredump.intra.peff.net","threadId":"54961","inReplyTo":"14b35d01a062f2dfdd710718b659064042dc21d6.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 04/20] write_reused_pack_verbatim(): convert to new revindex API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-12T08:50:26Z","receivedAt":"2021-01-12T08:51:25Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 08, 2021 at 01:16:56PM -0500, Taylor Blau wrote:\n\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index ea7df9270f..4341bc27b4 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -948,7 +948,7 @@ static size_t write_reused_pack_verbatim(struct hashfile *out,\n>  \t\toff_t to_write;\n>  \n>  \t\twritten = (pos * BITS_IN_EWORD);\n> -\t\tto_write = reuse_packfile->revindex[written].offset\n> +\t\tto_write = pack_pos_to_offset(reuse_packfile, written)\n>  \t\t\t- sizeof(struct pack_header);\n\nThis one is obviously correct.\n\n-Peff\n"},{"id":"414117","messageId":"X/1jHFiaKkMc2kqh@coredump.intra.peff.net","threadId":"54961","inReplyTo":"c47e77a30eb40d9841a60a28b620671860dc2461.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 05/20] check_object(): convert to new revindex API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-12T08:51:40Z","receivedAt":"2021-01-12T08:52:22Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 08, 2021 at 01:17:00PM -0500, Taylor Blau wrote:\n\n> Replace direct accesses to the revindex with calls to\n> 'offset_to_pack_pos()' and 'pack_pos_to_index()'.\n> \n> Since this caller already had some error checking (it can jump to the\n> 'give_up' label if it encounters an error), we can easily check whether\n> or not the provided offset points to an object in the given pack. This\n> error checking existed prior to this patch, too, since the caller checks\n> whether the return value from 'find_pack_revindex()' was NULL or not.\n\nYay. Happy again to see things getting more robust.\n\n-Peff\n"},{"id":"414118","messageId":"X/1jrSMU71BLWgm5@coredump.intra.peff.net","threadId":"54961","inReplyTo":"X/x5tdbM3PzkqbFQ@nand.local","subject":"Re: [PATCH 05/20] check_object(): convert to new revindex API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-12T08:54:05Z","receivedAt":"2021-01-12T08:54:47Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 11, 2021 at 11:15:49AM -0500, Taylor Blau wrote:\n\n> On Mon, Jan 11, 2021 at 06:43:23AM -0500, Derrick Stolee wrote:\n> > > @@ -1813,11 +1813,11 @@ static void check_object(struct object_entry *entry, uint32_t object_index)\n> > >  \t\t\t\tgoto give_up;\n> > >  \t\t\t}\n> > >  \t\t\tif (reuse_delta && !entry->preferred_base) {\n> > > -\t\t\t\tstruct revindex_entry *revidx;\n> > > -\t\t\t\trevidx = find_pack_revindex(p, ofs);\n> > > -\t\t\t\tif (!revidx)\n> > > +\t\t\t\tuint32_t pos;\n> > > +\t\t\t\tif (offset_to_pack_pos(p, ofs, &pos) < 0)\n> >\n> > The current implementation does not return a positive value. Only\n> > -1 on error and 0 on success. Is this \"< 0\" doing anything important?\n> > Seems like it would be easiest to do\n> >\n> > \tif (offset_to_pack_pos(p, ofs, &pos))\n> >\n> > [snip]\n> \n> Either would work, of course. I tend to find the '< 0' form easier to\n> read, but I may be in the minority there. For me, the negative return\n> value makes clear that the function encountered an error.\n\nI'll throw in my opinion that \"< 0\" to me much more clearly signals \"did\nan error occur\". And that same form can be used consistently with\nfunctions which _do_ have a positive return value on success, too. So I\nprefer it for readability.\n\n-Peff\n"},{"id":"414119","messageId":"X/1klwe44go+A+Xi@coredump.intra.peff.net","threadId":"54961","inReplyTo":"bc67bb462ae0c87b34e46568d54b170a8aec870b.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 07/20] show_objects_for_type(): convert to new revindex API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-12T08:57:59Z","receivedAt":"2021-01-12T08:59:00Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 08, 2021 at 01:17:09PM -0500, Taylor Blau wrote:\n\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index d6861ddd4d..80c57bde73 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -711,21 +711,22 @@ static void show_objects_for_type(\n>  \n>  \t\tfor (offset = 0; offset < BITS_IN_EWORD; ++offset) {\n>  \t\t\tstruct object_id oid;\n> -\t\t\tstruct revindex_entry *entry;\n> -\t\t\tuint32_t hash = 0;\n> +\t\t\tuint32_t hash = 0, n;\n> +\t\t\toff_t ofs;\n\nA minor nit, but \"n\" isn't very descriptive. It's not in scope for very\nlong, so that's not too bad, but there are two positions at work in this\nfunction: the pos/offset bit position, and the index position. Maybe\n\"index_pos\" would be better than \"n\" to keep the two clear?\n\n-Peff\n"},{"id":"414120","messageId":"X/1mpl/ZmL4NPIEm@coredump.intra.peff.net","threadId":"54961","inReplyTo":"54f4ad329f56808432549aa885f2847d5c9a8ac6.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 09/20] try_partial_reuse(): convert to new revindex API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-12T09:06:46Z","receivedAt":"2021-01-12T09:07:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 08, 2021 at 01:17:17PM -0500, Taylor Blau wrote:\n\n> Remove another instance of direct revindex manipulation by calling\n> 'pack_pos_to_offset()' instead (the caller here does not care about the\n> index position of the object at position 'pos').\n> \n> Somewhat confusingly, the subsequent call to unpack_object_header()\n> takes a pointer to &offset and then updates it with a new value. But,\n> try_partial_reuse() cares about the offset of both the base's header and\n> contents. The existing code made a copy of the offset field, and only\n> addresses and manipulates one of them.\n> \n> Instead, store the return of pack_pos_to_offset twice: once in header\n> and another in offset. Header will be left untouched, but offset will be\n> addressed and modified by unpack_object_header().\n\nI had to read these second two paragraphs a few times to parse them.\nReally we are just replacing revidx->offset with \"header\", and \"offset\"\nretains its same role within the function.\n\nSo it's definitely doing the right thing, but it makes more sense to me\nas:\n\n  Note that we cannot just use the existing \"offset\" variable to store\n  the value we get from pack_pos_to_offset(). It is incremented by\n  unpack_object_header(), but we later need the original value. Since\n  we'll no longer have revindex->offset to read it from, we'll store\n  that in a separate variable (\"header\" since it points to the entry's\n  header bytes).\n\nAnother option would be to just call pack_pos_to_offset() again for the\nlater call. Like the code it's replacing, it's constant-time anyway. But\nI think the \"header\" variable actually makes things more readable.\n\n-Peff\n"},{"id":"414122","messageId":"X/1n36/HtqAoKXrH@coredump.intra.peff.net","threadId":"54961","inReplyTo":"eab7ab1f35fa9703f56a99fa539839869fe4e54c.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 13/20] packed_object_info(): convert to new revindex API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-12T09:11:59Z","receivedAt":"2021-01-12T09:12:57Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 08, 2021 at 01:17:34PM -0500, Taylor Blau wrote:\n\n> Convert another call of 'find_pack_revindex()' to its replacement\n> 'pack_pos_to_offset()'. Likewise:\n> \n>   - Avoid manipulating `struct packed_git`'s `revindex` pointer directly\n>     by removing the pointer-as-array indexing.\n\nGood.\n\n>   - Add an additional guard to check that the offset 'obj_offset()'\n>     points to a real object. This should be the case with well-behaved\n>     callers to 'packed_object_info()', but isn't guarenteed.\n>\n>     Other blocks that fill in various other values from the 'struct\n>     object_info' request handle bad inputs by setting the type to\n>     'OBJ_BAD' and jumping to 'out'. Do the same when given a bad offset\n>     here.\n\nAlso good. I wonder if we need to call error() here, too. The caller\nwill probably say something like \"bad object\" or whatever, but the user\nwill have no clue that it's related to the revindex.\n\nThat would match other parts of the function (e.g., calling into\nunpack_entry() can generate lots of descriptive errors about exactly\nwhat went wrong).\n\n>     The previous code would have segfaulted when given a bad\n>     'obj_offset' value, since 'find_pack_revindex()' would return\n>     'NULL', and then the line that fills 'oi->disk_sizep' would try to\n>     access 'NULL[1]' with a stride of 16 bytes (the width of 'struct\n>     revindex_entry)'.\n\nYep. Again, I'm really happy to see these \"should never happen\" cases\nconverted to real errors or even BUG()s.\n\n-Peff\n"},{"id":"414123","messageId":"X/1qUphaPD1Pvk+X@coredump.intra.peff.net","threadId":"54961","inReplyTo":"13c49ed40ca72b7ab50939244616f0a90b5bf7f6.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 14/20] unpack_entry(): convert to new revindex API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-12T09:22:26Z","receivedAt":"2021-01-12T09:23:08Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 08, 2021 at 01:17:39PM -0500, Taylor Blau wrote:\n\n> diff --git a/packfile.c b/packfile.c\n> index 469c8d4f57..34467ea4a3 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -1692,11 +1692,19 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,\n>  \t\t}\n>  \n>  \t\tif (do_check_packed_object_crc && p->index_version > 1) {\n> -\t\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n> -\t\t\toff_t len = revidx[1].offset - obj_offset;\n> -\t\t\tif (check_pack_crc(p, &w_curs, obj_offset, len, revidx->nr)) {\n> +\t\t\tuint32_t pos, nr;\n\nWe have \"pos\" and \"nr\". What's the difference? :)\n\nI think pack_pos and index_pos might be harder to get confused.\n\n> +\t\t\toff_t len;\n> +\n> +\t\t\tif (offset_to_pack_pos(p, obj_offset, &pos) < 0) {\n> +\t\t\t\tdata = NULL;\n> +\t\t\t\tgoto out;\n> +\t\t\t}\n\nNice to see the error check here. As with the previous commit, we\nprobably want to error(), just as we would for errors below.\n\nDo we also need to call mark_bad_packed_object()? I guess we can't,\nbecause we only have the offset, and not the oid (the code below uses\nnth_packed_object_id(), but it is relying on the revindex, which we know\njust failed to work).\n\nI'm just wondering if an error here is going to put us into an infinite\nloop of retrying the lookup in the same pack over and over. Let's\nsee...our caller is ultimately packed_object_info(), but it too does not\nhave the oid. It returns an error up to do_oid_object_info_extended().\nWhich yes, does mark_bad_packed_object() itself. Good. So I think we are\nfine, and arguably these lower-level calls to mark_bad_packed_object()\nare not necessary. But they do not hurt either.\n\n-Peff\n"},{"id":"414124","messageId":"X/1roycRbYPjnI3l@coredump.intra.peff.net","threadId":"54961","inReplyTo":"13c28eca-81d5-10c9-c92c-162547416014@gmail.com","subject":"Re: [PATCH 16/20] builtin/gc.c: guess the size of the revindex","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-12T09:28:03Z","receivedAt":"2021-01-12T09:29:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 11, 2021 at 12:09:27PM -0500, Derrick Stolee wrote:\n\n> > The reason I didn't pursue it here was that we haven't yet loaded the\n> > reverse index by this point. So, you'd want a function that at least\n> > stats the '*.rev' file (and either does or doesn't parse it [1]), or\n> > aborts early to indicate otherwise.\n> \n> In this patch, I would expect it to use sizeof(struct revindex_entry).\n> Later, the method would know if a .rev file exists and do the right\n> thing instead. (Also, should mmap'd data count towards this estimate?)\n\nYeah, I think if we care about memory pressure, then the mmap would\ncount anyway. I agree that letting the revindex code decide which to use\nwould be the most accurate thing, but given that this whole chunk of\ncode is an estimate (that does not even seem to take into account the\nmemory used for the delta search!), I don't think it's worth trying to\nget to accurate.\n\n-Peff\n"},{"id":"414125","messageId":"X/1st6SrJXysoejt@coredump.intra.peff.net","threadId":"54961","inReplyTo":"d60411d524656f4680ac578765b2a8704325a060.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 18/20] pack-revindex: remove unused 'find_revindex_position()'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-12T09:32:39Z","receivedAt":"2021-01-12T09:33:38Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 08, 2021 at 01:17:57PM -0500, Taylor Blau wrote:\n\n> Now that all 'find_revindex_position()' callers have been removed (and\n> converted to the more descriptive 'offset_to_pack_pos()'), it is almost\n> safe to get rid of 'find_revindex_position()' entirely. Almost, except\n> for the fact that 'offset_to_pack_pos()' calls\n> 'find_revindex_position()'.\n> \n> Inline 'find_revindex_position()' into 'offset_to_pack_pos()', and\n> then remove 'find_revindex_position()' entirely.\n\nSounds good.\n\n> This is a straightforward refactoring with one minor snag.\n> 'offset_to_pack_pos()' used to load the index before calling\n> 'find_revindex_position()'. That means that by the time\n> 'find_revindex_position()' starts executing, 'p->num_objects' can be\n> safely read. After inlining, be careful to not read 'p->num_objects'\n> until _after_ 'load_pack_revindex()' (which loads the index as a\n> side-effect) has been called.\n\nGood catch. We might want to drop the initialization of \"lo\":\n\n>  \tint lo = 0;\n> -\tint hi = p->num_objects + 1;\n\ndown to here:\n\n> +\thi = p->num_objects + 1;\n\nto maintain symmetry (though it's quite a minor point).\n\nI notice these are signed ints, but we've taken care to use uint32_t\nelsewhere for positions. Shouldn't these be uint32_t, also (or at least\nunsigned)?\n\n-Peff\n"},{"id":"414126","messageId":"X/1tKaIXxj9u/QfM@coredump.intra.peff.net","threadId":"54961","inReplyTo":"7c0e4acc845d1135e684188b2ccc61cf358994dc.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 19/20] pack-revindex: hide the definition of 'revindex_entry'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-12T09:34:33Z","receivedAt":"2021-01-12T09:35:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 08, 2021 at 01:18:01PM -0500, Taylor Blau wrote:\n\n> Now that all spots outside of pack-revindex.c that reference 'struct\n> revindex_entry' directly have been removed, it is safe to hide the\n> implementation by moving it from pack-revindex.h to pack-revindex.c.\n\nThat was a lot of patches to get here, but this is a very nice outcome. :)\n\n-Peff\n"},{"id":"414127","messageId":"X/1txTIxV4pYt0Xo@coredump.intra.peff.net","threadId":"54961","inReplyTo":"eada1ffcfafc3fb57de80626e368672cb8b22318.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 20/20] pack-revindex.c: avoid direct revindex access in 'offset_to_pack_pos()'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-12T09:37:09Z","receivedAt":"2021-01-12T09:38:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 08, 2021 at 01:18:06PM -0500, Taylor Blau wrote:\n\n> To prepare for on-disk reverse indexes, remove a spot in\n> 'offset_to_pack_pos()' that looks at the 'revindex' array in 'struct\n> packed_git'.\n> \n> Even though this use of the revindex pointer is within pack-revindex.c,\n> this clean up is still worth doing. Since the 'revindex' pointer will be\n> NULL when reading from an on-disk reverse index (instead the\n> 'revindex_data' pointer will be mmaped to the 'pack-*.rev' file), this\n> call-site would have to include a conditional to lookup the offset for\n> position 'mi' each iteration through the search.\n> \n> So instead of open-coding 'pack_pos_to_offset()', call it directly from\n> within 'offset_to_pack_pos()'.\n\nThis definitely makes sense in the long run. I could take or leave it as\na final patch in _this_ series (as opposed to the first patch in a\nsubsequent series adding the rev files).\n\n>  \tdo {\n>  \t\tconst unsigned mi = lo + (hi - lo) / 2;\n> -\t\tif (revindex[mi].offset == ofs) {\n> +\t\toff_t got = pack_pos_to_offset(p, mi);\n\n\nThey're both constant-time, so performance should be the same big-O. The\nfunction has extra BUG() checks. I doubt those are measurable in\npractice, though.\n\n-Peff\n"},{"id":"414128","messageId":"X/1u46v1dhu0Aj8G@coredump.intra.peff.net","threadId":"54961","inReplyTo":"X/1guCOGWybOzIS7@coredump.intra.peff.net","subject":"Re: [PATCH 01/20] pack-revindex: introduce a new API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-12T09:41:55Z","receivedAt":"2021-01-12T09:42:37Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 12, 2021 at 03:41:28AM -0500, Jeff King wrote:\n\n> > +int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n> > +{\n> > +\tint ret;\n> > +\n> > +\tif (load_pack_revindex(p) < 0)\n> > +\t\treturn -1;\n> \n> This one lazy-loads the revindex for us, which seems handy...\n> \n> > +uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos)\n> > +{\n> > +\tif (!p->revindex)\n> > +\t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n> > +\tif (pos >= p->num_objects)\n> > +\t\tBUG(\"pack_pos_to_index: out-of-bounds object at %\"PRIu32, pos);\n> > +\treturn p->revindex[pos].nr;\n> > +}\n> \n> But these ones don't. I'm glad we at least catch it with a BUG(), but it\n> makes the API a little funny. Returning an error here would require a\n> similarly awkward out-parameter, I guess.\n\nHaving now looked at the callers through the series, I think adding an\nerror return to pack_pos_to_index() would be really awkward (since it\ncannot currently fail).\n\nWe _could_ insist that callers of offset_to_pack_pos() also make sure\nthe revindex is loaded themselves. But it would be annoying and\nerror-prone to check the existing callers. So I'm OK with leaving this\nasymmetry in the API.\n\n-Peff\n"},{"id":"414129","messageId":"X/1vy3D10wDEZNva@coredump.intra.peff.net","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-12T09:45:47Z","receivedAt":"2021-01-12T09:46:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 08, 2021 at 01:16:39PM -0500, Taylor Blau wrote:\n\n> Generating the reverse index in memory for repositories with large packs has two\n> significant drawbacks:\n> \n>   - It requires allocating sizeof(struct revindex_entry) per packed object.\n> \n>   - It requires us to sort the entries by their pack offset. This is implemented\n>     in sort_revindex() using a radix sort, but still takes considerable time (as\n>     benchmarks found in the second series demonstrate).\n\nOr thinking about it more fundamentally: any operation which touches the\nrevindex is now O(nr_objects_in_repo), even if it only cares about a few\nobjects. Ideally this will eventually be this O(log nr_objects_in_repo);\nwe can't do much better than that because of object lookups (unless we\nreplace the .idx with a perfect hash or something).\n\n> The goal of this series is to remove direct access of the `struct\n> revindex_entry` type, as well as `struct packed_git`'s `revindex` field. The\n> on-disk format will be mmap'd and accessed directly, but the format is\n> sufficiently different that the whole `revindex` array can't be written as-is.\n\nIt looks good overall to me. I left a few nits around documentation and\ninteger types that I think are worth a re-roll, but I think after\naddressing those it should be good.\n\n-Peff\n"},{"id":"414169","messageId":"X/3ODgaa9wr65M09@nand.local","threadId":"54961","inReplyTo":"X/1guCOGWybOzIS7@coredump.intra.peff.net","subject":"Re: [PATCH 01/20] pack-revindex: introduce a new API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-12T16:27:58Z","receivedAt":"2021-01-12T16:29:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 12, 2021 at 03:41:28AM -0500, Jeff King wrote:\n> > There are four ways to interact with the reverse index. Accordingly,\n> > four functions will be exported from 'pack-revindex.h' by the time that\n> > the existing API is removed. A caller may:\n>\n> This tells us what the new API functions do. That's useful, but should\n> it be in the header file itself, documenting each function?\n\nMm, that's a good idea. I took your suggestion for a comment at the top\nof pack-revindex.h directly, and then added some of my own commentary\nabove each function. I avoided documenting the functions we're about to\nremove for obvious reasons.\n\nI think that the commit message is OK as-is, since it provides more of a\nrationale of what operations need to exist, rather than the specifics of\neach implementation.\n\nWe'll have to update these again when the on-disk format exists (i.e.,\nbecause all of the runtimes become constant with the exception of going\nto the index), but that's a topic for another series.\n\n> > +int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n>\n> The types here make sense. off_t is clearly needed for a pack offset,\n> and uint32_t is correct for the position fields, because packs have a\n> 4-byte object count.\n>\n> Separating the error return from the out-parameter makes the interface\n> slightly more awkward, but is needed to use the properly-sized types.\n> Makes sense.\n\nYep, exactly.\n\n> > +int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n> > +{\n> > +\tint ret;\n> > +\n> > +\tif (load_pack_revindex(p) < 0)\n> > +\t\treturn -1;\n>\n> This one lazy-loads the revindex for us, which seems handy...\n>\n> > +uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos)\n> > +{\n> > +\tif (!p->revindex)\n> > +\t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n> > +\tif (pos >= p->num_objects)\n> > +\t\tBUG(\"pack_pos_to_index: out-of-bounds object at %\"PRIu32, pos);\n> > +\treturn p->revindex[pos].nr;\n> > +}\n>\n> But these ones don't. I'm glad we at least catch it with a BUG(), but it\n> makes the API a little funny. Returning an error here would require a\n> similarly awkward out-parameter, I guess.\n\nIt is awkward, but (as you note downthread) the callers of\npack_pos_to_index() make it difficult to do so, so I didn't pursue it\nhere.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"414170","messageId":"X/3O7D18gRLfSNJ8@nand.local","threadId":"54961","inReplyTo":"X/1iM0p5d8Zj8ucS@coredump.intra.peff.net","subject":"Re: [PATCH 02/20] write_reuse_object(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-12T16:31:40Z","receivedAt":"2021-01-12T16:32:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 12, 2021 at 03:47:47AM -0500, Jeff King wrote:\n> > @@ -436,10 +436,13 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n> >  \t\t\t\t\t      type, entry_size);\n> >\n> >  \toffset = entry->in_pack_offset;\n> > -\trevidx = find_pack_revindex(p, offset);\n> > -\tdatalen = revidx[1].offset - offset;\n> > +\tif (offset_to_pack_pos(p, offset, &pos) < 0)\n> > +\t\tdie(_(\"write_reuse_object: could not locate %s\"),\n> > +\t\t    oid_to_hex(&entry->idx.oid));\n>\n> If we believe the offset is bogus, should we print that in the error\n> message, too? Something like:\n>\n>   die(\"could not locate %s, expected at offset %\"PRIuMAX\" in pack %s\",\n>       oid_to_hex(&entry->idx.oid), (uintmax_t)offset, p->pack_name);\n\nGood idea, thanks.\n\n> > +\tdatalen = pack_pos_to_offset(p, pos + 1) - offset;\n>\n> This \"pos + 1\" means we may be looking one past the end of the array.\n> That's OK (at least for now), because our revindex always puts in an\n> extra dummy value exactly for computing these kinds of byte-distances.\n> That might be worth documenting in the API header.\n\nYeah, I made sure to document that when I was touching up the last\npatch. FWIW, that's a behavior that we're going to carry over even when\nthe reverse index is stored on-disk (not by writing four extra bytes\ninto the .rev file, but by handling queries for pos == p->num_objects\nseparately.)\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"414171","messageId":"X/3Pgn/HWVkncbgi@nand.local","threadId":"54961","inReplyTo":"X/1ivewkRCD5BpcZ@coredump.intra.peff.net","subject":"Re: [PATCH 03/20] write_reused_pack_one(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-12T16:34:10Z","receivedAt":"2021-01-12T16:35:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 12, 2021 at 03:50:05AM -0500, Jeff King wrote:\n> > @@ -887,11 +887,15 @@ static void write_reused_pack_one(size_t pos, struct hashfile *out,\n> >\n> >  \t\t/* Convert to REF_DELTA if we must... */\n> >  \t\tif (!allow_ofs_delta) {\n> > -\t\t\tint base_pos = find_revindex_position(reuse_packfile, base_offset);\n> > +\t\t\tuint32_t base_pos;\n> >  \t\t\tstruct object_id base_oid;\n> >\n> > +\t\t\tif (offset_to_pack_pos(reuse_packfile, base_offset, &base_pos) < 0)\n> > +\t\t\t\tdie(_(\"expected object at offset %\"PRIuMAX),\n> > +\t\t\t\t    (uintmax_t)base_offset);\n>\n> This error does mention the offset, which is good. But not the pack name\n> (nor the object name, but we don't have it!).\n\nIndeed we don't have the object name, but the pack is reuse_packfile\n(which is statistically initialized), so we could do something like:\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 10a16ced1e..8e40b19ee8 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -893,8 +893,10 @@ static void write_reused_pack_one(size_t pos, struct hashfile *out,\n \t\t\tstruct object_id base_oid;\n\n \t\t\tif (offset_to_pack_pos(reuse_packfile, base_offset, &base_pos) < 0)\n-\t\t\t\tdie(_(\"expected object at offset %\"PRIuMAX),\n-\t\t\t\t    (uintmax_t)base_offset);\n+\t\t\t\tdie(_(\"expected object at offset %\"PRIuMAX\" \"\n+\t\t\t\t      \"in pack %s\"),\n+\t\t\t\t    (uintmax_t)base_offset,\n+\t\t\t\t    reuse_packfile->pack_name);\n\n \t\t\tnth_packed_object_id(&base_oid, reuse_packfile,\n \t\t\t\t\t     pack_pos_to_index(reuse_packfile, base_pos));\n\nWhich I think would be clearer.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"414172","messageId":"X/3P1eM+VDyNWRSY@nand.local","threadId":"54961","inReplyTo":"X/1klwe44go+A+Xi@coredump.intra.peff.net","subject":"Re: [PATCH 07/20] show_objects_for_type(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-12T16:35:33Z","receivedAt":"2021-01-12T16:36:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 12, 2021 at 03:57:59AM -0500, Jeff King wrote:\n> A minor nit, but \"n\" isn't very descriptive. It's not in scope for very\n> long, so that's not too bad, but there are two positions at work in this\n> function: the pos/offset bit position, and the index position. Maybe\n> \"index_pos\" would be better than \"n\" to keep the two clear?\n\nMuch clearer, thank you.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"414174","messageId":"X/3SsBNNgztxJ24H@nand.local","threadId":"54961","inReplyTo":"X/1mpl/ZmL4NPIEm@coredump.intra.peff.net","subject":"Re: [PATCH 09/20] try_partial_reuse(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-12T16:47:44Z","receivedAt":"2021-01-12T16:48:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 12, 2021 at 04:06:46AM -0500, Jeff King wrote:\n> On Fri, Jan 08, 2021 at 01:17:17PM -0500, Taylor Blau wrote:\n> > Somewhat confusingly, the subsequent call to unpack_object_header()\n> > takes a pointer to &offset and then updates it with a new value. But,\n> > try_partial_reuse() cares about the offset of both the base's header and\n> > contents. The existing code made a copy of the offset field, and only\n> > addresses and manipulates one of them.\n> >\n> > Instead, store the return of pack_pos_to_offset twice: once in header\n> > and another in offset. Header will be left untouched, but offset will be\n> > addressed and modified by unpack_object_header().\n>\n> I had to read these second two paragraphs a few times to parse them.\n> Really we are just replacing revidx->offset with \"header\", and \"offset\"\n> retains its same role within the function.\n>\n> So it's definitely doing the right thing, but it makes more sense to me\n> as: [...]\n\nThanks. Reading these both back-to-back, I agree that yours is much\nclearer in describing what is actually going on.\n\nThanks,\nTaylor\n"},{"id":"414177","messageId":"X/3TgQrq1aiAByjJ@nand.local","threadId":"54961","inReplyTo":"X/1n36/HtqAoKXrH@coredump.intra.peff.net","subject":"Re: [PATCH 13/20] packed_object_info(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-12T16:51:13Z","receivedAt":"2021-01-12T16:52:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 12, 2021 at 04:11:59AM -0500, Jeff King wrote:\n> >   - Add an additional guard to check that the offset 'obj_offset()'\n> >     points to a real object. This should be the case with well-behaved\n> >     callers to 'packed_object_info()', but isn't guarenteed.\n> >\n> >     Other blocks that fill in various other values from the 'struct\n> >     object_info' request handle bad inputs by setting the type to\n> >     'OBJ_BAD' and jumping to 'out'. Do the same when given a bad offset\n> >     here.\n>\n> Also good. I wonder if we need to call error() here, too. The caller\n> will probably say something like \"bad object\" or whatever, but the user\n> will have no clue that it's related to the revindex.\n>\n> That would match other parts of the function (e.g., calling into\n> unpack_entry() can generate lots of descriptive errors about exactly\n> what went wrong).\n\nIndeed, and we have the object's offset and pack name to include, too,\nso our error can be quite descriptive.\n\n> Yep. Again, I'm really happy to see these \"should never happen\" cases\n> converted to real errors or even BUG()s.\n\n:-).\n\nThanks,\nTaylor\n"},{"id":"414178","messageId":"X/3UxI01yNmgQ0hq@nand.local","threadId":"54961","inReplyTo":"X/1qUphaPD1Pvk+X@coredump.intra.peff.net","subject":"Re: [PATCH 14/20] unpack_entry(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-12T16:56:36Z","receivedAt":"2021-01-12T16:57:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 12, 2021 at 04:22:26AM -0500, Jeff King wrote:\n> We have \"pos\" and \"nr\". What's the difference? :)\n>\n> I think pack_pos and index_pos might be harder to get confused.\n\nMuch clearer, thanks.\n\n> > +\t\t\toff_t len;\n> > +\n> > +\t\t\tif (offset_to_pack_pos(p, obj_offset, &pos) < 0) {\n> > +\t\t\t\tdata = NULL;\n> > +\t\t\t\tgoto out;\n> > +\t\t\t}\n>\n> Nice to see the error check here. As with the previous commit, we\n> probably want to error(), just as we would for errors below.\n\nYup, good call.\n\n> Do we also need to call mark_bad_packed_object()? I guess we can't,\n> because we only have the offset, and not the oid (the code below uses\n> nth_packed_object_id(), but it is relying on the revindex, which we know\n> just failed to work).\n>\n> I'm just wondering if an error here is going to put us into an infinite\n> loop of retrying the lookup in the same pack over and over. Let's\n> see...our caller is ultimately packed_object_info(), but it too does not\n> have the oid. It returns an error up to do_oid_object_info_extended().\n> Which yes, does mark_bad_packed_object() itself. Good. So I think we are\n> fine, and arguably these lower-level calls to mark_bad_packed_object()\n> are not necessary. But they do not hurt either.\n\nThanks, this rationale is helpful to have. I included an abridged\nversion of it in the patch message.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"414179","messageId":"X/3VfteeF3Ok2C+S@nand.local","threadId":"54961","inReplyTo":"X/1st6SrJXysoejt@coredump.intra.peff.net","subject":"Re: [PATCH 18/20] pack-revindex: remove unused 'find_revindex_position()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-12T16:59:42Z","receivedAt":"2021-01-12T17:00:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 12, 2021 at 04:32:39AM -0500, Jeff King wrote:\n> Good catch. We might want to drop the initialization of \"lo\":\n>\n> >  \tint lo = 0;\n> > -\tint hi = p->num_objects + 1;\n>\n> down to here:\n>\n> > +\thi = p->num_objects + 1;\n\n> to maintain symmetry (though it's quite a minor point).\n\n:-). I agree it's a minor point, but I think that the symmetry is nice,\nso it's worth doing.\n\n> I notice these are signed ints, but we've taken care to use uint32_t\n> elsewhere for positions. Shouldn't these be uint32_t, also (or at least\n> unsigned)?\n\nI'll let both of these be an raw unsigned, since the midpoint is already\nlabeled as an unsigned.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"414180","messageId":"X/3WDArP4zrdd0S2@nand.local","threadId":"54961","inReplyTo":"X/1txTIxV4pYt0Xo@coredump.intra.peff.net","subject":"Re: [PATCH 20/20] pack-revindex.c: avoid direct revindex access in 'offset_to_pack_pos()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-12T17:02:04Z","receivedAt":"2021-01-12T17:03:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 12, 2021 at 04:37:09AM -0500, Jeff King wrote:\n> This definitely makes sense in the long run. I could take or leave it as\n> a final patch in _this_ series (as opposed to the first patch in a\n> subsequent series adding the rev files).\n>\n> >  \tdo {\n> >  \t\tconst unsigned mi = lo + (hi - lo) / 2;\n> > -\t\tif (revindex[mi].offset == ofs) {\n> > +\t\toff_t got = pack_pos_to_offset(p, mi);\n>\n>\n> They're both constant-time, so performance should be the same big-O. The\n> function has extra BUG() checks. I doubt those are measurable in\n> practice, though.\n\nFunny enough, I have moved this patch between the two so many times\nbefore submitting this. I tend to agree that I don't think it makes a\ndifference in which series this patch goes, so I'm just as happy to\nleave it where it is and stop thinking about it ;-).\n\nIf others have strong feelings, this can be dropped when queuing and\nI'll send it along as the first commit in the second series (which will\nhave to be updated along with this one).\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"414182","messageId":"X/3Xbxf5P8303di5@nand.local","threadId":"54961","inReplyTo":"X/1vy3D10wDEZNva@coredump.intra.peff.net","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-12T17:07:59Z","receivedAt":"2021-01-12T17:08:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 12, 2021 at 04:45:47AM -0500, Jeff King wrote:\n> > The goal of this series is to remove direct access of the `struct\n> > revindex_entry` type, as well as `struct packed_git`'s `revindex` field. The\n> > on-disk format will be mmap'd and accessed directly, but the format is\n> > sufficiently different that the whole `revindex` array can't be written as-is.\n>\n> It looks good overall to me. I left a few nits around documentation and\n> integer types that I think are worth a re-roll, but I think after\n> addressing those it should be good.\n\nThanks so much for your review. I think I addressed all of your\nfeedback, but I'll sit on the revised patches for another day or so in\ncase other reviewers would like to chime in before v2.\n\nHopefully the this re-roll is the only one we'll need, since the next\nseries is much more interesting!\n\nThanks,\nTaylor\n"},{"id":"414237","messageId":"xmqqk0shznvf.fsf@gitster.c.googlers.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-13T00:23:00Z","receivedAt":"2021-01-13T00:47:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> This is the first of two series to introduce an on-disk alternative to the\n> reverse index which is currently generated once per-process and stored in\n> memory.\n\nQueued; seems to be killed on Windows but otherwise looking good.\n  \n  https://github.com/git/git/runs/1691849288?check_suite_focus=true#step:6:164\n\nThanks.\n"},{"id":"414239","messageId":"X/5ER+ml/MhDjROA@nand.local","threadId":"54961","inReplyTo":"xmqqk0shznvf.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T00:52:23Z","receivedAt":"2021-01-13T00:53:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 12, 2021 at 04:23:00PM -0800, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> > This is the first of two series to introduce an on-disk alternative to the\n> > reverse index which is currently generated once per-process and stored in\n> > memory.\n>\n> Queued; seems to be killed on Windows but otherwise looking good.\n\nThanks.\n\n>   https://github.com/git/git/runs/1691849288?check_suite_focus=true#step:6:164\n\nI will look into those failures. It looks like these are all due to\nrunning 'git index-pack' when the '*.idx' file already exists (resulting\nin Windows being unable to write over the file).\n\nDid you want to queue these two topics separately? It looks like you\npicked them up in 'seen' as one big topic in a8f367dae2 (Merge branch\n'tb/pack-revindex-api' into jch, 2021-01-12), but I thought you may want\nto pick parts one and two up separately to allow them to progress\nindependently of one another.\n\nIn either case, I am planning on sending a new version of the first\nseries tomorrow to address Peff's feedback.\n\n> Thanks.\n\nThanks,\nTaylor\n"},{"id":"414241","messageId":"xmqqft35ziog.fsf@gitster.c.googlers.com","threadId":"54961","inReplyTo":"X/5ER+ml/MhDjROA@nand.local","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-13T02:15:11Z","receivedAt":"2021-01-13T02:16:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> Did you want to queue these two topics separately?\n\nUnless it is a reasonable amount of certainty that the bottom topic\nwill not have to be rerolled, I'd rather not to have separate topics\non top of each other.\n\nIt is tedious and error prone, having to rebase the old iteration of\nthe top topic on top when a new iteration of the bottom topic comes\nout.  I'd rather see that the top one get rerolled whenever the\nbottom one gets rerolled to make life simpler.\n\nEven in a single topic, I would encourage people to put the\nfoundational and preparatory work early so that we can make them\nsolidify before review really gets to later parts of the series.\n\nAnd when a series is structured like so, it is perfectly fine to say\nsomething like:\n\n    Here is a new iteration of the last 7 patches---the early 13\n    patches are the same as the previous round, so reset the topic\n    to bb6414ab (packfile: prepare for the existence of '*.rev'\n    files, 2021-01-08) and then apply these 7.\n\nif you do not want to send all patches in a nontrivial series when\nit gets updated.\n\nThanks.\n"},{"id":"414242","messageId":"X/5nsw6uqKDCHGql@nand.local","threadId":"54961","inReplyTo":"xmqqft35ziog.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T03:23:31Z","receivedAt":"2021-01-13T03:24:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 12, 2021 at 06:15:11PM -0800, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> > Did you want to queue these two topics separately?\n>\n> Unless it is a reasonable amount of certainty that the bottom topic\n> will not have to be rerolled, I'd rather not to have separate topics\n> on top of each other.\n\nHmm. I certainly do not want to create any tedious work for you, but I\ndo think that we're in a situation where the bottom topic is stable and\nthe interesting discussion will happen in the newer topic.\n\n> It is tedious and error prone, having to rebase the old iteration of\n> the top topic on top when a new iteration of the bottom topic comes\n> out.  I'd rather see that the top one get rerolled whenever the\n> bottom one gets rerolled to make life simpler.\n\nIndeed, my local v2 of the bottom topic requires the top topic to be\nrebased onto it, and so I'll plan to send a v2 of each tomorrow morning.\n\n> Even in a single topic, I would encourage people to put thet\n> foundational and preparatory work early so that we can make them\n> solidify before review really gets to later parts of the series.\n>\n> And when a series is structured like so, it is perfectly fine to say\n> something like:\n>\n>     Here is a new iteration of the last 7 patches---the early 13\n>     patches are the same as the previous round, so reset the topic\n>     to bb6414ab (packfile: prepare for the existence of '*.rev'\n>     files, 2021-01-08) and then apply these 7.\n>\n> if you do not want to send all patches in a nontrivial series when\n> it gets updated.\n\nI am happy to do it this way if you would prefer. If so, you may want to\nrename this topic while queuing, since it is not just about a new\nrevindex API, but rather about introducing the on-disk format. (I would\nsuggest tb/on-disk-revindex, if you were looking for alternate names).\n\nIf you agree that the bottom topic is stable, I'd prefer to send the top\none separately. Otherwise, I can send both together. Let me know.\n\n> Thanks.\n\nThanks,\nTaylor\n"},{"id":"414245","messageId":"xmqqa6tdz2fo.fsf@gitster.c.googlers.com","threadId":"54961","inReplyTo":"fa6b8309088fd04410ca7276c5cf14db0fb82fb2.1610129796.git.me@ttaylorr.com","subject":"Re: [PATCH 01/20] pack-revindex: introduce a new API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-13T08:06:03Z","receivedAt":"2021-01-13T08:06:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> In the next several patches, we will prepare for loading a reverse index\n> either in memory, or from a yet-to-be-introduced on-disk format. To do\n\nDoes \"load revindex in memory\" (as opposed to \"from on-disk file\")\nmean the good old \"read the forward index and make inverse map\nin-core\", or something else?  \n\nIOW, is \"We will prepare a reverse index either by computing in\nmemory from forward index, or loading from on-disk file\" what we\nwant to say here?\n\n> There are four ways to interact with the reverse index. Accordingly,\n> four functions will be exported from 'pack-revindex.h' by the time that\n> the existing API is removed. A caller may:\n>\n>  1. Load the pack's reverse index. This involves opening up the index,\n>     generating an array, and then sorting it. Since opening the index\n>     can fail, this function ('load_pack_revindex()') returns an int.\n>     Accordingly, it takes only a single argument: the 'struct\n>     packed_git' the caller wants to build a reverse index for.\n>\n>     This function is well-suited for both the current and new API.\n>     Callers will have to continue to open the reverse index explicitly,\n>     but this function will eventually learn how to detect and load a\n>     reverse index from the on-disk format, if one exists. Otherwise, it\n>     will fallback to generating one in memory from scratch.\n\nOK.\n\n>  2. Convert a pack position into an offset. This operation is now\n>     called `pack_pos_to_offset()`. It takes a pack and a position, and\n>     returns the corresponding off_t.\n>\n>  3. Convert a pack position into an index position. Same as above; this\n>     takes a pack and a position, and returns a uint32_t. This operation\n>     is known as `pack_pos_to_index()`.\n>\n>  4. Find the pack position for a given offset. This operation is now\n>     known as `offset_to_pack_pos()`. It takes a pack, an offset, and a\n>     pointer to a uint32_t where the position is written, if an object\n>     exists at that offset. Otherwise, -1 is returned to indicate\n>     failure.\n\nWithout knowing what exactly \"pack position\", \"offset\" and \"index\nposition\" refer to, the above three are almost impossible to grok.\nCan we have one paragraph description for each?  Something along the\nlines of...\n\n - Pack position: a packstream consists of series of byte ranges,\n   each of which represents an object, so the objects can be\n   numbered from 0 (the object whose data is stored at the earliest\n   part in the packfile) to N (the object whose data is stored at\n   the tail end of the packfile).  The number corresponding to an\n   object in this order in the packfile is called the \"pack\n   position\" of the object.\n\n - Offset: The ofs_t distance between the beginning of a pack stream\n   and the beginning of data that represents an object is called the\n   \"offset\" of the object in the packfile.\n\n - Index position: for a single pack stream, there is a table that\n   maps object name to its offset and the entries in this table are\n   sorted by the object name (this is what pack \".idx\" file is).\n   The location (counting from 0) of an object in this table is\n   called the \"index position\" of the object in the packfile.\n\nI am not sure if the above correctly reflects what you meant by\n\"position\", though.\n\n>     Unlike some of the callers that used to access '->offset' and '->nr'\n>     directly, the error checking around this call is somewhat more\n>     robust. This is important since callers can pass an offset which\n>     does not contain an object.\n\nMeaning \"offset ought to point at the boundary between objects in\nthe pack stream, and the API, unlike the direct access, makes sure\nthat is the case\"?  That is a good thing.\n\n>     This will become important in a subsequent patch where a caller\n>     which does not but could check the return value treats the signed\n>     `-1` from `find_revindex_position()` as an index into the 'revindex'\n>     array.\n>\n> Signed-off-by: Taylor Blau <me@ttaylorr.com>\n> ---\n>  pack-revindex.c | 32 ++++++++++++++++++++++++++++++++\n>  pack-revindex.h |  4 ++++\n>  2 files changed, 36 insertions(+)\n>\n> diff --git a/pack-revindex.c b/pack-revindex.c\n> index ecdde39cf4..6d86a85208 100644\n> --- a/pack-revindex.c\n> +++ b/pack-revindex.c\n> @@ -203,3 +203,35 @@ struct revindex_entry *find_pack_revindex(struct packed_git *p, off_t ofs)\n>  \n>  \treturn p->revindex + pos;\n>  }\n> +\n> +int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n> +{\n> +\tint ret;\n> +\n> +\tif (load_pack_revindex(p) < 0)\n> +\t\treturn -1;\n> +\n> +\tret = find_revindex_position(p, ofs);\n> +\tif (ret < 0)\n> +\t\treturn -1;\n\nWhy not \"return ret\"?  We know that find_revindex_position() would\nsignal an error by returning -1, but is there a reason why we want\nto prevent it from returning richer errors in the future?\n\n> +\t*pos = ret;\n\nThe untold assumption is that uint32_t can fit the maximum returned\nvalue from find_revindex_position() and \"signed int\" can also big\nenough.  I guess it is OK to be limited to up-to 2 billion objects\non 32-bit systems.\n\n> +\treturn 0;\n> +}\n> +\n> +uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos)\n> +{\n> +\tif (!p->revindex)\n> +\t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n\nThe previous function lazy loaded the revindex, but this one and the\nnext one refuses to work without revindex.  Intended?\n\n> +\tif (pos >= p->num_objects)\n> +\t\tBUG(\"pack_pos_to_index: out-of-bounds object at %\"PRIu32, pos);\n\nPersonally I find it easier to place items on a single line in an\nascending order of magnitude, i.e.\n\n\tif (p->num_objects <= pos)\n\t\tBUG(\"...\");\n\nThe assertion requires pos to be strictly lower than p->num_objects,\nwhich is in line with how we usually count elements of an array of\nsize p->num_objects, but the next one allows pos == p->num_objects;\nintended?\n\np->revindex[] is an array of two-member struct, so if an element of\nthe array is invalid for its .nr member here because pos is exactly\nat p->num_objects, I would imagine it is also invalid for its .offset\nmember, too, no?\n\nAh, perhaps the \"offset beyond the end of the pack positions\" is a\nsentinel element to give the in-pack-stream size of the object at\nthe last pack position?  If that is the case, it deserves a comment,\nI would think.\n\n> +\treturn p->revindex[pos].nr;\n> +}\n> +\n> +off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n> +{\n> +\tif (!p->revindex)\n> +\t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n> +\tif (pos > p->num_objects)\n> +\t\tBUG(\"pack_pos_to_offset: out-of-bounds object at %\"PRIu32, pos);\n> +\treturn p->revindex[pos].offset;\n> +}\n> diff --git a/pack-revindex.h b/pack-revindex.h\n> index 848331d5d6..256c0a9106 100644\n> --- a/pack-revindex.h\n> +++ b/pack-revindex.h\n> @@ -13,4 +13,8 @@ int find_revindex_position(struct packed_git *p, off_t ofs);\n>  \n>  struct revindex_entry *find_pack_revindex(struct packed_git *p, off_t ofs);\n>  \n> +int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos);\n> +uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);\n> +off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos);\n> +\n>  #endif\n"},{"id":"414246","messageId":"xmqq4kjlz1qf.fsf@gitster.c.googlers.com","threadId":"54961","inReplyTo":"X/5nsw6uqKDCHGql@nand.local","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-13T08:21:12Z","receivedAt":"2021-01-13T08:22:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> If you agree that the bottom topic is stable, I'd prefer to send the top\n> one separately. Otherwise, I can send both together. Let me know.\n\nI do not expect the first 20 of the 20+8 patches to be stable from\nthe beginning---in fact, after reading 01/20 myself, and seeing a\nfew of Peff's reviews, I expect that you'll be redoing at least some\nof them.\n\nWhen we deal with this kind of situation (not limited to these\ntopics), let's make it a tradition to first pretend that we have a\nlong single topic, and expect that the early rounds of review focus\non two things:\n\n (1) to identify the best ordering of the commits (topics from\n     experienced contributors like you are likely to be already\n     structured in a good order, so reviewers may only have to say\n     \"the ordering looks good\", but sometimes they may want to say\n     thinks like \"it would be better to leave patches 5, 8 and 11 to\n     much later in the series\".\n\n (2) to find good points to divide the series into two (or more)\n     pieces, and spend more effort on helping the bottom part to\n     solidify faster.\n\nThat way, the bottom part can be merged sooner to 'next' than the\nrest.  It always is cumbersome to have some part of the series in\n'next' and remainder in 'seen', so at that point, the lower half\nwould naturally gain a different name before it gets merged to\n'next', I would think.\n\nThanks.\n\n"},{"id":"414248","messageId":"xmqqv9c1xllv.fsf@gitster.c.googlers.com","threadId":"54961","inReplyTo":"xmqqa6tdz2fo.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 01/20] pack-revindex: introduce a new API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-13T08:54:52Z","receivedAt":"2021-01-13T08:55:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Without knowing what exactly \"pack position\", \"offset\" and \"index\n> position\" refer to, the above three are almost impossible to grok.\n> Can we have one paragraph description for each?  Something along the\n> lines of...\n>\n>  - Pack position: a packstream consists of series of byte ranges,\n>    each of which represents an object, so the objects can be\n>    numbered from 0 (the object whose data is stored at the earliest\n>    part in the packfile) to N (the object whose data is stored at\n>    the tail end of the packfile).  The number corresponding to an\n>    object in this order in the packfile is called the \"pack\n>    position\" of the object.\n>\n>  - Offset: The ofs_t distance between the beginning of a pack stream\n>    and the beginning of data that represents an object is called the\n>    \"offset\" of the object in the packfile.\n>\n>  - Index position: for a single pack stream, there is a table that\n>    maps object name to its offset and the entries in this table are\n>    sorted by the object name (this is what pack \".idx\" file is).\n>    The location (counting from 0) of an object in this table is\n>    called the \"index position\" of the object in the packfile.\n>\n> I am not sure if the above correctly reflects what you meant by\n> \"position\", though.\n\nPlease scratch the above.  I see Peff already pointed out pretty\nmuch the same, and his phrasing looked a lot cleaner than my\nattempt.\n\nThanks.\n"},{"id":"414256","messageId":"X/7vb9Dnrfa1Yaut@coredump.intra.peff.net","threadId":"54961","inReplyTo":"X/3O7D18gRLfSNJ8@nand.local","subject":"Re: [PATCH 02/20] write_reuse_object(): convert to new revindex API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-13T13:02:39Z","receivedAt":"2021-01-13T13:03:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 12, 2021 at 11:31:40AM -0500, Taylor Blau wrote:\n\n> > > +\tdatalen = pack_pos_to_offset(p, pos + 1) - offset;\n> >\n> > This \"pos + 1\" means we may be looking one past the end of the array.\n> > That's OK (at least for now), because our revindex always puts in an\n> > extra dummy value exactly for computing these kinds of byte-distances.\n> > That might be worth documenting in the API header.\n> \n> Yeah, I made sure to document that when I was touching up the last\n> patch. FWIW, that's a behavior that we're going to carry over even when\n> the reverse index is stored on-disk (not by writing four extra bytes\n> into the .rev file, but by handling queries for pos == p->num_objects\n> separately.)\n\nI think the only reason to look past the end like that is to compute the\nsize of the final entry. So we _could_ abstract that away from the\ncallers with a separate function like:\n\n  off_t pack_pos_to_size(struct packed_git *p, uint32_t pos);\n\nBut as long as the behavior of passing p->num_objects is documented, I\ndo not mind overly mind spelling it one way or the other.\n\n-Peff\n"},{"id":"414257","messageId":"X/7wCy92mGPZB9PM@coredump.intra.peff.net","threadId":"54961","inReplyTo":"X/3VfteeF3Ok2C+S@nand.local","subject":"Re: [PATCH 18/20] pack-revindex: remove unused 'find_revindex_position()'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-13T13:05:15Z","receivedAt":"2021-01-13T13:06:01Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 12, 2021 at 11:59:42AM -0500, Taylor Blau wrote:\n\n> > I notice these are signed ints, but we've taken care to use uint32_t\n> > elsewhere for positions. Shouldn't these be uint32_t, also (or at least\n> > unsigned)?\n> \n> I'll let both of these be an raw unsigned, since the midpoint is already\n> labeled as an unsigned.\n\nYeah, I found that existing code doubly weird, since \"mi\" must by\ndefinition be bounded by \"lo\" and \"hi\". :)\n\nLooks like the bug was introduced in 92e5c77c37 (revindex: export new\nAPIs, 2013-10-24), which hacked up the existing loop into a new\nfunction. We should have caught it then (back then we were a lot less\ncareful about types, IMHO).\n\nAlso, the subject line of that commit is giving me deja vu. ;)\n\n-Peff\n"},{"id":"414258","messageId":"X/7yFdqUmSmRE8A0@coredump.intra.peff.net","threadId":"54961","inReplyTo":"xmqq4kjlz1qf.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-13T13:13:57Z","receivedAt":"2021-01-13T13:14:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 13, 2021 at 12:21:12AM -0800, Junio C Hamano wrote:\n\n> Taylor Blau <me@ttaylorr.com> writes:\n> \n> > If you agree that the bottom topic is stable, I'd prefer to send the top\n> > one separately. Otherwise, I can send both together. Let me know.\n> \n> I do not expect the first 20 of the 20+8 patches to be stable from\n> the beginning---in fact, after reading 01/20 myself, and seeing a\n> few of Peff's reviews, I expect that you'll be redoing at least some\n> of them.\n\nThey'll definitely need at least one re-roll. But I think Taylor is\nexpecting (and I do too) that the second half will probably have a lot\nmore back-and-forth over the on-disk format, and hence need more\nre-rolls.\n\nMy main concern is reviewer fatigue. 28 patches is a lot. If we can\nsolidify the first 20 and then let people focus on the final 8\nseparately, that helps. If you're OK with splitting a topic and saying\n\"this is a re-roll of just the last 8 patches\", then that problem is\nsolved. But IMHO it is easier to just point out that split from the\nstart than it is to come up with it after the fact. It tells reviewers\nwhat to expect from the get-go.\n\n>  (2) to find good points to divide the series into two (or more)\n>      pieces, and spend more effort on helping the bottom part to\n>      solidify faster.\n\nI think we just did that preemptively. ;) In these two particular\nseries, the first 20 (or at least the first 19) are an improvement by\nthemselves. I think they would be worth doing even without the final 8,\nboth to make calling code more readable and to add new error checks and\nassertions to revindex access.\n\n> That way, the bottom part can be merged sooner to 'next' than the\n> rest.  It always is cumbersome to have some part of the series in\n> 'next' and remainder in 'seen', so at that point, the lower half\n> would naturally gain a different name before it gets merged to\n> 'next', I would think.\n\nThat seems to me like it ends up being _more_ work than just making them\ninto two branches in the first place.\n\nSo I guess I remain skeptical that ad-hoc splitting of longer series is\neasier than doing so up front. But you're the one who does all of the\nbranch shuffling in the end, so if you really prefer longer series, I'm\nnot what I think matters that much. ;)\n\n-Peff\n"},{"id":"414259","messageId":"X/7zA3KjlNnS2HhJ@coredump.intra.peff.net","threadId":"54961","inReplyTo":"xmqqa6tdz2fo.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 01/20] pack-revindex: introduce a new API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-13T13:17:55Z","receivedAt":"2021-01-13T13:18:38Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 13, 2021 at 12:06:03AM -0800, Junio C Hamano wrote:\n\n> > +int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n> > +{\n> > +\tint ret;\n> [...]\n> > +\t*pos = ret;\n> \n> The untold assumption is that uint32_t can fit the maximum returned\n> value from find_revindex_position() and \"signed int\" can also big\n> enough.  I guess it is OK to be limited to up-to 2 billion objects\n> on 32-bit systems.\n\nThanks for pointing this out. I recalled there being an \"int\" problem\nsomewhere in the revindex code, but I didn't notice it on my\nread-through. This bug already exists (the problem is actually in the\nfind_revindex_position() interface), and is fixed when we inline that\ninto offset_to_pack_pos() in patch 18.\n\nIt might be worth mentioning the fix there.\n\n-Peff\n"},{"id":"414271","messageId":"X/8THO3ck3bjJH+K@nand.local","threadId":"54961","inReplyTo":"X/7yFdqUmSmRE8A0@coredump.intra.peff.net","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T15:34:52Z","receivedAt":"2021-01-13T15:35:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 13, 2021 at 08:13:57AM -0500, Jeff King wrote:\n> On Wed, Jan 13, 2021 at 12:21:12AM -0800, Junio C Hamano wrote:\n>\n> > Taylor Blau <me@ttaylorr.com> writes:\n> >\n> > > If you agree that the bottom topic is stable, I'd prefer to send the top\n> > > one separately. Otherwise, I can send both together. Let me know.\n> >\n> > I do not expect the first 20 of the 20+8 patches to be stable from\n> > the beginning---in fact, after reading 01/20 myself, and seeing a\n> > few of Peff's reviews, I expect that you'll be redoing at least some\n> > of them.\n>\n> They'll definitely need at least one re-roll. But I think Taylor is\n> expecting (and I do too) that the second half will probably have a lot\n> more back-and-forth over the on-disk format, and hence need more\n> re-rolls.\n\nIndeed, I find the first ~20 patches fairly benign, and I think that the\ninteresting discussion will and should take place over the final 8\npatches.\n\nFor what it's worth, I was referring to the pending re-roll I have of\nthe first 20 patches as the stable one. (Peff notes in [1] that he\nthought there wasn't much else to consider beyond his comments.)\n\n> My main concern is reviewer fatigue. 28 patches is a lot. If we can\n> solidify the first 20 and then let people focus on the final 8\n> separately, that helps. If you're OK with splitting a topic and saying\n> \"this is a re-roll of just the last 8 patches\", then that problem is\n> solved. But IMHO it is easier to just point out that split from the\n> start than it is to come up with it after the fact. It tells reviewers\n> what to expect from the get-go.\n\nYes, exactly.\n\n> > That way, the bottom part can be merged sooner to 'next' than the\n> > rest.  It always is cumbersome to have some part of the series in\n> > 'next' and remainder in 'seen', so at that point, the lower half\n> > would naturally gain a different name before it gets merged to\n> > 'next', I would think.\n>\n> That seems to me like it ends up being _more_ work than just making them\n> into two branches in the first place.\n\nI agree, but I also wasn't aware that you would consider queuing part of\na series. If that's the route you want to take, I'm OK with that. But I\ntend to agree with Peff that (in this case since a clear deliniation\nalready exists) it may save us time to just send two separate series\nfrom the get-go.\n\n> So I guess I remain skeptical that ad-hoc splitting of longer series is\n> easier than doing so up front. But you're the one who does all of the\n> branch shuffling in the end, so if you really prefer longer series, I'm\n> not what I think matters that much. ;)\n\nAgreed. Junio, let me know which you'd prefer in this case (I'm not sure\nif the additional context has changed your mind or not).\n\n> -Peff\n\nThanks,\nTaylor\n\n[1]: https://lore.kernel.org/git/X%2F1vy3D10wDEZNva@coredump.intra.peff.net/\n"},{"id":"414273","messageId":"X/8egmj9Tno3pvhC@nand.local","threadId":"54961","inReplyTo":"xmqqa6tdz2fo.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 01/20] pack-revindex: introduce a new API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T16:23:30Z","receivedAt":"2021-01-13T16:24:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 13, 2021 at 12:06:03AM -0800, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> > In the next several patches, we will prepare for loading a reverse index\n> > either in memory, or from a yet-to-be-introduced on-disk format. To do\n>\n> Does \"load revindex in memory\" (as opposed to \"from on-disk file\")\n> mean the good old \"read the forward index and make inverse map\n> in-core\", or something else?\n\nIndeed, that's what it means. I've made that clearer in the patch\nmessage by saying it explicitly.\n\n> IOW, is \"We will prepare a reverse index either by computing in\n> memory from forward index, or loading from on-disk file\" what we\n> want to say here?\n\nYep.\n\n> Without knowing what exactly \"pack position\", \"offset\" and \"index\n> position\" refer to, the above three are almost impossible to grok.\n> Can we have one paragraph description for each?  Something along the\n> lines of...\n\nYep, and I see later on in the thread that you want to discard this\nsuggestion since Peff has suggested similar changes in pack-revindex.h,\nwhich I've applied.\n\n> >     Unlike some of the callers that used to access '->offset' and '->nr'\n> >     directly, the error checking around this call is somewhat more\n> >     robust. This is important since callers can pass an offset which\n> >     does not contain an object.\n>\n> Meaning \"offset ought to point at the boundary between objects in\n> the pack stream, and the API, unlike the direct access, makes sure\n> that is the case\"?  That is a good thing.\n\nIndeed, and I've called that out directly in the patch message to\nhighlight it.\n\n> > diff --git a/pack-revindex.c b/pack-revindex.c\n> > index ecdde39cf4..6d86a85208 100644\n> > --- a/pack-revindex.c\n> > +++ b/pack-revindex.c\n> > @@ -203,3 +203,35 @@ struct revindex_entry *find_pack_revindex(struct packed_git *p, off_t ofs)\n> >\n> >  \treturn p->revindex + pos;\n> >  }\n> > +\n> > +int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n> > +{\n> > +\tint ret;\n> > +\n> > +\tif (load_pack_revindex(p) < 0)\n> > +\t\treturn -1;\n> > +\n> > +\tret = find_revindex_position(p, ofs);\n> > +\tif (ret < 0)\n> > +\t\treturn -1;\n>\n> Why not \"return ret\"?  We know that find_revindex_position() would\n> signal an error by returning -1, but is there a reason why we want\n> to prevent it from returning richer errors in the future?\n\nNo reason, I've changed it to 'return ret' instead.\n\n> > +\t*pos = ret;\n>\n> The untold assumption is that uint32_t can fit the maximum returned\n> value from find_revindex_position() and \"signed int\" can also big\n> enough.  I guess it is OK to be limited to up-to 2 billion objects\n> on 32-bit systems.\n\nIndeed, and that is fixed in a later on patch. I'll make sure to call it\nout there.\n\n> > +\treturn 0;\n> > +}\n> > +\n> > +uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos)\n> > +{\n> > +\tif (!p->revindex)\n> > +\t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n>\n> The previous function lazy loaded the revindex, but this one and the\n> next one refuses to work without revindex.  Intended?\n\nYes, and discussed a little bit more here [1]. Obviously that discussion\ndoesn't do any good to those reading the git log in the future, so I've\nsummarized the important detail (that some callers are equipped to deal\nwith errors but others aren't) in the patch.\n\n> > +\tif (pos >= p->num_objects)\n> > +\t\tBUG(\"pack_pos_to_index: out-of-bounds object at %\"PRIu32, pos);\n>\n> Personally I find it easier to place items on a single line in an\n> ascending order of magnitude, i.e.\n>\n> \tif (p->num_objects <= pos)\n> \t\tBUG(\"...\");\n\nMore readable, thanks.\n\n> The assertion requires pos to be strictly lower than p->num_objects,\n> which is in line with how we usually count elements of an array of\n> size p->num_objects, but the next one allows pos == p->num_objects;\n> intended?\n>\n> p->revindex[] is an array of two-member struct, so if an element of\n> the array is invalid for its .nr member here because pos is exactly\n> at p->num_objects, I would imagine it is also invalid for its .offset\n> member, too, no?\n>\n> Ah, perhaps the \"offset beyond the end of the pack positions\" is a\n> sentinel element to give the in-pack-stream size of the object at\n> the last pack position?  If that is the case, it deserves a comment,\n> I would think.\n\nExactly. I added a detail about that in the patch, too.\n\nThanks,\nTaylor\n\n[1]: https://lore.kernel.org/git/X%2F3ODgaa9wr65M09@nand.local/\n"},{"id":"414283","messageId":"xmqqft34y53j.fsf@gitster.c.googlers.com","threadId":"54961","inReplyTo":"X/8THO3ck3bjJH+K@nand.local","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-13T20:06:08Z","receivedAt":"2021-01-13T20:07:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n>> > That way, the bottom part can be merged sooner to 'next' than the\n>> > rest.  It always is cumbersome to have some part of the series in\n>> > 'next' and remainder in 'seen', so at that point, the lower half\n>> > would naturally gain a different name before it gets merged to\n>> > 'next', I would think.\n>>\n>> That seems to me like it ends up being _more_ work than just making them\n>> into two branches in the first place.\n\nMore work to contributors?  How?\n\nAs long as each of 20-patch and 8-patch series is marked clearly to\nmanage the risk of mistakes and confusion down to the same level as\na single long series, I am perfectly OK.\n\nExamples of help contributors could have made, which would have\navoided past confusion (these are not \"potential\" ones, but I had to\nredo day's intergration in the past because of one long topic\nbuilding on top of another) are:\n\n - When sending either topic, not limited to the initial round but\n   in all the subsequent rounds, remind that the top topic is to be\n   applied on top of the bottom topic.\n\n - When updating the bottom topic (e.g. 20-patch one in this case),\n   send out the top one (e.g. 8-patch one), too (or instruct me to\n   discard the top one tentatively).\n\nThe worst case that happened in the past was that a quite minor\ntweak was made to a bottom topic that was depended on another topic,\nso I just queued the new iteration of the bottom topic again,\nwithout realizing that the other one needed to be rebased.  We ended\nup two copies of the bottom topic commits in 'pu' (these days we\ncall it 'seen') as the tweak was so minor that the two topics\ncleanly merged into 'pu' without causing conflict.  The next bad\ncase was a similar situation with larger rewrite of the bottom\ntopic, which caused me to look at quite a big conflict and waste my\ntime until I realized that I was missing an updated top half.\n\nIf the inter-dependent topics that caused me trouble were managed as\na single long patch series, either with \"this is a full replacement\nof the new iteration\" or \"these are to update only the last 8\npatches; apply them after rewinding the topic to commit f0e1d2c3\n(gostak: distim doshes, 2021-01-08)\", would have had a lot less risk\nto introduce human error on this end.\n\n> I agree, but I also wasn't aware that you would consider queuing part of\n> a series. If that's the route you want to take, I'm OK with that.\n\nDiscarding broken part of a series and only queuing a good part can\nhappen with or without multiple topics.  Merging one topic to 'next'\nbut not the other also happens.  Merging early part of a topic to\n'next' while leaving the rest to 'seen' is possible but I'd prefer\nto avoid it.  Because of the last one, a single long topic, when a\nbottom part stabilizes enough, would likely to gain a separate name\nand its tip would be merged to 'next'.\n\n> But I\n> tend to agree with Peff that (in this case since a clear deliniation\n> already exists) it may save us time to just send two separate series\n> from the get-go.\n\nAs long as the two serieses are marked as such clearly, not just in\nthe initial round but in all subsequent rounds, it is OK.  But in an\nunproven initial round, you may regret having to move a patch across\ntopics, from the bottom one to the top one or vice versa, instead of\njust reordering inside a single topic.\n\n>> So I guess I remain skeptical that ad-hoc splitting of longer series is\n>> easier than doing so up front.\n\nNobody suggested ad-hoc splitting.  I was saying that splitting\nwould naturally grow out of reviews toward stabilization.\n"},{"id":"414285","messageId":"X/9UXrCzihwa+ivu@nand.local","threadId":"54961","inReplyTo":"xmqqft34y53j.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T20:13:18Z","receivedAt":"2021-01-13T20:14:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 13, 2021 at 12:06:08PM -0800, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> > But I tend to agree with Peff that (in this case since a clear\n> > deliniation already exists) it may save us time to just send two\n> > separate series from the get-go.\n>\n> As long as the two serieses are marked as such clearly, not just in\n> the initial round but in all subsequent rounds, it is OK.  But in an\n> unproven initial round, you may regret having to move a patch across\n> topics, from the bottom one to the top one or vice versa, instead of\n> just reordering inside a single topic.\n\nSounds good. Like I said, the last thing I want to do is create undue\nburden on your or anybody else when queuing or reviewing these topics.\n\nSo, I'll try to err on the side of stating the dependency between topics\ntoo often rather than not often enough. Hopefully the first ~20 patches\nare boring enough that they will make their way to master soon enough\nand we don't have to worry about it.\n\nOrdinarily, I would have held off on the second series until more the\nfirst one had graduated, but I felt that the pure refactorings didn't\nmake much sense on their own without the new file-based backend to\nmotivate them.\n\nThanks,\nTaylor\n"},{"id":"414287","messageId":"X/9WkdHzSW9jAJ3k@coredump.intra.peff.net","threadId":"54961","inReplyTo":"xmqqft34y53j.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-13T20:22:41Z","receivedAt":"2021-01-13T20:23:39Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 13, 2021 at 12:06:08PM -0800, Junio C Hamano wrote:\n\n> Taylor Blau <me@ttaylorr.com> writes:\n> \n> >> > That way, the bottom part can be merged sooner to 'next' than the\n> >> > rest.  It always is cumbersome to have some part of the series in\n> >> > 'next' and remainder in 'seen', so at that point, the lower half\n> >> > would naturally gain a different name before it gets merged to\n> >> > 'next', I would think.\n> >>\n> >> That seems to me like it ends up being _more_ work than just making them\n> >> into two branches in the first place.\n> \n> More work to contributors?  How?\n\nThe quoted part is from me, so I'll respond: I didn't mean contributors,\nbut it seems like more work to you. I.e., you are ending up with the\nsame multi-branch config _and_ you have to split the branches yourself\nlater after seeing review.\n\nBut reading what you wrote below, the advantage is that if this does not\nhappen until the first part hits \"next\", then there is no chance of it\nbeing rebased at that point (and thus getting rewritten out from under\nthe second topic).\n\n> The worst case that happened in the past was that a quite minor\n> tweak was made to a bottom topic that was depended on another topic,\n> so I just queued the new iteration of the bottom topic again,\n> without realizing that the other one needed to be rebased.  We ended\n> up two copies of the bottom topic commits in 'pu' (these days we\n> call it 'seen') as the tweak was so minor that the two topics\n> cleanly merged into 'pu' without causing conflict.  The next bad\n> case was a similar situation with larger rewrite of the bottom\n> topic, which caused me to look at quite a big conflict and waste my\n> time until I realized that I was missing an updated top half.\n\nI somehow assumed you had more automation there. On my end, since I\nrebase my topics aggressively, it is just a matter of pointing the\nbranch upstream in the right place. But of course that is not your\nworkflow at all.\n\nI know you do have the \"this branches uses\" logic in your what's cooking\nemails. In theory it could remind you of the situation, but I'm not sure\nwhere in the workflow you'd insert it (by the time you run the WC\nscript, it is hard to realize the rebasing that _should_ have been done\nearlier, unless you collate patch-ids, and even that is not 100%).\n\nI do wonder if setting the dependent branch's @{upstream} would be\nhelpful here. You do not rebase all of your topics, but the ones with a\nlocal-branch @{u} would be candidates for doing so.\n\nAll that said, I am also sensitive that my armchair \"you could do it\nlike this...\" suggesting may not be fully informed. So take it as idle\nthoughts, not necessarily arguments. :)\n\n> >> So I guess I remain skeptical that ad-hoc splitting of longer series is\n> >> easier than doing so up front.\n> \n> Nobody suggested ad-hoc splitting.  I was saying that splitting\n> would naturally grow out of reviews toward stabilization.\n\nThis quote is me again. By \"ad-hoc\" I meant exactly this \"after we see\nsome reviews\" (as opposed to drawing a line up front).\n\n-Peff\n"},{"id":"414294","messageId":"7676822a541bfef1861be01dc55d86d3d0cad494.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 03/20] write_reused_pack_one(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:23:39Z","receivedAt":"2021-01-14T02:05:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Replace direct revindex accesses with calls to 'pack_pos_to_offset()'\nand 'pack_pos_to_index()'.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/pack-objects.c | 14 ++++++++++----\n 1 file changed, 10 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ab1fd853f1..8e40b19ee8 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -868,8 +868,8 @@ static void write_reused_pack_one(size_t pos, struct hashfile *out,\n \tenum object_type type;\n \tunsigned long size;\n \n-\toffset = reuse_packfile->revindex[pos].offset;\n-\tnext = reuse_packfile->revindex[pos + 1].offset;\n+\toffset = pack_pos_to_offset(reuse_packfile, pos);\n+\tnext = pack_pos_to_offset(reuse_packfile, pos + 1);\n \n \trecord_reused_object(offset, offset - hashfile_total(out));\n \n@@ -889,11 +889,17 @@ static void write_reused_pack_one(size_t pos, struct hashfile *out,\n \n \t\t/* Convert to REF_DELTA if we must... */\n \t\tif (!allow_ofs_delta) {\n-\t\t\tint base_pos = find_revindex_position(reuse_packfile, base_offset);\n+\t\t\tuint32_t base_pos;\n \t\t\tstruct object_id base_oid;\n \n+\t\t\tif (offset_to_pack_pos(reuse_packfile, base_offset, &base_pos) < 0)\n+\t\t\t\tdie(_(\"expected object at offset %\"PRIuMAX\" \"\n+\t\t\t\t      \"in pack %s\"),\n+\t\t\t\t    (uintmax_t)base_offset,\n+\t\t\t\t    reuse_packfile->pack_name);\n+\n \t\t\tnth_packed_object_id(&base_oid, reuse_packfile,\n-\t\t\t\t\t     reuse_packfile->revindex[base_pos].nr);\n+\t\t\t\t\t     pack_pos_to_index(reuse_packfile, base_pos));\n \n \t\t\tlen = encode_in_pack_object_header(header, sizeof(header),\n \t\t\t\t\t\t\t   OBJ_REF_DELTA, size);\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414295","messageId":"e1aa89244ad3edb52aaeb28d6934cb2b0a0dc65a.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 01/20] pack-revindex: introduce a new API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:23:31Z","receivedAt":"2021-01-14T02:05:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In the next several patches, we will prepare for loading a reverse index\neither in memory (mapping the inverse of the .idx's contents in-core),\nor directly from a yet-to-be-introduced on-disk format. To prepare for\nthat, we'll introduce an API that avoids the caller explicitly indexing\nthe revindex pointer in the packed_git structure.\n\nThere are four ways to interact with the reverse index. Accordingly,\nfour functions will be exported from 'pack-revindex.h' by the time that\nthe existing API is removed. A caller may:\n\n 1. Load the pack's reverse index. This involves opening up the index,\n    generating an array, and then sorting it. Since opening the index\n    can fail, this function ('load_pack_revindex()') returns an int.\n    Accordingly, it takes only a single argument: the 'struct\n    packed_git' the caller wants to build a reverse index for.\n\n    This function is well-suited for both the current and new API.\n    Callers will have to continue to open the reverse index explicitly,\n    but this function will eventually learn how to detect and load a\n    reverse index from the on-disk format, if one exists. Otherwise, it\n    will fallback to generating one in memory from scratch.\n\n 2. Convert a pack position into an offset. This operation is now\n    called `pack_pos_to_offset()`. It takes a pack and a position, and\n    returns the corresponding off_t.\n\n    Any error simply calls BUG(), since the callers are not well-suited\n    to handle a failure and keep going.\n\n 3. Convert a pack position into an index position. Same as above; this\n    takes a pack and a position, and returns a uint32_t. This operation\n    is known as `pack_pos_to_index()`. The same thinking about error\n    conditions applies here as well.\n\n 4. Find the pack position for a given offset. This operation is now\n    known as `offset_to_pack_pos()`. It takes a pack, an offset, and a\n    pointer to a uint32_t where the position is written, if an object\n    exists at that offset. Otherwise, -1 is returned to indicate\n    failure.\n\n    Unlike some of the callers that used to access '->offset' and '->nr'\n    directly, the error checking around this call is somewhat more\n    robust. This is important since callers should always pass an offset\n    which points at the boundary of two objects. The API, unlike direct\n    access, enforces that that is the case.\n\n    This will become important in a subsequent patch where a caller\n    which does not but could check the return value treats the signed\n    `-1` from `find_revindex_position()` as an index into the 'revindex'\n    array.\n\nTwo design warts are carried over into the new API:\n\n  - Asking for the index position of an out-of-bounds object will result\n    in a BUG() (since no such object exists), but asking for the offset\n    of the non-existent object at the end of the pack returns the total\n    size of the pack.\n\n    This makes it convenient for callers who always want to take the\n    difference of two adjacent object's offsets (to compute the on-disk\n    size) but don't want to worry about boundaries at the end of the\n    pack.\n\n  - offset_to_pack_pos() lazily loads the reverse index, but\n    pack_pos_to_index() doesn't (callers of the former are well-suited\n    to handle errors, but callers of the latter are not).\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-revindex.c | 32 +++++++++++++++++++++++++++++\n pack-revindex.h | 54 +++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 86 insertions(+)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex ecdde39cf4..0ca3b54b45 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -203,3 +203,35 @@ struct revindex_entry *find_pack_revindex(struct packed_git *p, off_t ofs)\n \n \treturn p->revindex + pos;\n }\n+\n+int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n+{\n+\tint ret;\n+\n+\tif (load_pack_revindex(p) < 0)\n+\t\treturn -1;\n+\n+\tret = find_revindex_position(p, ofs);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\t*pos = ret;\n+\treturn 0;\n+}\n+\n+uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos)\n+{\n+\tif (!p->revindex)\n+\t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n+\tif (p->num_objects <= pos)\n+\t\tBUG(\"pack_pos_to_index: out-of-bounds object at %\"PRIu32, pos);\n+\treturn p->revindex[pos].nr;\n+}\n+\n+off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos)\n+{\n+\tif (!p->revindex)\n+\t\tBUG(\"pack_pos_to_index: reverse index not yet loaded\");\n+\tif (p->num_objects < pos)\n+\t\tBUG(\"pack_pos_to_offset: out-of-bounds object at %\"PRIu32, pos);\n+\treturn p->revindex[pos].offset;\n+}\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex 848331d5d6..5a218aaa66 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -1,6 +1,21 @@\n #ifndef PACK_REVINDEX_H\n #define PACK_REVINDEX_H\n \n+/**\n+ * A revindex allows converting efficiently between three properties\n+ * of an object within a pack:\n+ *\n+ * - index position: the numeric position within the list of sorted object ids\n+ *   found in the .idx file\n+ *\n+ * - pack position: the numeric position within the list of objects in their\n+ *   order within the actual .pack file (i.e., 0 is the first object in the\n+ *   .pack, 1 is the second, and so on)\n+ *\n+ * - offset: the byte offset within the .pack file at which the object contents\n+ *   can be found\n+ */\n+\n struct packed_git;\n \n struct revindex_entry {\n@@ -8,9 +23,48 @@ struct revindex_entry {\n \tunsigned int nr;\n };\n \n+/*\n+ * load_pack_revindex populates the revindex's internal data-structures for the\n+ * given pack, returning zero on success and a negative value otherwise.\n+ */\n int load_pack_revindex(struct packed_git *p);\n int find_revindex_position(struct packed_git *p, off_t ofs);\n \n struct revindex_entry *find_pack_revindex(struct packed_git *p, off_t ofs);\n \n+/*\n+ * offset_to_pack_pos converts an object offset to a pack position. This\n+ * function returns zero on success, and a negative number otherwise. The\n+ * parameter 'pos' is usable only on success.\n+ *\n+ * If the reverse index has not yet been loaded, this function loads it lazily,\n+ * and returns an negative number if an error was encountered.\n+ *\n+ * This function runs in time O(log N) with the number of objects in the pack.\n+ */\n+int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos);\n+\n+/*\n+ * pack_pos_to_index converts the given pack-relative position 'pos' by\n+ * returning an index-relative position.\n+ *\n+ * If the reverse index has not yet been loaded, or the position is out of\n+ * bounds, this function aborts.\n+ *\n+ * This function runs in constant time.\n+ */\n+uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);\n+\n+/*\n+ * pack_pos_to_offset converts the given pack-relative position 'pos' into a\n+ * pack offset. For a pack with 'N' objects, asking for position 'N' will return\n+ * the total size (in bytes) of the pack.\n+ *\n+ * If the reverse index has not yet been loaded, or the position is out of\n+ * bounds, this function aborts.\n+ *\n+ * This function runs in constant time.\n+ */\n+off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos);\n+\n #endif\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414297","messageId":"8400ff6c9615b4c999b198c46b2e673ec0f2b14f.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 20/20] pack-revindex.c: avoid direct revindex access in 'offset_to_pack_pos()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:25:10Z","receivedAt":"2021-01-14T02:05:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To prepare for on-disk reverse indexes, remove a spot in\n'offset_to_pack_pos()' that looks at the 'revindex' array in 'struct\npacked_git'.\n\nEven though this use of the revindex pointer is within pack-revindex.c,\nthis clean up is still worth doing. Since the 'revindex' pointer will be\nNULL when reading from an on-disk reverse index (instead the\n'revindex_data' pointer will be mmaped to the 'pack-*.rev' file), this\ncall-site would have to include a conditional to lookup the offset for\nposition 'mi' each iteration through the search.\n\nSo instead of open-coding 'pack_pos_to_offset()', call it directly from\nwithin 'offset_to_pack_pos()'.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-revindex.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex a508d5f0a4..5e69bc7372 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -177,21 +177,21 @@ int load_pack_revindex(struct packed_git *p)\n int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n {\n \tunsigned lo, hi;\n-\tconst struct revindex_entry *revindex;\n \n \tif (load_pack_revindex(p) < 0)\n \t\treturn -1;\n \n \tlo = 0;\n \thi = p->num_objects + 1;\n-\trevindex = p->revindex;\n \n \tdo {\n \t\tconst unsigned mi = lo + (hi - lo) / 2;\n-\t\tif (revindex[mi].offset == ofs) {\n+\t\toff_t got = pack_pos_to_offset(p, mi);\n+\n+\t\tif (got == ofs) {\n \t\t\t*pos = mi;\n \t\t\treturn 0;\n-\t\t} else if (ofs < revindex[mi].offset)\n+\t\t} else if (ofs < got)\n \t\t\thi = mi;\n \t\telse\n \t\t\tlo = mi + 1;\n-- \n2.30.0.138.g6d7191ea01\n"},{"id":"414298","messageId":"acd80069a2bea5cede6b68302b7ff8097924dcd0.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 09/20] try_partial_reuse(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:24:05Z","receivedAt":"2021-01-14T02:05:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Remove another instance of direct revindex manipulation by calling\n'pack_pos_to_offset()' instead (the caller here does not care about the\nindex position of the object at position 'pos').\n\nNote that we cannot just use the existing \"offset\" variable to store the\nvalue we get from pack_pos_to_offset(). It is incremented by\nunpack_object_header(), but we later need the original value. Since\nwe'll no longer have revindex->offset to read it from, we'll store that\nin a separate variable (\"header\" since it points to the entry's header\nbytes).\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 13 +++++--------\n 1 file changed, 5 insertions(+), 8 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 89a528a91b..1fdf7ce20a 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1069,23 +1069,21 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t\t      struct bitmap *reuse,\n \t\t\t      struct pack_window **w_curs)\n {\n-\tstruct revindex_entry *revidx;\n-\toff_t offset;\n+\toff_t offset, header;\n \tenum object_type type;\n \tunsigned long size;\n \n \tif (pos >= bitmap_git->pack->num_objects)\n \t\treturn; /* not actually in the pack */\n \n-\trevidx = &bitmap_git->pack->revindex[pos];\n-\toffset = revidx->offset;\n+\toffset = header = pack_pos_to_offset(bitmap_git->pack, pos);\n \ttype = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size);\n \tif (type < 0)\n \t\treturn; /* broken packfile, punt */\n \n \tif (type == OBJ_REF_DELTA || type == OBJ_OFS_DELTA) {\n \t\toff_t base_offset;\n-\t\tint base_pos;\n+\t\tuint32_t base_pos;\n \n \t\t/*\n \t\t * Find the position of the base object so we can look it up\n@@ -1096,11 +1094,10 @@ static void try_partial_reuse(struct bitmap_index *bitmap_git,\n \t\t * more detail.\n \t\t */\n \t\tbase_offset = get_delta_base(bitmap_git->pack, w_curs,\n-\t\t\t\t\t     &offset, type, revidx->offset);\n+\t\t\t\t\t     &offset, type, header);\n \t\tif (!base_offset)\n \t\t\treturn;\n-\t\tbase_pos = find_revindex_position(bitmap_git->pack, base_offset);\n-\t\tif (base_pos < 0)\n+\t\tif (offset_to_pack_pos(bitmap_git->pack, base_offset, &base_pos) < 0)\n \t\t\treturn;\n \n \t\t/*\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414299","messageId":"e7574763513294b71071b032d5cd3aa0976969dd.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 15/20] for_each_object_in_pack(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:24:49Z","receivedAt":"2021-01-14T02:05:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Avoid looking at the 'revindex' pointer directly and instead call\n'pack_pos_to_index()'.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n packfile.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 936ab3def5..7bb1750934 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2086,7 +2086,7 @@ int for_each_object_in_pack(struct packed_git *p,\n \t\tstruct object_id oid;\n \n \t\tif (flags & FOR_EACH_OBJECT_PACK_ORDER)\n-\t\t\tpos = p->revindex[i].nr;\n+\t\t\tpos = pack_pos_to_index(p, i);\n \t\telse\n \t\t\tpos = i;\n \n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414300","messageId":"41b2e00947bdac416e5f599dc50ebf4b0e3e238b.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 13/20] packed_object_info(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:24:41Z","receivedAt":"2021-01-14T02:05:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Convert another call of 'find_pack_revindex()' to its replacement\n'pack_pos_to_offset()'. Likewise:\n\n  - Avoid manipulating `struct packed_git`'s `revindex` pointer directly\n    by removing the pointer-as-array indexing.\n\n  - Add an additional guard to check that the offset 'obj_offset()'\n    points to a real object. This should be the case with well-behaved\n    callers to 'packed_object_info()', but isn't guarenteed.\n\n    Other blocks that fill in various other values from the 'struct\n    object_info' request handle bad inputs by setting the type to\n    'OBJ_BAD' and jumping to 'out'. Do the same when given a bad offset\n    here.\n\n    The previous code would have segfaulted when given a bad\n    'obj_offset' value, since 'find_pack_revindex()' would return\n    'NULL', and then the line that fills 'oi->disk_sizep' would try to\n    access 'NULL[1]' with a stride of 16 bytes (the width of 'struct\n    revindex_entry)'.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n packfile.c | 11 +++++++++--\n 1 file changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 7c37f9ec5c..bb4bb14671 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1537,8 +1537,15 @@ int packed_object_info(struct repository *r, struct packed_git *p,\n \t}\n \n \tif (oi->disk_sizep) {\n-\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n-\t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n+\t\tuint32_t pos;\n+\t\tif (offset_to_pack_pos(p, obj_offset, &pos) < 0) {\n+\t\t\terror(\"could not find object at offset %\"PRIuMAX\" \"\n+\t\t\t      \"in pack %s\", (uintmax_t)obj_offset, p->pack_name);\n+\t\t\ttype = OBJ_BAD;\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\t*oi->disk_sizep = pack_pos_to_offset(p, pos + 1) - obj_offset;\n \t}\n \n \tif (oi->typep || oi->type_name) {\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414301","messageId":"a500311e33a2f7e11a539dd0718ed946f4bd6bc8.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 16/20] builtin/gc.c: guess the size of the revindex","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:24:54Z","receivedAt":"2021-01-14T02:05:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"'estimate_repack_memory()' takes into account the amount of memory\nrequired to load the reverse index in memory by multiplying the assumed\nnumber of objects by the size of the 'revindex_entry' struct.\n\nPrepare for hiding the definition of 'struct revindex_entry' by removing\na 'sizeof()' of that type from outside of pack-revindex.c. Instead,\nguess that one off_t and one uint32_t are required per object. Strictly\nspeaking, this is a worse guess than asking for 'sizeof(struct\nrevindex_entry)' directly, since the true size of this struct is 16\nbytes with padding on the end of the struct in order to align the offset\nfield.\n\nBut, this is an approximation anyway, and it does remove a use of the\n'struct revindex_entry' from outside of pack-revindex internals.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/gc.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 4c24f41852..c60811f212 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -301,7 +301,7 @@ static uint64_t estimate_repack_memory(struct packed_git *pack)\n \t/* and then obj_hash[], underestimated in fact */\n \theap += sizeof(struct object *) * nr_objects;\n \t/* revindex is used also */\n-\theap += sizeof(struct revindex_entry) * nr_objects;\n+\theap += (sizeof(off_t) + sizeof(uint32_t)) * nr_objects;\n \t/*\n \t * read_sha1_file() (either at delta calculation phase, or\n \t * writing phase) also fills up the delta base cache\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414302","messageId":"cabafce4a105d4a09e561c05a0a4e14581e0e04f.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 19/20] pack-revindex: hide the definition of 'revindex_entry'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:25:06Z","receivedAt":"2021-01-14T02:06:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that all spots outside of pack-revindex.c that reference 'struct\nrevindex_entry' directly have been removed, it is safe to hide the\nimplementation by moving it from pack-revindex.h to pack-revindex.c.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-revindex.c | 5 +++++\n pack-revindex.h | 5 -----\n 2 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 282fe92640..a508d5f0a4 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -3,6 +3,11 @@\n #include \"object-store.h\"\n #include \"packfile.h\"\n \n+struct revindex_entry {\n+\toff_t offset;\n+\tunsigned int nr;\n+};\n+\n /*\n  * Pack index for existing packs give us easy access to the offsets into\n  * corresponding pack file where each object's data starts, but the entries\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex 746776be7f..6e0320b08b 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -18,11 +18,6 @@\n \n struct packed_git;\n \n-struct revindex_entry {\n-\toff_t offset;\n-\tunsigned int nr;\n-};\n-\n /*\n  * load_pack_revindex populates the revindex's internal data-structures for the\n  * given pack, returning zero on success and a negative value otherwise.\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414303","messageId":"67d14da04a147e1805807e405cbecf91e831f5cc.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 17/20] pack-revindex: remove unused 'find_pack_revindex()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:24:58Z","receivedAt":"2021-01-14T02:06:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that no callers of 'find_pack_revindex()' remain, remove the\nfunction's declaration and implementation entirely.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-revindex.c | 15 ---------------\n pack-revindex.h |  2 --\n 2 files changed, 17 deletions(-)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 0ca3b54b45..16baafb281 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -189,21 +189,6 @@ int find_revindex_position(struct packed_git *p, off_t ofs)\n \treturn -1;\n }\n \n-struct revindex_entry *find_pack_revindex(struct packed_git *p, off_t ofs)\n-{\n-\tint pos;\n-\n-\tif (load_pack_revindex(p))\n-\t\treturn NULL;\n-\n-\tpos = find_revindex_position(p, ofs);\n-\n-\tif (pos < 0)\n-\t\treturn NULL;\n-\n-\treturn p->revindex + pos;\n-}\n-\n int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n {\n \tint ret;\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex 5a218aaa66..f7094ba9a5 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -30,8 +30,6 @@ struct revindex_entry {\n int load_pack_revindex(struct packed_git *p);\n int find_revindex_position(struct packed_git *p, off_t ofs);\n \n-struct revindex_entry *find_pack_revindex(struct packed_git *p, off_t ofs);\n-\n /*\n  * offset_to_pack_pos converts an object offset to a pack position. This\n  * function returns zero on success, and a negative number otherwise. The\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414304","messageId":"569acdca7f4a5aa25b625280266dfe40dc4fdefe.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 10/20] rebuild_existing_bitmaps(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:24:27Z","receivedAt":"2021-01-14T02:06:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Remove another instance of looking at the revindex directly by instead\ncalling 'pack_pos_to_index()'. Unlike other patches, this caller only\ncares about the index position of each object in the loop.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 5 ++---\n 1 file changed, 2 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 1fdf7ce20a..60fe20fb87 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1392,11 +1392,10 @@ uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \n \tfor (i = 0; i < num_objects; ++i) {\n \t\tstruct object_id oid;\n-\t\tstruct revindex_entry *entry;\n \t\tstruct object_entry *oe;\n \n-\t\tentry = &bitmap_git->pack->revindex[i];\n-\t\tnth_packed_object_id(&oid, bitmap_git->pack, entry->nr);\n+\t\tnth_packed_object_id(&oid, bitmap_git->pack,\n+\t\t\t\t     pack_pos_to_index(bitmap_git->pack, i));\n \t\toe = packlist_find(mapping, &oid);\n \n \t\tif (oe)\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414305","messageId":"68794e9484d303b832e8a9f7163cf89e2c506479.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 07/20] show_objects_for_type(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:23:56Z","receivedAt":"2021-01-14T02:06:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Avoid storing the revindex entry directly, since this structure will\nsoon be removed from the public interface. Instead, store the offset and\nindex position by calling 'pack_pos_to_offset()' and\n'pack_pos_to_index()', respectively.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 13 +++++++------\n 1 file changed, 7 insertions(+), 6 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d6861ddd4d..27a7a8ac4c 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -711,21 +711,22 @@ static void show_objects_for_type(\n \n \t\tfor (offset = 0; offset < BITS_IN_EWORD; ++offset) {\n \t\t\tstruct object_id oid;\n-\t\t\tstruct revindex_entry *entry;\n-\t\t\tuint32_t hash = 0;\n+\t\t\tuint32_t hash = 0, index_pos;\n+\t\t\toff_t ofs;\n \n \t\t\tif ((word >> offset) == 0)\n \t\t\t\tbreak;\n \n \t\t\toffset += ewah_bit_ctz64(word >> offset);\n \n-\t\t\tentry = &bitmap_git->pack->revindex[pos + offset];\n-\t\t\tnth_packed_object_id(&oid, bitmap_git->pack, entry->nr);\n+\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n+\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n+\t\t\tnth_packed_object_id(&oid, bitmap_git->pack, index_pos);\n \n \t\t\tif (bitmap_git->hashes)\n-\t\t\t\thash = get_be32(bitmap_git->hashes + entry->nr);\n+\t\t\t\thash = get_be32(bitmap_git->hashes + index_pos);\n \n-\t\t\tshow_reach(&oid, object_type, 0, hash, bitmap_git->pack, entry->offset);\n+\t\t\tshow_reach(&oid, object_type, 0, hash, bitmap_git->pack, ofs);\n \t\t}\n \t}\n }\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414306","messageId":"98816377248b3112975d49d89a4af7c29d12554e.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 11/20] get_delta_base_oid(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:24:32Z","receivedAt":"2021-01-14T02:06:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Replace direct accesses to the 'struct revindex' type with a call to\n'pack_pos_to_index()'.\n\nLikewise drop the old-style 'find_pack_revindex()' with its replacement\n'offset_to_pack_pos()' (while continuing to perform the same error\nchecking).\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n packfile.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 86f5c8dbf6..3e3f391949 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1235,18 +1235,18 @@ static int get_delta_base_oid(struct packed_git *p,\n \t\toidread(oid, base);\n \t\treturn 0;\n \t} else if (type == OBJ_OFS_DELTA) {\n-\t\tstruct revindex_entry *revidx;\n+\t\tuint32_t base_pos;\n \t\toff_t base_offset = get_delta_base(p, w_curs, &curpos,\n \t\t\t\t\t\t   type, delta_obj_offset);\n \n \t\tif (!base_offset)\n \t\t\treturn -1;\n \n-\t\trevidx = find_pack_revindex(p, base_offset);\n-\t\tif (!revidx)\n+\t\tif (offset_to_pack_pos(p, base_offset, &base_pos) < 0)\n \t\t\treturn -1;\n \n-\t\treturn nth_packed_object_id(oid, p, revidx->nr);\n+\t\treturn nth_packed_object_id(oid, p,\n+\t\t\t\t\t    pack_pos_to_index(p, base_pos));\n \t} else\n \t\treturn -1;\n }\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414307","messageId":"df8bb571a55ae94d85b996326aae8a709d84777c.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 12/20] retry_bad_packed_offset(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:24:36Z","receivedAt":"2021-01-14T02:06:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Perform exactly the same conversion as in the previous commit to another\ncaller within 'packfile.c'.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n packfile.c | 7 +++----\n 1 file changed, 3 insertions(+), 4 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 3e3f391949..7c37f9ec5c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1256,12 +1256,11 @@ static int retry_bad_packed_offset(struct repository *r,\n \t\t\t\t   off_t obj_offset)\n {\n \tint type;\n-\tstruct revindex_entry *revidx;\n+\tuint32_t pos;\n \tstruct object_id oid;\n-\trevidx = find_pack_revindex(p, obj_offset);\n-\tif (!revidx)\n+\tif (offset_to_pack_pos(p, obj_offset, &pos) < 0)\n \t\treturn OBJ_BAD;\n-\tnth_packed_object_id(&oid, p, revidx->nr);\n+\tnth_packed_object_id(&oid, p, pack_pos_to_index(p, pos));\n \tmark_bad_packed_object(p, oid.hash);\n \ttype = oid_object_info(r, &oid, NULL);\n \tif (type <= OBJ_NONE)\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414308","messageId":"8ad49d231f5e00e258a2e64443cda16626e289c4.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 14/20] unpack_entry(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:24:45Z","receivedAt":"2021-01-14T02:06:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Remove direct manipulation of the 'struct revindex_entry' type as well\nas calls to the deprecated API in 'packfile.c:unpack_entry()'. Usual\nclean-up is performed (replacing '->nr' with calls to\n'pack_pos_to_index()' and so on).\n\nAdd an additional check to make sure that 'obj_offset()' points at a\nvalid object. In the case this check is violated, we cannot call\n'mark_bad_packed_object()' because we don't know the OID. At the top of\nthe call stack is do_oid_object_info_extended() (via\npacked_object_info()), which does mark the object.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n packfile.c | 26 ++++++++++++++++++--------\n 1 file changed, 18 insertions(+), 8 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex bb4bb14671..936ab3def5 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1694,11 +1694,21 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,\n \t\t}\n \n \t\tif (do_check_packed_object_crc && p->index_version > 1) {\n-\t\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n-\t\t\toff_t len = revidx[1].offset - obj_offset;\n-\t\t\tif (check_pack_crc(p, &w_curs, obj_offset, len, revidx->nr)) {\n+\t\t\tuint32_t pack_pos, index_pos;\n+\t\t\toff_t len;\n+\n+\t\t\tif (offset_to_pack_pos(p, obj_offset, &pack_pos) < 0) {\n+\t\t\t\terror(\"could not find object at offset %\"PRIuMAX\" in pack %s\",\n+\t\t\t\t      (uintmax_t)obj_offset, p->pack_name);\n+\t\t\t\tdata = NULL;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\tlen = pack_pos_to_offset(p, pack_pos + 1) - obj_offset;\n+\t\t\tindex_pos = pack_pos_to_index(p, pack_pos);\n+\t\t\tif (check_pack_crc(p, &w_curs, obj_offset, len, index_pos)) {\n \t\t\t\tstruct object_id oid;\n-\t\t\t\tnth_packed_object_id(&oid, p, revidx->nr);\n+\t\t\t\tnth_packed_object_id(&oid, p, index_pos);\n \t\t\t\terror(\"bad packed object CRC for %s\",\n \t\t\t\t      oid_to_hex(&oid));\n \t\t\t\tmark_bad_packed_object(p, oid.hash);\n@@ -1781,11 +1791,11 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,\n \t\t\t * This is costly but should happen only in the presence\n \t\t\t * of a corrupted pack, and is better than failing outright.\n \t\t\t */\n-\t\t\tstruct revindex_entry *revidx;\n+\t\t\tuint32_t pos;\n \t\t\tstruct object_id base_oid;\n-\t\t\trevidx = find_pack_revindex(p, obj_offset);\n-\t\t\tif (revidx) {\n-\t\t\t\tnth_packed_object_id(&base_oid, p, revidx->nr);\n+\t\t\tif (!(offset_to_pack_pos(p, obj_offset, &pos))) {\n+\t\t\t\tnth_packed_object_id(&base_oid, p,\n+\t\t\t\t\t\t     pack_pos_to_index(p, pos));\n \t\t\t\terror(\"failed to read delta base object %s\"\n \t\t\t\t      \" at offset %\"PRIuMAX\" from %s\",\n \t\t\t\t      oid_to_hex(&base_oid), (uintmax_t)obj_offset,\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414310","messageId":"3b5c92be684b95f04cbe224c791d87657be9ff79.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 18/20] pack-revindex: remove unused 'find_revindex_position()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:25:02Z","receivedAt":"2021-01-14T02:07:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that all 'find_revindex_position()' callers have been removed (and\nconverted to the more descriptive 'offset_to_pack_pos()'), it is almost\nsafe to get rid of 'find_revindex_position()' entirely. Almost, except\nfor the fact that 'offset_to_pack_pos()' calls\n'find_revindex_position()'.\n\nInline 'find_revindex_position()' into 'offset_to_pack_pos()', and\nthen remove 'find_revindex_position()' entirely.\n\nThis is a straightforward refactoring with one minor snag.\n'offset_to_pack_pos()' used to load the index before calling\n'find_revindex_position()'. That means that by the time\n'find_revindex_position()' starts executing, 'p->num_objects' can be\nsafely read. After inlining, be careful to not read 'p->num_objects'\nuntil _after_ 'load_pack_revindex()' (which loads the index as a\nside-effect) has been called.\n\nAnother small fix that is included is converting the upper- and\nlower-bounds to be unsigned's instead of ints. This dates back to\n92e5c77c37 (revindex: export new APIs, 2013-10-24)--ironically, the last\ntime we introduced new APIs here--but this unifies the types.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-revindex.c | 31 ++++++++++++-------------------\n pack-revindex.h |  1 -\n 2 files changed, 12 insertions(+), 20 deletions(-)\n\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 16baafb281..282fe92640 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -169,16 +169,23 @@ int load_pack_revindex(struct packed_git *p)\n \treturn 0;\n }\n \n-int find_revindex_position(struct packed_git *p, off_t ofs)\n+int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n {\n-\tint lo = 0;\n-\tint hi = p->num_objects + 1;\n-\tconst struct revindex_entry *revindex = p->revindex;\n+\tunsigned lo, hi;\n+\tconst struct revindex_entry *revindex;\n+\n+\tif (load_pack_revindex(p) < 0)\n+\t\treturn -1;\n+\n+\tlo = 0;\n+\thi = p->num_objects + 1;\n+\trevindex = p->revindex;\n \n \tdo {\n \t\tconst unsigned mi = lo + (hi - lo) / 2;\n \t\tif (revindex[mi].offset == ofs) {\n-\t\t\treturn mi;\n+\t\t\t*pos = mi;\n+\t\t\treturn 0;\n \t\t} else if (ofs < revindex[mi].offset)\n \t\t\thi = mi;\n \t\telse\n@@ -189,20 +196,6 @@ int find_revindex_position(struct packed_git *p, off_t ofs)\n \treturn -1;\n }\n \n-int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n-{\n-\tint ret;\n-\n-\tif (load_pack_revindex(p) < 0)\n-\t\treturn -1;\n-\n-\tret = find_revindex_position(p, ofs);\n-\tif (ret < 0)\n-\t\treturn ret;\n-\t*pos = ret;\n-\treturn 0;\n-}\n-\n uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos)\n {\n \tif (!p->revindex)\ndiff --git a/pack-revindex.h b/pack-revindex.h\nindex f7094ba9a5..746776be7f 100644\n--- a/pack-revindex.h\n+++ b/pack-revindex.h\n@@ -28,7 +28,6 @@ struct revindex_entry {\n  * given pack, returning zero on success and a negative value otherwise.\n  */\n int load_pack_revindex(struct packed_git *p);\n-int find_revindex_position(struct packed_git *p, off_t ofs);\n \n /*\n  * offset_to_pack_pos converts an object offset to a pack position. This\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414318","messageId":"31ac6f57033f942be5ab4eff96e482a93fda4196.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 08/20] get_size_by_pos(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:24:00Z","receivedAt":"2021-01-14T02:09:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Remove another caller that holds onto a 'struct revindex_entry' by\nreplacing the direct indexing with calls to 'pack_pos_to_offset()' and\n'pack_pos_to_index()'.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 27a7a8ac4c..89a528a91b 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -835,11 +835,11 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git,\n \toi.sizep = &size;\n \n \tif (pos < pack->num_objects) {\n-\t\tstruct revindex_entry *entry = &pack->revindex[pos];\n-\t\tif (packed_object_info(the_repository, pack,\n-\t\t\t\t       entry->offset, &oi) < 0) {\n+\t\toff_t ofs = pack_pos_to_offset(pack, pos);\n+\t\tif (packed_object_info(the_repository, pack, ofs, &oi) < 0) {\n \t\t\tstruct object_id oid;\n-\t\t\tnth_packed_object_id(&oid, pack, entry->nr);\n+\t\t\tnth_packed_object_id(&oid, pack,\n+\t\t\t\t\t     pack_pos_to_index(pack, pos));\n \t\t\tdie(_(\"unable to get size of %s\"), oid_to_hex(&oid));\n \t\t}\n \t} else {\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414319","messageId":"dd7133fdb76761b2758ff6986421e4a7755c54a9.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 04/20] write_reused_pack_verbatim(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:23:43Z","receivedAt":"2021-01-14T02:09:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Replace a direct access to the revindex array with\n'pack_pos_to_offset()'.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/pack-objects.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 8e40b19ee8..77ce5583a2 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -952,7 +952,7 @@ static size_t write_reused_pack_verbatim(struct hashfile *out,\n \t\toff_t to_write;\n \n \t\twritten = (pos * BITS_IN_EWORD);\n-\t\tto_write = reuse_packfile->revindex[written].offset\n+\t\tto_write = pack_pos_to_offset(reuse_packfile, written)\n \t\t\t- sizeof(struct pack_header);\n \n \t\t/* We're recording one chunk, not one object. */\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414320","messageId":"084bbf2145735ef1affb0bc051b09dcff8f306ce.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 06/20] bitmap_position_packfile(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:23:52Z","receivedAt":"2021-01-14T02:09:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Replace find_revindex_position() with its counterpart in the new API,\noffset_to_pack_pos().\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n pack-bitmap.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d88745fb02..d6861ddd4d 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -407,11 +407,14 @@ static inline int bitmap_position_extended(struct bitmap_index *bitmap_git,\n static inline int bitmap_position_packfile(struct bitmap_index *bitmap_git,\n \t\t\t\t\t   const struct object_id *oid)\n {\n+\tuint32_t pos;\n \toff_t offset = find_pack_entry_one(oid->hash, bitmap_git->pack);\n \tif (!offset)\n \t\treturn -1;\n \n-\treturn find_revindex_position(bitmap_git->pack, offset);\n+\tif (offset_to_pack_pos(bitmap_git->pack, offset, &pos) < 0)\n+\t\treturn -1;\n+\treturn pos;\n }\n \n static int bitmap_position(struct bitmap_index *bitmap_git,\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414321","messageId":"0fca7d5812185d482fd48f7df6c062ab44933055.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 02/20] write_reuse_object(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:23:35Z","receivedAt":"2021-01-14T02:10:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"First replace 'find_pack_revindex()' with its replacement\n'offset_to_pack_pos()'. This prevents any bogus OFS_DELTA that may make\nits way through until 'write_reuse_object()' from causing a bad memory\nread (if 'revidx' is 'NULL')\n\nNext, replace a direct access of '->nr' with the wrapper function\n'pack_pos_to_index()'.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/pack-objects.c | 13 +++++++++----\n 1 file changed, 9 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 2a00358f34..ab1fd853f1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -419,7 +419,7 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n {\n \tstruct packed_git *p = IN_PACK(entry);\n \tstruct pack_window *w_curs = NULL;\n-\tstruct revindex_entry *revidx;\n+\tuint32_t pos;\n \toff_t offset;\n \tenum object_type type = oe_type(entry);\n \toff_t datalen;\n@@ -436,10 +436,15 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,\n \t\t\t\t\t      type, entry_size);\n \n \toffset = entry->in_pack_offset;\n-\trevidx = find_pack_revindex(p, offset);\n-\tdatalen = revidx[1].offset - offset;\n+\tif (offset_to_pack_pos(p, offset, &pos) < 0)\n+\t\tdie(_(\"write_reuse_object: could not locate %s, expected at \"\n+\t\t      \"offset %\"PRIuMAX\" in pack %s\"),\n+\t\t    oid_to_hex(&entry->idx.oid), (uintmax_t)offset,\n+\t\t    p->pack_name);\n+\tdatalen = pack_pos_to_offset(p, pos + 1) - offset;\n \tif (!pack_to_stdout && p->index_version > 1 &&\n-\t    check_pack_crc(p, &w_curs, offset, datalen, revidx->nr)) {\n+\t    check_pack_crc(p, &w_curs, offset, datalen,\n+\t\t\t   pack_pos_to_index(p, pos))) {\n \t\terror(_(\"bad packed object CRC for %s\"),\n \t\t      oid_to_hex(&entry->idx.oid));\n \t\tunuse_pack(&w_curs);\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414324","messageId":"8e93ca38865a9fb9ff9bb264c6db9b6dc14e3029.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"[PATCH v2 05/20] check_object(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:23:47Z","receivedAt":"2021-01-14T02:12:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Replace direct accesses to the revindex with calls to\n'offset_to_pack_pos()' and 'pack_pos_to_index()'.\n\nSince this caller already had some error checking (it can jump to the\n'give_up' label if it encounters an error), we can easily check whether\nor not the provided offset points to an object in the given pack. This\nerror checking existed prior to this patch, too, since the caller checks\nwhether the return value from 'find_pack_revindex()' was NULL or not.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/pack-objects.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 77ce5583a2..5b0c4489e2 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1817,11 +1817,11 @@ static void check_object(struct object_entry *entry, uint32_t object_index)\n \t\t\t\tgoto give_up;\n \t\t\t}\n \t\t\tif (reuse_delta && !entry->preferred_base) {\n-\t\t\t\tstruct revindex_entry *revidx;\n-\t\t\t\trevidx = find_pack_revindex(p, ofs);\n-\t\t\t\tif (!revidx)\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (offset_to_pack_pos(p, ofs, &pos) < 0)\n \t\t\t\t\tgoto give_up;\n-\t\t\t\tif (!nth_packed_object_id(&base_ref, p, revidx->nr))\n+\t\t\t\tif (!nth_packed_object_id(&base_ref, p,\n+\t\t\t\t\t\t\t  pack_pos_to_index(p, pos)))\n \t\t\t\t\thave_base = 1;\n \t\t\t}\n \t\t\tentry->in_pack_header_size = used + used_0;\n-- \n2.30.0.138.g6d7191ea01\n\n"},{"id":"414325","messageId":"cover.1610576604.git.me@ttaylorr.com","threadId":"54961","inReplyTo":"cover.1610129796.git.me@ttaylorr.com","subject":"[PATCH v2 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-13T22:23:25Z","receivedAt":"2021-01-14T02:12:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi,\n\nHere is a revision of the first of two series to prepare for and introduce an\non-disk alternative for storing the reverse index.\n\nIn this revision, I addressed feedback from Junio, Peff, and Stolee. A\nrange-diff is included below, but the main changes are:\n\n  - Error messages are improved to include the pack and offset when applicable.\n  - Variable names were made clearer (e.g., n -> index_pos).\n  - Comments were added in pack-revindex.h to introduce relevant terminology,\n    and which methods convert between what orderings.\n  - int-sized lower- and upper-bounds were converted to be unsigned.\n\nI believe that this revision should be ready for queueing. I'll send a v2 of the\ncorresponding latter series shortly.\n\nThanks in advance for your review.\n\nTaylor Blau (20):\n  pack-revindex: introduce a new API\n  write_reuse_object(): convert to new revindex API\n  write_reused_pack_one(): convert to new revindex API\n  write_reused_pack_verbatim(): convert to new revindex API\n  check_object(): convert to new revindex API\n  bitmap_position_packfile(): convert to new revindex API\n  show_objects_for_type(): convert to new revindex API\n  get_size_by_pos(): convert to new revindex API\n  try_partial_reuse(): convert to new revindex API\n  rebuild_existing_bitmaps(): convert to new revindex API\n  get_delta_base_oid(): convert to new revindex API\n  retry_bad_packed_offset(): convert to new revindex API\n  packed_object_info(): convert to new revindex API\n  unpack_entry(): convert to new revindex API\n  for_each_object_in_pack(): convert to new revindex API\n  builtin/gc.c: guess the size of the revindex\n  pack-revindex: remove unused 'find_pack_revindex()'\n  pack-revindex: remove unused 'find_revindex_position()'\n  pack-revindex: hide the definition of 'revindex_entry'\n  pack-revindex.c: avoid direct revindex access in\n    'offset_to_pack_pos()'\n\n builtin/gc.c           |  2 +-\n builtin/pack-objects.c | 37 +++++++++++++++++---------\n pack-bitmap.c          | 44 +++++++++++++++----------------\n pack-revindex.c        | 51 ++++++++++++++++++++++-------------\n pack-revindex.h        | 60 +++++++++++++++++++++++++++++++++++++-----\n packfile.c             | 54 ++++++++++++++++++++++++-------------\n 6 files changed, 168 insertions(+), 80 deletions(-)\n\nRange-diff against v1:\n 1:  fa6b830908 <  -:  ---------- pack-revindex: introduce a new API\n -:  ---------- >  1:  e1aa89244a pack-revindex: introduce a new API\n 2:  00668523e1 !  2:  0fca7d5812 write_reuse_object(): convert to new revindex API\n    @@ builtin/pack-objects.c: static off_t write_reuse_object(struct hashfile *f, stru\n     -\trevidx = find_pack_revindex(p, offset);\n     -\tdatalen = revidx[1].offset - offset;\n     +\tif (offset_to_pack_pos(p, offset, &pos) < 0)\n    -+\t\tdie(_(\"write_reuse_object: could not locate %s\"),\n    -+\t\t    oid_to_hex(&entry->idx.oid));\n    ++\t\tdie(_(\"write_reuse_object: could not locate %s, expected at \"\n    ++\t\t      \"offset %\"PRIuMAX\" in pack %s\"),\n    ++\t\t    oid_to_hex(&entry->idx.oid), (uintmax_t)offset,\n    ++\t\t    p->pack_name);\n     +\tdatalen = pack_pos_to_offset(p, pos + 1) - offset;\n      \tif (!pack_to_stdout && p->index_version > 1 &&\n     -\t    check_pack_crc(p, &w_curs, offset, datalen, revidx->nr)) {\n 3:  81ab11e18c !  3:  7676822a54 write_reused_pack_one(): convert to new revindex API\n    @@ builtin/pack-objects.c: static void write_reused_pack_one(size_t pos, struct has\n      \t\t\tstruct object_id base_oid;\n      \n     +\t\t\tif (offset_to_pack_pos(reuse_packfile, base_offset, &base_pos) < 0)\n    -+\t\t\t\tdie(_(\"expected object at offset %\"PRIuMAX),\n    -+\t\t\t\t    (uintmax_t)base_offset);\n    ++\t\t\t\tdie(_(\"expected object at offset %\"PRIuMAX\" \"\n    ++\t\t\t\t      \"in pack %s\"),\n    ++\t\t\t\t    (uintmax_t)base_offset,\n    ++\t\t\t\t    reuse_packfile->pack_name);\n     +\n      \t\t\tnth_packed_object_id(&base_oid, reuse_packfile,\n     -\t\t\t\t\t     reuse_packfile->revindex[base_pos].nr);\n 4:  14b35d01a0 =  4:  dd7133fdb7 write_reused_pack_verbatim(): convert to new revindex API\n 5:  c47e77a30e =  5:  8e93ca3886 check_object(): convert to new revindex API\n 6:  3b170663dd =  6:  084bbf2145 bitmap_position_packfile(): convert to new revindex API\n 7:  bc67bb462a !  7:  68794e9484 show_objects_for_type(): convert to new revindex API\n    @@ pack-bitmap.c: static void show_objects_for_type(\n      \t\t\tstruct object_id oid;\n     -\t\t\tstruct revindex_entry *entry;\n     -\t\t\tuint32_t hash = 0;\n    -+\t\t\tuint32_t hash = 0, n;\n    ++\t\t\tuint32_t hash = 0, index_pos;\n     +\t\t\toff_t ofs;\n      \n      \t\t\tif ((word >> offset) == 0)\n    @@ pack-bitmap.c: static void show_objects_for_type(\n      \n     -\t\t\tentry = &bitmap_git->pack->revindex[pos + offset];\n     -\t\t\tnth_packed_object_id(&oid, bitmap_git->pack, entry->nr);\n    -+\t\t\tn = pack_pos_to_index(bitmap_git->pack, pos + offset);\n    ++\t\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, pos + offset);\n     +\t\t\tofs = pack_pos_to_offset(bitmap_git->pack, pos + offset);\n    -+\t\t\tnth_packed_object_id(&oid, bitmap_git->pack, n);\n    ++\t\t\tnth_packed_object_id(&oid, bitmap_git->pack, index_pos);\n      \n      \t\t\tif (bitmap_git->hashes)\n     -\t\t\t\thash = get_be32(bitmap_git->hashes + entry->nr);\n    -+\t\t\t\thash = get_be32(bitmap_git->hashes + n);\n    ++\t\t\t\thash = get_be32(bitmap_git->hashes + index_pos);\n      \n     -\t\t\tshow_reach(&oid, object_type, 0, hash, bitmap_git->pack, entry->offset);\n     +\t\t\tshow_reach(&oid, object_type, 0, hash, bitmap_git->pack, ofs);\n 8:  541fe679f3 =  8:  31ac6f5703 get_size_by_pos(): convert to new revindex API\n 9:  54f4ad329f !  9:  acd80069a2 try_partial_reuse(): convert to new revindex API\n    @@ Commit message\n         'pack_pos_to_offset()' instead (the caller here does not care about the\n         index position of the object at position 'pos').\n     \n    -    Somewhat confusingly, the subsequent call to unpack_object_header()\n    -    takes a pointer to &offset and then updates it with a new value. But,\n    -    try_partial_reuse() cares about the offset of both the base's header and\n    -    contents. The existing code made a copy of the offset field, and only\n    -    addresses and manipulates one of them.\n    -\n    -    Instead, store the return of pack_pos_to_offset twice: once in header\n    -    and another in offset. Header will be left untouched, but offset will be\n    -    addressed and modified by unpack_object_header().\n    +    Note that we cannot just use the existing \"offset\" variable to store the\n    +    value we get from pack_pos_to_offset(). It is incremented by\n    +    unpack_object_header(), but we later need the original value. Since\n    +    we'll no longer have revindex->offset to read it from, we'll store that\n    +    in a separate variable (\"header\" since it points to the entry's header\n    +    bytes).\n     \n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n     \n10:  97eaa7b2d6 = 10:  569acdca7f rebuild_existing_bitmaps(): convert to new revindex API\n11:  e00c434ab2 = 11:  9881637724 get_delta_base_oid(): convert to new revindex API\n12:  aae01d7029 = 12:  df8bb571a5 retry_bad_packed_offset(): convert to new revindex API\n13:  eab7ab1f35 ! 13:  41b2e00947 packed_object_info(): convert to new revindex API\n    @@ packfile.c: int packed_object_info(struct repository *r, struct packed_git *p,\n     -\t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n     +\t\tuint32_t pos;\n     +\t\tif (offset_to_pack_pos(p, obj_offset, &pos) < 0) {\n    ++\t\t\terror(\"could not find object at offset %\"PRIuMAX\" \"\n    ++\t\t\t      \"in pack %s\", (uintmax_t)obj_offset, p->pack_name);\n     +\t\t\ttype = OBJ_BAD;\n     +\t\t\tgoto out;\n     +\t\t}\n14:  13c49ed40c ! 14:  8ad49d231f unpack_entry(): convert to new revindex API\n    @@ Commit message\n         Remove direct manipulation of the 'struct revindex_entry' type as well\n         as calls to the deprecated API in 'packfile.c:unpack_entry()'. Usual\n         clean-up is performed (replacing '->nr' with calls to\n    -    'pack_pos_to_index()' and so on). Add an additional check to make\n    -    sure that 'obj_offset()' points at a valid object.\n    +    'pack_pos_to_index()' and so on).\n    +\n    +    Add an additional check to make sure that 'obj_offset()' points at a\n    +    valid object. In the case this check is violated, we cannot call\n    +    'mark_bad_packed_object()' because we don't know the OID. At the top of\n    +    the call stack is do_oid_object_info_extended() (via\n    +    packed_object_info()), which does mark the object.\n     \n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n     \n    @@ packfile.c: void *unpack_entry(struct repository *r, struct packed_git *p, off_t\n     -\t\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n     -\t\t\toff_t len = revidx[1].offset - obj_offset;\n     -\t\t\tif (check_pack_crc(p, &w_curs, obj_offset, len, revidx->nr)) {\n    -+\t\t\tuint32_t pos, nr;\n    ++\t\t\tuint32_t pack_pos, index_pos;\n     +\t\t\toff_t len;\n     +\n    -+\t\t\tif (offset_to_pack_pos(p, obj_offset, &pos) < 0) {\n    ++\t\t\tif (offset_to_pack_pos(p, obj_offset, &pack_pos) < 0) {\n    ++\t\t\t\terror(\"could not find object at offset %\"PRIuMAX\" in pack %s\",\n    ++\t\t\t\t      (uintmax_t)obj_offset, p->pack_name);\n     +\t\t\t\tdata = NULL;\n     +\t\t\t\tgoto out;\n     +\t\t\t}\n     +\n    -+\t\t\tlen = pack_pos_to_offset(p, pos + 1) - obj_offset;\n    -+\t\t\tnr = pack_pos_to_index(p, pos);\n    -+\t\t\tif (check_pack_crc(p, &w_curs, obj_offset, len, nr)) {\n    ++\t\t\tlen = pack_pos_to_offset(p, pack_pos + 1) - obj_offset;\n    ++\t\t\tindex_pos = pack_pos_to_index(p, pack_pos);\n    ++\t\t\tif (check_pack_crc(p, &w_curs, obj_offset, len, index_pos)) {\n      \t\t\t\tstruct object_id oid;\n     -\t\t\t\tnth_packed_object_id(&oid, p, revidx->nr);\n    -+\t\t\t\tnth_packed_object_id(&oid, p, nr);\n    ++\t\t\t\tnth_packed_object_id(&oid, p, index_pos);\n      \t\t\t\terror(\"bad packed object CRC for %s\",\n      \t\t\t\t      oid_to_hex(&oid));\n      \t\t\t\tmark_bad_packed_object(p, oid.hash);\n15:  a3249986f9 = 15:  e757476351 for_each_object_in_pack(): convert to new revindex API\n16:  7c17db7a7d = 16:  a500311e33 builtin/gc.c: guess the size of the revindex\n17:  c4c88bcc3d ! 17:  67d14da04a pack-revindex: remove unused 'find_pack_revindex()'\n    @@ pack-revindex.h: struct revindex_entry {\n      \n     -struct revindex_entry *find_pack_revindex(struct packed_git *p, off_t ofs);\n     -\n    - int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos);\n    - uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);\n    - off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos);\n    + /*\n    +  * offset_to_pack_pos converts an object offset to a pack position. This\n    +  * function returns zero on success, and a negative number otherwise. The\n18:  d60411d524 ! 18:  3b5c92be68 pack-revindex: remove unused 'find_revindex_position()'\n    @@ Commit message\n         until _after_ 'load_pack_revindex()' (which loads the index as a\n         side-effect) has been called.\n     \n    +    Another small fix that is included is converting the upper- and\n    +    lower-bounds to be unsigned's instead of ints. This dates back to\n    +    92e5c77c37 (revindex: export new APIs, 2013-10-24)--ironically, the last\n    +    time we introduced new APIs here--but this unifies the types.\n    +\n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n     \n      ## pack-revindex.c ##\n    @@ pack-revindex.c: int load_pack_revindex(struct packed_git *p)\n     -int find_revindex_position(struct packed_git *p, off_t ofs)\n     +int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n      {\n    - \tint lo = 0;\n    +-\tint lo = 0;\n     -\tint hi = p->num_objects + 1;\n     -\tconst struct revindex_entry *revindex = p->revindex;\n    -+\tint hi;\n    ++\tunsigned lo, hi;\n     +\tconst struct revindex_entry *revindex;\n     +\n     +\tif (load_pack_revindex(p) < 0)\n     +\t\treturn -1;\n     +\n    ++\tlo = 0;\n     +\thi = p->num_objects + 1;\n     +\trevindex = p->revindex;\n      \n    @@ pack-revindex.c: int find_revindex_position(struct packed_git *p, off_t ofs)\n     -\n     -\tret = find_revindex_position(p, ofs);\n     -\tif (ret < 0)\n    --\t\treturn -1;\n    +-\t\treturn ret;\n     -\t*pos = ret;\n     -\treturn 0;\n     -}\n    @@ pack-revindex.c: int find_revindex_position(struct packed_git *p, off_t ofs)\n     \n      ## pack-revindex.h ##\n     @@ pack-revindex.h: struct revindex_entry {\n    - };\n    - \n    +  * given pack, returning zero on success and a negative value otherwise.\n    +  */\n      int load_pack_revindex(struct packed_git *p);\n     -int find_revindex_position(struct packed_git *p, off_t ofs);\n      \n    - int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos);\n    - uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);\n    + /*\n    +  * offset_to_pack_pos converts an object offset to a pack position. This\n19:  7c0e4acc84 ! 19:  cabafce4a1 pack-revindex: hide the definition of 'revindex_entry'\n    @@ pack-revindex.h\n     -\tunsigned int nr;\n     -};\n     -\n    - int load_pack_revindex(struct packed_git *p);\n    - \n    - int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos);\n    + /*\n    +  * load_pack_revindex populates the revindex's internal data-structures for the\n    +  * given pack, returning zero on success and a negative value otherwise.\n20:  eada1ffcfa ! 20:  8400ff6c96 pack-revindex.c: avoid direct revindex access in 'offset_to_pack_pos()'\n    @@ Commit message\n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n     \n      ## pack-revindex.c ##\n    -@@ pack-revindex.c: int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n    +@@ pack-revindex.c: int load_pack_revindex(struct packed_git *p)\n    + int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n      {\n    - \tint lo = 0;\n    - \tint hi;\n    + \tunsigned lo, hi;\n     -\tconst struct revindex_entry *revindex;\n      \n      \tif (load_pack_revindex(p) < 0)\n      \t\treturn -1;\n      \n    + \tlo = 0;\n      \thi = p->num_objects + 1;\n     -\trevindex = p->revindex;\n      \n-- \n2.30.0.138.g6d7191ea01\n"},{"id":"414327","messageId":"xmqqwnwgyqn6.fsf@gitster.c.googlers.com","threadId":"54961","inReplyTo":"a500311e33a2f7e11a539dd0718ed946f4bd6bc8.1610576604.git.me@ttaylorr.com","subject":"Re: [PATCH v2 16/20] builtin/gc.c: guess the size of the revindex","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-14T06:33:01Z","receivedAt":"2021-01-14T06:33:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> 'estimate_repack_memory()' takes into account the amount of memory\n> required to load the reverse index in memory by multiplying the assumed\n> number of objects by the size of the 'revindex_entry' struct.\n>\n> Prepare for hiding the definition of 'struct revindex_entry' by removing\n> a 'sizeof()' of that type from outside of pack-revindex.c. Instead,\n> guess that one off_t and one uint32_t are required per object. Strictly\n> speaking, this is a worse guess than asking for 'sizeof(struct\n> revindex_entry)' directly, since the true size of this struct is 16\n> bytes with padding on the end of the struct in order to align the offset\n> field.\n\nMeaning that we under-estimate by 25%?\n\n> But, this is an approximation anyway, and it does remove a use of the\n> 'struct revindex_entry' from outside of pack-revindex internals.\n>\n> Signed-off-by: Taylor Blau <me@ttaylorr.com>\n> ---\n>  builtin/gc.c | 2 +-\n>  1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index 4c24f41852..c60811f212 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -301,7 +301,7 @@ static uint64_t estimate_repack_memory(struct packed_git *pack)\n>  \t/* and then obj_hash[], underestimated in fact */\n>  \theap += sizeof(struct object *) * nr_objects;\n>  \t/* revindex is used also */\n> -\theap += sizeof(struct revindex_entry) * nr_objects;\n> +\theap += (sizeof(off_t) + sizeof(uint32_t)) * nr_objects;\n>  \t/*\n>  \t * read_sha1_file() (either at delta calculation phase, or\n>  \t * writing phase) also fills up the delta base cache\n"},{"id":"414329","messageId":"xmqqmtxcyq7e.fsf@gitster.c.googlers.com","threadId":"54961","inReplyTo":"8400ff6c9615b4c999b198c46b2e673ec0f2b14f.1610576604.git.me@ttaylorr.com","subject":"Re: [PATCH v2 20/20] pack-revindex.c: avoid direct revindex access in 'offset_to_pack_pos()'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-14T06:42:29Z","receivedAt":"2021-01-14T06:43:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> To prepare for on-disk reverse indexes, remove a spot in\n> 'offset_to_pack_pos()' that looks at the 'revindex' array in 'struct\n> packed_git'.\n\nHmph, I somehow would have expected that this clean-up would be done\nbefore step [18/20], but that does not matter in the end.  The end\nresult looks fairly clean.\n\nI wonder if the call overhead to pack_pos_to_offset(), relative to\nthe direct indexing of an in-core array revindex[] followed by an\naccess to a member .offset that we used to do, makes a measurable\ndifference in this tight loop, though.\n\n> diff --git a/pack-revindex.c b/pack-revindex.c\n> index a508d5f0a4..5e69bc7372 100644\n> --- a/pack-revindex.c\n> +++ b/pack-revindex.c\n> @@ -177,21 +177,21 @@ int load_pack_revindex(struct packed_git *p)\n>  int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n>  {\n>  \tunsigned lo, hi;\n> -\tconst struct revindex_entry *revindex;\n>  \n>  \tif (load_pack_revindex(p) < 0)\n>  \t\treturn -1;\n>  \n>  \tlo = 0;\n>  \thi = p->num_objects + 1;\n> -\trevindex = p->revindex;\n>  \n>  \tdo {\n>  \t\tconst unsigned mi = lo + (hi - lo) / 2;\n> -\t\tif (revindex[mi].offset == ofs) {\n> +\t\toff_t got = pack_pos_to_offset(p, mi);\n> +\n> +\t\tif (got == ofs) {\n>  \t\t\t*pos = mi;\n>  \t\t\treturn 0;\n> -\t\t} else if (ofs < revindex[mi].offset)\n> +\t\t} else if (ofs < got)\n>  \t\t\thi = mi;\n>  \t\telse\n>  \t\t\tlo = mi + 1;\n"},{"id":"414330","messageId":"xmqqft34yq6y.fsf@gitster.c.googlers.com","threadId":"54961","inReplyTo":"3b5c92be684b95f04cbe224c791d87657be9ff79.1610576604.git.me@ttaylorr.com","subject":"Re: [PATCH v2 18/20] pack-revindex: remove unused 'find_revindex_position()'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-14T06:42:45Z","receivedAt":"2021-01-14T06:43:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> -int find_revindex_position(struct packed_git *p, off_t ofs)\n> +int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)\n>  {\n> -\tint lo = 0;\n> -\tint hi = p->num_objects + 1;\n> -\tconst struct revindex_entry *revindex = p->revindex;\n> +\tunsigned lo, hi;\n> +\tconst struct revindex_entry *revindex;\n> +\n> +\tif (load_pack_revindex(p) < 0)\n> +\t\treturn -1;\n> +\n> +\tlo = 0;\n> +\thi = p->num_objects + 1;\n> +\trevindex = p->revindex;\n>  \tdo {\n>  \t\tconst unsigned mi = lo + (hi - lo) / 2;\n>  \t\tif (revindex[mi].offset == ofs) {\n> -\t\t\treturn mi;\n> +\t\t\t*pos = mi;\n> +\t\t\treturn 0;\n>  \t\t} else if (ofs < revindex[mi].offset)\n>  \t\t\thi = mi;\n>  \t\telse\n\nOK, we can safely depend on \"unsigned int\" at least as wide as\n\"uint32_t\"; unlike the original that used \"int\", we won't risk\nlosing the upper half of 4G range.\n\nNice.\n"},{"id":"414331","messageId":"xmqq8s8wyq5i.fsf@gitster.c.googlers.com","threadId":"54961","inReplyTo":"e7574763513294b71071b032d5cd3aa0976969dd.1610576604.git.me@ttaylorr.com","subject":"Re: [PATCH v2 15/20] for_each_object_in_pack(): convert to new revindex API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-14T06:43:37Z","receivedAt":"2021-01-14T06:44:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> Avoid looking at the 'revindex' pointer directly and instead call\n> 'pack_pos_to_index()'.\n>\n> Signed-off-by: Taylor Blau <me@ttaylorr.com>\n> ---\n>  packfile.c | 2 +-\n>  1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/packfile.c b/packfile.c\n> index 936ab3def5..7bb1750934 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -2086,7 +2086,7 @@ int for_each_object_in_pack(struct packed_git *p,\n>  \t\tstruct object_id oid;\n>  \n>  \t\tif (flags & FOR_EACH_OBJECT_PACK_ORDER)\n> -\t\t\tpos = p->revindex[i].nr;\n> +\t\t\tpos = pack_pos_to_index(p, i);\n\nIt wasn't too bad before this series formally defined what\n\"position\", \"index\" and \"offset\" mean, but now this has become\nhighly misleading. The variable \"pos\" here holds what we consider\n\"index\" while \"i\" holds what we call \"position\" [*1*].\n\n>  \t\telse\n>  \t\t\tpos = i;\n\nPerhaps renaming \"uint32_t pos\" to \"nth\" would avoid confusion?\n\n-\tif (nth_packed_object_id(&oid, p, pos) < 0)\n+\tif (nth_packed_object_id(&oid, p, nth) < 0)\n\t\treturn error(...);\n\n\n[Footnote]\n\n*1* The nth_packed_object_id() call we make later using the value we\nobtain here should be documented to take \"index\" as its last\nparameter, now that is what we call the location in the index, which\nis in object name order.\n\n\n\n"},{"id":"414332","messageId":"xmqq1reoypzy.fsf@gitster.c.googlers.com","threadId":"54961","inReplyTo":"e1aa89244ad3edb52aaeb28d6934cb2b0a0dc65a.1610576604.git.me@ttaylorr.com","subject":"Re: [PATCH v2 01/20] pack-revindex: introduce a new API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-14T06:46:57Z","receivedAt":"2021-01-14T06:47:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> +/*\n> + * offset_to_pack_pos converts an object offset to a pack position. This\n> + * function returns zero on success, and a negative number otherwise. The\n> + * parameter 'pos' is usable only on success.\n> + *\n> + * If the reverse index has not yet been loaded, this function loads it lazily,\n> + * and returns an negative number if an error was encountered.\n\nIt is somewhat strange to see a function that yields a non-negative\n\"position\" on success and a negative value to signal a failure to\nhave a separate pointer to the location to receive the true return\nvalue.  Do we truly care the upper half of \"uint32_t\" (in other\nwords, do we seriously want to support more than 2G positions in a\npack)?\n\nWhat I'm trying to get at is that\n\n\t\tint pos = offset_to_pack_pos(...);\n\t\tif (pos < 0)\n\t\t\terror();\n\t\telse\n\t\t\tuse(pos);\n\nis more natural than\n\n\t\tuint32_t pos;\n                if (offset_to_pack_pos(..., &pos) < 0)\n\t\t\terror();\n\t\telse\n\t\t\tuse(pos);\n\nbut now I wrote it down and laid it out in front of my eyes, the\nlatter does not look too bad.\n\n\t... later comes back after reading through the series ...\n\n\tThe new callers all looked quite nice to eyes.  Because we\n\tdiscourage assignment inside if() condition, the converted\n\tresult does not make the code more verbose than the\n\toriginal.  In fact, it makes it even clearer that we are\n\tchecking for an error return from a function call.  \n\n\tQuite nice.\n\n> + * This function runs in time O(log N) with the number of objects in the pack.\n\nIs it a good idea to commit to such performance characteristics as a\npromise to callers like this (the comment applies to all three\nfunctions)?\n\nIt depends on how a developer is helped by this comment when\ndeciding whether to use this function, or find other ways, to\nimplement what s/he wants to do.\n\n> + */\n> +int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos);\n\n> +/*\n> + * pack_pos_to_index converts the given pack-relative position 'pos' by\n> + * returning an index-relative position.\n> + *\n> + * If the reverse index has not yet been loaded, or the position is out of\n> + * bounds, this function aborts.\n> + *\n> + * This function runs in constant time.\n> + */\n> +uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);\n> +\n> +/*\n> + * pack_pos_to_offset converts the given pack-relative position 'pos' into a\n> + * pack offset. For a pack with 'N' objects, asking for position 'N' will return\n> + * the total size (in bytes) of the pack.\n\nIf we talk about \"asking for 'N'\" and want it to mean \"one beyond\nthe last position\", it is better to clarify that we count from 0.\nBut see below.\n\n> + * If the reverse index has not yet been loaded, or the position is out of\n> + * bounds, this function aborts.\n\nI think it is easier to read if the \"unlike the above function, this\nallows pos that is one beyond the last object\" is explained next to\n\"if out of bounds, it is an error\", not as a part of the previous\nparagraph.\n\n> + * This function runs in constant time.\n> + */\n> +off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos);\n> +\n>  #endif\n"},{"id":"414352","messageId":"685081d3-0cca-4c61-52b2-e9a8006803b1@gmail.com","threadId":"54961","inReplyTo":"xmqq1reoypzy.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 01/20] pack-revindex: introduce a new API","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-01-14T12:00:10Z","receivedAt":"2021-01-14T12:00:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/14/2021 1:46 AM, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n> \n>> +/*\n>> + * offset_to_pack_pos converts an object offset to a pack position. This\n>> + * function returns zero on success, and a negative number otherwise. The\n>> + * parameter 'pos' is usable only on success.\n>> + *\n>> + * If the reverse index has not yet been loaded, this function loads it lazily,\n>> + * and returns an negative number if an error was encountered.\n> \n> It is somewhat strange to see a function that yields a non-negative\n> \"position\" on success and a negative value to signal a failure to\n> have a separate pointer to the location to receive the true return\n> value.  Do we truly care the upper half of \"uint32_t\" (in other\n> words, do we seriously want to support more than 2G positions in a\n> pack)?\n> \n> What I'm trying to get at is that\n> \n> \t\tint pos = offset_to_pack_pos(...);\n> \t\tif (pos < 0)\n> \t\t\terror();\n> \t\telse\n> \t\t\tuse(pos);\n> \n> is more natural than\n\nI agree that this is used commonly, but usually in the case that\nwe are finding a position in the list _or where such an item would\nbe inserted_. For example:\n\n\tpos = index_name_pos(istate, dirname, len);\n\tif (pos < 0)\n\t\tpos = -pos-1;\n\twhile (pos < istate->cache_nr) {\n\t\t...\n\nBut that does not apply in this case. Knowing that the requested\noffset lies between object 'i' and object 'i + 1' isn't helpful,\nsince the offset still does not correspond to the start of an\nobject.\n\n> \t\tuint32_t pos;\n>                 if (offset_to_pack_pos(..., &pos) < 0)\n> \t\t\terror();\n> \t\telse\n> \t\t\tuse(pos);\n> \n> but now I wrote it down and laid it out in front of my eyes, the\n> latter does not look too bad.\n> \n> \t... later comes back after reading through the series ...\n> \n> \tThe new callers all looked quite nice to eyes.  Because we\n> \tdiscourage assignment inside if() condition, the converted\n> \tresult does not make the code more verbose than the\n> \toriginal.  In fact, it makes it even clearer that we are\n> \tchecking for an error return from a function call.  \n> \n> \tQuite nice.\n\nAs someone who spends a decent amount of time working in C#, I\nalso like this pattern. The APIs in C# work this way, too, such\nas:\n\n\tif (!set.TryGetValue(key, out value))\n\t\treturn false;\n\n\t// Use 'value', which is initialized now.\n\nThanks,\n-Stolee\n"},{"id":"414361","messageId":"YAB3E6lgOQdNgGOr@nand.local","threadId":"54961","inReplyTo":"xmqqwnwgyqn6.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 16/20] builtin/gc.c: guess the size of the revindex","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-14T16:53:39Z","receivedAt":"2021-01-14T16:54:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 13, 2021 at 10:33:01PM -0800, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> > 'estimate_repack_memory()' takes into account the amount of memory\n> > required to load the reverse index in memory by multiplying the assumed\n> > number of objects by the size of the 'revindex_entry' struct.\n> >\n> > Prepare for hiding the definition of 'struct revindex_entry' by removing\n> > a 'sizeof()' of that type from outside of pack-revindex.c. Instead,\n> > guess that one off_t and one uint32_t are required per object. Strictly\n> > speaking, this is a worse guess than asking for 'sizeof(struct\n> > revindex_entry)' directly, since the true size of this struct is 16\n> > bytes with padding on the end of the struct in order to align the offset\n> > field.\n>\n> Meaning that we under-estimate by 25%?\n\nIn this area, yes. I'm skeptical that this estimate is all that\nimportant, since it doesn't seem to take into account the memory\nrequired to select delta/base candidates [1].\n\nThanks,\nTaylor\n\n[1]: https://lore.kernel.org/git/X%2F1roycRbYPjnI3l@coredump.intra.peff.net/\n"},{"id":"414362","messageId":"YAB3qax1O++wLasq@nand.local","threadId":"54961","inReplyTo":"xmqqmtxcyq7e.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 20/20] pack-revindex.c: avoid direct revindex access in 'offset_to_pack_pos()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-14T16:56:09Z","receivedAt":"2021-01-14T16:57:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 13, 2021 at 10:42:29PM -0800, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> > To prepare for on-disk reverse indexes, remove a spot in\n> > 'offset_to_pack_pos()' that looks at the 'revindex' array in 'struct\n> > packed_git'.\n>\n> Hmph, I somehow would have expected that this clean-up would be done\n> before step [18/20], but that does not matter in the end.  The end\n> result looks fairly clean.\n>\n> I wonder if the call overhead to pack_pos_to_offset(), relative to\n> the direct indexing of an in-core array revindex[] followed by an\n> access to a member .offset that we used to do, makes a measurable\n> difference in this tight loop, though.\n\nI'm skeptical that it does (take that with a grain of salt, since I\nhaven't done any per-function tests with perf, only \"how long does it\ntake to run 'git cat-file --batch-check=%(objectsize:disk)' and so on\").\n\nBut even if it were to make a difference, it'll get dwarfed in the next\nseries by the time that we now _don't_ have to spend building and\nsorting the reverse index in memory for each new process.\n\nThanks,\nTaylor\n"},{"id":"414363","messageId":"YAB4xfoPK7z5pmmW@nand.local","threadId":"54961","inReplyTo":"xmqq8s8wyq5i.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 15/20] for_each_object_in_pack(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-14T17:00:53Z","receivedAt":"2021-01-14T17:01:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 13, 2021 at 10:43:37PM -0800, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> > Avoid looking at the 'revindex' pointer directly and instead call\n> > 'pack_pos_to_index()'.\n> >\n> > Signed-off-by: Taylor Blau <me@ttaylorr.com>\n> > ---\n> >  packfile.c | 2 +-\n> >  1 file changed, 1 insertion(+), 1 deletion(-)\n> >\n> > diff --git a/packfile.c b/packfile.c\n> > index 936ab3def5..7bb1750934 100644\n> > --- a/packfile.c\n> > +++ b/packfile.c\n> > @@ -2086,7 +2086,7 @@ int for_each_object_in_pack(struct packed_git *p,\n> >  \t\tstruct object_id oid;\n> >\n> >  \t\tif (flags & FOR_EACH_OBJECT_PACK_ORDER)\n> > -\t\t\tpos = p->revindex[i].nr;\n> > +\t\t\tpos = pack_pos_to_index(p, i);\n>\n> It wasn't too bad before this series formally defined what\n> \"position\", \"index\" and \"offset\" mean, but now this has become\n> highly misleading. The variable \"pos\" here holds what we consider\n> \"index\" while \"i\" holds what we call \"position\" [*1*].\n>\n> >  \t\telse\n> >  \t\t\tpos = i;\n>\n> Perhaps renaming \"uint32_t pos\" to \"nth\" would avoid confusion?\n\nI agree that it can be confusing. Unfortunately in this spot, this\nvariable really does mean two things. If we set the\nFOR_EACH_OBJECT_PACK_ORDER bit in our flags, then the caller really\nwants the index position (and the objects to be delivered in pack\norder). But if we didn't set it, then the caller wants it in index\norder.\n\n> -\tif (nth_packed_object_id(&oid, p, pos) < 0)\n> +\tif (nth_packed_object_id(&oid, p, nth) < 0)\n> \t\treturn error(...);\n\nThis suggested diff makes me think that you understand all of that, so\nI'm mostly saying this for the benefit of others that haven't looked at\nthis code closely in the recent past.\n\nI'd be happy to send a replacement patch if you would like [1], but I'm\nhopeful that this is clear enough since there isn't much code between\nthe declaration, assignment(s), and use of 'pos'.\n\nThanks,\nTaylor\n\n[1]: I understand your general disdain for single replacement patches,\nbut I'd like to avoid sending the other 19 patches if possible to avoid\ndelivering more mail to list subscribers than is necessary.\n"},{"id":"414364","messageId":"YAB6DNk4wPBVbGtU@nand.local","threadId":"54961","inReplyTo":"xmqq1reoypzy.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 01/20] pack-revindex: introduce a new API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-14T17:06:20Z","receivedAt":"2021-01-14T17:07:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jan 13, 2021 at 10:46:57PM -0800, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> > +/*\n> > + * offset_to_pack_pos converts an object offset to a pack position. This\n> > + * function returns zero on success, and a negative number otherwise. The\n> > + * parameter 'pos' is usable only on success.\n> > + *\n> > + * If the reverse index has not yet been loaded, this function loads it lazily,\n> > + * and returns an negative number if an error was encountered.\n>\n> It is somewhat strange to see a function that yields a non-negative\n> \"position\" on success and a negative value to signal a failure to\n> have a separate pointer to the location to receive the true return\n> value.  Do we truly care the upper half of \"uint32_t\" (in other\n> words, do we seriously want to support more than 2G positions in a\n> pack)?\n\nI don't think that we care about that as much as we do about potential\nmisuse of a signed return value. There are indeed a couple of spots\nwhere a potential negative return value is ignored, and then used to\nlookup an object in a pack, or some such.\n\nAnd that's part of the goal of this API: we have strict guidelines about\nwhen the output parameter is and isn't usable. That makes it more\ndifficult to accidentally use an uninitialized value / negative number.\n\n> What I'm trying to get at is that [...] is more natural than [...] but\n> now I wrote it down and laid it out in front of my eyes, the latter\n> does not look too bad.\n\nOK, good :-).\n\n> \t... later comes back after reading through the series ...\n>\n> \tThe new callers all looked quite nice to eyes.  Because we\n> \tdiscourage assignment inside if() condition, the converted\n> \tresult does not make the code more verbose than the\n> \toriginal.  In fact, it makes it even clearer that we are\n> \tchecking for an error return from a function call.\n>\n> \tQuite nice.\n\nThank you :-D.\n\n> > + * This function runs in time O(log N) with the number of objects in the pack.\n>\n> Is it a good idea to commit to such performance characteristics as a\n> promise to callers like this (the comment applies to all three\n> functions)?\n>\n> It depends on how a developer is helped by this comment when\n> deciding whether to use this function, or find other ways, to\n> implement what s/he wants to do.\n\nI don't mind it. If they all had the same performance characteristics, I\nwouldn't be for it, but since they don't, I think that it's good to\nknow. Peff suggested this back in [1].\n\nThanks,\nTaylor\n\n[1]: https://lore.kernel.org/git/X%2F1guCOGWybOzIS7@coredump.intra.peff.net/\n"},{"id":"414374","messageId":"YACZLHm4NtrM3POZ@coredump.intra.peff.net","threadId":"54961","inReplyTo":"YAB6DNk4wPBVbGtU@nand.local","subject":"Re: [PATCH v2 01/20] pack-revindex: introduce a new API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-14T19:19:08Z","receivedAt":"2021-01-14T19:20:01Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 14, 2021 at 12:06:20PM -0500, Taylor Blau wrote:\n\n> > > + * This function runs in time O(log N) with the number of objects in the pack.\n> >\n> > Is it a good idea to commit to such performance characteristics as a\n> > promise to callers like this (the comment applies to all three\n> > functions)?\n> >\n> > It depends on how a developer is helped by this comment when\n> > deciding whether to use this function, or find other ways, to\n> > implement what s/he wants to do.\n> \n> I don't mind it. If they all had the same performance characteristics, I\n> wouldn't be for it, but since they don't, I think that it's good to\n> know. Peff suggested this back in [1].\n\nYeah, I asked for this. As somebody who has frequently worked on the\ncode which accesses the revindex (mostly bitmap stuff), I found it\nuseful to understand how expensive the operations were.  However, I also\nknow what their runtimes are at this point, and it is not like somebody\ninterested cannot look at the implementation. So it may not be that\nimportant.\n\nSo I do still think it is useful, but if somebody feels strongly against\nit, I don't mind it being removed.\n\n-Peff\n"},{"id":"414376","messageId":"YACcoNY/SiEbBSgh@coredump.intra.peff.net","threadId":"54961","inReplyTo":"xmqq8s8wyq5i.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 15/20] for_each_object_in_pack(): convert to new revindex API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-14T19:33:52Z","receivedAt":"2021-01-14T19:34:35Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 13, 2021 at 10:43:37PM -0800, Junio C Hamano wrote:\n\n> >  \t\tif (flags & FOR_EACH_OBJECT_PACK_ORDER)\n> > -\t\t\tpos = p->revindex[i].nr;\n> > +\t\t\tpos = pack_pos_to_index(p, i);\n> \n> It wasn't too bad before this series formally defined what\n> \"position\", \"index\" and \"offset\" mean, but now this has become\n> highly misleading. The variable \"pos\" here holds what we consider\n> \"index\" while \"i\" holds what we call \"position\" [*1*].\n\nI don't think \"position\" is a meaningful term by itself. I would say the\nuseful terms are \"pack position\", \"index position\", and \"offset\" (or\n\"pack offset\" if you like). I don't think anything in the definitions\nadded by earlier patches contradicts that, but perhaps we can make it\nmore clear.\n\nSo \"pos\" in this case is not wrong. But I agree that it could stand to\nbe more clear. Saying \"nth\" does not help things IMHO (there is an \"nth\"\npack position, as well).\n\nBut maybe this makes it more clear (or possibly just the name change\nwithout the comment):\n\ndiff --git a/packfile.c b/packfile.c\nindex de47c9f4f8..6035b80466 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2078,19 +2078,30 @@ int for_each_object_in_pack(struct packed_git *p,\n \t}\n \n \tfor (i = 0; i < p->num_objects; i++) {\n-\t\tuint32_t pos;\n+\t\tuint32_t index_pos;\n \t\tstruct object_id oid;\n \n+\t\t/*\n+\t\t * We are iterating \"i\" from 0 up to num_objects, but its\n+\t\t * meaning may be different:\n+\t\t *\n+\t\t *   - in object-name order, it is the same as the index order\n+\t\t *     given to us by nth_packed_object_id(), and we can use it\n+\t\t *     directly\n+\t\t *\n+\t\t *   - in pack-order, it is pack position, which we must\n+\t\t *     convert to an index position in order to get the oid.\n+\t\t */\n \t\tif (flags & FOR_EACH_OBJECT_PACK_ORDER)\n-\t\t\tpos = p->revindex[i].nr;\n+\t\t\tindex_pos = p->revindex[i].nr;\n \t\telse\n-\t\t\tpos = i;\n+\t\t\tindex_pos = i;\n \n-\t\tif (nth_packed_object_id(&oid, p, pos) < 0)\n+\t\tif (nth_packed_object_id(&oid, p, index_pos) < 0)\n \t\t\treturn error(\"unable to get sha1 of object %u in %s\",\n-\t\t\t\t     pos, p->pack_name);\n+\t\t\t\t     index_pos, p->pack_name);\n \n-\t\tr = cb(&oid, p, pos, data);\n+\t\tr = cb(&oid, p, index_pos, data);\n \t\tif (r)\n \t\t\tbreak;\n \t}\n\n\n> *1* The nth_packed_object_id() call we make later using the value we\n> obtain here should be documented to take \"index\" as its last\n> parameter, now that is what we call the location in the index, which\n> is in object name order.\n\nI would love to see the function given a more descriptive name. Having\nworked on the bitmap code a lot, where the norm is pack-order, saying\n\"nth\" is confusing and error-prone.\n\nBut I think that's out of scope for this series.\n\n-Peff\n"},{"id":"414377","messageId":"YACe3KBgdRmIZA3b@coredump.intra.peff.net","threadId":"54961","inReplyTo":"YAB3E6lgOQdNgGOr@nand.local","subject":"Re: [PATCH v2 16/20] builtin/gc.c: guess the size of the revindex","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-14T19:43:24Z","receivedAt":"2021-01-14T19:44:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 14, 2021 at 11:53:39AM -0500, Taylor Blau wrote:\n\n> On Wed, Jan 13, 2021 at 10:33:01PM -0800, Junio C Hamano wrote:\n> > Taylor Blau <me@ttaylorr.com> writes:\n> >\n> > > 'estimate_repack_memory()' takes into account the amount of memory\n> > > required to load the reverse index in memory by multiplying the assumed\n> > > number of objects by the size of the 'revindex_entry' struct.\n> > >\n> > > Prepare for hiding the definition of 'struct revindex_entry' by removing\n> > > a 'sizeof()' of that type from outside of pack-revindex.c. Instead,\n> > > guess that one off_t and one uint32_t are required per object. Strictly\n> > > speaking, this is a worse guess than asking for 'sizeof(struct\n> > > revindex_entry)' directly, since the true size of this struct is 16\n> > > bytes with padding on the end of the struct in order to align the offset\n> > > field.\n> >\n> > Meaning that we under-estimate by 25%?\n> \n> In this area, yes. I'm skeptical that this estimate is all that\n> important, since it doesn't seem to take into account the memory\n> required to select delta/base candidates [1].\n\nIt has many other inaccuracies:\n\n  - it assumes half of all objects are blobs, which is not really\n    accurate (linux.git is more like 60% trees, 12% commits, 28% blobs).\n    This underestimates because blobs are the smallest struct.\n\n  - since we moved a bunch of stuff out of \"struct object_entry\" into\n    lazily-initialized auxiliary structures, we are under-counting the\n    per-object cost when we have to spill into this structures\n\nSo I'm rather skeptical that this number is close to accurate. But\nsince there's a bunch of leeway (we are looking to use half of the\nsystem memory) I suspect it doesn't matter all that much. But I\ndefinitely don't think it's worth trying to micro-optimize its accuracy.\n\n-Peff\n"},{"id":"414378","messageId":"YACgyn029KBps/yx@coredump.intra.peff.net","threadId":"54961","inReplyTo":"cover.1610576604.git.me@ttaylorr.com","subject":"Re: [PATCH v2 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-14T19:51:38Z","receivedAt":"2021-01-14T19:52:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 13, 2021 at 05:23:25PM -0500, Taylor Blau wrote:\n\n> In this revision, I addressed feedback from Junio, Peff, and Stolee. A\n> range-diff is included below, but the main changes are:\n> \n>   - Error messages are improved to include the pack and offset when applicable.\n>   - Variable names were made clearer (e.g., n -> index_pos).\n>   - Comments were added in pack-revindex.h to introduce relevant terminology,\n>     and which methods convert between what orderings.\n>   - int-sized lower- and upper-bounds were converted to be unsigned.\n\nThanks, this addresses all of my nits. I responded to a few of Junio's\nreviews with some further comments/suggestions; the most interesting one\nis using \"index_pos\" to indicate the ordering more clearly in patch 15.\nI'm happy with or without including that.\n\n-Peff\n"},{"id":"414382","messageId":"YAClXle+utN/VnVZ@coredump.intra.peff.net","threadId":"54961","inReplyTo":"YACcoNY/SiEbBSgh@coredump.intra.peff.net","subject":"Re: [PATCH v2 15/20] for_each_object_in_pack(): convert to new revindex API","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-14T20:11:10Z","receivedAt":"2021-01-14T20:11:52Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 14, 2021 at 02:33:53PM -0500, Jeff King wrote:\n\n> So \"pos\" in this case is not wrong. But I agree that it could stand to\n> be more clear. Saying \"nth\" does not help things IMHO (there is an \"nth\"\n> pack position, as well).\n> \n> But maybe this makes it more clear (or possibly just the name change\n> without the comment):\n\nHere it is again, but with a signoff and commit message, and done on top\nof your series (so if we agree this is a good resolution, it can just be\npicked up on top, but I am also happy for it to be squashed into patch\n15).\n\n-- >8 --\nSubject: [PATCH] for_each_object_in_pack(): clarify pack vs index ordering\n\nWe may return objects in one of two orders: how they appear in the .idx\n(sorted by object id) or how they appear in the packfile itself. To\nfurther complicate matters, we have two ordering variables, \"i\" and\n\"pos\", and it is not clear to which order they apply.\n\nLet's clarify this by using an unambiguous name where possible, and\nleaving a comment for the variable that does double-duty.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n packfile.c | 24 ++++++++++++++++++------\n 1 file changed, 18 insertions(+), 6 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 7bb1750934..35d50e2c38 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2082,19 +2082,31 @@ int for_each_object_in_pack(struct packed_git *p,\n \t}\n \n \tfor (i = 0; i < p->num_objects; i++) {\n-\t\tuint32_t pos;\n+\t\tuint32_t index_pos;\n \t\tstruct object_id oid;\n \n+\t\t/*\n+\t\t * We are iterating \"i\" from 0 up to num_objects, but its\n+\t\t * meaning may be different, depending on the requested output\n+\t\t * order:\n+\t\t *\n+\t\t *   - in object-name order, it is the same as the index order\n+\t\t *     used by nth_packed_object_id(), so we can pass it\n+\t\t *     directly\n+\t\t *\n+\t\t *   - in pack-order, it is pack position, which we must\n+\t\t *     convert to an index position in order to get the oid.\n+\t\t */\n \t\tif (flags & FOR_EACH_OBJECT_PACK_ORDER)\n-\t\t\tpos = pack_pos_to_index(p, i);\n+\t\t\tindex_pos = pack_pos_to_index(p, i);\n \t\telse\n-\t\t\tpos = i;\n+\t\t\tindex_pos = i;\n \n-\t\tif (nth_packed_object_id(&oid, p, pos) < 0)\n+\t\tif (nth_packed_object_id(&oid, p, index_pos) < 0)\n \t\t\treturn error(\"unable to get sha1 of object %u in %s\",\n-\t\t\t\t     pos, p->pack_name);\n+\t\t\t\t     index_pos, p->pack_name);\n \n-\t\tr = cb(&oid, p, pos, data);\n+\t\tr = cb(&oid, p, index_pos, data);\n \t\tif (r)\n \t\t\tbreak;\n \t}\n-- \n2.30.0.578.g0a9fb12091\n\n"},{"id":"414384","messageId":"YACmcG4bNugX3WfK@nand.local","threadId":"54961","inReplyTo":"YAClXle+utN/VnVZ@coredump.intra.peff.net","subject":"Re: [PATCH v2 15/20] for_each_object_in_pack(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-01-14T20:15:44Z","receivedAt":"2021-01-14T20:16:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jan 14, 2021 at 03:11:10PM -0500, Jeff King wrote:\n> On Thu, Jan 14, 2021 at 02:33:53PM -0500, Jeff King wrote:\n>\n> > So \"pos\" in this case is not wrong. But I agree that it could stand to\n> > be more clear. Saying \"nth\" does not help things IMHO (there is an \"nth\"\n> > pack position, as well).\n> >\n> > But maybe this makes it more clear (or possibly just the name change\n> > without the comment):\n>\n> Here it is again, but with a signoff and commit message, and done on top\n> of your series (so if we agree this is a good resolution, it can just be\n> picked up on top, but I am also happy for it to be squashed into patch\n> 15).\n\nMuch appreciated. This looks good to me (and I have no opinion whether\nit is picked up on top, or squashed into patch 15).\n\n  Acked-by: Taylor Blau <me@ttaylorr.com>\n\nThanks,\nTaylor\n"},{"id":"414390","messageId":"xmqq35z3xn30.fsf@gitster.c.googlers.com","threadId":"54961","inReplyTo":"YACZLHm4NtrM3POZ@coredump.intra.peff.net","subject":"Re: [PATCH v2 01/20] pack-revindex: introduce a new API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-14T20:47:31Z","receivedAt":"2021-01-14T20:48:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> On Thu, Jan 14, 2021 at 12:06:20PM -0500, Taylor Blau wrote:\n>\n>> > > + * This function runs in time O(log N) with the number of objects in the pack.\n>> >\n>> > Is it a good idea to commit to such performance characteristics as a\n>> > promise to callers like this (the comment applies to all three\n>> > functions)?\n>> >\n>> > It depends on how a developer is helped by this comment when\n>> > deciding whether to use this function, or find other ways, to\n>> > implement what s/he wants to do.\n>> \n>> I don't mind it. If they all had the same performance characteristics, I\n>> wouldn't be for it, but since they don't, I think that it's good to\n>> know. Peff suggested this back in [1].\n>\n> Yeah, I asked for this. As somebody who has frequently worked on the\n> code which accesses the revindex (mostly bitmap stuff), I found it\n> useful to understand how expensive the operations were.  However, I also\n> know what their runtimes are at this point, and it is not like somebody\n> interested cannot look at the implementation. So it may not be that\n> important.\n>\n> So I do still think it is useful, but if somebody feels strongly against\n> it, I don't mind it being removed.\n\nThat won't be me.  It's not like you'd use pack_pos_to_index()\ncombined with pack_pos_to_offset() instead of offset_to_pack_pos()\nbecause the latter is more expensive than using the other two\nfunctions; the comment does not help those who want to know relative\nperformance of these functions for such a purpose.\n\nI was just curious who the comments were meant to help, that's all.\n\nThanks.\n"},{"id":"414391","messageId":"xmqqv9bzw8bb.fsf@gitster.c.googlers.com","threadId":"54961","inReplyTo":"YACcoNY/SiEbBSgh@coredump.intra.peff.net","subject":"Re: [PATCH v2 15/20] for_each_object_in_pack(): convert to new revindex API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-14T20:51:52Z","receivedAt":"2021-01-14T20:52:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n>  \tfor (i = 0; i < p->num_objects; i++) {\n> -\t\tuint32_t pos;\n> +\t\tuint32_t index_pos;\n> ...\n>> *1* The nth_packed_object_id() call we make later using the value we\n>> obtain here should be documented to take \"index\" as its last\n>> parameter, now that is what we call the location in the index, which\n>> is in object name order.\n>\n> I would love to see the function given a more descriptive name. Having\n> worked on the bitmap code a lot, where the norm is pack-order, saying\n> \"nth\" is confusing and error-prone.\n>\n> But I think that's out of scope for this series.\n\nYeah, an explicit index_pos (vs pack_order_pos) would be good names\nto use, and nth_packed_object_id() can also use somewhere in its\nname to hint that it is about the object name order, but I agree\nthat both are outside the scope of this series.\n\nThanks.\n"},{"id":"414392","messageId":"xmqqr1mnw88w.fsf@gitster.c.googlers.com","threadId":"54961","inReplyTo":"YACgyn029KBps/yx@coredump.intra.peff.net","subject":"Re: [PATCH v2 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-14T20:53:19Z","receivedAt":"2021-01-14T20:54:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> On Wed, Jan 13, 2021 at 05:23:25PM -0500, Taylor Blau wrote:\n>\n>> In this revision, I addressed feedback from Junio, Peff, and Stolee. A\n>> range-diff is included below, but the main changes are:\n>> \n>>   - Error messages are improved to include the pack and offset when applicable.\n>>   - Variable names were made clearer (e.g., n -> index_pos).\n>>   - Comments were added in pack-revindex.h to introduce relevant terminology,\n>>     and which methods convert between what orderings.\n>>   - int-sized lower- and upper-bounds were converted to be unsigned.\n>\n> Thanks, this addresses all of my nits. I responded to a few of Junio's\n> reviews with some further comments/suggestions; the most interesting one\n> is using \"index_pos\" to indicate the ordering more clearly in patch 15.\n> I'm happy with or without including that.\n\nI think it is better to leave it for future \"clean-up\" outside the\nseries, to be done after the series hits a released version.\n\n"},{"id":"414416","messageId":"xmqqo8hruef3.fsf@gitster.c.googlers.com","threadId":"54961","inReplyTo":"YACmcG4bNugX3WfK@nand.local","subject":"Re: [PATCH v2 15/20] for_each_object_in_pack(): convert to new revindex API","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-01-15T02:22:56Z","receivedAt":"2021-01-15T02:23:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> On Thu, Jan 14, 2021 at 03:11:10PM -0500, Jeff King wrote:\n>> On Thu, Jan 14, 2021 at 02:33:53PM -0500, Jeff King wrote:\n>>\n>> > So \"pos\" in this case is not wrong. But I agree that it could stand to\n>> > be more clear. Saying \"nth\" does not help things IMHO (there is an \"nth\"\n>> > pack position, as well).\n>> >\n>> > But maybe this makes it more clear (or possibly just the name change\n>> > without the comment):\n>>\n>> Here it is again, but with a signoff and commit message, and done on top\n>> of your series (so if we agree this is a good resolution, it can just be\n>> picked up on top, but I am also happy for it to be squashed into patch\n>> 15).\n>\n> Much appreciated. This looks good to me (and I have no opinion whether\n> it is picked up on top, or squashed into patch 15).\n>\n>   Acked-by: Taylor Blau <me@ttaylorr.com>\n\nOK, so I'll make this 21/20 and rebase the other one...\n"},{"id":"414418","messageId":"YAD983DhrWGGBMAQ@nand.local","threadId":"54961","inReplyTo":"xmqqo8hruef3.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 15/20] for_each_object_in_pack(): convert to new revindex API","fromName":"Taylor Blau","fromEmail":"ttaylorr@github.com","sentAt":"2021-01-15T02:29:23Z","receivedAt":"2021-01-15T02:30:24Z","isPatch":true,"sender":{"key":"ttaylorr@github.com","avatar":"https://gravatar.com/avatar/d5f3476f26b6f99cbb6b467e7ed7482f5762c8157bc73f569196e428bdcbea25?d=mp&s=160"},"body":"On Thu, Jan 14, 2021 at 06:22:56PM -0800, Junio C Hamano wrote:\n> > Much appreciated. This looks good to me (and I have no opinion whether\n> > it is picked up on top, or squashed into patch 15).\n> >\n> >   Acked-by: Taylor Blau <me@ttaylorr.com>\n>\n> OK, so I'll make this 21/20 and rebase the other one...\n\nI'm happy if you want to apply this separately on top of both series,\nsince I think we all agree that this isn't strictly necessary to go\nforward.\n\nThat said, if you do make this 21/20 and the rebase of the second series\nrequires any intervention, please let me know and I'll be happy to send\nout a version myself.\n\nIn the meantime, I don't mind if you want to eject the latter topic from\nseen while it picks up review. The alternative (keeping it in and\nrebasing it forward), of course, is much appreciated.\n\n\nThanks,\nTaylor\n"},{"id":"414440","messageId":"YAFhMmEjk97sp7Dx@coredump.intra.peff.net","threadId":"54961","inReplyTo":"xmqqr1mnw88w.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-15T09:32:34Z","receivedAt":"2021-01-15T09:33:39Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 14, 2021 at 12:53:19PM -0800, Junio C Hamano wrote:\n\n> Jeff King <peff@peff.net> writes:\n> \n> > On Wed, Jan 13, 2021 at 05:23:25PM -0500, Taylor Blau wrote:\n> >\n> >> In this revision, I addressed feedback from Junio, Peff, and Stolee. A\n> >> range-diff is included below, but the main changes are:\n> >> \n> >>   - Error messages are improved to include the pack and offset when applicable.\n> >>   - Variable names were made clearer (e.g., n -> index_pos).\n> >>   - Comments were added in pack-revindex.h to introduce relevant terminology,\n> >>     and which methods convert between what orderings.\n> >>   - int-sized lower- and upper-bounds were converted to be unsigned.\n> >\n> > Thanks, this addresses all of my nits. I responded to a few of Junio's\n> > reviews with some further comments/suggestions; the most interesting one\n> > is using \"index_pos\" to indicate the ordering more clearly in patch 15.\n> > I'm happy with or without including that.\n> \n> I think it is better to leave it for future \"clean-up\" outside the\n> series, to be done after the series hits a released version.\n\nOK. I'll hold on to it and re-send it after the area has settled.\n\n-Peff\n"},{"id":"414441","messageId":"YAFhaeO0TeKw0G7q@coredump.intra.peff.net","threadId":"54961","inReplyTo":"YAFhMmEjk97sp7Dx@coredump.intra.peff.net","subject":"Re: [PATCH v2 00/20] pack-revindex: prepare for on-disk reverse index","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-01-15T09:33:29Z","receivedAt":"2021-01-15T09:34:17Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 15, 2021 at 04:32:34AM -0500, Jeff King wrote:\n\n> On Thu, Jan 14, 2021 at 12:53:19PM -0800, Junio C Hamano wrote:\n> \n> > Jeff King <peff@peff.net> writes:\n> > \n> > > On Wed, Jan 13, 2021 at 05:23:25PM -0500, Taylor Blau wrote:\n> > >\n> > >> In this revision, I addressed feedback from Junio, Peff, and Stolee. A\n> > >> range-diff is included below, but the main changes are:\n> > >> \n> > >>   - Error messages are improved to include the pack and offset when applicable.\n> > >>   - Variable names were made clearer (e.g., n -> index_pos).\n> > >>   - Comments were added in pack-revindex.h to introduce relevant terminology,\n> > >>     and which methods convert between what orderings.\n> > >>   - int-sized lower- and upper-bounds were converted to be unsigned.\n> > >\n> > > Thanks, this addresses all of my nits. I responded to a few of Junio's\n> > > reviews with some further comments/suggestions; the most interesting one\n> > > is using \"index_pos\" to indicate the ordering more clearly in patch 15.\n> > > I'm happy with or without including that.\n> > \n> > I think it is better to leave it for future \"clean-up\" outside the\n> > series, to be done after the series hits a released version.\n> \n> OK. I'll hold on to it and re-send it after the area has settled.\n\nOh, nevermind, you said elsewhere you'd pick it up as 21/20. Assuming\nyou do that, I won't resend it. :)\n\n-Peff\n"}]}