git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 0/8] pack-revindex: introduce on-disk '.rev' format

From
Taylor Blau <me@ttaylorr.com>
Date
Jan 13, 2021, 22:28 UTC
Message-ID
<cover.1610576805.git.me@ttaylorr.com>
In-Reply-To
<cover.1610129989.git.me@ttaylorr.com>
Hi,

This is the second of two series to implement support for an on-disk format for storing the reverse index. Note that this depends on the patches in the previous series [1], which was recently updated).

This version is largely unchanged from the original, with the following exceptions:

  - It has been rebased onto the patches in the first series.
  - The operands of two comparisons in 'offset_to_pack_pos()' were swapped so
    that the smaller of the two appears on the left-hand side of the comparison.
  - A brown-paper-bag bug was fixed in tests so that they pass on Windows (last
    night's integration broke 'seen' on Windows).
  - The GIT_TEST_WRITE_REV_INDEX mode was enabled in the "all-features" test.
Thanks in advance for your review.
[1]: https://lore.kernel.org/git/cover.1610129796.git.me@ttaylorr.com/
Taylor Blau (8):
  packfile: prepare for the existence of '*.rev' files
  pack-write.c: prepare to write 'pack-*.rev' files
  builtin/index-pack.c: write reverse indexes
  builtin/pack-objects.c: respect 'pack.writeReverseIndex'
  Documentation/config/pack.txt: advertise 'pack.writeReverseIndex'
  t: prepare for GIT_TEST_WRITE_REV_INDEX
  t: support GIT_TEST_WRITE_REV_INDEX
  pack-revindex: ensure that on-disk reverse indexes are given
    precedence
 Documentation/config/pack.txt           |   7 ++
 Documentation/git-index-pack.txt        |  20 ++--
 Documentation/technical/pack-format.txt |  17 ++++
 builtin/index-pack.c                    |  67 +++++++++++--
 builtin/pack-objects.c                  |   9 ++
 builtin/repack.c                        |   1 +
 ci/run-build-and-tests.sh               |   1 +
 object-store.h                          |   3 +
 pack-revindex.c                         | 116 ++++++++++++++++++++--
 pack-revindex.h                         |  10 +-
 pack-write.c                            | 123 +++++++++++++++++++++++-
 pack.h                                  |   4 +
 packfile.c                              |  13 ++-
 packfile.h                              |   1 +
 t/README                                |   3 +
 t/t5319-multi-pack-index.sh             |   5 +-
 t/t5325-reverse-index.sh                |  97 +++++++++++++++++++
 t/t5604-clone-reference.sh              |   2 +-
 t/t5702-protocol-v2.sh                  |  12 ++-
 t/t6500-gc.sh                           |   6 +-
 t/t9300-fast-import.sh                  |   5 +-
 tmp-objdir.c                            |   4 +-
 22 files changed, 485 insertions(+), 41 deletions(-)
 create mode 100755 t/t5325-reverse-index.sh
 -:  ---------- >  1:  e1aa89244a pack-revindex: introduce a new API
 -:  ---------- >  2:  0fca7d5812 write_reuse_object(): convert to new revindex API
 -:  ---------- >  3:  7676822a54 write_reused_pack_one(): convert to new revindex API
 -:  ---------- >  4:  dd7133fdb7 write_reused_pack_verbatim(): convert to new revindex API
 -:  ---------- >  5:  8e93ca3886 check_object(): convert to new revindex API
 -:  ---------- >  6:  084bbf2145 bitmap_position_packfile(): convert to new revindex API
 -:  ---------- >  7:  68794e9484 show_objects_for_type(): convert to new revindex API
 -:  ---------- >  8:  31ac6f5703 get_size_by_pos(): convert to new revindex API
 -:  ---------- >  9:  acd80069a2 try_partial_reuse(): convert to new revindex API
 -:  ---------- > 10:  569acdca7f rebuild_existing_bitmaps(): convert to new revindex API
 -:  ---------- > 11:  9881637724 get_delta_base_oid(): convert to new revindex API
 -:  ---------- > 12:  df8bb571a5 retry_bad_packed_offset(): convert to new revindex API
 -:  ---------- > 13:  41b2e00947 packed_object_info(): convert to new revindex API
 -:  ---------- > 14:  8ad49d231f unpack_entry(): convert to new revindex API
 -:  ---------- > 15:  e757476351 for_each_object_in_pack(): convert to new revindex API
 -:  ---------- > 16:  a500311e33 builtin/gc.c: guess the size of the revindex
 -:  ---------- > 17:  67d14da04a pack-revindex: remove unused 'find_pack_revindex()'
 -:  ---------- > 18:  3b5c92be68 pack-revindex: remove unused 'find_revindex_position()'
 -:  ---------- > 19:  cabafce4a1 pack-revindex: hide the definition of 'revindex_entry'
 -:  ---------- > 20:  8400ff6c96 pack-revindex.c: avoid direct revindex access in 'offset_to_pack_pos()'
 1:  ddf47a0a50 ! 21:  6742c15c84 packfile: prepare for the existence of '*.rev' files
    @@ pack-revindex.c: static void create_pack_revindex(struct packed_git *p)
     +
      int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)
      {
    - 	int lo = 0;
    + 	unsigned lo, hi;
     @@ pack-revindex.c: int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos)
      uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos)
    @@ pack-revindex.c: int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_
     -	if (!p->revindex)
     +	if (!(p->revindex || p->revindex_data))
      		BUG("pack_pos_to_index: reverse index not yet loaded");
    - 	if (pos >= p->num_objects)
    + 	if (p->num_objects <= pos)
      		BUG("pack_pos_to_index: out-of-bounds object at %"PRIu32, pos);
     -	return p->revindex[pos].nr;
     +
    @@ pack-revindex.c: int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_
     -	if (!p->revindex)
     +	if (!(p->revindex || p->revindex_data))
      		BUG("pack_pos_to_index: reverse index not yet loaded");
    - 	if (pos > p->num_objects)
    + 	if (p->num_objects < pos)
      		BUG("pack_pos_to_offset: out-of-bounds object at %"PRIu32, pos);
     -	return p->revindex[pos].offset;
     +
    @@ pack-revindex.c: int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_
     +		return nth_packed_object_offset(p, pack_pos_to_index(p, pos));
      }
    + ## pack-revindex.h ##
    +@@ pack-revindex.h: struct packed_git;
    + /*
    +  * load_pack_revindex populates the revindex's internal data-structures for the
    +  * given pack, returning zero on success and a negative value otherwise.
    ++ *
    ++ * If a '.rev' file is present, it is checked for consistency, mmap'd, and
    ++ * pointers are assigned into it (instead of using the in-memory variant).
    +  */
    + int load_pack_revindex(struct packed_git *p);
    +
    +@@ pack-revindex.h: uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);
    +  * If the reverse index has not yet been loaded, or the position is out of
    +  * bounds, this function aborts.
    +  *
    +- * This function runs in constant time.
    ++ * This function runs in constant time under both in-memory and on-disk reverse
    ++ * indexes, but an additional step is taken to consult the corresponding .idx
    ++ * file when using the on-disk format.
    +  */
    + off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos);
    +
    +
      ## packfile.c ##
     @@ packfile.c: void close_pack_index(struct packed_git *p)
      	}
 2:  88393e2662 = 22:  8648c87fa7 pack-write.c: prepare to write 'pack-*.rev' files
 3:  b0a7329824 ! 23:  5b18ada611 builtin/index-pack.c: write reverse indexes
    @@ t/t5325-reverse-index.sh (new)
     +	rm -f $rev &&
     +	conf=$1 &&
     +	shift &&
    ++	# remove the index since Windows won't overwrite an existing file
    ++	rm $packdir/pack-$pack.idx &&
     +	git -c pack.writeReverseIndex=$conf index-pack "$@" \
     +		$packdir/pack-$pack.pack
     +}
 4:  e297a31875 = 24:  68bde3ea97 builtin/pack-objects.c: respect 'pack.writeReverseIndex'
 5:  5d3e96a498 = 25:  38a253d0ce Documentation/config/pack.txt: advertise 'pack.writeReverseIndex'
 6:  2288571fbe <  -:  ---------- t: prepare for GIT_TEST_WRITE_REV_INDEX
 -:  ---------- > 26:  12cdf2d67a t: prepare for GIT_TEST_WRITE_REV_INDEX
 7:  3525c4d114 ! 27:  6b647d9775 t: support GIT_TEST_WRITE_REV_INDEX
    @@ Commit message
         Add a new option that unconditionally enables the pack.writeReverseIndex
         setting in order to run the whole test suite in a mode that generates
    -    on-disk reverse indexes.
    +    on-disk reverse indexes. Additionally, enable this mode in the second
    +    run of tests under linux-gcc in 'ci/run-build-and-tests.sh'.
         Once on-disk reverse indexes are proven out over several releases, we
         can change the default value of that configuration to 'true', and drop
    @@ builtin/pack-objects.c: int cmd_pack_objects(int argc, const char **argv, const
      	progress = isatty(2);
      	argc = parse_options(argc, argv, prefix, pack_objects_options,
    + ## ci/run-build-and-tests.sh ##
    +@@ ci/run-build-and-tests.sh: linux-gcc)
    + 	export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1
    + 	export GIT_TEST_MULTI_PACK_INDEX=1
    + 	export GIT_TEST_ADD_I_USE_BUILTIN=1
    ++	export GIT_TEST_WRITE_REV_INDEX=1
    + 	make test
    + 	;;
    + linux-clang)
    +
      ## pack-revindex.h ##
     @@
    - #ifndef PACK_REVINDEX_H
    - #define PACK_REVINDEX_H
    +  *   can be found
    +  */
     +#define GIT_TEST_WRITE_REV_INDEX "GIT_TEST_WRITE_REV_INDEX"
     +
      struct packed_git;
    - int load_pack_revindex(struct packed_git *p);
    + /*
      ## t/README ##
     @@ t/README: GIT_TEST_DEFAULT_HASH=<hash-algo> specifies which hash algorithm to
 8:  6e580d43d1 ! 28:  48926ae182 pack-revindex: ensure that on-disk reverse indexes are given precedence
    @@ pack-revindex.c: static void create_pack_revindex(struct packed_git *p)
      ## pack-revindex.h ##
     @@
    - #define PACK_REVINDEX_H
    +  */
      #define GIT_TEST_WRITE_REV_INDEX "GIT_TEST_WRITE_REV_INDEX"
     +#define GIT_TEST_REV_INDEX_DIE_IN_MEMORY "GIT_TEST_REV_INDEX_DIE_IN_MEMORY"
    @@ t/t5325-reverse-index.sh: test_expect_success 'pack-objects respects pack.writeR
      '
     +test_expect_success 'reverse index is not generated when available on disk' '
    -+	git index-pack --rev-index $packdir/pack-$pack.pack &&
    ++	test_index_pack true &&
    ++	test_path_is_file $rev &&
     +
     +	git rev-parse HEAD >tip &&
     +	GIT_TEST_REV_INDEX_DIE_IN_MEMORY=1 git cat-file \
--
2.30.0.138.g6d7191ea01
Previous: Taylor BlauNext: Taylor Blau
Message 17 of 82 in “pack-revindex: introduce on-disk '.rev' format”
  1. 0/8 pack-revindex: introduce on-disk '.rev' formatTaylor Blau, Jan 8, 2021
  2. 2/8 pack-write.c: prepare to write 'pack-*.rev' filesTaylor Blau, Jan 8, 2021
  3. 8/8 pack-revindex: ensure that on-disk reverse indexes are given precedenceTaylor Blau, Jan 8, 2021
  4. 3/8 builtin/index-pack.c: write reverse indexesTaylor Blau, Jan 8, 2021
  5. 1/8 packfile: prepare for the existence of '*.rev' filesTaylor Blau, Jan 8, 2021
  6. 4/8 builtin/pack-objects.c: respect 'pack.writeReverseIndex'Taylor Blau, Jan 8, 2021
  7. 7/8 t: support GIT_TEST_WRITE_REV_INDEXTaylor Blau, Jan 8, 2021
  8. Derrick StoleeJan 12, 2021
  9. Taylor BlauJan 12, 2021
  10. Ævar Arnfjörð BjarmasonJan 12, 2021
  11. Derrick StoleeJan 12, 2021
  12. Taylor BlauJan 12, 2021
  13. 6/8 t: prepare for GIT_TEST_WRITE_REV_INDEXTaylor Blau, Jan 8, 2021
  14. Ævar Arnfjörð BjarmasonJan 12, 2021
  15. Taylor BlauJan 12, 2021
  16. 5/8 Documentation/config/pack.txt: advertise 'pack.writeReverseIndex'Taylor Blau, Jan 8, 2021
  17. 0/8 pack-revindex: introduce on-disk '.rev' formatTaylor Blau, Jan 13, 2021
  18. 8/8 pack-revindex: ensure that on-disk reverse indexes are given precedenceTaylor Blau, Jan 13, 2021
  19. 6/8 t: prepare for GIT_TEST_WRITE_REV_INDEXTaylor Blau, Jan 13, 2021
  20. 1/8 packfile: prepare for the existence of '*.rev' filesTaylor Blau, Jan 13, 2021
  21. Junio C HamanoJan 14, 2021
  22. Derrick StoleeJan 14, 2021
  23. Jeff KingJan 14, 2021
  24. Taylor BlauJan 14, 2021
  25. Junio C HamanoJan 14, 2021
  26. Taylor BlauJan 14, 2021
  27. Junio C HamanoJan 14, 2021
  28. Jeff KingJan 22, 2021
  29. Taylor BlauJan 25, 2021
  30. Jeff KingJan 25, 2021
  31. Junio C HamanoJan 25, 2021
  32. Taylor BlauJan 25, 2021
  33. 5/8 Documentation/config/pack.txt: advertise 'pack.writeReverseIndex'Taylor Blau, Jan 13, 2021
  34. 4/8 builtin/pack-objects.c: respect 'pack.writeReverseIndex'Taylor Blau, Jan 13, 2021
  35. Jeff KingJan 22, 2021
  36. Jeff KingJan 23, 2021
  37. Taylor BlauJan 25, 2021
  38. Jeff KingJan 25, 2021
  39. 2/8 pack-write.c: prepare to write 'pack-*.rev' filesTaylor Blau, Jan 13, 2021
  40. Jeff KingJan 22, 2021
  41. Taylor BlauJan 25, 2021
  42. Jeff KingJan 26, 2021
  43. 7/8 t: support GIT_TEST_WRITE_REV_INDEXTaylor Blau, Jan 13, 2021
  44. 3/8 builtin/index-pack.c: write reverse indexesTaylor Blau, Jan 13, 2021
  45. Jeff KingJan 22, 2021
  46. Taylor BlauJan 25, 2021
  47. 00/10 pack-revindex: introduce on-disk '.rev' formatTaylor Blau, Jan 25, 2021
  48. 01/10 packfile: prepare for the existence of '*.rev' filesTaylor Blau, Jan 25, 2021
  49. Jeff KingJan 29, 2021
  50. Taylor BlauJan 29, 2021
  51. Jeff KingJan 30, 2021
  52. 08/10 t: support GIT_TEST_WRITE_REV_INDEXTaylor Blau, Jan 25, 2021
  53. Jeff KingJan 29, 2021
  54. 07/10 t: prepare for GIT_TEST_WRITE_REV_INDEXTaylor Blau, Jan 25, 2021
  55. Jeff KingJan 29, 2021
  56. Eric SunshineJan 29, 2021
  57. Taylor BlauJan 29, 2021
  58. Jeff KingJan 30, 2021
  59. Junio C HamanoJan 29, 2021
  60. 05/10 builtin/pack-objects.c: respect 'pack.writeReverseIndex'Taylor Blau, Jan 25, 2021
  61. 09/10 pack-revindex: ensure that on-disk reverse indexes are given precedenceTaylor Blau, Jan 25, 2021
  62. Jeff KingJan 29, 2021
  63. Taylor BlauJan 29, 2021
  64. Jeff KingJan 30, 2021
  65. 10/10 t5325: check both on-disk and in-memory reverse indexTaylor Blau, Jan 25, 2021
  66. Jeff KingJan 29, 2021
  67. Jeff KingJan 29, 2021
  68. Taylor BlauJan 29, 2021
  69. Jeff KingJan 30, 2021
  70. 06/10 Documentation/config/pack.txt: advertise 'pack.writeReverseIndex'Taylor Blau, Jan 25, 2021
  71. Jeff KingJan 29, 2021
  72. Taylor BlauJan 29, 2021
  73. Jeff KingJan 30, 2021
  74. 02/10 pack-write.c: prepare to write 'pack-*.rev' filesTaylor Blau, Jan 25, 2021
  75. 04/10 builtin/index-pack.c: write reverse indexesTaylor Blau, Jan 25, 2021
  76. 03/10 builtin/index-pack.c: allow stripping arbitrary extensionsTaylor Blau, Jan 25, 2021
  77. Jeff KingJan 29, 2021
  78. Taylor BlauJan 29, 2021
  79. Junio C HamanoJan 26, 2021
  80. Taylor BlauJan 26, 2021
  81. Jeff KingJan 29, 2021
  82. Taylor BlauJan 29, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.