git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v4 0/4] Changed path filter hash fix and version bump

From
JTJonathan Tan <jonathantanmy@google.com>
Date
Jun 13, 2023, 17:39 UTC
Message-ID
<cover.1686677910.git.jonathantanmy@google.com>
In-Reply-To
<cover.1684790529.git.jonathantanmy@google.com>

Thanks Ramsay for spotting the errors and mentioning that I can use octal escapes. Here's an update taking into account their comments.

Jonathan Tan (4):
  gitformat-commit-graph: describe version 2 of BDAT
  t4216: test changed path filters with high bit paths
  repo-settings: introduce commitgraph.changedPathsVersion
  commit-graph: new filter ver. that fixes murmur3
 Documentation/config/commitgraph.txt     | 16 ++++-
 Documentation/gitformat-commit-graph.txt |  9 ++-
 bloom.c                                  | 65 +++++++++++++++++++-
 bloom.h                                  |  8 ++-
 commit-graph.c                           | 29 +++++++--
 oss-fuzz/fuzz-commit-graph.c             |  2 +-
 repo-settings.c                          |  6 +-
 repository.h                             |  2 +-
 t/helper/test-bloom.c                    |  9 ++-
 t/t0095-bloom.sh                         |  8 +++
 t/t4216-log-bloom.sh                     | 77 ++++++++++++++++++++++++
 11 files changed, 210 insertions(+), 21 deletions(-)
Range-diff against v3:
1:  d4b63945f6 ! 1:  a3b52af4c9 gitformat-commit-graph: describe version 2 of BDAT
    @@ Documentation/gitformat-commit-graph.txt: All multi-byte numbers are in network
      	described in https://doi.org/10.1007/978-3-540-30494-4_26 "Bloom Filters
     -	in Probabilistic Verification"
     +	in Probabilistic Verification". Version 1 bloom filters have a bug that appears
    -+	when int is signed and the repository has path names that have characters >=
    ++	when char is signed and the repository has path names that have characters >=
     +	0x80; Git supports reading and writing them, but this ability will be removed
     +	in a future version of Git.
            - The number of times a path is hashed and hence the number of bit positions
2:  aa4535776e ! 2:  f095e2b486 t4216: test changed path filters with high bit paths
    @@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation backfills empty comm
     +}
     +
     +# chosen to be the same under all Unicode normalization forms
    -+CENT=$(printf "\xc2\xa2")
    ++CENT=$(printf "\302\242")
     +
    -+# Some systems (in particular, Linux on the CI running on GitHub at the time of
    -+# writing) store into CENT a literal backslash, then "x", and so on (instead of
    -+# the high-bit characters needed). In these systems, do not run the following
    -+# tests.
    -+if test "$(printf $CENT | perl -0777 -ne 'no utf8; print ord($_)')" = "194"
    -+then
    -+	test_set_prereq HIGH_BIT
    -+fi
    -+
    -+test_expect_success HIGH_BIT 'set up repo with high bit path, version 1 changed-path' '
    ++test_expect_success 'set up repo with high bit path, version 1 changed-path' '
     +	git init highbit1 &&
     +	test_commit -C highbit1 c1 "$CENT" &&
     +	git -C highbit1 commit-graph write --reachable --changed-paths
     +'
     +
    -+test_expect_success HIGH_BIT 'setup check value of version 1 changed-path' '
    ++test_expect_success 'setup check value of version 1 changed-path' '
     +	(cd highbit1 &&
     +		printf "52a9" >expect &&
     +		get_first_changed_path_filter >actual)
     +'
     +
    -+# expect will not match actual if int is unsigned by default. Write the test
    ++# expect will not match actual if char is unsigned by default. Write the test
     +# in this way, so that a user running this test script can still see if the two
     +# files match. (It will appear as an ordinary success if they match, and a skip
     +# if not.)
     +if test_cmp highbit1/expect highbit1/actual
     +then
    -+	test_set_prereq SIGNED_INT_BY_DEFAULT
    ++	test_set_prereq SIGNED_CHAR_BY_DEFAULT
     +fi
    -+test_expect_success SIGNED_INT_BY_DEFAULT 'check value of version 1 changed-path' '
    ++test_expect_success SIGNED_CHAR_BY_DEFAULT 'check value of version 1 changed-path' '
     +	# Only the prereq matters for this test.
     +	true
     +'
     +
    -+test_expect_success HIGH_BIT 'version 1 changed-path used when version 1 requested' '
    ++test_expect_success 'version 1 changed-path used when version 1 requested' '
     +	(cd highbit1 &&
     +		test_bloom_filters_used "-- $CENT")
     +'
3:  d6982268a4 = 3:  6adfa53daf repo-settings: introduce commitgraph.changedPathsVersion
4:  e879483c42 ! 4:  5c65bf8a22 commit-graph: new filter ver. that fixes murmur3
    @@ t/t0095-bloom.sh: test_expect_success 'compute unseeded murmur3 hash for test st
      	Hashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|
     
      ## t/t4216-log-bloom.sh ##
    -@@ t/t4216-log-bloom.sh: test_expect_success HIGH_BIT 'version 1 changed-path used when version 1 request
    +@@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when version 1 requested' '
      		test_bloom_filters_used "-- $CENT")
      '
      
    -+test_expect_success HIGH_BIT 'version 1 changed-path not used when version 2 requested' '
    ++test_expect_success 'version 1 changed-path not used when version 2 requested' '
     +	(cd highbit1 &&
     +		git config --add commitgraph.changedPathsVersion 2 &&
     +		test_bloom_filters_not_used "-- $CENT")
     +'
     +
    -+test_expect_success HIGH_BIT 'set up repo with high bit path, version 2 changed-path' '
    ++test_expect_success 'set up repo with high bit path, version 2 changed-path' '
     +	git init highbit2 &&
     +	git -C highbit2 config --add commitgraph.changedPathsVersion 2 &&
     +	test_commit -C highbit2 c2 "$CENT" &&
     +	git -C highbit2 commit-graph write --reachable --changed-paths
     +'
     +
    -+test_expect_success HIGH_BIT 'check value of version 2 changed-path' '
    ++test_expect_success 'check value of version 2 changed-path' '
     +	(cd highbit2 &&
     +		printf "c01f" >expect &&
     +		get_first_changed_path_filter >actual &&
     +		test_cmp expect actual)
     +'
     +
    -+test_expect_success HIGH_BIT 'version 2 changed-path used when version 2 requested' '
    ++test_expect_success 'version 2 changed-path used when version 2 requested' '
     +	(cd highbit2 &&
     +		test_bloom_filters_used "-- $CENT")
     +'
     +
    -+test_expect_success HIGH_BIT 'version 2 changed-path not used when version 1 requested' '
    ++test_expect_success 'version 2 changed-path not used when version 1 requested' '
     +	(cd highbit2 &&
     +		git config --add commitgraph.changedPathsVersion 1 &&
     +		test_bloom_filters_not_used "-- $CENT")
-- 
2.41.0.162.gfafddb0af9-goog
Previous: Junio C HamanoNext: Jonathan Tan
Message 36 of 116 in “Changed path filter hash fix and version bump”
  1. 0/2 Changed path filter hash fix and version bumpJonathan Tan, May 22, 2023
  2. 1/2 t4216: test wrong bloom filter version rejectionJonathan Tan, May 22, 2023
  3. 2/2 commit-graph: fix murmur3, bump filter ver. to 2Jonathan Tan, May 22, 2023
  4. Derrick StoleeMay 23, 2023
  5. Jonathan TanMay 23, 2023
  6. Junio C HamanoMay 23, 2023
  7. Jonathan TanMay 24, 2023
  8. Derrick StoleeMay 26, 2023
  9. Jonathan TanMay 30, 2023
  10. Junio C HamanoMay 23, 2023
  11. 0/3 Changed path filter hash fix and version bumpJonathan Tan, May 31, 2023
  12. 1/3 t4216: test changed path filters with high bit pathsJonathan Tan, May 31, 2023
  13. 2/3 repo-settings: introduce commitgraph.changedPathsVersionJonathan Tan, May 31, 2023
  14. 3/3 commit-graph: new filter ver. that fixes murmur3Jonathan Tan, May 31, 2023
  15. Junio C HamanoJun 3, 2023
  16. Junio C HamanoJun 3, 2023
  17. Jonathan TanJun 7, 2023
  18. Jonathan TanJun 7, 2023
  19. 0/4 Changed path filter hash fix and version bumpJonathan Tan, Jun 8, 2023
  20. 1/4 gitformat-commit-graph: describe version 2 of BDATJonathan Tan, Jun 8, 2023
  21. Ramsay JonesJun 8, 2023
  22. Junio C HamanoJun 12, 2023
  23. 2/4 t4216: test changed path filters with high bit pathsJonathan Tan, Jun 8, 2023
  24. 3/4 repo-settings: introduce commitgraph.changedPathsVersionJonathan Tan, Jun 8, 2023
  25. 4/4 commit-graph: new filter ver. that fixes murmur3Jonathan Tan, Jun 8, 2023
  26. Ramsay JonesJun 8, 2023
  27. Jonathan TanJun 9, 2023
  28. Junio C HamanoJun 12, 2023
  29. Jonathan TanJun 13, 2023
  30. CodingGuidelines: use octal escapes, not hexJonathan Tan, Jun 13, 2023
  31. Eric SunshineJun 13, 2023
  32. Jonathan TanJun 13, 2023
  33. Eric SunshineJun 13, 2023
  34. Junio C HamanoJun 13, 2023
  35. Junio C HamanoJun 13, 2023
  36. 0/4 Changed path filter hash fix and version bumpJonathan Tan, Jun 13, 2023
  37. 2/4 t4216: test changed path filters with high bit pathsJonathan Tan, Jun 13, 2023
  38. 1/4 gitformat-commit-graph: describe version 2 of BDATJonathan Tan, Jun 13, 2023
  39. Junio C HamanoJun 13, 2023
  40. Derrick StoleeJun 20, 2023
  41. Taylor BlauJun 21, 2023
  42. Jonathan TanJun 22, 2023
  43. Derrick StoleeJun 23, 2023
  44. 3/4 repo-settings: introduce commitgraph.changedPathsVersionJonathan Tan, Jun 13, 2023
  45. Derrick StoleeJun 20, 2023
  46. Taylor BlauJun 21, 2023
  47. 4/4 commit-graph: new filter ver. that fixes murmur3Jonathan Tan, Jun 13, 2023
  48. Derrick StoleeJun 20, 2023
  49. Junio C HamanoJun 20, 2023
  50. Junio C HamanoJun 13, 2023
  51. Derrick StoleeJun 20, 2023
  52. Jonathan TanJun 20, 2023
  53. Taylor BlauJun 21, 2023
  54. Derrick StoleeJun 21, 2023
  55. Jonathan TanJun 22, 2023
  56. Derrick StoleeJun 23, 2023
  57. 0/4 Changed path filter hash fix and version bumpJonathan Tan, Jul 13, 2023
  58. 3/4 repo-settings: introduce commitgraph.changedPathsVersionJonathan Tan, Jul 13, 2023
  59. Taylor BlauJul 19, 2023
  60. Jonathan TanJul 20, 2023
  61. Taylor BlauJul 20, 2023
  62. 2/4 t4216: test changed path filters with high bit pathsJonathan Tan, Jul 13, 2023
  63. Junio C HamanoJul 13, 2023
  64. Taylor BlauJul 19, 2023
  65. 0/4 commit-graph: avoid looking at Bloom filter data directlyTaylor Blau, Jul 19, 2023
  66. 1/4 t/helper/test-read-graph.c: extract `dump_graph_info()`Taylor Blau, Jul 19, 2023
  67. 2/4 bloom.h: make `load_bloom_filter_from_graph()` publicTaylor Blau, Jul 19, 2023
  68. 3/4 t/helper/test-read-graph: implement `bloom-filters` modeTaylor Blau, Jul 19, 2023
  69. 4/4 fixup! t4216: test changed path filters with high bit pathsTaylor Blau, Jul 19, 2023
  70. Junio C HamanoJul 19, 2023
  71. Jonathan TanJul 20, 2023
  72. 1/4 gitformat-commit-graph: describe version 2 of BDATJonathan Tan, Jul 13, 2023
  73. Taylor BlauJul 19, 2023
  74. Jonathan TanJul 20, 2023
  75. Taylor BlauJul 21, 2023
  76. 4/4 commit-graph: new filter ver. that fixes murmur3Jonathan Tan, Jul 13, 2023
  77. Taylor BlauJul 19, 2023
  78. Jonathan TanJul 20, 2023
  79. Taylor BlauJul 26, 2023
  80. Junio C HamanoJul 13, 2023
  81. Junio C HamanoJul 13, 2023
  82. Jonathan TanJul 14, 2023
  83. 0/7 Changed path filter hash fix and version bumpJonathan Tan, Jul 20, 2023
  84. 1/7 gitformat-commit-graph: describe version 2 of BDATJonathan Tan, Jul 20, 2023
  85. 2/7 t/helper/test-read-graph.c: extract `dump_graph_info()`Jonathan Tan, Jul 20, 2023
  86. Taylor BlauJul 26, 2023
  87. 3/7 bloom.h: make `load_bloom_filter_from_graph()` publicJonathan Tan, Jul 20, 2023
  88. 5/7 t4216: test changed path filters with high bit pathsJonathan Tan, Jul 20, 2023
  89. Taylor BlauJul 26, 2023
  90. 4/7 t/helper/test-read-graph: implement `bloom-filters` modeJonathan Tan, Jul 20, 2023
  91. 6/7 repo-settings: introduce commitgraph.changedPathsVersionJonathan Tan, Jul 20, 2023
  92. 7/7 commit-graph: new filter ver. that fixes murmur3Jonathan Tan, Jul 20, 2023
  93. Junio C HamanoJul 25, 2023
  94. Junio C HamanoJul 26, 2023
  95. Taylor BlauJul 27, 2023
  96. Junio C HamanoJul 27, 2023
  97. Jonathan TanJul 27, 2023
  98. Taylor BlauJul 27, 2023
  99. Jonathan TanJul 27, 2023
  100. Taylor BlauAug 1, 2023
  101. Jonathan TanAug 1, 2023
  102. Taylor BlauAug 3, 2023
  103. Derrick StoleeAug 3, 2023
  104. Taylor BlauAug 3, 2023
  105. Junio C HamanoJul 27, 2023
  106. 0/7 Changed path filter hash fix and version bumpJonathan Tan, Aug 1, 2023
  107. 1/7 gitformat-commit-graph: describe version 2 of BDATJonathan Tan, Aug 1, 2023
  108. 2/7 t/helper/test-read-graph.c: extract `dump_graph_info()`Jonathan Tan, Aug 1, 2023
  109. 3/7 bloom.h: make `load_bloom_filter_from_graph()` publicJonathan Tan, Aug 1, 2023
  110. 4/7 t/helper/test-read-graph: implement `bloom-filters` modeJonathan Tan, Aug 1, 2023
  111. 5/7 t4216: test changed path filters with high bit pathsJonathan Tan, Aug 1, 2023
  112. 6/7 repo-settings: introduce commitgraph.changedPathsVersionJonathan Tan, Aug 1, 2023
  113. 7/7 commit-graph: new filter ver. that fixes murmur3Jonathan Tan, Aug 1, 2023
  114. Junio C HamanoAug 1, 2023
  115. Taylor BlauAug 1, 2023
  116. Junio C HamanoAug 1, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.