git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 00/15] bloom: changed-path Bloom filters v2

From
Taylor Blau <me@ttaylorr.com>
Date
Aug 21, 2023, 21:43 UTC
Message-ID
<cover.1692654233.git.me@ttaylorr.com>

This series combines the efforts in [1] and [2] to culminate in v2 of the changed-path Bloom filter format (which uses a bona-fide murmur3 implementation) as well as an efficient upgrade path for repositories with few non-ASCII paths.

The first seven patches are from [1], and are unchanged from that version. Most of the remaining patches are from [2], and they have been modified based on the review therein.

The final patch is new, and avoids leaking memory when the Bloom subsystem is initialized both in the read- and read/write-cases.

Thanks to Jonathan and Peff who have both helped a great deal in putting these patches together. And, as always, thanks in advance for your review!

[1]: https://lore.kernel.org/git/cover.1684790529.git.jonathantanmy@google.com/ [2]: https://lore.kernel.org/git/cover.1691426160.git.me@ttaylorr.com/

Jonathan Tan (4):
  gitformat-commit-graph: describe version 2 of BDAT
  t4216: test changed path filters with high bit paths
  repo-settings: introduce commitgraph.changedPathsVersion
  commit-graph: new filter ver. that fixes murmur3
Taylor Blau (11):
  t/helper/test-read-graph.c: extract `dump_graph_info()`
  bloom.h: make `load_bloom_filter_from_graph()` public
  t/helper/test-read-graph: implement `bloom-filters` mode
  bloom: annotate filters with hash version
  bloom: prepare to discard incompatible Bloom filters
  t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`
  commit-graph.c: unconditionally load Bloom filters
  commit-graph: drop unnecessary `graph_read_bloom_data_context`
  object.h: fix mis-aligned flag bits table
  commit-graph: reuse existing Bloom filters where possible
  bloom: introduce `deinit_bloom_filters()`
 Documentation/config/commitgraph.txt     |  26 ++-
 Documentation/gitformat-commit-graph.txt |   9 +-
 bloom.c                                  | 208 +++++++++++++++++++++--
 bloom.h                                  |  38 ++++-
 commit-graph.c                           |  36 +++-
 object.h                                 |   3 +-
 oss-fuzz/fuzz-commit-graph.c             |   2 +-
 repo-settings.c                          |   6 +-
 repository.h                             |   2 +-
 t/helper/test-bloom.c                    |   9 +-
 t/helper/test-read-graph.c               |  67 ++++++--
 t/t0095-bloom.sh                         |   8 +
 t/t4216-log-bloom.sh                     | 184 +++++++++++++++++++-
 13 files changed, 548 insertions(+), 50 deletions(-)
-- 
2.42.0.4.g52b49bb434
Next: Taylor Blau
Message 1 of 76 in “bloom: changed-path Bloom filters v2”
  1. 00/15 bloom: changed-path Bloom filters v2Taylor Blau, Aug 21, 2023
  2. 01/15 gitformat-commit-graph: describe version 2 of BDATTaylor Blau, Aug 21, 2023
  3. 02/15 t/helper/test-read-graph.c: extract `dump_graph_info()`Taylor Blau, Aug 21, 2023
  4. 03/15 bloom.h: make `load_bloom_filter_from_graph()` publicTaylor Blau, Aug 21, 2023
  5. 04/15 t/helper/test-read-graph: implement `bloom-filters` modeTaylor Blau, Aug 21, 2023
  6. 05/15 t4216: test changed path filters with high bit pathsTaylor Blau, Aug 21, 2023
  7. 06/15 repo-settings: introduce commitgraph.changedPathsVersionTaylor Blau, Aug 21, 2023
  8. 07/15 commit-graph: new filter ver. that fixes murmur3Taylor Blau, Aug 21, 2023
  9. SZEDER GáborAug 26, 2023
  10. Jonathan TanAug 29, 2023
  11. SZEDER GáborAug 30, 2023
  12. Jonathan TanSep 1, 2023
  13. Taylor BlauSep 25, 2023
  14. SZEDER GáborOct 8, 2023
  15. Taylor BlauOct 9, 2023
  16. Taylor BlauOct 9, 2023
  17. Junio C HamanoOct 9, 2023
  18. Taylor BlauOct 10, 2023
  19. 08/15 bloom: annotate filters with hash versionTaylor Blau, Aug 21, 2023
  20. 09/15 bloom: prepare to discard incompatible Bloom filtersTaylor Blau, Aug 21, 2023
  21. 10/15 t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`Taylor Blau, Aug 21, 2023
  22. 11/15 commit-graph.c: unconditionally load Bloom filtersTaylor Blau, Aug 21, 2023
  23. 12/15 commit-graph: drop unnecessary `graph_read_bloom_data_context`Taylor Blau, Aug 21, 2023
  24. 13/15 object.h: fix mis-aligned flag bits tableTaylor Blau, Aug 21, 2023
  25. 14/15 commit-graph: reuse existing Bloom filters where possibleTaylor Blau, Aug 21, 2023
  26. 15/15 bloom: introduce `deinit_bloom_filters()`Taylor Blau, Aug 21, 2023
  27. Jonathan TanAug 24, 2023
  28. Jonathan TanAug 25, 2023
  29. Jonathan TanAug 29, 2023
  30. Junio C HamanoAug 29, 2023
  31. 00/15 bloom: changed-path Bloom filters v2Jonathan Tan, Aug 30, 2023
  32. 13/15 object.h: fix mis-aligned flag bits tableJonathan Tan, Aug 30, 2023
  33. 06/15 repo-settings: introduce commitgraph.changedPathsVersionJonathan Tan, Aug 30, 2023
  34. 04/15 t/helper/test-read-graph: implement `bloom-filters` modeJonathan Tan, Aug 30, 2023
  35. 08/15 bloom: annotate filters with hash versionJonathan Tan, Aug 30, 2023
  36. 01/15 gitformat-commit-graph: describe version 2 of BDATJonathan Tan, Aug 30, 2023
  37. 11/15 commit-graph.c: unconditionally load Bloom filtersJonathan Tan, Aug 30, 2023
  38. 10/15 t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`Jonathan Tan, Aug 30, 2023
  39. 09/15 bloom: prepare to discard incompatible Bloom filtersJonathan Tan, Aug 30, 2023
  40. 12/15 commit-graph: drop unnecessary `graph_read_bloom_data_context`Jonathan Tan, Aug 30, 2023
  41. 07/15 commit-graph: new filter ver. that fixes murmur3Jonathan Tan, Aug 30, 2023
  42. 02/15 t/helper/test-read-graph.c: extract `dump_graph_info()`Jonathan Tan, Aug 30, 2023
  43. 05/15 t4216: test changed path filters with high bit pathsJonathan Tan, Aug 30, 2023
  44. 14/15 commit-graph: reuse existing Bloom filters where possibleJonathan Tan, Aug 30, 2023
  45. 03/15 bloom.h: make `load_bloom_filter_from_graph()` publicJonathan Tan, Aug 30, 2023
  46. 15/15 bloom: introduce `deinit_bloom_filters()`Jonathan Tan, Aug 30, 2023
  47. Junio C HamanoAug 30, 2023
  48. 00/17 bloom: changed-path Bloom filters v2 (& sundries)Taylor Blau, Oct 10, 2023
  49. 01/17 t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`Taylor Blau, Oct 10, 2023
  50. 02/17 revision.c: consult Bloom filters for root commitsTaylor Blau, Oct 10, 2023
  51. 04/17 gitformat-commit-graph: describe version 2 of BDATTaylor Blau, Oct 10, 2023
  52. 05/17 t/helper/test-read-graph.c: extract `dump_graph_info()`Taylor Blau, Oct 10, 2023
  53. Patrick SteinhardtOct 17, 2023
  54. Taylor BlauOct 18, 2023
  55. Junio C HamanoOct 18, 2023
  56. 03/17 commit-graph: ensure Bloom filters are read with consistent settingsTaylor Blau, Oct 10, 2023
  57. Patrick SteinhardtOct 17, 2023
  58. 06/17 bloom.h: make `load_bloom_filter_from_graph()` publicTaylor Blau, Oct 10, 2023
  59. 07/17 t/helper/test-read-graph: implement `bloom-filters` modeTaylor Blau, Oct 10, 2023
  60. 09/17 repo-settings: introduce commitgraph.changedPathsVersionTaylor Blau, Oct 10, 2023
  61. 15/17 object.h: fix mis-aligned flag bits tableTaylor Blau, Oct 10, 2023
  62. 12/17 bloom: prepare to discard incompatible Bloom filtersTaylor Blau, Oct 10, 2023
  63. 13/17 commit-graph.c: unconditionally load Bloom filtersTaylor Blau, Oct 10, 2023
  64. Patrick SteinhardtOct 17, 2023
  65. 11/17 bloom: annotate filters with hash versionTaylor Blau, Oct 10, 2023
  66. 14/17 commit-graph: drop unnecessary `graph_read_bloom_data_context`Taylor Blau, Oct 10, 2023
  67. 10/17 commit-graph: new filter ver. that fixes murmur3Taylor Blau, Oct 10, 2023
  68. Patrick SteinhardtOct 17, 2023
  69. Taylor BlauOct 18, 2023
  70. 08/17 t4216: test changed path filters with high bit pathsTaylor Blau, Oct 10, 2023
  71. Patrick SteinhardtOct 17, 2023
  72. Taylor BlauOct 18, 2023
  73. 16/17 commit-graph: reuse existing Bloom filters where possibleTaylor Blau, Oct 10, 2023
  74. 17/17 bloom: introduce `deinit_bloom_filters()`Taylor Blau, Oct 10, 2023
  75. Patrick SteinhardtOct 17, 2023
  76. Taylor BlauOct 18, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.