{"thread":{"id":"61166","subject":"[PATCH 00/24] pack-bitmap: pseudo-merge reachability bitmaps","startedAt":"2024-03-20T22:04:59Z","lastAt":"2024-05-25T03:26:31Z","messageCount":157,"participants":["Taylor Blau","Junio C Hamano","Jeff King","Patrick Steinhardt"],"isPatch":true,"patchVersion":1,"patchTotal":24},"messages":[{"id":"491043","messageId":"cover.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":null,"subject":"[PATCH 00/24] pack-bitmap: pseudo-merge reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:04:57Z","receivedAt":"2024-03-20T22:04:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This series implements a new idea in the pack-bitmap machinery called\n\"pseudo-merge reachability bitmaps\".\n\nBACKGROUND\n==========\n\nPseudo-merge bitmaps conceptually represent storing bitmaps for\noctopus-merges covering the reference tips in a repository not selected\nfor bitmaps, decreasing the size (number of parents) of consecutive\nmerges according to an inverse power-law.\n\n(Note that a complete description of the on-disk format can be found in\nDocumentation/technical/bitmap-format.txt, and the configuration\ndetails in the \"bitmapPseudoMerge\" section of 'git-config(1)').\n\nCommits are assigned into pseudo-merge group(s) according to a number of\nparameters unique to each defined group, like so:\n\n [bitmapPseudoMerge \"name\"]\n pattern = \"refs/*\"\n decay = 100\n sampleRate = 100\n threshold = 1.week.ago\n maxMerges = 64\n stableThreshold = 1.month.ago\n stableSize = 1024\n\nUsers can create zero or more pseudo-merge groups. Pseudo-merge groups\nlook at reference tips matching the given pattern, and use the commits\nat the tips of those (matching) references as candidates for forming the\npseudo-merge groups.\n\nThe full details are spelled out in git-config(1), but the gist is as\nfollows:\n\n - \"decay\" controls the decay rate between consecutive groups\n (effectively setting 'k' in the decay function 'f(n)=C*n^-k')\n\n - \"sampleRate\" controls how often (between 0 and 100, inclusive) to\n select a candidate for inclusion within a pseudo-merge group\n\n - \"threshold\" sets the minimum age for an un-bitmapped reference tip\n to join a pseudo-merge gruop\n\n - \"maxMerges\" controls how many unstable pseudo-merge groups we'll\n create\n\nIn addition, there are a pair of parameters controlling \"stable\"\npseudo-merge groups, meant to cover commits that are so old as to be\nhighly unlikely to change. \"stableThreshold\" controls the minimum age,\nand \"stableSize\" controls the fixed size of each of these groups. Stable\npseudo-merges do not follow the inverse power-law decay function as\nabove.\n\nPseudo-merges can be grouped evevn further if the given pattern has one\nor more capture groups, similar to delta islands. In a fork-network\nrepository (where each fork is a remote, and all objects live in\naggregate in a repository called \"network.git\"), you can group\nbranches/tags separately by fork ID like so:\n\n [bitmapPseudoMerge \"branches\"]\n pattern = \"refs/remotes/([0-9])+/heads/\"\n [...]\n\n [bitmapPseudoMerge \"tags\"]\n pattern = \"refs/remotes/([0-9])+/tags/\"\n [...]\n\nPseudo-merges are an additive optimization. A pseudo-merge may only be\nused when all of its parents are part of either the \"haves\" or \"wants\"\nside of a reachability query.\n\nThis is why groups with \"older\" commits tend to be larger (since they\nare less likely to change, thus more likely to be usable over time). The\nsame is true for \"newer\" pseudo-merge groups being smaller.\n\nUSAGE\n=====\n\nHere's a best-case scenario for using pseudo-merge bitmaps. This is a\nlocal copy of a relatively large (private) repository that has a large\nnumber of references (~44k) with poor bitmap coverage, spike-y branches,\nand deep-ish trees.\n\nFirst, we generate a new bitmap with 64 pseudo-merge commits covering\nthe un-bitmapped parts of this repository, like so.\n\n $ GIT_TRACE2_PERF=1 GIT_TRACE2_PERF_BRIEF=1 \\\n git.compile \\\n -c bitmapPseudoMerge.all.pattern='refs/' \\\n -c bitmapPseudoMerge.all.threshold=now \\\n -c bitmapPseudoMerge.all.stableThreshold=never \\\n -c bitmapPseudoMerge.all.maxMerges=64 \\\n -c pack.writeBitmapLookupTable=true \\\n repack -adb\n [...]\n d1 | main | data | r1 | 476.227928 | 0.058245 | progress | ....total_objects:1\n d1 | main | region_leave | r1 | 476.227938 | 0.058255 | progress | ..label:Selecting pseudo-merge commits\n d1 | main | region_enter | r1 | 476.227946 | | progress | ..label:Building bitmaps\n d1 | main | region_enter | r1 | 476.227949 | | pack-bitmap- | ....label:building_bitmaps_total\n d1 | main | data | r1 | 476.228348 | 0.000399 | bitmap | ......opened bitmap file:.git/objects/pack/pack-47e6bced8a8f64d82802a77f2f1cf5eeac24f295.pack\n d1 | main | data | r1 | 485.947246 | 9.719297 | pack-bitmap- | ......num_selected_commits:733\n d1 | main | data | r1 | 485.947269 | 9.719320 | pack-bitmap- | ......num_maximal_commits:115\n d1 | main | region_leave | r1 | 1033.008932 | 556.780983 | pack-bitmap- | ....label:building_bitmaps_total\n d1 | main | data | r1 | 1033.008953 | 556.781007 | pack-bitmap- | ....building_bitmaps_reused:700\n d1 | main | data | r1 | 1033.008957 | 556.781011 | pack-bitmap- | ....building_bitmaps_pseudo_merge_reused:0\n d1 | main | data | r1 | 1033.008968 | 556.781022 | progress | ....total_objects:733\n d1 | main | region_leave | r1 | 1033.008971 | 556.781025 | progress | ..label:Building bitmaps\n\n(Note: this repository had an up-to-date .bitmap prior to generating\npseudo-merges, so we spend about ~9 minutes (on an unoptimized build)\ngenerating psuedo-merges. This feels expensive, but maybe my intuition\nis wrong. I've spent quite a bit of time trying to drive this down, but\nI still think there is some medium-hanging fruit left here to get this\nto be even cheaper.)\n\nPERFORMANCE\n===========\n\nFrom there, we can compare the time it takes to generate a count of\nreachable objects with and without using pseudo-merge bitmaps:\n\n $ hyperfine -L v ,.compile 'git{v} rev-list --all --objects --count --use-bitmap-index'\n Benchmark 1: git rev-list --all --objects --count --use-bitmap-index\n Time (mean ± σ): 16.129 s ± 0.079 s [User: 15.681 s, System: 0.446 s]\n Range (min … max): 16.029 s … 16.243 s 10 runs\n\n Benchmark 2: git.compile rev-list --all --objects --count --use-bitmap-index\n Time (mean ± σ): 874.9 ms ± 20.4 ms [User: 611.4 ms, System: 263.3 ms]\n Range (min … max): 847.1 ms … 904.3 ms 10 runs\n\n Summary\n git.compile rev-list --all --objects --count --use-bitmap-index ran\n 18.43 ± 0.44 times faster than git rev-list --all --objects --count --use-bitmap-index\n\nSimilarly for clone performance (this impact here is less dramatic than\nabove, primarily because we end up accumulating most objects via\npack-reuse, but enumeration is faster by roughly the same quantity as\nabove):\n\n $ hyperfine --runs=3 -L v ,.compile 'git{v} pack-objects \\\n --all --use-bitmap-index --delta-base-offset \\\n --stdout --all-progress /dev/null'\n Benchmark 1: git pack-objects \\\n --all --use-bitmap-index --delta-base-offset \\\n --stdout --all-progress /dev/null\n Time (mean ± σ): 93.811 s ± 2.202 s [User: 80.905 s, System: 7.674 s]\n Range (min … max): 92.136 s … 96.305 s 3 runs\n\n Benchmark 2: git.compile pack-objects \\\n --all --use-bitmap-index --delta-base-offset \\\n --stdout --all-progress /dev/null\n Time (mean ± σ): 79.608 s ± 0.764 s [User: 68.304 s, System: 7.215 s]\n Range (min … max): 78.912 s … 80.425 s 3 runs\n\n Summary\n git.compile pack-objects --all --use-bitmap-index --delta-base-offset --stdout --all-progress /dev/null ran\n 1.18 ± 0.03 times faster than git pack-objects --all --use-bitmap-index --delta-base-offset --stdout --all-progress /dev/null\n\nIn a smaller repository, like git.git, the perfomrance number are much\nless dramatic (here we're generating the numbers from the new p5333\nperformance test), but show that we do not pay a performance penalty for\nusing pseudo-merges:\n\n Test this tree\n -----------------------------------------------------------------------------------\n 5333.2: git rev-list --count --all --objects (no bitmaps) 3.46(3.37+0.09)\n 5333.3: git rev-list --count --all --objects (no pseudo-merges) 0.13(0.11+0.01)\n 5333.4: git rev-list --count --all --objects (with pseudo-merges) 0.12(0.11+0.01)\n\nCONCLUSION\n==========\n\nThis series turned out to be quite the adventure to work on ;-). Many\nthanks to Peff, who developed this idea together with me during the end\nof October, 2021.\n\nReview on this large topic is greatly appreciated. Likewise, if folks\nhave good ideas on either (a) how to speed-up generating these bitmaps,\nor (b) a sense of whether or not the current performance is acceptable,\nI am all ears.\n\nThanks in advance for anyone interested in reviewing this topic, and I\nlook forward to your feedback!\n\nTaylor Blau (24):\n Documentation/technical: describe pseudo-merge bitmaps format\n config: repo_config_get_expiry()\n ewah: implement `ewah_bitmap_is_subset()`\n pack-bitmap: drop unused `max_bitmaps` parameter\n pack-bitmap: move some initialization to `bitmap_writer_init()`\n pseudo-merge.ch: initial commit\n pack-bitmap-write: support storing pseudo-merge commits\n pack-bitmap: implement `bitmap_writer_has_bitmapped_object_id()`\n pack-bitmap: make `bitmap_writer_push_bitmapped_commit()` public\n pseudo-merge: implement support for selecting pseudo-merge commits\n pack-bitmap-write.c: select pseudo-merge commits\n pack-bitmap-write.c: write pseudo-merge table\n pack-bitmap: extract `read_bitmap()` function\n pseudo-merge: scaffolding for reads\n pack-bitmap.c: read pseudo-merge extension\n pseudo-merge: implement support for reading pseudo-merge commits\n ewah: implement `ewah_bitmap_popcount()`\n pack-bitmap: implement test helpers for pseudo-merge\n t/test-lib-functions.sh: support `--date` in `test_commit_bulk()`\n pack-bitmap.c: use pseudo-merges during traversal\n pack-bitmap: extra trace2 information\n ewah: `bitmap_equals_ewah()`\n pseudo-merge: implement support for finding existing merges\n t/perf: implement performace tests for pseudo-merge bitmaps\n\n Documentation/config.txt | 2 +\n Documentation/config/bitmap-pseudo-merge.txt | 75 ++\n Documentation/technical/bitmap-format.txt | 205 +++++\n Makefile | 1 +\n builtin/pack-objects.c | 3 +-\n config.c | 18 +\n config.h | 2 +\n ewah/bitmap.c | 76 ++\n ewah/ewok.h | 3 +\n midx.c | 3 +-\n pack-bitmap-write.c | 275 ++++++-\n pack-bitmap.c | 359 ++++++++-\n pack-bitmap.h | 16 +-\n pseudo-merge.c | 739 +++++++++++++++++++\n pseudo-merge.h | 218 ++++++\n t/helper/test-bitmap.c | 34 +-\n t/perf/p5333-pseudo-merge-bitmaps.sh | 32 +\n t/t5333-pseudo-merge-bitmaps.sh | 389 ++++++++++\n t/test-lib-functions.sh | 12 +-\n 19 files changed, 2401 insertions(+), 61 deletions(-)\n create mode 100644 Documentation/config/bitmap-pseudo-merge.txt\n create mode 100644 pseudo-merge.c\n create mode 100644 pseudo-merge.h\n create mode 100755 t/perf/p5333-pseudo-merge-bitmaps.sh\n create mode 100755 t/t5333-pseudo-merge-bitmaps.sh\n\n\nbase-commit: 3bd955d26919e149552f34aacf8a4e6368c26cec\n-- \n2.44.0.303.g1dc5e5b124c\n"},{"id":"491045","messageId":"76e7e3b9cca7fb957384ed98f2cd32cf11ff8488.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 01/24] Documentation/technical: describe pseudo-merge bitmaps format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:02Z","receivedAt":"2024-03-20T22:05:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to implement pseudo-merge bitmaps over the next several commits\nby first describing the serialization format which will store the new\npseudo-merge bitmaps themselves.\n\nThis format is implemented as an optional extension within the bitmap v1\nformat, making it compatible with previous versions of Git, as well as\nthe original .bitmap implementation within JGit.\n\nThe format (as well as a general description of pseudo-merge bitmaps,\nand motivating use-case(s)) is described in detail in the patch contents\nbelow, but the high-level description is as follows:\n\n - An array of pseudo-merge bitmaps, each containing a pair of EWAH\n bitmaps: one describing the set of pseudo-merge \"parents\", and\n another describing the set of object(s) reachable from those\n parents.\n\n - A lookup table to determine which pseudo-merge(s) a given commit\n appears in. An optional extended lookup table follows when there is\n at least one commit which appears in multiple pseudo-merge groups.\n\n - Trailing metadata, including the number of pseudo-merge(s), number\n of unique parents, the offset within the .bitmap file for the\n pseudo-merge commit lookup table, and the size of the optional\n extension itself.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/technical/bitmap-format.txt | 179 ++++++++++++++++++++++\n 1 file changed, 179 insertions(+)\n\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex f5d200939b0..63a7177ac08 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -255,3 +255,182 @@ triplet is -\n \txor_row (4 byte integer, network byte order): ::\n \tThe position of the triplet whose bitmap is used to compress\n \tthis one, or `0xffffffff` if no such bitmap exists.\n+\n+Pseudo-merge bitmaps\n+--------------------\n+\n+If the `BITMAP_OPT_PSEUDO_MERGES` flag is set, a variable number of\n+bytes (preceding the name-hash cache, commit lookup table, and trailing\n+checksum) of the `.bitmap` file is used to store pseudo-merge bitmaps.\n+\n+A \"pseudo-merge bitmap\" is used to refer to a pair of bitmaps, as\n+follows:\n+\n+Commit bitmap::\n+\n+ A bitmap whose set bits describe the set of commits included in the\n+ pseudo-merge's \"merge\" bitmap (as below).\n+\n+Merge bitmap::\n+\n+ A bitmap whose set bits describe the reachability closure over the set\n+ of commits in the pseudo-merge's \"commits\" bitmap (as above). An\n+ identical bitmap would be generated for an octopus merge with the same\n+ set of parents as described in the commits bitmap.\n+\n+Pseudo-merge bitmaps can accelerate bitmap traversals when all commits\n+for a given pseudo-merge are listed on either side of the traversal,\n+either directly (by explicitly asking for them as part of the `HAVES`\n+or `WANTS`) or indirectly (by encountering them during a fill-in\n+traversal).\n+\n+=== Use-cases\n+\n+For example, suppose there exists a pseudo-merge bitmap with a large\n+number of commits, all of which are listed in the `WANTS` section of\n+some bitmap traversal query. When pseudo-merge bitmaps are enabled, the\n+bitmap machinery can quickly determine there is a pseudo-merge which\n+satisfies some subset of the wanted objects on either side of the query.\n+Then, we can inflate the EWAH-compressed bitmap, and `OR` it in to the\n+resulting bitmap. By contrast, without pseudo-merge bitmaps, we would\n+have to repeat the decompression and `OR`-ing step over a potentially\n+large number of individual bitmaps, which can take proportionally more\n+time.\n+\n+Another benefit of pseudo-merges arises when there is some combination\n+of (a) a large number of references, with (b) poor bitmap coverage, and\n+(c) deep, nested trees, making fill-in traversal relatively expensive.\n+For example, suppose that there are a large enough number of tags where\n+bitmapping each of the tags individually is infeasible. Without\n+pseudo-merge bitmaps, computing the result of, say, `git rev-list\n+--use-bitmap-index --count --objects --tags` would likely require a\n+large amount of fill-in traversal. But when a large quantity of those\n+tags are stored together in a pseudo-merge bitmap, the bitmap machinery\n+can take advantage of the fact that we only care about the union of\n+objects reachable from all of those tags, and answer the query much\n+faster.\n+\n+=== File format\n+\n+If enabled, pseudo-merge bitmaps are stored in an optional section at\n+the end of a `.bitmap` file. The format is as follows:\n+\n+....\n++-------------------------------------------+\n+| .bitmap File |\n++-------------------------------------------+\n+| |\n+| Pseudo-merge bitmaps (Variable Length) |\n+| +---------------------------+ |\n+| | commits_bitmap (EWAH) | |\n+| +---------------------------+ |\n+| | merge_bitmap (EWAH) | |\n+| +---------------------------+ |\n+| |\n++-------------------------------------------+\n+| |\n+| Lookup Table |\n+| +------------+--------------+ |\n+| | commit_pos | offset | |\n+| +------------+--------------+ |\n+| | 4 bytes | 8 bytes | |\n+| +------------+--------------+ |\n+| |\n+| Offset Cases: |\n+| ------------- |\n+| |\n+| 1. MSB Unset: single pseudo-merge bitmap |\n+| + offset to pseudo-merge bitmap |\n+| |\n+| 2. MSB Set: multiple pseudo-merges |\n+| + offset to extended lookup table |\n+| |\n++-------------------------------------------+\n+| |\n+| Extended Lookup Table (Optional) |\n+| |\n+| +----+----------+----------+----------+ |\n+| | N | Offset 1 | .... | Offset N | |\n+| +----+----------+----------+----------+ |\n+| | | 8 bytes | .... | 8 bytes | |\n+| +----+----------+----------+----------+ |\n+| |\n++-------------------------------------------+\n+| |\n+| Pseudo-merge Metadata |\n+| +------------------+----------------+ |\n+| | # pseudo-merges | # Commits | |\n+| +------------------+----------------+ |\n+| | 4 bytes | 4 bytes | |\n+| +------------------+----------------+ |\n+| |\n+| +------------------+----------------+ |\n+| | Lookup offset | Extension size | |\n+| +------------------+----------------+ |\n+| | 8 bytes | 8 bytes | |\n+| +------------------+----------------+ |\n+| |\n++-------------------------------------------+\n+....\n+\n+* One or more pseudo-merge bitmaps, each containing:\n+\n+ ** `commits_bitmap`, an EWAH-compressed bitmap describing the set of\n+ commits included in the this psuedo-merge.\n+\n+ ** `merge_bitmap`, an EWAH-compressed bitmap describing the union of\n+ the set of objects reachable from all commits listed in the\n+ `commits_bitmap`.\n+\n+* A lookup table, mapping pseudo-merged commits to the pseudo-merges\n+ they belong to. Entries appear in increasing order of each commit's\n+ bit position. Each entry is 12 bytes wide, and is comprised of the\n+ following:\n+\n+ ** `commit_pos`, a 4-byte unsigned value (in network byte-order)\n+ containing the bit position for this commit.\n+\n+ ** `offset`, an 8-byte unsigned value (also in network byte-order)\n+ containing either one of two possible offsets, depending on whether or\n+ not the most-significant bit is set.\n+\n+ *** If unset (i.e. `offset & ((uint64_t)1<<63) == 0`), the offset\n+\t(relative to the beginning of the `.bitmap` file) at which the\n+\tpseudo-merge bitmap for this commit can be read. This indicates\n+\tonly a single pseudo-merge bitmap contains this commit.\n+\n+ *** If set (i.e. `offset & ((uint64_t)1<<63) != 0`), the offset\n+\t(again relative to the beginning of the `.bitmap` file) at which\n+\tthe extended offset table can be located describing the set of\n+\tpseudo-merge bitmaps which contain this commit. This indicates\n+\tthat multiple pseudo-merge bitmaps contain this commit.\n+\n+* An (optional) extended lookup table (written if and only if there is\n+ at least one commit which appears in more than one pseudo-merge).\n+ There are as many entries as commits which appear in multiple\n+ pseudo-merges. Each entry contains the following:\n+\n+ ** `N`, a 4-byte unsigned value equal to the number of pseudo-merges\n+ which contain a given commit.\n+\n+ ** An array of `N` 8-byte unsigned values, each of which is\n+ interpreted as an offset (relative to the beginning of the\n+ `.bitmap` file) at which a pseudo-merge bitmap for this commit can\n+ be read. These values occur in no particular order.\n+\n+* Positions for all pseudo-merges, each stored as an 8-byte unsigned\n+ value (in network byte-order) containing the offset (relative to the\n+ beginnign of the `.bitmap` file) of each consecutive pseudo-merge.\n+\n+* A 4-byte unsigned value (in network byte-order) equal to the number of\n+ pseudo-merges.\n+\n+* A 4-byte unsigned value (in network byte-order) equal to the number of\n+ unique commits which appear in any pseudo-merge.\n+\n+* An 8-byte unsigned value (in network byte-order) equal to the number\n+ of bytes between the start of the pseudo-merge section and the\n+ beginning of the lookup table.\n+\n+* An 8-byte unsigned value (in network byte-order) equal to the number\n+ of bytes in the pseudo-merge section (including this field).\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491044","messageId":"21d8f9dc2b4ddc8ac3f4e8f6b21bfb762fc6ab77.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 02/24] config: repo_config_get_expiry()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:05Z","receivedAt":"2024-03-20T22:05:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Callers interested in parsing an approxidate from configuration\ncurrently make use of the `git_config_get_expiry()` function via the\nstandard `git_config()` callback.\n\nIntroduce a `repo_config_get_expiry()` variant in the style of functions\nintroduced by 3b256228a6 (config: read config from a repository object,\n2017-06-22) to read a single value without requiring the git_config()\ncallback-style approach.\n\nThis new function is similar to the existing implementation in\n`git_config_get_expiry()`, however it differs in that it fills out a\n`timestamp_t` value through a pointer, instead of merely checking and\ndiscarding the result (and returning it as a string).\n\nThis function will gain its first caller in a subsequent commit to parse\na \"threshold\" parameter for excluding too-recent commits from\npseudo-merge groups.\n\nSigned-off-by: Taylor Blau \n---\n config.c | 18 ++++++++++++++++++\n config.h | 2 ++\n 2 files changed, 20 insertions(+)\n\ndiff --git a/config.c b/config.c\nindex 3cfeb3d8bd9..8512da92273 100644\n--- a/config.c\n+++ b/config.c\n@@ -2627,6 +2627,24 @@ int repo_config_get_pathname(struct repository *repo,\n \treturn ret;\n }\n \n+int repo_config_get_expiry(struct repository *repo,\n+\t\t\t const char *key, const char **dest)\n+{\n+\tint ret;\n+\n+\tgit_config_check_init(repo);\n+\n+\tret = repo_config_get_string(repo, key, (char **)dest);\n+\tif (ret)\n+\t\treturn ret;\n+\tif (strcmp(*dest, \"now\")) {\n+\t\ttimestamp_t now = approxidate(\"now\");\n+\t\tif (approxidate(*dest) >= now)\n+\t\t\tgit_die_config(key, _(\"Invalid %s: '%s'\"), key, *dest);\n+\t}\n+\treturn ret;\n+}\n+\n /* Read values into protected_config. */\n static void read_protected_config(void)\n {\ndiff --git a/config.h b/config.h\nindex 5dba984f770..619db01bc27 100644\n--- a/config.h\n+++ b/config.h\n@@ -576,6 +576,8 @@ int repo_config_get_maybe_bool(struct repository *repo,\n \t\t\t const char *key, int *dest);\n int repo_config_get_pathname(struct repository *repo,\n \t\t\t const char *key, const char **dest);\n+int repo_config_get_expiry(struct repository *repo,\n+\t\t\t const char *key, const char **dest);\n \n /*\n * Functions for reading protected config. By definition, protected\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491046","messageId":"1347571ef4ca6329de58394bfea71927c8e08151.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 03/24] ewah: implement `ewah_bitmap_is_subset()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:08Z","receivedAt":"2024-03-20T22:05:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In order to know whether a given pseudo-merge (comprised of a \"parents\"\nand \"objects\" bitmaps) is \"satisfied\" and can be OR'd into the bitmap\nresult, we need to be able to quickly determine whether the \"parents\"\nbitmap is a subset of the current set of objects reachable on either\nside of a traversal.\n\nImplement a helper function to prepare for that, which determines\nwhether an EWAH bitmap (the parents bitmap from the pseudo-merge) is a\nsubset of a non-EWAH bitmap (in this case, the results bitmap from\neither side of the traversal).\n\nThis function makes use of the EWAH iterator to avoid inflating any part\nof the EWAH bitmap after we determine it is not a subset of the non-EWAH\nbitmap. This \"fail-fast\" allows us to avoid a potentially large amount\nof wasted effort.\n\nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 43 +++++++++++++++++++++++++++++++++++++++++++\n ewah/ewok.h | 1 +\n 2 files changed, 44 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex ac7e0af622a..5bdae3fb07b 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -138,6 +138,49 @@ void bitmap_or(struct bitmap *self, const struct bitmap *other)\n \t\tself->words[i] |= other->words[i];\n }\n \n+int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other)\n+{\n+\tstruct ewah_iterator it;\n+\teword_t word;\n+\tsize_t i;\n+\n+\tewah_iterator_init(&it, self);\n+\n+\tfor (i = 0; i < other->word_alloc; i++) {\n+\t\tif (!ewah_iterator_next(&word, &it)) {\n+\t\t\t/*\n+\t\t\t * If we reached the end of `self`, and haven't\n+\t\t\t * rejected `self` as a possible subset of\n+\t\t\t * `other` yet, then we are done and `self` is\n+\t\t\t * indeed a subset of `other`.\n+\t\t\t */\n+\t\t\treturn 1;\n+\t\t}\n+\t\tif (word & ~other->words[i]) {\n+\t\t\t/*\n+\t\t\t * Otherwise, compare the next two pairs of\n+\t\t\t * words. If the word from `self` has bit(s) not\n+\t\t\t * in the word from `other`, `self` is not a\n+\t\t\t * proper subset of `other`.\n+\t\t\t */\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * If we got to this point, there may be zero or more words\n+\t * remaining in `self`, with no remaining words left in `other`.\n+\t * If there are any bits set in the remaining word(s) in `self`,\n+\t * then `self` is not a proper subset of `other`.\n+\t */\n+\twhile (ewah_iterator_next(&word, &it))\n+\t\tif (word)\n+\t\t\treturn 0;\n+\n+\t/* `self` is definitely a subset of `other` */\n+\treturn 1;\n+}\n+\n void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other)\n {\n \tsize_t original_size = self->word_alloc;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex c11d76c6f33..c334833b201 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -180,6 +180,7 @@ int bitmap_get(struct bitmap *self, size_t pos);\n void bitmap_free(struct bitmap *self);\n int bitmap_equals(struct bitmap *self, struct bitmap *other);\n int bitmap_is_subset(struct bitmap *self, struct bitmap *other);\n+int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other);\n \n struct ewah_bitmap * bitmap_to_ewah(struct bitmap *bitmap);\n struct bitmap *ewah_to_bitmap(struct ewah_bitmap *ewah);\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491048","messageId":"c6a08dae037351f0a5ba8f1044837c894c471b4f.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 04/24] pack-bitmap: drop unused `max_bitmaps` parameter","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:11Z","receivedAt":"2024-03-20T22:05:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `max_bitmaps` parameter in `bitmap_writer_select_commits()` was\nintroduced back in 7cc8f97108 (pack-objects: implement bitmap writing,\n2013-12-21), making it original to the bitmap implementation in Git\nitself.\n\nWhen that patch was merged via 0f9e62e084 (Merge branch\n'jk/pack-bitmap', 2014-02-27), its sole caller in builtin/pack-objects.c\npassed a value of \"-1\" for `max_bitmaps`, indicating no limit.\n\nSince then, the only other caller (in midx.c, added via c528e17966\n(pack-bitmap: write multi-pack bitmaps, 2021-08-31)) also uses a value\nof \"-1\" for `max_bitmaps`.\n\nSince no callers have needed a finite limit for the `max_bitmaps`\nparameter in the nearly decade that has passed since 0f9e62e084, let's\nremove the parameter and any dead pieces of code connected to it.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 2 +-\n midx.c | 2 +-\n pack-bitmap-write.c | 8 +-------\n pack-bitmap.h | 2 +-\n 4 files changed, 4 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 329aeac8043..41281cae91f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1359,7 +1359,7 @@ static void write_pack_file(void)\n \t\t\t\tstop_progress(&progress_state);\n \n \t\t\t\tbitmap_writer_show_progress(progress);\n-\t\t\t\tbitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);\n+\t\t\t\tbitmap_writer_select_commits(indexed_commits, indexed_commits_nr);\n \t\t\t\tif (bitmap_writer_build(&to_pack) < 0)\n \t\t\t\t\tdie(_(\"failed to write bitmap index\"));\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\ndiff --git a/midx.c b/midx.c\nindex 85e1c2cd128..366bfbe18c8 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1330,7 +1330,7 @@ static int write_midx_bitmap(const char *midx_name,\n \tfor (i = 0; i < pdata->nr_objects; i++)\n \t\tindex[pack_order[i]] = &pdata->objects[i].idx;\n \n-\tbitmap_writer_select_commits(commits, commits_nr, -1);\n+\tbitmap_writer_select_commits(commits, commits_nr);\n \tret = bitmap_writer_build(pdata);\n \tif (ret < 0)\n \t\tgoto cleanup;\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 990a9498d73..3dc2408eca7 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -591,8 +591,7 @@ static int date_compare(const void *_a, const void *_b)\n }\n \n void bitmap_writer_select_commits(struct commit **indexed_commits,\n-\t\t\t\t unsigned int indexed_commits_nr,\n-\t\t\t\t int max_bitmaps)\n+\t\t\t\t unsigned int indexed_commits_nr)\n {\n \tunsigned int i = 0, j, next;\n \n@@ -615,11 +614,6 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tif (i + next >= indexed_commits_nr)\n \t\t\tbreak;\n \n-\t\tif (max_bitmaps > 0 && writer.selected_nr >= max_bitmaps) {\n-\t\t\twriter.selected_nr = max_bitmaps;\n-\t\t\tbreak;\n-\t\t}\n-\n \t\tif (next == 0) {\n \t\t\tchosen = indexed_commits[i];\n \t\t} else {\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex c7dea13217a..3f96608d5c1 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -110,7 +110,7 @@ int rebuild_bitmap(const uint32_t *reposition,\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t struct commit *commit);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n-\t\tunsigned int indexed_commits_nr, int max_bitmaps);\n+\t\t\t\t unsigned int indexed_commits_nr);\n int bitmap_writer_build(struct packing_data *to_pack);\n void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr,\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491047","messageId":"a65316567391160dddae70fb036f03858772014f.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 05/24] pack-bitmap: move some initialization to `bitmap_writer_init()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:13Z","receivedAt":"2024-03-20T22:05:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pack-bitmap-writer machinery uses a oidmap (backed by khash.h) to\nmap from commits selected for bitmaps (by OID) to a bitmapped_commit\nstructure (containing the bitmap itself, among other things like its XOR\noffset, etc.)\n\nThis map was initialized at the end of `bitmap_writer_build()`. New\nentries are added in `pack-bitmap-write.c::store_selected()`, which is\ncalled by the bitmap_builder machinery (which is responsible for\ntraversing history and generating the actual bitmaps).\n\nReorganize when this field is initialized and when entries are added to\nit so that we can quickly determine whether a commit is a candidate for\npseudo-merge selection, or not (since it was already selected to receive\na bitmap, and thus is ineligible for pseudo-merge inclusion).\n\nThe changes are as follows:\n\n - Introduce a new `bitmap_writer_init()` function which initializes\n the `writer.bitmaps` field (instead of waiting until the end of\n `bitmap_writer_build()`).\n\n - Add map entries in `push_bitmapped_commit()` (which is called via\n `bitmap_writer_select_commits()`) with OID keys and NULL values to\n track whether or not we *expect* to write a bitmap for some given\n commit.\n\n - Validate that a NULL entry is found matching the given key when we\n store a selected bitmap.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 1 +\n midx.c | 1 +\n pack-bitmap-write.c | 23 ++++++++++++++++++-----\n pack-bitmap.h | 1 +\n 4 files changed, 21 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 41281cae91f..34a431e3856 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1339,6 +1339,7 @@ static void write_pack_file(void)\n \t\t\t\t hash_to_hex(hash));\n \n \t\t\tif (write_bitmap_index) {\n+\t\t\t\tbitmap_writer_init(the_repository);\n \t\t\t\tbitmap_writer_set_checksum(hash);\n \t\t\t\tbitmap_writer_build_type_index(\n \t\t\t\t\t&to_pack, written_list, nr_written);\ndiff --git a/midx.c b/midx.c\nindex 366bfbe18c8..24d98120852 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1311,6 +1311,7 @@ static int write_midx_bitmap(const char *midx_name,\n \tfor (i = 0; i < pdata->nr_objects; i++)\n \t\tindex[i] = &pdata->objects[i].idx;\n \n+\tbitmap_writer_init(the_repository);\n \tbitmap_writer_show_progress(flags & MIDX_PROGRESS);\n \tbitmap_writer_build_type_index(pdata, index, pdata->nr_objects);\n \ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 3dc2408eca7..ad768959633 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -46,6 +46,11 @@ struct bitmap_writer {\n \n static struct bitmap_writer writer;\n \n+void bitmap_writer_init(struct repository *r)\n+{\n+\twriter.bitmaps = kh_init_oid_map();\n+}\n+\n void bitmap_writer_show_progress(int show)\n {\n \twriter.show_progress = show;\n@@ -117,11 +122,20 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n \n static inline void push_bitmapped_commit(struct commit *commit)\n {\n+\tint hash_ret;\n+\tkhiter_t hash_pos;\n+\n \tif (writer.selected_nr >= writer.selected_alloc) {\n \t\twriter.selected_alloc = (writer.selected_alloc + 32) * 2;\n \t\tREALLOC_ARRAY(writer.selected, writer.selected_alloc);\n \t}\n \n+\thash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);\n+\tif (!hash_ret)\n+\t\tdie(_(\"duplicate entry when writing bitmap index: %s\"),\n+\t\t oid_to_hex(&commit->object.oid));\n+\tkh_value(writer.bitmaps, hash_pos) = NULL;\n+\n \twriter.selected[writer.selected_nr].commit = commit;\n \twriter.selected[writer.selected_nr].bitmap = NULL;\n \twriter.selected[writer.selected_nr].flags = 0;\n@@ -466,14 +480,14 @@ static void store_selected(struct bb_commit *ent, struct commit *commit)\n {\n \tstruct bitmapped_commit *stored = &writer.selected[ent->idx];\n \tkhiter_t hash_pos;\n-\tint hash_ret;\n \n \tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n \n-\thash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);\n-\tif (hash_ret == 0)\n-\t\tdie(\"Duplicate entry when writing index: %s\",\n+\thash_pos = kh_get_oid_map(writer.bitmaps, commit->object.oid);\n+\tif (hash_pos == kh_end(writer.bitmaps))\n+\t\tdie(_(\"attempted to store non-selected commit: '%s'\"),\n \t\t oid_to_hex(&commit->object.oid));\n+\n \tkh_value(writer.bitmaps, hash_pos) = stored;\n }\n \n@@ -488,7 +502,6 @@ int bitmap_writer_build(struct packing_data *to_pack)\n \tuint32_t *mapping;\n \tint closed = 1; /* until proven otherwise */\n \n-\twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n \n \tif (writer.show_progress)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3f96608d5c1..dae2d68a338 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -97,6 +97,7 @@ int bitmap_has_oid_in_uninteresting(struct bitmap_index *, const struct object_i\n \n off_t get_disk_usage_from_bitmap(struct bitmap_index *, struct rev_info *);\n \n+void bitmap_writer_init(struct repository *r);\n void bitmap_writer_show_progress(int show);\n void bitmap_writer_set_checksum(const unsigned char *sha1);\n void bitmap_writer_build_type_index(struct packing_data *to_pack,\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491049","messageId":"c6f9170af0fad0f2b0c2062f43b211b12a362642.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 06/24] pseudo-merge.ch: initial commit","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:17Z","receivedAt":"2024-03-20T22:05:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Add a new (empty) header file to contain the implementation for\nselecting, reading, and applying pseudo-merge bitmaps.\n\nFor now this header and its corresponding implementation are left\nempty, but they will evolve over the course of subsequent commit(s).\n\nSigned-off-by: Taylor Blau \n---\n Makefile | 1 +\n pseudo-merge.c | 2 ++\n pseudo-merge.h | 6 ++++++\n 3 files changed, 9 insertions(+)\n create mode 100644 pseudo-merge.c\n create mode 100644 pseudo-merge.h\n\ndiff --git a/Makefile b/Makefile\nindex 4e255c81f22..fd050bd9d68 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1114,6 +1114,7 @@ LIB_OBJS += prompt.o\n LIB_OBJS += protocol.o\n LIB_OBJS += protocol-caps.o\n LIB_OBJS += prune-packed.o\n+LIB_OBJS += pseudo-merge.o\n LIB_OBJS += quote.o\n LIB_OBJS += range-diff.o\n LIB_OBJS += reachable.o\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nnew file mode 100644\nindex 00000000000..37e037ba272\n--- /dev/null\n+++ b/pseudo-merge.c\n@@ -0,0 +1,2 @@\n+#include \"git-compat-util.h\"\n+#include \"pseudo-merge.h\"\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nnew file mode 100644\nindex 00000000000..cab8ff6960a\n--- /dev/null\n+++ b/pseudo-merge.h\n@@ -0,0 +1,6 @@\n+#ifndef PSEUDO_MERGE_H\n+#define PSEUDO_MERGE_H\n+\n+#include \"git-compat-util.h\"\n+\n+#endif\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491050","messageId":"7acdee2b5f2eddfb143afa2982f40bb0136ccdd1.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 07/24] pack-bitmap-write: support storing pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:19Z","receivedAt":"2024-03-20T22:05:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to write pseudo-merge bitmaps by annotating individual bitmapped\ncommits (which are represented by the `bitmapped_commit` structure) with\nan extra bit indicating whether or not they are a pseudo-merge.\n\nIn subsequent commits, pseudo-merge bitmaps will be generated by\nallocating a fake commit node with parents covering the full set of\ncommits represented by the pseudo-merge bitmap. These commits will be\nadded to the set of \"selected\" commits as usual, but will be written\nspecially instead of being included with the rest of the selected\ncommits.\n\nMechanically speaking, there are two parts of this change:\n\n - The bitmapped_commit struct gets a new bit indicating whether it is\n a pseudo-merge, or an ordinary commit selected for bitmaps.\n\n - A handful of changes to only write out the non-pseudo-merge commits\n when enumerating through the selected array (see the new\n `bitmap_writer_selected_nr()` function). Pseudo-merge commits appear\n after all non-pseudo-merge commits, so it is safe to enumerate\n through the selected array like so:\n\n for (i = 0; i < bitmap_writer_selected_nr(); i++)\n if (writer.selected[i].pseudo_merge)\n BUG(\"unexpected pseudo-merge\");\n\n without encountering the BUG().\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 100 +++++++++++++++++++++++++++++---------------\n pack-bitmap.h | 1 +\n 2 files changed, 67 insertions(+), 34 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex ad768959633..b1e8a0ad66d 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -24,7 +24,7 @@ struct bitmapped_commit {\n \tstruct ewah_bitmap *write_as;\n \tint flags;\n \tint xor_offset;\n-\tuint32_t commit_pos;\n+\tunsigned pseudo_merge : 1;\n };\n \n struct bitmap_writer {\n@@ -39,6 +39,8 @@ struct bitmap_writer {\n \tstruct bitmapped_commit *selected;\n \tunsigned int selected_nr, selected_alloc;\n \n+\tuint32_t pseudo_merges_nr;\n+\n \tstruct progress *progress;\n \tint show_progress;\n \tunsigned char pack_checksum[GIT_MAX_RAWSZ];\n@@ -46,6 +48,11 @@ struct bitmap_writer {\n \n static struct bitmap_writer writer;\n \n+static inline int bitmap_writer_selected_nr(void)\n+{\n+\treturn writer.selected_nr - writer.pseudo_merges_nr;\n+}\n+\n void bitmap_writer_init(struct repository *r)\n {\n \twriter.bitmaps = kh_init_oid_map();\n@@ -120,25 +127,30 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n * Compute the actual bitmaps\n */\n \n-static inline void push_bitmapped_commit(struct commit *commit)\n+static void bitmap_writer_push_bitmapped_commit(struct commit *commit,\n+\t\t\t\t\t\tunsigned pseudo_merge)\n {\n-\tint hash_ret;\n-\tkhiter_t hash_pos;\n-\n \tif (writer.selected_nr >= writer.selected_alloc) {\n \t\twriter.selected_alloc = (writer.selected_alloc + 32) * 2;\n \t\tREALLOC_ARRAY(writer.selected, writer.selected_alloc);\n \t}\n \n-\thash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);\n-\tif (!hash_ret)\n-\t\tdie(_(\"duplicate entry when writing bitmap index: %s\"),\n-\t\t oid_to_hex(&commit->object.oid));\n-\tkh_value(writer.bitmaps, hash_pos) = NULL;\n+\tif (!pseudo_merge) {\n+\t\tint hash_ret;\n+\t\tkhiter_t hash_pos = kh_put_oid_map(writer.bitmaps,\n+\t\t\t\t\t\t commit->object.oid,\n+\t\t\t\t\t\t &hash_ret);\n+\n+\t\tif (!hash_ret)\n+\t\t\tdie(_(\"duplicate entry when writing bitmap index: %s\"),\n+\t\t\t oid_to_hex(&commit->object.oid));\n+\t\tkh_value(writer.bitmaps, hash_pos) = NULL;\n+\t}\n \n \twriter.selected[writer.selected_nr].commit = commit;\n \twriter.selected[writer.selected_nr].bitmap = NULL;\n \twriter.selected[writer.selected_nr].flags = 0;\n+\twriter.selected[writer.selected_nr].pseudo_merge = pseudo_merge;\n \n \twriter.selected_nr++;\n }\n@@ -168,16 +180,20 @@ static void compute_xor_offsets(void)\n \n \twhile (next < writer.selected_nr) {\n \t\tstruct bitmapped_commit *stored = &writer.selected[next];\n-\n \t\tint best_offset = 0;\n \t\tstruct ewah_bitmap *best_bitmap = stored->bitmap;\n \t\tstruct ewah_bitmap *test_xor;\n \n+\t\tif (stored->pseudo_merge)\n+\t\t\tgoto next;\n+\n \t\tfor (i = 1; i <= MAX_XOR_OFFSET_SEARCH; ++i) {\n \t\t\tint curr = next - i;\n \n \t\t\tif (curr < 0)\n \t\t\t\tbreak;\n+\t\t\tif (writer.selected[curr].pseudo_merge)\n+\t\t\t\tcontinue;\n \n \t\t\ttest_xor = ewah_pool_new();\n \t\t\tewah_xor(writer.selected[curr].bitmap, stored->bitmap, test_xor);\n@@ -193,6 +209,7 @@ static void compute_xor_offsets(void)\n \t\t\t}\n \t\t}\n \n+next:\n \t\tstored->xor_offset = best_offset;\n \t\tstored->write_as = best_bitmap;\n \n@@ -205,7 +222,8 @@ struct bb_commit {\n \tstruct bitmap *commit_mask;\n \tstruct bitmap *bitmap;\n \tunsigned selected:1,\n-\t\t maximal:1;\n+\t\t maximal:1,\n+\t\t pseudo_merge:1;\n \tunsigned idx; /* within selected array */\n };\n \n@@ -243,17 +261,18 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \trevs.first_parent_only = 1;\n \n \tfor (i = 0; i < writer->selected_nr; i++) {\n-\t\tstruct commit *c = writer->selected[i].commit;\n-\t\tstruct bb_commit *ent = bb_data_at(&bb->data, c);\n+\t\tstruct bitmapped_commit *bc = &writer->selected[i];\n+\t\tstruct bb_commit *ent = bb_data_at(&bb->data, bc->commit);\n \n \t\tent->selected = 1;\n \t\tent->maximal = 1;\n+\t\tent->pseudo_merge = bc->pseudo_merge;\n \t\tent->idx = i;\n \n \t\tent->commit_mask = bitmap_new();\n \t\tbitmap_set(ent->commit_mask, i);\n \n-\t\tadd_pending_object(&revs, &c->object, \"\");\n+\t\tadd_pending_object(&revs, &bc->commit->object, \"\");\n \t}\n \n \tif (prepare_revision_walk(&revs))\n@@ -430,8 +449,13 @@ static int fill_bitmap_commit(struct bb_commit *ent,\n \t\tstruct commit *c = prio_queue_get(queue);\n \n \t\tif (old_bitmap && mapping) {\n-\t\t\tstruct ewah_bitmap *old = bitmap_for_commit(old_bitmap, c);\n+\t\t\tstruct ewah_bitmap *old;\n \t\t\tstruct bitmap *remapped = bitmap_new();\n+\n+\t\t\tif (commit->object.flags & BITMAP_PSEUDO_MERGE)\n+\t\t\t\told = NULL;\n+\t\t\telse\n+\t\t\t\told = bitmap_for_commit(old_bitmap, c);\n \t\t\t/*\n \t\t\t * If this commit has an old bitmap, then translate that\n \t\t\t * bitmap and add its bits to this one. No need to walk\n@@ -450,12 +474,14 @@ static int fill_bitmap_commit(struct bb_commit *ent,\n \t\t * Mark ourselves and queue our tree. The commit\n \t\t * walk ensures we cover all parents.\n \t\t */\n-\t\tpos = find_object_pos(&c->object.oid, &found);\n-\t\tif (!found)\n-\t\t\treturn -1;\n-\t\tbitmap_set(ent->bitmap, pos);\n-\t\tprio_queue_put(tree_queue,\n-\t\t\t repo_get_commit_tree(the_repository, c));\n+\t\tif (!(c->object.flags & BITMAP_PSEUDO_MERGE)) {\n+\t\t\tpos = find_object_pos(&c->object.oid, &found);\n+\t\t\tif (!found)\n+\t\t\t\treturn -1;\n+\t\t\tbitmap_set(ent->bitmap, pos);\n+\t\t\tprio_queue_put(tree_queue,\n+\t\t\t\t repo_get_commit_tree(the_repository, c));\n+\t\t}\n \n \t\tfor (p = c->parents; p; p = p->next) {\n \t\t\tpos = find_object_pos(&p->item->object.oid, &found);\n@@ -483,6 +509,9 @@ static void store_selected(struct bb_commit *ent, struct commit *commit)\n \n \tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n \n+\tif (ent->pseudo_merge)\n+\t\treturn;\n+\n \thash_pos = kh_get_oid_map(writer.bitmaps, commit->object.oid);\n \tif (hash_pos == kh_end(writer.bitmaps))\n \t\tdie(_(\"attempted to store non-selected commit: '%s'\"),\n@@ -612,7 +641,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n-\t\t\tpush_bitmapped_commit(indexed_commits[i]);\n+\t\t\tbitmap_writer_push_bitmapped_commit(indexed_commits[i], 0);\n \t\treturn;\n \t}\n \n@@ -645,7 +674,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\t\t}\n \t\t}\n \n-\t\tpush_bitmapped_commit(chosen);\n+\t\tbitmap_writer_push_bitmapped_commit(chosen, 0);\n \n \t\ti += next + 1;\n \t\tdisplay_progress(writer.progress, i);\n@@ -683,8 +712,11 @@ static void write_selected_commits_v1(struct hashfile *f,\n {\n \tint i;\n \n-\tfor (i = 0; i < writer.selected_nr; ++i) {\n+\tfor (i = 0; i < bitmap_writer_selected_nr(); ++i) {\n \t\tstruct bitmapped_commit *stored = &writer.selected[i];\n+\t\tif (stored->pseudo_merge)\n+\t\t\tBUG(\"unexpected pseudo-merge among selected: %s\",\n+\t\t\t oid_to_hex(&stored->commit->object.oid));\n \n \t\tif (offsets)\n \t\t\toffsets[i] = hashfile_total(f);\n@@ -718,10 +750,10 @@ static void write_lookup_table(struct hashfile *f,\n \tuint32_t i;\n \tuint32_t *table, *table_inv;\n \n-\tALLOC_ARRAY(table, writer.selected_nr);\n-\tALLOC_ARRAY(table_inv, writer.selected_nr);\n+\tALLOC_ARRAY(table, bitmap_writer_selected_nr());\n+\tALLOC_ARRAY(table_inv, bitmap_writer_selected_nr());\n \n-\tfor (i = 0; i < writer.selected_nr; i++)\n+\tfor (i = 0; i < bitmap_writer_selected_nr(); i++)\n \t\ttable[i] = i;\n \n \t/*\n@@ -729,16 +761,16 @@ static void write_lookup_table(struct hashfile *f,\n \t * bitmap corresponds to j'th bitmapped commit (among the selected\n \t * commits) in lex order of OIDs.\n \t */\n-\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n+\tQSORT_S(table, bitmap_writer_selected_nr(), table_cmp, commit_positions);\n \n \t/* table_inv helps us discover that relationship (i'th bitmap\n \t * to j'th commit by j = table_inv[i])\n \t */\n-\tfor (i = 0; i < writer.selected_nr; i++)\n+\tfor (i = 0; i < bitmap_writer_selected_nr(); i++)\n \t\ttable_inv[table[i]] = i;\n \n \ttrace2_region_enter(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n-\tfor (i = 0; i < writer.selected_nr; i++) {\n+\tfor (i = 0; i < bitmap_writer_selected_nr(); i++) {\n \t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n \t\tuint32_t xor_offset = selected->xor_offset;\n \t\tuint32_t xor_row;\n@@ -809,7 +841,7 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \tmemcpy(header.magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE));\n \theader.version = htons(default_version);\n \theader.options = htons(flags | options);\n-\theader.entry_count = htonl(writer.selected_nr);\n+\theader.entry_count = htonl(bitmap_writer_selected_nr());\n \thashcpy(header.checksum, writer.pack_checksum);\n \n \thashwrite(f, &header, sizeof(header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz);\n@@ -821,9 +853,9 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n \t\tCALLOC_ARRAY(offsets, index_nr);\n \n-\tALLOC_ARRAY(commit_positions, writer.selected_nr);\n+\tALLOC_ARRAY(commit_positions, bitmap_writer_selected_nr());\n \n-\tfor (i = 0; i < writer.selected_nr; i++) {\n+\tfor (i = 0; i < bitmap_writer_selected_nr(); i++) {\n \t\tstruct bitmapped_commit *stored = &writer.selected[i];\n \t\tint commit_pos = oid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex dae2d68a338..ca9acd2f735 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -21,6 +21,7 @@ struct bitmap_disk_header {\n \tunsigned char checksum[GIT_MAX_RAWSZ];\n };\n \n+#define BITMAP_PSEUDO_MERGE (1u<<21)\n #define NEEDS_BITMAP (1u<<22)\n \n /*\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491051","messageId":"4fdd7dda2744a938f19b76324c76196b033dc2fc.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 08/24] pack-bitmap: implement `bitmap_writer_has_bitmapped_object_id()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:23Z","receivedAt":"2024-03-20T22:05:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to implement pseudo-merge bitmap selection by implementing a\nnecessary new function, `bitmap_writer_has_bitmapped_object_id()`.\n\nThis function returns whether or not the bitmap_writer selected the\ngiven object ID for bitmapping. This will allow the pseudo-merge\nmachinery to reject candidates for pseudo-merges if they have already\nbeen selected as an ordinary bitmap tip.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 5 +++++\n pack-bitmap.h | 2 ++\n 2 files changed, 7 insertions(+)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex b1e8a0ad66d..cd528f89a76 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -123,6 +123,11 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n \t}\n }\n \n+int bitmap_writer_has_bitmapped_object_id(const struct object_id *oid)\n+{\n+\treturn kh_get_oid_map(writer.bitmaps, *oid) != kh_end(writer.bitmaps);\n+}\n+\n /**\n * Compute the actual bitmaps\n */\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex ca9acd2f735..995d664cc89 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -98,6 +98,8 @@ int bitmap_has_oid_in_uninteresting(struct bitmap_index *, const struct object_i\n \n off_t get_disk_usage_from_bitmap(struct bitmap_index *, struct rev_info *);\n \n+int bitmap_writer_has_bitmapped_object_id(const struct object_id *oid);\n+\n void bitmap_writer_init(struct repository *r);\n void bitmap_writer_show_progress(int show);\n void bitmap_writer_set_checksum(const unsigned char *sha1);\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491052","messageId":"d74cf3e484d4c2bb8267717df1eb998108278510.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 09/24] pack-bitmap: make `bitmap_writer_push_bitmapped_commit()` public","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:26Z","receivedAt":"2024-03-20T22:05:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pseudo-merge selection code will be added in a subsequent commit,\nand will need a way to push the allocated commit structures into the\nbitmap writer from a separate compilation unit.\n\nMake the `bitmap_writer_push_bitmapped_commit()` function part of the\npack-bitmap.h header in order to make this possible.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 4 ++--\n pack-bitmap.h | 2 ++\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex cd528f89a76..e46978d494c 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -132,8 +132,8 @@ int bitmap_writer_has_bitmapped_object_id(const struct object_id *oid)\n * Compute the actual bitmaps\n */\n \n-static void bitmap_writer_push_bitmapped_commit(struct commit *commit,\n-\t\t\t\t\t\tunsigned pseudo_merge)\n+void bitmap_writer_push_bitmapped_commit(struct commit *commit,\n+\t\t\t\t\t unsigned pseudo_merge)\n {\n \tif (writer.selected_nr >= writer.selected_alloc) {\n \t\twriter.selected_alloc = (writer.selected_alloc + 32) * 2;\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 995d664cc89..0f539d79cfd 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -99,6 +99,8 @@ int bitmap_has_oid_in_uninteresting(struct bitmap_index *, const struct object_i\n off_t get_disk_usage_from_bitmap(struct bitmap_index *, struct rev_info *);\n \n int bitmap_writer_has_bitmapped_object_id(const struct object_id *oid);\n+void bitmap_writer_push_bitmapped_commit(struct commit *commit,\n+\t\t\t\t\t unsigned pseudo_merge);\n \n void bitmap_writer_init(struct repository *r);\n void bitmap_writer_show_progress(int show);\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491053","messageId":"323e1250b247f44212879c37092e1c4cbdc0b310.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 10/24] pseudo-merge: implement support for selecting pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:29Z","receivedAt":"2024-03-20T22:05:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Teach the new pseudo-merge machinery how to select non-bitmapped commits\nfor inclusion in different pseudo-merge group(s) based on a handful of\ncriteria.\n\nPseudo-merges are derived first from named pseudo-merge groups (see the\n`bitmapPseudoMerge..*` configuration options). They are\n(optionally) further segmented within an individual pseudo-merge group\nbased on any capture group(s) within the pseudo-merge group's pattern.\n\nFor example, a configuration like so:\n\n [bitmapPseudoMerge \"all\"]\n pattern = \"refs/\"\n threshold = now\n stableThreshold = never\n sampleRate = 100\n maxMerges = 64\n\nwould group all non-bitmapped commits into up to 64 individual\npseudo-merge commits.\n\nIf you wanted to separate tags from branches when generating\npseudo-merge commits, and further segment them by which fork they\noriginate from (using the same \"refs/virtual/\" scheme as in the delta\nislands documentation), you would instead write something like:\n\n [bitmapPseudoMerge \"all\"]\n pattern = \"refs/virtual/([0-9]+)/(heads|tags)/\"\n threshold = now\n stableThreshold = never\n sampleRate = 100\n maxMerges = 64\n\nWhich would generate pseudo-merge group identifiers like \"1234-heads\",\nand \"5678-tags\" (for branches in fork \"1234\", and tags in remote \"5678\",\nrespectively).\n\nWithin pseudo-merge groups, there are a handful of other options used to\ncontrol the distribution of matching commits among individual\npseudo-merge commits:\n\n - bitmapPseudoMerge..decay\n - bitmapPseudoMerge..sampleRate\n - bitmapPseudoMerge..threshold\n - bitmapPseudoMerge..maxMerges\n - bitmapPseudoMerge..stableThreshold\n - bitmapPseudoMerge..stableSize\n\nThe decay parameter roughly corresponds to \"k\" in `f(n) = C*n^(-k/100)`,\nwhere `f(n)` describes the size of the `n`-th pseudo-merge group. The\nsample rate controls what percentage of eligible commits are considered\nas candidates. The threshold parameter indicates the minimum age (so as\nto avoid including too-recent commits in a pseudo-merge group, making it\nless likely to be valid). The \"maxMerges\" parameter sets an upper-bound\non the number of pseudo-merge commits an individual group\n\nThe latter two \"stable\"-related parameters control \"stable\" pseudo-merge\ngroups, comprised of a fixed number of commits which are older than the\nconfigured \"stable threshold\" value and may be grouped together in\nchunks of \"stableSize\" in order of age.\n\nThis patch implements the aforementioned selection routine, as well as\nparsing the relevant configuration options.\n\nSigned-off-by: Taylor Blau \n---\n pseudo-merge.c | 441 +++++++++++++++++++++++++++++++++++++++++++++++++\n pseudo-merge.h | 96 +++++++++++\n 2 files changed, 537 insertions(+)\n\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex 37e037ba272..caccef942a1 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -1,2 +1,443 @@\n #include \"git-compat-util.h\"\n #include \"pseudo-merge.h\"\n+#include \"date.h\"\n+#include \"oid-array.h\"\n+#include \"strbuf.h\"\n+#include \"config.h\"\n+#include \"string-list.h\"\n+#include \"refs.h\"\n+#include \"pack-bitmap.h\"\n+#include \"commit.h\"\n+#include \"alloc.h\"\n+#include \"progress.h\"\n+\n+#define DEFAULT_PSEUDO_MERGE_DECAY 1.0f\n+#define DEFAULT_PSEUDO_MERGE_MAX_MERGES 64\n+#define DEFAULT_PSEUDO_MERGE_SAMPLE_RATE 100\n+#define DEFAULT_PSEUDO_MERGE_THRESHOLD approxidate(\"1.week.ago\")\n+#define DEFAULT_PSEUDO_MERGE_STABLE_THRESHOLD approxidate(\"1.month.ago\")\n+#define DEFAULT_PSEUDO_MERGE_STABLE_SIZE 512\n+\n+static float gitexp(float base, int exp)\n+{\n+\tfloat result = 1;\n+\twhile (1) {\n+\t\tif (exp % 2)\n+\t\t\tresult *= base;\n+\t\texp >>= 1;\n+\t\tif (!exp)\n+\t\t\tbreak;\n+\t\tbase *= base;\n+\t}\n+\treturn result;\n+}\n+\n+static uint32_t pseudo_merge_group_size(const struct pseudo_merge_group *group,\n+\t\t\t\t\tconst struct pseudo_merge_matches *matches,\n+\t\t\t\t\tuint32_t i)\n+{\n+\tfloat C = 0.0f;\n+\tuint32_t n;\n+\n+\t/*\n+\t * The size of pseudo-merge groups decays according to a power series,\n+\t * which looks like:\n+\t *\n+\t * f(n) = C * n^-k\n+\t *\n+\t * , where 'n' is the n-th pseudo-merge group, 'f(n)' is its size, 'k'\n+\t * is the decay rate, and 'C' is a scaling value.\n+\t *\n+\t * The value of C depends on the number of groups, decay rate, and total\n+\t * number of commits. It is computed such that if there are M and N\n+\t * total groups and commits, respectively, that:\n+\t *\n+\t * N = f(0) + f(1) + ... f(M-1)\n+\t *\n+\t * Rearranging to isolate C, we get:\n+\t *\n+\t * N = \\sum_{n=1}^M C / n^k\n+\t *\n+\t * N / C = \\sum_{n=1}^M n^-k\n+\t *\n+\t * C = N / \\sum_{n=1}^M n^-k\n+\t *\n+\t * For example, if we have a decay rate of 'k' being equal to 1.5, 'N'\n+\t * total commits equal to 10,000, and 'M' being equal to 6 groups, then\n+\t * the (rounded) group sizes are:\n+\t *\n+\t * { 5469, 1934, 1053, 684, 489, 372 }\n+\t *\n+\t * increasing the number of total groups, say to 10, scales the group\n+\t * sizes appropriately:\n+\t *\n+\t * { 5012, 1772, 964, 626, 448, 341, 271, 221, 186, 158 }\n+\t */\n+\tfor (n = 0; n < group->max_merges; n++)\n+\t\tC += 1.0f / gitexp(n + 1, group->decay);\n+\tC = matches->unstable_nr / C;\n+\n+\treturn (int)((C / gitexp(i + 1, group->decay)) + 0.5);\n+}\n+\n+static void init_pseudo_merge_group(struct pseudo_merge_group *group)\n+{\n+\tmemset(group, 0, sizeof(struct pseudo_merge_group));\n+\n+\tstrmap_init_with_options(&group->matches, NULL, 0);\n+\n+\tgroup->decay = DEFAULT_PSEUDO_MERGE_DECAY;\n+\tgroup->max_merges = DEFAULT_PSEUDO_MERGE_MAX_MERGES;\n+\tgroup->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE;\n+\tgroup->threshold = DEFAULT_PSEUDO_MERGE_THRESHOLD;\n+\tgroup->stable_threshold = DEFAULT_PSEUDO_MERGE_STABLE_THRESHOLD;\n+\tgroup->stable_size = DEFAULT_PSEUDO_MERGE_STABLE_SIZE;\n+}\n+\n+static int pseudo_merge_config(const char *var, const char *value,\n+\t\t\t const struct config_context *ctx,\n+\t\t\t void *cb_data)\n+{\n+\tstruct string_list *list = cb_data;\n+\tstruct string_list_item *item;\n+\tstruct pseudo_merge_group *group;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *sub, *key;\n+\tsize_t sub_len;\n+\n+\tif (parse_config_key(var, \"bitmappseudomerge\", &sub, &sub_len, &key))\n+\t\treturn 0;\n+\n+\tif (!sub_len)\n+\t\treturn 0;\n+\n+\tstrbuf_add(&buf, sub, sub_len);\n+\n+\titem = string_list_lookup(list, buf.buf);\n+\tif (!item) {\n+\t\titem = string_list_insert(list, buf.buf);\n+\n+\t\titem->util = xmalloc(sizeof(struct pseudo_merge_group));\n+\t\tinit_pseudo_merge_group(item->util);\n+\t}\n+\n+\tgroup = item->util;\n+\n+\tif (!strcmp(key, \"pattern\")) {\n+\t\tstruct strbuf re = STRBUF_INIT;\n+\n+\t\tfree(group->pattern);\n+\t\tif (*value != '^')\n+\t\t\tstrbuf_addch(&re, '^');\n+\t\tstrbuf_addstr(&re, value);\n+\n+\t\tgroup->pattern = xcalloc(1, sizeof(regex_t));\n+\t\tif (regcomp(group->pattern, re.buf, REG_EXTENDED))\n+\t\t\tdie(_(\"failed to load pseudo-merge regex for %s: '%s'\"),\n+\t\t\t sub, re.buf);\n+\n+\t\tstrbuf_release(&re);\n+\t} else if (!strcmp(key, \"decay\")) {\n+\t\tgroup->decay = git_config_int(var, value, ctx->kvi);\n+\t\tif (group->decay < 0) {\n+\t\t\twarning(_(\"%s must be non-negative, using default\"), var);\n+\t\t\tgroup->decay = DEFAULT_PSEUDO_MERGE_DECAY;\n+\t\t}\n+\t} else if (!strcmp(key, \"samplerate\")) {\n+\t\tgroup->sample_rate = git_config_int(var, value, ctx->kvi);\n+\t\tif (!(0 <= group->sample_rate && group->sample_rate <= 100)) {\n+\t\t\twarning(_(\"%s must be between 0 and 100, using default\"), var);\n+\t\t\tgroup->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE;\n+\t\t}\n+\t} else if (!strcmp(key, \"threshold\")) {\n+\t\tif (git_config_expiry_date(&group->threshold, var, value)) {\n+\t\t\tstrbuf_release(&buf);\n+\t\t\treturn -1;\n+\t\t}\n+\t} else if (!strcmp(key, \"maxmerges\")) {\n+\t\tgroup->max_merges = git_config_int(var, value, ctx->kvi);\n+\t\tif (group->max_merges < 0) {\n+\t\t\twarning(_(\"%s must be non-negative, using default\"), var);\n+\t\t\tgroup->max_merges = DEFAULT_PSEUDO_MERGE_MAX_MERGES;\n+\t\t}\n+\t} else if (!strcmp(key, \"stablethreshold\")) {\n+\t\tif (git_config_expiry_date(&group->stable_threshold, var, value)) {\n+\t\t\tstrbuf_release(&buf);\n+\t\t\treturn -1;\n+\t\t}\n+\t} else if (!strcmp(key, \"stablesize\")) {\n+\t\tgroup->stable_size = git_config_int(var, value, ctx->kvi);\n+\t\tif (group->stable_size <= 0) {\n+\t\t\twarning(_(\"%s must be positive, using default\"), var);\n+\t\t\tgroup->stable_size = DEFAULT_PSEUDO_MERGE_STABLE_SIZE;\n+\t\t}\n+\t}\n+\n+\tstrbuf_release(&buf);\n+\n+\treturn 0;\n+}\n+\n+void load_pseudo_merges_from_config(struct string_list *list)\n+{\n+\tstruct string_list_item *item;\n+\n+\tgit_config(pseudo_merge_config, list);\n+\n+\tfor_each_string_list_item(item, list) {\n+\t\tstruct pseudo_merge_group *group = item->util;\n+\t\tif (!group->pattern)\n+\t\t\tdie(_(\"pseudo-merge group '%s' missing required pattern\"),\n+\t\t\t item->string);\n+\t\tif (group->threshold < group->stable_threshold)\n+\t\t\tdie(_(\"pseudo-merge group '%s' has unstable threshold \"\n+\t\t\t \"before stable one\"), item->string);\n+\t}\n+}\n+\n+static int find_pseudo_merge_group_for_ref(const char *refname,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t int flags UNUSED,\n+\t\t\t\t\t void *_data)\n+{\n+\tstruct string_list *list = _data;\n+\tstruct object_id peeled;\n+\tstruct commit *c;\n+\tuint32_t i;\n+\tint has_bitmap;\n+\n+\tif (!peel_iterated_oid(oid, &peeled))\n+\t\toid = &peeled;\n+\n+\tc = lookup_commit(the_repository, oid);\n+\tif (!c)\n+\t\treturn 0;\n+\n+\thas_bitmap = bitmap_writer_has_bitmapped_object_id(oid);\n+\n+\tfor (i = 0; i < list->nr; i++) {\n+\t\tstruct pseudo_merge_group *group;\n+\t\tstruct pseudo_merge_matches *matches;\n+\t\tstruct strbuf group_name = STRBUF_INIT;\n+\t\tregmatch_t captures[16];\n+\t\tsize_t j;\n+\n+\t\tgroup = list->items[i].util;\n+\t\tif (regexec(group->pattern, refname, ARRAY_SIZE(captures),\n+\t\t\t captures, 0))\n+\t\t\tcontinue;\n+\n+\t\tif (captures[ARRAY_SIZE(captures) - 1].rm_so != -1)\n+\t\t\twarning(_(\"pseudo-merge regex from config has too many capture \"\n+\t\t\t\t \"groups (max=%\"PRIuMAX\")\"),\n+\t\t\t\t(uintmax_t)ARRAY_SIZE(captures) - 2);\n+\n+\t\tfor (j = !!group->pattern->re_nsub; j < ARRAY_SIZE(captures); j++) {\n+\t\t\tregmatch_t *match = &captures[j];\n+\t\t\tif (match->rm_so == -1)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (group_name.len)\n+\t\t\t\tstrbuf_addch(&group_name, '-');\n+\n+\t\t\tstrbuf_add(&group_name, refname + match->rm_so,\n+\t\t\t\t match->rm_eo - match->rm_so);\n+\t\t}\n+\n+\t\tmatches = strmap_get(&group->matches, group_name.buf);\n+\t\tif (!matches) {\n+\t\t\tmatches = xcalloc(1, sizeof(*matches));\n+\t\t\tstrmap_put(&group->matches, strbuf_detach(&group_name, NULL),\n+\t\t\t\t matches);\n+\t\t}\n+\n+\t\tif (c->date <= group->stable_threshold) {\n+\t\t\tALLOC_GROW(matches->stable, matches->stable_nr + 1,\n+\t\t\t\t matches->stable_alloc);\n+\t\t\tmatches->stable[matches->stable_nr++] = c;\n+\t\t} else if (c->date <= group->threshold && !has_bitmap) {\n+\t\t\tALLOC_GROW(matches->unstable, matches->unstable_nr + 1,\n+\t\t\t\t matches->unstable_alloc);\n+\t\t\tmatches->unstable[matches->unstable_nr++] = c;\n+\t\t}\n+\n+\t\tstrbuf_release(&group_name);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static struct commit *push_pseudo_merge(struct pseudo_merge_group *group)\n+{\n+\tstruct commit *merge;\n+\n+\tALLOC_GROW(group->merges, group->merges_nr + 1, group->merges_alloc);\n+\n+\tmerge = alloc_commit_node(the_repository);\n+\tmerge->object.parsed = 1;\n+\tmerge->object.flags |= BITMAP_PSEUDO_MERGE;\n+\n+\tgroup->merges[group->merges_nr++] = merge;\n+\n+\treturn merge;\n+}\n+\n+static struct pseudo_merge_commit_idx *pseudo_merge_idx(kh_oid_map_t *pseudo_merge_commits,\n+\t\t\t\t\t\t\tconst struct object_id *oid)\n+\n+{\n+\tstruct pseudo_merge_commit_idx *pmc;\n+\tkhiter_t hash_pos;\n+\n+\thash_pos = kh_get_oid_map(pseudo_merge_commits, *oid);\n+\tif (hash_pos == kh_end(pseudo_merge_commits)) {\n+\t\tint hash_ret;\n+\t\thash_pos = kh_put_oid_map(pseudo_merge_commits, *oid, &hash_ret);\n+\n+\t\tCALLOC_ARRAY(pmc, 1);\n+\n+\t\tkh_value(pseudo_merge_commits, hash_pos) = pmc;\n+\t} else {\n+\t\tpmc = kh_value(pseudo_merge_commits, hash_pos);\n+\t}\n+\n+\treturn pmc;\n+}\n+\n+#define MIN_PSEUDO_MERGE_SIZE 8\n+\n+static void select_pseudo_merges_1(struct pseudo_merge_group *group,\n+\t\t\t\t struct pseudo_merge_matches *matches,\n+\t\t\t\t kh_oid_map_t *pseudo_merge_commits,\n+\t\t\t\t uint32_t *pseudo_merges_nr)\n+{\n+\tuint32_t i, j;\n+\tuint32_t stable_merges_nr;\n+\n+\tif (!matches->stable_nr && !matches->unstable_nr)\n+\t\treturn; /* all tips in this group already have bitmaps */\n+\n+\tstable_merges_nr = matches->stable_nr / group->stable_size;\n+\tif (matches->stable_nr % group->stable_size)\n+\t\tstable_merges_nr++;\n+\n+\t/* make stable_merges_nr pseudo merges for stable commits */\n+\tfor (i = 0, j = 0; i < stable_merges_nr; i++) {\n+\t\tstruct commit *merge;\n+\t\tstruct commit_list **p;\n+\n+\t\tmerge = push_pseudo_merge(group);\n+\t\tp = &merge->parents;\n+\n+\t\tdo {\n+\t\t\tstruct commit *c;\n+\t\t\tstruct pseudo_merge_commit_idx *pmc;\n+\n+\t\t\tif (j >= matches->stable_nr)\n+\t\t\t\tbreak;\n+\n+\t\t\tc = matches->stable[j++];\n+\t\t\tpmc = pseudo_merge_idx(pseudo_merge_commits,\n+\t\t\t\t\t &c->object.oid);\n+\n+\t\t\tALLOC_GROW(pmc->pseudo_merge, pmc->nr + 1, pmc->alloc);\n+\n+\t\t\tpmc->pseudo_merge[pmc->nr++] = *pseudo_merges_nr;\n+\t\t\tp = commit_list_append(c, p);\n+\t\t} while (j % group->stable_size);\n+\n+\t\tbitmap_writer_push_bitmapped_commit(merge, 1);\n+\t\t(*pseudo_merges_nr)++;\n+\t}\n+\n+\t/* make up to group->max_merges pseudo merges for unstable commits */\n+\tfor (i = 0, j = 0; i < group->max_merges; i++) {\n+\t\tstruct commit *merge;\n+\t\tstruct commit_list **p;\n+\t\tuint32_t size, end;\n+\n+\t\tmerge = push_pseudo_merge(group);\n+\t\tp = &merge->parents;\n+\n+\t\tsize = pseudo_merge_group_size(group, matches, i);\n+\t\tend = size < MIN_PSEUDO_MERGE_SIZE ? matches->unstable_nr : j + size;\n+\n+\t\tfor (; j < end && j < matches->unstable_nr; j++) {\n+\t\t\tstruct commit *c = matches->unstable[j];\n+\t\t\tstruct pseudo_merge_commit_idx *pmc;\n+\n+\t\t\tif (j % (100 / group->sample_rate))\n+\t\t\t\tcontinue;\n+\n+\t\t\tpmc = pseudo_merge_idx(pseudo_merge_commits,\n+\t\t\t\t\t &c->object.oid);\n+\n+\t\t\tALLOC_GROW(pmc->pseudo_merge, pmc->nr + 1, pmc->alloc);\n+\n+\t\t\tpmc->pseudo_merge[pmc->nr++] = *pseudo_merges_nr;\n+\t\t\tp = commit_list_append(c, p);\n+\t\t}\n+\n+\t\tbitmap_writer_push_bitmapped_commit(merge, 1);\n+\t\t(*pseudo_merges_nr)++;\n+\t\tif (end >= matches->unstable_nr)\n+\t\t\tbreak;\n+\t}\n+}\n+\n+static int commit_date_cmp(const void *va, const void *vb)\n+{\n+\ttimestamp_t a = (*(const struct commit **)va)->date;\n+\ttimestamp_t b = (*(const struct commit **)vb)->date;\n+\n+\tif (a < b)\n+\t\treturn -1;\n+\telse if (a > b)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static void sort_pseudo_merge_matches(struct pseudo_merge_matches *matches)\n+{\n+\tQSORT(matches->stable, matches->stable_nr, commit_date_cmp);\n+\tQSORT(matches->unstable, matches->unstable_nr, commit_date_cmp);\n+}\n+\n+void select_pseudo_merges(struct string_list *list,\n+\t\t\t struct commit **commits, size_t commits_nr,\n+\t\t\t kh_oid_map_t *pseudo_merge_commits,\n+\t\t\t uint32_t *pseudo_merges_nr,\n+\t\t\t unsigned show_progress)\n+{\n+\tstruct progress *progress = NULL;\n+\tuint32_t i;\n+\n+\tif (!list->nr)\n+\t\treturn;\n+\n+\tif (show_progress)\n+\t\tprogress = start_progress(\"Selecting pseudo-merge commits\", list->nr);\n+\n+\tfor_each_ref(find_pseudo_merge_group_for_ref, list);\n+\n+\tfor (i = 0; i < list->nr; i++) {\n+\t\tstruct pseudo_merge_group *group;\n+\t\tstruct hashmap_iter iter;\n+\t\tstruct strmap_entry *e;\n+\n+\t\tgroup = list->items[i].util;\n+\t\tstrmap_for_each_entry(&group->matches, &iter, e) {\n+\t\t\tstruct pseudo_merge_matches *matches = e->value;\n+\n+\t\t\tsort_pseudo_merge_matches(matches);\n+\n+\t\t\tselect_pseudo_merges_1(group, matches,\n+\t\t\t\t\t pseudo_merge_commits,\n+\t\t\t\t\t pseudo_merges_nr);\n+\t\t}\n+\n+\t\tdisplay_progress(progress, i + 1);\n+\t}\n+\n+\tstop_progress(&progress);\n+}\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nindex cab8ff6960a..81888731864 100644\n--- a/pseudo-merge.h\n+++ b/pseudo-merge.h\n@@ -2,5 +2,101 @@\n #define PSEUDO_MERGE_H\n \n #include \"git-compat-util.h\"\n+#include \"strmap.h\"\n+#include \"khash.h\"\n+#include \"ewah/ewok.h\"\n+\n+struct commit;\n+struct string_list;\n+struct bitmap_index;\n+\n+/*\n+ * A pseudo-merge group tracks the set of non-bitmapped reference tips\n+ * that match the given pattern.\n+ *\n+ * Within those matches, they are further segmented by separating\n+ * consecutive capture groups with '-' dash character capture groups\n+ * with '-' dash characters.\n+ *\n+ * Those groups are then ordered by committer date and partitioned\n+ * into individual pseudo-merge(s) according to the decay, max_merges,\n+ * sample_rate, and threshold parameters.\n+ */\n+struct pseudo_merge_group {\n+\tregex_t *pattern;\n+\n+\t/* capture group(s) -> struct pseudo_merge_matches */\n+\tstruct strmap matches;\n+\n+\t/*\n+\t * The individual pseudo-merge(s) that are generated from the\n+\t * above array of matches, partitioned according to the below\n+\t * parameters.\n+\t */\n+\tstruct commit **merges;\n+\tsize_t merges_nr;\n+\tsize_t merges_alloc;\n+\n+\t/*\n+\t * Pseudo-merge grouping parameters. See git-config(1) for\n+\t * more information.\n+\t */\n+\tfloat decay;\n+\tint max_merges;\n+\tint sample_rate;\n+\tint stable_size;\n+\ttimestamp_t threshold;\n+\ttimestamp_t stable_threshold;\n+};\n+\n+struct pseudo_merge_matches {\n+\tstruct commit **stable;\n+\tstruct commit **unstable;\n+\tsize_t stable_nr, stable_alloc;\n+\tsize_t unstable_nr, unstable_alloc;\n+};\n+\n+/*\n+ * Read the repository's configuration:\n+ *\n+ * - bitmapPseudoMerge..pattern\n+ * - bitmapPseudoMerge..decay\n+ * - bitmapPseudoMerge..sampleRate\n+ * - bitmapPseudoMerge..threshold\n+ * - bitmapPseudoMerge..maxMerges\n+ * - bitmapPseudoMerge..stableThreshold\n+ * - bitmapPseudoMerge..stableSize\n+ *\n+ * and populates the given `list` with pseudo-merge groups. String\n+ * entry keys are the pseudo-merge group names, and the values are\n+ * pointers to the pseudo_merge_group structure itself.\n+ */\n+void load_pseudo_merges_from_config(struct string_list *list);\n+\n+/*\n+ * A pseudo-merge commit index (pseudo_merge_commit_idx) maps a\n+ * particular (non-pseudo-merge) commit to the list of pseudo-merge(s)\n+ * it appears in.\n+ */\n+struct pseudo_merge_commit_idx {\n+\tuint32_t *pseudo_merge;\n+\tsize_t nr, alloc;\n+};\n+\n+/*\n+ * Selects pseudo-merges from a list of commits, populating the given\n+ * string_list of pseudo-merge groups.\n+ *\n+ * Populates the pseudo_merge_commits map with a commit_idx\n+ * corresponding to each commit in the list. Counts the total number\n+ * of pseudo-merges generated.\n+ *\n+ * Optionally shows a progress meter.\n+ */\n+void select_pseudo_merges(struct string_list *list,\n+\t\t\t struct commit **commits, size_t commits_nr,\n+\t\t\t kh_oid_map_t *pseudo_merge_commits,\n+\t\t\t uint32_t *pseudo_merges_nr,\n+\t\t\t unsigned show_progress);\n \n #endif\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491054","messageId":"bf6b0d8601e98061a84c5eb90128c2d82053988a.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 11/24] pack-bitmap-write.c: select pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:32Z","receivedAt":"2024-03-20T22:05:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the pseudo-merge machinery has learned how to select\nnon-bitmapped commits and assign them into different pseudo-merge\ngroup(s), invoke this new API from within the pack-bitmap internals and\nstore the results off.\n\nNote that the selected pseudo-merge commits aren't actually used or\nwritten anywhere yet. This will be done in the following commit.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config.txt | 2 +\n Documentation/config/bitmap-pseudo-merge.txt | 75 ++++++++++++++++++++\n Documentation/technical/bitmap-format.txt | 26 +++++++\n pack-bitmap-write.c | 14 ++++\n 4 files changed, 117 insertions(+)\n create mode 100644 Documentation/config/bitmap-pseudo-merge.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 782c2bab906..e5a7170c9e0 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -381,6 +381,8 @@ include::config/apply.txt[]\n \n include::config/attr.txt[]\n \n+include::config/bitmap-pseudo-merge.txt[]\n+\n include::config/blame.txt[]\n \n include::config/branch.txt[]\ndiff --git a/Documentation/config/bitmap-pseudo-merge.txt b/Documentation/config/bitmap-pseudo-merge.txt\nnew file mode 100644\nindex 00000000000..90b72522046\n--- /dev/null\n+++ b/Documentation/config/bitmap-pseudo-merge.txt\n@@ -0,0 +1,75 @@\n+bitmapPseudoMerge..pattern::\n+\tRegular expression used to match reference names. Commits\n+\tpointed to by references matching this pattern (and meeting\n+\tthe below criteria, like `bitmapPseudoMerge..sampleRate`\n+\tand `bitmapPseudoMerge..threshold`) will be considered\n+\tfor inclusion in a pseudo-merge bitmap.\n++\n+Commits are grouped into pseudo-merge groups based on whether or not\n+any reference(s) that point at a given commit match the pattern, which\n+is an extended regular expression.\n++\n+Within a pseudo-merge group, commits may be further grouped into\n+sub-groups based on the capture groups in the pattern. These\n+sub-groupings are formed from the regular expressions by concatenating\n+any capture groups from the regular expression, with a '-' dash in\n+between.\n++\n+For example, if the pattern is `refs/tags/`, then all tags (provided\n+they meet the below criteria) will be considered candidates for the\n+same pseudo-merge group. However, if the pattern is instead\n+`refs/remotes/([0-9])+/tags/`, then tags from different remotes will\n+be grouped into separate pseudo-merge groups, based on the remote\n+number.\n+\n+bitmapPseudoMerge..decay::\n+\tDetermines the rate at which consecutive pseudo-merge bitmap\n+\tgroups decrease in size. Must be non-negative. This parameter\n+\tcan be thought of as `k` in the function `f(n) = C *\n+\tn^(-k/100)`, where `f(n)` is the size of the `n`th group.\n++\n+Setting the decay rate equal to `0` will cause all groups to be the\n+same size. Setting the decay rate equal to `100` will cause the `n`th\n+group to be `1/n` the size of the initial group. Higher values of the\n+decay rate cause consecutive groups to shrink at an increasing rate.\n+The default is `100`.\n+\n+bitmapPseudoMerge..sampleRate::\n+\tDetermines the proportion of non-bitmapped commits (among\n+\treference tips) which are selected for inclusion in an\n+\tunstable pseudo-merge bitmap. Must be between `0` and `100`\n+\t(inclusive). The default is `100`.\n+\n+bitmapPseudoMerge..threshold::\n+\tDetermines the minimum age of non-bitmapped commits (among\n+\treference tips, as above) which are candidates for inclusion\n+\tin an unstable pseudo-merge bitmap. The default is\n+\t`1.week.ago`.\n+\n+bitmapPseudoMerge..maxMerges::\n+\tDetermines the maximum number of pseudo-merge commits among\n+\twhich commits may be distributed.\n++\n+For pseudo-merge groups whose pattern does not contain any capture\n+groups, this setting is applied for all commits matching the regular\n+expression. For patterns that have one or more capture groups, this\n+setting is applied for each distinct capture group.\n++\n+For example, if your capture group is `refs/tags/`, then this setting\n+will distribute all tags into a maximum of `maxMerges` pseudo-merge\n+commits. However, if your capture group is, say,\n+`refs/remotes/([0-9]+)/tags/`, then this setting will be applied to\n+each remote's set of tags individually.\n++\n+Must be non-negative. The default value is 64.\n+\n+bitmapPseudoMerge..stableThreshold::\n+\tDetermines the minimum age of commits (among reference tips,\n+\tas above, however stable commits are still considered\n+\tcandidates even when they have been covered by a bitmap) which\n+\tare candidates for a stable a pseudo-merge bitmap. The default\n+\tis `1.month.ago`.\n+\n+bitmapPseudoMerge..stableSize::\n+\tDetermines the size (in number of commits) of a stable\n+\tpsuedo-merge bitmap. The default is `512`.\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex 63a7177ac08..ed7edf98034 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -434,3 +434,29 @@ the end of a `.bitmap` file. The format is as follows:\n \n * An 8-byte unsigned value (in network byte-order) equal to the number\n of bytes in the pseudo-merge section (including this field).\n+\n+=== Pseudo-merge selection\n+\n+Pseudo-merge commits are selected among non-bitmapped commits at the\n+tip of one or more reference(s). In addition, there are a handful of\n+constraints to further refine this selection:\n+\n+`pack.bitmapPseudoMergeDecay`:: Defines the \"decay rate\", which\n+corresponds to how quickly (or not) consecutive pseudo-merges decrease\n+in size relative to one another.\n+\n+`pack.bitmapPseudoMergeGroups`:: Defines the maximum number of\n+pseudo-merge groups.\n+\n+`pack.bitmapPseudoMergeSampleRate`:: Defines the percentage of commits\n+(matching the above criteria) which are selected.\n+\n+`pack.bitmapPseudoMergeThreshold`:: Defines the minimum age of a commit\n+in order to be considered for inclusion within one or more pseudo-merge\n+bitmaps.\n+\n+The size of consecutive pseudo-merge groups decays according to a\n+power-law decay function, where the size of the `n`-th group is `f(n) =\n+C*n^-k`. The value of `C` is chosen accordingly to match the number of\n+desired groups, and `k` is 1/100th of the value of\n+`pack.bitmapPseudoMergeDecay`.\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex e46978d494c..db1c38f4e46 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -17,6 +17,7 @@\n #include \"trace2.h\"\n #include \"tree.h\"\n #include \"tree-walk.h\"\n+#include \"pseudo-merge.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -39,6 +40,8 @@ struct bitmap_writer {\n \tstruct bitmapped_commit *selected;\n \tunsigned int selected_nr, selected_alloc;\n \n+\tstruct string_list pseudo_merge_groups;\n+\tkh_oid_map_t *pseudo_merge_commits; /* oid -> pseudo merge(s) */\n \tuint32_t pseudo_merges_nr;\n \n \tstruct progress *progress;\n@@ -56,6 +59,11 @@ static inline int bitmap_writer_selected_nr(void)\n void bitmap_writer_init(struct repository *r)\n {\n \twriter.bitmaps = kh_init_oid_map();\n+\twriter.pseudo_merge_commits = kh_init_oid_map();\n+\n+\tstring_list_init_dup(&writer.pseudo_merge_groups);\n+\n+\tload_pseudo_merges_from_config(&writer.pseudo_merge_groups);\n }\n \n void bitmap_writer_show_progress(int show)\n@@ -686,6 +694,12 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t}\n \n \tstop_progress(&writer.progress);\n+\n+\tselect_pseudo_merges(&writer.pseudo_merge_groups,\n+\t\t\t indexed_commits, indexed_commits_nr,\n+\t\t\t writer.pseudo_merge_commits,\n+\t\t\t &writer.pseudo_merges_nr,\n+\t\t\t writer.show_progress);\n }\n \n \n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491055","messageId":"4c594f3faa875a6f54a801daf4250e2f8750a87c.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 12/24] pack-bitmap-write.c: write pseudo-merge table","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:35Z","receivedAt":"2024-03-20T22:05:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the pack-bitmap writer machinery understands how to select and\nstore pseudo-merge commits, teach it how to write the new optional\npseudo-merge .bitmap extension.\n\nNo readers yet exist for this new extension to the .bitmap format. The\nfollowing commits will take any preparatory step(s) necessary before\nthen implementing the routines necessary to read this new table.\n\nIn the meantime, the new `write_pseudo_merges()` function implements\nwriting this new format as described by a previous commit in\nDocumentation/technical/bitmap-format.txt.\n\nWriting this table is fairly straightforward and consists of a few\nsub-components:\n\n - a pair of bitmaps for each pseudo-merge (one for the pseudo-merge\n \"parents\", and another for the objects reachable from those parents)\n\n - for each commit, the offset of either (a) the pseudo-merge it\n belongs to, or (b) an extended lookup table if it belongs to >1\n pseudo-merge groups\n\n - if there are any commits belonging to >1 pseudo-merge group, the\n extended lookup tables (which each consist of the number of\n pseudo-merge groups a commit appears in, and then that many 4-byte\n unsigned )\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 128 ++++++++++++++++++++++++++++++++++++++++++++\n pack-bitmap.h | 1 +\n 2 files changed, 129 insertions(+)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex db1c38f4e46..2d1b202fcd9 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -18,6 +18,7 @@\n #include \"tree.h\"\n #include \"tree-walk.h\"\n #include \"pseudo-merge.h\"\n+#include \"oid-array.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -748,6 +749,127 @@ static void write_selected_commits_v1(struct hashfile *f,\n \t}\n }\n \n+static void write_pseudo_merges(struct hashfile *f)\n+{\n+\tstruct oid_array commits = OID_ARRAY_INIT;\n+\tstruct bitmap **commits_bitmap = NULL;\n+\toff_t *pseudo_merge_ofs = NULL;\n+\toff_t start, table_start, next_ext;\n+\n+\tuint32_t base = bitmap_writer_selected_nr();\n+\tsize_t i, j = 0;\n+\n+\tCALLOC_ARRAY(commits_bitmap, writer.pseudo_merges_nr);\n+\tCALLOC_ARRAY(pseudo_merge_ofs, writer.pseudo_merges_nr);\n+\n+\tfor (i = 0; i < writer.pseudo_merges_nr; i++) {\n+\t\tstruct bitmapped_commit *merge = &writer.selected[base + i];\n+\t\tstruct commit_list *p;\n+\n+\t\tif (!merge->pseudo_merge)\n+\t\t\tBUG(\"found non-pseudo merge commit at %\"PRIuMAX, (uintmax_t)i);\n+\n+\t\tcommits_bitmap[i] = bitmap_new();\n+\n+\t\tfor (p = merge->commit->parents; p; p = p->next)\n+\t\t\tbitmap_set(commits_bitmap[i],\n+\t\t\t\t find_object_pos(&p->item->object.oid, NULL));\n+\t}\n+\n+\tstart = hashfile_total(f);\n+\n+\tfor (i = 0; i < writer.pseudo_merges_nr; i++) {\n+\t\tstruct ewah_bitmap *commits_ewah = bitmap_to_ewah(commits_bitmap[i]);\n+\n+\t\tpseudo_merge_ofs[i] = hashfile_total(f);\n+\n+\t\tdump_bitmap(f, commits_ewah);\n+\t\tdump_bitmap(f, writer.selected[base+i].write_as);\n+\n+\t\tewah_free(commits_ewah);\n+\t}\n+\n+\tnext_ext = st_add(hashfile_total(f),\n+\t\t\t st_mult(kh_size(writer.pseudo_merge_commits),\n+\t\t\t\t sizeof(uint64_t)));\n+\n+\ttable_start = hashfile_total(f);\n+\n+\tcommits.alloc = kh_size(writer.pseudo_merge_commits);\n+\tCALLOC_ARRAY(commits.oid, commits.alloc);\n+\n+\tfor (i = kh_begin(writer.pseudo_merge_commits); i != kh_end(writer.pseudo_merge_commits); i++) {\n+\t\tif (!kh_exist(writer.pseudo_merge_commits, i))\n+\t\t\tcontinue;\n+\t\toid_array_append(&commits, &kh_key(writer.pseudo_merge_commits, i));\n+\t}\n+\n+\toid_array_sort(&commits);\n+\n+\t/* write lookup table (non-extended) */\n+\tfor (i = 0; i < commits.nr; i++) {\n+\t\tint hash_pos;\n+\t\tstruct pseudo_merge_commit_idx *c;\n+\n+\t\thash_pos = kh_get_oid_map(writer.pseudo_merge_commits,\n+\t\t\t\t\t commits.oid[i]);\n+\t\tif (hash_pos == kh_end(writer.pseudo_merge_commits))\n+\t\t\tBUG(\"could not find pseudo-merge commit %s\",\n+\t\t\t oid_to_hex(&commits.oid[i]));\n+\n+\t\tc = kh_value(writer.pseudo_merge_commits, hash_pos);\n+\n+\t\thashwrite_be32(f, find_object_pos(&commits.oid[i], NULL));\n+\t\tif (c->nr == 1)\n+\t\t\thashwrite_be64(f, pseudo_merge_ofs[c->pseudo_merge[0]]);\n+\t\telse if (c->nr > 1) {\n+\t\t\tif (next_ext & ((uint64_t)1<<63))\n+\t\t\t\tdie(_(\"too many pseudo-merges\"));\n+\t\t\thashwrite_be64(f, next_ext | ((uint64_t)1<<63));\n+\t\t\tnext_ext = st_add3(next_ext,\n+\t\t\t\t\t sizeof(uint32_t),\n+\t\t\t\t\t st_mult(c->nr, sizeof(uint64_t)));\n+\t\t} else\n+\t\t\tBUG(\"expected commit '%s' to have at least one \"\n+\t\t\t \"pseudo-merge\", oid_to_hex(&commits.oid[i]));\n+\t}\n+\n+\t/* write lookup table (extended) */\n+\tfor (i = 0; i < commits.nr; i++) {\n+\t\tint hash_pos;\n+\t\tstruct pseudo_merge_commit_idx *c;\n+\n+\t\thash_pos = kh_get_oid_map(writer.pseudo_merge_commits,\n+\t\t\t\t\t commits.oid[i]);\n+\t\tif (hash_pos == kh_end(writer.pseudo_merge_commits))\n+\t\t\tBUG(\"could not find pseudo-merge commit %s\",\n+\t\t\t oid_to_hex(&commits.oid[i]));\n+\n+\t\tc = kh_value(writer.pseudo_merge_commits, hash_pos);\n+\t\tif (c->nr == 1)\n+\t\t\tcontinue;\n+\n+\t\thashwrite_be32(f, c->nr);\n+\t\tfor (j = 0; j < c->nr; j++)\n+\t\t\thashwrite_be64(f, pseudo_merge_ofs[c->pseudo_merge[j]]);\n+\t}\n+\n+\t/* write positions for all pseudo merges */\n+\tfor (i = 0; i < writer.pseudo_merges_nr; i++)\n+\t\thashwrite_be64(f, pseudo_merge_ofs[i]);\n+\n+\thashwrite_be32(f, writer.pseudo_merges_nr);\n+\thashwrite_be32(f, kh_size(writer.pseudo_merge_commits));\n+\thashwrite_be64(f, table_start - start);\n+\thashwrite_be64(f, hashfile_total(f) - start + sizeof(uint64_t));\n+\n+\tfor (i = 0; i < writer.pseudo_merges_nr; i++)\n+\t\tbitmap_free(commits_bitmap[i]);\n+\n+\tfree(pseudo_merge_ofs);\n+\tfree(commits_bitmap);\n+}\n+\n static int table_cmp(const void *_va, const void *_vb, void *_data)\n {\n \tuint32_t *commit_positions = _data;\n@@ -855,6 +977,9 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \n \tint fd = odb_mkstemp(&tmp_file, \"pack/tmp_bitmap_XXXXXX\");\n \n+\tif (writer.pseudo_merges_nr)\n+\t\toptions |= BITMAP_OPT_PSEUDO_MERGES;\n+\n \tf = hashfd(fd, tmp_file.buf);\n \n \tmemcpy(header.magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE));\n@@ -886,6 +1011,9 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \n \twrite_selected_commits_v1(f, commit_positions, offsets);\n \n+\tif (options & BITMAP_OPT_PSEUDO_MERGES)\n+\t\twrite_pseudo_merges(f);\n+\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n \t\twrite_lookup_table(f, commit_positions, offsets);\n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 0f539d79cfd..55527f61cd9 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -37,6 +37,7 @@ enum pack_bitmap_opts {\n \tBITMAP_OPT_FULL_DAG = 0x1,\n \tBITMAP_OPT_HASH_CACHE = 0x4,\n \tBITMAP_OPT_LOOKUP_TABLE = 0x10,\n+\tBITMAP_OPT_PSEUDO_MERGES = 0x20,\n };\n \n enum pack_bitmap_flags {\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491056","messageId":"7a31a932ab327681e7d91454e5dee3f903f279d9.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 13/24] pack-bitmap: extract `read_bitmap()` function","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:38Z","receivedAt":"2024-03-20T22:05:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pack-bitmap machinery uses the `read_bitmap_1()` function to read a\nbitmap from within the mmap'd region corresponding to the .bitmap file.\nAs as side-effect of calling this function, `read_bitmap_1()` increments\nthe `index->map_pos` variable to reflect the number of bytes read.\n\nExtract the core of this routine to a separate function (that operates\nover a `const unsigned char *`, a `size_t` and a `size_t *` pointer)\ninstead of a `struct bitmap_index *` pointer.\n\nThis function (called `read_bitmap()`) is part of the pack-bitmap.h API\nso that it can be used within the upcoming portion of the implementation\nin pseduo-merge.ch.\n\nRewrite the existing function, `read_bitmap_1()`, in terms of its more\ngeneric counterpart.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 24 +++++++++++++++---------\n pack-bitmap.h | 2 ++\n 2 files changed, 17 insertions(+), 9 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 2baeabacee1..b3b6f9aad21 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -129,17 +129,13 @@ static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st)\n \treturn composed;\n }\n \n-/*\n- * Read a bitmap from the current read position on the mmaped\n- * index, and increase the read position accordingly\n- */\n-static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n+struct ewah_bitmap *read_bitmap(const unsigned char *map,\n+\t\t\t\tsize_t map_size, size_t *map_pos)\n {\n \tstruct ewah_bitmap *b = ewah_pool_new();\n \n-\tssize_t bitmap_size = ewah_read_mmap(b,\n-\t\tindex->map + index->map_pos,\n-\t\tindex->map_size - index->map_pos);\n+\tssize_t bitmap_size = ewah_read_mmap(b, map + *map_pos,\n+\t\t\t\t\t map_size - *map_pos);\n \n \tif (bitmap_size < 0) {\n \t\terror(_(\"failed to load bitmap index (corrupted?)\"));\n@@ -147,10 +143,20 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \t\treturn NULL;\n \t}\n \n-\tindex->map_pos += bitmap_size;\n+\t*map_pos += bitmap_size;\n+\n \treturn b;\n }\n \n+/*\n+ * Read a bitmap from the current read position on the mmaped\n+ * index, and increase the read position accordingly\n+ */\n+static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n+{\n+\treturn read_bitmap(index->map, index->map_size, &index->map_pos);\n+}\n+\n static uint32_t bitmap_num_objects(struct bitmap_index *index)\n {\n \tif (index->midx)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 55527f61cd9..a5fe4f305ef 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -133,4 +133,6 @@ int bitmap_is_preferred_refname(struct repository *r, const char *refname);\n \n int verify_bitmap_files(struct repository *r);\n \n+struct ewah_bitmap *read_bitmap(const unsigned char *map,\n+\t\t\t\tsize_t map_size, size_t *map_pos);\n #endif\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491057","messageId":"7e4d051f37a42e9e44b13acfc60b42fbb2a891b5.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 14/24] pseudo-merge: scaffolding for reads","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:41Z","receivedAt":"2024-03-20T22:05:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement scaffolding within the new pseudo-merge compilation unit\nnecessary to use the pseudo-merge API from within the pack-bitmap.c\nmachinery.\n\nThe core of this scaffolding is two-fold:\n\n - The `pseudo_merge` structure itself, which represents an individual\n pseudo-merge bitmap. It has fields for both bitmaps, as well as\n metadata about its position within the memory-mapped region, and\n a few extra bits indicating whether or not it is satisfied, and\n which bitmaps(s, if any) have been read, since they are initialized\n lazily.\n\n - The `pseudo_merge_map` structure, which holds an array of\n pseudo_merges, as well as a pointer to the memory-mapped region\n containing the pseudo-merge serialization from within a .bitmap\n file.\n\nNote that the `bitmap_index` structure is defined statically within the\npack-bitmap.o compilation unit, so we can't take in a `struct\nbitmap_index *`. Instead, wrap the primary components necessary to read\nthe pseudo-merges in this new structure to avoid exposing the\nimplementation details of the `bitmap_index` structure.\n\nSigned-off-by: Taylor Blau \n---\n pseudo-merge.c | 10 ++++++++\n pseudo-merge.h | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 75 insertions(+)\n\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex caccef942a1..d18de0a266b 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -441,3 +441,13 @@ void select_pseudo_merges(struct string_list *list,\n \n \tstop_progress(&progress);\n }\n+\n+void free_pseudo_merge_map(struct pseudo_merge_map *pm)\n+{\n+\tuint32_t i;\n+\tfor (i = 0; i < pm->nr; i++) {\n+\t\tewah_pool_free(pm->v[i].commits);\n+\t\tewah_pool_free(pm->v[i].bitmap);\n+\t}\n+\tfree(pm->v);\n+}\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nindex 81888731864..2f652fc6767 100644\n--- a/pseudo-merge.h\n+++ b/pseudo-merge.h\n@@ -99,4 +99,69 @@ void select_pseudo_merges(struct string_list *list,\n \t\t\t uint32_t *pseudo_merges_nr,\n \t\t\t unsigned show_progress);\n \n+/*\n+ * Represents a serialized view of a file containing pseudo-merge(s)\n+ * (see Documentation/technical/bitmap-format.txt for a specification\n+ * of the format).\n+ */\n+struct pseudo_merge_map {\n+\t/*\n+\t * An array of pseudo-merge(s), lazily loaded from the .bitmap\n+\t * file.\n+\t */\n+\tstruct pseudo_merge *v;\n+\tsize_t nr;\n+\tsize_t commits_nr;\n+\n+\t/*\n+\t * Pointers into a memory-mapped view of the .bitmap file:\n+\t *\n+\t * - map: the beginning of the .bitmap file\n+\t * - commits: the beginning of the pseudo-merge commit index\n+\t * - map_size: the size of the .bitmap file\n+\t */\n+\tconst unsigned char *map;\n+\tconst unsigned char *commits;\n+\n+\tsize_t map_size;\n+};\n+\n+/*\n+ * An individual pseudo-merge, storing a pair of lazily-loaded\n+ * bitmaps:\n+ *\n+ * - commits: the set of commit(s) that are part of the pseudo-merge\n+ * - bitmap: the set of object(s) reachable from the above set of\n+ * commits.\n+ *\n+ * The `at` and `bitmap_at` fields are used to store the locations of\n+ * each of the above bitmaps in the .bitmap file.\n+ */\n+struct pseudo_merge {\n+\tstruct ewah_bitmap *commits;\n+\tstruct ewah_bitmap *bitmap;\n+\n+\toff_t at;\n+\toff_t bitmap_at;\n+\n+\t/*\n+\t * `satisfied` indicates whether the given pseudo-merge has been\n+\t * used.\n+\t *\n+\t * `loaded_commits` and `loaded_bitmap` indicate whether the\n+\t * respective bitmaps have been loaded and read from the\n+\t * .bitmap file.\n+\t */\n+\tunsigned satisfied : 1,\n+\t\t loaded_commits : 1,\n+\t\t loaded_bitmap : 1;\n+};\n+\n+/*\n+ * Frees the given pseudo-merge map, releasing any memory held by (a)\n+ * parsed EWAH bitmaps, or (b) the array of pseudo-merges itself. Does\n+ * not free the memory-mapped view of the .bitmap file.\n+ */\n+void free_pseudo_merge_map(struct pseudo_merge_map *pm);\n+\n #endif\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491058","messageId":"7bb644b2b0c3478c65e16c355be41127f32c9787.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 15/24] pack-bitmap.c: read pseudo-merge extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:44Z","receivedAt":"2024-03-20T22:05:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the scaffolding for reading the pseudo-merge extension has been\nlaid, teach the pack-bitmap machinery to read the pseudo-merge extension\nwhen present.\n\nNote that pseudo-merges themselves are not yet used during traversal,\nthis step will be taken by a future commit.\n\nIn the meantime, read the table and initialize the pseudo_merge_map\nstructure introduced by a previous commit. When the pseudo-merge\nextension is present, `load_bitmap_header()` performs basic sanity\nchecks to make sure that the table is well-formed.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 39 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 39 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex b3b6f9aad21..e0f191b7581 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -20,6 +20,7 @@\n #include \"list-objects-filter-options.h\"\n #include \"midx.h\"\n #include \"config.h\"\n+#include \"pseudo-merge.h\"\n \n /*\n * An entry on the bitmap index, representing the bitmap for a given\n@@ -86,6 +87,9 @@ struct bitmap_index {\n \t */\n \tunsigned char *table_lookup;\n \n+\t/* This contains the pseudo-merge cache within 'map' (if found). */\n+\tstruct pseudo_merge_map pseudo_merges;\n+\n \t/*\n \t * Extended index.\n \t *\n@@ -205,6 +209,41 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t\t\t\tindex->table_lookup = (void *)(index_end - table_size);\n \t\t\tindex_end -= table_size;\n \t\t}\n+\n+\t\tif (flags & BITMAP_OPT_PSEUDO_MERGES) {\n+\t\t\tunsigned char *pseudo_merge_ofs;\n+\t\t\tsize_t table_size;\n+\t\t\tuint32_t i;\n+\n+\t\t\tif (sizeof(table_size) > index_end - index->map - header_size)\n+\t\t\t\treturn error(_(\"corrupted bitmap index file (too short to fit pseudo-merge table header)\"));\n+\n+\t\t\ttable_size = get_be64(index_end - 8);\n+\t\t\tif (table_size > index_end - index->map - header_size)\n+\t\t\t\treturn error(_(\"corrupted bitmap index file (too short to fit pseudo-merge table)\"));\n+\n+\t\t\tif (git_env_bool(\"GIT_TEST_USE_PSEUDO_MERGES\", 1)) {\n+\t\t\t\tconst unsigned char *ext = (index_end - table_size);\n+\n+\t\t\t\tindex->pseudo_merges.map = index->map;\n+\t\t\t\tindex->pseudo_merges.map_size = index->map_size;\n+\t\t\t\tindex->pseudo_merges.commits = ext + get_be64(index_end - 16);\n+\t\t\t\tindex->pseudo_merges.commits_nr = get_be32(index_end - 20);\n+\t\t\t\tindex->pseudo_merges.nr = get_be32(index_end - 24);\n+\n+\t\t\t\tCALLOC_ARRAY(index->pseudo_merges.v,\n+\t\t\t\t\t index->pseudo_merges.nr);\n+\n+\t\t\t\tpseudo_merge_ofs = index_end - 24 -\n+\t\t\t\t\t(index->pseudo_merges.nr * sizeof(uint64_t));\n+\t\t\t\tfor (i = 0; i < index->pseudo_merges.nr; i++) {\n+\t\t\t\t\tindex->pseudo_merges.v[i].at = get_be64(pseudo_merge_ofs);\n+\t\t\t\t\tpseudo_merge_ofs += sizeof(uint64_t);\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tindex_end -= table_size;\n+\t\t}\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491059","messageId":"792cc863154a2671291efad6a64ac8a034dd4bb4.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 16/24] pseudo-merge: implement support for reading pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:47Z","receivedAt":"2024-03-20T22:05:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement the basic API for reading pseudo-merge bitmaps, which consists\nof four basic functions:\n\n - pseudo_merge_bitmap()\n - use_pseudo_merge()\n - apply_pseudo_merges_for_commit()\n - cascade_pseudo_merges()\n\nThese functions are all documented in pseudo-merge.h, but their rough\ndescriptions are as follows:\n\n - pseudo_merge_bitmap() reads and inflates the objects EWAH bitmap for\n a given pseudo-merge\n\n - use_pseudo_merge() does the same as pseudo_merge_bitmap(), but on\n the commits EWAH bitmap, not the objects bitmap\n\n - apply_pseudo_merges_for_commit() applies all satisfied pseudo-merge\n commits for a given result set, and cascades any yet-unsatisfied\n pseudo-merges if any were applied in the previous step\n\n - cascade_pseudo_merges() applies all pseudo-merges which are\n satisfied but have not been previously applied, repeating this\n process until no more pseudo-merges can be applied\n\nThe core of the API is the latter two functions, which are responsible\nfor applying pseudo-merges during the object traversal implemented in\nthe pack-bitmap machinery.\n\nThe other two functions (pseudo_merge_bitmap(), and use_pseudo_merge())\nare low-level ways to interact with the pseudo-merge machinery, which\nwill be useful in future commits.\n\nSigned-off-by: Taylor Blau \n---\n pseudo-merge.c | 231 +++++++++++++++++++++++++++++++++++++++++++++++++\n pseudo-merge.h | 44 ++++++++++\n 2 files changed, 275 insertions(+)\n\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex d18de0a266b..e111c9cd1a6 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -10,6 +10,7 @@\n #include \"commit.h\"\n #include \"alloc.h\"\n #include \"progress.h\"\n+#include \"hex.h\"\n \n #define DEFAULT_PSEUDO_MERGE_DECAY 1.0f\n #define DEFAULT_PSEUDO_MERGE_MAX_MERGES 64\n@@ -451,3 +452,233 @@ void free_pseudo_merge_map(struct pseudo_merge_map *pm)\n \t}\n \tfree(pm->v);\n }\n+\n+struct pseudo_merge_commit_ext {\n+\tuint32_t nr;\n+\tconst unsigned char *ptr;\n+};\n+\n+static int pseudo_merge_ext_at(const struct pseudo_merge_map *pm,\n+\t\t\t struct pseudo_merge_commit_ext *ext, size_t at)\n+{\n+\tif (at >= pm->map_size)\n+\t\treturn error(_(\"extended pseudo-merge read out-of-bounds \"\n+\t\t\t \"(%\"PRIuMAX\" >= %\"PRIuMAX\")\"),\n+\t\t\t (uintmax_t)at, (uintmax_t)pm->map_size);\n+\n+\text->nr = get_be32(pm->map + at);\n+\text->ptr = pm->map + at + sizeof(uint32_t);\n+\n+\treturn 0;\n+}\n+\n+struct ewah_bitmap *pseudo_merge_bitmap(const struct pseudo_merge_map *pm,\n+\t\t\t\t\tstruct pseudo_merge *merge)\n+{\n+\tif (!merge->loaded_commits)\n+\t\tBUG(\"cannot use unloaded pseudo-merge bitmap\");\n+\n+\tif (!merge->loaded_bitmap) {\n+\t\tsize_t at = merge->bitmap_at;\n+\n+\t\tmerge->bitmap = read_bitmap(pm->map, pm->map_size, &at);\n+\t\tmerge->loaded_bitmap = 1;\n+\t}\n+\n+\treturn merge->bitmap;\n+}\n+\n+struct pseudo_merge *use_pseudo_merge(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct pseudo_merge *merge)\n+{\n+\tif (!merge->loaded_commits) {\n+\t\tsize_t pos = merge->at;\n+\n+\t\tmerge->commits = read_bitmap(pm->map, pm->map_size, &pos);\n+\t\tmerge->bitmap_at = pos;\n+\t\tmerge->loaded_commits = 1;\n+\t}\n+\treturn merge;\n+}\n+\n+static struct pseudo_merge *pseudo_merge_at(const struct pseudo_merge_map *pm,\n+\t\t\t\t\t struct object_id *oid,\n+\t\t\t\t\t size_t want)\n+{\n+\tsize_t lo = 0;\n+\tsize_t hi = pm->nr;\n+\n+\twhile (lo < hi) {\n+\t\tsize_t mi = lo + (hi - lo) / 2;\n+\t\tsize_t got = pm->v[mi].at;\n+\n+\t\tif (got == want)\n+\t\t\treturn use_pseudo_merge(pm, &pm->v[mi]);\n+\t\telse if (got < want)\n+\t\t\thi = mi;\n+\t\telse\n+\t\t\tlo = mi + 1;\n+\t}\n+\n+\twarning(_(\"could not find pseudo-merge for commit %s at offset %\"PRIuMAX),\n+\t\toid_to_hex(oid), (uintmax_t)want);\n+\n+\treturn NULL;\n+}\n+\n+struct pseudo_merge_commit {\n+\tuint32_t commit_pos;\n+\tuint64_t pseudo_merge_ofs;\n+};\n+\n+#define PSEUDO_MERGE_COMMIT_RAWSZ (sizeof(uint32_t)+sizeof(uint64_t))\n+\n+static void read_pseudo_merge_commit_at(struct pseudo_merge_commit *merge,\n+\t\t\t\t\tconst unsigned char *at)\n+{\n+\tmerge->commit_pos = get_be32(at);\n+\tmerge->pseudo_merge_ofs = get_be64(at + sizeof(uint32_t));\n+}\n+\n+static int nth_pseudo_merge_ext(const struct pseudo_merge_map *pm,\n+\t\t\t\tstruct pseudo_merge_commit_ext *ext,\n+\t\t\t\tstruct pseudo_merge_commit *merge,\n+\t\t\t\tuint32_t n)\n+{\n+\tsize_t ofs;\n+\n+\tif (n >= ext->nr)\n+\t\treturn error(_(\"extended pseudo-merge lookup out-of-bounds \"\n+\t\t\t \"(%\"PRIu32\" >= %\"PRIu32\")\"), n, ext->nr);\n+\n+\tofs = get_be64(ext->ptr + st_mult(n, sizeof(uint64_t)));\n+\tif (ofs >= pm->map_size)\n+\t\treturn error(_(\"out-of-bounds read: (%\"PRIuMAX\" >= %\"PRIuMAX\")\"),\n+\t\t\t (uintmax_t)ofs, (uintmax_t)pm->map_size);\n+\n+\tread_pseudo_merge_commit_at(merge, pm->map + ofs);\n+\n+\treturn 0;\n+}\n+\n+static unsigned apply_pseudo_merge(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct pseudo_merge *merge,\n+\t\t\t\t struct bitmap *result,\n+\t\t\t\t struct bitmap *roots)\n+{\n+\tif (merge->satisfied)\n+\t\treturn 0;\n+\n+\tif (!ewah_bitmap_is_subset(merge->commits, roots ? roots : result))\n+\t\treturn 0;\n+\n+\tbitmap_or_ewah(result, pseudo_merge_bitmap(pm, merge));\n+\tif (roots)\n+\t\tbitmap_or_ewah(roots, pseudo_merge_bitmap(pm, merge));\n+\tmerge->satisfied = 1;\n+\n+\treturn 1;\n+}\n+\n+static int pseudo_merge_commit_cmp(const void *va, const void *vb)\n+{\n+\tstruct pseudo_merge_commit merge;\n+\tuint32_t key = *(uint32_t*)va;\n+\n+\tread_pseudo_merge_commit_at(&merge, vb);\n+\n+\tif (key < merge.commit_pos)\n+\t\treturn -1;\n+\tif (key > merge.commit_pos)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static struct pseudo_merge_commit *find_pseudo_merge(const struct pseudo_merge_map *pm,\n+\t\t\t\t\t\t uint32_t pos)\n+{\n+\tif (!pm->commits_nr)\n+\t\treturn NULL;\n+\n+\treturn bsearch(&pos, pm->commits, pm->commits_nr,\n+\t\t PSEUDO_MERGE_COMMIT_RAWSZ, pseudo_merge_commit_cmp);\n+}\n+\n+int apply_pseudo_merges_for_commit(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct bitmap *result,\n+\t\t\t\t struct commit *commit, uint32_t commit_pos)\n+{\n+\tstruct pseudo_merge *merge;\n+\tstruct pseudo_merge_commit *merge_commit;\n+\tint ret = 0;\n+\n+\tmerge_commit = find_pseudo_merge(pm, commit_pos);\n+\tif (!merge_commit)\n+\t\treturn 0;\n+\n+\tif (merge_commit->pseudo_merge_ofs & ((uint64_t)1<<63)) {\n+\t\tstruct pseudo_merge_commit_ext ext = { 0 };\n+\t\toff_t ofs = merge_commit->pseudo_merge_ofs & ~((uint64_t)1<<63);\n+\t\tuint32_t i;\n+\n+\t\tif (pseudo_merge_ext_at(pm, &ext, ofs) < -1) {\n+\t\t\twarning(_(\"could not read extended pseudo-merge table \"\n+\t\t\t\t \"for commit %s\"),\n+\t\t\t\toid_to_hex(&commit->object.oid));\n+\t\t\treturn ret;\n+\t\t}\n+\n+\t\tfor (i = 0; i < ext.nr; i++) {\n+\t\t\tif (nth_pseudo_merge_ext(pm, &ext, merge_commit, i) < 0)\n+\t\t\t\treturn ret;\n+\n+\t\t\tmerge = pseudo_merge_at(pm, &commit->object.oid,\n+\t\t\t\t\t\tmerge_commit->pseudo_merge_ofs);\n+\n+\t\t\tif (!merge)\n+\t\t\t\treturn ret;\n+\n+\t\t\tif (apply_pseudo_merge(pm, merge, result, NULL))\n+\t\t\t\tret++;\n+\t\t}\n+\t} else {\n+\t\tmerge = pseudo_merge_at(pm, &commit->object.oid,\n+\t\t\t\t\tmerge_commit->pseudo_merge_ofs);\n+\n+\t\tif (!merge)\n+\t\t\treturn ret;\n+\n+\t\tif (apply_pseudo_merge(pm, merge, result, NULL))\n+\t\t\tret++;\n+\t}\n+\n+\tif (ret)\n+\t\tcascade_pseudo_merges(pm, result, NULL);\n+\n+\treturn ret;\n+}\n+\n+int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n+\t\t\t struct bitmap *result,\n+\t\t\t struct bitmap *roots)\n+{\n+\tunsigned any_satisfied;\n+\tint ret = 0;\n+\n+\tdo {\n+\t\tstruct pseudo_merge *merge;\n+\t\tuint32_t i;\n+\n+\t\tany_satisfied = 0;\n+\n+\t\tfor (i = 0; i < pm->nr; i++) {\n+\t\t\tmerge = use_pseudo_merge(pm, &pm->v[i]);\n+\t\t\tif (apply_pseudo_merge(pm, merge, result, roots)) {\n+\t\t\t\tany_satisfied |= 1;\n+\t\t\t\tret++;\n+\t\t\t}\n+\t\t}\n+\t} while (any_satisfied);\n+\n+\treturn ret;\n+}\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nindex 2f652fc6767..cc14e947e86 100644\n--- a/pseudo-merge.h\n+++ b/pseudo-merge.h\n@@ -164,4 +164,48 @@ struct pseudo_merge {\n */\n void free_pseudo_merge_map(struct pseudo_merge_map *pm);\n \n+/*\n+ * Loads the bitmap corresponding to the given pseudo-merge from the\n+ * map, if it has not already been loaded.\n+ */\n+struct ewah_bitmap *pseudo_merge_bitmap(const struct pseudo_merge_map *pm,\n+\t\t\t\t\tstruct pseudo_merge *merge);\n+\n+/*\n+ * Loads the pseudo-merge and its commits bitmap from the given\n+ * pseudo-merge map, if it has not already been loaded.\n+ */\n+struct pseudo_merge *use_pseudo_merge(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct pseudo_merge *merge);\n+\n+/*\n+ * Applies pseudo-merge(s) containing the given commit to the bitmap\n+ * \"result\".\n+ *\n+ * If any pseudo-merge(s) were satisfied, returns the number\n+ * satisfied, otherwise returns 0. If any were satisfied, the\n+ * remaining unsatisfied pseudo-merges are cascaded (see below).\n+ */\n+int apply_pseudo_merges_for_commit(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct bitmap *result,\n+\t\t\t\t struct commit *commit, uint32_t commit_pos);\n+\n+/*\n+ * Applies pseudo-merge(s) which are satisfied according to the\n+ * current bitmap in result (or roots, see below). If any\n+ * pseudo-merges were satisfied, repeat the process over unsatisfied\n+ * pseudo-merge commits until no more pseudo-merges are satisfied.\n+ *\n+ * Result is the bitmap to which the pseudo-merge(s) are applied.\n+ * Roots (if given) is a bitmap of the traversal tip(s) for either\n+ * side of a reachability traversal.\n+ *\n+ * Roots may given instead of a populated results bitmap at the\n+ * beginning of a traversal on either side where the reachability\n+ * closure over tips is not yet known.\n+ */\n+int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n+\t\t\t struct bitmap *result,\n+\t\t\t struct bitmap *roots);\n+\n #endif\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491060","messageId":"8fb7f7ab37b467bb5026296e9c7ae10632163525.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 17/24] ewah: implement `ewah_bitmap_popcount()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:49Z","receivedAt":"2024-03-20T22:05:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Some of the pseudo-merge test helpers (which will be introduced in the\nfollowing commit) will want to indicate the total number of commits in\nor objects reachable from a pseudo-merge.\n\nImplement a popcount() function that operates on EWAH bitmaps to quickly\ndetermine how many bits are set in each of the respective bitmaps.\n\nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 14 ++++++++++++++\n ewah/ewok.h | 1 +\n 2 files changed, 15 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex 5bdae3fb07b..a41fa152cbd 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -212,6 +212,20 @@ size_t bitmap_popcount(struct bitmap *self)\n \treturn count;\n }\n \n+size_t ewah_bitmap_popcount(struct ewah_bitmap *self)\n+{\n+\tstruct ewah_iterator it;\n+\teword_t word;\n+\tsize_t count = 0;\n+\n+\tewah_iterator_init(&it, self);\n+\n+\twhile (ewah_iterator_next(&word, &it))\n+\t\tcount += ewah_bit_popcount64(word);\n+\n+\treturn count;\n+}\n+\n int bitmap_is_empty(struct bitmap *self)\n {\n \tsize_t i;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex c334833b201..d7e9fb67715 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -190,6 +190,7 @@ void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other);\n void bitmap_or(struct bitmap *self, const struct bitmap *other);\n \n size_t bitmap_popcount(struct bitmap *self);\n+size_t ewah_bitmap_popcount(struct ewah_bitmap *self);\n int bitmap_is_empty(struct bitmap *self);\n \n #endif\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491061","messageId":"c839e1fed15a4a4cb827bbd177c120c8c3f74556.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 18/24] pack-bitmap: implement test helpers for pseudo-merge","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:52Z","receivedAt":"2024-03-20T22:05:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement three new sub-commands for the \"bitmap\" test-helper:\n\n - t/helper test-tool bitmap dump-pseudo-merges\n - t/helper test-tool bitmap dump-pseudo-merge-commits \n - t/helper test-tool bitmap dump-pseudo-merge-objects \n\nThese three helpers dump the list of pseudo merges, the \"parents\" of the\nnth pseudo-merges, and the set of objects reachable from those parents,\nrespectively.\n\nThese helpers will be useful in subsequent patches when we add test\ncoverage for pseudo-merge bitmaps.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 126 +++++++++++++++++++++++++++++++++++++++++\n pack-bitmap.h | 3 +\n t/helper/test-bitmap.c | 34 ++++++++---\n 3 files changed, 156 insertions(+), 7 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex e0f191b7581..7188dd75eaf 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -2442,6 +2442,132 @@ int test_bitmap_hashes(struct repository *r)\n \treturn 0;\n }\n \n+static void bit_pos_to_object_id(struct bitmap_index *bitmap_git,\n+\t\t\t\t uint32_t bit_pos,\n+\t\t\t\t struct object_id *oid)\n+{\n+\tuint32_t index_pos;\n+\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tindex_pos = pack_pos_to_midx(bitmap_git->midx, bit_pos);\n+\telse\n+\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, bit_pos);\n+\n+\tnth_bitmap_object_oid(bitmap_git, oid, index_pos);\n+}\n+\n+int test_bitmap_pseudo_merges(struct repository *r)\n+{\n+\tstruct bitmap_index *bitmap_git;\n+\tuint32_t i;\n+\n+\tbitmap_git = prepare_bitmap_git(r);\n+\tif (!bitmap_git || !bitmap_git->pseudo_merges.nr)\n+\t\tgoto cleanup;\n+\n+\tfor (i = 0; i < bitmap_git->pseudo_merges.nr; i++) {\n+\t\tstruct pseudo_merge *merge;\n+\t\tstruct ewah_bitmap *commits_bitmap, *merge_bitmap;\n+\n+\t\tmerge = use_pseudo_merge(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t &bitmap_git->pseudo_merges.v[i]);\n+\t\tcommits_bitmap = merge->commits;\n+\t\tmerge_bitmap = pseudo_merge_bitmap(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t\t merge);\n+\n+\t\tprintf(\"at=%\"PRIuMAX\", commits=%\"PRIuMAX\", objects=%\"PRIuMAX\"\\n\",\n+\t\t (uintmax_t)merge->at,\n+\t\t (uintmax_t)ewah_bitmap_popcount(commits_bitmap),\n+\t\t (uintmax_t)ewah_bitmap_popcount(merge_bitmap));\n+\t}\n+\n+cleanup:\n+\tfree_bitmap_index(bitmap_git);\n+\treturn 0;\n+}\n+\n+static void dump_ewah_object_ids(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct ewah_bitmap *bitmap)\n+\n+{\n+\tstruct ewah_iterator it;\n+\teword_t word;\n+\tuint32_t pos = 0;\n+\n+\tewah_iterator_init(&it, bitmap);\n+\n+\twhile (ewah_iterator_next(&word, &it)) {\n+\t\tstruct object_id oid;\n+\t\tuint32_t offset;\n+\n+\t\tfor (offset = 0; offset < BITS_IN_EWORD; offset++) {\n+\t\t\tif (!(word >> offset))\n+\t\t\t\tbreak;\n+\n+\t\t\toffset += ewah_bit_ctz64(word >> offset);\n+\n+\t\t\tbit_pos_to_object_id(bitmap_git, pos + offset, &oid);\n+\t\t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n+\t\t}\n+\t\tpos += BITS_IN_EWORD;\n+\t}\n+}\n+\n+int test_bitmap_pseudo_merge_commits(struct repository *r, uint32_t n)\n+{\n+\tstruct bitmap_index *bitmap_git;\n+\tstruct pseudo_merge *merge;\n+\tint ret = 0;\n+\n+\tbitmap_git = prepare_bitmap_git(r);\n+\tif (!bitmap_git || !bitmap_git->pseudo_merges.nr)\n+\t\tgoto cleanup;\n+\n+\tif (n >= bitmap_git->pseudo_merges.nr) {\n+\t\tret = error(_(\"pseudo-merge index out of range \"\n+\t\t\t \"(%\"PRIu32\" >= %\"PRIuMAX\")\"),\n+\t\t\t n, (uintmax_t)bitmap_git->pseudo_merges.nr);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tmerge = use_pseudo_merge(&bitmap_git->pseudo_merges,\n+\t\t\t\t &bitmap_git->pseudo_merges.v[n]);\n+\tdump_ewah_object_ids(bitmap_git, merge->commits);\n+\n+cleanup:\n+\tfree_bitmap_index(bitmap_git);\n+\treturn ret;\n+}\n+\n+int test_bitmap_pseudo_merge_objects(struct repository *r, uint32_t n)\n+{\n+\tstruct bitmap_index *bitmap_git;\n+\tstruct pseudo_merge *merge;\n+\tint ret = 0;\n+\n+\tbitmap_git = prepare_bitmap_git(r);\n+\tif (!bitmap_git || !bitmap_git->pseudo_merges.nr)\n+\t\tgoto cleanup;\n+\n+\tif (n >= bitmap_git->pseudo_merges.nr) {\n+\t\tret = error(_(\"pseudo-merge index out of range \"\n+\t\t\t \"(%\"PRIu32\" >= %\"PRIuMAX\")\"),\n+\t\t\t n, (uintmax_t)bitmap_git->pseudo_merges.nr);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tmerge = use_pseudo_merge(&bitmap_git->pseudo_merges,\n+\t\t\t\t &bitmap_git->pseudo_merges.v[n]);\n+\n+\tdump_ewah_object_ids(bitmap_git,\n+\t\t\t pseudo_merge_bitmap(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t\t merge));\n+\n+cleanup:\n+\tfree_bitmap_index(bitmap_git);\n+\treturn ret;\n+}\n+\n int rebuild_bitmap(const uint32_t *reposition,\n \t\t struct ewah_bitmap *source,\n \t\t struct bitmap *dest)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex a5fe4f305ef..25d3b8e604a 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -73,6 +73,9 @@ void traverse_bitmap_commit_list(struct bitmap_index *,\n void test_bitmap_walk(struct rev_info *revs);\n int test_bitmap_commits(struct repository *r);\n int test_bitmap_hashes(struct repository *r);\n+int test_bitmap_pseudo_merges(struct repository *r);\n+int test_bitmap_pseudo_merge_commits(struct repository *r, uint32_t n);\n+int test_bitmap_pseudo_merge_objects(struct repository *r, uint32_t n);\n \n #define GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL \\\n \t\"GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL\"\ndiff --git a/t/helper/test-bitmap.c b/t/helper/test-bitmap.c\nindex af43ee1cb5e..6af2b42678f 100644\n--- a/t/helper/test-bitmap.c\n+++ b/t/helper/test-bitmap.c\n@@ -13,21 +13,41 @@ static int bitmap_dump_hashes(void)\n \treturn test_bitmap_hashes(the_repository);\n }\n \n+static int bitmap_dump_pseudo_merges(void)\n+{\n+\treturn test_bitmap_pseudo_merges(the_repository);\n+}\n+\n+static int bitmap_dump_pseudo_merge_commits(uint32_t n)\n+{\n+\treturn test_bitmap_pseudo_merge_commits(the_repository, n);\n+}\n+\n+static int bitmap_dump_pseudo_merge_objects(uint32_t n)\n+{\n+\treturn test_bitmap_pseudo_merge_objects(the_repository, n);\n+}\n+\n int cmd__bitmap(int argc, const char **argv)\n {\n \tsetup_git_directory();\n \n-\tif (argc != 2)\n-\t\tgoto usage;\n-\n-\tif (!strcmp(argv[1], \"list-commits\"))\n+\tif (argc == 2 && !strcmp(argv[1], \"list-commits\"))\n \t\treturn bitmap_list_commits();\n-\tif (!strcmp(argv[1], \"dump-hashes\"))\n+\tif (argc == 2 && !strcmp(argv[1], \"dump-hashes\"))\n \t\treturn bitmap_dump_hashes();\n+\tif (argc == 2 && !strcmp(argv[1], \"dump-pseudo-merges\"))\n+\t\treturn bitmap_dump_pseudo_merges();\n+\tif (argc == 3 && !strcmp(argv[1], \"dump-pseudo-merge-commits\"))\n+\t\treturn bitmap_dump_pseudo_merge_commits(atoi(argv[2]));\n+\tif (argc == 3 && !strcmp(argv[1], \"dump-pseudo-merge-objects\"))\n+\t\treturn bitmap_dump_pseudo_merge_objects(atoi(argv[2]));\n \n-usage:\n \tusage(\"\\ttest-tool bitmap list-commits\\n\"\n-\t \"\\ttest-tool bitmap dump-hashes\");\n+\t \"\\ttest-tool bitmap dump-hashes\\n\"\n+\t \"\\ttest-tool bitmap dump-pseudo-merges\\n\"\n+\t \"\\ttest-tool bitmap dump-pseudo-merge-commits \\n\"\n+\t \"\\ttest-tool bitmap dump-pseudo-merge-objects \");\n \n \treturn -1;\n }\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491062","messageId":"7d3b88e6fd68724c1a984396be64e1043411dcb1.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 19/24] t/test-lib-functions.sh: support `--date` in `test_commit_bulk()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:55Z","receivedAt":"2024-03-20T22:05:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"One of the tests we'll want to add for pseudo-merge bitmaps needs to be\nable to generate a large number of commits at a specific date.\n\nSupport the `--date` option (with identical semantics to the `--date`\noption for `test_commit()`) within `test_commit_bulk` as a prerequisite\nfor that.\n\nSigned-off-by: Taylor Blau \n---\n t/test-lib-functions.sh | 12 +++++++++++-\n 1 file changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 6eaf116346b..312cc5d4c79 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -458,6 +458,7 @@ test_commit_bulk () {\n \tindir=.\n \tref=HEAD\n \tn=1\n+\tnotick=\n \tmessage='commit %s'\n \tfilename='%s.t'\n \tcontents='content %s'\n@@ -488,6 +489,12 @@ test_commit_bulk () {\n \t\t\tfilename=\"${1#--*=}-%s.t\"\n \t\t\tcontents=\"${1#--*=} %s\"\n \t\t\t;;\n+\t\t--date)\n+\t\t\tnotick=yes\n+\t\t\tGIT_COMMITTER_DATE=\"$2\"\n+\t\t\tGIT_AUTHOR_DATE=\"$2\"\n+\t\t\tshift\n+\t\t\t;;\n \t\t-*)\n \t\t\tBUG \"invalid test_commit_bulk option: $1\"\n \t\t\t;;\n@@ -507,7 +514,10 @@ test_commit_bulk () {\n \n \twhile test \"$total\" -gt 0\n \tdo\n-\t\ttest_tick &&\n+\t\tif test -z \"$notick\"\n+\t\tthen\n+\t\t\ttest_tick\n+\t\tfi &&\n \t\techo \"commit $ref\"\n \t\tprintf 'author %s <%s> %s\\n' \\\n \t\t\t\"$GIT_AUTHOR_NAME\" \\\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491063","messageId":"c18694ade2abad813cac7cd6c80a0b8951dd70cd.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 20/24] pack-bitmap.c: use pseudo-merges during traversal","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:05:58Z","receivedAt":"2024-03-20T22:06:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that all of the groundwork has been laid to support reading and\nusing pseudo-merges, make use of that work in this commit by teaching\nthe pack-bitmap machinery to use pseudo-merge(s) when available during\ntraversal.\n\nThe basic operation is as follows:\n\n - When enumerating objects on either side of a reachability query,\n first see if any subset of the roots satisfies some pseudo-merge\n bitmap. If it does, apply that pseudo-merge bitmap.\n\n - If any pseudo-merge bitmap(s) were applied in the previous step, OR\n them into the result[^1]. Then repeat the process over all\n pseudo-merge bitmaps (we'll refer to this as \"cascading\"\n pseudo-merges). Once this is done, OR in the resulting bitmap.\n\n - If there is no fill-in traversal to be done, return the bitmap for\n that side of the reachability query. If there is fill-in traversal,\n then for each commit we encounter via show_commit(), check to see if\n any unsatisfied pseudo-merges containing that commit as one of its\n parents has been made satisfied by the presence of that commit.\n\n If so, OR in the object set from that pseudo-merge bitmap, and then\n cascade. If not, continue traversal.\n\nA similar implementation is present in the boundary-based bitmap\ntraversal routines.\n\n[^1]: Importantly, we cannot OR in the entire set of roots along with\n the objects reachable from whatever pseudo-merge bitmaps were\n satisfied. This may leave some dangling bits corresponding to any\n unsatisfied root(s) getting OR'd into the resulting bitmap, tricking\n other parts of the traversal into thinking we already have a\n reachability closure over those commit(s) when we do not.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 112 ++++++++++-\n t/t5333-pseudo-merge-bitmaps.sh | 325 ++++++++++++++++++++++++++++++++\n 2 files changed, 436 insertions(+), 1 deletion(-)\n create mode 100755 t/t5333-pseudo-merge-bitmaps.sh\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 7188dd75eaf..a7c36a977bd 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -114,6 +114,9 @@ struct bitmap_index {\n \tunsigned int version;\n };\n \n+static int pseudo_merges_satisfied_nr;\n+static int pseudo_merges_cascades_nr;\n+\n static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st)\n {\n \tstruct ewah_bitmap *parent;\n@@ -1006,6 +1009,22 @@ static void show_commit(struct commit *commit UNUSED,\n {\n }\n \n+static unsigned apply_pseudo_merges_for_commit_1(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct bitmap *result,\n+\t\t\t\t\t\t struct commit *commit,\n+\t\t\t\t\t\t uint32_t commit_pos)\n+{\n+\tint ret;\n+\n+\tret = apply_pseudo_merges_for_commit(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t result, commit, commit_pos);\n+\n+\tif (ret)\n+\t\tpseudo_merges_satisfied_nr += ret;\n+\n+\treturn ret;\n+}\n+\n static int add_to_include_set(struct bitmap_index *bitmap_git,\n \t\t\t struct include_data *data,\n \t\t\t struct commit *commit,\n@@ -1026,6 +1045,10 @@ static int add_to_include_set(struct bitmap_index *bitmap_git,\n \t}\n \n \tbitmap_set(data->base, bitmap_pos);\n+\tif (apply_pseudo_merges_for_commit_1(bitmap_git, data->base, commit,\n+\t\t\t\t\t bitmap_pos))\n+\t\treturn 0;\n+\n \treturn 1;\n }\n \n@@ -1151,6 +1174,20 @@ static void show_boundary_object(struct object *object UNUSED,\n \tBUG(\"should not be called\");\n }\n \n+static unsigned cascade_pseudo_merges_1(struct bitmap_index *bitmap_git,\n+\t\t\t\t\tstruct bitmap *result,\n+\t\t\t\t\tstruct bitmap *roots)\n+{\n+\tint ret = cascade_pseudo_merges(&bitmap_git->pseudo_merges,\n+\t\t\t\t\tresult, roots);\n+\tif (ret) {\n+\t\tpseudo_merges_cascades_nr++;\n+\t\tpseudo_merges_satisfied_nr += ret;\n+\t}\n+\n+\treturn ret;\n+}\n+\n static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \t\t\t\t\t struct rev_info *revs,\n \t\t\t\t\t struct object_list *roots)\n@@ -1160,6 +1197,7 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \tunsigned int i;\n \tunsigned int tmp_blobs, tmp_trees, tmp_tags;\n \tint any_missing = 0;\n+\tint existing_bitmaps = 0;\n \n \tcb.bitmap_git = bitmap_git;\n \tcb.base = bitmap_new();\n@@ -1167,6 +1205,25 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \n \trevs->ignore_missing_links = 1;\n \n+\tif (bitmap_git->pseudo_merges.nr) {\n+\t\tstruct bitmap *roots_bitmap = bitmap_new();\n+\t\tstruct object_list *objects = NULL;\n+\n+\t\tfor (objects = roots; objects; objects = objects->next) {\n+\t\t\tstruct object *object = objects->item;\n+\t\t\tint pos;\n+\n+\t\t\tpos = bitmap_position(bitmap_git, &object->oid);\n+\t\t\tif (pos < 0)\n+\t\t\t\tcontinue;\n+\n+\t\t\tbitmap_set(roots_bitmap, pos);\n+\t\t}\n+\n+\t\tif (!cascade_pseudo_merges_1(bitmap_git, cb.base, roots_bitmap))\n+\t\t\tbitmap_free(roots_bitmap);\n+\t}\n+\n \t/*\n \t * OR in any existing reachability bitmaps among `roots` into\n \t * `cb.base`.\n@@ -1178,8 +1235,10 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \t\t\tcontinue;\n \n \t\tif (add_commit_to_bitmap(bitmap_git, &cb.base,\n-\t\t\t\t\t (struct commit *)object))\n+\t\t\t\t\t (struct commit *)object)) {\n+\t\t\texisting_bitmaps = 1;\n \t\t\tcontinue;\n+\t\t}\n \n \t\tany_missing = 1;\n \t}\n@@ -1187,6 +1246,9 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \tif (!any_missing)\n \t\tgoto cleanup;\n \n+\tif (existing_bitmaps)\n+\t\tcascade_pseudo_merges_1(bitmap_git, cb.base, NULL);\n+\n \ttmp_blobs = revs->blob_objects;\n \ttmp_trees = revs->tree_objects;\n \ttmp_tags = revs->blob_objects;\n@@ -1242,6 +1304,13 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \treturn cb.base;\n }\n \n+static void unsatisfy_all_pseudo_merges(struct bitmap_index *bitmap_git)\n+{\n+\tuint32_t i;\n+\tfor (i = 0; i < bitmap_git->pseudo_merges.nr; i++)\n+\t\tbitmap_git->pseudo_merges.v[i].satisfied = 0;\n+}\n+\n static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\t\t\t struct rev_info *revs,\n \t\t\t\t struct object_list *roots,\n@@ -1249,9 +1318,32 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n {\n \tstruct bitmap *base = NULL;\n \tint needs_walk = 0;\n+\tunsigned existing_bitmaps = 0;\n \n \tstruct object_list *not_mapped = NULL;\n \n+\tunsatisfy_all_pseudo_merges(bitmap_git);\n+\n+\tif (bitmap_git->pseudo_merges.nr) {\n+\t\tstruct bitmap *roots_bitmap = bitmap_new();\n+\t\tstruct object_list *objects = NULL;\n+\n+\t\tfor (objects = roots; objects; objects = objects->next) {\n+\t\t\tstruct object *object = objects->item;\n+\t\t\tint pos;\n+\n+\t\t\tpos = bitmap_position(bitmap_git, &object->oid);\n+\t\t\tif (pos < 0)\n+\t\t\t\tcontinue;\n+\n+\t\t\tbitmap_set(roots_bitmap, pos);\n+\t\t}\n+\n+\t\tbase = bitmap_new();\n+\t\tif (!cascade_pseudo_merges_1(bitmap_git, base, roots_bitmap))\n+\t\t\tbitmap_free(roots_bitmap);\n+\t}\n+\n \t/*\n \t * Go through all the roots for the walk. The ones that have bitmaps\n \t * on the bitmap index will be `or`ed together to form an initial\n@@ -1262,11 +1354,21 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t */\n \twhile (roots) {\n \t\tstruct object *object = roots->item;\n+\n \t\troots = roots->next;\n \n+\t\tif (base) {\n+\t\t\tint pos = bitmap_position(bitmap_git, &object->oid);\n+\t\t\tif (pos > 0 && bitmap_get(base, pos)) {\n+\t\t\t\tobject->flags |= SEEN;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n \t\tif (object->type == OBJ_COMMIT &&\n \t\t add_commit_to_bitmap(bitmap_git, &base, (struct commit *)object)) {\n \t\t\tobject->flags |= SEEN;\n+\t\t\texisting_bitmaps = 1;\n \t\t\tcontinue;\n \t\t}\n \n@@ -1282,6 +1384,9 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \n \troots = not_mapped;\n \n+\tif (existing_bitmaps)\n+\t\tcascade_pseudo_merges_1(bitmap_git, base, NULL);\n+\n \t/*\n \t * Let's iterate through all the roots that don't have bitmaps to\n \t * check if we can determine them to be reachable from the existing\n@@ -1866,6 +1971,11 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tobject_list_free(&wants);\n \tobject_list_free(&haves);\n \n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"pseudo_merges_satisfied\",\n+\t\t\t pseudo_merges_satisfied_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"pseudo_merges_cascades\",\n+\t\t\t pseudo_merges_cascades_nr);\n+\n \treturn bitmap_git;\n \n cleanup:\ndiff --git a/t/t5333-pseudo-merge-bitmaps.sh b/t/t5333-pseudo-merge-bitmaps.sh\nnew file mode 100755\nindex 00000000000..909c17e301e\n--- /dev/null\n+++ b/t/t5333-pseudo-merge-bitmaps.sh\n@@ -0,0 +1,325 @@\n+#!/bin/sh\n+\n+test_description='pseudo-merge bitmaps'\n+\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n+. ./test-lib.sh\n+\n+test_pseudo_merges () {\n+\ttest-tool bitmap dump-pseudo-merges\n+}\n+\n+test_pseudo_merge_commits () {\n+\ttest-tool bitmap dump-pseudo-merge-commits \"$1\"\n+}\n+\n+test_pseudo_merges_satisfied () {\n+\ttest_trace2_data bitmap pseudo_merges_satisfied \"$1\"\n+}\n+\n+test_pseudo_merges_cascades () {\n+\ttest_trace2_data bitmap pseudo_merges_cascades \"$1\"\n+}\n+\n+tag_everything () {\n+\tgit rev-list --all --no-object-names >in &&\n+\tperl -lne '\n+\t\tprint \"create refs/tags/\" . $. . \" \" . $1 if /([0-9a-f]+)/\n+\t' expect &&\n+\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\n+\ttest_pseudo_merges_satisfied 0 merges &&\n+\ttest_must_be_empty merges &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'pseudo-merges accurately represent their objects' '\n+\ttest_config bitmapPseudoMerge.test.pattern \"refs/tags/\" &&\n+\ttest_config bitmapPseudoMerge.test.maxMerges 8 &&\n+\ttest_config bitmapPseudoMerge.test.stableThreshold never &&\n+\n+\tgit repack -adb &&\n+\n+\ttest_pseudo_merges >merges &&\n+\ttest_line_count = 8 merges &&\n+\n+\tfor i in $(test_seq 0 $(($(wc -l commits &&\n+\n+\t\tgit rev-list --objects --no-object-names --stdin expect.raw &&\n+\t\ttest-tool bitmap dump-pseudo-merge-objects $i >actual.raw &&\n+\n+\t\tsort -u expect &&\n+\t\tsort -u actual &&\n+\n+\t\ttest_cmp expect actual || return 1\n+\tdone\n+'\n+\n+test_expect_success 'bitmap traversal with pseudo-merges' '\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\tgit rev-list --count --all --objects >expect &&\n+\n+\ttest_pseudo_merges_satisfied 8 trace2.txt &&\n+\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\tgit rev-list --count --all --objects >expect &&\n+\n+\ttest_pseudo_merges_satisfied 8 merges &&\n+\t\tfor i in $(test_seq 0 $(($(wc -l commits &&\n+\n+\t\ttest-tool bitmap list-commits >bitmaps &&\n+\t\tbitmaps_nr=\"$(wc -l expect &&\n+\n+\t\ttest $(cat expect) -eq $(wc -l merges &&\n+\t\ttest_line_count = 1 merges &&\n+\n+\t\ttest_pseudo_merge_commits 0 >oids &&\n+\t\tgit cat-file --batch commits &&\n+\n+\t\ttest $(wc -l in &&\n+\t\tgit update-ref --stdin merges &&\n+\t\tmerges_nr=\"$(wc -l oids &&\n+\t\t\tgit cat-file --batch commits &&\n+\n+\t\t\texpect=\"$(grep -c \"^committer.*$old +0000$\" commits)\" &&\n+\t\t\tactual=\"$(wc -l oids &&\n+\t\tgit cat-file --batch commits &&\n+\t\ttest $(wc -l err &&\n+\n+\tcat >expect <<-EOF &&\n+\tfatal: pseudo-merge group ${SQ}test${SQ} has unstable threshold before stable one\n+\tEOF\n+\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'pseudo-merge pattern with capture groups' '\n+\tgit init pseudo-merge-captures &&\n+\t(\n+\t\tcd pseudo-merge-captures &&\n+\n+\t\ttest_commit_bulk 128 &&\n+\t\ttag_everything &&\n+\n+\t\tfor r in $(test_seq 8)\n+\t\tdo\n+\t\t\ttest_commit_bulk 16 &&\n+\n+\t\t\tgit rev-list HEAD~16.. >in &&\n+\n+\t\t\tperl -lne \"print \\\"create refs/remotes/$r/tags/\\$. \\$_\\\"\" refs &&\n+\n+\t\ttest_pseudo_merges >merges &&\n+\t\tfor m in $(test_seq 0 $(($(wc -l oids &&\n+\t\t\tgrep -f oids refs |\n+\t\t\tperl -lne \"print \\$1 if /refs\\/remotes\\/([0-9]+)/\" |\n+\t\t\tsort -u || return 1\n+\t\tdone >remotes &&\n+\n+\t\ttest $(wc -l merges &&\n+\t\ttest_line_count = 2 merges &&\n+\n+\t\ttest_pseudo_merge_commits 0 >commits-0.raw &&\n+\t\ttest_pseudo_merge_commits 1 >commits-1.raw &&\n+\n+\t\tsort commits-0.raw >commits-0 &&\n+\t\tsort commits-1.raw >commits-1 &&\n+\n+\t\tcomm -12 commits-0 commits-1 >overlap &&\n+\n+\t\ttest_line_count -gt 0 overlap\n+\t)\n+'\n+\n+test_expect_success 'pseudo-merge overlap traversal' '\n+\t(\n+\t\tcd pseudo-merge-overlap &&\n+\n+\t\t: >trace2.txt &&\n+\t\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\t\tgit rev-list --count --all --objects >expect &&\n+\n+\t\ttest_pseudo_merges_satisfied 2 trace2.txt &&\n+\t\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\t\tgit rev-list --count --all --objects >expect &&\n+\n+\t\ttest_pseudo_merges_satisfied 2 \n---\n pack-bitmap.c | 26 +++++++++++++++++++++++++-\n 1 file changed, 25 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex a7c36a977bd..be65f637cf5 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -116,6 +116,10 @@ struct bitmap_index {\n \n static int pseudo_merges_satisfied_nr;\n static int pseudo_merges_cascades_nr;\n+static int existing_bitmaps_hits_nr;\n+static int existing_bitmaps_misses_nr;\n+static int roots_with_bitmaps_nr;\n+static int roots_without_bitmaps_nr;\n \n static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st)\n {\n@@ -1040,10 +1044,14 @@ static int add_to_include_set(struct bitmap_index *bitmap_git,\n \n \tpartial = bitmap_for_commit(bitmap_git, commit);\n \tif (partial) {\n+\t\texisting_bitmaps_hits_nr++;\n+\n \t\tbitmap_or_ewah(data->base, partial);\n \t\treturn 0;\n \t}\n \n+\texisting_bitmaps_misses_nr++;\n+\n \tbitmap_set(data->base, bitmap_pos);\n \tif (apply_pseudo_merges_for_commit_1(bitmap_git, data->base, commit,\n \t\t\t\t\t bitmap_pos))\n@@ -1099,8 +1107,12 @@ static int add_commit_to_bitmap(struct bitmap_index *bitmap_git,\n {\n \tstruct ewah_bitmap *or_with = bitmap_for_commit(bitmap_git, commit);\n \n-\tif (!or_with)\n+\tif (!or_with) {\n+\t\texisting_bitmaps_misses_nr++;\n \t\treturn 0;\n+\t}\n+\n+\texisting_bitmaps_hits_nr++;\n \n \tif (!*base)\n \t\t*base = ewah_to_bitmap(or_with);\n@@ -1407,8 +1419,12 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\t\tobject->flags &= ~UNINTERESTING;\n \t\t\tadd_pending_object(revs, object, \"\");\n \t\t\tneeds_walk = 1;\n+\n+\t\t\troots_without_bitmaps_nr++;\n \t\t} else {\n \t\t\tobject->flags |= SEEN;\n+\n+\t\t\troots_with_bitmaps_nr++;\n \t\t}\n \t}\n \n@@ -1975,6 +1991,14 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\t pseudo_merges_satisfied_nr);\n \ttrace2_data_intmax(\"bitmap\", the_repository, \"pseudo_merges_cascades\",\n \t\t\t pseudo_merges_cascades_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"bitmap/hits\",\n+\t\t\t existing_bitmaps_hits_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"bitmap/misses\",\n+\t\t\t existing_bitmaps_misses_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"bitmap/roots_with_bitmap\",\n+\t\t\t roots_with_bitmaps_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"bitmap/roots_without_bitmap\",\n+\t\t\t roots_without_bitmaps_nr);\n \n \treturn bitmap_git;\n \n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491064","messageId":"1eb10c190ba1f045b3eab8c1975a77e6a046c8ee.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 22/24] ewah: `bitmap_equals_ewah()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:06:04Z","receivedAt":"2024-03-20T22:06:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to reuse existing pseudo-merge bitmaps by implementing a\n`bitmap_equals_ewah()` helper.\n\nThis helper will be used to see if a raw bitmap (containing the set of\nparents for some pseudo-merge) is equal to any existing pseudo-merge's\ncommits bitmap (which are stored as EWAH-compressed bitmaps on disk).\n\nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 19 +++++++++++++++++++\n ewah/ewok.h | 1 +\n 2 files changed, 20 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex a41fa152cbd..59dc77a08f6 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -261,6 +261,25 @@ int bitmap_equals(struct bitmap *self, struct bitmap *other)\n \treturn 1;\n }\n \n+int bitmap_equals_ewah(struct bitmap *self, struct ewah_bitmap *other)\n+{\n+\tstruct ewah_iterator it;\n+\teword_t word;\n+\tsize_t i = 0;\n+\n+\tewah_iterator_init(&it, other);\n+\n+\twhile (ewah_iterator_next(&word, &it))\n+\t\tif (word != (i < self->word_alloc ? self->words[i++] : 0))\n+\t\t\treturn 0;\n+\n+\tfor (; i < self->word_alloc; i++)\n+\t\tif (self->words[i])\n+\t\t\treturn 0;\n+\n+\treturn 1;\n+}\n+\n int bitmap_is_subset(struct bitmap *self, struct bitmap *other)\n {\n \tsize_t common_size, i;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex d7e9fb67715..0d49ec00618 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -179,6 +179,7 @@ void bitmap_unset(struct bitmap *self, size_t pos);\n int bitmap_get(struct bitmap *self, size_t pos);\n void bitmap_free(struct bitmap *self);\n int bitmap_equals(struct bitmap *self, struct bitmap *other);\n+int bitmap_equals_ewah(struct bitmap *self, struct ewah_bitmap *other);\n int bitmap_is_subset(struct bitmap *self, struct bitmap *other);\n int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other);\n \n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491066","messageId":"4ae4f0eaae5ebe9495968e8585f4b2692d2cbec2.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 23/24] pseudo-merge: implement support for finding existing merges","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:06:07Z","receivedAt":"2024-03-20T22:06:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This patch implements support for reusing existing pseudo-merge commits\nwhen writing bitmaps when there is an existing pseudo-merge bitmap which\nhas exactly the same set of parents as one that we are about to write.\n\nNote that unstable pseudo-merges are likely to change between\nconsecutive repacks, and so are generally poor candidates for reuse.\nHowever, stable pseudo-merges (see the configuration option\n'bitmapPseudoMerge..stableThreshold') are by definition unlikely\nto change between runs (as they represent long-running branches).\n\nBecause there is no index from a *set* of pseudo-merge parents to a\nmatching pseudo-merge bitmap, we have to construct the bitmap\ncorresponding to the set of parents for each pending pseudo-merge commit\nand see if a matching bitmap exists.\n\nThis is technically quadratic in the number of pseudo-merges, but is OK\nin practice for a couple of reasons:\n\n - non-matching pseudo-merge bitmaps are rejected quickly as soon as\n they differ in a single bit\n\n - already-matched pseudo-merge bitmaps are discarded from subsequent\n rounds of search\n\n - the number of pseudo-merges is generally small, even for large\n repositories\n\nIn order to do this, implement (a) a function that finds a matching\npseudo-merge given some uncompressed bitset describing its parents, (b)\na function that computes the bitset of parents for a given pseudo-merge\ncommit, and (c) call that function before computing the set of reachable\nobjects for some pending pseudo-merge.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 15 ++++++--\n pack-bitmap.c | 32 +++++++++++++++++\n pack-bitmap.h | 2 ++\n pseudo-merge.c | 55 ++++++++++++++++++++++++++++\n pseudo-merge.h | 7 ++++\n t/t5333-pseudo-merge-bitmaps.sh | 64 +++++++++++++++++++++++++++++++++\n 6 files changed, 173 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 2d1b202fcd9..fdd84d31a68 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -19,6 +19,10 @@\n #include \"tree-walk.h\"\n #include \"pseudo-merge.h\"\n #include \"oid-array.h\"\n+#include \"config.h\"\n+#include \"alloc.h\"\n+#include \"refs.h\"\n+#include \"strmap.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -443,6 +447,7 @@ static int fill_bitmap_tree(struct bitmap *bitmap,\n }\n \n static int reused_bitmaps_nr;\n+static int reused_pseudo_merge_bitmaps_nr;\n \n static int fill_bitmap_commit(struct bb_commit *ent,\n \t\t\t struct commit *commit,\n@@ -467,7 +472,7 @@ static int fill_bitmap_commit(struct bb_commit *ent,\n \t\t\tstruct bitmap *remapped = bitmap_new();\n \n \t\t\tif (commit->object.flags & BITMAP_PSEUDO_MERGE)\n-\t\t\t\told = NULL;\n+\t\t\t\told = pseudo_merge_bitmap_for_commit(old_bitmap, c);\n \t\t\telse\n \t\t\t\told = bitmap_for_commit(old_bitmap, c);\n \t\t\t/*\n@@ -478,7 +483,10 @@ static int fill_bitmap_commit(struct bb_commit *ent,\n \t\t\tif (old && !rebuild_bitmap(mapping, old, remapped)) {\n \t\t\t\tbitmap_or(ent->bitmap, remapped);\n \t\t\t\tbitmap_free(remapped);\n-\t\t\t\treused_bitmaps_nr++;\n+\t\t\t\tif (commit->object.flags & BITMAP_PSEUDO_MERGE)\n+\t\t\t\t\treused_pseudo_merge_bitmaps_nr++;\n+\t\t\t\telse\n+\t\t\t\t\treused_bitmaps_nr++;\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tbitmap_free(remapped);\n@@ -604,6 +612,9 @@ int bitmap_writer_build(struct packing_data *to_pack)\n \t\t\t the_repository);\n \ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n \t\t\t \"building_bitmaps_reused\", reused_bitmaps_nr);\n+\ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n+\t\t\t \"building_bitmaps_pseudo_merge_reused\",\n+\t\t\t reused_pseudo_merge_bitmaps_nr);\n \n \tstop_progress(&writer.progress);\n \ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex be65f637cf5..5a5f8b7e69f 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1316,6 +1316,37 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \treturn cb.base;\n }\n \n+struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct commit *commit)\n+{\n+\tstruct commit_list *p;\n+\tstruct bitmap *parents;\n+\tstruct pseudo_merge *match = NULL;\n+\n+\tif (!bitmap_git->pseudo_merges.nr)\n+\t\treturn NULL;\n+\n+\tparents = bitmap_new();\n+\n+\tfor (p = commit->parents; p; p = p->next) {\n+\t\tint pos = bitmap_position(bitmap_git, &p->item->object.oid);\n+\t\tif (pos < 0 || pos >= bitmap_num_objects(bitmap_git))\n+\t\t\tgoto done;\n+\n+\t\tbitmap_set(parents, pos);\n+\t}\n+\n+\tmatch = pseudo_merge_for_parents(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t\tparents);\n+\n+done:\n+\tbitmap_free(parents);\n+\tif (match)\n+\t\treturn pseudo_merge_bitmap(&bitmap_git->pseudo_merges, match);\n+\n+\treturn NULL;\n+}\n+\n static void unsatisfy_all_pseudo_merges(struct bitmap_index *bitmap_git)\n {\n \tuint32_t i;\n@@ -2808,6 +2839,7 @@ void free_bitmap_index(struct bitmap_index *b)\n \t\t */\n \t\tclose_midx_revindex(b->midx);\n \t}\n+\tfree_pseudo_merge_map(&b->pseudo_merges);\n \tfree(b);\n }\n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 25d3b8e604a..0fefef39bec 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -119,6 +119,8 @@ int rebuild_bitmap(const uint32_t *reposition,\n \t\t struct bitmap *dest);\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t struct commit *commit);\n+struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct commit *commit);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\t\t\t unsigned int indexed_commits_nr);\n int bitmap_writer_build(struct packing_data *to_pack);\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex e111c9cd1a6..9e21fbb5062 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -682,3 +682,58 @@ int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n \n \treturn ret;\n }\n+\n+struct pseudo_merge *pseudo_merge_for_parents(const struct pseudo_merge_map *pm,\n+\t\t\t\t\t struct bitmap *parents)\n+{\n+\tstruct pseudo_merge *match = NULL;\n+\tsize_t i;\n+\n+\tif (!pm->nr)\n+\t\treturn NULL;\n+\n+\t/*\n+\t * NOTE: this loop is quadratic in the worst-case (where no\n+\t * matching pseudo-merge bitmaps are found), but in practice\n+\t * this is OK for a few reasons:\n+\t *\n+\t * - Rejecting pseudo-merge bitmaps that do not match the\n+\t * given commit is done quickly (i.e. `bitmap_equals_ewah()`\n+\t * returns early when we know the two bitmaps aren't equal.\n+\t *\n+\t * - Already matched pseudo-merge bitmaps (which we track with\n+\t * the `->satisfied` bit here) are skipped as potential\n+\t * candidates.\n+\t *\n+\t * - The number of pseudo-merges should be small (in the\n+\t * hundreds for most repositories).\n+\t *\n+\t * If in the future this semi-quadratic behavior does become a\n+\t * problem, another approach would be to keep track of which\n+\t * pseudo-merges are still \"viable\" after enumerating the\n+\t * pseudo-merge commit's parents:\n+\t *\n+\t * - A pseudo-merge bitmap becomes non-viable when the bit(s)\n+\t * corresponding to one or more parent(s) of the given\n+\t * commit are not set in a candidate pseudo-merge's commits\n+\t * bitmap.\n+\t *\n+\t * - After processing all bits, enumerate the remaining set of\n+\t * viable pseudo-merge bitmaps, and check that their\n+\t * popcount() matches the number of parents in the given\n+\t * commit.\n+\t */\n+\tfor (i = 0; i < pm->nr; i++) {\n+\t\tstruct pseudo_merge *candidate = use_pseudo_merge(pm, &pm->v[i]);\n+\t\tif (!candidate || candidate->satisfied)\n+\t\t\tcontinue;\n+\t\tif (!bitmap_equals_ewah(parents, candidate->commits))\n+\t\t\tcontinue;\n+\n+\t\tmatch = candidate;\n+\t\tmatch->satisfied = 1;\n+\t\tbreak;\n+\t}\n+\n+\treturn match;\n+}\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nindex cc14e947e86..33acd00a3e5 100644\n--- a/pseudo-merge.h\n+++ b/pseudo-merge.h\n@@ -208,4 +208,11 @@ int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n \t\t\t struct bitmap *result,\n \t\t\t struct bitmap *roots);\n \n+/*\n+ * Returns a pseudo-merge which contains the exact set of commits\n+ * listed in the \"parents\" bitamp, or NULL if none could be found.\n+ */\n+struct pseudo_merge *pseudo_merge_for_parents(const struct pseudo_merge_map *pm,\n+\t\t\t\t\t struct bitmap *parents);\n+\n #endif\ndiff --git a/t/t5333-pseudo-merge-bitmaps.sh b/t/t5333-pseudo-merge-bitmaps.sh\nindex 909c17e301e..531f1924af4 100755\n--- a/t/t5333-pseudo-merge-bitmaps.sh\n+++ b/t/t5333-pseudo-merge-bitmaps.sh\n@@ -22,6 +22,10 @@ test_pseudo_merges_cascades () {\n \ttest_trace2_data bitmap pseudo_merges_cascades \"$1\"\n }\n \n+test_pseudo_merges_reused () {\n+\ttest_trace2_data pack-bitmap-write building_bitmaps_pseudo_merge_reused \"$1\"\n+}\n+\n tag_everything () {\n \tgit rev-list --all --no-object-names >in &&\n \tperl -lne '\n@@ -322,4 +326,64 @@ test_expect_success 'pseudo-merge overlap stale traversal' '\n \t)\n '\n \n+test_expect_success 'pseudo-merge reuse' '\n+\tgit init pseudo-merge-reuse &&\n+\t(\n+\t\tcd pseudo-merge-reuse &&\n+\n+\t\tstable=\"1641013200\" && # 2022-01-01\n+\t\tunstable=\"1672549200\" && # 2023-01-01\n+\n+\t\tfor date in $stable $unstable\n+\t\tdo\n+\t\t\ttest_commit_bulk --date \"$date +0000\" 128 &&\n+\t\t\ttest_tick || return 1\n+\t\tdone &&\n+\n+\t\ttag_everything &&\n+\n+\t\tgit \\\n+\t\t\t-c bitmapPseudoMerge.test.pattern=\"refs/tags/\" \\\n+\t\t\t-c bitmapPseudoMerge.test.maxMerges=1 \\\n+\t\t\t-c bitmapPseudoMerge.test.threshold=now \\\n+\t\t\t-c bitmapPseudoMerge.test.stableThreshold=$(($unstable - 1)) \\\n+\t\t\t-c bitmapPseudoMerge.test.stableSize=512 \\\n+\t\t\trepack -adb &&\n+\n+\t\ttest_pseudo_merges >merges &&\n+\t\ttest_line_count = 2 merges &&\n+\n+\t\ttest_pseudo_merge_commits 0 >stable-oids.before &&\n+\t\ttest_pseudo_merge_commits 1 >unstable-oids.before &&\n+\n+\t\t: >trace2.txt &&\n+\t\tGIT_TRACE2_EVENT=$PWD/trace2.txt git \\\n+\t\t\t-c bitmapPseudoMerge.test.pattern=\"refs/tags/\" \\\n+\t\t\t-c bitmapPseudoMerge.test.maxMerges=2 \\\n+\t\t\t-c bitmapPseudoMerge.test.threshold=now \\\n+\t\t\t-c bitmapPseudoMerge.test.stableThreshold=$(($unstable - 1)) \\\n+\t\t\t-c bitmapPseudoMerge.test.stableSize=512 \\\n+\t\t\trepack -adb &&\n+\n+\t\ttest_pseudo_merges_reused 1 merges &&\n+\t\ttest_line_count = 3 merges &&\n+\n+\t\ttest_pseudo_merge_commits 0 >stable-oids.after &&\n+\t\tfor i in 1 2\n+\t\tdo\n+\t\t\ttest_pseudo_merge_commits $i || return 1\n+\t\tdone >unstable-oids.after &&\n+\n+\t\tsort -u expect &&\n+\t\tsort -u actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tsort -u expect &&\n+\t\tsort -u actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.44.0.303.g1dc5e5b124c\n\n"},{"id":"491067","messageId":"a05ad42202d08b419deee5af68e109f10ebb14e7.1710972293.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH 24/24] t/perf: implement performace tests for pseudo-merge bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-20T22:06:10Z","receivedAt":"2024-03-20T22:06:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement a straightforward performance test demonstrating the benefit\nof pseudo-merge bitmaps by measuring how long it takes to count\nreachable objects in a few different scenarios:\n\n - without bitmaps, to demonstrate a reasonable baseline\n - with bitmaps, but without pseudo-merges\n - with bitmaps and pseudo-merges\n\nResults from running this test on git.git are as follows:\n\n Test this tree\n -----------------------------------------------------------------------------------\n 5333.2: git rev-list --count --all --objects (no bitmaps) 3.46(3.37+0.09)\n 5333.3: git rev-list --count --all --objects (no pseudo-merges) 0.13(0.11+0.01)\n 5333.4: git rev-list --count --all --objects (with pseudo-merges) 0.12(0.11+0.01)\n\nSigned-off-by: Taylor Blau \n---\n t/perf/p5333-pseudo-merge-bitmaps.sh | 32 ++++++++++++++++++++++++++++\n 1 file changed, 32 insertions(+)\n create mode 100755 t/perf/p5333-pseudo-merge-bitmaps.sh\n\ndiff --git a/t/perf/p5333-pseudo-merge-bitmaps.sh b/t/perf/p5333-pseudo-merge-bitmaps.sh\nnew file mode 100755\nindex 00000000000..4bec409d10e\n--- /dev/null\n+++ b/t/perf/p5333-pseudo-merge-bitmaps.sh\n@@ -0,0 +1,32 @@\n+#!/bin/sh\n+\n+test_description='pseudo-merge bitmaps'\n+. ./perf-lib.sh\n+\n+test_perf_large_repo\n+\n+test_expect_success 'setup' '\n+\tgit \\\n+\t\t-c bitmapPseudoMerge.all.pattern=\"refs/\" \\\n+\t\t-c bitmapPseudoMerge.all.threshold=now \\\n+\t\t-c bitmapPseudoMerge.all.stableThreshold=never \\\n+\t\t-c bitmapPseudoMerge.all.maxMerges=64 \\\n+\t\t-c pack.writeBitmapLookupTable=true \\\n+\t\trepack -adb\n+'\n+\n+test_perf 'git rev-list --count --all --objects (no bitmaps)' '\n+\tgit rev-list --objects --all\n+'\n+\n+test_perf 'git rev-list --count --all --objects (no pseudo-merges)' '\n+\tGIT_TEST_USE_PSEDUO_MERGES=0 \\\n+\t\tgit rev-list --objects --all --use-bitmap-index\n+'\n+\n+test_perf 'git rev-list --count --all --objects (with pseudo-merges)' '\n+\tGIT_TEST_USE_PSEDUO_MERGES=1 \\\n+\t\tgit rev-list --objects --all --use-bitmap-index\n+'\n+\n+test_done\n-- \n2.44.0.303.g1dc5e5b124c\n"},{"id":"491170","messageId":"xmqqzfur74a7.fsf@gitster.g","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"Re: [PATCH 00/24] pack-bitmap: pseudo-merge reachability bitmaps","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-03-21T19:50:40Z","receivedAt":"2024-03-21T19:50:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> This series implements a new idea in the pack-bitmap machinery called\n> \"pseudo-merge reachability bitmaps\".\n\nWhen you work on your next series, please describe the feature in a\ndifferent order. Giving name, and then mechanism (from coarse to\ndetailed), and then finally what it is good for, is a sure way to\ndiscourage people from reading the long description. What is the\nproblem, what insight led to a new solution to the problem, and what\nthe mechansim looks like, is probably an order to keep the reader\nengaged better.\n\n"},{"id":"491177","messageId":"xmqqcyrn6zyd.fsf@gitster.g","threadId":"61166","inReplyTo":"76e7e3b9cca7fb957384ed98f2cd32cf11ff8488.1710972293.git.me@ttaylorr.com","subject":"Re: [PATCH 01/24] Documentation/technical: describe pseudo-merge bitmaps format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-03-21T21:24:10Z","receivedAt":"2024-03-21T21:24:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Prepare to implement pseudo-merge bitmaps over the next several commits\n> by first describing the serialization format which will store the new\n> pseudo-merge bitmaps themselves.\n\nBefore talking about what problem, which is not addressed adequately\nwith existing mechanisms, it will solve?\n\n> +Pseudo-merge bitmaps\n> +--------------------\n> +\n> +If the `BITMAP_OPT_PSEUDO_MERGES` flag is set, a variable number of\n> +bytes (preceding the name-hash cache, commit lookup table, and trailing\n> +checksum) of the `.bitmap` file is used to store pseudo-merge bitmaps.\n> +\n> +A \"pseudo-merge bitmap\" is used to refer to a pair of bitmaps, as\n> +follows:\n> +\n> +Commit bitmap::\n> +\n> + A bitmap whose set bits describe the set of commits included in the\n> + pseudo-merge's \"merge\" bitmap (as below).\n> +\n> +Merge bitmap::\n> +\n> + A bitmap whose set bits describe the reachability closure over the set\n> + of commits in the pseudo-merge's \"commits\" bitmap (as above). An\n> + identical bitmap would be generated for an octopus merge with the same\n> + set of parents as described in the commits bitmap.\n> +\n> +Pseudo-merge bitmaps can accelerate bitmap traversals when all commits\n> +for a given pseudo-merge are listed on either side of the traversal,\n> +either directly (by explicitly asking for them as part of the `HAVES`\n> +or `WANTS`) or indirectly (by encountering them during a fill-in\n> +traversal).\n\n\"either side of\" implies there are two sides. Is it correct to\nunderstand that they are \"the side reachable from HAVE\" and \"the\nother side that is reachable from WANT\"?\n\n> +=== Use-cases\n> +\n> +For example, suppose there exists a pseudo-merge bitmap with a large\n> +number of commits, all of which are listed in the `WANTS` section of\n> +some bitmap traversal query. When pseudo-merge bitmaps are enabled, the\n> +bitmap machinery can quickly determine there is a pseudo-merge which\n> +satisfies some subset of the wanted objects on either side of the query.\n\nHere you only talk about WANT but still mention \"either side of\".\nHow would the HAVE side of the query contribute to the computation?\n\n> + ** `commits_bitmap`, an EWAH-compressed bitmap describing the set of\n> + commits included in the this psuedo-merge.\n> +\n> + ** `merge_bitmap`, an EWAH-compressed bitmap describing the union of\n> + the set of objects reachable from all commits listed in the\n> + `commits_bitmap`.\n\n\"union of the set of objects reachable from all\", meaning if an\nobject is listed here, all commits in commits_bitmap are guaranteed\nto reach that object? It sounds more like the intersction of sets\nthan union.\n\n> +* A lookup table, mapping pseudo-merged commits to the pseudo-merges\n> + they belong to. Entries appear in increasing order of each commit's\n> + bit position. Each entry is 12 bytes wide, and is comprised of the\n> + following:\n\n\"a pseudo-merged commit\" is a new term. It was explained what \"a\npseudo-merge bitmap\" is already, and it was explained that \"a\npseudo-merge bitmap\" consists of a pair of bitmaps (commit bitmap\nthat records which commit belongs to the \"pseudo-merge\", and merge\nbitmap that records objects reachable from all commits in the commit\nbitmap). But we haven't heard of \"a pseudo-merged commit\", or what\nthe verb \"to pseudo-merge a commit\" means.\n\nDoes it merely mean \"a commit that is recorded in the commit-bitmap\nhalf of a pseudo-merge bitmap\"? It still is unclear at this point\nin the description if a commit can be part of only one such\ncommit-bitmap and makes readers reading hiccup, until a few\nparagraphs below where extended table is there to help a commit\nrecorded in commit-bitmap of more than one pseudo-merge bitmaps.\n\nI'll stop here for now, but this made me even more convinced that\nthe presentation order needs to be rethought to sell why this whole\nthing is a good idea by telling readers what problem it is solving,\nwhy a new data structure helps and how, etc. Perhaps you can start\nby trying to write a paragraph of description for the topic suitable\nfor the \"What's cooking\" report, which needs to do a good elevator\npitch.\n\nThanks.\n\n> + ** `commit_pos`, a 4-byte unsigned value (in network byte-order)\n> + containing the bit position for this commit.\n> +\n> + ** `offset`, an 8-byte unsigned value (also in network byte-order)\n> + containing either one of two possible offsets, depending on whether or\n> + not the most-significant bit is set.\n> +\n> + *** If unset (i.e. `offset & ((uint64_t)1<<63) == 0`), the offset\n> +\t(relative to the beginning of the `.bitmap` file) at which the\n> +\tpseudo-merge bitmap for this commit can be read. This indicates\n> +\tonly a single pseudo-merge bitmap contains this commit.\n> +\n> + *** If set (i.e. `offset & ((uint64_t)1<<63) != 0`), the offset\n> +\t(again relative to the beginning of the `.bitmap` file) at which\n> +\tthe extended offset table can be located describing the set of\n> +\tpseudo-merge bitmaps which contain this commit. This indicates\n> +\tthat multiple pseudo-merge bitmaps contain this commit.\n> +\n> +* An (optional) extended lookup table (written if and only if there is\n> + at least one commit which appears in more than one pseudo-merge).\n> + There are as many entries as commits which appear in multiple\n> + pseudo-merges. Each entry contains the following:\n> +\n> + ** `N`, a 4-byte unsigned value equal to the number of pseudo-merges\n> + which contain a given commit.\n> +\n> + ** An array of `N` 8-byte unsigned values, each of which is\n> + interpreted as an offset (relative to the beginning of the\n> + `.bitmap` file) at which a pseudo-merge bitmap for this commit can\n> + be read. These values occur in no particular order.\n> +\n> +* Positions for all pseudo-merges, each stored as an 8-byte unsigned\n> + value (in network byte-order) containing the offset (relative to the\n> + beginnign of the `.bitmap` file) of each consecutive pseudo-merge.\n> +\n> +* A 4-byte unsigned value (in network byte-order) equal to the number of\n> + pseudo-merges.\n> +\n> +* A 4-byte unsigned value (in network byte-order) equal to the number of\n> + unique commits which appear in any pseudo-merge.\n> +\n> +* An 8-byte unsigned value (in network byte-order) equal to the number\n> + of bytes between the start of the pseudo-merge section and the\n> + beginning of the lookup table.\n> +\n> +* An 8-byte unsigned value (in network byte-order) equal to the number\n> + of bytes in the pseudo-merge section (including this field).\n"},{"id":"491178","messageId":"ZfyxCLpjbaScIdWA@nand.local","threadId":"61166","inReplyTo":"xmqqcyrn6zyd.fsf@gitster.g","subject":"Re: [PATCH 01/24] Documentation/technical: describe pseudo-merge bitmaps format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-03-21T22:13:28Z","receivedAt":"2024-03-21T22:13:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Mar 21, 2024 at 02:24:10PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > +Pseudo-merge bitmaps can accelerate bitmap traversals when all commits\n> > +for a given pseudo-merge are listed on either side of the traversal,\n> > +either directly (by explicitly asking for them as part of the `HAVES`\n> > +or `WANTS`) or indirectly (by encountering them during a fill-in\n> > +traversal).\n>\n> \"either side of\" implies there are two sides. Is it correct to\n> understand that they are \"the side reachable from HAVE\" and \"the\n> other side that is reachable from WANT\"?\n\nYes, exactly.\n\n> > +=== Use-cases\n> > +\n> > +For example, suppose there exists a pseudo-merge bitmap with a large\n> > +number of commits, all of which are listed in the `WANTS` section of\n> > +some bitmap traversal query. When pseudo-merge bitmaps are enabled, the\n> > +bitmap machinery can quickly determine there is a pseudo-merge which\n> > +satisfies some subset of the wanted objects on either side of the query.\n>\n> Here you only talk about WANT but still mention \"either side of\".\n> How would the HAVE side of the query contribute to the computation?\n\nApologies for the confusion. In the first sentence, I'm talking about a\nspecific case where all parents of a pseudo-merge commit are listed\nin/reachable from the WANTS side of a traversal.\n\nThe second sentence describes the general case. The order should be\nswapped so that the second sentence comes first, and vice-versa for the\nsentence beginning with \"For example, [...]\".\n\n> > + ** `commits_bitmap`, an EWAH-compressed bitmap describing the set of\n> > + commits included in the this psuedo-merge.\n> > +\n> > + ** `merge_bitmap`, an EWAH-compressed bitmap describing the union of\n> > + the set of objects reachable from all commits listed in the\n> > + `commits_bitmap`.\n>\n> \"union of the set of objects reachable from all\", meaning if an\n> object is listed here, all commits in commits_bitmap are guaranteed\n> to reach that object? It sounds more like the intersction of sets\n> than union.\n\nOops, yes, I definitely meant intersection here. Thanks for a close\nread.\n\n> > +* A lookup table, mapping pseudo-merged commits to the pseudo-merges\n> > + they belong to. Entries appear in increasing order of each commit's\n> > + bit position. Each entry is 12 bytes wide, and is comprised of the\n> > + following:\n>\n> \"a pseudo-merged commit\" is a new term. It was explained what \"a\n> pseudo-merge bitmap\" is already, and it was explained that \"a\n> pseudo-merge bitmap\" consists of a pair of bitmaps (commit bitmap\n> that records which commit belongs to the \"pseudo-merge\", and merge\n> bitmap that records objects reachable from all commits in the commit\n> bitmap). But we haven't heard of \"a pseudo-merged commit\", or what\n> the verb \"to pseudo-merge a commit\" means.\n>\n> Does it merely mean \"a commit that is recorded in the commit-bitmap\n> half of a pseudo-merge bitmap\"? It still is unclear at this point\n> in the description if a commit can be part of only one such\n> commit-bitmap and makes readers reading hiccup, until a few\n> paragraphs below where extended table is there to help a commit\n> recorded in commit-bitmap of more than one pseudo-merge bitmaps.\n\nSorry for being unclear here. A pseudo-merge \"commit\" refers to a\nconceptual octopus-merge commit whose parents are the ones listed in the\n\"parents\" bitmap of the pseudo-merge bitmap, as defined. The \"objects\"\nbitmap is the set of objects reachable from that imaginary commit, or,\nequivalently, the intersection of objects reachable from the commits\nlisted in the parents bitmap.\n\nI'll make this more clear in the next version, thanks.\n\n> I'll stop here for now, but this made me even more convinced that\n> the presentation order needs to be rethought to sell why this whole\n> thing is a good idea by telling readers what problem it is solving,\n> why a new data structure helps and how, etc. Perhaps you can start\n> by trying to write a paragraph of description for the topic suitable\n> for the \"What's cooking\" report, which needs to do a good elevator\n> pitch.\n\nI'm sorry that the ordering was sub-optimal here. For the purposes of\nthe WC report, I would write the following if I were queuing this topic:\n\n * tb/pseudo-merge-bitmaps (2024-03-21) 24 commits\n - t/perf: implement performace tests for pseudo-merge bitmaps\n - pseudo-merge: implement support for finding existing merges\n - ewah: `bitmap_equals_ewah()`\n - pack-bitmap: extra trace2 information\n - pack-bitmap.c: use pseudo-merges during traversal\n - t/test-lib-functions.sh: support `--date` in `test_commit_bulk()`\n - pack-bitmap: implement test helpers for pseudo-merge\n - ewah: implement `ewah_bitmap_popcount()`\n - pseudo-merge: implement support for reading pseudo-merge commits\n - pack-bitmap.c: read pseudo-merge extension\n - pseudo-merge: scaffolding for reads\n - pack-bitmap: extract `read_bitmap()` function\n - pack-bitmap-write.c: write pseudo-merge table\n - pack-bitmap-write.c: select pseudo-merge commits\n - pseudo-merge: implement support for selecting pseudo-merge commits\n - pack-bitmap: make `bitmap_writer_push_bitmapped_commit()` public\n - pack-bitmap: implement `bitmap_writer_has_bitmapped_object_id()`\n - pack-bitmap-write: support storing pseudo-merge commits\n - pseudo-merge.ch: initial commit\n - pack-bitmap: move some initialization to `bitmap_writer_init()`\n - pack-bitmap: drop unused `max_bitmaps` parameter\n - ewah: implement `ewah_bitmap_is_subset()`\n - config: repo_config_get_expiry()\n - Documentation/technical: describe pseudo-merge bitmaps format\n\n The pack-bitmap machinery has been extended to write bitmaps for\n pseudo-merges, which are imaginary commits which act as octopus\n merges covering groups of the un-bitmapped parts of history at\n reference tips.\n\nThanks,\nTaylor\n"},{"id":"491179","messageId":"xmqq5xxf6x8q.fsf@gitster.g","threadId":"61166","inReplyTo":"ZfyxCLpjbaScIdWA@nand.local","subject":"Re: [PATCH 01/24] Documentation/technical: describe pseudo-merge bitmaps format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-03-21T22:22:45Z","receivedAt":"2024-03-21T22:22:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> The pack-bitmap machinery has been extended to write bitmaps for\n> pseudo-merges, which are imaginary commits which act as octopus\n> merges covering groups of the un-bitmapped parts of history at\n> reference tips.\n\nThat is \"what the topic does\", and does not cover \"why does it do\nso\" and/or \"for what effect\".\n\nI can sort-of see that allowing us to record a pre-combined bitmap\nfor octopus merges that do not exist, we have more flexibility\ncompared to the original bitmap machinery where we can only put\nbitmap to commits that exist. What is not clear is what this\nadditional flexibility is used for.\n\nDoes the approach takes advantage of that additional flexibility to\nreduce redundancy, allowing us to have the same bitmap coverage with\nsmaller number of bitmaps?\n\nThanks.\n"},{"id":"492729","messageId":"20240410175432.GA2260545@coredump.intra.peff.net","threadId":"61166","inReplyTo":"21d8f9dc2b4ddc8ac3f4e8f6b21bfb762fc6ab77.1710972293.git.me@ttaylorr.com","subject":"Re: [PATCH 02/24] config: repo_config_get_expiry()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-04-10T17:54:32Z","receivedAt":"2024-04-10T17:54:41Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 20, 2024 at 06:05:05PM -0400, Taylor Blau wrote:\n\n> Introduce a `repo_config_get_expiry()` variant in the style of functions\n> introduced by 3b256228a6 (config: read config from a repository object,\n> 2017-06-22) to read a single value without requiring the git_config()\n> callback-style approach.\n> \n> This new function is similar to the existing implementation in\n> `git_config_get_expiry()`, however it differs in that it fills out a\n> `timestamp_t` value through a pointer, instead of merely checking and\n> discarding the result (and returning it as a string).\n\nThe existing git_config_get_expiry() is a funny interface. That makes me\nwonder how its existing callers handle this issue. In most cases we are\njust grabbing values for git-gc to pass along as strings to\nsub-programs. The only other case resolves via approxidate immediately,\nin get_shared_index_expire_date(). Which sort of leaks the string,\nthough it is technically stuffed away in a global that we never look at.\n\nSo I can see the value of an interface which just returns the parsed\ntimestamp and handles the string itself. Weirdly we even have\ngit_config_get_expiry_in_days() which works like that, but always scales\nthe timestamp! So we could implement that in terms of this new function.\n\nBut...\n\n> +int repo_config_get_expiry(struct repository *repo,\n> +\t\t\t const char *key, const char **dest)\n> +{\n> +\tint ret;\n> +\n> +\tgit_config_check_init(repo);\n> +\n> +\tret = repo_config_get_string(repo, key, (char **)dest);\n> +\tif (ret)\n> +\t\treturn ret;\n> +\tif (strcmp(*dest, \"now\")) {\n> +\t\ttimestamp_t now = approxidate(\"now\");\n> +\t\tif (approxidate(*dest) >= now)\n> +\t\t\tgit_die_config(key, _(\"Invalid %s: '%s'\"), key, *dest);\n> +\t}\n> +\treturn ret;\n> +}\n\n...does this actually do what the commit message says? It looks\nidentical to git_config_get_expiry() except that it takes a repository\nparameter?\n\n> This function will gain its first caller in a subsequent commit to parse\n> a \"threshold\" parameter for excluding too-recent commits from\n> pseudo-merge groups.\n\nSo presumably you call approxidate() there in that new caller. Looks\nlike that would be patch 10. But I don't see a call to the new function\nat all! It just uses git_config_expiry_date(), which does what you need\n(it doesn't use the configset, but it looks like you ended up doing a\nconfig callback approach anyway).\n\nSo can this patch be dropped?\n\n-Peff\n"},{"id":"492730","messageId":"20240410180505.GB2260545@coredump.intra.peff.net","threadId":"61166","inReplyTo":"1347571ef4ca6329de58394bfea71927c8e08151.1710972293.git.me@ttaylorr.com","subject":"Re: [PATCH 03/24] ewah: implement `ewah_bitmap_is_subset()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-04-10T18:05:05Z","receivedAt":"2024-04-10T18:05:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 20, 2024 at 06:05:08PM -0400, Taylor Blau wrote:\n\n> In order to know whether a given pseudo-merge (comprised of a \"parents\"\n> and \"objects\" bitmaps) is \"satisfied\" and can be OR'd into the bitmap\n> result, we need to be able to quickly determine whether the \"parents\"\n> bitmap is a subset of the current set of objects reachable on either\n> side of a traversal.\n> \n> Implement a helper function to prepare for that, which determines\n> whether an EWAH bitmap (the parents bitmap from the pseudo-merge) is a\n> subset of a non-EWAH bitmap (in this case, the results bitmap from\n> either side of the traversal).\n> \n> This function makes use of the EWAH iterator to avoid inflating any part\n> of the EWAH bitmap after we determine it is not a subset of the non-EWAH\n> bitmap. This \"fail-fast\" allows us to avoid a potentially large amount\n> of wasted effort.\n\nMakes sense, as we have an expanded bitmap_is_subset() already, and this\nshould be more efficient.\n\n> diff --git a/ewah/bitmap.c b/ewah/bitmap.c\n> index ac7e0af622a..5bdae3fb07b 100644\n> --- a/ewah/bitmap.c\n> +++ b/ewah/bitmap.c\n> @@ -138,6 +138,49 @@ void bitmap_or(struct bitmap *self, const struct bitmap *other)\n> \t\tself->words[i] |= other->words[i];\n> }\n> \n> +int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other)\n\nIt wasn't immediately obvious to me if we were checking that \"other\" is\na subset of \"self\" or vice versa. I wonder if we could use different\nnames here to make it more clear (though really it matters more in the\ndeclaration, not the implementation).\n\nI think bitmap_is_subset() suffers from the same issue (and is even more\nconfusing because the two have the same type!). Maybe just a header file\ncomment would help?\n\n> +{\n> +\tstruct ewah_iterator it;\n> +\teword_t word;\n> +\tsize_t i;\n> +\n> +\tewah_iterator_init(&it, self);\n> +\n> +\tfor (i = 0; i < other->word_alloc; i++) {\n> +\t\tif (!ewah_iterator_next(&word, &it)) {\n> +\t\t\t/*\n> +\t\t\t * If we reached the end of `self`, and haven't\n> +\t\t\t * rejected `self` as a possible subset of\n> +\t\t\t * `other` yet, then we are done and `self` is\n> +\t\t\t * indeed a subset of `other`.\n> +\t\t\t */\n> +\t\t\treturn 1;\n> +\t\t}\n> +\t\tif (word & ~other->words[i]) {\n> +\t\t\t/*\n> +\t\t\t * Otherwise, compare the next two pairs of\n> +\t\t\t * words. If the word from `self` has bit(s) not\n> +\t\t\t * in the word from `other`, `self` is not a\n> +\t\t\t * proper subset of `other`.\n> +\t\t\t */\n> +\t\t\treturn 0;\n> +\t\t}\n> +\t}\n\nOK, so we expand the ewah as we go, comparing words, and then quit early\nif we can. That's the best we can do when comparing to a regular bitmap.\nI suspect we could do more clever things for ewah-to-ewah (like saying\n\"oh, they both have a run of 10,000 zeroes\" without expanding), but\nthat wouldn't be helpful here, as your use case will be comparing\nagainst a bitmap we're building in memory.\n\nI think your use of the phrase \"proper subset\" is a little confusing\nhere, as it is not a subset at all, let alone the distinction between a\nregular and proper one (in the mathematical definitions).\n\n> +\t/*\n> +\t * If we got to this point, there may be zero or more words\n> +\t * remaining in `self`, with no remaining words left in `other`.\n> +\t * If there are any bits set in the remaining word(s) in `self`,\n> +\t * then `self` is not a proper subset of `other`.\n> +\t */\n> +\twhile (ewah_iterator_next(&word, &it))\n> +\t\tif (word)\n> +\t\t\treturn 0;\n\nOK, so here we keep expanding to see if there are any bits set, meaning\nwe may read past a bunch of 0-words that we don't care about. I suspect\nthis could be optimized to just ask the ewah \"are there any bits left?\"\nbut I don't think we have an easy function for that. And it's not clear\nto me that it would produce measurable speedups anyway, so probably not\nworth worrying about.\n\nAs above, ditto on the use of \"proper subset\" here.\n\n-Peff\n"},{"id":"492731","messageId":"20240410180604.GC2260545@coredump.intra.peff.net","threadId":"61166","inReplyTo":"c6a08dae037351f0a5ba8f1044837c894c471b4f.1710972293.git.me@ttaylorr.com","subject":"Re: [PATCH 04/24] pack-bitmap: drop unused `max_bitmaps` parameter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-04-10T18:06:04Z","receivedAt":"2024-04-10T18:06:05Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 20, 2024 at 06:05:11PM -0400, Taylor Blau wrote:\n\n> The `max_bitmaps` parameter in `bitmap_writer_select_commits()` was\n> introduced back in 7cc8f97108 (pack-objects: implement bitmap writing,\n> 2013-12-21), making it original to the bitmap implementation in Git\n> itself.\n> \n> When that patch was merged via 0f9e62e084 (Merge branch\n> 'jk/pack-bitmap', 2014-02-27), its sole caller in builtin/pack-objects.c\n> passed a value of \"-1\" for `max_bitmaps`, indicating no limit.\n> \n> Since then, the only other caller (in midx.c, added via c528e17966\n> (pack-bitmap: write multi-pack bitmaps, 2021-08-31)) also uses a value\n> of \"-1\" for `max_bitmaps`.\n> \n> Since no callers have needed a finite limit for the `max_bitmaps`\n> parameter in the nearly decade that has passed since 0f9e62e084, let's\n> remove the parameter and any dead pieces of code connected to it.\n\nGreat, I'm happy to see dead code being cleaned up. And thanks (as\nalways) for digging into the history to explain how we got here.\n\n-Peff\n"},{"id":"492732","messageId":"20240410181020.GD2260545@coredump.intra.peff.net","threadId":"61166","inReplyTo":"a65316567391160dddae70fb036f03858772014f.1710972293.git.me@ttaylorr.com","subject":"Re: [PATCH 05/24] pack-bitmap: move some initialization to `bitmap_writer_init()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-04-10T18:10:20Z","receivedAt":"2024-04-10T18:10:21Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 20, 2024 at 06:05:13PM -0400, Taylor Blau wrote:\n\n> The pack-bitmap-writer machinery uses a oidmap (backed by khash.h) to\n> map from commits selected for bitmaps (by OID) to a bitmapped_commit\n> structure (containing the bitmap itself, among other things like its XOR\n> offset, etc.)\n> \n> This map was initialized at the end of `bitmap_writer_build()`. New\n> entries are added in `pack-bitmap-write.c::store_selected()`, which is\n> called by the bitmap_builder machinery (which is responsible for\n> traversing history and generating the actual bitmaps).\n> \n> Reorganize when this field is initialized and when entries are added to\n> it so that we can quickly determine whether a commit is a candidate for\n> pseudo-merge selection, or not (since it was already selected to receive\n> a bitmap, and thus is ineligible for pseudo-merge inclusion).\n\nOK, makes sense. I don't think this should violate any assumptions in\nthe current bitmap code (and the sanity checks for duplicate/missing\nentries in the hash seem right to me).\n\n-Peff\n"},{"id":"493679","messageId":"Zi/3jRyTQ44dCtaX@nand.local","threadId":"61166","inReplyTo":"20240410175432.GA2260545@coredump.intra.peff.net","subject":"Re: [PATCH 02/24] config: repo_config_get_expiry()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T19:39:57Z","receivedAt":"2024-04-29T19:40:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Apr 10, 2024 at 01:54:32PM -0400, Jeff King wrote:\n> > This function will gain its first caller in a subsequent commit to parse\n> > a \"threshold\" parameter for excluding too-recent commits from\n> > pseudo-merge groups.\n>\n> So presumably you call approxidate() there in that new caller. Looks\n> like that would be patch 10. But I don't see a call to the new function\n> at all! It just uses git_config_expiry_date(), which does what you need\n> (it doesn't use the configset, but it looks like you ended up doing a\n> config callback approach anyway).\n>\n> So can this patch be dropped?\n\nWow, yes, definitely -- this patch can be absolutely dropped.\n\nI suspect what happened here is that this patch is a relic from before I\nintroduced pseudo-merge.c::pseudo_merge_config(), which is a standard\ncallback for `git_config()`.\n\nI *think* the reason for the change is that I wanted to use the\nparse_config_key() function to parse different named pseudo-merge groups\nseparately. The idea of having multiple named pseudo-merge groups was\nintroduced after I wrote this patch, and I suspect that I never realized\nthat this patch could be dropped as a result.\n\nThanks for catching, I can cleanly just drop this patch from the series\nand everything works as expected.\n\nThanks,\nTaylor\n"},{"id":"493681","messageId":"Zi/5RWSFTFadna55@nand.local","threadId":"61166","inReplyTo":"20240410180505.GB2260545@coredump.intra.peff.net","subject":"Re: [PATCH 03/24] ewah: implement `ewah_bitmap_is_subset()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T19:47:17Z","receivedAt":"2024-04-29T19:47:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Apr 10, 2024 at 02:05:05PM -0400, Jeff King wrote:\n> > This function makes use of the EWAH iterator to avoid inflating any part\n> > of the EWAH bitmap after we determine it is not a subset of the non-EWAH\n> > bitmap. This \"fail-fast\" allows us to avoid a potentially large amount\n> > of wasted effort.\n>\n> Makes sense, as we have an expanded bitmap_is_subset() already, and this\n> should be more efficient.\n\nYep, the idea is that we already have a deflated non-EWAH bitmap in\nmemory, but we're comparing it against a potentially large EWAH\ncompressed bitmap.\n\nIf we can determine early that the EWAH bitmap has bits that are *not*\nin the non-EWAH bitmap, then we can avoid inflating a large part of the\nEWAH bitmap.\n\n> > diff --git a/ewah/bitmap.c b/ewah/bitmap.c\n> > index ac7e0af622a..5bdae3fb07b 100644\n> > --- a/ewah/bitmap.c\n> > +++ b/ewah/bitmap.c\n> > @@ -138,6 +138,49 @@ void bitmap_or(struct bitmap *self, const struct bitmap *other)\n> > \t\tself->words[i] |= other->words[i];\n> > }\n> >\n> > +int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other)\n>\n> It wasn't immediately obvious to me if we were checking that \"other\" is\n> a subset of \"self\" or vice versa. I wonder if we could use different\n> names here to make it more clear (though really it matters more in the\n> declaration, not the implementation).\n\nWe check whether 'self' is a subset of 'other', but I'll document it\nhere for both functions.\n\n> I think bitmap_is_subset() suffers from the same issue (and is even more\n> confusing because the two have the same type!). Maybe just a header file\n> comment would help?\n\nYeah, agreed.\n\n> I think your use of the phrase \"proper subset\" is a little confusing\n> here, as it is not a subset at all, let alone the distinction between a\n> regular and proper one (in the mathematical definitions).\n\nThanks for catching, this should definitely say just \"subset\"\n(indicating that 'self' and 'other' can have an identical set of bits in\neach and self is still considered a subset of other).\n\n> > +\t/*\n> > +\t * If we got to this point, there may be zero or more words\n> > +\t * remaining in `self`, with no remaining words left in `other`.\n> > +\t * If there are any bits set in the remaining word(s) in `self`,\n> > +\t * then `self` is not a proper subset of `other`.\n> > +\t */\n> > +\twhile (ewah_iterator_next(&word, &it))\n> > +\t\tif (word)\n> > +\t\t\treturn 0;\n>\n> OK, so here we keep expanding to see if there are any bits set, meaning\n> we may read past a bunch of 0-words that we don't care about. I suspect\n> this could be optimized to just ask the ewah \"are there any bits left?\"\n> but I don't think we have an easy function for that. And it's not clear\n> to me that it would produce measurable speedups anyway, so probably not\n> worth worrying about.\n\nYep.\n\n> As above, ditto on the use of \"proper subset\" here.\n\nThanks, fixed.\n\nThanks,\nTaylor\n"},{"id":"493683","messageId":"43fd5e3597151a86254e18e08ffd8cadbcb6e4f0.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 01/23] Documentation/technical: describe pseudo-merge bitmaps format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:42:54Z","receivedAt":"2024-04-29T20:42:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to implement pseudo-merge bitmaps over the next several commits\nby first describing the serialization format which will store the new\npseudo-merge bitmaps themselves.\n\nThis format is implemented as an optional extension within the bitmap v1\nformat, making it compatible with previous versions of Git, as well as\nthe original .bitmap implementation within JGit.\n\nThe format (as well as a general description of pseudo-merge bitmaps,\nand motivating use-case(s)) is described in detail in the patch contents\nbelow, but the high-level description is as follows:\n\n - An array of pseudo-merge bitmaps, each containing a pair of EWAH\n bitmaps: one describing the set of pseudo-merge \"parents\", and\n another describing the set of object(s) reachable from those\n parents.\n\n - A lookup table to determine which pseudo-merge(s) a given commit\n appears in. An optional extended lookup table follows when there is\n at least one commit which appears in multiple pseudo-merge groups.\n\n - Trailing metadata, including the number of pseudo-merge(s), number\n of unique parents, the offset within the .bitmap file for the\n pseudo-merge commit lookup table, and the size of the optional\n extension itself.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/technical/bitmap-format.txt | 179 ++++++++++++++++++++++\n 1 file changed, 179 insertions(+)\n\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex f5d200939b0..63a7177ac08 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -255,3 +255,182 @@ triplet is -\n \txor_row (4 byte integer, network byte order): ::\n \tThe position of the triplet whose bitmap is used to compress\n \tthis one, or `0xffffffff` if no such bitmap exists.\n+\n+Pseudo-merge bitmaps\n+--------------------\n+\n+If the `BITMAP_OPT_PSEUDO_MERGES` flag is set, a variable number of\n+bytes (preceding the name-hash cache, commit lookup table, and trailing\n+checksum) of the `.bitmap` file is used to store pseudo-merge bitmaps.\n+\n+A \"pseudo-merge bitmap\" is used to refer to a pair of bitmaps, as\n+follows:\n+\n+Commit bitmap::\n+\n+ A bitmap whose set bits describe the set of commits included in the\n+ pseudo-merge's \"merge\" bitmap (as below).\n+\n+Merge bitmap::\n+\n+ A bitmap whose set bits describe the reachability closure over the set\n+ of commits in the pseudo-merge's \"commits\" bitmap (as above). An\n+ identical bitmap would be generated for an octopus merge with the same\n+ set of parents as described in the commits bitmap.\n+\n+Pseudo-merge bitmaps can accelerate bitmap traversals when all commits\n+for a given pseudo-merge are listed on either side of the traversal,\n+either directly (by explicitly asking for them as part of the `HAVES`\n+or `WANTS`) or indirectly (by encountering them during a fill-in\n+traversal).\n+\n+=== Use-cases\n+\n+For example, suppose there exists a pseudo-merge bitmap with a large\n+number of commits, all of which are listed in the `WANTS` section of\n+some bitmap traversal query. When pseudo-merge bitmaps are enabled, the\n+bitmap machinery can quickly determine there is a pseudo-merge which\n+satisfies some subset of the wanted objects on either side of the query.\n+Then, we can inflate the EWAH-compressed bitmap, and `OR` it in to the\n+resulting bitmap. By contrast, without pseudo-merge bitmaps, we would\n+have to repeat the decompression and `OR`-ing step over a potentially\n+large number of individual bitmaps, which can take proportionally more\n+time.\n+\n+Another benefit of pseudo-merges arises when there is some combination\n+of (a) a large number of references, with (b) poor bitmap coverage, and\n+(c) deep, nested trees, making fill-in traversal relatively expensive.\n+For example, suppose that there are a large enough number of tags where\n+bitmapping each of the tags individually is infeasible. Without\n+pseudo-merge bitmaps, computing the result of, say, `git rev-list\n+--use-bitmap-index --count --objects --tags` would likely require a\n+large amount of fill-in traversal. But when a large quantity of those\n+tags are stored together in a pseudo-merge bitmap, the bitmap machinery\n+can take advantage of the fact that we only care about the union of\n+objects reachable from all of those tags, and answer the query much\n+faster.\n+\n+=== File format\n+\n+If enabled, pseudo-merge bitmaps are stored in an optional section at\n+the end of a `.bitmap` file. The format is as follows:\n+\n+....\n++-------------------------------------------+\n+| .bitmap File |\n++-------------------------------------------+\n+| |\n+| Pseudo-merge bitmaps (Variable Length) |\n+| +---------------------------+ |\n+| | commits_bitmap (EWAH) | |\n+| +---------------------------+ |\n+| | merge_bitmap (EWAH) | |\n+| +---------------------------+ |\n+| |\n++-------------------------------------------+\n+| |\n+| Lookup Table |\n+| +------------+--------------+ |\n+| | commit_pos | offset | |\n+| +------------+--------------+ |\n+| | 4 bytes | 8 bytes | |\n+| +------------+--------------+ |\n+| |\n+| Offset Cases: |\n+| ------------- |\n+| |\n+| 1. MSB Unset: single pseudo-merge bitmap |\n+| + offset to pseudo-merge bitmap |\n+| |\n+| 2. MSB Set: multiple pseudo-merges |\n+| + offset to extended lookup table |\n+| |\n++-------------------------------------------+\n+| |\n+| Extended Lookup Table (Optional) |\n+| |\n+| +----+----------+----------+----------+ |\n+| | N | Offset 1 | .... | Offset N | |\n+| +----+----------+----------+----------+ |\n+| | | 8 bytes | .... | 8 bytes | |\n+| +----+----------+----------+----------+ |\n+| |\n++-------------------------------------------+\n+| |\n+| Pseudo-merge Metadata |\n+| +------------------+----------------+ |\n+| | # pseudo-merges | # Commits | |\n+| +------------------+----------------+ |\n+| | 4 bytes | 4 bytes | |\n+| +------------------+----------------+ |\n+| |\n+| +------------------+----------------+ |\n+| | Lookup offset | Extension size | |\n+| +------------------+----------------+ |\n+| | 8 bytes | 8 bytes | |\n+| +------------------+----------------+ |\n+| |\n++-------------------------------------------+\n+....\n+\n+* One or more pseudo-merge bitmaps, each containing:\n+\n+ ** `commits_bitmap`, an EWAH-compressed bitmap describing the set of\n+ commits included in the this psuedo-merge.\n+\n+ ** `merge_bitmap`, an EWAH-compressed bitmap describing the union of\n+ the set of objects reachable from all commits listed in the\n+ `commits_bitmap`.\n+\n+* A lookup table, mapping pseudo-merged commits to the pseudo-merges\n+ they belong to. Entries appear in increasing order of each commit's\n+ bit position. Each entry is 12 bytes wide, and is comprised of the\n+ following:\n+\n+ ** `commit_pos`, a 4-byte unsigned value (in network byte-order)\n+ containing the bit position for this commit.\n+\n+ ** `offset`, an 8-byte unsigned value (also in network byte-order)\n+ containing either one of two possible offsets, depending on whether or\n+ not the most-significant bit is set.\n+\n+ *** If unset (i.e. `offset & ((uint64_t)1<<63) == 0`), the offset\n+\t(relative to the beginning of the `.bitmap` file) at which the\n+\tpseudo-merge bitmap for this commit can be read. This indicates\n+\tonly a single pseudo-merge bitmap contains this commit.\n+\n+ *** If set (i.e. `offset & ((uint64_t)1<<63) != 0`), the offset\n+\t(again relative to the beginning of the `.bitmap` file) at which\n+\tthe extended offset table can be located describing the set of\n+\tpseudo-merge bitmaps which contain this commit. This indicates\n+\tthat multiple pseudo-merge bitmaps contain this commit.\n+\n+* An (optional) extended lookup table (written if and only if there is\n+ at least one commit which appears in more than one pseudo-merge).\n+ There are as many entries as commits which appear in multiple\n+ pseudo-merges. Each entry contains the following:\n+\n+ ** `N`, a 4-byte unsigned value equal to the number of pseudo-merges\n+ which contain a given commit.\n+\n+ ** An array of `N` 8-byte unsigned values, each of which is\n+ interpreted as an offset (relative to the beginning of the\n+ `.bitmap` file) at which a pseudo-merge bitmap for this commit can\n+ be read. These values occur in no particular order.\n+\n+* Positions for all pseudo-merges, each stored as an 8-byte unsigned\n+ value (in network byte-order) containing the offset (relative to the\n+ beginnign of the `.bitmap` file) of each consecutive pseudo-merge.\n+\n+* A 4-byte unsigned value (in network byte-order) equal to the number of\n+ pseudo-merges.\n+\n+* A 4-byte unsigned value (in network byte-order) equal to the number of\n+ unique commits which appear in any pseudo-merge.\n+\n+* An 8-byte unsigned value (in network byte-order) equal to the number\n+ of bytes between the start of the pseudo-merge section and the\n+ beginning of the lookup table.\n+\n+* An 8-byte unsigned value (in network byte-order) equal to the number\n+ of bytes in the pseudo-merge section (including this field).\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493684","messageId":"cover.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH v2 00/23] pack-bitmap: pseudo-merge reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:42:58Z","receivedAt":"2024-04-29T20:43:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is a reroll my topic to introduce pseudo-merge bitmaps. Much is\nunchanged since last time, but notable changes (in response to Peff's\nreview of the first five or so patches of this topic) include:\n\n - Rebased onto 2.45, so this is now based on 'master', which is at\n 786a3e4b8d (Git 2.45, 2024-04-29) at the time of writing.\n\n - Dropped patch 2/24 from the first round as it is no longer\n necessary.\n\n - Introduced some documentation and fixed a couple of comments\n around ewah_bitmap_is_subset() and bitmap_is_subset() to clarify\n which argument is supposed to be a subset of the other.\n\nOtherwise, a range-diff is included below for convenience. Thanks in\nadvance for your review!\n\nTaylor Blau (23):\n Documentation/technical: describe pseudo-merge bitmaps format\n ewah: implement `ewah_bitmap_is_subset()`\n pack-bitmap: drop unused `max_bitmaps` parameter\n pack-bitmap: move some initialization to `bitmap_writer_init()`\n pseudo-merge.ch: initial commit\n pack-bitmap-write: support storing pseudo-merge commits\n pack-bitmap: implement `bitmap_writer_has_bitmapped_object_id()`\n pack-bitmap: make `bitmap_writer_push_bitmapped_commit()` public\n pseudo-merge: implement support for selecting pseudo-merge commits\n pack-bitmap-write.c: select pseudo-merge commits\n pack-bitmap-write.c: write pseudo-merge table\n pack-bitmap: extract `read_bitmap()` function\n pseudo-merge: scaffolding for reads\n pack-bitmap.c: read pseudo-merge extension\n pseudo-merge: implement support for reading pseudo-merge commits\n ewah: implement `ewah_bitmap_popcount()`\n pack-bitmap: implement test helpers for pseudo-merge\n t/test-lib-functions.sh: support `--date` in `test_commit_bulk()`\n pack-bitmap.c: use pseudo-merges during traversal\n pack-bitmap: extra trace2 information\n ewah: `bitmap_equals_ewah()`\n pseudo-merge: implement support for finding existing merges\n t/perf: implement performace tests for pseudo-merge bitmaps\n\n Documentation/config.txt | 2 +\n Documentation/config/bitmap-pseudo-merge.txt | 75 ++\n Documentation/technical/bitmap-format.txt | 205 +++++\n Makefile | 1 +\n builtin/pack-objects.c | 3 +-\n ewah/bitmap.c | 76 ++\n ewah/ewok.h | 8 +\n midx-write.c | 3 +-\n pack-bitmap-write.c | 275 ++++++-\n pack-bitmap.c | 359 ++++++++-\n pack-bitmap.h | 16 +-\n pseudo-merge.c | 739 +++++++++++++++++++\n pseudo-merge.h | 218 ++++++\n t/helper/test-bitmap.c | 34 +-\n t/perf/p5333-pseudo-merge-bitmaps.sh | 32 +\n t/t5333-pseudo-merge-bitmaps.sh | 389 ++++++++++\n t/test-lib-functions.sh | 12 +-\n 17 files changed, 2386 insertions(+), 61 deletions(-)\n create mode 100644 Documentation/config/bitmap-pseudo-merge.txt\n create mode 100644 pseudo-merge.c\n create mode 100644 pseudo-merge.h\n create mode 100755 t/perf/p5333-pseudo-merge-bitmaps.sh\n create mode 100755 t/t5333-pseudo-merge-bitmaps.sh\n\nRange-diff against v1:\n 1: 76e7e3b9cca = 1: 43fd5e35971 Documentation/technical: describe pseudo-merge bitmaps format\n 2: 21d8f9dc2b4 < -: ----------- config: repo_config_get_expiry()\n 3: 1347571ef4c ! 2: 290d928325d ewah: implement `ewah_bitmap_is_subset()`\n @@ ewah/bitmap.c: void bitmap_or(struct bitmap *self, const struct bitmap *other)\n +\t\t\t * Otherwise, compare the next two pairs of\n +\t\t\t * words. If the word from `self` has bit(s) not\n +\t\t\t * in the word from `other`, `self` is not a\n -+\t\t\t * proper subset of `other`.\n ++\t\t\t * subset of `other`.\n +\t\t\t */\n +\t\t\treturn 0;\n +\t\t}\n @@ ewah/bitmap.c: void bitmap_or(struct bitmap *self, const struct bitmap *other)\n +\t * If we got to this point, there may be zero or more words\n +\t * remaining in `self`, with no remaining words left in `other`.\n +\t * If there are any bits set in the remaining word(s) in `self`,\n -+\t * then `self` is not a proper subset of `other`.\n ++\t * then `self` is not a subset of `other`.\n +\t */\n +\twhile (ewah_iterator_next(&word, &it))\n +\t\tif (word)\n @@ ewah/bitmap.c: void bitmap_or(struct bitmap *self, const struct bitmap *other)\n \tsize_t original_size = self->word_alloc;\n \n ## ewah/ewok.h ##\n -@@ ewah/ewok.h: int bitmap_get(struct bitmap *self, size_t pos);\n +@@ ewah/ewok.h: void bitmap_unset(struct bitmap *self, size_t pos);\n + int bitmap_get(struct bitmap *self, size_t pos);\n void bitmap_free(struct bitmap *self);\n int bitmap_equals(struct bitmap *self, struct bitmap *other);\n ++\n ++/*\n ++ * Both `bitmap_is_subset()` and `ewah_bitmap_is_subset()` return 1 if the set\n ++ * of bits in 'self' are a subset of the bits in 'other'. Returns 0 otherwise.\n ++ */\n int bitmap_is_subset(struct bitmap *self, struct bitmap *other);\n +int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other);\n \n 4: c6a08dae037 ! 3: 5160859f7f3 pack-bitmap: drop unused `max_bitmaps` parameter\n @@ builtin/pack-objects.c: static void write_pack_file(void)\n \t\t\t\t\tdie(_(\"failed to write bitmap index\"));\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\n \n - ## midx.c ##\n -@@ midx.c: static int write_midx_bitmap(const char *midx_name,\n + ## midx-write.c ##\n +@@ midx-write.c: static int write_midx_bitmap(const char *midx_name,\n \tfor (i = 0; i < pdata->nr_objects; i++)\n \t\tindex[pack_order[i]] = &pdata->objects[i].idx;\n \n 5: a6531656739 ! 4: 3d7d930b1c5 pack-bitmap: move some initialization to `bitmap_writer_init()`\n @@ builtin/pack-objects.c: static void write_pack_file(void)\n \t\t\t\tbitmap_writer_build_type_index(\n \t\t\t\t\t&to_pack, written_list, nr_written);\n \n - ## midx.c ##\n -@@ midx.c: static int write_midx_bitmap(const char *midx_name,\n + ## midx-write.c ##\n +@@ midx-write.c: static int write_midx_bitmap(const char *midx_name,\n \tfor (i = 0; i < pdata->nr_objects; i++)\n \t\tindex[i] = &pdata->objects[i].idx;\n \n 6: c6f9170af0f = 5: e7a87cf7d4e pseudo-merge.ch: initial commit\n 7: 7acdee2b5f2 = 6: ee33a703245 pack-bitmap-write: support storing pseudo-merge commits\n 8: 4fdd7dda274 = 7: 9c6d09bf874 pack-bitmap: implement `bitmap_writer_has_bitmapped_object_id()`\n 9: d74cf3e484d = 8: dfd4b73d12e pack-bitmap: make `bitmap_writer_push_bitmapped_commit()` public\n10: 323e1250b24 = 9: 86a1e4b8b9b pseudo-merge: implement support for selecting pseudo-merge commits\n11: bf6b0d8601e = 10: 12b432e3a8a pack-bitmap-write.c: select pseudo-merge commits\n12: 4c594f3faa8 = 11: 6ce805d061e pack-bitmap-write.c: write pseudo-merge table\n13: 7a31a932ab3 = 12: 60f6b310213 pack-bitmap: extract `read_bitmap()` function\n14: 7e4d051f37a = 13: 9465313691b pseudo-merge: scaffolding for reads\n15: 7bb644b2b0c = 14: 5894f3d5369 pack-bitmap.c: read pseudo-merge extension\n16: 792cc863154 = 15: 7dbee8bcbdf pseudo-merge: implement support for reading pseudo-merge commits\n17: 8fb7f7ab37b = 16: 09650aa53e9 ewah: implement `ewah_bitmap_popcount()`\n18: c839e1fed15 = 17: 7b5ea56d053 pack-bitmap: implement test helpers for pseudo-merge\n19: 7d3b88e6fd6 = 18: 006abdd1698 t/test-lib-functions.sh: support `--date` in `test_commit_bulk()`\n20: c18694ade2a = 19: 3f85e5b90f5 pack-bitmap.c: use pseudo-merges during traversal\n21: d38ebeba419 = 20: 5fac186df64 pack-bitmap: extra trace2 information\n22: 1eb10c190ba ! 21: b5aea8e57f8 ewah: `bitmap_equals_ewah()`\n @@ ewah/ewok.h: void bitmap_unset(struct bitmap *self, size_t pos);\n void bitmap_free(struct bitmap *self);\n int bitmap_equals(struct bitmap *self, struct bitmap *other);\n +int bitmap_equals_ewah(struct bitmap *self, struct ewah_bitmap *other);\n - int bitmap_is_subset(struct bitmap *self, struct bitmap *other);\n - int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other);\n \n + /*\n + * Both `bitmap_is_subset()` and `ewah_bitmap_is_subset()` return 1 if the set\n23: 4ae4f0eaae5 = 22: 61ddb574285 pseudo-merge: implement support for finding existing merges\n24: a05ad42202d = 23: 2bd830d35dd t/perf: implement performace tests for pseudo-merge bitmaps\n\nbase-commit: 786a3e4b8d754d2b14b1208b98eeb0a554ef19a8\n-- \n2.45.0.23.gc6f94b99219\n"},{"id":"493685","messageId":"290d928325dedd89d8e95aa12e643434b0dd2501.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 02/23] ewah: implement `ewah_bitmap_is_subset()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:43:01Z","receivedAt":"2024-04-29T20:43:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In order to know whether a given pseudo-merge (comprised of a \"parents\"\nand \"objects\" bitmaps) is \"satisfied\" and can be OR'd into the bitmap\nresult, we need to be able to quickly determine whether the \"parents\"\nbitmap is a subset of the current set of objects reachable on either\nside of a traversal.\n\nImplement a helper function to prepare for that, which determines\nwhether an EWAH bitmap (the parents bitmap from the pseudo-merge) is a\nsubset of a non-EWAH bitmap (in this case, the results bitmap from\neither side of the traversal).\n\nThis function makes use of the EWAH iterator to avoid inflating any part\nof the EWAH bitmap after we determine it is not a subset of the non-EWAH\nbitmap. This \"fail-fast\" allows us to avoid a potentially large amount\nof wasted effort.\n\nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 43 +++++++++++++++++++++++++++++++++++++++++++\n ewah/ewok.h | 6 ++++++\n 2 files changed, 49 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex ac7e0af622a..d352fec54ce 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -138,6 +138,49 @@ void bitmap_or(struct bitmap *self, const struct bitmap *other)\n \t\tself->words[i] |= other->words[i];\n }\n \n+int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other)\n+{\n+\tstruct ewah_iterator it;\n+\teword_t word;\n+\tsize_t i;\n+\n+\tewah_iterator_init(&it, self);\n+\n+\tfor (i = 0; i < other->word_alloc; i++) {\n+\t\tif (!ewah_iterator_next(&word, &it)) {\n+\t\t\t/*\n+\t\t\t * If we reached the end of `self`, and haven't\n+\t\t\t * rejected `self` as a possible subset of\n+\t\t\t * `other` yet, then we are done and `self` is\n+\t\t\t * indeed a subset of `other`.\n+\t\t\t */\n+\t\t\treturn 1;\n+\t\t}\n+\t\tif (word & ~other->words[i]) {\n+\t\t\t/*\n+\t\t\t * Otherwise, compare the next two pairs of\n+\t\t\t * words. If the word from `self` has bit(s) not\n+\t\t\t * in the word from `other`, `self` is not a\n+\t\t\t * subset of `other`.\n+\t\t\t */\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * If we got to this point, there may be zero or more words\n+\t * remaining in `self`, with no remaining words left in `other`.\n+\t * If there are any bits set in the remaining word(s) in `self`,\n+\t * then `self` is not a subset of `other`.\n+\t */\n+\twhile (ewah_iterator_next(&word, &it))\n+\t\tif (word)\n+\t\t\treturn 0;\n+\n+\t/* `self` is definitely a subset of `other` */\n+\treturn 1;\n+}\n+\n void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other)\n {\n \tsize_t original_size = self->word_alloc;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex c11d76c6f33..2b6c4ac499c 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -179,7 +179,13 @@ void bitmap_unset(struct bitmap *self, size_t pos);\n int bitmap_get(struct bitmap *self, size_t pos);\n void bitmap_free(struct bitmap *self);\n int bitmap_equals(struct bitmap *self, struct bitmap *other);\n+\n+/*\n+ * Both `bitmap_is_subset()` and `ewah_bitmap_is_subset()` return 1 if the set\n+ * of bits in 'self' are a subset of the bits in 'other'. Returns 0 otherwise.\n+ */\n int bitmap_is_subset(struct bitmap *self, struct bitmap *other);\n+int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other);\n \n struct ewah_bitmap * bitmap_to_ewah(struct bitmap *bitmap);\n struct bitmap *ewah_to_bitmap(struct ewah_bitmap *ewah);\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493686","messageId":"5160859f7f3cf72de03a4644ee3d3743eaba2bc2.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 03/23] pack-bitmap: drop unused `max_bitmaps` parameter","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:43:05Z","receivedAt":"2024-04-29T20:43:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `max_bitmaps` parameter in `bitmap_writer_select_commits()` was\nintroduced back in 7cc8f97108 (pack-objects: implement bitmap writing,\n2013-12-21), making it original to the bitmap implementation in Git\nitself.\n\nWhen that patch was merged via 0f9e62e084 (Merge branch\n'jk/pack-bitmap', 2014-02-27), its sole caller in builtin/pack-objects.c\npassed a value of \"-1\" for `max_bitmaps`, indicating no limit.\n\nSince then, the only other caller (in midx.c, added via c528e17966\n(pack-bitmap: write multi-pack bitmaps, 2021-08-31)) also uses a value\nof \"-1\" for `max_bitmaps`.\n\nSince no callers have needed a finite limit for the `max_bitmaps`\nparameter in the nearly decade that has passed since 0f9e62e084, let's\nremove the parameter and any dead pieces of code connected to it.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 2 +-\n midx-write.c | 2 +-\n pack-bitmap-write.c | 8 +-------\n pack-bitmap.h | 2 +-\n 4 files changed, 4 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex baf0090fc8d..5060ce2dfba 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1359,7 +1359,7 @@ static void write_pack_file(void)\n \t\t\t\tstop_progress(&progress_state);\n \n \t\t\t\tbitmap_writer_show_progress(progress);\n-\t\t\t\tbitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);\n+\t\t\t\tbitmap_writer_select_commits(indexed_commits, indexed_commits_nr);\n \t\t\t\tif (bitmap_writer_build(&to_pack) < 0)\n \t\t\t\t\tdie(_(\"failed to write bitmap index\"));\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\ndiff --git a/midx-write.c b/midx-write.c\nindex 65e69d2de78..469cceaa583 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -838,7 +838,7 @@ static int write_midx_bitmap(const char *midx_name,\n \tfor (i = 0; i < pdata->nr_objects; i++)\n \t\tindex[pack_order[i]] = &pdata->objects[i].idx;\n \n-\tbitmap_writer_select_commits(commits, commits_nr, -1);\n+\tbitmap_writer_select_commits(commits, commits_nr);\n \tret = bitmap_writer_build(pdata);\n \tif (ret < 0)\n \t\tgoto cleanup;\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex c6c8f94cc51..c35bc81d00f 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -591,8 +591,7 @@ static int date_compare(const void *_a, const void *_b)\n }\n \n void bitmap_writer_select_commits(struct commit **indexed_commits,\n-\t\t\t\t unsigned int indexed_commits_nr,\n-\t\t\t\t int max_bitmaps)\n+\t\t\t\t unsigned int indexed_commits_nr)\n {\n \tunsigned int i = 0, j, next;\n \n@@ -615,11 +614,6 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\tif (i + next >= indexed_commits_nr)\n \t\t\tbreak;\n \n-\t\tif (max_bitmaps > 0 && writer.selected_nr >= max_bitmaps) {\n-\t\t\twriter.selected_nr = max_bitmaps;\n-\t\t\tbreak;\n-\t\t}\n-\n \t\tif (next == 0) {\n \t\t\tchosen = indexed_commits[i];\n \t\t} else {\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex c7dea13217a..3f96608d5c1 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -110,7 +110,7 @@ int rebuild_bitmap(const uint32_t *reposition,\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t struct commit *commit);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n-\t\tunsigned int indexed_commits_nr, int max_bitmaps);\n+\t\t\t\t unsigned int indexed_commits_nr);\n int bitmap_writer_build(struct packing_data *to_pack);\n void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\t\t uint32_t index_nr,\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493687","messageId":"3d7d930b1c5c4d122d8731ef0dc3fc90115573a2.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 04/23] pack-bitmap: move some initialization to `bitmap_writer_init()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:43:08Z","receivedAt":"2024-04-29T20:43:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pack-bitmap-writer machinery uses a oidmap (backed by khash.h) to\nmap from commits selected for bitmaps (by OID) to a bitmapped_commit\nstructure (containing the bitmap itself, among other things like its XOR\noffset, etc.)\n\nThis map was initialized at the end of `bitmap_writer_build()`. New\nentries are added in `pack-bitmap-write.c::store_selected()`, which is\ncalled by the bitmap_builder machinery (which is responsible for\ntraversing history and generating the actual bitmaps).\n\nReorganize when this field is initialized and when entries are added to\nit so that we can quickly determine whether a commit is a candidate for\npseudo-merge selection, or not (since it was already selected to receive\na bitmap, and thus is ineligible for pseudo-merge inclusion).\n\nThe changes are as follows:\n\n - Introduce a new `bitmap_writer_init()` function which initializes\n the `writer.bitmaps` field (instead of waiting until the end of\n `bitmap_writer_build()`).\n\n - Add map entries in `push_bitmapped_commit()` (which is called via\n `bitmap_writer_select_commits()`) with OID keys and NULL values to\n track whether or not we *expect* to write a bitmap for some given\n commit.\n\n - Validate that a NULL entry is found matching the given key when we\n store a selected bitmap.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 1 +\n midx-write.c | 1 +\n pack-bitmap-write.c | 23 ++++++++++++++++++-----\n pack-bitmap.h | 1 +\n 4 files changed, 21 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5060ce2dfba..2958cdda499 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1339,6 +1339,7 @@ static void write_pack_file(void)\n \t\t\t\t hash_to_hex(hash));\n \n \t\t\tif (write_bitmap_index) {\n+\t\t\t\tbitmap_writer_init(the_repository);\n \t\t\t\tbitmap_writer_set_checksum(hash);\n \t\t\t\tbitmap_writer_build_type_index(\n \t\t\t\t\t&to_pack, written_list, nr_written);\ndiff --git a/midx-write.c b/midx-write.c\nindex 469cceaa583..ed5f8b72b9c 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -819,6 +819,7 @@ static int write_midx_bitmap(const char *midx_name,\n \tfor (i = 0; i < pdata->nr_objects; i++)\n \t\tindex[i] = &pdata->objects[i].idx;\n \n+\tbitmap_writer_init(the_repository);\n \tbitmap_writer_show_progress(flags & MIDX_PROGRESS);\n \tbitmap_writer_build_type_index(pdata, index, pdata->nr_objects);\n \ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex c35bc81d00f..9bc41a9e145 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -46,6 +46,11 @@ struct bitmap_writer {\n \n static struct bitmap_writer writer;\n \n+void bitmap_writer_init(struct repository *r)\n+{\n+\twriter.bitmaps = kh_init_oid_map();\n+}\n+\n void bitmap_writer_show_progress(int show)\n {\n \twriter.show_progress = show;\n@@ -117,11 +122,20 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n \n static inline void push_bitmapped_commit(struct commit *commit)\n {\n+\tint hash_ret;\n+\tkhiter_t hash_pos;\n+\n \tif (writer.selected_nr >= writer.selected_alloc) {\n \t\twriter.selected_alloc = (writer.selected_alloc + 32) * 2;\n \t\tREALLOC_ARRAY(writer.selected, writer.selected_alloc);\n \t}\n \n+\thash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);\n+\tif (!hash_ret)\n+\t\tdie(_(\"duplicate entry when writing bitmap index: %s\"),\n+\t\t oid_to_hex(&commit->object.oid));\n+\tkh_value(writer.bitmaps, hash_pos) = NULL;\n+\n \twriter.selected[writer.selected_nr].commit = commit;\n \twriter.selected[writer.selected_nr].bitmap = NULL;\n \twriter.selected[writer.selected_nr].flags = 0;\n@@ -466,14 +480,14 @@ static void store_selected(struct bb_commit *ent, struct commit *commit)\n {\n \tstruct bitmapped_commit *stored = &writer.selected[ent->idx];\n \tkhiter_t hash_pos;\n-\tint hash_ret;\n \n \tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n \n-\thash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);\n-\tif (hash_ret == 0)\n-\t\tdie(\"Duplicate entry when writing index: %s\",\n+\thash_pos = kh_get_oid_map(writer.bitmaps, commit->object.oid);\n+\tif (hash_pos == kh_end(writer.bitmaps))\n+\t\tdie(_(\"attempted to store non-selected commit: '%s'\"),\n \t\t oid_to_hex(&commit->object.oid));\n+\n \tkh_value(writer.bitmaps, hash_pos) = stored;\n }\n \n@@ -488,7 +502,6 @@ int bitmap_writer_build(struct packing_data *to_pack)\n \tuint32_t *mapping;\n \tint closed = 1; /* until proven otherwise */\n \n-\twriter.bitmaps = kh_init_oid_map();\n \twriter.to_pack = to_pack;\n \n \tif (writer.show_progress)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3f96608d5c1..dae2d68a338 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -97,6 +97,7 @@ int bitmap_has_oid_in_uninteresting(struct bitmap_index *, const struct object_i\n \n off_t get_disk_usage_from_bitmap(struct bitmap_index *, struct rev_info *);\n \n+void bitmap_writer_init(struct repository *r);\n void bitmap_writer_show_progress(int show);\n void bitmap_writer_set_checksum(const unsigned char *sha1);\n void bitmap_writer_build_type_index(struct packing_data *to_pack,\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493688","messageId":"e7a87cf7d4ea0249547cb170f2a4b14a80187ed7.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 05/23] pseudo-merge.ch: initial commit","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:43:11Z","receivedAt":"2024-04-29T20:43:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Add a new (empty) header file to contain the implementation for\nselecting, reading, and applying pseudo-merge bitmaps.\n\nFor now this header and its corresponding implementation are left\nempty, but they will evolve over the course of subsequent commit(s).\n\nSigned-off-by: Taylor Blau \n---\n Makefile | 1 +\n pseudo-merge.c | 2 ++\n pseudo-merge.h | 6 ++++++\n 3 files changed, 9 insertions(+)\n create mode 100644 pseudo-merge.c\n create mode 100644 pseudo-merge.h\n\ndiff --git a/Makefile b/Makefile\nindex 1e31acc72ec..6a3d164fdf8 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1119,6 +1119,7 @@ LIB_OBJS += prompt.o\n LIB_OBJS += protocol.o\n LIB_OBJS += protocol-caps.o\n LIB_OBJS += prune-packed.o\n+LIB_OBJS += pseudo-merge.o\n LIB_OBJS += quote.o\n LIB_OBJS += range-diff.o\n LIB_OBJS += reachable.o\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nnew file mode 100644\nindex 00000000000..37e037ba272\n--- /dev/null\n+++ b/pseudo-merge.c\n@@ -0,0 +1,2 @@\n+#include \"git-compat-util.h\"\n+#include \"pseudo-merge.h\"\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nnew file mode 100644\nindex 00000000000..cab8ff6960a\n--- /dev/null\n+++ b/pseudo-merge.h\n@@ -0,0 +1,6 @@\n+#ifndef PSEUDO_MERGE_H\n+#define PSEUDO_MERGE_H\n+\n+#include \"git-compat-util.h\"\n+\n+#endif\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493689","messageId":"ee33a70324589a98c2239530b03cc2d7afbdfb9e.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 06/23] pack-bitmap-write: support storing pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:43:15Z","receivedAt":"2024-04-29T20:43:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to write pseudo-merge bitmaps by annotating individual bitmapped\ncommits (which are represented by the `bitmapped_commit` structure) with\nan extra bit indicating whether or not they are a pseudo-merge.\n\nIn subsequent commits, pseudo-merge bitmaps will be generated by\nallocating a fake commit node with parents covering the full set of\ncommits represented by the pseudo-merge bitmap. These commits will be\nadded to the set of \"selected\" commits as usual, but will be written\nspecially instead of being included with the rest of the selected\ncommits.\n\nMechanically speaking, there are two parts of this change:\n\n - The bitmapped_commit struct gets a new bit indicating whether it is\n a pseudo-merge, or an ordinary commit selected for bitmaps.\n\n - A handful of changes to only write out the non-pseudo-merge commits\n when enumerating through the selected array (see the new\n `bitmap_writer_selected_nr()` function). Pseudo-merge commits appear\n after all non-pseudo-merge commits, so it is safe to enumerate\n through the selected array like so:\n\n for (i = 0; i < bitmap_writer_selected_nr(); i++)\n if (writer.selected[i].pseudo_merge)\n BUG(\"unexpected pseudo-merge\");\n\n without encountering the BUG().\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 100 +++++++++++++++++++++++++++++---------------\n pack-bitmap.h | 1 +\n 2 files changed, 67 insertions(+), 34 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 9bc41a9e145..fef02cd745a 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -24,7 +24,7 @@ struct bitmapped_commit {\n \tstruct ewah_bitmap *write_as;\n \tint flags;\n \tint xor_offset;\n-\tuint32_t commit_pos;\n+\tunsigned pseudo_merge : 1;\n };\n \n struct bitmap_writer {\n@@ -39,6 +39,8 @@ struct bitmap_writer {\n \tstruct bitmapped_commit *selected;\n \tunsigned int selected_nr, selected_alloc;\n \n+\tuint32_t pseudo_merges_nr;\n+\n \tstruct progress *progress;\n \tint show_progress;\n \tunsigned char pack_checksum[GIT_MAX_RAWSZ];\n@@ -46,6 +48,11 @@ struct bitmap_writer {\n \n static struct bitmap_writer writer;\n \n+static inline int bitmap_writer_selected_nr(void)\n+{\n+\treturn writer.selected_nr - writer.pseudo_merges_nr;\n+}\n+\n void bitmap_writer_init(struct repository *r)\n {\n \twriter.bitmaps = kh_init_oid_map();\n@@ -120,25 +127,30 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n * Compute the actual bitmaps\n */\n \n-static inline void push_bitmapped_commit(struct commit *commit)\n+static void bitmap_writer_push_bitmapped_commit(struct commit *commit,\n+\t\t\t\t\t\tunsigned pseudo_merge)\n {\n-\tint hash_ret;\n-\tkhiter_t hash_pos;\n-\n \tif (writer.selected_nr >= writer.selected_alloc) {\n \t\twriter.selected_alloc = (writer.selected_alloc + 32) * 2;\n \t\tREALLOC_ARRAY(writer.selected, writer.selected_alloc);\n \t}\n \n-\thash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);\n-\tif (!hash_ret)\n-\t\tdie(_(\"duplicate entry when writing bitmap index: %s\"),\n-\t\t oid_to_hex(&commit->object.oid));\n-\tkh_value(writer.bitmaps, hash_pos) = NULL;\n+\tif (!pseudo_merge) {\n+\t\tint hash_ret;\n+\t\tkhiter_t hash_pos = kh_put_oid_map(writer.bitmaps,\n+\t\t\t\t\t\t commit->object.oid,\n+\t\t\t\t\t\t &hash_ret);\n+\n+\t\tif (!hash_ret)\n+\t\t\tdie(_(\"duplicate entry when writing bitmap index: %s\"),\n+\t\t\t oid_to_hex(&commit->object.oid));\n+\t\tkh_value(writer.bitmaps, hash_pos) = NULL;\n+\t}\n \n \twriter.selected[writer.selected_nr].commit = commit;\n \twriter.selected[writer.selected_nr].bitmap = NULL;\n \twriter.selected[writer.selected_nr].flags = 0;\n+\twriter.selected[writer.selected_nr].pseudo_merge = pseudo_merge;\n \n \twriter.selected_nr++;\n }\n@@ -168,16 +180,20 @@ static void compute_xor_offsets(void)\n \n \twhile (next < writer.selected_nr) {\n \t\tstruct bitmapped_commit *stored = &writer.selected[next];\n-\n \t\tint best_offset = 0;\n \t\tstruct ewah_bitmap *best_bitmap = stored->bitmap;\n \t\tstruct ewah_bitmap *test_xor;\n \n+\t\tif (stored->pseudo_merge)\n+\t\t\tgoto next;\n+\n \t\tfor (i = 1; i <= MAX_XOR_OFFSET_SEARCH; ++i) {\n \t\t\tint curr = next - i;\n \n \t\t\tif (curr < 0)\n \t\t\t\tbreak;\n+\t\t\tif (writer.selected[curr].pseudo_merge)\n+\t\t\t\tcontinue;\n \n \t\t\ttest_xor = ewah_pool_new();\n \t\t\tewah_xor(writer.selected[curr].bitmap, stored->bitmap, test_xor);\n@@ -193,6 +209,7 @@ static void compute_xor_offsets(void)\n \t\t\t}\n \t\t}\n \n+next:\n \t\tstored->xor_offset = best_offset;\n \t\tstored->write_as = best_bitmap;\n \n@@ -205,7 +222,8 @@ struct bb_commit {\n \tstruct bitmap *commit_mask;\n \tstruct bitmap *bitmap;\n \tunsigned selected:1,\n-\t\t maximal:1;\n+\t\t maximal:1,\n+\t\t pseudo_merge:1;\n \tunsigned idx; /* within selected array */\n };\n \n@@ -243,17 +261,18 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \trevs.first_parent_only = 1;\n \n \tfor (i = 0; i < writer->selected_nr; i++) {\n-\t\tstruct commit *c = writer->selected[i].commit;\n-\t\tstruct bb_commit *ent = bb_data_at(&bb->data, c);\n+\t\tstruct bitmapped_commit *bc = &writer->selected[i];\n+\t\tstruct bb_commit *ent = bb_data_at(&bb->data, bc->commit);\n \n \t\tent->selected = 1;\n \t\tent->maximal = 1;\n+\t\tent->pseudo_merge = bc->pseudo_merge;\n \t\tent->idx = i;\n \n \t\tent->commit_mask = bitmap_new();\n \t\tbitmap_set(ent->commit_mask, i);\n \n-\t\tadd_pending_object(&revs, &c->object, \"\");\n+\t\tadd_pending_object(&revs, &bc->commit->object, \"\");\n \t}\n \n \tif (prepare_revision_walk(&revs))\n@@ -430,8 +449,13 @@ static int fill_bitmap_commit(struct bb_commit *ent,\n \t\tstruct commit *c = prio_queue_get(queue);\n \n \t\tif (old_bitmap && mapping) {\n-\t\t\tstruct ewah_bitmap *old = bitmap_for_commit(old_bitmap, c);\n+\t\t\tstruct ewah_bitmap *old;\n \t\t\tstruct bitmap *remapped = bitmap_new();\n+\n+\t\t\tif (commit->object.flags & BITMAP_PSEUDO_MERGE)\n+\t\t\t\told = NULL;\n+\t\t\telse\n+\t\t\t\told = bitmap_for_commit(old_bitmap, c);\n \t\t\t/*\n \t\t\t * If this commit has an old bitmap, then translate that\n \t\t\t * bitmap and add its bits to this one. No need to walk\n@@ -450,12 +474,14 @@ static int fill_bitmap_commit(struct bb_commit *ent,\n \t\t * Mark ourselves and queue our tree. The commit\n \t\t * walk ensures we cover all parents.\n \t\t */\n-\t\tpos = find_object_pos(&c->object.oid, &found);\n-\t\tif (!found)\n-\t\t\treturn -1;\n-\t\tbitmap_set(ent->bitmap, pos);\n-\t\tprio_queue_put(tree_queue,\n-\t\t\t repo_get_commit_tree(the_repository, c));\n+\t\tif (!(c->object.flags & BITMAP_PSEUDO_MERGE)) {\n+\t\t\tpos = find_object_pos(&c->object.oid, &found);\n+\t\t\tif (!found)\n+\t\t\t\treturn -1;\n+\t\t\tbitmap_set(ent->bitmap, pos);\n+\t\t\tprio_queue_put(tree_queue,\n+\t\t\t\t repo_get_commit_tree(the_repository, c));\n+\t\t}\n \n \t\tfor (p = c->parents; p; p = p->next) {\n \t\t\tpos = find_object_pos(&p->item->object.oid, &found);\n@@ -483,6 +509,9 @@ static void store_selected(struct bb_commit *ent, struct commit *commit)\n \n \tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n \n+\tif (ent->pseudo_merge)\n+\t\treturn;\n+\n \thash_pos = kh_get_oid_map(writer.bitmaps, commit->object.oid);\n \tif (hash_pos == kh_end(writer.bitmaps))\n \t\tdie(_(\"attempted to store non-selected commit: '%s'\"),\n@@ -612,7 +641,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n-\t\t\tpush_bitmapped_commit(indexed_commits[i]);\n+\t\t\tbitmap_writer_push_bitmapped_commit(indexed_commits[i], 0);\n \t\treturn;\n \t}\n \n@@ -645,7 +674,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\t\t}\n \t\t}\n \n-\t\tpush_bitmapped_commit(chosen);\n+\t\tbitmap_writer_push_bitmapped_commit(chosen, 0);\n \n \t\ti += next + 1;\n \t\tdisplay_progress(writer.progress, i);\n@@ -683,8 +712,11 @@ static void write_selected_commits_v1(struct hashfile *f,\n {\n \tint i;\n \n-\tfor (i = 0; i < writer.selected_nr; ++i) {\n+\tfor (i = 0; i < bitmap_writer_selected_nr(); ++i) {\n \t\tstruct bitmapped_commit *stored = &writer.selected[i];\n+\t\tif (stored->pseudo_merge)\n+\t\t\tBUG(\"unexpected pseudo-merge among selected: %s\",\n+\t\t\t oid_to_hex(&stored->commit->object.oid));\n \n \t\tif (offsets)\n \t\t\toffsets[i] = hashfile_total(f);\n@@ -718,10 +750,10 @@ static void write_lookup_table(struct hashfile *f,\n \tuint32_t i;\n \tuint32_t *table, *table_inv;\n \n-\tALLOC_ARRAY(table, writer.selected_nr);\n-\tALLOC_ARRAY(table_inv, writer.selected_nr);\n+\tALLOC_ARRAY(table, bitmap_writer_selected_nr());\n+\tALLOC_ARRAY(table_inv, bitmap_writer_selected_nr());\n \n-\tfor (i = 0; i < writer.selected_nr; i++)\n+\tfor (i = 0; i < bitmap_writer_selected_nr(); i++)\n \t\ttable[i] = i;\n \n \t/*\n@@ -729,16 +761,16 @@ static void write_lookup_table(struct hashfile *f,\n \t * bitmap corresponds to j'th bitmapped commit (among the selected\n \t * commits) in lex order of OIDs.\n \t */\n-\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n+\tQSORT_S(table, bitmap_writer_selected_nr(), table_cmp, commit_positions);\n \n \t/* table_inv helps us discover that relationship (i'th bitmap\n \t * to j'th commit by j = table_inv[i])\n \t */\n-\tfor (i = 0; i < writer.selected_nr; i++)\n+\tfor (i = 0; i < bitmap_writer_selected_nr(); i++)\n \t\ttable_inv[table[i]] = i;\n \n \ttrace2_region_enter(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n-\tfor (i = 0; i < writer.selected_nr; i++) {\n+\tfor (i = 0; i < bitmap_writer_selected_nr(); i++) {\n \t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n \t\tuint32_t xor_offset = selected->xor_offset;\n \t\tuint32_t xor_row;\n@@ -809,7 +841,7 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \tmemcpy(header.magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE));\n \theader.version = htons(default_version);\n \theader.options = htons(flags | options);\n-\theader.entry_count = htonl(writer.selected_nr);\n+\theader.entry_count = htonl(bitmap_writer_selected_nr());\n \thashcpy(header.checksum, writer.pack_checksum);\n \n \thashwrite(f, &header, sizeof(header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz);\n@@ -821,9 +853,9 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n \t\tCALLOC_ARRAY(offsets, index_nr);\n \n-\tALLOC_ARRAY(commit_positions, writer.selected_nr);\n+\tALLOC_ARRAY(commit_positions, bitmap_writer_selected_nr());\n \n-\tfor (i = 0; i < writer.selected_nr; i++) {\n+\tfor (i = 0; i < bitmap_writer_selected_nr(); i++) {\n \t\tstruct bitmapped_commit *stored = &writer.selected[i];\n \t\tint commit_pos = oid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex dae2d68a338..ca9acd2f735 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -21,6 +21,7 @@ struct bitmap_disk_header {\n \tunsigned char checksum[GIT_MAX_RAWSZ];\n };\n \n+#define BITMAP_PSEUDO_MERGE (1u<<21)\n #define NEEDS_BITMAP (1u<<22)\n \n /*\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493690","messageId":"9c6d09bf8742f880647067af91da16d656bab672.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 07/23] pack-bitmap: implement `bitmap_writer_has_bitmapped_object_id()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:43:18Z","receivedAt":"2024-04-29T20:43:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to implement pseudo-merge bitmap selection by implementing a\nnecessary new function, `bitmap_writer_has_bitmapped_object_id()`.\n\nThis function returns whether or not the bitmap_writer selected the\ngiven object ID for bitmapping. This will allow the pseudo-merge\nmachinery to reject candidates for pseudo-merges if they have already\nbeen selected as an ordinary bitmap tip.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 5 +++++\n pack-bitmap.h | 2 ++\n 2 files changed, 7 insertions(+)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex fef02cd745a..c7514a58407 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -123,6 +123,11 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack,\n \t}\n }\n \n+int bitmap_writer_has_bitmapped_object_id(const struct object_id *oid)\n+{\n+\treturn kh_get_oid_map(writer.bitmaps, *oid) != kh_end(writer.bitmaps);\n+}\n+\n /**\n * Compute the actual bitmaps\n */\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex ca9acd2f735..995d664cc89 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -98,6 +98,8 @@ int bitmap_has_oid_in_uninteresting(struct bitmap_index *, const struct object_i\n \n off_t get_disk_usage_from_bitmap(struct bitmap_index *, struct rev_info *);\n \n+int bitmap_writer_has_bitmapped_object_id(const struct object_id *oid);\n+\n void bitmap_writer_init(struct repository *r);\n void bitmap_writer_show_progress(int show);\n void bitmap_writer_set_checksum(const unsigned char *sha1);\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493691","messageId":"dfd4b73d12edc1833e5090c956cda6f28046c9e4.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 08/23] pack-bitmap: make `bitmap_writer_push_bitmapped_commit()` public","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:43:22Z","receivedAt":"2024-04-29T20:43:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pseudo-merge selection code will be added in a subsequent commit,\nand will need a way to push the allocated commit structures into the\nbitmap writer from a separate compilation unit.\n\nMake the `bitmap_writer_push_bitmapped_commit()` function part of the\npack-bitmap.h header in order to make this possible.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 4 ++--\n pack-bitmap.h | 2 ++\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex c7514a58407..dab5bdea806 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -132,8 +132,8 @@ int bitmap_writer_has_bitmapped_object_id(const struct object_id *oid)\n * Compute the actual bitmaps\n */\n \n-static void bitmap_writer_push_bitmapped_commit(struct commit *commit,\n-\t\t\t\t\t\tunsigned pseudo_merge)\n+void bitmap_writer_push_bitmapped_commit(struct commit *commit,\n+\t\t\t\t\t unsigned pseudo_merge)\n {\n \tif (writer.selected_nr >= writer.selected_alloc) {\n \t\twriter.selected_alloc = (writer.selected_alloc + 32) * 2;\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 995d664cc89..0f539d79cfd 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -99,6 +99,8 @@ int bitmap_has_oid_in_uninteresting(struct bitmap_index *, const struct object_i\n off_t get_disk_usage_from_bitmap(struct bitmap_index *, struct rev_info *);\n \n int bitmap_writer_has_bitmapped_object_id(const struct object_id *oid);\n+void bitmap_writer_push_bitmapped_commit(struct commit *commit,\n+\t\t\t\t\t unsigned pseudo_merge);\n \n void bitmap_writer_init(struct repository *r);\n void bitmap_writer_show_progress(int show);\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493692","messageId":"86a1e4b8b9be99563836d1539fbf2ed4c4a6920d.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 09/23] pseudo-merge: implement support for selecting pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:43:37Z","receivedAt":"2024-04-29T20:43:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Teach the new pseudo-merge machinery how to select non-bitmapped commits\nfor inclusion in different pseudo-merge group(s) based on a handful of\ncriteria.\n\nPseudo-merges are derived first from named pseudo-merge groups (see the\n`bitmapPseudoMerge..*` configuration options). They are\n(optionally) further segmented within an individual pseudo-merge group\nbased on any capture group(s) within the pseudo-merge group's pattern.\n\nFor example, a configuration like so:\n\n [bitmapPseudoMerge \"all\"]\n pattern = \"refs/\"\n threshold = now\n stableThreshold = never\n sampleRate = 100\n maxMerges = 64\n\nwould group all non-bitmapped commits into up to 64 individual\npseudo-merge commits.\n\nIf you wanted to separate tags from branches when generating\npseudo-merge commits, and further segment them by which fork they\noriginate from (using the same \"refs/virtual/\" scheme as in the delta\nislands documentation), you would instead write something like:\n\n [bitmapPseudoMerge \"all\"]\n pattern = \"refs/virtual/([0-9]+)/(heads|tags)/\"\n threshold = now\n stableThreshold = never\n sampleRate = 100\n maxMerges = 64\n\nWhich would generate pseudo-merge group identifiers like \"1234-heads\",\nand \"5678-tags\" (for branches in fork \"1234\", and tags in remote \"5678\",\nrespectively).\n\nWithin pseudo-merge groups, there are a handful of other options used to\ncontrol the distribution of matching commits among individual\npseudo-merge commits:\n\n - bitmapPseudoMerge..decay\n - bitmapPseudoMerge..sampleRate\n - bitmapPseudoMerge..threshold\n - bitmapPseudoMerge..maxMerges\n - bitmapPseudoMerge..stableThreshold\n - bitmapPseudoMerge..stableSize\n\nThe decay parameter roughly corresponds to \"k\" in `f(n) = C*n^(-k/100)`,\nwhere `f(n)` describes the size of the `n`-th pseudo-merge group. The\nsample rate controls what percentage of eligible commits are considered\nas candidates. The threshold parameter indicates the minimum age (so as\nto avoid including too-recent commits in a pseudo-merge group, making it\nless likely to be valid). The \"maxMerges\" parameter sets an upper-bound\non the number of pseudo-merge commits an individual group\n\nThe latter two \"stable\"-related parameters control \"stable\" pseudo-merge\ngroups, comprised of a fixed number of commits which are older than the\nconfigured \"stable threshold\" value and may be grouped together in\nchunks of \"stableSize\" in order of age.\n\nThis patch implements the aforementioned selection routine, as well as\nparsing the relevant configuration options.\n\nSigned-off-by: Taylor Blau \n---\n pseudo-merge.c | 441 +++++++++++++++++++++++++++++++++++++++++++++++++\n pseudo-merge.h | 96 +++++++++++\n 2 files changed, 537 insertions(+)\n\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex 37e037ba272..caccef942a1 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -1,2 +1,443 @@\n #include \"git-compat-util.h\"\n #include \"pseudo-merge.h\"\n+#include \"date.h\"\n+#include \"oid-array.h\"\n+#include \"strbuf.h\"\n+#include \"config.h\"\n+#include \"string-list.h\"\n+#include \"refs.h\"\n+#include \"pack-bitmap.h\"\n+#include \"commit.h\"\n+#include \"alloc.h\"\n+#include \"progress.h\"\n+\n+#define DEFAULT_PSEUDO_MERGE_DECAY 1.0f\n+#define DEFAULT_PSEUDO_MERGE_MAX_MERGES 64\n+#define DEFAULT_PSEUDO_MERGE_SAMPLE_RATE 100\n+#define DEFAULT_PSEUDO_MERGE_THRESHOLD approxidate(\"1.week.ago\")\n+#define DEFAULT_PSEUDO_MERGE_STABLE_THRESHOLD approxidate(\"1.month.ago\")\n+#define DEFAULT_PSEUDO_MERGE_STABLE_SIZE 512\n+\n+static float gitexp(float base, int exp)\n+{\n+\tfloat result = 1;\n+\twhile (1) {\n+\t\tif (exp % 2)\n+\t\t\tresult *= base;\n+\t\texp >>= 1;\n+\t\tif (!exp)\n+\t\t\tbreak;\n+\t\tbase *= base;\n+\t}\n+\treturn result;\n+}\n+\n+static uint32_t pseudo_merge_group_size(const struct pseudo_merge_group *group,\n+\t\t\t\t\tconst struct pseudo_merge_matches *matches,\n+\t\t\t\t\tuint32_t i)\n+{\n+\tfloat C = 0.0f;\n+\tuint32_t n;\n+\n+\t/*\n+\t * The size of pseudo-merge groups decays according to a power series,\n+\t * which looks like:\n+\t *\n+\t * f(n) = C * n^-k\n+\t *\n+\t * , where 'n' is the n-th pseudo-merge group, 'f(n)' is its size, 'k'\n+\t * is the decay rate, and 'C' is a scaling value.\n+\t *\n+\t * The value of C depends on the number of groups, decay rate, and total\n+\t * number of commits. It is computed such that if there are M and N\n+\t * total groups and commits, respectively, that:\n+\t *\n+\t * N = f(0) + f(1) + ... f(M-1)\n+\t *\n+\t * Rearranging to isolate C, we get:\n+\t *\n+\t * N = \\sum_{n=1}^M C / n^k\n+\t *\n+\t * N / C = \\sum_{n=1}^M n^-k\n+\t *\n+\t * C = N / \\sum_{n=1}^M n^-k\n+\t *\n+\t * For example, if we have a decay rate of 'k' being equal to 1.5, 'N'\n+\t * total commits equal to 10,000, and 'M' being equal to 6 groups, then\n+\t * the (rounded) group sizes are:\n+\t *\n+\t * { 5469, 1934, 1053, 684, 489, 372 }\n+\t *\n+\t * increasing the number of total groups, say to 10, scales the group\n+\t * sizes appropriately:\n+\t *\n+\t * { 5012, 1772, 964, 626, 448, 341, 271, 221, 186, 158 }\n+\t */\n+\tfor (n = 0; n < group->max_merges; n++)\n+\t\tC += 1.0f / gitexp(n + 1, group->decay);\n+\tC = matches->unstable_nr / C;\n+\n+\treturn (int)((C / gitexp(i + 1, group->decay)) + 0.5);\n+}\n+\n+static void init_pseudo_merge_group(struct pseudo_merge_group *group)\n+{\n+\tmemset(group, 0, sizeof(struct pseudo_merge_group));\n+\n+\tstrmap_init_with_options(&group->matches, NULL, 0);\n+\n+\tgroup->decay = DEFAULT_PSEUDO_MERGE_DECAY;\n+\tgroup->max_merges = DEFAULT_PSEUDO_MERGE_MAX_MERGES;\n+\tgroup->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE;\n+\tgroup->threshold = DEFAULT_PSEUDO_MERGE_THRESHOLD;\n+\tgroup->stable_threshold = DEFAULT_PSEUDO_MERGE_STABLE_THRESHOLD;\n+\tgroup->stable_size = DEFAULT_PSEUDO_MERGE_STABLE_SIZE;\n+}\n+\n+static int pseudo_merge_config(const char *var, const char *value,\n+\t\t\t const struct config_context *ctx,\n+\t\t\t void *cb_data)\n+{\n+\tstruct string_list *list = cb_data;\n+\tstruct string_list_item *item;\n+\tstruct pseudo_merge_group *group;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *sub, *key;\n+\tsize_t sub_len;\n+\n+\tif (parse_config_key(var, \"bitmappseudomerge\", &sub, &sub_len, &key))\n+\t\treturn 0;\n+\n+\tif (!sub_len)\n+\t\treturn 0;\n+\n+\tstrbuf_add(&buf, sub, sub_len);\n+\n+\titem = string_list_lookup(list, buf.buf);\n+\tif (!item) {\n+\t\titem = string_list_insert(list, buf.buf);\n+\n+\t\titem->util = xmalloc(sizeof(struct pseudo_merge_group));\n+\t\tinit_pseudo_merge_group(item->util);\n+\t}\n+\n+\tgroup = item->util;\n+\n+\tif (!strcmp(key, \"pattern\")) {\n+\t\tstruct strbuf re = STRBUF_INIT;\n+\n+\t\tfree(group->pattern);\n+\t\tif (*value != '^')\n+\t\t\tstrbuf_addch(&re, '^');\n+\t\tstrbuf_addstr(&re, value);\n+\n+\t\tgroup->pattern = xcalloc(1, sizeof(regex_t));\n+\t\tif (regcomp(group->pattern, re.buf, REG_EXTENDED))\n+\t\t\tdie(_(\"failed to load pseudo-merge regex for %s: '%s'\"),\n+\t\t\t sub, re.buf);\n+\n+\t\tstrbuf_release(&re);\n+\t} else if (!strcmp(key, \"decay\")) {\n+\t\tgroup->decay = git_config_int(var, value, ctx->kvi);\n+\t\tif (group->decay < 0) {\n+\t\t\twarning(_(\"%s must be non-negative, using default\"), var);\n+\t\t\tgroup->decay = DEFAULT_PSEUDO_MERGE_DECAY;\n+\t\t}\n+\t} else if (!strcmp(key, \"samplerate\")) {\n+\t\tgroup->sample_rate = git_config_int(var, value, ctx->kvi);\n+\t\tif (!(0 <= group->sample_rate && group->sample_rate <= 100)) {\n+\t\t\twarning(_(\"%s must be between 0 and 100, using default\"), var);\n+\t\t\tgroup->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE;\n+\t\t}\n+\t} else if (!strcmp(key, \"threshold\")) {\n+\t\tif (git_config_expiry_date(&group->threshold, var, value)) {\n+\t\t\tstrbuf_release(&buf);\n+\t\t\treturn -1;\n+\t\t}\n+\t} else if (!strcmp(key, \"maxmerges\")) {\n+\t\tgroup->max_merges = git_config_int(var, value, ctx->kvi);\n+\t\tif (group->max_merges < 0) {\n+\t\t\twarning(_(\"%s must be non-negative, using default\"), var);\n+\t\t\tgroup->max_merges = DEFAULT_PSEUDO_MERGE_MAX_MERGES;\n+\t\t}\n+\t} else if (!strcmp(key, \"stablethreshold\")) {\n+\t\tif (git_config_expiry_date(&group->stable_threshold, var, value)) {\n+\t\t\tstrbuf_release(&buf);\n+\t\t\treturn -1;\n+\t\t}\n+\t} else if (!strcmp(key, \"stablesize\")) {\n+\t\tgroup->stable_size = git_config_int(var, value, ctx->kvi);\n+\t\tif (group->stable_size <= 0) {\n+\t\t\twarning(_(\"%s must be positive, using default\"), var);\n+\t\t\tgroup->stable_size = DEFAULT_PSEUDO_MERGE_STABLE_SIZE;\n+\t\t}\n+\t}\n+\n+\tstrbuf_release(&buf);\n+\n+\treturn 0;\n+}\n+\n+void load_pseudo_merges_from_config(struct string_list *list)\n+{\n+\tstruct string_list_item *item;\n+\n+\tgit_config(pseudo_merge_config, list);\n+\n+\tfor_each_string_list_item(item, list) {\n+\t\tstruct pseudo_merge_group *group = item->util;\n+\t\tif (!group->pattern)\n+\t\t\tdie(_(\"pseudo-merge group '%s' missing required pattern\"),\n+\t\t\t item->string);\n+\t\tif (group->threshold < group->stable_threshold)\n+\t\t\tdie(_(\"pseudo-merge group '%s' has unstable threshold \"\n+\t\t\t \"before stable one\"), item->string);\n+\t}\n+}\n+\n+static int find_pseudo_merge_group_for_ref(const char *refname,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t int flags UNUSED,\n+\t\t\t\t\t void *_data)\n+{\n+\tstruct string_list *list = _data;\n+\tstruct object_id peeled;\n+\tstruct commit *c;\n+\tuint32_t i;\n+\tint has_bitmap;\n+\n+\tif (!peel_iterated_oid(oid, &peeled))\n+\t\toid = &peeled;\n+\n+\tc = lookup_commit(the_repository, oid);\n+\tif (!c)\n+\t\treturn 0;\n+\n+\thas_bitmap = bitmap_writer_has_bitmapped_object_id(oid);\n+\n+\tfor (i = 0; i < list->nr; i++) {\n+\t\tstruct pseudo_merge_group *group;\n+\t\tstruct pseudo_merge_matches *matches;\n+\t\tstruct strbuf group_name = STRBUF_INIT;\n+\t\tregmatch_t captures[16];\n+\t\tsize_t j;\n+\n+\t\tgroup = list->items[i].util;\n+\t\tif (regexec(group->pattern, refname, ARRAY_SIZE(captures),\n+\t\t\t captures, 0))\n+\t\t\tcontinue;\n+\n+\t\tif (captures[ARRAY_SIZE(captures) - 1].rm_so != -1)\n+\t\t\twarning(_(\"pseudo-merge regex from config has too many capture \"\n+\t\t\t\t \"groups (max=%\"PRIuMAX\")\"),\n+\t\t\t\t(uintmax_t)ARRAY_SIZE(captures) - 2);\n+\n+\t\tfor (j = !!group->pattern->re_nsub; j < ARRAY_SIZE(captures); j++) {\n+\t\t\tregmatch_t *match = &captures[j];\n+\t\t\tif (match->rm_so == -1)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (group_name.len)\n+\t\t\t\tstrbuf_addch(&group_name, '-');\n+\n+\t\t\tstrbuf_add(&group_name, refname + match->rm_so,\n+\t\t\t\t match->rm_eo - match->rm_so);\n+\t\t}\n+\n+\t\tmatches = strmap_get(&group->matches, group_name.buf);\n+\t\tif (!matches) {\n+\t\t\tmatches = xcalloc(1, sizeof(*matches));\n+\t\t\tstrmap_put(&group->matches, strbuf_detach(&group_name, NULL),\n+\t\t\t\t matches);\n+\t\t}\n+\n+\t\tif (c->date <= group->stable_threshold) {\n+\t\t\tALLOC_GROW(matches->stable, matches->stable_nr + 1,\n+\t\t\t\t matches->stable_alloc);\n+\t\t\tmatches->stable[matches->stable_nr++] = c;\n+\t\t} else if (c->date <= group->threshold && !has_bitmap) {\n+\t\t\tALLOC_GROW(matches->unstable, matches->unstable_nr + 1,\n+\t\t\t\t matches->unstable_alloc);\n+\t\t\tmatches->unstable[matches->unstable_nr++] = c;\n+\t\t}\n+\n+\t\tstrbuf_release(&group_name);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static struct commit *push_pseudo_merge(struct pseudo_merge_group *group)\n+{\n+\tstruct commit *merge;\n+\n+\tALLOC_GROW(group->merges, group->merges_nr + 1, group->merges_alloc);\n+\n+\tmerge = alloc_commit_node(the_repository);\n+\tmerge->object.parsed = 1;\n+\tmerge->object.flags |= BITMAP_PSEUDO_MERGE;\n+\n+\tgroup->merges[group->merges_nr++] = merge;\n+\n+\treturn merge;\n+}\n+\n+static struct pseudo_merge_commit_idx *pseudo_merge_idx(kh_oid_map_t *pseudo_merge_commits,\n+\t\t\t\t\t\t\tconst struct object_id *oid)\n+\n+{\n+\tstruct pseudo_merge_commit_idx *pmc;\n+\tkhiter_t hash_pos;\n+\n+\thash_pos = kh_get_oid_map(pseudo_merge_commits, *oid);\n+\tif (hash_pos == kh_end(pseudo_merge_commits)) {\n+\t\tint hash_ret;\n+\t\thash_pos = kh_put_oid_map(pseudo_merge_commits, *oid, &hash_ret);\n+\n+\t\tCALLOC_ARRAY(pmc, 1);\n+\n+\t\tkh_value(pseudo_merge_commits, hash_pos) = pmc;\n+\t} else {\n+\t\tpmc = kh_value(pseudo_merge_commits, hash_pos);\n+\t}\n+\n+\treturn pmc;\n+}\n+\n+#define MIN_PSEUDO_MERGE_SIZE 8\n+\n+static void select_pseudo_merges_1(struct pseudo_merge_group *group,\n+\t\t\t\t struct pseudo_merge_matches *matches,\n+\t\t\t\t kh_oid_map_t *pseudo_merge_commits,\n+\t\t\t\t uint32_t *pseudo_merges_nr)\n+{\n+\tuint32_t i, j;\n+\tuint32_t stable_merges_nr;\n+\n+\tif (!matches->stable_nr && !matches->unstable_nr)\n+\t\treturn; /* all tips in this group already have bitmaps */\n+\n+\tstable_merges_nr = matches->stable_nr / group->stable_size;\n+\tif (matches->stable_nr % group->stable_size)\n+\t\tstable_merges_nr++;\n+\n+\t/* make stable_merges_nr pseudo merges for stable commits */\n+\tfor (i = 0, j = 0; i < stable_merges_nr; i++) {\n+\t\tstruct commit *merge;\n+\t\tstruct commit_list **p;\n+\n+\t\tmerge = push_pseudo_merge(group);\n+\t\tp = &merge->parents;\n+\n+\t\tdo {\n+\t\t\tstruct commit *c;\n+\t\t\tstruct pseudo_merge_commit_idx *pmc;\n+\n+\t\t\tif (j >= matches->stable_nr)\n+\t\t\t\tbreak;\n+\n+\t\t\tc = matches->stable[j++];\n+\t\t\tpmc = pseudo_merge_idx(pseudo_merge_commits,\n+\t\t\t\t\t &c->object.oid);\n+\n+\t\t\tALLOC_GROW(pmc->pseudo_merge, pmc->nr + 1, pmc->alloc);\n+\n+\t\t\tpmc->pseudo_merge[pmc->nr++] = *pseudo_merges_nr;\n+\t\t\tp = commit_list_append(c, p);\n+\t\t} while (j % group->stable_size);\n+\n+\t\tbitmap_writer_push_bitmapped_commit(merge, 1);\n+\t\t(*pseudo_merges_nr)++;\n+\t}\n+\n+\t/* make up to group->max_merges pseudo merges for unstable commits */\n+\tfor (i = 0, j = 0; i < group->max_merges; i++) {\n+\t\tstruct commit *merge;\n+\t\tstruct commit_list **p;\n+\t\tuint32_t size, end;\n+\n+\t\tmerge = push_pseudo_merge(group);\n+\t\tp = &merge->parents;\n+\n+\t\tsize = pseudo_merge_group_size(group, matches, i);\n+\t\tend = size < MIN_PSEUDO_MERGE_SIZE ? matches->unstable_nr : j + size;\n+\n+\t\tfor (; j < end && j < matches->unstable_nr; j++) {\n+\t\t\tstruct commit *c = matches->unstable[j];\n+\t\t\tstruct pseudo_merge_commit_idx *pmc;\n+\n+\t\t\tif (j % (100 / group->sample_rate))\n+\t\t\t\tcontinue;\n+\n+\t\t\tpmc = pseudo_merge_idx(pseudo_merge_commits,\n+\t\t\t\t\t &c->object.oid);\n+\n+\t\t\tALLOC_GROW(pmc->pseudo_merge, pmc->nr + 1, pmc->alloc);\n+\n+\t\t\tpmc->pseudo_merge[pmc->nr++] = *pseudo_merges_nr;\n+\t\t\tp = commit_list_append(c, p);\n+\t\t}\n+\n+\t\tbitmap_writer_push_bitmapped_commit(merge, 1);\n+\t\t(*pseudo_merges_nr)++;\n+\t\tif (end >= matches->unstable_nr)\n+\t\t\tbreak;\n+\t}\n+}\n+\n+static int commit_date_cmp(const void *va, const void *vb)\n+{\n+\ttimestamp_t a = (*(const struct commit **)va)->date;\n+\ttimestamp_t b = (*(const struct commit **)vb)->date;\n+\n+\tif (a < b)\n+\t\treturn -1;\n+\telse if (a > b)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static void sort_pseudo_merge_matches(struct pseudo_merge_matches *matches)\n+{\n+\tQSORT(matches->stable, matches->stable_nr, commit_date_cmp);\n+\tQSORT(matches->unstable, matches->unstable_nr, commit_date_cmp);\n+}\n+\n+void select_pseudo_merges(struct string_list *list,\n+\t\t\t struct commit **commits, size_t commits_nr,\n+\t\t\t kh_oid_map_t *pseudo_merge_commits,\n+\t\t\t uint32_t *pseudo_merges_nr,\n+\t\t\t unsigned show_progress)\n+{\n+\tstruct progress *progress = NULL;\n+\tuint32_t i;\n+\n+\tif (!list->nr)\n+\t\treturn;\n+\n+\tif (show_progress)\n+\t\tprogress = start_progress(\"Selecting pseudo-merge commits\", list->nr);\n+\n+\tfor_each_ref(find_pseudo_merge_group_for_ref, list);\n+\n+\tfor (i = 0; i < list->nr; i++) {\n+\t\tstruct pseudo_merge_group *group;\n+\t\tstruct hashmap_iter iter;\n+\t\tstruct strmap_entry *e;\n+\n+\t\tgroup = list->items[i].util;\n+\t\tstrmap_for_each_entry(&group->matches, &iter, e) {\n+\t\t\tstruct pseudo_merge_matches *matches = e->value;\n+\n+\t\t\tsort_pseudo_merge_matches(matches);\n+\n+\t\t\tselect_pseudo_merges_1(group, matches,\n+\t\t\t\t\t pseudo_merge_commits,\n+\t\t\t\t\t pseudo_merges_nr);\n+\t\t}\n+\n+\t\tdisplay_progress(progress, i + 1);\n+\t}\n+\n+\tstop_progress(&progress);\n+}\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nindex cab8ff6960a..81888731864 100644\n--- a/pseudo-merge.h\n+++ b/pseudo-merge.h\n@@ -2,5 +2,101 @@\n #define PSEUDO_MERGE_H\n \n #include \"git-compat-util.h\"\n+#include \"strmap.h\"\n+#include \"khash.h\"\n+#include \"ewah/ewok.h\"\n+\n+struct commit;\n+struct string_list;\n+struct bitmap_index;\n+\n+/*\n+ * A pseudo-merge group tracks the set of non-bitmapped reference tips\n+ * that match the given pattern.\n+ *\n+ * Within those matches, they are further segmented by separating\n+ * consecutive capture groups with '-' dash character capture groups\n+ * with '-' dash characters.\n+ *\n+ * Those groups are then ordered by committer date and partitioned\n+ * into individual pseudo-merge(s) according to the decay, max_merges,\n+ * sample_rate, and threshold parameters.\n+ */\n+struct pseudo_merge_group {\n+\tregex_t *pattern;\n+\n+\t/* capture group(s) -> struct pseudo_merge_matches */\n+\tstruct strmap matches;\n+\n+\t/*\n+\t * The individual pseudo-merge(s) that are generated from the\n+\t * above array of matches, partitioned according to the below\n+\t * parameters.\n+\t */\n+\tstruct commit **merges;\n+\tsize_t merges_nr;\n+\tsize_t merges_alloc;\n+\n+\t/*\n+\t * Pseudo-merge grouping parameters. See git-config(1) for\n+\t * more information.\n+\t */\n+\tfloat decay;\n+\tint max_merges;\n+\tint sample_rate;\n+\tint stable_size;\n+\ttimestamp_t threshold;\n+\ttimestamp_t stable_threshold;\n+};\n+\n+struct pseudo_merge_matches {\n+\tstruct commit **stable;\n+\tstruct commit **unstable;\n+\tsize_t stable_nr, stable_alloc;\n+\tsize_t unstable_nr, unstable_alloc;\n+};\n+\n+/*\n+ * Read the repository's configuration:\n+ *\n+ * - bitmapPseudoMerge..pattern\n+ * - bitmapPseudoMerge..decay\n+ * - bitmapPseudoMerge..sampleRate\n+ * - bitmapPseudoMerge..threshold\n+ * - bitmapPseudoMerge..maxMerges\n+ * - bitmapPseudoMerge..stableThreshold\n+ * - bitmapPseudoMerge..stableSize\n+ *\n+ * and populates the given `list` with pseudo-merge groups. String\n+ * entry keys are the pseudo-merge group names, and the values are\n+ * pointers to the pseudo_merge_group structure itself.\n+ */\n+void load_pseudo_merges_from_config(struct string_list *list);\n+\n+/*\n+ * A pseudo-merge commit index (pseudo_merge_commit_idx) maps a\n+ * particular (non-pseudo-merge) commit to the list of pseudo-merge(s)\n+ * it appears in.\n+ */\n+struct pseudo_merge_commit_idx {\n+\tuint32_t *pseudo_merge;\n+\tsize_t nr, alloc;\n+};\n+\n+/*\n+ * Selects pseudo-merges from a list of commits, populating the given\n+ * string_list of pseudo-merge groups.\n+ *\n+ * Populates the pseudo_merge_commits map with a commit_idx\n+ * corresponding to each commit in the list. Counts the total number\n+ * of pseudo-merges generated.\n+ *\n+ * Optionally shows a progress meter.\n+ */\n+void select_pseudo_merges(struct string_list *list,\n+\t\t\t struct commit **commits, size_t commits_nr,\n+\t\t\t kh_oid_map_t *pseudo_merge_commits,\n+\t\t\t uint32_t *pseudo_merges_nr,\n+\t\t\t unsigned show_progress);\n \n #endif\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493693","messageId":"12b432e3a8adcda6228beae2b41b2363a6ce82a0.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 10/23] pack-bitmap-write.c: select pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:43:41Z","receivedAt":"2024-04-29T20:43:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the pseudo-merge machinery has learned how to select\nnon-bitmapped commits and assign them into different pseudo-merge\ngroup(s), invoke this new API from within the pack-bitmap internals and\nstore the results off.\n\nNote that the selected pseudo-merge commits aren't actually used or\nwritten anywhere yet. This will be done in the following commit.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config.txt | 2 +\n Documentation/config/bitmap-pseudo-merge.txt | 75 ++++++++++++++++++++\n Documentation/technical/bitmap-format.txt | 26 +++++++\n pack-bitmap-write.c | 14 ++++\n 4 files changed, 117 insertions(+)\n create mode 100644 Documentation/config/bitmap-pseudo-merge.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 70b448b1326..bbedb7b9a06 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -383,6 +383,8 @@ include::config/apply.txt[]\n \n include::config/attr.txt[]\n \n+include::config/bitmap-pseudo-merge.txt[]\n+\n include::config/blame.txt[]\n \n include::config/branch.txt[]\ndiff --git a/Documentation/config/bitmap-pseudo-merge.txt b/Documentation/config/bitmap-pseudo-merge.txt\nnew file mode 100644\nindex 00000000000..90b72522046\n--- /dev/null\n+++ b/Documentation/config/bitmap-pseudo-merge.txt\n@@ -0,0 +1,75 @@\n+bitmapPseudoMerge..pattern::\n+\tRegular expression used to match reference names. Commits\n+\tpointed to by references matching this pattern (and meeting\n+\tthe below criteria, like `bitmapPseudoMerge..sampleRate`\n+\tand `bitmapPseudoMerge..threshold`) will be considered\n+\tfor inclusion in a pseudo-merge bitmap.\n++\n+Commits are grouped into pseudo-merge groups based on whether or not\n+any reference(s) that point at a given commit match the pattern, which\n+is an extended regular expression.\n++\n+Within a pseudo-merge group, commits may be further grouped into\n+sub-groups based on the capture groups in the pattern. These\n+sub-groupings are formed from the regular expressions by concatenating\n+any capture groups from the regular expression, with a '-' dash in\n+between.\n++\n+For example, if the pattern is `refs/tags/`, then all tags (provided\n+they meet the below criteria) will be considered candidates for the\n+same pseudo-merge group. However, if the pattern is instead\n+`refs/remotes/([0-9])+/tags/`, then tags from different remotes will\n+be grouped into separate pseudo-merge groups, based on the remote\n+number.\n+\n+bitmapPseudoMerge..decay::\n+\tDetermines the rate at which consecutive pseudo-merge bitmap\n+\tgroups decrease in size. Must be non-negative. This parameter\n+\tcan be thought of as `k` in the function `f(n) = C *\n+\tn^(-k/100)`, where `f(n)` is the size of the `n`th group.\n++\n+Setting the decay rate equal to `0` will cause all groups to be the\n+same size. Setting the decay rate equal to `100` will cause the `n`th\n+group to be `1/n` the size of the initial group. Higher values of the\n+decay rate cause consecutive groups to shrink at an increasing rate.\n+The default is `100`.\n+\n+bitmapPseudoMerge..sampleRate::\n+\tDetermines the proportion of non-bitmapped commits (among\n+\treference tips) which are selected for inclusion in an\n+\tunstable pseudo-merge bitmap. Must be between `0` and `100`\n+\t(inclusive). The default is `100`.\n+\n+bitmapPseudoMerge..threshold::\n+\tDetermines the minimum age of non-bitmapped commits (among\n+\treference tips, as above) which are candidates for inclusion\n+\tin an unstable pseudo-merge bitmap. The default is\n+\t`1.week.ago`.\n+\n+bitmapPseudoMerge..maxMerges::\n+\tDetermines the maximum number of pseudo-merge commits among\n+\twhich commits may be distributed.\n++\n+For pseudo-merge groups whose pattern does not contain any capture\n+groups, this setting is applied for all commits matching the regular\n+expression. For patterns that have one or more capture groups, this\n+setting is applied for each distinct capture group.\n++\n+For example, if your capture group is `refs/tags/`, then this setting\n+will distribute all tags into a maximum of `maxMerges` pseudo-merge\n+commits. However, if your capture group is, say,\n+`refs/remotes/([0-9]+)/tags/`, then this setting will be applied to\n+each remote's set of tags individually.\n++\n+Must be non-negative. The default value is 64.\n+\n+bitmapPseudoMerge..stableThreshold::\n+\tDetermines the minimum age of commits (among reference tips,\n+\tas above, however stable commits are still considered\n+\tcandidates even when they have been covered by a bitmap) which\n+\tare candidates for a stable a pseudo-merge bitmap. The default\n+\tis `1.month.ago`.\n+\n+bitmapPseudoMerge..stableSize::\n+\tDetermines the size (in number of commits) of a stable\n+\tpsuedo-merge bitmap. The default is `512`.\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex 63a7177ac08..ed7edf98034 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -434,3 +434,29 @@ the end of a `.bitmap` file. The format is as follows:\n \n * An 8-byte unsigned value (in network byte-order) equal to the number\n of bytes in the pseudo-merge section (including this field).\n+\n+=== Pseudo-merge selection\n+\n+Pseudo-merge commits are selected among non-bitmapped commits at the\n+tip of one or more reference(s). In addition, there are a handful of\n+constraints to further refine this selection:\n+\n+`pack.bitmapPseudoMergeDecay`:: Defines the \"decay rate\", which\n+corresponds to how quickly (or not) consecutive pseudo-merges decrease\n+in size relative to one another.\n+\n+`pack.bitmapPseudoMergeGroups`:: Defines the maximum number of\n+pseudo-merge groups.\n+\n+`pack.bitmapPseudoMergeSampleRate`:: Defines the percentage of commits\n+(matching the above criteria) which are selected.\n+\n+`pack.bitmapPseudoMergeThreshold`:: Defines the minimum age of a commit\n+in order to be considered for inclusion within one or more pseudo-merge\n+bitmaps.\n+\n+The size of consecutive pseudo-merge groups decays according to a\n+power-law decay function, where the size of the `n`-th group is `f(n) =\n+C*n^-k`. The value of `C` is chosen accordingly to match the number of\n+desired groups, and `k` is 1/100th of the value of\n+`pack.bitmapPseudoMergeDecay`.\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex dab5bdea806..e06930e10b9 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -17,6 +17,7 @@\n #include \"trace2.h\"\n #include \"tree.h\"\n #include \"tree-walk.h\"\n+#include \"pseudo-merge.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -39,6 +40,8 @@ struct bitmap_writer {\n \tstruct bitmapped_commit *selected;\n \tunsigned int selected_nr, selected_alloc;\n \n+\tstruct string_list pseudo_merge_groups;\n+\tkh_oid_map_t *pseudo_merge_commits; /* oid -> pseudo merge(s) */\n \tuint32_t pseudo_merges_nr;\n \n \tstruct progress *progress;\n@@ -56,6 +59,11 @@ static inline int bitmap_writer_selected_nr(void)\n void bitmap_writer_init(struct repository *r)\n {\n \twriter.bitmaps = kh_init_oid_map();\n+\twriter.pseudo_merge_commits = kh_init_oid_map();\n+\n+\tstring_list_init_dup(&writer.pseudo_merge_groups);\n+\n+\tload_pseudo_merges_from_config(&writer.pseudo_merge_groups);\n }\n \n void bitmap_writer_show_progress(int show)\n@@ -686,6 +694,12 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t}\n \n \tstop_progress(&writer.progress);\n+\n+\tselect_pseudo_merges(&writer.pseudo_merge_groups,\n+\t\t\t indexed_commits, indexed_commits_nr,\n+\t\t\t writer.pseudo_merge_commits,\n+\t\t\t &writer.pseudo_merges_nr,\n+\t\t\t writer.show_progress);\n }\n \n \n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493694","messageId":"6ce805d061e1e51dfd1b2ab3b8cd081292f42f3a.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 11/23] pack-bitmap-write.c: write pseudo-merge table","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:43:45Z","receivedAt":"2024-04-29T20:43:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the pack-bitmap writer machinery understands how to select and\nstore pseudo-merge commits, teach it how to write the new optional\npseudo-merge .bitmap extension.\n\nNo readers yet exist for this new extension to the .bitmap format. The\nfollowing commits will take any preparatory step(s) necessary before\nthen implementing the routines necessary to read this new table.\n\nIn the meantime, the new `write_pseudo_merges()` function implements\nwriting this new format as described by a previous commit in\nDocumentation/technical/bitmap-format.txt.\n\nWriting this table is fairly straightforward and consists of a few\nsub-components:\n\n - a pair of bitmaps for each pseudo-merge (one for the pseudo-merge\n \"parents\", and another for the objects reachable from those parents)\n\n - for each commit, the offset of either (a) the pseudo-merge it\n belongs to, or (b) an extended lookup table if it belongs to >1\n pseudo-merge groups\n\n - if there are any commits belonging to >1 pseudo-merge group, the\n extended lookup tables (which each consist of the number of\n pseudo-merge groups a commit appears in, and then that many 4-byte\n unsigned )\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 128 ++++++++++++++++++++++++++++++++++++++++++++\n pack-bitmap.h | 1 +\n 2 files changed, 129 insertions(+)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex e06930e10b9..d4894ace9ee 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -18,6 +18,7 @@\n #include \"tree.h\"\n #include \"tree-walk.h\"\n #include \"pseudo-merge.h\"\n+#include \"oid-array.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -748,6 +749,127 @@ static void write_selected_commits_v1(struct hashfile *f,\n \t}\n }\n \n+static void write_pseudo_merges(struct hashfile *f)\n+{\n+\tstruct oid_array commits = OID_ARRAY_INIT;\n+\tstruct bitmap **commits_bitmap = NULL;\n+\toff_t *pseudo_merge_ofs = NULL;\n+\toff_t start, table_start, next_ext;\n+\n+\tuint32_t base = bitmap_writer_selected_nr();\n+\tsize_t i, j = 0;\n+\n+\tCALLOC_ARRAY(commits_bitmap, writer.pseudo_merges_nr);\n+\tCALLOC_ARRAY(pseudo_merge_ofs, writer.pseudo_merges_nr);\n+\n+\tfor (i = 0; i < writer.pseudo_merges_nr; i++) {\n+\t\tstruct bitmapped_commit *merge = &writer.selected[base + i];\n+\t\tstruct commit_list *p;\n+\n+\t\tif (!merge->pseudo_merge)\n+\t\t\tBUG(\"found non-pseudo merge commit at %\"PRIuMAX, (uintmax_t)i);\n+\n+\t\tcommits_bitmap[i] = bitmap_new();\n+\n+\t\tfor (p = merge->commit->parents; p; p = p->next)\n+\t\t\tbitmap_set(commits_bitmap[i],\n+\t\t\t\t find_object_pos(&p->item->object.oid, NULL));\n+\t}\n+\n+\tstart = hashfile_total(f);\n+\n+\tfor (i = 0; i < writer.pseudo_merges_nr; i++) {\n+\t\tstruct ewah_bitmap *commits_ewah = bitmap_to_ewah(commits_bitmap[i]);\n+\n+\t\tpseudo_merge_ofs[i] = hashfile_total(f);\n+\n+\t\tdump_bitmap(f, commits_ewah);\n+\t\tdump_bitmap(f, writer.selected[base+i].write_as);\n+\n+\t\tewah_free(commits_ewah);\n+\t}\n+\n+\tnext_ext = st_add(hashfile_total(f),\n+\t\t\t st_mult(kh_size(writer.pseudo_merge_commits),\n+\t\t\t\t sizeof(uint64_t)));\n+\n+\ttable_start = hashfile_total(f);\n+\n+\tcommits.alloc = kh_size(writer.pseudo_merge_commits);\n+\tCALLOC_ARRAY(commits.oid, commits.alloc);\n+\n+\tfor (i = kh_begin(writer.pseudo_merge_commits); i != kh_end(writer.pseudo_merge_commits); i++) {\n+\t\tif (!kh_exist(writer.pseudo_merge_commits, i))\n+\t\t\tcontinue;\n+\t\toid_array_append(&commits, &kh_key(writer.pseudo_merge_commits, i));\n+\t}\n+\n+\toid_array_sort(&commits);\n+\n+\t/* write lookup table (non-extended) */\n+\tfor (i = 0; i < commits.nr; i++) {\n+\t\tint hash_pos;\n+\t\tstruct pseudo_merge_commit_idx *c;\n+\n+\t\thash_pos = kh_get_oid_map(writer.pseudo_merge_commits,\n+\t\t\t\t\t commits.oid[i]);\n+\t\tif (hash_pos == kh_end(writer.pseudo_merge_commits))\n+\t\t\tBUG(\"could not find pseudo-merge commit %s\",\n+\t\t\t oid_to_hex(&commits.oid[i]));\n+\n+\t\tc = kh_value(writer.pseudo_merge_commits, hash_pos);\n+\n+\t\thashwrite_be32(f, find_object_pos(&commits.oid[i], NULL));\n+\t\tif (c->nr == 1)\n+\t\t\thashwrite_be64(f, pseudo_merge_ofs[c->pseudo_merge[0]]);\n+\t\telse if (c->nr > 1) {\n+\t\t\tif (next_ext & ((uint64_t)1<<63))\n+\t\t\t\tdie(_(\"too many pseudo-merges\"));\n+\t\t\thashwrite_be64(f, next_ext | ((uint64_t)1<<63));\n+\t\t\tnext_ext = st_add3(next_ext,\n+\t\t\t\t\t sizeof(uint32_t),\n+\t\t\t\t\t st_mult(c->nr, sizeof(uint64_t)));\n+\t\t} else\n+\t\t\tBUG(\"expected commit '%s' to have at least one \"\n+\t\t\t \"pseudo-merge\", oid_to_hex(&commits.oid[i]));\n+\t}\n+\n+\t/* write lookup table (extended) */\n+\tfor (i = 0; i < commits.nr; i++) {\n+\t\tint hash_pos;\n+\t\tstruct pseudo_merge_commit_idx *c;\n+\n+\t\thash_pos = kh_get_oid_map(writer.pseudo_merge_commits,\n+\t\t\t\t\t commits.oid[i]);\n+\t\tif (hash_pos == kh_end(writer.pseudo_merge_commits))\n+\t\t\tBUG(\"could not find pseudo-merge commit %s\",\n+\t\t\t oid_to_hex(&commits.oid[i]));\n+\n+\t\tc = kh_value(writer.pseudo_merge_commits, hash_pos);\n+\t\tif (c->nr == 1)\n+\t\t\tcontinue;\n+\n+\t\thashwrite_be32(f, c->nr);\n+\t\tfor (j = 0; j < c->nr; j++)\n+\t\t\thashwrite_be64(f, pseudo_merge_ofs[c->pseudo_merge[j]]);\n+\t}\n+\n+\t/* write positions for all pseudo merges */\n+\tfor (i = 0; i < writer.pseudo_merges_nr; i++)\n+\t\thashwrite_be64(f, pseudo_merge_ofs[i]);\n+\n+\thashwrite_be32(f, writer.pseudo_merges_nr);\n+\thashwrite_be32(f, kh_size(writer.pseudo_merge_commits));\n+\thashwrite_be64(f, table_start - start);\n+\thashwrite_be64(f, hashfile_total(f) - start + sizeof(uint64_t));\n+\n+\tfor (i = 0; i < writer.pseudo_merges_nr; i++)\n+\t\tbitmap_free(commits_bitmap[i]);\n+\n+\tfree(pseudo_merge_ofs);\n+\tfree(commits_bitmap);\n+}\n+\n static int table_cmp(const void *_va, const void *_vb, void *_data)\n {\n \tuint32_t *commit_positions = _data;\n@@ -855,6 +977,9 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \n \tint fd = odb_mkstemp(&tmp_file, \"pack/tmp_bitmap_XXXXXX\");\n \n+\tif (writer.pseudo_merges_nr)\n+\t\toptions |= BITMAP_OPT_PSEUDO_MERGES;\n+\n \tf = hashfd(fd, tmp_file.buf);\n \n \tmemcpy(header.magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE));\n@@ -886,6 +1011,9 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \n \twrite_selected_commits_v1(f, commit_positions, offsets);\n \n+\tif (options & BITMAP_OPT_PSEUDO_MERGES)\n+\t\twrite_pseudo_merges(f);\n+\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n \t\twrite_lookup_table(f, commit_positions, offsets);\n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 0f539d79cfd..55527f61cd9 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -37,6 +37,7 @@ enum pack_bitmap_opts {\n \tBITMAP_OPT_FULL_DAG = 0x1,\n \tBITMAP_OPT_HASH_CACHE = 0x4,\n \tBITMAP_OPT_LOOKUP_TABLE = 0x10,\n+\tBITMAP_OPT_PSEUDO_MERGES = 0x20,\n };\n \n enum pack_bitmap_flags {\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493695","messageId":"60f6b310213b86de25ecf23002cee7a3c8460415.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 12/23] pack-bitmap: extract `read_bitmap()` function","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:43:48Z","receivedAt":"2024-04-29T20:43:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pack-bitmap machinery uses the `read_bitmap_1()` function to read a\nbitmap from within the mmap'd region corresponding to the .bitmap file.\nAs as side-effect of calling this function, `read_bitmap_1()` increments\nthe `index->map_pos` variable to reflect the number of bytes read.\n\nExtract the core of this routine to a separate function (that operates\nover a `const unsigned char *`, a `size_t` and a `size_t *` pointer)\ninstead of a `struct bitmap_index *` pointer.\n\nThis function (called `read_bitmap()`) is part of the pack-bitmap.h API\nso that it can be used within the upcoming portion of the implementation\nin pseduo-merge.ch.\n\nRewrite the existing function, `read_bitmap_1()`, in terms of its more\ngeneric counterpart.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 24 +++++++++++++++---------\n pack-bitmap.h | 2 ++\n 2 files changed, 17 insertions(+), 9 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 35c5ef9d3cd..3519edb896b 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -129,17 +129,13 @@ static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st)\n \treturn composed;\n }\n \n-/*\n- * Read a bitmap from the current read position on the mmaped\n- * index, and increase the read position accordingly\n- */\n-static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n+struct ewah_bitmap *read_bitmap(const unsigned char *map,\n+\t\t\t\tsize_t map_size, size_t *map_pos)\n {\n \tstruct ewah_bitmap *b = ewah_pool_new();\n \n-\tssize_t bitmap_size = ewah_read_mmap(b,\n-\t\tindex->map + index->map_pos,\n-\t\tindex->map_size - index->map_pos);\n+\tssize_t bitmap_size = ewah_read_mmap(b, map + *map_pos,\n+\t\t\t\t\t map_size - *map_pos);\n \n \tif (bitmap_size < 0) {\n \t\terror(_(\"failed to load bitmap index (corrupted?)\"));\n@@ -147,10 +143,20 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \t\treturn NULL;\n \t}\n \n-\tindex->map_pos += bitmap_size;\n+\t*map_pos += bitmap_size;\n+\n \treturn b;\n }\n \n+/*\n+ * Read a bitmap from the current read position on the mmaped\n+ * index, and increase the read position accordingly\n+ */\n+static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n+{\n+\treturn read_bitmap(index->map, index->map_size, &index->map_pos);\n+}\n+\n static uint32_t bitmap_num_objects(struct bitmap_index *index)\n {\n \tif (index->midx)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 55527f61cd9..a5fe4f305ef 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -133,4 +133,6 @@ int bitmap_is_preferred_refname(struct repository *r, const char *refname);\n \n int verify_bitmap_files(struct repository *r);\n \n+struct ewah_bitmap *read_bitmap(const unsigned char *map,\n+\t\t\t\tsize_t map_size, size_t *map_pos);\n #endif\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493696","messageId":"9465313691be43c00c6f03c0be58710edbeb9caf.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 13/23] pseudo-merge: scaffolding for reads","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:43:52Z","receivedAt":"2024-04-29T20:43:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement scaffolding within the new pseudo-merge compilation unit\nnecessary to use the pseudo-merge API from within the pack-bitmap.c\nmachinery.\n\nThe core of this scaffolding is two-fold:\n\n - The `pseudo_merge` structure itself, which represents an individual\n pseudo-merge bitmap. It has fields for both bitmaps, as well as\n metadata about its position within the memory-mapped region, and\n a few extra bits indicating whether or not it is satisfied, and\n which bitmaps(s, if any) have been read, since they are initialized\n lazily.\n\n - The `pseudo_merge_map` structure, which holds an array of\n pseudo_merges, as well as a pointer to the memory-mapped region\n containing the pseudo-merge serialization from within a .bitmap\n file.\n\nNote that the `bitmap_index` structure is defined statically within the\npack-bitmap.o compilation unit, so we can't take in a `struct\nbitmap_index *`. Instead, wrap the primary components necessary to read\nthe pseudo-merges in this new structure to avoid exposing the\nimplementation details of the `bitmap_index` structure.\n\nSigned-off-by: Taylor Blau \n---\n pseudo-merge.c | 10 ++++++++\n pseudo-merge.h | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 75 insertions(+)\n\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex caccef942a1..d18de0a266b 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -441,3 +441,13 @@ void select_pseudo_merges(struct string_list *list,\n \n \tstop_progress(&progress);\n }\n+\n+void free_pseudo_merge_map(struct pseudo_merge_map *pm)\n+{\n+\tuint32_t i;\n+\tfor (i = 0; i < pm->nr; i++) {\n+\t\tewah_pool_free(pm->v[i].commits);\n+\t\tewah_pool_free(pm->v[i].bitmap);\n+\t}\n+\tfree(pm->v);\n+}\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nindex 81888731864..2f652fc6767 100644\n--- a/pseudo-merge.h\n+++ b/pseudo-merge.h\n@@ -99,4 +99,69 @@ void select_pseudo_merges(struct string_list *list,\n \t\t\t uint32_t *pseudo_merges_nr,\n \t\t\t unsigned show_progress);\n \n+/*\n+ * Represents a serialized view of a file containing pseudo-merge(s)\n+ * (see Documentation/technical/bitmap-format.txt for a specification\n+ * of the format).\n+ */\n+struct pseudo_merge_map {\n+\t/*\n+\t * An array of pseudo-merge(s), lazily loaded from the .bitmap\n+\t * file.\n+\t */\n+\tstruct pseudo_merge *v;\n+\tsize_t nr;\n+\tsize_t commits_nr;\n+\n+\t/*\n+\t * Pointers into a memory-mapped view of the .bitmap file:\n+\t *\n+\t * - map: the beginning of the .bitmap file\n+\t * - commits: the beginning of the pseudo-merge commit index\n+\t * - map_size: the size of the .bitmap file\n+\t */\n+\tconst unsigned char *map;\n+\tconst unsigned char *commits;\n+\n+\tsize_t map_size;\n+};\n+\n+/*\n+ * An individual pseudo-merge, storing a pair of lazily-loaded\n+ * bitmaps:\n+ *\n+ * - commits: the set of commit(s) that are part of the pseudo-merge\n+ * - bitmap: the set of object(s) reachable from the above set of\n+ * commits.\n+ *\n+ * The `at` and `bitmap_at` fields are used to store the locations of\n+ * each of the above bitmaps in the .bitmap file.\n+ */\n+struct pseudo_merge {\n+\tstruct ewah_bitmap *commits;\n+\tstruct ewah_bitmap *bitmap;\n+\n+\toff_t at;\n+\toff_t bitmap_at;\n+\n+\t/*\n+\t * `satisfied` indicates whether the given pseudo-merge has been\n+\t * used.\n+\t *\n+\t * `loaded_commits` and `loaded_bitmap` indicate whether the\n+\t * respective bitmaps have been loaded and read from the\n+\t * .bitmap file.\n+\t */\n+\tunsigned satisfied : 1,\n+\t\t loaded_commits : 1,\n+\t\t loaded_bitmap : 1;\n+};\n+\n+/*\n+ * Frees the given pseudo-merge map, releasing any memory held by (a)\n+ * parsed EWAH bitmaps, or (b) the array of pseudo-merges itself. Does\n+ * not free the memory-mapped view of the .bitmap file.\n+ */\n+void free_pseudo_merge_map(struct pseudo_merge_map *pm);\n+\n #endif\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493697","messageId":"5894f3d536980abbb4115aa842e27995e93326af.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 14/23] pack-bitmap.c: read pseudo-merge extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:43:56Z","receivedAt":"2024-04-29T20:44:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the scaffolding for reading the pseudo-merge extension has been\nlaid, teach the pack-bitmap machinery to read the pseudo-merge extension\nwhen present.\n\nNote that pseudo-merges themselves are not yet used during traversal,\nthis step will be taken by a future commit.\n\nIn the meantime, read the table and initialize the pseudo_merge_map\nstructure introduced by a previous commit. When the pseudo-merge\nextension is present, `load_bitmap_header()` performs basic sanity\nchecks to make sure that the table is well-formed.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 39 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 39 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 3519edb896b..fc9c3e2fc43 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -20,6 +20,7 @@\n #include \"list-objects-filter-options.h\"\n #include \"midx.h\"\n #include \"config.h\"\n+#include \"pseudo-merge.h\"\n \n /*\n * An entry on the bitmap index, representing the bitmap for a given\n@@ -86,6 +87,9 @@ struct bitmap_index {\n \t */\n \tunsigned char *table_lookup;\n \n+\t/* This contains the pseudo-merge cache within 'map' (if found). */\n+\tstruct pseudo_merge_map pseudo_merges;\n+\n \t/*\n \t * Extended index.\n \t *\n@@ -205,6 +209,41 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t\t\t\tindex->table_lookup = (void *)(index_end - table_size);\n \t\t\tindex_end -= table_size;\n \t\t}\n+\n+\t\tif (flags & BITMAP_OPT_PSEUDO_MERGES) {\n+\t\t\tunsigned char *pseudo_merge_ofs;\n+\t\t\tsize_t table_size;\n+\t\t\tuint32_t i;\n+\n+\t\t\tif (sizeof(table_size) > index_end - index->map - header_size)\n+\t\t\t\treturn error(_(\"corrupted bitmap index file (too short to fit pseudo-merge table header)\"));\n+\n+\t\t\ttable_size = get_be64(index_end - 8);\n+\t\t\tif (table_size > index_end - index->map - header_size)\n+\t\t\t\treturn error(_(\"corrupted bitmap index file (too short to fit pseudo-merge table)\"));\n+\n+\t\t\tif (git_env_bool(\"GIT_TEST_USE_PSEUDO_MERGES\", 1)) {\n+\t\t\t\tconst unsigned char *ext = (index_end - table_size);\n+\n+\t\t\t\tindex->pseudo_merges.map = index->map;\n+\t\t\t\tindex->pseudo_merges.map_size = index->map_size;\n+\t\t\t\tindex->pseudo_merges.commits = ext + get_be64(index_end - 16);\n+\t\t\t\tindex->pseudo_merges.commits_nr = get_be32(index_end - 20);\n+\t\t\t\tindex->pseudo_merges.nr = get_be32(index_end - 24);\n+\n+\t\t\t\tCALLOC_ARRAY(index->pseudo_merges.v,\n+\t\t\t\t\t index->pseudo_merges.nr);\n+\n+\t\t\t\tpseudo_merge_ofs = index_end - 24 -\n+\t\t\t\t\t(index->pseudo_merges.nr * sizeof(uint64_t));\n+\t\t\t\tfor (i = 0; i < index->pseudo_merges.nr; i++) {\n+\t\t\t\t\tindex->pseudo_merges.v[i].at = get_be64(pseudo_merge_ofs);\n+\t\t\t\t\tpseudo_merge_ofs += sizeof(uint64_t);\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tindex_end -= table_size;\n+\t\t}\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493698","messageId":"7dbee8bcbdf2b4f7f35fb76c3d5b0ad6a465539e.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 15/23] pseudo-merge: implement support for reading pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:44:00Z","receivedAt":"2024-04-29T20:44:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement the basic API for reading pseudo-merge bitmaps, which consists\nof four basic functions:\n\n - pseudo_merge_bitmap()\n - use_pseudo_merge()\n - apply_pseudo_merges_for_commit()\n - cascade_pseudo_merges()\n\nThese functions are all documented in pseudo-merge.h, but their rough\ndescriptions are as follows:\n\n - pseudo_merge_bitmap() reads and inflates the objects EWAH bitmap for\n a given pseudo-merge\n\n - use_pseudo_merge() does the same as pseudo_merge_bitmap(), but on\n the commits EWAH bitmap, not the objects bitmap\n\n - apply_pseudo_merges_for_commit() applies all satisfied pseudo-merge\n commits for a given result set, and cascades any yet-unsatisfied\n pseudo-merges if any were applied in the previous step\n\n - cascade_pseudo_merges() applies all pseudo-merges which are\n satisfied but have not been previously applied, repeating this\n process until no more pseudo-merges can be applied\n\nThe core of the API is the latter two functions, which are responsible\nfor applying pseudo-merges during the object traversal implemented in\nthe pack-bitmap machinery.\n\nThe other two functions (pseudo_merge_bitmap(), and use_pseudo_merge())\nare low-level ways to interact with the pseudo-merge machinery, which\nwill be useful in future commits.\n\nSigned-off-by: Taylor Blau \n---\n pseudo-merge.c | 231 +++++++++++++++++++++++++++++++++++++++++++++++++\n pseudo-merge.h | 44 ++++++++++\n 2 files changed, 275 insertions(+)\n\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex d18de0a266b..e111c9cd1a6 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -10,6 +10,7 @@\n #include \"commit.h\"\n #include \"alloc.h\"\n #include \"progress.h\"\n+#include \"hex.h\"\n \n #define DEFAULT_PSEUDO_MERGE_DECAY 1.0f\n #define DEFAULT_PSEUDO_MERGE_MAX_MERGES 64\n@@ -451,3 +452,233 @@ void free_pseudo_merge_map(struct pseudo_merge_map *pm)\n \t}\n \tfree(pm->v);\n }\n+\n+struct pseudo_merge_commit_ext {\n+\tuint32_t nr;\n+\tconst unsigned char *ptr;\n+};\n+\n+static int pseudo_merge_ext_at(const struct pseudo_merge_map *pm,\n+\t\t\t struct pseudo_merge_commit_ext *ext, size_t at)\n+{\n+\tif (at >= pm->map_size)\n+\t\treturn error(_(\"extended pseudo-merge read out-of-bounds \"\n+\t\t\t \"(%\"PRIuMAX\" >= %\"PRIuMAX\")\"),\n+\t\t\t (uintmax_t)at, (uintmax_t)pm->map_size);\n+\n+\text->nr = get_be32(pm->map + at);\n+\text->ptr = pm->map + at + sizeof(uint32_t);\n+\n+\treturn 0;\n+}\n+\n+struct ewah_bitmap *pseudo_merge_bitmap(const struct pseudo_merge_map *pm,\n+\t\t\t\t\tstruct pseudo_merge *merge)\n+{\n+\tif (!merge->loaded_commits)\n+\t\tBUG(\"cannot use unloaded pseudo-merge bitmap\");\n+\n+\tif (!merge->loaded_bitmap) {\n+\t\tsize_t at = merge->bitmap_at;\n+\n+\t\tmerge->bitmap = read_bitmap(pm->map, pm->map_size, &at);\n+\t\tmerge->loaded_bitmap = 1;\n+\t}\n+\n+\treturn merge->bitmap;\n+}\n+\n+struct pseudo_merge *use_pseudo_merge(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct pseudo_merge *merge)\n+{\n+\tif (!merge->loaded_commits) {\n+\t\tsize_t pos = merge->at;\n+\n+\t\tmerge->commits = read_bitmap(pm->map, pm->map_size, &pos);\n+\t\tmerge->bitmap_at = pos;\n+\t\tmerge->loaded_commits = 1;\n+\t}\n+\treturn merge;\n+}\n+\n+static struct pseudo_merge *pseudo_merge_at(const struct pseudo_merge_map *pm,\n+\t\t\t\t\t struct object_id *oid,\n+\t\t\t\t\t size_t want)\n+{\n+\tsize_t lo = 0;\n+\tsize_t hi = pm->nr;\n+\n+\twhile (lo < hi) {\n+\t\tsize_t mi = lo + (hi - lo) / 2;\n+\t\tsize_t got = pm->v[mi].at;\n+\n+\t\tif (got == want)\n+\t\t\treturn use_pseudo_merge(pm, &pm->v[mi]);\n+\t\telse if (got < want)\n+\t\t\thi = mi;\n+\t\telse\n+\t\t\tlo = mi + 1;\n+\t}\n+\n+\twarning(_(\"could not find pseudo-merge for commit %s at offset %\"PRIuMAX),\n+\t\toid_to_hex(oid), (uintmax_t)want);\n+\n+\treturn NULL;\n+}\n+\n+struct pseudo_merge_commit {\n+\tuint32_t commit_pos;\n+\tuint64_t pseudo_merge_ofs;\n+};\n+\n+#define PSEUDO_MERGE_COMMIT_RAWSZ (sizeof(uint32_t)+sizeof(uint64_t))\n+\n+static void read_pseudo_merge_commit_at(struct pseudo_merge_commit *merge,\n+\t\t\t\t\tconst unsigned char *at)\n+{\n+\tmerge->commit_pos = get_be32(at);\n+\tmerge->pseudo_merge_ofs = get_be64(at + sizeof(uint32_t));\n+}\n+\n+static int nth_pseudo_merge_ext(const struct pseudo_merge_map *pm,\n+\t\t\t\tstruct pseudo_merge_commit_ext *ext,\n+\t\t\t\tstruct pseudo_merge_commit *merge,\n+\t\t\t\tuint32_t n)\n+{\n+\tsize_t ofs;\n+\n+\tif (n >= ext->nr)\n+\t\treturn error(_(\"extended pseudo-merge lookup out-of-bounds \"\n+\t\t\t \"(%\"PRIu32\" >= %\"PRIu32\")\"), n, ext->nr);\n+\n+\tofs = get_be64(ext->ptr + st_mult(n, sizeof(uint64_t)));\n+\tif (ofs >= pm->map_size)\n+\t\treturn error(_(\"out-of-bounds read: (%\"PRIuMAX\" >= %\"PRIuMAX\")\"),\n+\t\t\t (uintmax_t)ofs, (uintmax_t)pm->map_size);\n+\n+\tread_pseudo_merge_commit_at(merge, pm->map + ofs);\n+\n+\treturn 0;\n+}\n+\n+static unsigned apply_pseudo_merge(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct pseudo_merge *merge,\n+\t\t\t\t struct bitmap *result,\n+\t\t\t\t struct bitmap *roots)\n+{\n+\tif (merge->satisfied)\n+\t\treturn 0;\n+\n+\tif (!ewah_bitmap_is_subset(merge->commits, roots ? roots : result))\n+\t\treturn 0;\n+\n+\tbitmap_or_ewah(result, pseudo_merge_bitmap(pm, merge));\n+\tif (roots)\n+\t\tbitmap_or_ewah(roots, pseudo_merge_bitmap(pm, merge));\n+\tmerge->satisfied = 1;\n+\n+\treturn 1;\n+}\n+\n+static int pseudo_merge_commit_cmp(const void *va, const void *vb)\n+{\n+\tstruct pseudo_merge_commit merge;\n+\tuint32_t key = *(uint32_t*)va;\n+\n+\tread_pseudo_merge_commit_at(&merge, vb);\n+\n+\tif (key < merge.commit_pos)\n+\t\treturn -1;\n+\tif (key > merge.commit_pos)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static struct pseudo_merge_commit *find_pseudo_merge(const struct pseudo_merge_map *pm,\n+\t\t\t\t\t\t uint32_t pos)\n+{\n+\tif (!pm->commits_nr)\n+\t\treturn NULL;\n+\n+\treturn bsearch(&pos, pm->commits, pm->commits_nr,\n+\t\t PSEUDO_MERGE_COMMIT_RAWSZ, pseudo_merge_commit_cmp);\n+}\n+\n+int apply_pseudo_merges_for_commit(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct bitmap *result,\n+\t\t\t\t struct commit *commit, uint32_t commit_pos)\n+{\n+\tstruct pseudo_merge *merge;\n+\tstruct pseudo_merge_commit *merge_commit;\n+\tint ret = 0;\n+\n+\tmerge_commit = find_pseudo_merge(pm, commit_pos);\n+\tif (!merge_commit)\n+\t\treturn 0;\n+\n+\tif (merge_commit->pseudo_merge_ofs & ((uint64_t)1<<63)) {\n+\t\tstruct pseudo_merge_commit_ext ext = { 0 };\n+\t\toff_t ofs = merge_commit->pseudo_merge_ofs & ~((uint64_t)1<<63);\n+\t\tuint32_t i;\n+\n+\t\tif (pseudo_merge_ext_at(pm, &ext, ofs) < -1) {\n+\t\t\twarning(_(\"could not read extended pseudo-merge table \"\n+\t\t\t\t \"for commit %s\"),\n+\t\t\t\toid_to_hex(&commit->object.oid));\n+\t\t\treturn ret;\n+\t\t}\n+\n+\t\tfor (i = 0; i < ext.nr; i++) {\n+\t\t\tif (nth_pseudo_merge_ext(pm, &ext, merge_commit, i) < 0)\n+\t\t\t\treturn ret;\n+\n+\t\t\tmerge = pseudo_merge_at(pm, &commit->object.oid,\n+\t\t\t\t\t\tmerge_commit->pseudo_merge_ofs);\n+\n+\t\t\tif (!merge)\n+\t\t\t\treturn ret;\n+\n+\t\t\tif (apply_pseudo_merge(pm, merge, result, NULL))\n+\t\t\t\tret++;\n+\t\t}\n+\t} else {\n+\t\tmerge = pseudo_merge_at(pm, &commit->object.oid,\n+\t\t\t\t\tmerge_commit->pseudo_merge_ofs);\n+\n+\t\tif (!merge)\n+\t\t\treturn ret;\n+\n+\t\tif (apply_pseudo_merge(pm, merge, result, NULL))\n+\t\t\tret++;\n+\t}\n+\n+\tif (ret)\n+\t\tcascade_pseudo_merges(pm, result, NULL);\n+\n+\treturn ret;\n+}\n+\n+int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n+\t\t\t struct bitmap *result,\n+\t\t\t struct bitmap *roots)\n+{\n+\tunsigned any_satisfied;\n+\tint ret = 0;\n+\n+\tdo {\n+\t\tstruct pseudo_merge *merge;\n+\t\tuint32_t i;\n+\n+\t\tany_satisfied = 0;\n+\n+\t\tfor (i = 0; i < pm->nr; i++) {\n+\t\t\tmerge = use_pseudo_merge(pm, &pm->v[i]);\n+\t\t\tif (apply_pseudo_merge(pm, merge, result, roots)) {\n+\t\t\t\tany_satisfied |= 1;\n+\t\t\t\tret++;\n+\t\t\t}\n+\t\t}\n+\t} while (any_satisfied);\n+\n+\treturn ret;\n+}\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nindex 2f652fc6767..cc14e947e86 100644\n--- a/pseudo-merge.h\n+++ b/pseudo-merge.h\n@@ -164,4 +164,48 @@ struct pseudo_merge {\n */\n void free_pseudo_merge_map(struct pseudo_merge_map *pm);\n \n+/*\n+ * Loads the bitmap corresponding to the given pseudo-merge from the\n+ * map, if it has not already been loaded.\n+ */\n+struct ewah_bitmap *pseudo_merge_bitmap(const struct pseudo_merge_map *pm,\n+\t\t\t\t\tstruct pseudo_merge *merge);\n+\n+/*\n+ * Loads the pseudo-merge and its commits bitmap from the given\n+ * pseudo-merge map, if it has not already been loaded.\n+ */\n+struct pseudo_merge *use_pseudo_merge(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct pseudo_merge *merge);\n+\n+/*\n+ * Applies pseudo-merge(s) containing the given commit to the bitmap\n+ * \"result\".\n+ *\n+ * If any pseudo-merge(s) were satisfied, returns the number\n+ * satisfied, otherwise returns 0. If any were satisfied, the\n+ * remaining unsatisfied pseudo-merges are cascaded (see below).\n+ */\n+int apply_pseudo_merges_for_commit(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct bitmap *result,\n+\t\t\t\t struct commit *commit, uint32_t commit_pos);\n+\n+/*\n+ * Applies pseudo-merge(s) which are satisfied according to the\n+ * current bitmap in result (or roots, see below). If any\n+ * pseudo-merges were satisfied, repeat the process over unsatisfied\n+ * pseudo-merge commits until no more pseudo-merges are satisfied.\n+ *\n+ * Result is the bitmap to which the pseudo-merge(s) are applied.\n+ * Roots (if given) is a bitmap of the traversal tip(s) for either\n+ * side of a reachability traversal.\n+ *\n+ * Roots may given instead of a populated results bitmap at the\n+ * beginning of a traversal on either side where the reachability\n+ * closure over tips is not yet known.\n+ */\n+int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n+\t\t\t struct bitmap *result,\n+\t\t\t struct bitmap *roots);\n+\n #endif\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493699","messageId":"09650aa53e9207191e7607f9f7ec77e2e7bc796b.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 16/23] ewah: implement `ewah_bitmap_popcount()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:44:03Z","receivedAt":"2024-04-29T20:44:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Some of the pseudo-merge test helpers (which will be introduced in the\nfollowing commit) will want to indicate the total number of commits in\nor objects reachable from a pseudo-merge.\n\nImplement a popcount() function that operates on EWAH bitmaps to quickly\ndetermine how many bits are set in each of the respective bitmaps.\n\nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 14 ++++++++++++++\n ewah/ewok.h | 1 +\n 2 files changed, 15 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex d352fec54ce..dc2ca190f12 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -212,6 +212,20 @@ size_t bitmap_popcount(struct bitmap *self)\n \treturn count;\n }\n \n+size_t ewah_bitmap_popcount(struct ewah_bitmap *self)\n+{\n+\tstruct ewah_iterator it;\n+\teword_t word;\n+\tsize_t count = 0;\n+\n+\tewah_iterator_init(&it, self);\n+\n+\twhile (ewah_iterator_next(&word, &it))\n+\t\tcount += ewah_bit_popcount64(word);\n+\n+\treturn count;\n+}\n+\n int bitmap_is_empty(struct bitmap *self)\n {\n \tsize_t i;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 2b6c4ac499c..7074a6347b7 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -195,6 +195,7 @@ void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other);\n void bitmap_or(struct bitmap *self, const struct bitmap *other);\n \n size_t bitmap_popcount(struct bitmap *self);\n+size_t ewah_bitmap_popcount(struct ewah_bitmap *self);\n int bitmap_is_empty(struct bitmap *self);\n \n #endif\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493700","messageId":"7b5ea56d053bc023780c34385570be4617da6705.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 17/23] pack-bitmap: implement test helpers for pseudo-merge","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:44:07Z","receivedAt":"2024-04-29T20:44:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement three new sub-commands for the \"bitmap\" test-helper:\n\n - t/helper test-tool bitmap dump-pseudo-merges\n - t/helper test-tool bitmap dump-pseudo-merge-commits \n - t/helper test-tool bitmap dump-pseudo-merge-objects \n\nThese three helpers dump the list of pseudo merges, the \"parents\" of the\nnth pseudo-merges, and the set of objects reachable from those parents,\nrespectively.\n\nThese helpers will be useful in subsequent patches when we add test\ncoverage for pseudo-merge bitmaps.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 126 +++++++++++++++++++++++++++++++++++++++++\n pack-bitmap.h | 3 +\n t/helper/test-bitmap.c | 34 ++++++++---\n 3 files changed, 156 insertions(+), 7 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex fc9c3e2fc43..c13074673af 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -2443,6 +2443,132 @@ int test_bitmap_hashes(struct repository *r)\n \treturn 0;\n }\n \n+static void bit_pos_to_object_id(struct bitmap_index *bitmap_git,\n+\t\t\t\t uint32_t bit_pos,\n+\t\t\t\t struct object_id *oid)\n+{\n+\tuint32_t index_pos;\n+\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tindex_pos = pack_pos_to_midx(bitmap_git->midx, bit_pos);\n+\telse\n+\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, bit_pos);\n+\n+\tnth_bitmap_object_oid(bitmap_git, oid, index_pos);\n+}\n+\n+int test_bitmap_pseudo_merges(struct repository *r)\n+{\n+\tstruct bitmap_index *bitmap_git;\n+\tuint32_t i;\n+\n+\tbitmap_git = prepare_bitmap_git(r);\n+\tif (!bitmap_git || !bitmap_git->pseudo_merges.nr)\n+\t\tgoto cleanup;\n+\n+\tfor (i = 0; i < bitmap_git->pseudo_merges.nr; i++) {\n+\t\tstruct pseudo_merge *merge;\n+\t\tstruct ewah_bitmap *commits_bitmap, *merge_bitmap;\n+\n+\t\tmerge = use_pseudo_merge(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t &bitmap_git->pseudo_merges.v[i]);\n+\t\tcommits_bitmap = merge->commits;\n+\t\tmerge_bitmap = pseudo_merge_bitmap(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t\t merge);\n+\n+\t\tprintf(\"at=%\"PRIuMAX\", commits=%\"PRIuMAX\", objects=%\"PRIuMAX\"\\n\",\n+\t\t (uintmax_t)merge->at,\n+\t\t (uintmax_t)ewah_bitmap_popcount(commits_bitmap),\n+\t\t (uintmax_t)ewah_bitmap_popcount(merge_bitmap));\n+\t}\n+\n+cleanup:\n+\tfree_bitmap_index(bitmap_git);\n+\treturn 0;\n+}\n+\n+static void dump_ewah_object_ids(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct ewah_bitmap *bitmap)\n+\n+{\n+\tstruct ewah_iterator it;\n+\teword_t word;\n+\tuint32_t pos = 0;\n+\n+\tewah_iterator_init(&it, bitmap);\n+\n+\twhile (ewah_iterator_next(&word, &it)) {\n+\t\tstruct object_id oid;\n+\t\tuint32_t offset;\n+\n+\t\tfor (offset = 0; offset < BITS_IN_EWORD; offset++) {\n+\t\t\tif (!(word >> offset))\n+\t\t\t\tbreak;\n+\n+\t\t\toffset += ewah_bit_ctz64(word >> offset);\n+\n+\t\t\tbit_pos_to_object_id(bitmap_git, pos + offset, &oid);\n+\t\t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n+\t\t}\n+\t\tpos += BITS_IN_EWORD;\n+\t}\n+}\n+\n+int test_bitmap_pseudo_merge_commits(struct repository *r, uint32_t n)\n+{\n+\tstruct bitmap_index *bitmap_git;\n+\tstruct pseudo_merge *merge;\n+\tint ret = 0;\n+\n+\tbitmap_git = prepare_bitmap_git(r);\n+\tif (!bitmap_git || !bitmap_git->pseudo_merges.nr)\n+\t\tgoto cleanup;\n+\n+\tif (n >= bitmap_git->pseudo_merges.nr) {\n+\t\tret = error(_(\"pseudo-merge index out of range \"\n+\t\t\t \"(%\"PRIu32\" >= %\"PRIuMAX\")\"),\n+\t\t\t n, (uintmax_t)bitmap_git->pseudo_merges.nr);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tmerge = use_pseudo_merge(&bitmap_git->pseudo_merges,\n+\t\t\t\t &bitmap_git->pseudo_merges.v[n]);\n+\tdump_ewah_object_ids(bitmap_git, merge->commits);\n+\n+cleanup:\n+\tfree_bitmap_index(bitmap_git);\n+\treturn ret;\n+}\n+\n+int test_bitmap_pseudo_merge_objects(struct repository *r, uint32_t n)\n+{\n+\tstruct bitmap_index *bitmap_git;\n+\tstruct pseudo_merge *merge;\n+\tint ret = 0;\n+\n+\tbitmap_git = prepare_bitmap_git(r);\n+\tif (!bitmap_git || !bitmap_git->pseudo_merges.nr)\n+\t\tgoto cleanup;\n+\n+\tif (n >= bitmap_git->pseudo_merges.nr) {\n+\t\tret = error(_(\"pseudo-merge index out of range \"\n+\t\t\t \"(%\"PRIu32\" >= %\"PRIuMAX\")\"),\n+\t\t\t n, (uintmax_t)bitmap_git->pseudo_merges.nr);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tmerge = use_pseudo_merge(&bitmap_git->pseudo_merges,\n+\t\t\t\t &bitmap_git->pseudo_merges.v[n]);\n+\n+\tdump_ewah_object_ids(bitmap_git,\n+\t\t\t pseudo_merge_bitmap(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t\t merge));\n+\n+cleanup:\n+\tfree_bitmap_index(bitmap_git);\n+\treturn ret;\n+}\n+\n int rebuild_bitmap(const uint32_t *reposition,\n \t\t struct ewah_bitmap *source,\n \t\t struct bitmap *dest)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex a5fe4f305ef..25d3b8e604a 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -73,6 +73,9 @@ void traverse_bitmap_commit_list(struct bitmap_index *,\n void test_bitmap_walk(struct rev_info *revs);\n int test_bitmap_commits(struct repository *r);\n int test_bitmap_hashes(struct repository *r);\n+int test_bitmap_pseudo_merges(struct repository *r);\n+int test_bitmap_pseudo_merge_commits(struct repository *r, uint32_t n);\n+int test_bitmap_pseudo_merge_objects(struct repository *r, uint32_t n);\n \n #define GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL \\\n \t\"GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL\"\ndiff --git a/t/helper/test-bitmap.c b/t/helper/test-bitmap.c\nindex af43ee1cb5e..6af2b42678f 100644\n--- a/t/helper/test-bitmap.c\n+++ b/t/helper/test-bitmap.c\n@@ -13,21 +13,41 @@ static int bitmap_dump_hashes(void)\n \treturn test_bitmap_hashes(the_repository);\n }\n \n+static int bitmap_dump_pseudo_merges(void)\n+{\n+\treturn test_bitmap_pseudo_merges(the_repository);\n+}\n+\n+static int bitmap_dump_pseudo_merge_commits(uint32_t n)\n+{\n+\treturn test_bitmap_pseudo_merge_commits(the_repository, n);\n+}\n+\n+static int bitmap_dump_pseudo_merge_objects(uint32_t n)\n+{\n+\treturn test_bitmap_pseudo_merge_objects(the_repository, n);\n+}\n+\n int cmd__bitmap(int argc, const char **argv)\n {\n \tsetup_git_directory();\n \n-\tif (argc != 2)\n-\t\tgoto usage;\n-\n-\tif (!strcmp(argv[1], \"list-commits\"))\n+\tif (argc == 2 && !strcmp(argv[1], \"list-commits\"))\n \t\treturn bitmap_list_commits();\n-\tif (!strcmp(argv[1], \"dump-hashes\"))\n+\tif (argc == 2 && !strcmp(argv[1], \"dump-hashes\"))\n \t\treturn bitmap_dump_hashes();\n+\tif (argc == 2 && !strcmp(argv[1], \"dump-pseudo-merges\"))\n+\t\treturn bitmap_dump_pseudo_merges();\n+\tif (argc == 3 && !strcmp(argv[1], \"dump-pseudo-merge-commits\"))\n+\t\treturn bitmap_dump_pseudo_merge_commits(atoi(argv[2]));\n+\tif (argc == 3 && !strcmp(argv[1], \"dump-pseudo-merge-objects\"))\n+\t\treturn bitmap_dump_pseudo_merge_objects(atoi(argv[2]));\n \n-usage:\n \tusage(\"\\ttest-tool bitmap list-commits\\n\"\n-\t \"\\ttest-tool bitmap dump-hashes\");\n+\t \"\\ttest-tool bitmap dump-hashes\\n\"\n+\t \"\\ttest-tool bitmap dump-pseudo-merges\\n\"\n+\t \"\\ttest-tool bitmap dump-pseudo-merge-commits \\n\"\n+\t \"\\ttest-tool bitmap dump-pseudo-merge-objects \");\n \n \treturn -1;\n }\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493701","messageId":"006abdd1698999539eda7f1b08c2b5a99efaed94.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 18/23] t/test-lib-functions.sh: support `--date` in `test_commit_bulk()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:44:11Z","receivedAt":"2024-04-29T20:44:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"One of the tests we'll want to add for pseudo-merge bitmaps needs to be\nable to generate a large number of commits at a specific date.\n\nSupport the `--date` option (with identical semantics to the `--date`\noption for `test_commit()`) within `test_commit_bulk` as a prerequisite\nfor that.\n\nSigned-off-by: Taylor Blau \n---\n t/test-lib-functions.sh | 12 +++++++++++-\n 1 file changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 862d80c9748..16fd585e34b 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -458,6 +458,7 @@ test_commit_bulk () {\n \tindir=.\n \tref=HEAD\n \tn=1\n+\tnotick=\n \tmessage='commit %s'\n \tfilename='%s.t'\n \tcontents='content %s'\n@@ -488,6 +489,12 @@ test_commit_bulk () {\n \t\t\tfilename=\"${1#--*=}-%s.t\"\n \t\t\tcontents=\"${1#--*=} %s\"\n \t\t\t;;\n+\t\t--date)\n+\t\t\tnotick=yes\n+\t\t\tGIT_COMMITTER_DATE=\"$2\"\n+\t\t\tGIT_AUTHOR_DATE=\"$2\"\n+\t\t\tshift\n+\t\t\t;;\n \t\t-*)\n \t\t\tBUG \"invalid test_commit_bulk option: $1\"\n \t\t\t;;\n@@ -507,7 +514,10 @@ test_commit_bulk () {\n \n \twhile test \"$total\" -gt 0\n \tdo\n-\t\ttest_tick &&\n+\t\tif test -z \"$notick\"\n+\t\tthen\n+\t\t\ttest_tick\n+\t\tfi &&\n \t\techo \"commit $ref\"\n \t\tprintf 'author %s <%s> %s\\n' \\\n \t\t\t\"$GIT_AUTHOR_NAME\" \\\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493702","messageId":"3f85e5b90f5fc65b4d6aa610cfbec40d2cd8fdf2.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 19/23] pack-bitmap.c: use pseudo-merges during traversal","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:44:14Z","receivedAt":"2024-04-29T20:44:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that all of the groundwork has been laid to support reading and\nusing pseudo-merges, make use of that work in this commit by teaching\nthe pack-bitmap machinery to use pseudo-merge(s) when available during\ntraversal.\n\nThe basic operation is as follows:\n\n - When enumerating objects on either side of a reachability query,\n first see if any subset of the roots satisfies some pseudo-merge\n bitmap. If it does, apply that pseudo-merge bitmap.\n\n - If any pseudo-merge bitmap(s) were applied in the previous step, OR\n them into the result[^1]. Then repeat the process over all\n pseudo-merge bitmaps (we'll refer to this as \"cascading\"\n pseudo-merges). Once this is done, OR in the resulting bitmap.\n\n - If there is no fill-in traversal to be done, return the bitmap for\n that side of the reachability query. If there is fill-in traversal,\n then for each commit we encounter via show_commit(), check to see if\n any unsatisfied pseudo-merges containing that commit as one of its\n parents has been made satisfied by the presence of that commit.\n\n If so, OR in the object set from that pseudo-merge bitmap, and then\n cascade. If not, continue traversal.\n\nA similar implementation is present in the boundary-based bitmap\ntraversal routines.\n\n[^1]: Importantly, we cannot OR in the entire set of roots along with\n the objects reachable from whatever pseudo-merge bitmaps were\n satisfied. This may leave some dangling bits corresponding to any\n unsatisfied root(s) getting OR'd into the resulting bitmap, tricking\n other parts of the traversal into thinking we already have a\n reachability closure over those commit(s) when we do not.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 112 ++++++++++-\n t/t5333-pseudo-merge-bitmaps.sh | 325 ++++++++++++++++++++++++++++++++\n 2 files changed, 436 insertions(+), 1 deletion(-)\n create mode 100755 t/t5333-pseudo-merge-bitmaps.sh\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex c13074673af..e61058dada6 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -114,6 +114,9 @@ struct bitmap_index {\n \tunsigned int version;\n };\n \n+static int pseudo_merges_satisfied_nr;\n+static int pseudo_merges_cascades_nr;\n+\n static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st)\n {\n \tstruct ewah_bitmap *parent;\n@@ -1006,6 +1009,22 @@ static void show_commit(struct commit *commit UNUSED,\n {\n }\n \n+static unsigned apply_pseudo_merges_for_commit_1(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct bitmap *result,\n+\t\t\t\t\t\t struct commit *commit,\n+\t\t\t\t\t\t uint32_t commit_pos)\n+{\n+\tint ret;\n+\n+\tret = apply_pseudo_merges_for_commit(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t result, commit, commit_pos);\n+\n+\tif (ret)\n+\t\tpseudo_merges_satisfied_nr += ret;\n+\n+\treturn ret;\n+}\n+\n static int add_to_include_set(struct bitmap_index *bitmap_git,\n \t\t\t struct include_data *data,\n \t\t\t struct commit *commit,\n@@ -1026,6 +1045,10 @@ static int add_to_include_set(struct bitmap_index *bitmap_git,\n \t}\n \n \tbitmap_set(data->base, bitmap_pos);\n+\tif (apply_pseudo_merges_for_commit_1(bitmap_git, data->base, commit,\n+\t\t\t\t\t bitmap_pos))\n+\t\treturn 0;\n+\n \treturn 1;\n }\n \n@@ -1151,6 +1174,20 @@ static void show_boundary_object(struct object *object UNUSED,\n \tBUG(\"should not be called\");\n }\n \n+static unsigned cascade_pseudo_merges_1(struct bitmap_index *bitmap_git,\n+\t\t\t\t\tstruct bitmap *result,\n+\t\t\t\t\tstruct bitmap *roots)\n+{\n+\tint ret = cascade_pseudo_merges(&bitmap_git->pseudo_merges,\n+\t\t\t\t\tresult, roots);\n+\tif (ret) {\n+\t\tpseudo_merges_cascades_nr++;\n+\t\tpseudo_merges_satisfied_nr += ret;\n+\t}\n+\n+\treturn ret;\n+}\n+\n static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \t\t\t\t\t struct rev_info *revs,\n \t\t\t\t\t struct object_list *roots)\n@@ -1160,6 +1197,7 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \tunsigned int i;\n \tunsigned int tmp_blobs, tmp_trees, tmp_tags;\n \tint any_missing = 0;\n+\tint existing_bitmaps = 0;\n \n \tcb.bitmap_git = bitmap_git;\n \tcb.base = bitmap_new();\n@@ -1167,6 +1205,25 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \n \trevs->ignore_missing_links = 1;\n \n+\tif (bitmap_git->pseudo_merges.nr) {\n+\t\tstruct bitmap *roots_bitmap = bitmap_new();\n+\t\tstruct object_list *objects = NULL;\n+\n+\t\tfor (objects = roots; objects; objects = objects->next) {\n+\t\t\tstruct object *object = objects->item;\n+\t\t\tint pos;\n+\n+\t\t\tpos = bitmap_position(bitmap_git, &object->oid);\n+\t\t\tif (pos < 0)\n+\t\t\t\tcontinue;\n+\n+\t\t\tbitmap_set(roots_bitmap, pos);\n+\t\t}\n+\n+\t\tif (!cascade_pseudo_merges_1(bitmap_git, cb.base, roots_bitmap))\n+\t\t\tbitmap_free(roots_bitmap);\n+\t}\n+\n \t/*\n \t * OR in any existing reachability bitmaps among `roots` into\n \t * `cb.base`.\n@@ -1178,8 +1235,10 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \t\t\tcontinue;\n \n \t\tif (add_commit_to_bitmap(bitmap_git, &cb.base,\n-\t\t\t\t\t (struct commit *)object))\n+\t\t\t\t\t (struct commit *)object)) {\n+\t\t\texisting_bitmaps = 1;\n \t\t\tcontinue;\n+\t\t}\n \n \t\tany_missing = 1;\n \t}\n@@ -1187,6 +1246,9 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \tif (!any_missing)\n \t\tgoto cleanup;\n \n+\tif (existing_bitmaps)\n+\t\tcascade_pseudo_merges_1(bitmap_git, cb.base, NULL);\n+\n \ttmp_blobs = revs->blob_objects;\n \ttmp_trees = revs->tree_objects;\n \ttmp_tags = revs->blob_objects;\n@@ -1242,6 +1304,13 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \treturn cb.base;\n }\n \n+static void unsatisfy_all_pseudo_merges(struct bitmap_index *bitmap_git)\n+{\n+\tuint32_t i;\n+\tfor (i = 0; i < bitmap_git->pseudo_merges.nr; i++)\n+\t\tbitmap_git->pseudo_merges.v[i].satisfied = 0;\n+}\n+\n static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\t\t\t struct rev_info *revs,\n \t\t\t\t struct object_list *roots,\n@@ -1249,9 +1318,32 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n {\n \tstruct bitmap *base = NULL;\n \tint needs_walk = 0;\n+\tunsigned existing_bitmaps = 0;\n \n \tstruct object_list *not_mapped = NULL;\n \n+\tunsatisfy_all_pseudo_merges(bitmap_git);\n+\n+\tif (bitmap_git->pseudo_merges.nr) {\n+\t\tstruct bitmap *roots_bitmap = bitmap_new();\n+\t\tstruct object_list *objects = NULL;\n+\n+\t\tfor (objects = roots; objects; objects = objects->next) {\n+\t\t\tstruct object *object = objects->item;\n+\t\t\tint pos;\n+\n+\t\t\tpos = bitmap_position(bitmap_git, &object->oid);\n+\t\t\tif (pos < 0)\n+\t\t\t\tcontinue;\n+\n+\t\t\tbitmap_set(roots_bitmap, pos);\n+\t\t}\n+\n+\t\tbase = bitmap_new();\n+\t\tif (!cascade_pseudo_merges_1(bitmap_git, base, roots_bitmap))\n+\t\t\tbitmap_free(roots_bitmap);\n+\t}\n+\n \t/*\n \t * Go through all the roots for the walk. The ones that have bitmaps\n \t * on the bitmap index will be `or`ed together to form an initial\n@@ -1262,11 +1354,21 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t */\n \twhile (roots) {\n \t\tstruct object *object = roots->item;\n+\n \t\troots = roots->next;\n \n+\t\tif (base) {\n+\t\t\tint pos = bitmap_position(bitmap_git, &object->oid);\n+\t\t\tif (pos > 0 && bitmap_get(base, pos)) {\n+\t\t\t\tobject->flags |= SEEN;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n \t\tif (object->type == OBJ_COMMIT &&\n \t\t add_commit_to_bitmap(bitmap_git, &base, (struct commit *)object)) {\n \t\t\tobject->flags |= SEEN;\n+\t\t\texisting_bitmaps = 1;\n \t\t\tcontinue;\n \t\t}\n \n@@ -1282,6 +1384,9 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \n \troots = not_mapped;\n \n+\tif (existing_bitmaps)\n+\t\tcascade_pseudo_merges_1(bitmap_git, base, NULL);\n+\n \t/*\n \t * Let's iterate through all the roots that don't have bitmaps to\n \t * check if we can determine them to be reachable from the existing\n@@ -1866,6 +1971,11 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tobject_list_free(&wants);\n \tobject_list_free(&haves);\n \n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"pseudo_merges_satisfied\",\n+\t\t\t pseudo_merges_satisfied_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"pseudo_merges_cascades\",\n+\t\t\t pseudo_merges_cascades_nr);\n+\n \treturn bitmap_git;\n \n cleanup:\ndiff --git a/t/t5333-pseudo-merge-bitmaps.sh b/t/t5333-pseudo-merge-bitmaps.sh\nnew file mode 100755\nindex 00000000000..909c17e301e\n--- /dev/null\n+++ b/t/t5333-pseudo-merge-bitmaps.sh\n@@ -0,0 +1,325 @@\n+#!/bin/sh\n+\n+test_description='pseudo-merge bitmaps'\n+\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n+. ./test-lib.sh\n+\n+test_pseudo_merges () {\n+\ttest-tool bitmap dump-pseudo-merges\n+}\n+\n+test_pseudo_merge_commits () {\n+\ttest-tool bitmap dump-pseudo-merge-commits \"$1\"\n+}\n+\n+test_pseudo_merges_satisfied () {\n+\ttest_trace2_data bitmap pseudo_merges_satisfied \"$1\"\n+}\n+\n+test_pseudo_merges_cascades () {\n+\ttest_trace2_data bitmap pseudo_merges_cascades \"$1\"\n+}\n+\n+tag_everything () {\n+\tgit rev-list --all --no-object-names >in &&\n+\tperl -lne '\n+\t\tprint \"create refs/tags/\" . $. . \" \" . $1 if /([0-9a-f]+)/\n+\t' expect &&\n+\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\n+\ttest_pseudo_merges_satisfied 0 merges &&\n+\ttest_must_be_empty merges &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'pseudo-merges accurately represent their objects' '\n+\ttest_config bitmapPseudoMerge.test.pattern \"refs/tags/\" &&\n+\ttest_config bitmapPseudoMerge.test.maxMerges 8 &&\n+\ttest_config bitmapPseudoMerge.test.stableThreshold never &&\n+\n+\tgit repack -adb &&\n+\n+\ttest_pseudo_merges >merges &&\n+\ttest_line_count = 8 merges &&\n+\n+\tfor i in $(test_seq 0 $(($(wc -l commits &&\n+\n+\t\tgit rev-list --objects --no-object-names --stdin expect.raw &&\n+\t\ttest-tool bitmap dump-pseudo-merge-objects $i >actual.raw &&\n+\n+\t\tsort -u expect &&\n+\t\tsort -u actual &&\n+\n+\t\ttest_cmp expect actual || return 1\n+\tdone\n+'\n+\n+test_expect_success 'bitmap traversal with pseudo-merges' '\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\tgit rev-list --count --all --objects >expect &&\n+\n+\ttest_pseudo_merges_satisfied 8 trace2.txt &&\n+\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\tgit rev-list --count --all --objects >expect &&\n+\n+\ttest_pseudo_merges_satisfied 8 merges &&\n+\t\tfor i in $(test_seq 0 $(($(wc -l commits &&\n+\n+\t\ttest-tool bitmap list-commits >bitmaps &&\n+\t\tbitmaps_nr=\"$(wc -l expect &&\n+\n+\t\ttest $(cat expect) -eq $(wc -l merges &&\n+\t\ttest_line_count = 1 merges &&\n+\n+\t\ttest_pseudo_merge_commits 0 >oids &&\n+\t\tgit cat-file --batch commits &&\n+\n+\t\ttest $(wc -l in &&\n+\t\tgit update-ref --stdin merges &&\n+\t\tmerges_nr=\"$(wc -l oids &&\n+\t\t\tgit cat-file --batch commits &&\n+\n+\t\t\texpect=\"$(grep -c \"^committer.*$old +0000$\" commits)\" &&\n+\t\t\tactual=\"$(wc -l oids &&\n+\t\tgit cat-file --batch commits &&\n+\t\ttest $(wc -l err &&\n+\n+\tcat >expect <<-EOF &&\n+\tfatal: pseudo-merge group ${SQ}test${SQ} has unstable threshold before stable one\n+\tEOF\n+\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'pseudo-merge pattern with capture groups' '\n+\tgit init pseudo-merge-captures &&\n+\t(\n+\t\tcd pseudo-merge-captures &&\n+\n+\t\ttest_commit_bulk 128 &&\n+\t\ttag_everything &&\n+\n+\t\tfor r in $(test_seq 8)\n+\t\tdo\n+\t\t\ttest_commit_bulk 16 &&\n+\n+\t\t\tgit rev-list HEAD~16.. >in &&\n+\n+\t\t\tperl -lne \"print \\\"create refs/remotes/$r/tags/\\$. \\$_\\\"\" refs &&\n+\n+\t\ttest_pseudo_merges >merges &&\n+\t\tfor m in $(test_seq 0 $(($(wc -l oids &&\n+\t\t\tgrep -f oids refs |\n+\t\t\tperl -lne \"print \\$1 if /refs\\/remotes\\/([0-9]+)/\" |\n+\t\t\tsort -u || return 1\n+\t\tdone >remotes &&\n+\n+\t\ttest $(wc -l merges &&\n+\t\ttest_line_count = 2 merges &&\n+\n+\t\ttest_pseudo_merge_commits 0 >commits-0.raw &&\n+\t\ttest_pseudo_merge_commits 1 >commits-1.raw &&\n+\n+\t\tsort commits-0.raw >commits-0 &&\n+\t\tsort commits-1.raw >commits-1 &&\n+\n+\t\tcomm -12 commits-0 commits-1 >overlap &&\n+\n+\t\ttest_line_count -gt 0 overlap\n+\t)\n+'\n+\n+test_expect_success 'pseudo-merge overlap traversal' '\n+\t(\n+\t\tcd pseudo-merge-overlap &&\n+\n+\t\t: >trace2.txt &&\n+\t\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\t\tgit rev-list --count --all --objects >expect &&\n+\n+\t\ttest_pseudo_merges_satisfied 2 trace2.txt &&\n+\t\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\t\tgit rev-list --count --all --objects >expect &&\n+\n+\t\ttest_pseudo_merges_satisfied 2 \n---\n pack-bitmap.c | 26 +++++++++++++++++++++++++-\n 1 file changed, 25 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex e61058dada6..1966b3b95f1 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -116,6 +116,10 @@ struct bitmap_index {\n \n static int pseudo_merges_satisfied_nr;\n static int pseudo_merges_cascades_nr;\n+static int existing_bitmaps_hits_nr;\n+static int existing_bitmaps_misses_nr;\n+static int roots_with_bitmaps_nr;\n+static int roots_without_bitmaps_nr;\n \n static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st)\n {\n@@ -1040,10 +1044,14 @@ static int add_to_include_set(struct bitmap_index *bitmap_git,\n \n \tpartial = bitmap_for_commit(bitmap_git, commit);\n \tif (partial) {\n+\t\texisting_bitmaps_hits_nr++;\n+\n \t\tbitmap_or_ewah(data->base, partial);\n \t\treturn 0;\n \t}\n \n+\texisting_bitmaps_misses_nr++;\n+\n \tbitmap_set(data->base, bitmap_pos);\n \tif (apply_pseudo_merges_for_commit_1(bitmap_git, data->base, commit,\n \t\t\t\t\t bitmap_pos))\n@@ -1099,8 +1107,12 @@ static int add_commit_to_bitmap(struct bitmap_index *bitmap_git,\n {\n \tstruct ewah_bitmap *or_with = bitmap_for_commit(bitmap_git, commit);\n \n-\tif (!or_with)\n+\tif (!or_with) {\n+\t\texisting_bitmaps_misses_nr++;\n \t\treturn 0;\n+\t}\n+\n+\texisting_bitmaps_hits_nr++;\n \n \tif (!*base)\n \t\t*base = ewah_to_bitmap(or_with);\n@@ -1407,8 +1419,12 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\t\tobject->flags &= ~UNINTERESTING;\n \t\t\tadd_pending_object(revs, object, \"\");\n \t\t\tneeds_walk = 1;\n+\n+\t\t\troots_without_bitmaps_nr++;\n \t\t} else {\n \t\t\tobject->flags |= SEEN;\n+\n+\t\t\troots_with_bitmaps_nr++;\n \t\t}\n \t}\n \n@@ -1975,6 +1991,14 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\t pseudo_merges_satisfied_nr);\n \ttrace2_data_intmax(\"bitmap\", the_repository, \"pseudo_merges_cascades\",\n \t\t\t pseudo_merges_cascades_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"bitmap/hits\",\n+\t\t\t existing_bitmaps_hits_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"bitmap/misses\",\n+\t\t\t existing_bitmaps_misses_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"bitmap/roots_with_bitmap\",\n+\t\t\t roots_with_bitmaps_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"bitmap/roots_without_bitmap\",\n+\t\t\t roots_without_bitmaps_nr);\n \n \treturn bitmap_git;\n \n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493704","messageId":"b5aea8e57f8342115a8d6416bd6ae2f7669bac99.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 21/23] ewah: `bitmap_equals_ewah()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:44:22Z","receivedAt":"2024-04-29T20:44:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to reuse existing pseudo-merge bitmaps by implementing a\n`bitmap_equals_ewah()` helper.\n\nThis helper will be used to see if a raw bitmap (containing the set of\nparents for some pseudo-merge) is equal to any existing pseudo-merge's\ncommits bitmap (which are stored as EWAH-compressed bitmaps on disk).\n\nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 19 +++++++++++++++++++\n ewah/ewok.h | 1 +\n 2 files changed, 20 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex dc2ca190f12..55928dada86 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -261,6 +261,25 @@ int bitmap_equals(struct bitmap *self, struct bitmap *other)\n \treturn 1;\n }\n \n+int bitmap_equals_ewah(struct bitmap *self, struct ewah_bitmap *other)\n+{\n+\tstruct ewah_iterator it;\n+\teword_t word;\n+\tsize_t i = 0;\n+\n+\tewah_iterator_init(&it, other);\n+\n+\twhile (ewah_iterator_next(&word, &it))\n+\t\tif (word != (i < self->word_alloc ? self->words[i++] : 0))\n+\t\t\treturn 0;\n+\n+\tfor (; i < self->word_alloc; i++)\n+\t\tif (self->words[i])\n+\t\t\treturn 0;\n+\n+\treturn 1;\n+}\n+\n int bitmap_is_subset(struct bitmap *self, struct bitmap *other)\n {\n \tsize_t common_size, i;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 7074a6347b7..5e357e24933 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -179,6 +179,7 @@ void bitmap_unset(struct bitmap *self, size_t pos);\n int bitmap_get(struct bitmap *self, size_t pos);\n void bitmap_free(struct bitmap *self);\n int bitmap_equals(struct bitmap *self, struct bitmap *other);\n+int bitmap_equals_ewah(struct bitmap *self, struct ewah_bitmap *other);\n \n /*\n * Both `bitmap_is_subset()` and `ewah_bitmap_is_subset()` return 1 if the set\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493705","messageId":"61ddb5742850868d0fd192f37048527c3b06e853.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 22/23] pseudo-merge: implement support for finding existing merges","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:44:26Z","receivedAt":"2024-04-29T20:44:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This patch implements support for reusing existing pseudo-merge commits\nwhen writing bitmaps when there is an existing pseudo-merge bitmap which\nhas exactly the same set of parents as one that we are about to write.\n\nNote that unstable pseudo-merges are likely to change between\nconsecutive repacks, and so are generally poor candidates for reuse.\nHowever, stable pseudo-merges (see the configuration option\n'bitmapPseudoMerge..stableThreshold') are by definition unlikely\nto change between runs (as they represent long-running branches).\n\nBecause there is no index from a *set* of pseudo-merge parents to a\nmatching pseudo-merge bitmap, we have to construct the bitmap\ncorresponding to the set of parents for each pending pseudo-merge commit\nand see if a matching bitmap exists.\n\nThis is technically quadratic in the number of pseudo-merges, but is OK\nin practice for a couple of reasons:\n\n - non-matching pseudo-merge bitmaps are rejected quickly as soon as\n they differ in a single bit\n\n - already-matched pseudo-merge bitmaps are discarded from subsequent\n rounds of search\n\n - the number of pseudo-merges is generally small, even for large\n repositories\n\nIn order to do this, implement (a) a function that finds a matching\npseudo-merge given some uncompressed bitset describing its parents, (b)\na function that computes the bitset of parents for a given pseudo-merge\ncommit, and (c) call that function before computing the set of reachable\nobjects for some pending pseudo-merge.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 15 ++++++--\n pack-bitmap.c | 32 +++++++++++++++++\n pack-bitmap.h | 2 ++\n pseudo-merge.c | 55 ++++++++++++++++++++++++++++\n pseudo-merge.h | 7 ++++\n t/t5333-pseudo-merge-bitmaps.sh | 64 +++++++++++++++++++++++++++++++++\n 6 files changed, 173 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex d4894ace9ee..f7245d7d6fa 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -19,6 +19,10 @@\n #include \"tree-walk.h\"\n #include \"pseudo-merge.h\"\n #include \"oid-array.h\"\n+#include \"config.h\"\n+#include \"alloc.h\"\n+#include \"refs.h\"\n+#include \"strmap.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -443,6 +447,7 @@ static int fill_bitmap_tree(struct bitmap *bitmap,\n }\n \n static int reused_bitmaps_nr;\n+static int reused_pseudo_merge_bitmaps_nr;\n \n static int fill_bitmap_commit(struct bb_commit *ent,\n \t\t\t struct commit *commit,\n@@ -467,7 +472,7 @@ static int fill_bitmap_commit(struct bb_commit *ent,\n \t\t\tstruct bitmap *remapped = bitmap_new();\n \n \t\t\tif (commit->object.flags & BITMAP_PSEUDO_MERGE)\n-\t\t\t\told = NULL;\n+\t\t\t\told = pseudo_merge_bitmap_for_commit(old_bitmap, c);\n \t\t\telse\n \t\t\t\told = bitmap_for_commit(old_bitmap, c);\n \t\t\t/*\n@@ -478,7 +483,10 @@ static int fill_bitmap_commit(struct bb_commit *ent,\n \t\t\tif (old && !rebuild_bitmap(mapping, old, remapped)) {\n \t\t\t\tbitmap_or(ent->bitmap, remapped);\n \t\t\t\tbitmap_free(remapped);\n-\t\t\t\treused_bitmaps_nr++;\n+\t\t\t\tif (commit->object.flags & BITMAP_PSEUDO_MERGE)\n+\t\t\t\t\treused_pseudo_merge_bitmaps_nr++;\n+\t\t\t\telse\n+\t\t\t\t\treused_bitmaps_nr++;\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tbitmap_free(remapped);\n@@ -604,6 +612,9 @@ int bitmap_writer_build(struct packing_data *to_pack)\n \t\t\t the_repository);\n \ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n \t\t\t \"building_bitmaps_reused\", reused_bitmaps_nr);\n+\ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n+\t\t\t \"building_bitmaps_pseudo_merge_reused\",\n+\t\t\t reused_pseudo_merge_bitmaps_nr);\n \n \tstop_progress(&writer.progress);\n \ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 1966b3b95f1..70230e26479 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1316,6 +1316,37 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \treturn cb.base;\n }\n \n+struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct commit *commit)\n+{\n+\tstruct commit_list *p;\n+\tstruct bitmap *parents;\n+\tstruct pseudo_merge *match = NULL;\n+\n+\tif (!bitmap_git->pseudo_merges.nr)\n+\t\treturn NULL;\n+\n+\tparents = bitmap_new();\n+\n+\tfor (p = commit->parents; p; p = p->next) {\n+\t\tint pos = bitmap_position(bitmap_git, &p->item->object.oid);\n+\t\tif (pos < 0 || pos >= bitmap_num_objects(bitmap_git))\n+\t\t\tgoto done;\n+\n+\t\tbitmap_set(parents, pos);\n+\t}\n+\n+\tmatch = pseudo_merge_for_parents(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t\tparents);\n+\n+done:\n+\tbitmap_free(parents);\n+\tif (match)\n+\t\treturn pseudo_merge_bitmap(&bitmap_git->pseudo_merges, match);\n+\n+\treturn NULL;\n+}\n+\n static void unsatisfy_all_pseudo_merges(struct bitmap_index *bitmap_git)\n {\n \tuint32_t i;\n@@ -2809,6 +2840,7 @@ void free_bitmap_index(struct bitmap_index *b)\n \t\t */\n \t\tclose_midx_revindex(b->midx);\n \t}\n+\tfree_pseudo_merge_map(&b->pseudo_merges);\n \tfree(b);\n }\n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 25d3b8e604a..0fefef39bec 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -119,6 +119,8 @@ int rebuild_bitmap(const uint32_t *reposition,\n \t\t struct bitmap *dest);\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t struct commit *commit);\n+struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct commit *commit);\n void bitmap_writer_select_commits(struct commit **indexed_commits,\n \t\t\t\t unsigned int indexed_commits_nr);\n int bitmap_writer_build(struct packing_data *to_pack);\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex e111c9cd1a6..9e21fbb5062 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -682,3 +682,58 @@ int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n \n \treturn ret;\n }\n+\n+struct pseudo_merge *pseudo_merge_for_parents(const struct pseudo_merge_map *pm,\n+\t\t\t\t\t struct bitmap *parents)\n+{\n+\tstruct pseudo_merge *match = NULL;\n+\tsize_t i;\n+\n+\tif (!pm->nr)\n+\t\treturn NULL;\n+\n+\t/*\n+\t * NOTE: this loop is quadratic in the worst-case (where no\n+\t * matching pseudo-merge bitmaps are found), but in practice\n+\t * this is OK for a few reasons:\n+\t *\n+\t * - Rejecting pseudo-merge bitmaps that do not match the\n+\t * given commit is done quickly (i.e. `bitmap_equals_ewah()`\n+\t * returns early when we know the two bitmaps aren't equal.\n+\t *\n+\t * - Already matched pseudo-merge bitmaps (which we track with\n+\t * the `->satisfied` bit here) are skipped as potential\n+\t * candidates.\n+\t *\n+\t * - The number of pseudo-merges should be small (in the\n+\t * hundreds for most repositories).\n+\t *\n+\t * If in the future this semi-quadratic behavior does become a\n+\t * problem, another approach would be to keep track of which\n+\t * pseudo-merges are still \"viable\" after enumerating the\n+\t * pseudo-merge commit's parents:\n+\t *\n+\t * - A pseudo-merge bitmap becomes non-viable when the bit(s)\n+\t * corresponding to one or more parent(s) of the given\n+\t * commit are not set in a candidate pseudo-merge's commits\n+\t * bitmap.\n+\t *\n+\t * - After processing all bits, enumerate the remaining set of\n+\t * viable pseudo-merge bitmaps, and check that their\n+\t * popcount() matches the number of parents in the given\n+\t * commit.\n+\t */\n+\tfor (i = 0; i < pm->nr; i++) {\n+\t\tstruct pseudo_merge *candidate = use_pseudo_merge(pm, &pm->v[i]);\n+\t\tif (!candidate || candidate->satisfied)\n+\t\t\tcontinue;\n+\t\tif (!bitmap_equals_ewah(parents, candidate->commits))\n+\t\t\tcontinue;\n+\n+\t\tmatch = candidate;\n+\t\tmatch->satisfied = 1;\n+\t\tbreak;\n+\t}\n+\n+\treturn match;\n+}\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nindex cc14e947e86..33acd00a3e5 100644\n--- a/pseudo-merge.h\n+++ b/pseudo-merge.h\n@@ -208,4 +208,11 @@ int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n \t\t\t struct bitmap *result,\n \t\t\t struct bitmap *roots);\n \n+/*\n+ * Returns a pseudo-merge which contains the exact set of commits\n+ * listed in the \"parents\" bitamp, or NULL if none could be found.\n+ */\n+struct pseudo_merge *pseudo_merge_for_parents(const struct pseudo_merge_map *pm,\n+\t\t\t\t\t struct bitmap *parents);\n+\n #endif\ndiff --git a/t/t5333-pseudo-merge-bitmaps.sh b/t/t5333-pseudo-merge-bitmaps.sh\nindex 909c17e301e..531f1924af4 100755\n--- a/t/t5333-pseudo-merge-bitmaps.sh\n+++ b/t/t5333-pseudo-merge-bitmaps.sh\n@@ -22,6 +22,10 @@ test_pseudo_merges_cascades () {\n \ttest_trace2_data bitmap pseudo_merges_cascades \"$1\"\n }\n \n+test_pseudo_merges_reused () {\n+\ttest_trace2_data pack-bitmap-write building_bitmaps_pseudo_merge_reused \"$1\"\n+}\n+\n tag_everything () {\n \tgit rev-list --all --no-object-names >in &&\n \tperl -lne '\n@@ -322,4 +326,64 @@ test_expect_success 'pseudo-merge overlap stale traversal' '\n \t)\n '\n \n+test_expect_success 'pseudo-merge reuse' '\n+\tgit init pseudo-merge-reuse &&\n+\t(\n+\t\tcd pseudo-merge-reuse &&\n+\n+\t\tstable=\"1641013200\" && # 2022-01-01\n+\t\tunstable=\"1672549200\" && # 2023-01-01\n+\n+\t\tfor date in $stable $unstable\n+\t\tdo\n+\t\t\ttest_commit_bulk --date \"$date +0000\" 128 &&\n+\t\t\ttest_tick || return 1\n+\t\tdone &&\n+\n+\t\ttag_everything &&\n+\n+\t\tgit \\\n+\t\t\t-c bitmapPseudoMerge.test.pattern=\"refs/tags/\" \\\n+\t\t\t-c bitmapPseudoMerge.test.maxMerges=1 \\\n+\t\t\t-c bitmapPseudoMerge.test.threshold=now \\\n+\t\t\t-c bitmapPseudoMerge.test.stableThreshold=$(($unstable - 1)) \\\n+\t\t\t-c bitmapPseudoMerge.test.stableSize=512 \\\n+\t\t\trepack -adb &&\n+\n+\t\ttest_pseudo_merges >merges &&\n+\t\ttest_line_count = 2 merges &&\n+\n+\t\ttest_pseudo_merge_commits 0 >stable-oids.before &&\n+\t\ttest_pseudo_merge_commits 1 >unstable-oids.before &&\n+\n+\t\t: >trace2.txt &&\n+\t\tGIT_TRACE2_EVENT=$PWD/trace2.txt git \\\n+\t\t\t-c bitmapPseudoMerge.test.pattern=\"refs/tags/\" \\\n+\t\t\t-c bitmapPseudoMerge.test.maxMerges=2 \\\n+\t\t\t-c bitmapPseudoMerge.test.threshold=now \\\n+\t\t\t-c bitmapPseudoMerge.test.stableThreshold=$(($unstable - 1)) \\\n+\t\t\t-c bitmapPseudoMerge.test.stableSize=512 \\\n+\t\t\trepack -adb &&\n+\n+\t\ttest_pseudo_merges_reused 1 merges &&\n+\t\ttest_line_count = 3 merges &&\n+\n+\t\ttest_pseudo_merge_commits 0 >stable-oids.after &&\n+\t\tfor i in 1 2\n+\t\tdo\n+\t\t\ttest_pseudo_merge_commits $i || return 1\n+\t\tdone >unstable-oids.after &&\n+\n+\t\tsort -u expect &&\n+\t\tsort -u actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tsort -u expect &&\n+\t\tsort -u actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.45.0.23.gc6f94b99219\n\n"},{"id":"493706","messageId":"2bd830d35dd79a7b1201655df70fc0039cc44d7e.1714422410.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"[PATCH v2 23/23] t/perf: implement performace tests for pseudo-merge bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-04-29T20:44:30Z","receivedAt":"2024-04-29T20:44:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement a straightforward performance test demonstrating the benefit\nof pseudo-merge bitmaps by measuring how long it takes to count\nreachable objects in a few different scenarios:\n\n - without bitmaps, to demonstrate a reasonable baseline\n - with bitmaps, but without pseudo-merges\n - with bitmaps and pseudo-merges\n\nResults from running this test on git.git are as follows:\n\n Test this tree\n -----------------------------------------------------------------------------------\n 5333.2: git rev-list --count --all --objects (no bitmaps) 3.46(3.37+0.09)\n 5333.3: git rev-list --count --all --objects (no pseudo-merges) 0.13(0.11+0.01)\n 5333.4: git rev-list --count --all --objects (with pseudo-merges) 0.12(0.11+0.01)\n\nSigned-off-by: Taylor Blau \n---\n t/perf/p5333-pseudo-merge-bitmaps.sh | 32 ++++++++++++++++++++++++++++\n 1 file changed, 32 insertions(+)\n create mode 100755 t/perf/p5333-pseudo-merge-bitmaps.sh\n\ndiff --git a/t/perf/p5333-pseudo-merge-bitmaps.sh b/t/perf/p5333-pseudo-merge-bitmaps.sh\nnew file mode 100755\nindex 00000000000..4bec409d10e\n--- /dev/null\n+++ b/t/perf/p5333-pseudo-merge-bitmaps.sh\n@@ -0,0 +1,32 @@\n+#!/bin/sh\n+\n+test_description='pseudo-merge bitmaps'\n+. ./perf-lib.sh\n+\n+test_perf_large_repo\n+\n+test_expect_success 'setup' '\n+\tgit \\\n+\t\t-c bitmapPseudoMerge.all.pattern=\"refs/\" \\\n+\t\t-c bitmapPseudoMerge.all.threshold=now \\\n+\t\t-c bitmapPseudoMerge.all.stableThreshold=never \\\n+\t\t-c bitmapPseudoMerge.all.maxMerges=64 \\\n+\t\t-c pack.writeBitmapLookupTable=true \\\n+\t\trepack -adb\n+'\n+\n+test_perf 'git rev-list --count --all --objects (no bitmaps)' '\n+\tgit rev-list --objects --all\n+'\n+\n+test_perf 'git rev-list --count --all --objects (no pseudo-merges)' '\n+\tGIT_TEST_USE_PSEDUO_MERGES=0 \\\n+\t\tgit rev-list --objects --all --use-bitmap-index\n+'\n+\n+test_perf 'git rev-list --count --all --objects (with pseudo-merges)' '\n+\tGIT_TEST_USE_PSEDUO_MERGES=1 \\\n+\t\tgit rev-list --objects --all --use-bitmap-index\n+'\n+\n+test_done\n-- \n2.45.0.23.gc6f94b99219\n"},{"id":"493816","messageId":"xmqq34r2zkgp.fsf@gitster.g","threadId":"61166","inReplyTo":"cover.1714422410.git.me@ttaylorr.com","subject":"Re: [PATCH v2 00/23] pack-bitmap: pseudo-merge reachability bitmaps","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-30T20:03:50Z","receivedAt":"2024-04-30T20:03:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> - Rebased onto 2.45, so this is now based on 'master', which is at\n> 786a3e4b8d (Git 2.45, 2024-04-29) at the time of writing.\n\nIs there any notable reason for the rebase (other than \"2.45 is out\nnow\") that needs to be called out? Something along the lines of\n\"topic X and Y has graduated and the helper function used by this\ntopic has changed its external interface\"?\n\nThanks, queued.\n\n\n"},{"id":"493829","messageId":"ZjJUTlzOijT6E8pr@nand.local","threadId":"61166","inReplyTo":"xmqq34r2zkgp.fsf@gitster.g","subject":"Re: [PATCH v2 00/23] pack-bitmap: pseudo-merge reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-01T14:40:14Z","receivedAt":"2024-05-01T14:40:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Apr 30, 2024 at 01:03:50PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > - Rebased onto 2.45, so this is now based on 'master', which is at\n> > 786a3e4b8d (Git 2.45, 2024-04-29) at the time of writing.\n>\n> Is there any notable reason for the rebase (other than \"2.45 is out\n> now\") that needs to be called out? Something along the lines of\n> \"topic X and Y has graduated and the helper function used by this\n> topic has changed its external interface\"?\n\nIt's mostly that 2.45 is out, but there were a couple of topics that\nmerged that produced minor conflicts that I wanted to resolve for you:\n\n- 625ef1c6f1 (Merge branch 'tb/t7700-fixup', 2024-04-16) introduced a\n minor conflict (both sides add GIT_TEST_MULTI_PACK_INDEX=0 to the\n relevant test invocations).\n\n- d8360a86ed (Merge branch 'tb/midx-write', 2024-04-12) introduced a\n conflict where this branch adds a call to the new\n `bitmap_writer_init()` function in midx.c, but tb/midx-write moved all\n of that code over to midx-write.c\n\nThose are the two main ones, but I mostly just wanted to take care of it\nsince we're on the other side of 2.45.\n\n> Thanks, queued.\n\nThanks,\nTaylor\n"},{"id":"494149","messageId":"ZjjEjNRp2BAMWJ47@tanuki","threadId":"61166","inReplyTo":"43fd5e3597151a86254e18e08ffd8cadbcb6e4f0.1714422410.git.me@ttaylorr.com","subject":"Re: [PATCH v2 01/23] Documentation/technical: describe pseudo-merge bitmaps format","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-06T11:52:44Z","receivedAt":"2024-05-06T11:52:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 29, 2024 at 04:42:54PM -0400, Taylor Blau wrote:\n> Prepare to implement pseudo-merge bitmaps over the next several commits\n> by first describing the serialization format which will store the new\n> pseudo-merge bitmaps themselves.\n> \n> This format is implemented as an optional extension within the bitmap v1\n> format, making it compatible with previous versions of Git, as well as\n> the original .bitmap implementation within JGit.\n> \n> The format (as well as a general description of pseudo-merge bitmaps,\n> and motivating use-case(s)) is described in detail in the patch contents\n> below, but the high-level description is as follows:\n> \n> - An array of pseudo-merge bitmaps, each containing a pair of EWAH\n> bitmaps: one describing the set of pseudo-merge \"parents\", and\n> another describing the set of object(s) reachable from those\n> parents.\n> \n> - A lookup table to determine which pseudo-merge(s) a given commit\n> appears in. An optional extended lookup table follows when there is\n> at least one commit which appears in multiple pseudo-merge groups.\n> \n> - Trailing metadata, including the number of pseudo-merge(s), number\n> of unique parents, the offset within the .bitmap file for the\n> pseudo-merge commit lookup table, and the size of the optional\n> extension itself.\n> \n> Signed-off-by: Taylor Blau \n> ---\n> Documentation/technical/bitmap-format.txt | 179 ++++++++++++++++++++++\n> 1 file changed, 179 insertions(+)\n> \n> diff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\n> index f5d200939b0..63a7177ac08 100644\n> --- a/Documentation/technical/bitmap-format.txt\n> +++ b/Documentation/technical/bitmap-format.txt\n> @@ -255,3 +255,182 @@ triplet is -\n> \txor_row (4 byte integer, network byte order): ::\n> \tThe position of the triplet whose bitmap is used to compress\n> \tthis one, or `0xffffffff` if no such bitmap exists.\n> +\n> +Pseudo-merge bitmaps\n> +--------------------\n> +\n> +If the `BITMAP_OPT_PSEUDO_MERGES` flag is set, a variable number of\n> +bytes (preceding the name-hash cache, commit lookup table, and trailing\n> +checksum) of the `.bitmap` file is used to store pseudo-merge bitmaps.\n\nHere you say that the section is supposed to come before some other\nsections, whereas the first sentence in the \"File format\" section says\nthat it is the last section in a bitmap file.\n\n> +A \"pseudo-merge bitmap\" is used to refer to a pair of bitmaps, as\n> +follows:\n> +\n> +Commit bitmap::\n> +\n> + A bitmap whose set bits describe the set of commits included in the\n> + pseudo-merge's \"merge\" bitmap (as below).\n> +\n> +Merge bitmap::\n> +\n> + A bitmap whose set bits describe the reachability closure over the set\n> + of commits in the pseudo-merge's \"commits\" bitmap (as above). An\n> + identical bitmap would be generated for an octopus merge with the same\n> + set of parents as described in the commits bitmap.\n> +\n> +Pseudo-merge bitmaps can accelerate bitmap traversals when all commits\n> +for a given pseudo-merge are listed on either side of the traversal,\n> +either directly (by explicitly asking for them as part of the `HAVES`\n> +or `WANTS`) or indirectly (by encountering them during a fill-in\n> +traversal).\n> +\n> +=== Use-cases\n\nI feel like starting with the problems that the whole feature is\nintended to solve would help the reading flow quite a bit. So I'd move\nthis whole section up.\n\n> +For example, suppose there exists a pseudo-merge bitmap with a large\n> +number of commits, all of which are listed in the `WANTS` section of\n> +some bitmap traversal query. When pseudo-merge bitmaps are enabled, the\n> +bitmap machinery can quickly determine there is a pseudo-merge which\n> +satisfies some subset of the wanted objects on either side of the query.\n> +Then, we can inflate the EWAH-compressed bitmap, and `OR` it in to the\n> +resulting bitmap. By contrast, without pseudo-merge bitmaps, we would\n> +have to repeat the decompression and `OR`-ing step over a potentially\n> +large number of individual bitmaps, which can take proportionally more\n> +time.\n> +\n> +Another benefit of pseudo-merges arises when there is some combination\n> +of (a) a large number of references, with (b) poor bitmap coverage, and\n> +(c) deep, nested trees, making fill-in traversal relatively expensive.\n> +For example, suppose that there are a large enough number of tags where\n> +bitmapping each of the tags individually is infeasible. Without\n> +pseudo-merge bitmaps, computing the result of, say, `git rev-list\n> +--use-bitmap-index --count --objects --tags` would likely require a\n> +large amount of fill-in traversal. But when a large quantity of those\n> +tags are stored together in a pseudo-merge bitmap, the bitmap machinery\n> +can take advantage of the fact that we only care about the union of\n> +objects reachable from all of those tags, and answer the query much\n> +faster.\n\nI would start the explanation with a discussion of the problem before\npresenting the solution to those problems. In the current version it's\nthe other way round, you present a solution to a problem that isn't yet\nexplained\n\nIt might also be helpful to discuss a bit who is supposed to create\nthose pseudo-merge bitmaps. Does Git do so automatically for all tags?\nDoes the admin have to configure this? If the latter, when do you want\nto create those and what strategies are there to create them?\n\n> +=== File format\n> +\n> +If enabled, pseudo-merge bitmaps are stored in an optional section at\n> +the end of a `.bitmap` file. The format is as follows:\n> +\n> +....\n> ++-------------------------------------------+\n> +| .bitmap File |\n> ++-------------------------------------------+\n> +| |\n> +| Pseudo-merge bitmaps (Variable Length) |\n> +| +---------------------------+ |\n> +| | commits_bitmap (EWAH) | |\n> +| +---------------------------+ |\n> +| | merge_bitmap (EWAH) | |\n> +| +---------------------------+ |\n> +| |\n> ++-------------------------------------------+\n> +| |\n> +| Lookup Table |\n> +| +------------+--------------+ |\n> +| | commit_pos | offset | |\n> +| +------------+--------------+ |\n> +| | 4 bytes | 8 bytes | |\n> +| +------------+--------------+ |\n\nIt's a bit confusing that in the EWAH section above you have the type of\nthe fields in the same line as the field itself, whereas here you have\nthem formatted in a separate box. This makes the reader wonder at first\nwhether this is two or four fields. How about the following instead:\n\n | Lookup Table |\n | +---------------------------+ |\n | | commit_pos (4 bytes) | |\n | +---------------------------+ |\n | | offset (8 bytes) | |\n | +---------------------------+ |\n\nThe same comment applies to the other sections further down.\n\n> +| |\n> +| Offset Cases: |\n> +| ------------- |\n> +| |\n> +| 1. MSB Unset: single pseudo-merge bitmap |\n> +| + offset to pseudo-merge bitmap |\n> +| |\n> +| 2. MSB Set: multiple pseudo-merges |\n> +| + offset to extended lookup table |\n> +| |\n> ++-------------------------------------------+\n> +| |\n> +| Extended Lookup Table (Optional) |\n> +| |\n> +| +----+----------+----------+----------+ |\n> +| | N | Offset 1 | .... | Offset N | |\n> +| +----+----------+----------+----------+ |\n> +| | | 8 bytes | .... | 8 bytes | |\n> +| +----+----------+----------+----------+ |\n> +| |\n> ++-------------------------------------------+\n> +| |\n> +| Pseudo-merge Metadata |\n> +| +------------------+----------------+ |\n> +| | # pseudo-merges | # Commits | |\n> +| +------------------+----------------+ |\n> +| | 4 bytes | 4 bytes | |\n> +| +------------------+----------------+ |\n> +| |\n> +| +------------------+----------------+ |\n> +| | Lookup offset | Extension size | |\n> +| +------------------+----------------+ |\n> +| | 8 bytes | 8 bytes | |\n> +| +------------------+----------------+ |\n> +| |\n> ++-------------------------------------------+\n> +....\n> +\n> +* One or more pseudo-merge bitmaps, each containing:\n\nIn case you have multiple pseudo-merge bitmaps, is the whole of the\nabove repeated for each bitmap or is it just parts of it?\n\n> + ** `commits_bitmap`, an EWAH-compressed bitmap describing the set of\n> + commits included in the this psuedo-merge.\n> +\n> + ** `merge_bitmap`, an EWAH-compressed bitmap describing the union of\n> + the set of objects reachable from all commits listed in the\n> + `commits_bitmap`.\n> +\n> +* A lookup table, mapping pseudo-merged commits to the pseudo-merges\n> + they belong to. Entries appear in increasing order of each commit's\n> + bit position. Each entry is 12 bytes wide, and is comprised of the\n> + following:\n> +\n> + ** `commit_pos`, a 4-byte unsigned value (in network byte-order)\n> + containing the bit position for this commit.\n> +\n> + ** `offset`, an 8-byte unsigned value (also in network byte-order)\n> + containing either one of two possible offsets, depending on whether or\n> + not the most-significant bit is set.\n> +\n> + *** If unset (i.e. `offset & ((uint64_t)1<<63) == 0`), the offset\n> +\t(relative to the beginning of the `.bitmap` file) at which the\n> +\tpseudo-merge bitmap for this commit can be read. This indicates\n> +\tonly a single pseudo-merge bitmap contains this commit.\n> +\n> + *** If set (i.e. `offset & ((uint64_t)1<<63) != 0`), the offset\n> +\t(again relative to the beginning of the `.bitmap` file) at which\n> +\tthe extended offset table can be located describing the set of\n> +\tpseudo-merge bitmaps which contain this commit. This indicates\n> +\tthat multiple pseudo-merge bitmaps contain this commit.\n> +\n> +* An (optional) extended lookup table (written if and only if there is\n> + at least one commit which appears in more than one pseudo-merge).\n> + There are as many entries as commits which appear in multiple\n> + pseudo-merges. Each entry contains the following:\n> +\n> + ** `N`, a 4-byte unsigned value equal to the number of pseudo-merges\n> + which contain a given commit.\n\nHow exactly is the given commit identified? Or in other words, given an\nentry in the lookup table here, how do I figure out what commit it\nbelongs to?\n\n> + ** An array of `N` 8-byte unsigned values, each of which is\n> + interpreted as an offset (relative to the beginning of the\n> + `.bitmap` file) at which a pseudo-merge bitmap for this commit can\n> + be read. These values occur in no particular order.\n> +\n> +* Positions for all pseudo-merges, each stored as an 8-byte unsigned\n> + value (in network byte-order) containing the offset (relative to the\n> + beginnign of the `.bitmap` file) of each consecutive pseudo-merge.\n\ns/beginnign/beginning\n\nPatrick\n"},{"id":"494150","messageId":"ZjjEkivTeqC0xn_q@tanuki","threadId":"61166","inReplyTo":"3d7d930b1c5c4d122d8731ef0dc3fc90115573a2.1714422410.git.me@ttaylorr.com","subject":"Re: [PATCH v2 04/23] pack-bitmap: move some initialization to `bitmap_writer_init()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-06T11:52:50Z","receivedAt":"2024-05-06T11:52:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 29, 2024 at 04:43:08PM -0400, Taylor Blau wrote:\n> The pack-bitmap-writer machinery uses a oidmap (backed by khash.h) to\n> map from commits selected for bitmaps (by OID) to a bitmapped_commit\n> structure (containing the bitmap itself, among other things like its XOR\n> offset, etc.)\n> \n> This map was initialized at the end of `bitmap_writer_build()`. New\n> entries are added in `pack-bitmap-write.c::store_selected()`, which is\n> called by the bitmap_builder machinery (which is responsible for\n> traversing history and generating the actual bitmaps).\n> \n> Reorganize when this field is initialized and when entries are added to\n> it so that we can quickly determine whether a commit is a candidate for\n> pseudo-merge selection, or not (since it was already selected to receive\n> a bitmap, and thus is ineligible for pseudo-merge inclusion).\n\nI feel like this last sentence here could use some more explanation as\nthe restriction has never been explained before. Is this a strict\nrequirement, or is this rather \"It would be wasted anyway\"?\n\n> The changes are as follows:\n> \n> - Introduce a new `bitmap_writer_init()` function which initializes\n> the `writer.bitmaps` field (instead of waiting until the end of\n> `bitmap_writer_build()`).\n> \n> - Add map entries in `push_bitmapped_commit()` (which is called via\n> `bitmap_writer_select_commits()`) with OID keys and NULL values to\n> track whether or not we *expect* to write a bitmap for some given\n> commit.\n> \n> - Validate that a NULL entry is found matching the given key when we\n> store a selected bitmap.\n\nIt would be great if this refactoring went way further. Right now it's\nquite hard to verify whether the writer has really been initialized in\nall the right places because it is a global variable. Ideally, the whole\ninterface should be refactored to take the writer as input instead,\nwhere `bitmap_writer_init()` would then initialize the local variables.\n\nThat'd of course be a bigger refactoring and may or may not be a good\nfit for this patch series. But I'd very much love to see such a refactor\nas a follow-up series.\n\n[snip]\n> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index c35bc81d00f..9bc41a9e145 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -46,6 +46,11 @@ struct bitmap_writer {\n> \n> static struct bitmap_writer writer;\n> \n> +void bitmap_writer_init(struct repository *r)\n> +{\n> +\twriter.bitmaps = kh_init_oid_map();\n> +}\n\nGiven the other safety belts, do we also want to BUG here in case the\nbitmap has already been initialized?\n\nPatrick\n"},{"id":"494151","messageId":"ZjjEmGHYnw20wVBg@tanuki","threadId":"61166","inReplyTo":"ee33a70324589a98c2239530b03cc2d7afbdfb9e.1714422410.git.me@ttaylorr.com","subject":"Re: [PATCH v2 06/23] pack-bitmap-write: support storing pseudo-merge commits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-06T11:52:56Z","receivedAt":"2024-05-06T11:53:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 29, 2024 at 04:43:15PM -0400, Taylor Blau wrote:\n[snip]\n> @@ -46,6 +48,11 @@ struct bitmap_writer {\n> \n> static struct bitmap_writer writer;\n> \n> +static inline int bitmap_writer_selected_nr(void)\n> +{\n> +\treturn writer.selected_nr - writer.pseudo_merges_nr;\n> +}\n\nThis function may use a comment to explain what its meaning actually is.\nLike, `bitmap_writer_selected_nr()` is obviously not the same as the\n`selected_nr` of the `bitmap_writer`, which is quite confusing. So why\ndo we subtract values and why are there two different `selected_nr`s?\n\n[snip]\n> diff --git a/pack-bitmap.h b/pack-bitmap.h\n> index dae2d68a338..ca9acd2f735 100644\n> --- a/pack-bitmap.h\n> +++ b/pack-bitmap.h\n> @@ -21,6 +21,7 @@ struct bitmap_disk_header {\n> \tunsigned char checksum[GIT_MAX_RAWSZ];\n> };\n> \n> +#define BITMAP_PSEUDO_MERGE (1u<<21)\n> #define NEEDS_BITMAP (1u<<22)\n\nThis flag is already used by \"builtin/pack-objects.c\", which may be fine.\nBut in any case, shouldn't we update \"object.h\" with both of these flags?\n\nPatrick\n"},{"id":"494152","messageId":"ZjjEnVukDDpuP6X6@tanuki","threadId":"61166","inReplyTo":"86a1e4b8b9be99563836d1539fbf2ed4c4a6920d.1714422410.git.me@ttaylorr.com","subject":"Re: [PATCH v2 09/23] pseudo-merge: implement support for selecting pseudo-merge commits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-06T11:53:01Z","receivedAt":"2024-05-06T11:53:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 29, 2024 at 04:43:37PM -0400, Taylor Blau wrote:\n[snip]\n> +static uint32_t pseudo_merge_group_size(const struct pseudo_merge_group *group,\n> +\t\t\t\t\tconst struct pseudo_merge_matches *matches,\n> +\t\t\t\t\tuint32_t i)\n> +{\n> +\tfloat C = 0.0f;\n> +\tuint32_t n;\n> +\n> +\t/*\n> +\t * The size of pseudo-merge groups decays according to a power series,\n> +\t * which looks like:\n> +\t *\n> +\t * f(n) = C * n^-k\n> +\t *\n> +\t * , where 'n' is the n-th pseudo-merge group, 'f(n)' is its size, 'k'\n> +\t * is the decay rate, and 'C' is a scaling value.\n> +\t *\n> +\t * The value of C depends on the number of groups, decay rate, and total\n> +\t * number of commits. It is computed such that if there are M and N\n> +\t * total groups and commits, respectively, that:\n> +\t *\n> +\t * N = f(0) + f(1) + ... f(M-1)\n> +\t *\n> +\t * Rearranging to isolate C, we get:\n> +\t *\n> +\t * N = \\sum_{n=1}^M C / n^k\n> +\t *\n> +\t * N / C = \\sum_{n=1}^M n^-k\n> +\t *\n> +\t * C = N / \\sum_{n=1}^M n^-k\n> +\t *\n> +\t * For example, if we have a decay rate of 'k' being equal to 1.5, 'N'\n> +\t * total commits equal to 10,000, and 'M' being equal to 6 groups, then\n> +\t * the (rounded) group sizes are:\n> +\t *\n> +\t * { 5469, 1934, 1053, 684, 489, 372 }\n> +\t *\n> +\t * increasing the number of total groups, say to 10, scales the group\n> +\t * sizes appropriately:\n> +\t *\n> +\t * { 5012, 1772, 964, 626, 448, 341, 271, 221, 186, 158 }\n> +\t */\n> +\tfor (n = 0; n < group->max_merges; n++)\n> +\t\tC += 1.0f / gitexp(n + 1, group->decay);\n> +\tC = matches->unstable_nr / C;\n> +\n> +\treturn (int)((C / gitexp(i + 1, group->decay)) + 0.5);\n\nWhy do we cast the return to `int` when the function returns a\n`uint32_t`?\n\n> +}\n> +\n> +static void init_pseudo_merge_group(struct pseudo_merge_group *group)\n\nNit: Should't the name rather be `pseudo_merge_group_init()`?\n\n[snip]\n> +\t} else if (!strcmp(key, \"decay\")) {\n> +\t\tgroup->decay = git_config_int(var, value, ctx->kvi);\n> +\t\tif (group->decay < 0) {\n> +\t\t\twarning(_(\"%s must be non-negative, using default\"), var);\n> +\t\t\tgroup->decay = DEFAULT_PSEUDO_MERGE_DECAY;\n> +\t\t}\n\nThe decay is a float, and your decay rate examples mention a rate of\n1.5f. It's impossible to specify fractional rates though because we use\n`git_config_int()`. Should we introduce a new `git_config_float()`\nfunction to implement this properly?\n\n> +\t} else if (!strcmp(key, \"samplerate\")) {\n> +\t\tgroup->sample_rate = git_config_int(var, value, ctx->kvi);\n> +\t\tif (!(0 <= group->sample_rate && group->sample_rate <= 100)) {\n> +\t\t\twarning(_(\"%s must be between 0 and 100, using default\"), var);\n> +\t\t\tgroup->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE;\n> +\t\t}\n> +\t} else if (!strcmp(key, \"threshold\")) {\n> +\t\tif (git_config_expiry_date(&group->threshold, var, value)) {\n> +\t\t\tstrbuf_release(&buf);\n\nInstead of having multiple exit paths where we need to release `buf` we\nshould likely have a comment exit path.\n\n[snip]\n> +static struct commit *push_pseudo_merge(struct pseudo_merge_group *group)\n> +{\n> +\tstruct commit *merge;\n> +\n> +\tALLOC_GROW(group->merges, group->merges_nr + 1, group->merges_alloc);\n> +\n> +\tmerge = alloc_commit_node(the_repository);\n> +\tmerge->object.parsed = 1;\n\nWhy can we mark the object as parsed here?\n\n> +\tmerge->object.flags |= BITMAP_PSEUDO_MERGE;\n> +\n> +\tgroup->merges[group->merges_nr++] = merge;\n> +\n> +\treturn merge;\n> +}\n> +\n> +static struct pseudo_merge_commit_idx *pseudo_merge_idx(kh_oid_map_t *pseudo_merge_commits,\n> +\t\t\t\t\t\t\tconst struct object_id *oid)\n> +\n> +{\n> +\tstruct pseudo_merge_commit_idx *pmc;\n> +\tkhiter_t hash_pos;\n> +\n> +\thash_pos = kh_get_oid_map(pseudo_merge_commits, *oid);\n> +\tif (hash_pos == kh_end(pseudo_merge_commits)) {\n> +\t\tint hash_ret;\n> +\t\thash_pos = kh_put_oid_map(pseudo_merge_commits, *oid, &hash_ret);\n> +\t\tCALLOC_ARRAY(pmc, 1);\n> +\n> +\t\tkh_value(pseudo_merge_commits, hash_pos) = pmc;\n> +\t} else {\n> +\t\tpmc = kh_value(pseudo_merge_commits, hash_pos);\n> +\t}\n> +\n> +\treturn pmc;\n> +}\n\nCan't we simplify this to the following (untested):\n\nstatic struct pseudo_merge_commit_idx *pseudo_merge_idx(kh_oid_map_t *pseudo_merge_commits,\n const struct object_id *oid)\n{\n struct pseudo_merge_commit_idx *pmc;\n khiter_t hash_pos;\n int hash_ret;\n\n hash_pos = kh_put_oid_map(pseudo_merge_commits, *oid, &hash_ret);\n if (hash_ret) {\n CALLOC_ARRAY(pmc, 1);\n kh_value(pseudo_merge_commits, hash_pos) = pmc;\n } else {\n pmc = kh_value(pseudo_merge_commits, hash_pos);\n }\n\n return pmc;\n}\n\n> +\n> +#define MIN_PSEUDO_MERGE_SIZE 8\n> +\n> +static void select_pseudo_merges_1(struct pseudo_merge_group *group,\n> +\t\t\t\t struct pseudo_merge_matches *matches,\n> +\t\t\t\t kh_oid_map_t *pseudo_merge_commits,\n> +\t\t\t\t uint32_t *pseudo_merges_nr)\n> +{\n> +\tuint32_t i, j;\n> +\tuint32_t stable_merges_nr;\n> +\n> +\tif (!matches->stable_nr && !matches->unstable_nr)\n> +\t\treturn; /* all tips in this group already have bitmaps */\n\nIt's nice that there are some comments, but there are quite a lot of\nnon-obvious things going on in this function that would warrant an\nexplanation that expands a bit more into what exactly it is that we are\ndoing here.\n\nI may only be speaking for myself, but I basically have no clue what we\ndo here :) Something something pseudo merges, I guess. But there is no\nin-code explanation at all what a \"stable\" or \"unstable\" commit is, how\nexactly we match commits and other higher-level ideas.\n\nPatrick\n"},{"id":"494153","messageId":"ZjjEo8XU5hPPQSYf@tanuki","threadId":"61166","inReplyTo":"12b432e3a8adcda6228beae2b41b2363a6ce82a0.1714422410.git.me@ttaylorr.com","subject":"Re: [PATCH v2 10/23] pack-bitmap-write.c: select pseudo-merge commits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-06T11:53:07Z","receivedAt":"2024-05-06T11:53:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 29, 2024 at 04:43:41PM -0400, Taylor Blau wrote:\n[snip]\n> diff --git a/Documentation/config/bitmap-pseudo-merge.txt b/Documentation/config/bitmap-pseudo-merge.txt\n> new file mode 100644\n> index 00000000000..90b72522046\n> --- /dev/null\n> +++ b/Documentation/config/bitmap-pseudo-merge.txt\n> @@ -0,0 +1,75 @@\n> +bitmapPseudoMerge..pattern::\n> +\tRegular expression used to match reference names. Commits\n> +\tpointed to by references matching this pattern (and meeting\n> +\tthe below criteria, like `bitmapPseudoMerge..sampleRate`\n> +\tand `bitmapPseudoMerge..threshold`) will be considered\n> +\tfor inclusion in a pseudo-merge bitmap.\n> ++\n> +Commits are grouped into pseudo-merge groups based on whether or not\n> +any reference(s) that point at a given commit match the pattern, which\n> +is an extended regular expression.\n> ++\n> +Within a pseudo-merge group, commits may be further grouped into\n> +sub-groups based on the capture groups in the pattern. These\n> +sub-groupings are formed from the regular expressions by concatenating\n> +any capture groups from the regular expression, with a '-' dash in\n> +between.\n> ++\n> +For example, if the pattern is `refs/tags/`, then all tags (provided\n> +they meet the below criteria) will be considered candidates for the\n> +same pseudo-merge group. However, if the pattern is instead\n> +`refs/remotes/([0-9])+/tags/`, then tags from different remotes will\n> +be grouped into separate pseudo-merge groups, based on the remote\n> +number.\n> +\n> +bitmapPseudoMerge..decay::\n> +\tDetermines the rate at which consecutive pseudo-merge bitmap\n> +\tgroups decrease in size. Must be non-negative. This parameter\n> +\tcan be thought of as `k` in the function `f(n) = C *\n> +\tn^(-k/100)`, where `f(n)` is the size of the `n`th group.\n> ++\n> +Setting the decay rate equal to `0` will cause all groups to be the\n> +same size. Setting the decay rate equal to `100` will cause the `n`th\n> +group to be `1/n` the size of the initial group. Higher values of the\n> +decay rate cause consecutive groups to shrink at an increasing rate.\n> +The default is `100`.\n> +\n> +bitmapPseudoMerge..sampleRate::\n> +\tDetermines the proportion of non-bitmapped commits (among\n> +\treference tips) which are selected for inclusion in an\n> +\tunstable pseudo-merge bitmap. Must be between `0` and `100`\n> +\t(inclusive). The default is `100`.\n\nI think for this config to be actionable for anybody we need to explain\nwhat \"unstable\" or \"stable\" bitmaps are and what the tradeoff is that\nthe user needs to pick here. Like, why would I want to set this higher\nor lower than the default value, or modify it at all?\n\nI think the same is true for most of the other parts of the docs here,\nas well. We explain what those configs do, but basically leave the\nreader on their own to figure out what the real-world consequences are\nand why they would even want to configure those in the first place.\n\nI spent quite some time on this series now, so I'll stop reading at this\npoint. Thanks!\n\nPatrick\n"},{"id":"494162","messageId":"ZjkHT9XVl7ua8E14@nand.local","threadId":"61166","inReplyTo":"ZjjEjNRp2BAMWJ47@tanuki","subject":"Re: [PATCH v2 01/23] Documentation/technical: describe pseudo-merge bitmaps format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-06T16:37:35Z","receivedAt":"2024-05-06T16:37:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, May 06, 2024 at 01:52:44PM +0200, Patrick Steinhardt wrote:\n> > diff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\n> > index f5d200939b0..63a7177ac08 100644\n> > --- a/Documentation/technical/bitmap-format.txt\n> > +++ b/Documentation/technical/bitmap-format.txt\n> > @@ -255,3 +255,182 @@ triplet is -\n> > \txor_row (4 byte integer, network byte order): ::\n> > \tThe position of the triplet whose bitmap is used to compress\n> > \tthis one, or `0xffffffff` if no such bitmap exists.\n> > +\n> > +Pseudo-merge bitmaps\n> > +--------------------\n> > +\n> > +If the `BITMAP_OPT_PSEUDO_MERGES` flag is set, a variable number of\n> > +bytes (preceding the name-hash cache, commit lookup table, and trailing\n> > +checksum) of the `.bitmap` file is used to store pseudo-merge bitmaps.\n>\n> Here you say that the section is supposed to come before some other\n> sections, whereas the first sentence in the \"File format\" section says\n> that it is the last section in a bitmap file.\n\nThis is a quirk of the on-disk .bitmap format. New sections are added\nbefore existing sections, so if you were reading the file from beginning\nto end, you'd see the pseudo-merges extension, then the lookup table,\nthen the name-hash cache (assuming all were written).\n\nI think that describing them in the order they were introduced here\nmakes more sense, leaving their layout within the .bitmap file as an\nimplementation detail.\n\nIf you feel strongly otherwise, let's clean it up outside of this series\nsince this whole portion of the documentation would need to be\nreordered.\n\n> > +A \"pseudo-merge bitmap\" is used to refer to a pair of bitmaps, as\n> > +follows:\n> > +\n> > +Commit bitmap::\n> > +\n> > + A bitmap whose set bits describe the set of commits included in the\n> > + pseudo-merge's \"merge\" bitmap (as below).\n> > +\n> > +Merge bitmap::\n> > +\n> > + A bitmap whose set bits describe the reachability closure over the set\n> > + of commits in the pseudo-merge's \"commits\" bitmap (as above). An\n> > + identical bitmap would be generated for an octopus merge with the same\n> > + set of parents as described in the commits bitmap.\n> > +\n> > +Pseudo-merge bitmaps can accelerate bitmap traversals when all commits\n> > +for a given pseudo-merge are listed on either side of the traversal,\n> > +either directly (by explicitly asking for them as part of the `HAVES`\n> > +or `WANTS`) or indirectly (by encountering them during a fill-in\n> > +traversal).\n> > +\n> > +=== Use-cases\n>\n> I feel like starting with the problems that the whole feature is\n> intended to solve would help the reading flow quite a bit. So I'd move\n> this whole section up.\n\nI think we may want something in the middle, more like a \"problem\nstatement\". I wrote up a section that aims to do that, and tries to both\nbriefly describe the problem (first), as well as a small overview of\nwhat the solution is. Let me know what you think:\n\n--- 8< ---\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex 63a7177ac0..144f377e35 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -263,6 +263,26 @@ If the `BITMAP_OPT_PSEUDO_MERGES` flag is set, a variable number of\n bytes (preceding the name-hash cache, commit lookup table, and trailing\n checksum) of the `.bitmap` file is used to store pseudo-merge bitmaps.\n\n+=== Problem\n+\n+When a bitmap traversal is performed, the client may have to do some\n+amount of \"fill-in\" traversal to find the set of objects reachable from\n+references which do not have bitmap coverage within the repository.\n+\n+This fill-in traversal can be expensive, and can become a significant\n+bottleneck in repositories that have a large number of references with\n+poor bitmap coverage. Ideally every single reference would have\n+reachability bitmap coverage, but this is also not feasible as doing so\n+would reduce cache locality when reading the .bitmap file, and we would\n+also spend a significant amount of time XOR'ing individual bitmaps\n+together to generate a result.\n+\n+This section describes \"pseudo-merge bitmaps\", a new kind of\n+reachability bitmap that describes the set of objects reachable from a\n+group of references, rather than an individual reference.\n+\n+=== Overview\n+\n A \"pseudo-merge bitmap\" is used to refer to a pair of bitmaps, as\n follows:\n--- >8 ---\n\n> > +For example, suppose there exists a pseudo-merge bitmap with a large\n> > +number of commits, all of which are listed in the `WANTS` section of\n> > +some bitmap traversal query. When pseudo-merge bitmaps are enabled, the\n> > +bitmap machinery can quickly determine there is a pseudo-merge which\n> > +satisfies some subset of the wanted objects on either side of the query.\n> > +Then, we can inflate the EWAH-compressed bitmap, and `OR` it in to the\n> > +resulting bitmap. By contrast, without pseudo-merge bitmaps, we would\n> > +have to repeat the decompression and `OR`-ing step over a potentially\n> > +large number of individual bitmaps, which can take proportionally more\n> > +time.\n> > +\n> > +Another benefit of pseudo-merges arises when there is some combination\n> > +of (a) a large number of references, with (b) poor bitmap coverage, and\n> > +(c) deep, nested trees, making fill-in traversal relatively expensive.\n> > +For example, suppose that there are a large enough number of tags where\n> > +bitmapping each of the tags individually is infeasible. Without\n> > +pseudo-merge bitmaps, computing the result of, say, `git rev-list\n> > +--use-bitmap-index --count --objects --tags` would likely require a\n> > +large amount of fill-in traversal. But when a large quantity of those\n> > +tags are stored together in a pseudo-merge bitmap, the bitmap machinery\n> > +can take advantage of the fact that we only care about the union of\n> > +objects reachable from all of those tags, and answer the query much\n> > +faster.\n>\n> I would start the explanation with a discussion of the problem before\n> presenting the solution to those problems. In the current version it's\n> the other way round, you present a solution to a problem that isn't yet\n> explained\n>\n> It might also be helpful to discuss a bit who is supposed to create\n> those pseudo-merge bitmaps. Does Git do so automatically for all tags?\n> Does the admin have to configure this? If the latter, when do you want\n> to create those and what strategies are there to create them?\n\nThe pseudo-merge bitmaps are created by Git itself, configured via the\noptions described later on in this series. I'm happy to add a specific\ncall-out, but I would rather do it elsewhere outside of\nDocumentation/technical/bitmap-format.txt, which I think should be\nmostly focused on the on-disk format.\n\n> > +=== File format\n> > +\n> > +If enabled, pseudo-merge bitmaps are stored in an optional section at\n> > +the end of a `.bitmap` file. The format is as follows:\n> > +\n> > +....\n> > ++-------------------------------------------+\n> > +| .bitmap File |\n> > ++-------------------------------------------+\n> > +| |\n> > +| Pseudo-merge bitmaps (Variable Length) |\n> > +| +---------------------------+ |\n> > +| | commits_bitmap (EWAH) | |\n> > +| +---------------------------+ |\n> > +| | merge_bitmap (EWAH) | |\n> > +| +---------------------------+ |\n> > +| |\n> > ++-------------------------------------------+\n> > +| |\n> > +| Lookup Table |\n> > +| +------------+--------------+ |\n> > +| | commit_pos | offset | |\n> > +| +------------+--------------+ |\n> > +| | 4 bytes | 8 bytes | |\n> > +| +------------+--------------+ |\n>\n> It's a bit confusing that in the EWAH section above you have the type of\n> the fields in the same line as the field itself, whereas here you have\n> them formatted in a separate box. This makes the reader wonder at first\n> whether this is two or four fields. How about the following instead:\n>\n> | Lookup Table |\n> | +---------------------------+ |\n> | | commit_pos (4 bytes) | |\n> | +---------------------------+ |\n> | | offset (8 bytes) | |\n> | +---------------------------+ |\n>\n> The same comment applies to the other sections further down.\n\nMuch preferable, thanks. I made the change to use the suggested format\nhere and elsewhere in the document.\n\n> In case you have multiple pseudo-merge bitmaps, is the whole of the\n> above repeated for each bitmap or is it just parts of it?\n\nThe \"pseudo-merge bitmaps\" section contains a variable number of pairs\nof EWAH bitmaps, one pair for each pseudo-merge bitmap. I think this is\ncovered below where it says \"one or more pseudo-merge bitmaps, each\ncontaining: [...]\", but let me know if I should be more explicit.\n\n> > +* An (optional) extended lookup table (written if and only if there is\n> > + at least one commit which appears in more than one pseudo-merge).\n> > + There are as many entries as commits which appear in multiple\n> > + pseudo-merges. Each entry contains the following:\n> > +\n> > + ** `N`, a 4-byte unsigned value equal to the number of pseudo-merges\n> > + which contain a given commit.\n>\n> How exactly is the given commit identified? Or in other words, given an\n> entry in the lookup table here, how do I figure out what commit it\n> belongs to?\n\nThey aren't identified within this section. The extended lookup table is\nindexed into via the lookup table with an offset that is stored in the\n`offset` field when the MSB is set.\n\n> > + ** An array of `N` 8-byte unsigned values, each of which is\n> > + interpreted as an offset (relative to the beginning of the\n> > + `.bitmap` file) at which a pseudo-merge bitmap for this commit can\n> > + be read. These values occur in no particular order.\n> > +\n> > +* Positions for all pseudo-merges, each stored as an 8-byte unsigned\n> > + value (in network byte-order) containing the offset (relative to the\n> > + beginnign of the `.bitmap` file) of each consecutive pseudo-merge.\n>\n> s/beginnign/beginning\n\nGreat catch, thanks!\n\nThanks,\nTaylor\n"},{"id":"494175","messageId":"ZjkgcKzruDu7Qbmt@nand.local","threadId":"61166","inReplyTo":"ZjjEkivTeqC0xn_q@tanuki","subject":"Re: [PATCH v2 04/23] pack-bitmap: move some initialization to `bitmap_writer_init()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-06T18:24:48Z","receivedAt":"2024-05-06T18:24:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, May 06, 2024 at 01:52:50PM +0200, Patrick Steinhardt wrote:\n> On Mon, Apr 29, 2024 at 04:43:08PM -0400, Taylor Blau wrote:\n> > The pack-bitmap-writer machinery uses a oidmap (backed by khash.h) to\n> > map from commits selected for bitmaps (by OID) to a bitmapped_commit\n> > structure (containing the bitmap itself, among other things like its XOR\n> > offset, etc.)\n> >\n> > This map was initialized at the end of `bitmap_writer_build()`. New\n> > entries are added in `pack-bitmap-write.c::store_selected()`, which is\n> > called by the bitmap_builder machinery (which is responsible for\n> > traversing history and generating the actual bitmaps).\n> >\n> > Reorganize when this field is initialized and when entries are added to\n> > it so that we can quickly determine whether a commit is a candidate for\n> > pseudo-merge selection, or not (since it was already selected to receive\n> > a bitmap, and thus is ineligible for pseudo-merge inclusion).\n>\n> I feel like this last sentence here could use some more explanation as\n> the restriction has never been explained before. Is this a strict\n> requirement, or is this rather \"It would be wasted anyway\"?\n\nThanks, that's a great call out. I reworded this sentence to clarify\nthat it's redundant, but not a strict requirement. I think that's a\nsufficient amount of detail to motivate the change, but not so much that\nit distracts from the change at hand.\n\n> > The changes are as follows:\n> >\n> > - Introduce a new `bitmap_writer_init()` function which initializes\n> > the `writer.bitmaps` field (instead of waiting until the end of\n> > `bitmap_writer_build()`).\n> >\n> > - Add map entries in `push_bitmapped_commit()` (which is called via\n> > `bitmap_writer_select_commits()`) with OID keys and NULL values to\n> > track whether or not we *expect* to write a bitmap for some given\n> > commit.\n> >\n> > - Validate that a NULL entry is found matching the given key when we\n> > store a selected bitmap.\n>\n> It would be great if this refactoring went way further. Right now it's\n> quite hard to verify whether the writer has really been initialized in\n> all the right places because it is a global variable. Ideally, the whole\n> interface should be refactored to take the writer as input instead,\n> where `bitmap_writer_init()` would then initialize the local variables.\n>\n> That'd of course be a bigger refactoring and may or may not be a good\n> fit for this patch series. But I'd very much love to see such a refactor\n> as a follow-up series.\n\nYeah, I definitely agree here ;-). I will plan on doing this as a\nfollow-up to this series.\n\n> [snip]\n> > diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> > index c35bc81d00f..9bc41a9e145 100644\n> > --- a/pack-bitmap-write.c\n> > +++ b/pack-bitmap-write.c\n> > @@ -46,6 +46,11 @@ struct bitmap_writer {\n> >\n> > static struct bitmap_writer writer;\n> >\n> > +void bitmap_writer_init(struct repository *r)\n> > +{\n> > +\twriter.bitmaps = kh_init_oid_map();\n> > +}\n>\n> Given the other safety belts, do we also want to BUG here in case the\n> bitmap has already been initialized?\n\nGreat suggestion, thanks.\n\nThanks,\nTaylor\n"},{"id":"494178","messageId":"Zjkl6vMHCeDsJKdR@nand.local","threadId":"61166","inReplyTo":"ZjjEmGHYnw20wVBg@tanuki","subject":"Re: [PATCH v2 06/23] pack-bitmap-write: support storing pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-06T18:48:10Z","receivedAt":"2024-05-06T18:48:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, May 06, 2024 at 01:52:56PM +0200, Patrick Steinhardt wrote:\n> On Mon, Apr 29, 2024 at 04:43:15PM -0400, Taylor Blau wrote:\n> [snip]\n> > @@ -46,6 +48,11 @@ struct bitmap_writer {\n> >\n> > static struct bitmap_writer writer;\n> >\n> > +static inline int bitmap_writer_selected_nr(void)\n> > +{\n> > +\treturn writer.selected_nr - writer.pseudo_merges_nr;\n> > +}\n>\n> This function may use a comment to explain what its meaning actually is.\n> Like, `bitmap_writer_selected_nr()` is obviously not the same as the\n> `selected_nr` of the `bitmap_writer`, which is quite confusing. So why\n> do we subtract values and why are there two different `selected_nr`s?\n\nselected_nr is the total number of bitmaps we are writing (including\npseudo-merges), and writer.pseudo_merges_nr is the number of those\nbitmaps which are pseudo-merges.\n\nI renamed this function to bitmap_writer_nr_selected_commits() which\nshould clarify things, let me know if that works!\n\n> [snip]\n> > diff --git a/pack-bitmap.h b/pack-bitmap.h\n> > index dae2d68a338..ca9acd2f735 100644\n> > --- a/pack-bitmap.h\n> > +++ b/pack-bitmap.h\n> > @@ -21,6 +21,7 @@ struct bitmap_disk_header {\n> > \tunsigned char checksum[GIT_MAX_RAWSZ];\n> > };\n> >\n> > +#define BITMAP_PSEUDO_MERGE (1u<<21)\n> > #define NEEDS_BITMAP (1u<<22)\n>\n> This flag is already used by \"builtin/pack-objects.c\", which may be fine.\n> But in any case, shouldn't we update \"object.h\" with both of these flags?\n\nI can't see where in builtin/pack-objects.c this flag is used. The table\nin object.h says that bit 21 is used in:\n\n - list-objects-filter.c\n - builtin/index-pack.c\n - builtin/unpack-objects.c\n\nBut I think those are all fine. We don't call unpack-objects from the\nbitmap writing paths, and the same is true of index-pack (since we're\nwriting the pack out directly).\n\nlist-objects-filter.c should also be OK, since I am 99% sure that these\ntwo code paths do not collide, but even if they do, that field is only\nset on tree objects from the list-objects-filter.c code path, and the\nnew bits in pack-bitmap.h are only set on commit objects.\n\nRegardless, let me not forget to update the table in object.h! Thanks\nfor reminding me.\n\nThanks,\nTaylor\n"},{"id":"494194","messageId":"Zjk2UIV3kEwZUDW+@nand.local","threadId":"61166","inReplyTo":"ZjjEnVukDDpuP6X6@tanuki","subject":"Re: [PATCH v2 09/23] pseudo-merge: implement support for selecting pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-06T19:58:08Z","receivedAt":"2024-05-06T19:58:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, May 06, 2024 at 01:53:01PM +0200, Patrick Steinhardt wrote:\n> > +\tfor (n = 0; n < group->max_merges; n++)\n> > +\t\tC += 1.0f / gitexp(n + 1, group->decay);\n> > +\tC = matches->unstable_nr / C;\n> > +\n> > +\treturn (int)((C / gitexp(i + 1, group->decay)) + 0.5);\n>\n> Why do we cast the return to `int` when the function returns a\n> `uint32_t`?\n\nOops, great catch. This should cast to a uint32_t, not a signed type.\n\n> > +}\n> > +\n> > +static void init_pseudo_merge_group(struct pseudo_merge_group *group)\n>\n> Nit: Should't the name rather be `pseudo_merge_group_init()`?\n\nSure, I can change that.\n\n> [snip]\n> > +\t} else if (!strcmp(key, \"decay\")) {\n> > +\t\tgroup->decay = git_config_int(var, value, ctx->kvi);\n> > +\t\tif (group->decay < 0) {\n> > +\t\t\twarning(_(\"%s must be non-negative, using default\"), var);\n> > +\t\t\tgroup->decay = DEFAULT_PSEUDO_MERGE_DECAY;\n> > +\t\t}\n>\n> The decay is a float, and your decay rate examples mention a rate of\n> 1.5f. It's impossible to specify fractional rates though because we use\n> `git_config_int()`. Should we introduce a new `git_config_float()`\n> function to implement this properly?\n\nGood idea. I had addressed this with the sample rate by making the\nconfigured value a percentage (0-100) that was scaled down by 100, but\nfor some reason I neglected to do the same here. I'll introduce a new\nfloat parser.\n\n> > +\t} else if (!strcmp(key, \"samplerate\")) {\n> > +\t\tgroup->sample_rate = git_config_int(var, value, ctx->kvi);\n> > +\t\tif (!(0 <= group->sample_rate && group->sample_rate <= 100)) {\n> > +\t\t\twarning(_(\"%s must be between 0 and 100, using default\"), var);\n> > +\t\t\tgroup->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE;\n> > +\t\t}\n> > +\t} else if (!strcmp(key, \"threshold\")) {\n> > +\t\tif (git_config_expiry_date(&group->threshold, var, value)) {\n> > +\t\t\tstrbuf_release(&buf);\n>\n> Instead of having multiple exit paths where we need to release `buf` we\n> should likely have a comment exit path.\n\nGood call, thanks!\n\n> [snip]\n> > +static struct commit *push_pseudo_merge(struct pseudo_merge_group *group)\n> > +{\n> > +\tstruct commit *merge;\n> > +\n> > +\tALLOC_GROW(group->merges, group->merges_nr + 1, group->merges_alloc);\n> > +\n> > +\tmerge = alloc_commit_node(the_repository);\n> > +\tmerge->object.parsed = 1;\n>\n> Why can we mark the object as parsed here?\n\nWe have to mark it as parsed since there is no object buffer underlying\nthis fake commit node. If we try and parse it later on it will fail\nsince we won't be able to find a corresponding buffer.\n\n> > +\tmerge->object.flags |= BITMAP_PSEUDO_MERGE;\n> > +\n> > +\tgroup->merges[group->merges_nr++] = merge;\n> > +\n> > +\treturn merge;\n> > +}\n> > +\n> > +static struct pseudo_merge_commit_idx *pseudo_merge_idx(kh_oid_map_t *pseudo_merge_commits,\n> > +\t\t\t\t\t\t\tconst struct object_id *oid)\n> > +\n> > +{\n> > +\tstruct pseudo_merge_commit_idx *pmc;\n> > +\tkhiter_t hash_pos;\n> > +\n> > +\thash_pos = kh_get_oid_map(pseudo_merge_commits, *oid);\n> > +\tif (hash_pos == kh_end(pseudo_merge_commits)) {\n> > +\t\tint hash_ret;\n> > +\t\thash_pos = kh_put_oid_map(pseudo_merge_commits, *oid, &hash_ret);\n> > +\t\tCALLOC_ARRAY(pmc, 1);\n> > +\n> > +\t\tkh_value(pseudo_merge_commits, hash_pos) = pmc;\n> > +\t} else {\n> > +\t\tpmc = kh_value(pseudo_merge_commits, hash_pos);\n> > +\t}\n> > +\n> > +\treturn pmc;\n> > +}\n>\n> Can't we simplify this to the following (untested):\n>\n> static struct pseudo_merge_commit_idx *pseudo_merge_idx(kh_oid_map_t *pseudo_merge_commits,\n> const struct object_id *oid)\n> {\n> struct pseudo_merge_commit_idx *pmc;\n> khiter_t hash_pos;\n> int hash_ret;\n>\n> hash_pos = kh_put_oid_map(pseudo_merge_commits, *oid, &hash_ret);\n> if (hash_ret) {\n> CALLOC_ARRAY(pmc, 1);\n> kh_value(pseudo_merge_commits, hash_pos) = pmc;\n>\n> } else {\n> pmc = kh_value(pseudo_merge_commits, hash_pos);\n> }\n>\n> return pmc;\n> }\n\nNice suggestion, I think that should work great.\n\n> > +\n> > +#define MIN_PSEUDO_MERGE_SIZE 8\n> > +\n> > +static void select_pseudo_merges_1(struct pseudo_merge_group *group,\n> > +\t\t\t\t struct pseudo_merge_matches *matches,\n> > +\t\t\t\t kh_oid_map_t *pseudo_merge_commits,\n> > +\t\t\t\t uint32_t *pseudo_merges_nr)\n> > +{\n> > +\tuint32_t i, j;\n> > +\tuint32_t stable_merges_nr;\n> > +\n> > +\tif (!matches->stable_nr && !matches->unstable_nr)\n> > +\t\treturn; /* all tips in this group already have bitmaps */\n>\n> It's nice that there are some comments, but there are quite a lot of\n> non-obvious things going on in this function that would warrant an\n> explanation that expands a bit more into what exactly it is that we are\n> doing here.\n>\n> I may only be speaking for myself, but I basically have no clue what we\n> do here :) Something something pseudo merges, I guess. But there is no\n> in-code explanation at all what a \"stable\" or \"unstable\" commit is, how\n> exactly we match commits and other higher-level ideas.\n\nVery fair. I added some comments throughout to try and make this\nfunction's purpose more clear. Thanks for all of the great review so\nfar!\n\nThanks,\nTaylor\n"},{"id":"494198","messageId":"Zjk4EqblA5M6BbFV@nand.local","threadId":"61166","inReplyTo":"ZjjEo8XU5hPPQSYf@tanuki","subject":"Re: [PATCH v2 10/23] pack-bitmap-write.c: select pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-06T20:05:38Z","receivedAt":"2024-05-06T20:05:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, May 06, 2024 at 01:53:07PM +0200, Patrick Steinhardt wrote:\n> I think the same is true for most of the other parts of the docs here,\n> as well. We explain what those configs do, but basically leave the\n> reader on their own to figure out what the real-world consequences are\n> and why they would even want to configure those in the first place.\n\nI think that's a fair point, and I'm in the same boat as you are to a\nlarge extent. I have an idea of what these settings do and why you\nmight want to set them, but since there is no real-world deployment of\nthis series, I don't have any solid guidelines on when you\nshould/shouldn't set these settings.\n\nMy hope is that there are enough knobs to tweak here that anyone\ndeploying this feature could find a configuration that works for them.\n\nBut I'll try and add some general guidance on why you would want to\nchange certain settings, etc.\n\nThanks,\nTaylor\n"},{"id":"494463","messageId":"Zj4JM3ATSMice5do@tanuki","threadId":"61166","inReplyTo":"ZjkHT9XVl7ua8E14@nand.local","subject":"Re: [PATCH v2 01/23] Documentation/technical: describe pseudo-merge bitmaps format","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T11:46:59Z","receivedAt":"2024-05-10T11:47:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 06, 2024 at 12:37:35PM -0400, Taylor Blau wrote:\n> On Mon, May 06, 2024 at 01:52:44PM +0200, Patrick Steinhardt wrote:\n> > > diff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\n> > > index f5d200939b0..63a7177ac08 100644\n> > > --- a/Documentation/technical/bitmap-format.txt\n> > > +++ b/Documentation/technical/bitmap-format.txt\n> > > @@ -255,3 +255,182 @@ triplet is -\n> > > \txor_row (4 byte integer, network byte order): ::\n> > > \tThe position of the triplet whose bitmap is used to compress\n> > > \tthis one, or `0xffffffff` if no such bitmap exists.\n> > > +\n> > > +Pseudo-merge bitmaps\n> > > +--------------------\n> > > +\n> > > +If the `BITMAP_OPT_PSEUDO_MERGES` flag is set, a variable number of\n> > > +bytes (preceding the name-hash cache, commit lookup table, and trailing\n> > > +checksum) of the `.bitmap` file is used to store pseudo-merge bitmaps.\n> >\n> > Here you say that the section is supposed to come before some other\n> > sections, whereas the first sentence in the \"File format\" section says\n> > that it is the last section in a bitmap file.\n> \n> This is a quirk of the on-disk .bitmap format. New sections are added\n> before existing sections, so if you were reading the file from beginning\n> to end, you'd see the pseudo-merges extension, then the lookup table,\n> then the name-hash cache (assuming all were written).\n> \n> I think that describing them in the order they were introduced here\n> makes more sense, leaving their layout within the .bitmap file as an\n> implementation detail.\n> \n> If you feel strongly otherwise, let's clean it up outside of this series\n> since this whole portion of the documentation would need to be\n> reordered.\n\nI don't, thanks for the explanation.\n\n[snip]\n> +=== Overview\n> +\n> A \"pseudo-merge bitmap\" is used to refer to a pair of bitmaps, as\n> follows:\n> --- >8 ---\n> \n> > > +For example, suppose there exists a pseudo-merge bitmap with a large\n> > > +number of commits, all of which are listed in the `WANTS` section of\n> > > +some bitmap traversal query. When pseudo-merge bitmaps are enabled, the\n> > > +bitmap machinery can quickly determine there is a pseudo-merge which\n> > > +satisfies some subset of the wanted objects on either side of the query.\n> > > +Then, we can inflate the EWAH-compressed bitmap, and `OR` it in to the\n> > > +resulting bitmap. By contrast, without pseudo-merge bitmaps, we would\n> > > +have to repeat the decompression and `OR`-ing step over a potentially\n> > > +large number of individual bitmaps, which can take proportionally more\n> > > +time.\n> > > +\n> > > +Another benefit of pseudo-merges arises when there is some combination\n> > > +of (a) a large number of references, with (b) poor bitmap coverage, and\n> > > +(c) deep, nested trees, making fill-in traversal relatively expensive.\n> > > +For example, suppose that there are a large enough number of tags where\n> > > +bitmapping each of the tags individually is infeasible. Without\n> > > +pseudo-merge bitmaps, computing the result of, say, `git rev-list\n> > > +--use-bitmap-index --count --objects --tags` would likely require a\n> > > +large amount of fill-in traversal. But when a large quantity of those\n> > > +tags are stored together in a pseudo-merge bitmap, the bitmap machinery\n> > > +can take advantage of the fact that we only care about the union of\n> > > +objects reachable from all of those tags, and answer the query much\n> > > +faster.\n> >\n> > I would start the explanation with a discussion of the problem before\n> > presenting the solution to those problems. In the current version it's\n> > the other way round, you present a solution to a problem that isn't yet\n> > explained\n> >\n> > It might also be helpful to discuss a bit who is supposed to create\n> > those pseudo-merge bitmaps. Does Git do so automatically for all tags?\n> > Does the admin have to configure this? If the latter, when do you want\n> > to create those and what strategies are there to create them?\n> \n> The pseudo-merge bitmaps are created by Git itself, configured via the\n> options described later on in this series. I'm happy to add a specific\n> call-out, but I would rather do it elsewhere outside of\n> Documentation/technical/bitmap-format.txt, which I think should be\n> mostly focused on the on-disk format.\n\nI think what throws me off here is that you already go into the\nnon-technical somewhat by explaining their usecases. This causes us to\nend up halfwhere between \"We motivate the changes\" and \"We document the\ntechnical parts, only\".\n\n[snip]\n> > In case you have multiple pseudo-merge bitmaps, is the whole of the\n> > above repeated for each bitmap or is it just parts of it?\n> \n> The \"pseudo-merge bitmaps\" section contains a variable number of pairs\n> of EWAH bitmaps, one pair for each pseudo-merge bitmap. I think this is\n> covered below where it says \"one or more pseudo-merge bitmaps, each\n> containing: [...]\", but let me know if I should be more explicit.\n> \n> > > +* An (optional) extended lookup table (written if and only if there is\n> > > + at least one commit which appears in more than one pseudo-merge).\n> > > + There are as many entries as commits which appear in multiple\n> > > + pseudo-merges. Each entry contains the following:\n> > > +\n> > > + ** `N`, a 4-byte unsigned value equal to the number of pseudo-merges\n> > > + which contain a given commit.\n> >\n> > How exactly is the given commit identified? Or in other words, given an\n> > entry in the lookup table here, how do I figure out what commit it\n> > belongs to?\n> \n> They aren't identified within this section. The extended lookup table is\n> indexed into via the lookup table with an offset that is stored in the\n> `offset` field when the MSB is set.\n\nOkay. Would this explanation be a good addition to the document?\n\nPatrick\n"},{"id":"494464","messageId":"Zj4JOv4wJQh7GuIB@tanuki","threadId":"61166","inReplyTo":"Zjkl6vMHCeDsJKdR@nand.local","subject":"Re: [PATCH v2 06/23] pack-bitmap-write: support storing pseudo-merge commits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T11:47:06Z","receivedAt":"2024-05-10T11:47:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 06, 2024 at 02:48:10PM -0400, Taylor Blau wrote:\n> On Mon, May 06, 2024 at 01:52:56PM +0200, Patrick Steinhardt wrote:\n> > On Mon, Apr 29, 2024 at 04:43:15PM -0400, Taylor Blau wrote:\n> > [snip]\n> > > @@ -46,6 +48,11 @@ struct bitmap_writer {\n> > >\n> > > static struct bitmap_writer writer;\n> > >\n> > > +static inline int bitmap_writer_selected_nr(void)\n> > > +{\n> > > +\treturn writer.selected_nr - writer.pseudo_merges_nr;\n> > > +}\n> >\n> > This function may use a comment to explain what its meaning actually is.\n> > Like, `bitmap_writer_selected_nr()` is obviously not the same as the\n> > `selected_nr` of the `bitmap_writer`, which is quite confusing. So why\n> > do we subtract values and why are there two different `selected_nr`s?\n> \n> selected_nr is the total number of bitmaps we are writing (including\n> pseudo-merges), and writer.pseudo_merges_nr is the number of those\n> bitmaps which are pseudo-merges.\n> \n> I renamed this function to bitmap_writer_nr_selected_commits() which\n> should clarify things, let me know if that works!\n\nYup, that's clearer, thanks.\n\n> > [snip]\n> > > diff --git a/pack-bitmap.h b/pack-bitmap.h\n> > > index dae2d68a338..ca9acd2f735 100644\n> > > --- a/pack-bitmap.h\n> > > +++ b/pack-bitmap.h\n> > > @@ -21,6 +21,7 @@ struct bitmap_disk_header {\n> > > \tunsigned char checksum[GIT_MAX_RAWSZ];\n> > > };\n> > >\n> > > +#define BITMAP_PSEUDO_MERGE (1u<<21)\n> > > #define NEEDS_BITMAP (1u<<22)\n> >\n> > This flag is already used by \"builtin/pack-objects.c\", which may be fine.\n> > But in any case, shouldn't we update \"object.h\" with both of these flags?\n> \n> I can't see where in builtin/pack-objects.c this flag is used. The table\n> in object.h says that bit 21 is used in:\n\nYeah, no idea. I must've been seeing ghosts here.\n\n> - list-objects-filter.c\n> - builtin/index-pack.c\n> - builtin/unpack-objects.c\n> \n> But I think those are all fine. We don't call unpack-objects from the\n> bitmap writing paths, and the same is true of index-pack (since we're\n> writing the pack out directly).\n> \n> list-objects-filter.c should also be OK, since I am 99% sure that these\n> two code paths do not collide, but even if they do, that field is only\n> set on tree objects from the list-objects-filter.c code path, and the\n> new bits in pack-bitmap.h are only set on commit objects.\n\nOkay, thanks for the explanation!\n\nPatrick\n"},{"id":"494465","messageId":"Zj4JQQa0aBHkgXSn@tanuki","threadId":"61166","inReplyTo":"Zjk4EqblA5M6BbFV@nand.local","subject":"Re: [PATCH v2 10/23] pack-bitmap-write.c: select pseudo-merge commits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T11:47:13Z","receivedAt":"2024-05-10T11:47:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 06, 2024 at 04:05:38PM -0400, Taylor Blau wrote:\n> On Mon, May 06, 2024 at 01:53:07PM +0200, Patrick Steinhardt wrote:\n> > I think the same is true for most of the other parts of the docs here,\n> > as well. We explain what those configs do, but basically leave the\n> > reader on their own to figure out what the real-world consequences are\n> > and why they would even want to configure those in the first place.\n> \n> I think that's a fair point, and I'm in the same boat as you are to a\n> large extent. I have an idea of what these settings do and why you\n> might want to set them, but since there is no real-world deployment of\n> this series, I don't have any solid guidelines on when you\n> should/shouldn't set these settings.\n> \n> My hope is that there are enough knobs to tweak here that anyone\n> deploying this feature could find a configuration that works for them.\n> \n> But I'll try and add some general guidance on why you would want to\n> change certain settings, etc.\n\nFair enough, I can certainly relate to that. It's also the same for me\nin my patch series that introduces config knobs for the reftable writer\n[1]. I have of course benchmarked things, but benchmarking only goes so\nfar and thus isn't quite reflective of the real world.\n\nPatrick\n\n[1]: https://lore.kernel.org/git/cover.1714630191.git.ps@pks.im/\n"},{"id":"494645","messageId":"20240513184246.GA2675586@coredump.intra.peff.net","threadId":"61166","inReplyTo":"ee33a70324589a98c2239530b03cc2d7afbdfb9e.1714422410.git.me@ttaylorr.com","subject":"Re: [PATCH v2 06/23] pack-bitmap-write: support storing pseudo-merge commits","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-13T18:42:46Z","receivedAt":"2024-05-13T18:42:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 29, 2024 at 04:43:15PM -0400, Taylor Blau wrote:\n\n> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index 9bc41a9e145..fef02cd745a 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -24,7 +24,7 @@ struct bitmapped_commit {\n> \tstruct ewah_bitmap *write_as;\n> \tint flags;\n> \tint xor_offset;\n> -\tuint32_t commit_pos;\n> +\tunsigned pseudo_merge : 1;\n> };\n\nThe addition of the bit flag here makes sense, but dropping commit_pos\ncaught me by surprise. But...it looks like that flag is simply unused\ncruft even before this patch?\n\nIt might be worth noting that in the commit message, or better still,\npulling its removal out to a preparatory patch.\n\n> +static inline int bitmap_writer_selected_nr(void)\n> +{\n> +\treturn writer.selected_nr - writer.pseudo_merges_nr;\n> +}\n\nOK, so now most spots should use this new function instead of looking at\nwriter.selected_nr directly. But if anybody accidentally uses the old\nfield directly, it is presumably disastrous. Is it worth renaming it to\nmake sure we caught all references?\n\nThe downside would be that spots which _do_ want the complete\nselected_nr would need to be updated to use the new name. It doesn't\nlook like there are that many, though. OTOH, that means that it's also\neasy to inspect them and see that you covered all of the relevant cases\n(as far as I can see). I guess the biggest value in changing the field\nname would be catching any topics in flight (or long-running forks).\n\n-Peff\n"},{"id":"494646","messageId":"20240513185055.GB2675586@coredump.intra.peff.net","threadId":"61166","inReplyTo":"dfd4b73d12edc1833e5090c956cda6f28046c9e4.1714422410.git.me@ttaylorr.com","subject":"Re: [PATCH v2 08/23] pack-bitmap: make `bitmap_writer_push_bitmapped_commit()` public","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-13T18:50:55Z","receivedAt":"2024-05-13T18:50:56Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 29, 2024 at 04:43:22PM -0400, Taylor Blau wrote:\n\n> The pseudo-merge selection code will be added in a subsequent commit,\n> and will need a way to push the allocated commit structures into the\n> bitmap writer from a separate compilation unit.\n> \n> Make the `bitmap_writer_push_bitmapped_commit()` function part of the\n> pack-bitmap.h header in order to make this possible.\n\nI was a little surprised that this function and the one in the previous\ncommit needed to be public, since this whole topic is restricted to\nwriting, which is mostly contained to pack-bitmap-write.c. But you've\npulled the pseudo-merge bits out to pseudo-merge.[ch], and they need\naccess, which makes sense.\n\nOne could argue that it could all get stuffed into pack-bitmap-write.c,\nbut that is already getting to be a pretty large and complex file. So\nthis is probably the best route.\n\n-Peff\n"},{"id":"494650","messageId":"20240513190340.GC2675586@coredump.intra.peff.net","threadId":"61166","inReplyTo":"86a1e4b8b9be99563836d1539fbf2ed4c4a6920d.1714422410.git.me@ttaylorr.com","subject":"Re: [PATCH v2 09/23] pseudo-merge: implement support for selecting pseudo-merge commits","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-13T19:03:40Z","receivedAt":"2024-05-13T19:03:41Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 29, 2024 at 04:43:37PM -0400, Taylor Blau wrote:\n\n> Teach the new pseudo-merge machinery how to select non-bitmapped commits\n> for inclusion in different pseudo-merge group(s) based on a handful of\n> criteria.\n> \n> Pseudo-merges are derived first from named pseudo-merge groups (see the\n> `bitmapPseudoMerge..*` configuration options). They are\n> (optionally) further segmented within an individual pseudo-merge group\n> based on any capture group(s) within the pseudo-merge group's pattern.\n> \n> For example, a configuration like so:\n> \n> [bitmapPseudoMerge \"all\"]\n> pattern = \"refs/\"\n> threshold = now\n> stableThreshold = never\n> sampleRate = 100\n> maxMerges = 64\n> \n> would group all non-bitmapped commits into up to 64 individual\n> pseudo-merge commits.\n\nI was going to complain that explanatory text like this should probably\ngo into the documentation, not a commit message. But I see you do later\nadd documentation. Which seems to happen when this code is actually\nwired up to the bitmap-writer. Maybe a moot point now that I figured it\nout, but I think we'd be better off with the two commits squashed\ntogether.\n\nAnd consider whether this commit message can be shortened a lot to just\nrefer to the embedded docs (and especially if there is any useful info\nhere that is not covered in the docs, and should be moved there). I do\nthink some of these explanatory examples are good for users, but we\ndon't necessarily have a good spot to put them. The git-config\ndocumentation is more of a reference, and huge example sections would\nprobably bog it down. Maybe in the EXAMPLES section of pack-objects?\nIt's already a pretty big manpage, though, and this is just one tiny\ncorner.\n\nSo I dunno. I just want to make sure we don't bury useful information in\na commit message that most people won't see (something I've definitely\nbeen guilty of in the past, and which has later caused problems).\n\n\nI've got to break here in reviewing for the moment, but I think in a lot\nof ways this commit is going to be the most interesting one, because the\nusefulness of the whole pseudo-merge scheme depends on picking good sets\nof commits (ones that cover a lot of ground but have a low chance of\nbeing invalidated). So I'll pick up again with a careful look at this\none.\n\n-Peff\n"},{"id":"494658","messageId":"ZkJuNSWLRcPjPT6S@nand.local","threadId":"61166","inReplyTo":"Zj4JM3ATSMice5do@tanuki","subject":"Re: [PATCH v2 01/23] Documentation/technical: describe pseudo-merge bitmaps format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-13T19:47:01Z","receivedAt":"2024-05-13T19:47:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, May 10, 2024 at 01:46:59PM +0200, Patrick Steinhardt wrote:\n> > > > +* An (optional) extended lookup table (written if and only if there is\n> > > > + at least one commit which appears in more than one pseudo-merge).\n> > > > + There are as many entries as commits which appear in multiple\n> > > > + pseudo-merges. Each entry contains the following:\n> > > > +\n> > > > + ** `N`, a 4-byte unsigned value equal to the number of pseudo-merges\n> > > > + which contain a given commit.\n> > >\n> > > How exactly is the given commit identified? Or in other words, given an\n> > > entry in the lookup table here, how do I figure out what commit it\n> > > belongs to?\n> >\n> > They aren't identified within this section. The extended lookup table is\n> > indexed into via the lookup table with an offset that is stored in the\n> > `offset` field when the MSB is set.\n>\n> Okay. Would this explanation be a good addition to the document?\n\nI think we already have this written down in the section above. See in\nthe previous bullet point the section reading \"containing either one of\ntwo possible offsets, deepening on whether or not the most-significant\nbit is set: [...]\"\n\nDoes that work?\n\nThanks,\nTaylor\n"},{"id":"494662","messageId":"ZkJ14LHreSas23ZV@nand.local","threadId":"61166","inReplyTo":"20240513184246.GA2675586@coredump.intra.peff.net","subject":"Re: [PATCH v2 06/23] pack-bitmap-write: support storing pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-13T20:19:44Z","receivedAt":"2024-05-13T20:19:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, May 13, 2024 at 02:42:46PM -0400, Jeff King wrote:\n> On Mon, Apr 29, 2024 at 04:43:15PM -0400, Taylor Blau wrote:\n>\n> > diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> > index 9bc41a9e145..fef02cd745a 100644\n> > --- a/pack-bitmap-write.c\n> > +++ b/pack-bitmap-write.c\n> > @@ -24,7 +24,7 @@ struct bitmapped_commit {\n> > \tstruct ewah_bitmap *write_as;\n> > \tint flags;\n> > \tint xor_offset;\n> > -\tuint32_t commit_pos;\n> > +\tunsigned pseudo_merge : 1;\n> > };\n>\n> The addition of the bit flag here makes sense, but dropping commit_pos\n> caught me by surprise. But...it looks like that flag is simply unused\n> cruft even before this patch?\n>\n> It might be worth noting that in the commit message, or better still,\n> pulling its removal out to a preparatory patch.\n\nHah, so this is a funny one :-).\n\nI was following your suggestion to pull out the deletion into its own\npatch[^1] and starting to dig out back-references to indicate why it was\nsafe to remove this field.\n\nBut the only reference to commit_pos is from 7cc8f971085 (pack-objects:\nimplement bitmap writing, 2013-12-21), which is the commit that added\nthis field in the first place. Looking at:\n\n $ git log -p -S commit_pos 7cc8f971085 -- pack-bitmap-write.c\n\ndoesn't really show us anything interesting, either.\n\nBut! There is an array called commit_positions, which I suspected was\nfor holding the values of commit_pos in the same order as they appear in\nthe writer.selected array.\n\nSo I think the right patch is something like this (which I'll put in the\nnext round of this series):\n\n--- 8< ---\nSubject: [PATCH] pack-bitmap-write.c: move commit_positions into commit_pos\n fields\n\nIn 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21), the\nbitmapped_commit struct was introduced, including the 'commit_pos'\nfield, which has been unused ever since its introduction more than a\ndecade ago.\n\nInstead, we have used the nearby `commit_positions` array leaving the\nbitmapped_commit struct with an unused 4-byte field.\n\nWe could drop the `commit_pos` field as unused, and continue to store\nthe values in the auxiliary array. But we could also drop the array and\nstore the data for each bitmapped_commit struct inside of the structure\nitself, which is what this patch does.\n\nIn any spot that we previously read `commit_positions[i]`, we can now\ninstead read `writer.selected[i].commit_pos`. There are a few spots that\nneed changing as a result:\n\n - write_selected_commits_v1() is a simple transformation, since we're\n just reading the field. As a result, the function no longer needs an\n explicit argument to pass the commit_positions array.\n\n - write_lookup_table() also no longer needs the explicit\n commit_positions array passed in as an argument. But it still needs\n to sort an array of indices into the writer.selected array to read\n them in commit_pos order, so table_cmp() is adjusted accordingly.\n\n - bitmap_writer_finish() no longer needs to allocate, populate, and\n free the commit_positions table. Instead, we can just write the data\n directly into each struct bitmapped_commit.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 42 ++++++++++++++++--------------------------\n 1 file changed, 16 insertions(+), 26 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 473a0fa0d40..26f57e48804 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -679,9 +679,7 @@ static const struct object_id *oid_access(size_t pos, const void *table)\n \treturn &index[pos]->oid;\n }\n\n-static void write_selected_commits_v1(struct hashfile *f,\n-\t\t\t\t uint32_t *commit_positions,\n-\t\t\t\t off_t *offsets)\n+static void write_selected_commits_v1(struct hashfile *f, off_t *offsets)\n {\n \tint i;\n\n@@ -691,7 +689,7 @@ static void write_selected_commits_v1(struct hashfile *f,\n \t\tif (offsets)\n \t\t\toffsets[i] = hashfile_total(f);\n\n-\t\thashwrite_be32(f, commit_positions[i]);\n+\t\thashwrite_be32(f, stored->commit_pos);\n \t\thashwrite_u8(f, stored->xor_offset);\n \t\thashwrite_u8(f, stored->flags);\n\n@@ -699,23 +697,20 @@ static void write_selected_commits_v1(struct hashfile *f,\n \t}\n }\n\n-static int table_cmp(const void *_va, const void *_vb, void *_data)\n+static int table_cmp(const void *_va, const void *_vb)\n {\n-\tuint32_t *commit_positions = _data;\n-\tuint32_t a = commit_positions[*(uint32_t *)_va];\n-\tuint32_t b = commit_positions[*(uint32_t *)_vb];\n+\tstruct bitmapped_commit *a = &writer.selected[*(uint32_t *)_va];\n+\tstruct bitmapped_commit *b = &writer.selected[*(uint32_t *)_vb];\n\n-\tif (a > b)\n+\tif (a->commit_pos < b->commit_pos)\n+\t\treturn -1;\n+\telse if (a->commit_pos > b->commit_pos)\n \t\treturn 1;\n-\telse if (a < b)\n-\t\treturn -1;\n\n \treturn 0;\n }\n\n-static void write_lookup_table(struct hashfile *f,\n-\t\t\t uint32_t *commit_positions,\n-\t\t\t off_t *offsets)\n+static void write_lookup_table(struct hashfile *f, off_t *offsets)\n {\n \tuint32_t i;\n \tuint32_t *table, *table_inv;\n@@ -731,7 +726,7 @@ static void write_lookup_table(struct hashfile *f,\n \t * bitmap corresponds to j'th bitmapped commit (among the selected\n \t * commits) in lex order of OIDs.\n \t */\n-\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n+\tQSORT(table, writer.selected_nr, table_cmp);\n\n \t/* table_inv helps us discover that relationship (i'th bitmap\n \t * to j'th commit by j = table_inv[i])\n@@ -762,7 +757,7 @@ static void write_lookup_table(struct hashfile *f,\n \t\t\txor_row = 0xffffffff;\n \t\t}\n\n-\t\thashwrite_be32(f, commit_positions[table[i]]);\n+\t\thashwrite_be32(f, writer.selected[table[i]].commit_pos);\n \t\thashwrite_be64(f, (uint64_t)offsets[table[i]]);\n \t\thashwrite_be32(f, xor_row);\n \t}\n@@ -798,7 +793,6 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \tstatic uint16_t flags = BITMAP_OPT_FULL_DAG;\n \tstruct strbuf tmp_file = STRBUF_INIT;\n \tstruct hashfile *f;\n-\tuint32_t *commit_positions = NULL;\n \toff_t *offsets = NULL;\n \tuint32_t i;\n\n@@ -823,22 +817,19 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n \t\tCALLOC_ARRAY(offsets, index_nr);\n\n-\tALLOC_ARRAY(commit_positions, writer.selected_nr);\n-\n \tfor (i = 0; i < writer.selected_nr; i++) {\n \t\tstruct bitmapped_commit *stored = &writer.selected[i];\n-\t\tint commit_pos = oid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n+\t\tstored->commit_pos = oid_pos(&stored->commit->object.oid, index,\n+\t\t\t\t\t index_nr, oid_access);\n\n-\t\tif (commit_pos < 0)\n+\t\tif (stored->commit_pos < 0)\n \t\t\tBUG(_(\"trying to write commit not in index\"));\n-\n-\t\tcommit_positions[i] = commit_pos;\n \t}\n\n-\twrite_selected_commits_v1(f, commit_positions, offsets);\n+\twrite_selected_commits_v1(f, offsets);\n\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n-\t\twrite_lookup_table(f, commit_positions, offsets);\n+\t\twrite_lookup_table(f, offsets);\n\n \tif (options & BITMAP_OPT_HASH_CACHE)\n \t\twrite_hash_cache(f, index, index_nr);\n@@ -853,6 +844,5 @@ void bitmap_writer_finish(struct pack_idx_entry **index,\n \t\tdie_errno(\"unable to rename temporary bitmap file to '%s'\", filename);\n\n \tstrbuf_release(&tmp_file);\n-\tfree(commit_positions);\n \tfree(offsets);\n }\n\n--\n2.45.0.57.gee4186f79f3\n\n--- >8 ---\n\n> > +static inline int bitmap_writer_selected_nr(void)\n> > +{\n> > +\treturn writer.selected_nr - writer.pseudo_merges_nr;\n> > +}\n>\n> OK, so now most spots should use this new function instead of looking at\n> writer.selected_nr directly. But if anybody accidentally uses the old\n> field directly, it is presumably disastrous. Is it worth renaming it to\n> make sure we caught all references?\n\nWe only need to check within this file, since the bitmap_writer\nstructure definition is defined within the pack-bitmap-writer.c\ncompilation unit.\n\nI took a careful look through the file, and am confident that we touched\nall of the spots that needed attention.\n\nThanks,\nTaylor\n\n[^1]: If memory serves, that was my original intention when writing this\n series for the first time, but I must have forgotten when I was\n actually splitting out the individual patches and staged the removal\n alongside the rest of this change.\n"},{"id":"494685","messageId":"ZkK2Y06dgyLanNEA@nand.local","threadId":"61166","inReplyTo":"20240513185055.GB2675586@coredump.intra.peff.net","subject":"Re: [PATCH v2 08/23] pack-bitmap: make `bitmap_writer_push_bitmapped_commit()` public","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-14T00:54:59Z","receivedAt":"2024-05-14T00:55:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, May 13, 2024 at 02:50:55PM -0400, Jeff King wrote:\n> On Mon, Apr 29, 2024 at 04:43:22PM -0400, Taylor Blau wrote:\n>\n> > The pseudo-merge selection code will be added in a subsequent commit,\n> > and will need a way to push the allocated commit structures into the\n> > bitmap writer from a separate compilation unit.\n> >\n> > Make the `bitmap_writer_push_bitmapped_commit()` function part of the\n> > pack-bitmap.h header in order to make this possible.\n>\n> I was a little surprised that this function and the one in the previous\n> commit needed to be public, since this whole topic is restricted to\n> writing, which is mostly contained to pack-bitmap-write.c. But you've\n> pulled the pseudo-merge bits out to pseudo-merge.[ch], and they need\n> access, which makes sense.\n>\n> One could argue that it could all get stuffed into pack-bitmap-write.c,\n> but that is already getting to be a pretty large and complex file. So\n> this is probably the best route.\n\nI had originally written the series like that, but the new bits nearly\ndoubled the line count of pack-bitmap-write.c:\n\n $ wc -l pack-bitmap-write.c pseudo-merge.c\n 1031 pack-bitmap-write.c\n 752 pseudo-merge.c\n 1783 total\n\nso I ended up splitting it out into pseudo-merge.ch in the end.\n\nThanks,\nTaylor\n"},{"id":"494686","messageId":"ZkK3KYkA6+zUxu9A@nand.local","threadId":"61166","inReplyTo":"20240513190340.GC2675586@coredump.intra.peff.net","subject":"Re: [PATCH v2 09/23] pseudo-merge: implement support for selecting pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-14T00:58:17Z","receivedAt":"2024-05-14T00:58:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, May 13, 2024 at 03:03:40PM -0400, Jeff King wrote:\n> On Mon, Apr 29, 2024 at 04:43:37PM -0400, Taylor Blau wrote:\n>\n> > Teach the new pseudo-merge machinery how to select non-bitmapped commits\n> > for inclusion in different pseudo-merge group(s) based on a handful of\n> > criteria.\n> >\n> > Pseudo-merges are derived first from named pseudo-merge groups (see the\n> > `bitmapPseudoMerge..*` configuration options). They are\n> > (optionally) further segmented within an individual pseudo-merge group\n> > based on any capture group(s) within the pseudo-merge group's pattern.\n> >\n> > For example, a configuration like so:\n> >\n> > [bitmapPseudoMerge \"all\"]\n> > pattern = \"refs/\"\n> > threshold = now\n> > stableThreshold = never\n> > sampleRate = 100\n> > maxMerges = 64\n> >\n> > would group all non-bitmapped commits into up to 64 individual\n> > pseudo-merge commits.\n>\n> I was going to complain that explanatory text like this should probably\n> go into the documentation, not a commit message. But I see you do later\n> add documentation. Which seems to happen when this code is actually\n> wired up to the bitmap-writer. Maybe a moot point now that I figured it\n> out, but I think we'd be better off with the two commits squashed\n> together.\n\nI dunno. This commit is already rather large, and I like the split of\n\"here's how we select these things\", versus \"now we actually start\nselecting/writing them\".\n\nBut maybe it results in a slightly awkward break in the middle that\nleaves some of the stuff that would otherwise fit well in the EXAMPLES\nsection (as you mention below) in a weird limbo state.\n\n> And consider whether this commit message can be shortened a lot to just\n> refer to the embedded docs (and especially if there is any useful info\n> here that is not covered in the docs, and should be moved there). I do\n> think some of these explanatory examples are good for users, but we\n> don't necessarily have a good spot to put them. The git-config\n> documentation is more of a reference, and huge example sections would\n> probably bog it down. Maybe in the EXAMPLES section of pack-objects?\n> It's already a pretty big manpage, though, and this is just one tiny\n> corner.\n\nThere's a good amount of information already in\nDocumentation/technical/bitmap-format.txt, though perhaps some of the\npieces mentioned here could be added there. Let me know if you think one\nis missing something the other has (or if we could move significant\nportions of the latter into the former).\n\n> I've got to break here in reviewing for the moment, but I think in a lot\n> of ways this commit is going to be the most interesting one, because the\n> usefulness of the whole pseudo-merge scheme depends on picking good sets\n> of commits (ones that cover a lot of ground but have a low chance of\n> being invalidated). So I'll pick up again with a careful look at this\n> one.\n\nI agree that this is where things start to get interesting ;-). I'm\nlooking forward to your review when you get back to it!\n\nThanks,\nTaylor\n"},{"id":"494701","messageId":"ZkMFx6byw2ersCXg@tanuki","threadId":"61166","inReplyTo":"ZkJuNSWLRcPjPT6S@nand.local","subject":"Re: [PATCH v2 01/23] Documentation/technical: describe pseudo-merge bitmaps format","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-14T06:33:43Z","receivedAt":"2024-05-14T06:33:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 13, 2024 at 03:47:01PM -0400, Taylor Blau wrote:\n> On Fri, May 10, 2024 at 01:46:59PM +0200, Patrick Steinhardt wrote:\n> > > > > +* An (optional) extended lookup table (written if and only if there is\n> > > > > + at least one commit which appears in more than one pseudo-merge).\n> > > > > + There are as many entries as commits which appear in multiple\n> > > > > + pseudo-merges. Each entry contains the following:\n> > > > > +\n> > > > > + ** `N`, a 4-byte unsigned value equal to the number of pseudo-merges\n> > > > > + which contain a given commit.\n> > > >\n> > > > How exactly is the given commit identified? Or in other words, given an\n> > > > entry in the lookup table here, how do I figure out what commit it\n> > > > belongs to?\n> > >\n> > > They aren't identified within this section. The extended lookup table is\n> > > indexed into via the lookup table with an offset that is stored in the\n> > > `offset` field when the MSB is set.\n> >\n> > Okay. Would this explanation be a good addition to the document?\n> \n> I think we already have this written down in the section above. See in\n> the previous bullet point the section reading \"containing either one of\n> two possible offsets, deepening on whether or not the most-significant\n> bit is set: [...]\"\n> \n> Does that work?\n\nOne could have a back-reference to that section here. But I don't mind\nit too much overall.\n\nThanks!\n\nPatrick\n"},{"id":"494875","messageId":"20240516080704.GA85197@coredump.intra.peff.net","threadId":"61166","inReplyTo":"ZkK3KYkA6+zUxu9A@nand.local","subject":"Re: [PATCH v2 09/23] pseudo-merge: implement support for selecting pseudo-merge commits","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-16T08:07:04Z","receivedAt":"2024-05-16T08:07:05Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, May 13, 2024 at 08:58:17PM -0400, Taylor Blau wrote:\n\n> > I was going to complain that explanatory text like this should probably\n> > go into the documentation, not a commit message. But I see you do later\n> > add documentation. Which seems to happen when this code is actually\n> > wired up to the bitmap-writer. Maybe a moot point now that I figured it\n> > out, but I think we'd be better off with the two commits squashed\n> > together.\n> \n> I dunno. This commit is already rather large, and I like the split of\n> \"here's how we select these things\", versus \"now we actually start\n> selecting/writing them\".\n> \n> But maybe it results in a slightly awkward break in the middle that\n> leaves some of the stuff that would otherwise fit well in the EXAMPLES\n> section (as you mention below) in a weird limbo state.\n\nIt's not the break to me so much as the fact that you end up explaining\nthe concepts twice. Is it the same material presented in two ways? Or is\nthere stuff in one spot that is not in the other? I think the answer is\nthat it's a little bit of both. And as a reviewer (and an author) it's\nhard to put yourself in the shoes of a user who is only going to see\nwhat's in the docs.\n\nSo I'd rather even see the first patch as \"here's some config; don't\nworry about what it does too much, as we'll explain it in the next\ncommit\" and then in the second patch say \"go look at the config added by\nthis patch\". And then we know we're looking at the same thing a user\nwill.\n\n> There's a good amount of information already in\n> Documentation/technical/bitmap-format.txt, though perhaps some of the\n> pieces mentioned here could be added there. Let me know if you think one\n> is missing something the other has (or if we could move significant\n> portions of the latter into the former).\n\nI don't think we should expect most users to read anything in\nDocumentation/technical. Now I don't expect most users to fiddle with\nthis feature at all. But reading over the config docs added by the\nsubsequent patch, it's not at all clear to me when I would want to tweak\nthe knobs or why.\n\nI think there might need to be an \"advanced packing concepts\"\nuser-directed manual. Either as part of git-pack-objects(1), or maybe\nbroken out into its own page (\"gitpacking(7)\" or similar). Specifically\nfor this feature, I think it would want to cover:\n\n - what is this thing, and why would I want it. You cover this in the\n format doc, but I think it makes more sense in a user-directed doc\n (and to leave the format doc strictly as a technical reference).\n\n - what you wrote in \"use cases\" there is still, IMHO, introducing\n things in the wrong order for a regular user. They're going to come\n to the documentation either with a specific problem, or with an idea\n to browse around for things that might help them.\n\n So I feel like it needs to start with the concept and the\n motivation. Something like (assuming we're following a section on\n bitmaps in the \"advanced packing\" page, something I recognize also\n does not yet exist):\n\n Reachability bitmaps are most efficient when we have on-disk\n stored bitmaps for one or more of the starting points of a\n traversal. For this reason, Git prefers storing bitmaps for\n commits at the tips of refs, because traversals tend to start with\n those points.\n\n But if you have a large number of refs, it's not feasible to store\n a bitmap for _every_ ref tip. It takes up space, and just OR-ing\n all of those bitmaps together is expensive.\n\n One way we can deal with that is to create bitmaps that represent\n _groups_ of refs. When a traversal asks about the entire group,\n then we can use this single bitmap instead of considering each ref\n individually. Because these bitmaps represent the set of objects\n which would be reachable in a hypothetical merge of all of the\n commits, we call them pseudo-merge bitmaps.\n\n I don't think this is saying anything that your technical doc\n doesn't, but I think it's more important what it _isn't_ saying.\n We don't need to talk about commit bitmaps and merge bitmaps at all.\n We just want the user to have the concept of grouping refs. And then\n that would hopefully lead naturally into \"OK, so how do we group\n them\".\n\n - OK, so how do we group them? ;) I think there are two concerns here.\n One is that traversals can only use a pseudo-merge bitmap if _every_\n commit in its group is included in the traversal. So we want to\n group the refs along logical boundaries (e.g., tags vs branches vs\n remotes). Or in the case of shared-object repositories (like\n GitHub's), by boundaries which only the user knows about.\n\n And two is that we want groups that don't become invalidated when a\n ref changes or is removed. So you want a bunch of old, stable tags\n together, and probably don't want recent branches grouped at all.\n\n And then when we describe the config knobs you can turn, it should\n be in the user's mind how they can use them to influence those two\n things. For \"logical boundaries\" part, I think the commit message\n for patch 9 does some of that with the \"refs/virtual\" example. But\n that's something I don't see as clearly in the config documentation\n added in patch 10.\n\n The knobs for handling age are more complicated and harder to\n explain. ;) You do mention the power-law decay thing in patch 10,\n but it's in the technical format docs. I think it should be\n somewhere more user-accessible.\n\nSo hopefully that kind of lays out how I'm thinking about it. Both where\nthe docs go, but what I think are the useful ways to be thinking about\nthe feature. And not just for users, but as we see if the design is\ndoing a good job of fulfilling those needs.\n\nI think the name/pattern config you introduce does cover the logical\nboundaries in a clean and easy-to-understand way. The \"age\" stuff is\nmuch fuzzier in my mind. Your power-law decay makes sense to me, though\nit does have a lot of knobs, and I don't think we'll really understand\nhow it performs without real-world experimentation.\n\nI do wonder if something stupidly simple like \"just make a single group\nincluding tags older than 3 months, and ignore everything else\" (where\n\"single\" is \"single per logical boundary defined by the user\") would\nperform OK in practice. The point is to help \"--all\" and \"--not --all\"\nwhen you have a bunch of crufty old refs. So really, the challenge is\nmostly just identifying crufty old refs. ;)\n\nBut I do think your power-law stuff should be a functional superset of\nthat. And while it's complicated to reason about where the knobs should\nbe set, I don't think the code is very complex. And the fact that it\n_has_ knobs gives something to tweak and gather data with.\n\nAll of which is to say, I guess, that I think the code is going in a\nreasonable direction. It's hard to say much more without spending a\nbunch of time benchmarking real repositories, their traversal queries,\nand so on.\n\n-Peff\n"},{"id":"494917","messageId":"xmqqzfspml8g.fsf@gitster.g","threadId":"61166","inReplyTo":"20240516080704.GA85197@coredump.intra.peff.net","subject":"Re: [PATCH v2 09/23] pseudo-merge: implement support for selecting pseudo-merge commits","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-16T22:43:11Z","receivedAt":"2024-05-16T22:43:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> motivation. Something like (assuming we're following a section on\n> bitmaps in the \"advanced packing\" page, something I recognize also\n> does not yet exist):\n>\n> Reachability bitmaps are most efficient when we have on-disk\n> stored bitmaps for one or more of the starting points of a\n> traversal. For this reason, Git prefers storing bitmaps for\n> commits at the tips of refs, because traversals tend to start with\n> those points.\n>\n> But if you have a large number of refs, it's not feasible to store\n> a bitmap for _every_ ref tip. It takes up space, and just OR-ing\n> all of those bitmaps together is expensive.\n>\n> One way we can deal with that is to create bitmaps that represent\n> _groups_ of refs. When a traversal asks about the entire group,\n> then we can use this single bitmap instead of considering each ref\n> individually. Because these bitmaps represent the set of objects\n> which would be reachable in a hypothetical merge of all of the\n> commits, we call them pseudo-merge bitmaps.\n\nNicely put. I wish there were something like the above in the\npatches when I read these patches for the first time. The concept\nof \"pseudo-merge\" was the first hump in the road to understanding.\nEventually I figured it out, but a simple write-up like the above\nwould have helped readers a lot.\n\nThanks.\n"},{"id":"495183","messageId":"cover.1716318088.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH v3 00/30] pack-bitmap: pseudo-merge reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:01:38Z","receivedAt":"2024-05-21T19:01:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is another reroll my topic to introduce pseudo-merge bitmaps.\n\nThe implementation is relatively unchanged compared to last time, save\nfor the review that Patrick provided on the first half or so of this\nseries.\n\nThe notable change from last time is a significant reorganization of the\ndocumentation, description, motivation, and examples around what\npseudo-merge bitmaps are why you might want to use them. I took Peff's\nsuggestion to create a new gitpacking(7) manual page which I hope will\neventually grow to cover many of the more advanced concepts related to\npacking objects [^1].\n\nThe other change is that this series is now based on\n'tb/pack-bitmap-write-cleanups'.\n\nOtherwise, a range-diff is included below for convenience. Thanks in\nadvance for your review!\n\n[^1]: At present, the new manual page is rather bare :-). I want to\n separate the task of collecting all of the existing documentation\n around advanced packing concepts from pseudo-merge bitmaps and not add\n a new dependency here.\n\nTaylor Blau (30):\n object.h: add flags allocated by pack-bitmap.h\n pack-bitmap-write.c: move commit_positions into commit_pos fields\n pack-bitmap: avoid use of static `bitmap_writer`\n pack-bitmap: drop unused `max_bitmaps` parameter\n pack-bitmap-write.c: avoid uninitialized 'write_as' field\n pack-bitmap: introduce `bitmap_writer_free()`\n Documentation/gitpacking.txt: initial commit\n Documentation/gitpacking.txt: describe pseudo-merge bitmaps\n Documentation/technical: describe pseudo-merge bitmaps format\n ewah: implement `ewah_bitmap_is_subset()`\n pack-bitmap: move some initialization to `bitmap_writer_init()`\n pseudo-merge.ch: initial commit\n pack-bitmap-write: support storing pseudo-merge commits\n pack-bitmap: implement `bitmap_writer_has_bitmapped_object_id()`\n pack-bitmap: make `bitmap_writer_push_bitmapped_commit()` public\n config: introduce git_config_float()\n pseudo-merge: implement support for selecting pseudo-merge commits\n pack-bitmap-write.c: write pseudo-merge table\n pack-bitmap: extract `read_bitmap()` function\n pseudo-merge: scaffolding for reads\n pack-bitmap.c: read pseudo-merge extension\n pseudo-merge: implement support for reading pseudo-merge commits\n ewah: implement `ewah_bitmap_popcount()`\n pack-bitmap: implement test helpers for pseudo-merge\n t/test-lib-functions.sh: support `--date` in `test_commit_bulk()`\n pack-bitmap.c: use pseudo-merges during traversal\n pack-bitmap: extra trace2 information\n ewah: `bitmap_equals_ewah()`\n pseudo-merge: implement support for finding existing merges\n t/perf: implement performace tests for pseudo-merge bitmaps\n\n Documentation/Makefile | 1 +\n Documentation/config.txt | 2 +\n Documentation/config/bitmap-pseudo-merge.txt | 90 +++\n Documentation/gitpacking.txt | 186 +++++\n Documentation/technical/bitmap-format.txt | 132 ++++\n Makefile | 1 +\n builtin/pack-objects.c | 20 +-\n config.c | 9 +\n config.h | 6 +\n ewah/bitmap.c | 76 ++\n ewah/ewok.h | 8 +\n midx-write.c | 17 +-\n object.h | 1 +\n pack-bitmap-write.c | 488 ++++++++----\n pack-bitmap.c | 359 ++++++++-\n pack-bitmap.h | 55 +-\n parse.c | 29 +\n parse.h | 1 +\n pseudo-merge.c | 752 +++++++++++++++++++\n pseudo-merge.h | 216 ++++++\n t/helper/test-bitmap.c | 34 +-\n t/perf/p5333-pseudo-merge-bitmaps.sh | 32 +\n t/t5333-pseudo-merge-bitmaps.sh | 387 ++++++++++\n t/test-lib-functions.sh | 12 +-\n 24 files changed, 2742 insertions(+), 172 deletions(-)\n create mode 100644 Documentation/config/bitmap-pseudo-merge.txt\n create mode 100644 Documentation/gitpacking.txt\n create mode 100644 pseudo-merge.c\n create mode 100644 pseudo-merge.h\n create mode 100755 t/perf/p5333-pseudo-merge-bitmaps.sh\n create mode 100755 t/t5333-pseudo-merge-bitmaps.sh\n\nRange-diff against v2:\n -: ----------- > 1: 38c96fc1909 object.h: add flags allocated by pack-bitmap.h\n -: ----------- > 2: cbedff02ed1 pack-bitmap-write.c: move commit_positions into commit_pos fields\n -: ----------- > 3: 65ee69acfeb pack-bitmap: avoid use of static `bitmap_writer`\n -: ----------- > 4: b38dd5464d5 pack-bitmap: drop unused `max_bitmaps` parameter\n -: ----------- > 5: f16175295f5 pack-bitmap-write.c: avoid uninitialized 'write_as' field\n -: ----------- > 6: bf65967764f pack-bitmap: introduce `bitmap_writer_free()`\n -: ----------- > 7: 0f20c9becf4 Documentation/gitpacking.txt: initial commit\n -: ----------- > 8: 528b591bd84 Documentation/gitpacking.txt: describe pseudo-merge bitmaps\n 1: 43fd5e35971 ! 9: 12f318b3d7e Documentation/technical: describe pseudo-merge bitmaps format\n @@ Commit message\n format, making it compatible with previous versions of Git, as well as\n the original .bitmap implementation within JGit.\n \n - The format (as well as a general description of pseudo-merge bitmaps,\n - and motivating use-case(s)) is described in detail in the patch contents\n - below, but the high-level description is as follows:\n + The format is described in detail in the patch contents below, but the\n + high-level description is as follows:\n \n - An array of pseudo-merge bitmaps, each containing a pair of EWAH\n bitmaps: one describing the set of pseudo-merge \"parents\", and\n @@ Documentation/technical/bitmap-format.txt: triplet is -\n +bytes (preceding the name-hash cache, commit lookup table, and trailing\n +checksum) of the `.bitmap` file is used to store pseudo-merge bitmaps.\n +\n -+A \"pseudo-merge bitmap\" is used to refer to a pair of bitmaps, as\n -+follows:\n -+\n -+Commit bitmap::\n -+\n -+ A bitmap whose set bits describe the set of commits included in the\n -+ pseudo-merge's \"merge\" bitmap (as below).\n -+\n -+Merge bitmap::\n -+\n -+ A bitmap whose set bits describe the reachability closure over the set\n -+ of commits in the pseudo-merge's \"commits\" bitmap (as above). An\n -+ identical bitmap would be generated for an octopus merge with the same\n -+ set of parents as described in the commits bitmap.\n -+\n -+Pseudo-merge bitmaps can accelerate bitmap traversals when all commits\n -+for a given pseudo-merge are listed on either side of the traversal,\n -+either directly (by explicitly asking for them as part of the `HAVES`\n -+or `WANTS`) or indirectly (by encountering them during a fill-in\n -+traversal).\n -+\n -+=== Use-cases\n -+\n -+For example, suppose there exists a pseudo-merge bitmap with a large\n -+number of commits, all of which are listed in the `WANTS` section of\n -+some bitmap traversal query. When pseudo-merge bitmaps are enabled, the\n -+bitmap machinery can quickly determine there is a pseudo-merge which\n -+satisfies some subset of the wanted objects on either side of the query.\n -+Then, we can inflate the EWAH-compressed bitmap, and `OR` it in to the\n -+resulting bitmap. By contrast, without pseudo-merge bitmaps, we would\n -+have to repeat the decompression and `OR`-ing step over a potentially\n -+large number of individual bitmaps, which can take proportionally more\n -+time.\n -+\n -+Another benefit of pseudo-merges arises when there is some combination\n -+of (a) a large number of references, with (b) poor bitmap coverage, and\n -+(c) deep, nested trees, making fill-in traversal relatively expensive.\n -+For example, suppose that there are a large enough number of tags where\n -+bitmapping each of the tags individually is infeasible. Without\n -+pseudo-merge bitmaps, computing the result of, say, `git rev-list\n -+--use-bitmap-index --count --objects --tags` would likely require a\n -+large amount of fill-in traversal. But when a large quantity of those\n -+tags are stored together in a pseudo-merge bitmap, the bitmap machinery\n -+can take advantage of the fact that we only care about the union of\n -+objects reachable from all of those tags, and answer the query much\n -+faster.\n ++For more information on what pseudo-merges are, why they are useful, and\n ++how to configure them, see the information in linkgit:gitpacking[7].\n +\n +=== File format\n +\n @@ Documentation/technical/bitmap-format.txt: triplet is -\n ++-------------------------------------------+\n +| |\n +| Lookup Table |\n -+| +------------+--------------+ |\n -+| | commit_pos | offset | |\n -+| +------------+--------------+ |\n -+| | 4 bytes | 8 bytes | |\n ++| +---------------------------+ |\n ++| | commit_pos (4 bytes) | |\n ++| +---------------------------+ |\n ++| | offset (8 bytes) | |\n +| +------------+--------------+ |\n +| |\n +| Offset Cases: |\n @@ Documentation/technical/bitmap-format.txt: triplet is -\n ++-------------------------------------------+\n +| |\n +| Extended Lookup Table (Optional) |\n -+| |\n +| +----+----------+----------+----------+ |\n +| | N | Offset 1 | .... | Offset N | |\n +| +----+----------+----------+----------+ |\n @@ Documentation/technical/bitmap-format.txt: triplet is -\n ++-------------------------------------------+\n +| |\n +| Pseudo-merge Metadata |\n -+| +------------------+----------------+ |\n -+| | # pseudo-merges | # Commits | |\n -+| +------------------+----------------+ |\n -+| | 4 bytes | 4 bytes | |\n -+| +------------------+----------------+ |\n -+| |\n -+| +------------------+----------------+ |\n -+| | Lookup offset | Extension size | |\n -+| +------------------+----------------+ |\n -+| | 8 bytes | 8 bytes | |\n -+| +------------------+----------------+ |\n ++| +-----------------------------------+ |\n ++| | # pseudo-merges (4 bytes) | |\n ++| +-----------------------------------+ |\n ++| | # commits (4 bytes) | |\n ++| +-----------------------------------+ |\n ++| | Lookup offset (8 bytes) | |\n ++| +-----------------------------------+ |\n ++| | Extension size (8 bytes) | |\n ++| +-----------------------------------+ |\n +| |\n ++-------------------------------------------+\n +....\n @@ Documentation/technical/bitmap-format.txt: triplet is -\n +\n +* Positions for all pseudo-merges, each stored as an 8-byte unsigned\n + value (in network byte-order) containing the offset (relative to the\n -+ beginnign of the `.bitmap` file) of each consecutive pseudo-merge.\n ++ beginning of the `.bitmap` file) of each consecutive pseudo-merge.\n +\n +* A 4-byte unsigned value (in network byte-order) equal to the number of\n + pseudo-merges.\n 2: 290d928325d = 10: 40eb6137618 ewah: implement `ewah_bitmap_is_subset()`\n 3: 5160859f7f3 < -: ----------- pack-bitmap: drop unused `max_bitmaps` parameter\n 4: 3d7d930b1c5 < -: ----------- pack-bitmap: move some initialization to `bitmap_writer_init()`\n -: ----------- > 11: 487fb7c6e9c pack-bitmap: move some initialization to `bitmap_writer_init()`\n 5: e7a87cf7d4e = 12: 827732acf99 pseudo-merge.ch: initial commit\n 6: ee33a703245 ! 13: 8608dd1860f pack-bitmap-write: support storing pseudo-merge commits\n @@ Commit message\n \n Signed-off-by: Taylor Blau \n \n + ## object.h ##\n +@@ object.h: void object_array_init(struct object_array *array);\n + * reflog.c: 10--12\n + * builtin/show-branch.c: 0-------------------------------------------26\n + * builtin/unpack-objects.c: 2021\n +- * pack-bitmap.h: 22\n ++ * pack-bitmap.h: 2122\n + */\n + #define FLAG_BITS 28\n + \n +\n ## pack-bitmap-write.c ##\n @@ pack-bitmap-write.c: struct bitmapped_commit {\n - \tstruct ewah_bitmap *write_as;\n \tint flags;\n \tint xor_offset;\n --\tuint32_t commit_pos;\n + \tuint32_t commit_pos;\n +\tunsigned pseudo_merge : 1;\n };\n \n - struct bitmap_writer {\n -@@ pack-bitmap-write.c: struct bitmap_writer {\n - \tstruct bitmapped_commit *selected;\n - \tunsigned int selected_nr, selected_alloc;\n - \n -+\tuint32_t pseudo_merges_nr;\n -+\n - \tstruct progress *progress;\n - \tint show_progress;\n - \tunsigned char pack_checksum[GIT_MAX_RAWSZ];\n -@@ pack-bitmap-write.c: struct bitmap_writer {\n - \n - static struct bitmap_writer writer;\n - \n -+static inline int bitmap_writer_selected_nr(void)\n ++static inline int bitmap_writer_nr_selected_commits(struct bitmap_writer *writer)\n +{\n -+\treturn writer.selected_nr - writer.pseudo_merges_nr;\n ++\treturn writer->selected_nr - writer->pseudo_merges_nr;\n +}\n +\n - void bitmap_writer_init(struct repository *r)\n + void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r)\n {\n - \twriter.bitmaps = kh_init_oid_map();\n -@@ pack-bitmap-write.c: void bitmap_writer_build_type_index(struct packing_data *to_pack,\n - * Compute the actual bitmaps\n + \tmemset(writer, 0, sizeof(struct bitmap_writer));\n +@@ pack-bitmap-write.c: void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n */\n \n --static inline void push_bitmapped_commit(struct commit *commit)\n -+static void bitmap_writer_push_bitmapped_commit(struct commit *commit,\n -+\t\t\t\t\t\tunsigned pseudo_merge)\n + static inline void push_bitmapped_commit(struct bitmap_writer *writer,\n +-\t\t\t\t\t struct commit *commit)\n ++\t\t\t\t\t struct commit *commit,\n ++\t\t\t\t\t unsigned pseudo_merge)\n {\n -\tint hash_ret;\n -\tkhiter_t hash_pos;\n -\n - \tif (writer.selected_nr >= writer.selected_alloc) {\n - \t\twriter.selected_alloc = (writer.selected_alloc + 32) * 2;\n - \t\tREALLOC_ARRAY(writer.selected, writer.selected_alloc);\n + \tif (writer->selected_nr >= writer->selected_alloc) {\n + \t\twriter->selected_alloc = (writer->selected_alloc + 32) * 2;\n + \t\tREALLOC_ARRAY(writer->selected, writer->selected_alloc);\n \t}\n \n --\thash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret);\n +-\thash_pos = kh_put_oid_map(writer->bitmaps, commit->object.oid,\n +-\t\t\t\t &hash_ret);\n -\tif (!hash_ret)\n -\t\tdie(_(\"duplicate entry when writing bitmap index: %s\"),\n -\t\t oid_to_hex(&commit->object.oid));\n --\tkh_value(writer.bitmaps, hash_pos) = NULL;\n +-\tkh_value(writer->bitmaps, hash_pos) = NULL;\n +\tif (!pseudo_merge) {\n +\t\tint hash_ret;\n -+\t\tkhiter_t hash_pos = kh_put_oid_map(writer.bitmaps,\n ++\t\tkhiter_t hash_pos = kh_put_oid_map(writer->bitmaps,\n +\t\t\t\t\t\t commit->object.oid,\n +\t\t\t\t\t\t &hash_ret);\n +\n +\t\tif (!hash_ret)\n +\t\t\tdie(_(\"duplicate entry when writing bitmap index: %s\"),\n +\t\t\t oid_to_hex(&commit->object.oid));\n -+\t\tkh_value(writer.bitmaps, hash_pos) = NULL;\n ++\t\tkh_value(writer->bitmaps, hash_pos) = NULL;\n +\t}\n \n - \twriter.selected[writer.selected_nr].commit = commit;\n - \twriter.selected[writer.selected_nr].bitmap = NULL;\n - \twriter.selected[writer.selected_nr].flags = 0;\n -+\twriter.selected[writer.selected_nr].pseudo_merge = pseudo_merge;\n + \twriter->selected[writer->selected_nr].commit = commit;\n + \twriter->selected[writer->selected_nr].bitmap = NULL;\n + \twriter->selected[writer->selected_nr].write_as = NULL;\n + \twriter->selected[writer->selected_nr].flags = 0;\n ++\twriter->selected[writer->selected_nr].pseudo_merge = pseudo_merge;\n \n - \twriter.selected_nr++;\n + \twriter->selected_nr++;\n }\n -@@ pack-bitmap-write.c: static void compute_xor_offsets(void)\n +@@ pack-bitmap-write.c: static void compute_xor_offsets(struct bitmap_writer *writer)\n \n - \twhile (next < writer.selected_nr) {\n - \t\tstruct bitmapped_commit *stored = &writer.selected[next];\n + \twhile (next < writer->selected_nr) {\n + \t\tstruct bitmapped_commit *stored = &writer->selected[next];\n -\n \t\tint best_offset = 0;\n \t\tstruct ewah_bitmap *best_bitmap = stored->bitmap;\n @@ pack-bitmap-write.c: static void compute_xor_offsets(void)\n \n \t\t\tif (curr < 0)\n \t\t\t\tbreak;\n -+\t\t\tif (writer.selected[curr].pseudo_merge)\n ++\t\t\tif (writer->selected[curr].pseudo_merge)\n +\t\t\t\tcontinue;\n \n \t\t\ttest_xor = ewah_pool_new();\n - \t\t\tewah_xor(writer.selected[curr].bitmap, stored->bitmap, test_xor);\n -@@ pack-bitmap-write.c: static void compute_xor_offsets(void)\n + \t\t\tewah_xor(writer->selected[curr].bitmap, stored->bitmap, test_xor);\n +@@ pack-bitmap-write.c: static void compute_xor_offsets(struct bitmap_writer *writer)\n \t\t\t}\n \t\t}\n \n @@ pack-bitmap-write.c: static void bitmap_builder_init(struct bitmap_builder *bb,\n \t}\n \n \tif (prepare_revision_walk(&revs))\n -@@ pack-bitmap-write.c: static int fill_bitmap_commit(struct bb_commit *ent,\n +@@ pack-bitmap-write.c: static int fill_bitmap_commit(struct bitmap_writer *writer,\n \t\tstruct commit *c = prio_queue_get(queue);\n \n \t\tif (old_bitmap && mapping) {\n @@ pack-bitmap-write.c: static int fill_bitmap_commit(struct bb_commit *ent,\n \t\t\t/*\n \t\t\t * If this commit has an old bitmap, then translate that\n \t\t\t * bitmap and add its bits to this one. No need to walk\n -@@ pack-bitmap-write.c: static int fill_bitmap_commit(struct bb_commit *ent,\n +@@ pack-bitmap-write.c: static int fill_bitmap_commit(struct bitmap_writer *writer,\n \t\t * Mark ourselves and queue our tree. The commit\n \t\t * walk ensures we cover all parents.\n \t\t */\n --\t\tpos = find_object_pos(&c->object.oid, &found);\n +-\t\tpos = find_object_pos(writer, &c->object.oid, &found);\n -\t\tif (!found)\n -\t\t\treturn -1;\n -\t\tbitmap_set(ent->bitmap, pos);\n -\t\tprio_queue_put(tree_queue,\n -\t\t\t repo_get_commit_tree(the_repository, c));\n +\t\tif (!(c->object.flags & BITMAP_PSEUDO_MERGE)) {\n -+\t\t\tpos = find_object_pos(&c->object.oid, &found);\n ++\t\t\tpos = find_object_pos(writer, &c->object.oid, &found);\n +\t\t\tif (!found)\n +\t\t\t\treturn -1;\n +\t\t\tbitmap_set(ent->bitmap, pos);\n @@ pack-bitmap-write.c: static int fill_bitmap_commit(struct bb_commit *ent,\n +\t\t}\n \n \t\tfor (p = c->parents; p; p = p->next) {\n - \t\t\tpos = find_object_pos(&p->item->object.oid, &found);\n -@@ pack-bitmap-write.c: static void store_selected(struct bb_commit *ent, struct commit *commit)\n + \t\t\tpos = find_object_pos(writer, &p->item->object.oid,\n +@@ pack-bitmap-write.c: static void store_selected(struct bitmap_writer *writer,\n \n \tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n \n +\tif (ent->pseudo_merge)\n +\t\treturn;\n +\n - \thash_pos = kh_get_oid_map(writer.bitmaps, commit->object.oid);\n - \tif (hash_pos == kh_end(writer.bitmaps))\n + \thash_pos = kh_get_oid_map(writer->bitmaps, commit->object.oid);\n + \tif (hash_pos == kh_end(writer->bitmaps))\n \t\tdie(_(\"attempted to store non-selected commit: '%s'\"),\n -@@ pack-bitmap-write.c: void bitmap_writer_select_commits(struct commit **indexed_commits,\n +@@ pack-bitmap-write.c: void bitmap_writer_select_commits(struct bitmap_writer *writer,\n \n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n --\t\t\tpush_bitmapped_commit(indexed_commits[i]);\n -+\t\t\tbitmap_writer_push_bitmapped_commit(indexed_commits[i], 0);\n +-\t\t\tpush_bitmapped_commit(writer, indexed_commits[i]);\n ++\t\t\tpush_bitmapped_commit(writer, indexed_commits[i], 0);\n \t\treturn;\n \t}\n \n -@@ pack-bitmap-write.c: void bitmap_writer_select_commits(struct commit **indexed_commits,\n +@@ pack-bitmap-write.c: void bitmap_writer_select_commits(struct bitmap_writer *writer,\n \t\t\t}\n \t\t}\n \n --\t\tpush_bitmapped_commit(chosen);\n -+\t\tbitmap_writer_push_bitmapped_commit(chosen, 0);\n +-\t\tpush_bitmapped_commit(writer, chosen);\n ++\t\tpush_bitmapped_commit(writer, chosen, 0);\n \n \t\ti += next + 1;\n - \t\tdisplay_progress(writer.progress, i);\n -@@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n + \t\tdisplay_progress(writer->progress, i);\n +@@ pack-bitmap-write.c: static void write_selected_commits_v1(struct bitmap_writer *writer,\n {\n \tint i;\n \n --\tfor (i = 0; i < writer.selected_nr; ++i) {\n -+\tfor (i = 0; i < bitmap_writer_selected_nr(); ++i) {\n - \t\tstruct bitmapped_commit *stored = &writer.selected[i];\n +-\tfor (i = 0; i < writer->selected_nr; ++i) {\n ++\tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); ++i) {\n + \t\tstruct bitmapped_commit *stored = &writer->selected[i];\n +\t\tif (stored->pseudo_merge)\n +\t\t\tBUG(\"unexpected pseudo-merge among selected: %s\",\n +\t\t\t oid_to_hex(&stored->commit->object.oid));\n \n \t\tif (offsets)\n \t\t\toffsets[i] = hashfile_total(f);\n -@@ pack-bitmap-write.c: static void write_lookup_table(struct hashfile *f,\n +@@ pack-bitmap-write.c: static void write_lookup_table(struct bitmap_writer *writer, struct hashfile *f,\n \tuint32_t i;\n \tuint32_t *table, *table_inv;\n \n --\tALLOC_ARRAY(table, writer.selected_nr);\n --\tALLOC_ARRAY(table_inv, writer.selected_nr);\n -+\tALLOC_ARRAY(table, bitmap_writer_selected_nr());\n -+\tALLOC_ARRAY(table_inv, bitmap_writer_selected_nr());\n +-\tALLOC_ARRAY(table, writer->selected_nr);\n +-\tALLOC_ARRAY(table_inv, writer->selected_nr);\n ++\tALLOC_ARRAY(table, bitmap_writer_nr_selected_commits(writer));\n ++\tALLOC_ARRAY(table_inv, bitmap_writer_nr_selected_commits(writer));\n \n --\tfor (i = 0; i < writer.selected_nr; i++)\n -+\tfor (i = 0; i < bitmap_writer_selected_nr(); i++)\n +-\tfor (i = 0; i < writer->selected_nr; i++)\n ++\tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++)\n \t\ttable[i] = i;\n \n \t/*\n -@@ pack-bitmap-write.c: static void write_lookup_table(struct hashfile *f,\n +@@ pack-bitmap-write.c: static void write_lookup_table(struct bitmap_writer *writer, struct hashfile *f,\n \t * bitmap corresponds to j'th bitmapped commit (among the selected\n \t * commits) in lex order of OIDs.\n \t */\n --\tQSORT_S(table, writer.selected_nr, table_cmp, commit_positions);\n -+\tQSORT_S(table, bitmap_writer_selected_nr(), table_cmp, commit_positions);\n +-\tQSORT_S(table, writer->selected_nr, table_cmp, writer);\n ++\tQSORT_S(table, bitmap_writer_nr_selected_commits(writer), table_cmp, writer);\n \n \t/* table_inv helps us discover that relationship (i'th bitmap\n \t * to j'th commit by j = table_inv[i])\n \t */\n --\tfor (i = 0; i < writer.selected_nr; i++)\n -+\tfor (i = 0; i < bitmap_writer_selected_nr(); i++)\n +-\tfor (i = 0; i < writer->selected_nr; i++)\n ++\tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++)\n \t\ttable_inv[table[i]] = i;\n \n \ttrace2_region_enter(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n --\tfor (i = 0; i < writer.selected_nr; i++) {\n -+\tfor (i = 0; i < bitmap_writer_selected_nr(); i++) {\n - \t\tstruct bitmapped_commit *selected = &writer.selected[table[i]];\n +-\tfor (i = 0; i < writer->selected_nr; i++) {\n ++\tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++) {\n + \t\tstruct bitmapped_commit *selected = &writer->selected[table[i]];\n \t\tuint32_t xor_offset = selected->xor_offset;\n \t\tuint32_t xor_row;\n -@@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n +@@ pack-bitmap-write.c: void bitmap_writer_finish(struct bitmap_writer *writer,\n \tmemcpy(header.magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE));\n \theader.version = htons(default_version);\n \theader.options = htons(flags | options);\n --\theader.entry_count = htonl(writer.selected_nr);\n -+\theader.entry_count = htonl(bitmap_writer_selected_nr());\n - \thashcpy(header.checksum, writer.pack_checksum);\n +-\theader.entry_count = htonl(writer->selected_nr);\n ++\theader.entry_count = htonl(bitmap_writer_nr_selected_commits(writer));\n + \thashcpy(header.checksum, writer->pack_checksum);\n \n \thashwrite(f, &header, sizeof(header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz);\n -@@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n +@@ pack-bitmap-write.c: void bitmap_writer_finish(struct bitmap_writer *writer,\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n \t\tCALLOC_ARRAY(offsets, index_nr);\n \n --\tALLOC_ARRAY(commit_positions, writer.selected_nr);\n -+\tALLOC_ARRAY(commit_positions, bitmap_writer_selected_nr());\n - \n --\tfor (i = 0; i < writer.selected_nr; i++) {\n -+\tfor (i = 0; i < bitmap_writer_selected_nr(); i++) {\n - \t\tstruct bitmapped_commit *stored = &writer.selected[i];\n - \t\tint commit_pos = oid_pos(&stored->commit->object.oid, index, index_nr, oid_access);\n - \n +-\tfor (i = 0; i < writer->selected_nr; i++) {\n ++\tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++) {\n + \t\tstruct bitmapped_commit *stored = &writer->selected[i];\n + \t\tint commit_pos = oid_pos(&stored->commit->object.oid, index,\n + \t\t\t\t\t index_nr, oid_access);\n \n ## pack-bitmap.h ##\n @@ pack-bitmap.h: struct bitmap_disk_header {\n @@ pack-bitmap.h: struct bitmap_disk_header {\n #define NEEDS_BITMAP (1u<<22)\n \n /*\n +@@ pack-bitmap.h: struct bitmap_writer {\n + \tstruct bitmapped_commit *selected;\n + \tunsigned int selected_nr, selected_alloc;\n + \n ++\tuint32_t pseudo_merges_nr;\n ++\n + \tstruct progress *progress;\n + \tint show_progress;\n + \tunsigned char pack_checksum[GIT_MAX_RAWSZ];\n 7: 9c6d09bf874 < -: ----------- pack-bitmap: implement `bitmap_writer_has_bitmapped_object_id()`\n 8: dfd4b73d12e < -: ----------- pack-bitmap: make `bitmap_writer_push_bitmapped_commit()` public\n -: ----------- > 14: 99d2b6872ba pack-bitmap: implement `bitmap_writer_has_bitmapped_object_id()`\n -: ----------- > 15: e7209c60fa5 pack-bitmap: make `bitmap_writer_push_bitmapped_commit()` public\n -: ----------- > 16: 3070135eb4b config: introduce git_config_float()\n 9: 86a1e4b8b9b ! 17: 3029473c094 pseudo-merge: implement support for selecting pseudo-merge commits\n @@ Commit message\n for inclusion in different pseudo-merge group(s) based on a handful of\n criteria.\n \n - Pseudo-merges are derived first from named pseudo-merge groups (see the\n - `bitmapPseudoMerge..*` configuration options). They are\n - (optionally) further segmented within an individual pseudo-merge group\n - based on any capture group(s) within the pseudo-merge group's pattern.\n -\n - For example, a configuration like so:\n -\n - [bitmapPseudoMerge \"all\"]\n - pattern = \"refs/\"\n - threshold = now\n - stableThreshold = never\n - sampleRate = 100\n - maxMerges = 64\n -\n - would group all non-bitmapped commits into up to 64 individual\n - pseudo-merge commits.\n -\n - If you wanted to separate tags from branches when generating\n - pseudo-merge commits, and further segment them by which fork they\n - originate from (using the same \"refs/virtual/\" scheme as in the delta\n - islands documentation), you would instead write something like:\n -\n - [bitmapPseudoMerge \"all\"]\n - pattern = \"refs/virtual/([0-9]+)/(heads|tags)/\"\n - threshold = now\n - stableThreshold = never\n - sampleRate = 100\n - maxMerges = 64\n -\n - Which would generate pseudo-merge group identifiers like \"1234-heads\",\n - and \"5678-tags\" (for branches in fork \"1234\", and tags in remote \"5678\",\n - respectively).\n -\n - Within pseudo-merge groups, there are a handful of other options used to\n - control the distribution of matching commits among individual\n - pseudo-merge commits:\n -\n - - bitmapPseudoMerge..decay\n - - bitmapPseudoMerge..sampleRate\n - - bitmapPseudoMerge..threshold\n - - bitmapPseudoMerge..maxMerges\n - - bitmapPseudoMerge..stableThreshold\n - - bitmapPseudoMerge..stableSize\n -\n - The decay parameter roughly corresponds to \"k\" in `f(n) = C*n^(-k/100)`,\n - where `f(n)` describes the size of the `n`-th pseudo-merge group. The\n - sample rate controls what percentage of eligible commits are considered\n - as candidates. The threshold parameter indicates the minimum age (so as\n - to avoid including too-recent commits in a pseudo-merge group, making it\n - less likely to be valid). The \"maxMerges\" parameter sets an upper-bound\n - on the number of pseudo-merge commits an individual group\n -\n - The latter two \"stable\"-related parameters control \"stable\" pseudo-merge\n - groups, comprised of a fixed number of commits which are older than the\n - configured \"stable threshold\" value and may be grouped together in\n - chunks of \"stableSize\" in order of age.\n -\n - This patch implements the aforementioned selection routine, as well as\n - parsing the relevant configuration options.\n + Note that the selected pseudo-merge commits aren't actually used or\n + written anywhere yet. This will be done in the following commit.\n \n Signed-off-by: Taylor Blau \n \n + ## Documentation/config.txt ##\n +@@ Documentation/config.txt: include::config/apply.txt[]\n + \n + include::config/attr.txt[]\n + \n ++include::config/bitmap-pseudo-merge.txt[]\n ++\n + include::config/blame.txt[]\n + \n + include::config/branch.txt[]\n +\n + ## Documentation/config/bitmap-pseudo-merge.txt (new) ##\n +@@\n ++NOTE: The configuration options in `bitmapPseudoMerge.*` are considered\n ++EXPERIMENTAL and may be subject to change or be removed entirely in the\n ++future.\n ++\n ++bitmapPseudoMerge..pattern::\n ++\tRegular expression used to match reference names. Commits\n ++\tpointed to by references matching this pattern (and meeting\n ++\tthe below criteria, like `bitmapPseudoMerge..sampleRate`\n ++\tand `bitmapPseudoMerge..threshold`) will be considered\n ++\tfor inclusion in a pseudo-merge bitmap.\n +++\n ++Commits are grouped into pseudo-merge groups based on whether or not\n ++any reference(s) that point at a given commit match the pattern, which\n ++is an extended regular expression.\n +++\n ++Within a pseudo-merge group, commits may be further grouped into\n ++sub-groups based on the capture groups in the pattern. These\n ++sub-groupings are formed from the regular expressions by concatenating\n ++any capture groups from the regular expression, with a '-' dash in\n ++between.\n +++\n ++For example, if the pattern is `refs/tags/`, then all tags (provided\n ++they meet the below criteria) will be considered candidates for the\n ++same pseudo-merge group. However, if the pattern is instead\n ++`refs/remotes/([0-9])+/tags/`, then tags from different remotes will\n ++be grouped into separate pseudo-merge groups, based on the remote\n ++number.\n ++\n ++bitmapPseudoMerge..decay::\n ++\tDetermines the rate at which consecutive pseudo-merge bitmap\n ++\tgroups decrease in size. Must be non-negative. This parameter\n ++\tcan be thought of as `k` in the function `f(n) = C * n^-k`,\n ++\twhere `f(n)` is the size of the `n`th group.\n +++\n ++Setting the decay rate equal to `0` will cause all groups to be the\n ++same size. Setting the decay rate equal to `1` will cause the `n`th\n ++group to be `1/n` the size of the initial group. Higher values of the\n ++decay rate cause consecutive groups to shrink at an increasing rate.\n ++The default is `1`.\n +++\n ++If all groups are the same size, it is possible that groups containing\n ++newer commits will be able to be used less often than earlier groups,\n ++since it is more likely that the references pointing at newer commits\n ++will be updated more often than a reference pointing at an old commit.\n ++\n ++bitmapPseudoMerge..sampleRate::\n ++\tDetermines the proportion of non-bitmapped commits (among\n ++\treference tips) which are selected for inclusion in an\n ++\tunstable pseudo-merge bitmap. Must be between `0` and `1`\n ++\t(inclusive). The default is `1`.\n ++\n ++bitmapPseudoMerge..threshold::\n ++\tDetermines the minimum age of non-bitmapped commits (among\n ++\treference tips, as above) which are candidates for inclusion\n ++\tin an unstable pseudo-merge bitmap. The default is\n ++\t`1.week.ago`.\n ++\n ++bitmapPseudoMerge..maxMerges::\n ++\tDetermines the maximum number of pseudo-merge commits among\n ++\twhich commits may be distributed.\n +++\n ++For pseudo-merge groups whose pattern does not contain any capture\n ++groups, this setting is applied for all commits matching the regular\n ++expression. For patterns that have one or more capture groups, this\n ++setting is applied for each distinct capture group.\n +++\n ++For example, if your capture group is `refs/tags/`, then this setting\n ++will distribute all tags into a maximum of `maxMerges` pseudo-merge\n ++commits. However, if your capture group is, say,\n ++`refs/remotes/([0-9]+)/tags/`, then this setting will be applied to\n ++each remote's set of tags individually.\n +++\n ++Must be non-negative. The default value is 64.\n ++\n ++bitmapPseudoMerge..stableThreshold::\n ++\tDetermines the minimum age of commits (among reference tips,\n ++\tas above, however stable commits are still considered\n ++\tcandidates even when they have been covered by a bitmap) which\n ++\tare candidates for a stable a pseudo-merge bitmap. The default\n ++\tis `1.month.ago`.\n +++\n ++Setting this threshold to a smaller value (e.g., 1.week.ago) will cause\n ++more stable groups to be generated (which impose a one-time generation\n ++cost) but those groups will likely become stale over time. Using a\n ++larger value incurs the opposite penalty (fewer stable groups which are\n ++more useful).\n ++\n ++bitmapPseudoMerge..stableSize::\n ++\tDetermines the size (in number of commits) of a stable\n ++\tpsuedo-merge bitmap. The default is `512`.\n +\n + ## Documentation/gitpacking.txt ##\n +@@ Documentation/gitpacking.txt: can take advantage of the fact that we only care about the union of\n + objects reachable from all of those tags, and answer the query much\n + faster.\n + \n ++=== Configuration\n ++\n ++Reference tips are grouped into different pseudo-merge groups according\n ++to two criteria. A reference name matches one or more of the defined\n ++pseudo-merge patterns, and optionally one or more capture groups within\n ++that pattern which further partition the group.\n ++\n ++Within a group, commits may be considered \"stable\", or \"unstable\"\n ++depending on their age. These are adjusted by setting the\n ++`bitmapPseudoMerge..stableThreshold` and\n ++`bitmapPseudoMerge..threshold` configuration values, respectively.\n ++\n ++All stable commits are grouped into pseudo-merges of equal size\n ++(`bitmapPseudoMerge..stableSize`). If the `stableSize`\n ++configuration is set to, say, 100, then the first 100 commits (ordered\n ++by committer date) which are older than the `stableThreshold` value will\n ++form one group, the next 100 commits will form another group, and so on.\n ++\n ++Among unstable commits, the pseudo-merge machinery will attempt to\n ++combine older commits into large groups as opposed to newer commits\n ++which will appear in smaller groups. This is based on the heuristic that\n ++references whose tip commit is older are less likely to be modified to\n ++point at a different commit than a reference whose tip commit is newer.\n ++\n ++The size of groups is determined by a power-law decay function, and the\n ++decay parameter roughly corresponds to \"k\" in `f(n) = C*n^(-k/100)`,\n ++where `f(n)` describes the size of the `n`-th pseudo-merge group. The\n ++sample rate controls what percentage of eligible commits are considered\n ++as candidates. The threshold parameter indicates the minimum age (so as\n ++to avoid including too-recent commits in a pseudo-merge group, making it\n ++less likely to be valid). The \"maxMerges\" parameter sets an upper-bound\n ++on the number of pseudo-merge commits an individual group\n ++\n ++The \"stable\"-related parameters control \"stable\" pseudo-merge groups,\n ++comprised of a fixed number of commits which are older than the\n ++configured \"stable threshold\" value and may be grouped together in\n ++chunks of \"stableSize\" in order of age.\n ++\n ++The exact configuration for pseudo-merges is as follows:\n ++\n ++include::config/bitmap-pseudo-merge.txt[]\n ++\n ++=== Examples\n ++\n ++Suppose that you have a repository with a large number of references,\n ++and you want a bare-bones configuration of pseudo-merge bitmaps that\n ++will enhance bitmap coverage of the `refs/` namespace. You may start\n ++wiht a configuration like so:\n ++\n ++ [bitmapPseudoMerge \"all\"]\n ++\tpattern = \"refs/\"\n ++\tthreshold = now\n ++\tstableThreshold = never\n ++\tsampleRate = 100\n ++\tmaxMerges = 64\n ++\n ++This will create pseudo-merge bitmaps for all references, regardless of\n ++their age, and group them into 64 pseudo-merge commits.\n ++\n ++If you wanted to separate tags from branches when generating\n ++pseudo-merge commits, you would instead define the pattern with a\n ++capture group, like so:\n ++\n ++ [bitmapPseudoMerge \"all\"]\n ++\tpattern = \"refs/(heads/tags)/\"\n ++\n ++Suppose instead that you are working in a fork-network repository, with\n ++each fork specified by some numeric ID, and whose refs reside in\n ++`refs/virtual/NNN/` (where `NNN` is the numeric ID corresponding to some\n ++fork) in the network. In this instance, you may instead write something\n ++like:\n ++\n ++ [bitmapPseudoMerge \"all\"]\n ++\tpattern = \"refs/virtual/([0-9]+)/(heads|tags)/\"\n ++\tthreshold = now\n ++\tstableThreshold = never\n ++\tsampleRate = 100\n ++\tmaxMerges = 64\n ++\n ++Which would generate pseudo-merge group identifiers like \"1234-heads\",\n ++and \"5678-tags\" (for branches in fork \"1234\", and tags in remote \"5678\",\n ++respectively).\n ++\n + SEE ALSO\n + --------\n + linkgit:git-pack-objects[1]\n +\n + ## pack-bitmap-write.c ##\n +@@\n + #include \"trace2.h\"\n + #include \"tree.h\"\n + #include \"tree-walk.h\"\n ++#include \"pseudo-merge.h\"\n + \n + struct bitmapped_commit {\n + \tstruct commit *commit;\n +@@ pack-bitmap-write.c: void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r)\n + \tif (writer->bitmaps)\n + \t\tBUG(\"bitmap writer already initialized\");\n + \twriter->bitmaps = kh_init_oid_map();\n ++\twriter->pseudo_merge_commits = kh_init_oid_map();\n ++\n ++\tstring_list_init_dup(&writer->pseudo_merge_groups);\n ++\n ++\tload_pseudo_merges_from_config(&writer->pseudo_merge_groups);\n ++}\n ++\n ++static void free_pseudo_merge_commit_idx(struct pseudo_merge_commit_idx *idx)\n ++{\n ++\tif (!idx)\n ++\t\treturn;\n ++\tfree(idx->pseudo_merge);\n ++\tfree(idx);\n + }\n + \n + void bitmap_writer_free(struct bitmap_writer *writer)\n + {\n + \tuint32_t i;\n ++\tstruct pseudo_merge_commit_idx *idx;\n + \n + \tif (!writer)\n + \t\treturn;\n +@@ pack-bitmap-write.c: void bitmap_writer_free(struct bitmap_writer *writer)\n + \n + \tkh_destroy_oid_map(writer->bitmaps);\n + \n ++\tkh_foreach_value(writer->pseudo_merge_commits, idx,\n ++\t\t\t free_pseudo_merge_commit_idx(idx));\n ++\tkh_destroy_oid_map(writer->pseudo_merge_commits);\n ++\n + \tfor (i = 0; i < writer->selected_nr; i++) {\n + \t\tstruct bitmapped_commit *bc = &writer->selected[i];\n + \t\tif (bc->write_as != bc->bitmap)\n +@@ pack-bitmap-write.c: void bitmap_writer_select_commits(struct bitmap_writer *writer,\n + \t}\n + \n + \tstop_progress(&writer->progress);\n ++\n ++\tselect_pseudo_merges(writer, indexed_commits, indexed_commits_nr);\n + }\n + \n + \n +\n + ## pack-bitmap.h ##\n +@@ pack-bitmap.h: struct bitmap_writer {\n + \tstruct bitmapped_commit *selected;\n + \tunsigned int selected_nr, selected_alloc;\n + \n ++\tstruct string_list pseudo_merge_groups;\n ++\tkh_oid_map_t *pseudo_merge_commits; /* oid -> pseudo merge(s) */\n + \tuint32_t pseudo_merges_nr;\n + \n + \tstruct progress *progress;\n +\n ## pseudo-merge.c ##\n @@\n #include \"git-compat-util.h\"\n @@ pseudo-merge.c\n +\n +#define DEFAULT_PSEUDO_MERGE_DECAY 1.0f\n +#define DEFAULT_PSEUDO_MERGE_MAX_MERGES 64\n -+#define DEFAULT_PSEUDO_MERGE_SAMPLE_RATE 100\n ++#define DEFAULT_PSEUDO_MERGE_SAMPLE_RATE 1\n +#define DEFAULT_PSEUDO_MERGE_THRESHOLD approxidate(\"1.week.ago\")\n +#define DEFAULT_PSEUDO_MERGE_STABLE_THRESHOLD approxidate(\"1.month.ago\")\n +#define DEFAULT_PSEUDO_MERGE_STABLE_SIZE 512\n @@ pseudo-merge.c\n +\t\tC += 1.0f / gitexp(n + 1, group->decay);\n +\tC = matches->unstable_nr / C;\n +\n -+\treturn (int)((C / gitexp(i + 1, group->decay)) + 0.5);\n ++\treturn (uint32_t)((C / gitexp(i + 1, group->decay)) + 0.5);\n +}\n +\n -+static void init_pseudo_merge_group(struct pseudo_merge_group *group)\n ++static void pseudo_merge_group_init(struct pseudo_merge_group *group)\n +{\n +\tmemset(group, 0, sizeof(struct pseudo_merge_group));\n +\n @@ pseudo-merge.c\n +\tstruct strbuf buf = STRBUF_INIT;\n +\tconst char *sub, *key;\n +\tsize_t sub_len;\n ++\tint ret = 0;\n +\n +\tif (parse_config_key(var, \"bitmappseudomerge\", &sub, &sub_len, &key))\n -+\t\treturn 0;\n ++\t\tgoto done;\n +\n +\tif (!sub_len)\n -+\t\treturn 0;\n ++\t\tgoto done;\n +\n +\tstrbuf_add(&buf, sub, sub_len);\n +\n @@ pseudo-merge.c\n +\t\titem = string_list_insert(list, buf.buf);\n +\n +\t\titem->util = xmalloc(sizeof(struct pseudo_merge_group));\n -+\t\tinit_pseudo_merge_group(item->util);\n ++\t\tpseudo_merge_group_init(item->util);\n +\t}\n +\n +\tgroup = item->util;\n @@ pseudo-merge.c\n +\n +\t\tstrbuf_release(&re);\n +\t} else if (!strcmp(key, \"decay\")) {\n -+\t\tgroup->decay = git_config_int(var, value, ctx->kvi);\n ++\t\tgroup->decay = git_config_float(var, value, ctx->kvi);\n +\t\tif (group->decay < 0) {\n +\t\t\twarning(_(\"%s must be non-negative, using default\"), var);\n +\t\t\tgroup->decay = DEFAULT_PSEUDO_MERGE_DECAY;\n +\t\t}\n +\t} else if (!strcmp(key, \"samplerate\")) {\n -+\t\tgroup->sample_rate = git_config_int(var, value, ctx->kvi);\n -+\t\tif (!(0 <= group->sample_rate && group->sample_rate <= 100)) {\n -+\t\t\twarning(_(\"%s must be between 0 and 100, using default\"), var);\n ++\t\tgroup->sample_rate = git_config_float(var, value, ctx->kvi);\n ++\t\tif (!(0 <= group->sample_rate && group->sample_rate <= 1)) {\n ++\t\t\twarning(_(\"%s must be between 0 and 1, using default\"), var);\n +\t\t\tgroup->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE;\n +\t\t}\n +\t} else if (!strcmp(key, \"threshold\")) {\n +\t\tif (git_config_expiry_date(&group->threshold, var, value)) {\n -+\t\t\tstrbuf_release(&buf);\n -+\t\t\treturn -1;\n ++\t\t\tret = -1;\n ++\t\t\tgoto done;\n +\t\t}\n +\t} else if (!strcmp(key, \"maxmerges\")) {\n +\t\tgroup->max_merges = git_config_int(var, value, ctx->kvi);\n @@ pseudo-merge.c\n +\t\t}\n +\t} else if (!strcmp(key, \"stablethreshold\")) {\n +\t\tif (git_config_expiry_date(&group->stable_threshold, var, value)) {\n -+\t\t\tstrbuf_release(&buf);\n -+\t\t\treturn -1;\n ++\t\t\tret = -1;\n ++\t\t\tgoto done;\n +\t\t}\n +\t} else if (!strcmp(key, \"stablesize\")) {\n +\t\tgroup->stable_size = git_config_int(var, value, ctx->kvi);\n @@ pseudo-merge.c\n +\t\t}\n +\t}\n +\n ++done:\n +\tstrbuf_release(&buf);\n +\n -+\treturn 0;\n ++\treturn ret;\n +}\n +\n +void load_pseudo_merges_from_config(struct string_list *list)\n @@ pseudo-merge.c\n +\t\t\t\t\t int flags UNUSED,\n +\t\t\t\t\t void *_data)\n +{\n -+\tstruct string_list *list = _data;\n ++\tstruct bitmap_writer *writer = _data;\n +\tstruct object_id peeled;\n +\tstruct commit *c;\n +\tuint32_t i;\n @@ pseudo-merge.c\n +\tif (!c)\n +\t\treturn 0;\n +\n -+\thas_bitmap = bitmap_writer_has_bitmapped_object_id(oid);\n ++\thas_bitmap = bitmap_writer_has_bitmapped_object_id(writer, oid);\n +\n -+\tfor (i = 0; i < list->nr; i++) {\n ++\tfor (i = 0; i < writer->pseudo_merge_groups.nr; i++) {\n +\t\tstruct pseudo_merge_group *group;\n +\t\tstruct pseudo_merge_matches *matches;\n +\t\tstruct strbuf group_name = STRBUF_INIT;\n +\t\tregmatch_t captures[16];\n +\t\tsize_t j;\n +\n -+\t\tgroup = list->items[i].util;\n ++\t\tgroup = writer->pseudo_merge_groups.items[i].util;\n +\t\tif (regexec(group->pattern, refname, ARRAY_SIZE(captures),\n +\t\t\t captures, 0))\n +\t\t\tcontinue;\n @@ pseudo-merge.c\n +\n +{\n +\tstruct pseudo_merge_commit_idx *pmc;\n -+\tkhiter_t hash_pos;\n -+\n -+\thash_pos = kh_get_oid_map(pseudo_merge_commits, *oid);\n -+\tif (hash_pos == kh_end(pseudo_merge_commits)) {\n -+\t\tint hash_ret;\n -+\t\thash_pos = kh_put_oid_map(pseudo_merge_commits, *oid, &hash_ret);\n ++\tint hash_ret;\n ++\tkhiter_t hash_pos = kh_put_oid_map(pseudo_merge_commits, *oid,\n ++\t\t\t\t\t &hash_ret);\n +\n ++\tif (hash_ret) {\n +\t\tCALLOC_ARRAY(pmc, 1);\n -+\n +\t\tkh_value(pseudo_merge_commits, hash_pos) = pmc;\n +\t} else {\n +\t\tpmc = kh_value(pseudo_merge_commits, hash_pos);\n @@ pseudo-merge.c\n +\n +#define MIN_PSEUDO_MERGE_SIZE 8\n +\n -+static void select_pseudo_merges_1(struct pseudo_merge_group *group,\n -+\t\t\t\t struct pseudo_merge_matches *matches,\n -+\t\t\t\t kh_oid_map_t *pseudo_merge_commits,\n -+\t\t\t\t uint32_t *pseudo_merges_nr)\n ++static void select_pseudo_merges_1(struct bitmap_writer *writer,\n ++\t\t\t\t struct pseudo_merge_group *group,\n ++\t\t\t\t struct pseudo_merge_matches *matches)\n +{\n +\tuint32_t i, j;\n +\tuint32_t stable_merges_nr;\n @@ pseudo-merge.c\n +\t\tmerge = push_pseudo_merge(group);\n +\t\tp = &merge->parents;\n +\n ++\t\t/*\n ++\t\t * For each pseudo-merge created above, add parents to the\n ++\t\t * allocated commit node from the stable set of commits\n ++\t\t * (un-bitmapped, newer than the stable threshold).\n ++\t\t */\n +\t\tdo {\n +\t\t\tstruct commit *c;\n +\t\t\tstruct pseudo_merge_commit_idx *pmc;\n @@ pseudo-merge.c\n +\t\t\t\tbreak;\n +\n +\t\t\tc = matches->stable[j++];\n -+\t\t\tpmc = pseudo_merge_idx(pseudo_merge_commits,\n ++\t\t\t/*\n ++\t\t\t * Here and below, make sure that we keep our mapping of\n ++\t\t\t * commits -> pseudo-merge(s) which include the key'd\n ++\t\t\t * commit up-to-date.\n ++\t\t\t */\n ++\t\t\tpmc = pseudo_merge_idx(writer->pseudo_merge_commits,\n +\t\t\t\t\t &c->object.oid);\n +\n +\t\t\tALLOC_GROW(pmc->pseudo_merge, pmc->nr + 1, pmc->alloc);\n +\n -+\t\t\tpmc->pseudo_merge[pmc->nr++] = *pseudo_merges_nr;\n ++\t\t\tpmc->pseudo_merge[pmc->nr++] = writer->pseudo_merges_nr;\n +\t\t\tp = commit_list_append(c, p);\n +\t\t} while (j % group->stable_size);\n +\n -+\t\tbitmap_writer_push_bitmapped_commit(merge, 1);\n -+\t\t(*pseudo_merges_nr)++;\n ++\t\tbitmap_writer_push_commit(writer, merge, 1);\n ++\t\twriter->pseudo_merges_nr++;\n +\t}\n +\n +\t/* make up to group->max_merges pseudo merges for unstable commits */\n @@ pseudo-merge.c\n +\t\tsize = pseudo_merge_group_size(group, matches, i);\n +\t\tend = size < MIN_PSEUDO_MERGE_SIZE ? matches->unstable_nr : j + size;\n +\n ++\t\t/*\n ++\t\t * For each pseudo-merge commit created above, add parents to\n ++\t\t * the allocated commit node from the unstable set of commits\n ++\t\t * (newer than the stable threshold).\n ++\t\t *\n ++\t\t * Account for the sample rate, since not every candidate from\n ++\t\t * the set of stable commits will be included as a pseudo-merge\n ++\t\t * parent.\n ++\t\t */\n +\t\tfor (; j < end && j < matches->unstable_nr; j++) {\n +\t\t\tstruct commit *c = matches->unstable[j];\n +\t\t\tstruct pseudo_merge_commit_idx *pmc;\n +\n -+\t\t\tif (j % (100 / group->sample_rate))\n ++\t\t\tif (j % (uint32_t)(1.0f / group->sample_rate))\n +\t\t\t\tcontinue;\n +\n -+\t\t\tpmc = pseudo_merge_idx(pseudo_merge_commits,\n ++\t\t\tpmc = pseudo_merge_idx(writer->pseudo_merge_commits,\n +\t\t\t\t\t &c->object.oid);\n +\n +\t\t\tALLOC_GROW(pmc->pseudo_merge, pmc->nr + 1, pmc->alloc);\n +\n -+\t\t\tpmc->pseudo_merge[pmc->nr++] = *pseudo_merges_nr;\n ++\t\t\tpmc->pseudo_merge[pmc->nr++] = writer->pseudo_merges_nr;\n +\t\t\tp = commit_list_append(c, p);\n +\t\t}\n +\n -+\t\tbitmap_writer_push_bitmapped_commit(merge, 1);\n -+\t\t(*pseudo_merges_nr)++;\n ++\t\tbitmap_writer_push_commit(writer, merge, 1);\n ++\t\twriter->pseudo_merges_nr++;\n +\t\tif (end >= matches->unstable_nr)\n +\t\t\tbreak;\n +\t}\n @@ pseudo-merge.c\n +\tQSORT(matches->unstable, matches->unstable_nr, commit_date_cmp);\n +}\n +\n -+void select_pseudo_merges(struct string_list *list,\n -+\t\t\t struct commit **commits, size_t commits_nr,\n -+\t\t\t kh_oid_map_t *pseudo_merge_commits,\n -+\t\t\t uint32_t *pseudo_merges_nr,\n -+\t\t\t unsigned show_progress)\n ++void select_pseudo_merges(struct bitmap_writer *writer,\n ++\t\t\t struct commit **commits, size_t commits_nr)\n +{\n +\tstruct progress *progress = NULL;\n +\tuint32_t i;\n +\n -+\tif (!list->nr)\n ++\tif (!writer->pseudo_merge_groups.nr)\n +\t\treturn;\n +\n -+\tif (show_progress)\n -+\t\tprogress = start_progress(\"Selecting pseudo-merge commits\", list->nr);\n ++\tif (writer->show_progress)\n ++\t\tprogress = start_progress(\"Selecting pseudo-merge commits\",\n ++\t\t\t\t\t writer->pseudo_merge_groups.nr);\n +\n -+\tfor_each_ref(find_pseudo_merge_group_for_ref, list);\n ++\tfor_each_ref(find_pseudo_merge_group_for_ref, writer);\n +\n -+\tfor (i = 0; i < list->nr; i++) {\n ++\tfor (i = 0; i < writer->pseudo_merge_groups.nr; i++) {\n +\t\tstruct pseudo_merge_group *group;\n +\t\tstruct hashmap_iter iter;\n +\t\tstruct strmap_entry *e;\n +\n -+\t\tgroup = list->items[i].util;\n ++\t\tgroup = writer->pseudo_merge_groups.items[i].util;\n +\t\tstrmap_for_each_entry(&group->matches, &iter, e) {\n +\t\t\tstruct pseudo_merge_matches *matches = e->value;\n +\n +\t\t\tsort_pseudo_merge_matches(matches);\n +\n -+\t\t\tselect_pseudo_merges_1(group, matches,\n -+\t\t\t\t\t pseudo_merge_commits,\n -+\t\t\t\t\t pseudo_merges_nr);\n ++\t\t\tselect_pseudo_merges_1(writer, group, matches);\n +\t\t}\n +\n +\t\tdisplay_progress(progress, i + 1);\n @@ pseudo-merge.h\n +struct commit;\n +struct string_list;\n +struct bitmap_index;\n ++struct bitmap_writer;\n +\n +/*\n + * A pseudo-merge group tracks the set of non-bitmapped reference tips\n @@ pseudo-merge.h\n +\t */\n +\tfloat decay;\n +\tint max_merges;\n -+\tint sample_rate;\n ++\tfloat sample_rate;\n +\tint stable_size;\n +\ttimestamp_t threshold;\n +\ttimestamp_t stable_threshold;\n @@ pseudo-merge.h\n + *\n + * Optionally shows a progress meter.\n + */\n -+void select_pseudo_merges(struct string_list *list,\n -+\t\t\t struct commit **commits, size_t commits_nr,\n -+\t\t\t kh_oid_map_t *pseudo_merge_commits,\n -+\t\t\t uint32_t *pseudo_merges_nr,\n -+\t\t\t unsigned show_progress);\n ++void select_pseudo_merges(struct bitmap_writer *writer,\n ++\t\t\t struct commit **commits, size_t commits_nr);\n \n #endif\n10: 12b432e3a8a < -: ----------- pack-bitmap-write.c: select pseudo-merge commits\n11: 6ce805d061e ! 18: 311226f65c2 pack-bitmap-write.c: write pseudo-merge table\n @@ pack-bitmap-write.c\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n -@@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n +@@ pack-bitmap-write.c: static void write_selected_commits_v1(struct bitmap_writer *writer,\n \t}\n }\n \n -+static void write_pseudo_merges(struct hashfile *f)\n ++static void write_pseudo_merges(struct bitmap_writer *writer,\n ++\t\t\t\tstruct hashfile *f)\n +{\n +\tstruct oid_array commits = OID_ARRAY_INIT;\n +\tstruct bitmap **commits_bitmap = NULL;\n +\toff_t *pseudo_merge_ofs = NULL;\n +\toff_t start, table_start, next_ext;\n +\n -+\tuint32_t base = bitmap_writer_selected_nr();\n ++\tuint32_t base = bitmap_writer_nr_selected_commits(writer);\n +\tsize_t i, j = 0;\n +\n -+\tCALLOC_ARRAY(commits_bitmap, writer.pseudo_merges_nr);\n -+\tCALLOC_ARRAY(pseudo_merge_ofs, writer.pseudo_merges_nr);\n ++\tCALLOC_ARRAY(commits_bitmap, writer->pseudo_merges_nr);\n ++\tCALLOC_ARRAY(pseudo_merge_ofs, writer->pseudo_merges_nr);\n +\n -+\tfor (i = 0; i < writer.pseudo_merges_nr; i++) {\n -+\t\tstruct bitmapped_commit *merge = &writer.selected[base + i];\n ++\tfor (i = 0; i < writer->pseudo_merges_nr; i++) {\n ++\t\tstruct bitmapped_commit *merge = &writer->selected[base + i];\n +\t\tstruct commit_list *p;\n +\n +\t\tif (!merge->pseudo_merge)\n @@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n +\n +\t\tfor (p = merge->commit->parents; p; p = p->next)\n +\t\t\tbitmap_set(commits_bitmap[i],\n -+\t\t\t\t find_object_pos(&p->item->object.oid, NULL));\n ++\t\t\t\t find_object_pos(writer, &p->item->object.oid,\n ++\t\t\t\t\t\t NULL));\n +\t}\n +\n +\tstart = hashfile_total(f);\n +\n -+\tfor (i = 0; i < writer.pseudo_merges_nr; i++) {\n ++\tfor (i = 0; i < writer->pseudo_merges_nr; i++) {\n +\t\tstruct ewah_bitmap *commits_ewah = bitmap_to_ewah(commits_bitmap[i]);\n +\n +\t\tpseudo_merge_ofs[i] = hashfile_total(f);\n +\n +\t\tdump_bitmap(f, commits_ewah);\n -+\t\tdump_bitmap(f, writer.selected[base+i].write_as);\n ++\t\tdump_bitmap(f, writer->selected[base+i].write_as);\n +\n +\t\tewah_free(commits_ewah);\n +\t}\n +\n +\tnext_ext = st_add(hashfile_total(f),\n -+\t\t\t st_mult(kh_size(writer.pseudo_merge_commits),\n ++\t\t\t st_mult(kh_size(writer->pseudo_merge_commits),\n +\t\t\t\t sizeof(uint64_t)));\n +\n +\ttable_start = hashfile_total(f);\n +\n -+\tcommits.alloc = kh_size(writer.pseudo_merge_commits);\n ++\tcommits.alloc = kh_size(writer->pseudo_merge_commits);\n +\tCALLOC_ARRAY(commits.oid, commits.alloc);\n +\n -+\tfor (i = kh_begin(writer.pseudo_merge_commits); i != kh_end(writer.pseudo_merge_commits); i++) {\n -+\t\tif (!kh_exist(writer.pseudo_merge_commits, i))\n ++\tfor (i = kh_begin(writer->pseudo_merge_commits); i != kh_end(writer->pseudo_merge_commits); i++) {\n ++\t\tif (!kh_exist(writer->pseudo_merge_commits, i))\n +\t\t\tcontinue;\n -+\t\toid_array_append(&commits, &kh_key(writer.pseudo_merge_commits, i));\n ++\t\toid_array_append(&commits, &kh_key(writer->pseudo_merge_commits, i));\n +\t}\n +\n +\toid_array_sort(&commits);\n @@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n +\t\tint hash_pos;\n +\t\tstruct pseudo_merge_commit_idx *c;\n +\n -+\t\thash_pos = kh_get_oid_map(writer.pseudo_merge_commits,\n ++\t\thash_pos = kh_get_oid_map(writer->pseudo_merge_commits,\n +\t\t\t\t\t commits.oid[i]);\n -+\t\tif (hash_pos == kh_end(writer.pseudo_merge_commits))\n ++\t\tif (hash_pos == kh_end(writer->pseudo_merge_commits))\n +\t\t\tBUG(\"could not find pseudo-merge commit %s\",\n +\t\t\t oid_to_hex(&commits.oid[i]));\n +\n -+\t\tc = kh_value(writer.pseudo_merge_commits, hash_pos);\n ++\t\tc = kh_value(writer->pseudo_merge_commits, hash_pos);\n +\n -+\t\thashwrite_be32(f, find_object_pos(&commits.oid[i], NULL));\n ++\t\thashwrite_be32(f, find_object_pos(writer, &commits.oid[i],\n ++\t\t\t\t\t\t NULL));\n +\t\tif (c->nr == 1)\n +\t\t\thashwrite_be64(f, pseudo_merge_ofs[c->pseudo_merge[0]]);\n +\t\telse if (c->nr > 1) {\n @@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n +\t\tint hash_pos;\n +\t\tstruct pseudo_merge_commit_idx *c;\n +\n -+\t\thash_pos = kh_get_oid_map(writer.pseudo_merge_commits,\n ++\t\thash_pos = kh_get_oid_map(writer->pseudo_merge_commits,\n +\t\t\t\t\t commits.oid[i]);\n -+\t\tif (hash_pos == kh_end(writer.pseudo_merge_commits))\n ++\t\tif (hash_pos == kh_end(writer->pseudo_merge_commits))\n +\t\t\tBUG(\"could not find pseudo-merge commit %s\",\n +\t\t\t oid_to_hex(&commits.oid[i]));\n +\n -+\t\tc = kh_value(writer.pseudo_merge_commits, hash_pos);\n ++\t\tc = kh_value(writer->pseudo_merge_commits, hash_pos);\n +\t\tif (c->nr == 1)\n +\t\t\tcontinue;\n +\n @@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n +\t}\n +\n +\t/* write positions for all pseudo merges */\n -+\tfor (i = 0; i < writer.pseudo_merges_nr; i++)\n ++\tfor (i = 0; i < writer->pseudo_merges_nr; i++)\n +\t\thashwrite_be64(f, pseudo_merge_ofs[i]);\n +\n -+\thashwrite_be32(f, writer.pseudo_merges_nr);\n -+\thashwrite_be32(f, kh_size(writer.pseudo_merge_commits));\n ++\thashwrite_be32(f, writer->pseudo_merges_nr);\n ++\thashwrite_be32(f, kh_size(writer->pseudo_merge_commits));\n +\thashwrite_be64(f, table_start - start);\n +\thashwrite_be64(f, hashfile_total(f) - start + sizeof(uint64_t));\n +\n -+\tfor (i = 0; i < writer.pseudo_merges_nr; i++)\n ++\tfor (i = 0; i < writer->pseudo_merges_nr; i++)\n +\t\tbitmap_free(commits_bitmap[i]);\n +\n +\tfree(pseudo_merge_ofs);\n @@ pack-bitmap-write.c: static void write_selected_commits_v1(struct hashfile *f,\n +\n static int table_cmp(const void *_va, const void *_vb, void *_data)\n {\n - \tuint32_t *commit_positions = _data;\n -@@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n + \tstruct bitmap_writer *writer = _data;\n +@@ pack-bitmap-write.c: void bitmap_writer_finish(struct bitmap_writer *writer,\n \n \tint fd = odb_mkstemp(&tmp_file, \"pack/tmp_bitmap_XXXXXX\");\n \n -+\tif (writer.pseudo_merges_nr)\n ++\tif (writer->pseudo_merges_nr)\n +\t\toptions |= BITMAP_OPT_PSEUDO_MERGES;\n +\n \tf = hashfd(fd, tmp_file.buf);\n \n \tmemcpy(header.magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE));\n -@@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,\n +@@ pack-bitmap-write.c: void bitmap_writer_finish(struct bitmap_writer *writer,\n \n - \twrite_selected_commits_v1(f, commit_positions, offsets);\n + \twrite_selected_commits_v1(writer, f, offsets);\n \n +\tif (options & BITMAP_OPT_PSEUDO_MERGES)\n -+\t\twrite_pseudo_merges(f);\n ++\t\twrite_pseudo_merges(writer, f);\n +\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n - \t\twrite_lookup_table(f, commit_positions, offsets);\n + \t\twrite_lookup_table(writer, f, offsets);\n \n \n ## pack-bitmap.h ##\n12: 60f6b310213 = 19: 55dd7a8023e pack-bitmap: extract `read_bitmap()` function\n13: 9465313691b ! 20: 3cc5434e44e pseudo-merge: scaffolding for reads\n @@ Commit message\n Signed-off-by: Taylor Blau \n \n ## pseudo-merge.c ##\n -@@ pseudo-merge.c: void select_pseudo_merges(struct string_list *list,\n +@@ pseudo-merge.c: void select_pseudo_merges(struct bitmap_writer *writer,\n \n \tstop_progress(&progress);\n }\n @@ pseudo-merge.c: void select_pseudo_merges(struct string_list *list,\n +}\n \n ## pseudo-merge.h ##\n -@@ pseudo-merge.h: void select_pseudo_merges(struct string_list *list,\n - \t\t\t uint32_t *pseudo_merges_nr,\n - \t\t\t unsigned show_progress);\n +@@ pseudo-merge.h: struct pseudo_merge_commit_idx {\n + void select_pseudo_merges(struct bitmap_writer *writer,\n + \t\t\t struct commit **commits, size_t commits_nr);\n \n +/*\n + * Represents a serialized view of a file containing pseudo-merge(s)\n14: 5894f3d5369 = 21: 7664f5f9648 pack-bitmap.c: read pseudo-merge extension\n15: 7dbee8bcbdf = 22: 8ba0a9c5402 pseudo-merge: implement support for reading pseudo-merge commits\n16: 09650aa53e9 = 23: 2c02f303b6f ewah: implement `ewah_bitmap_popcount()`\n17: 7b5ea56d053 = 24: 82cce72bf55 pack-bitmap: implement test helpers for pseudo-merge\n18: 006abdd1698 = 25: 890f6c4b9de t/test-lib-functions.sh: support `--date` in `test_commit_bulk()`\n19: 3f85e5b90f5 ! 26: 41691824f78 pack-bitmap.c: use pseudo-merges during traversal\n @@ t/t5333-pseudo-merge-bitmaps.sh (new)\n +\n +test_expect_success 'bitmapPseudoMerge.sampleRate adjusts commit selection rate' '\n +\ttest_config bitmapPseudoMerge.test.pattern \"refs/tags/\" &&\n -+\ttest_config bitmapPseudoMerge.test.maxMerges 8 &&\n ++\ttest_config bitmapPseudoMerge.test.maxMerges 1 &&\n +\ttest_config bitmapPseudoMerge.test.stableThreshold never &&\n +\n +\tcommits_nr=$(git rev-list --all --count) &&\n +\n -+\tfor rate in 100 50 10\n ++\tfor rate in 1.0 0.5 0.25\n +\tdo\n +\t\tgit -c bitmapPseudoMerge.test.sampleRate=$rate repack -adb &&\n +\n +\t\ttest_pseudo_merges >merges &&\n -+\t\tfor i in $(test_seq 0 $(($(wc -l commits &&\n ++\t\ttest_line_count = 1 merges &&\n ++\t\ttest_pseudo_merge_commits 0 >commits &&\n +\n +\t\ttest-tool bitmap list-commits >bitmaps &&\n +\t\tbitmaps_nr=\"$(wc -l expect &&\n +\n +\t\ttest $(cat expect) -eq $(wc -l object.flags & BITMAP_PSEUDO_MERGE)\n @@ pack-bitmap-write.c: static int fill_bitmap_commit(struct bb_commit *ent,\n \t\t\telse\n \t\t\t\told = bitmap_for_commit(old_bitmap, c);\n \t\t\t/*\n -@@ pack-bitmap-write.c: static int fill_bitmap_commit(struct bb_commit *ent,\n +@@ pack-bitmap-write.c: static int fill_bitmap_commit(struct bitmap_writer *writer,\n \t\t\tif (old && !rebuild_bitmap(mapping, old, remapped)) {\n \t\t\t\tbitmap_or(ent->bitmap, remapped);\n \t\t\t\tbitmap_free(remapped);\n @@ pack-bitmap-write.c: static int fill_bitmap_commit(struct bb_commit *ent,\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tbitmap_free(remapped);\n -@@ pack-bitmap-write.c: int bitmap_writer_build(struct packing_data *to_pack)\n +@@ pack-bitmap-write.c: int bitmap_writer_build(struct bitmap_writer *writer,\n \t\t\t the_repository);\n \ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n \t\t\t \"building_bitmaps_reused\", reused_bitmaps_nr);\n @@ pack-bitmap-write.c: int bitmap_writer_build(struct packing_data *to_pack)\n +\t\t\t \"building_bitmaps_pseudo_merge_reused\",\n +\t\t\t reused_pseudo_merge_bitmaps_nr);\n \n - \tstop_progress(&writer.progress);\n + \tstop_progress(&writer->progress);\n \n \n ## pack-bitmap.c ##\n @@ pack-bitmap.h: int rebuild_bitmap(const uint32_t *reposition,\n \t\t\t\t struct commit *commit);\n +struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_git,\n +\t\t\t\t\t\t struct commit *commit);\n - void bitmap_writer_select_commits(struct commit **indexed_commits,\n + void bitmap_writer_select_commits(struct bitmap_writer *writer,\n + \t\t\t\t struct commit **indexed_commits,\n \t\t\t\t unsigned int indexed_commits_nr);\n - int bitmap_writer_build(struct packing_data *to_pack);\n \n ## pseudo-merge.c ##\n @@ pseudo-merge.c: int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n23: 2bd830d35dd = 30: 6a6d88fa512 t/perf: implement performace tests for pseudo-merge bitmaps\n-- \n2.45.1.175.gbea44add9db\n"},{"id":"495184","messageId":"38c96fc1909162a4d9c188f55b7c708cfc1b14b9.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 01/30] object.h: add flags allocated by pack-bitmap.h","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:01:48Z","receivedAt":"2024-05-21T19:01:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In commit 7cc8f971085 (pack-objects: implement bitmap writing,\n2013-12-21) the NEEDS_BITMAP flag was introduced into pack-bitmap.h, but\nno object flags allocation table existed at the time.\n\nIn 208acbfb82f (object.h: centralize object flag allocation, 2014-03-25)\nwhen that table was first introduced, we never added the flags from\n7cc8f971085, which has remained the case since.\n\nRectify this by including the flag bit used by pack-bitmap.h into the\ncentralized table in object.h.\n\nSigned-off-by: Taylor Blau \n---\n object.h | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/object.h b/object.h\nindex 9293e703ccc..99b9c8f114c 100644\n--- a/object.h\n+++ b/object.h\n@@ -81,6 +81,7 @@ void object_array_init(struct object_array *array);\n * reflog.c: 10--12\n * builtin/show-branch.c: 0-------------------------------------------26\n * builtin/unpack-objects.c: 2021\n+ * pack-bitmap.h: 22\n */\n #define FLAG_BITS 28\n \n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495185","messageId":"0f20c9becf452ef7a7e931b36336ccddba0f1d13.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 07/30] Documentation/gitpacking.txt: initial commit","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:01:59Z","receivedAt":"2024-05-21T19:02:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a new manual page, gitpacking(7) to collect useful information\nabout advanced packing concepts in Git.\n\nIn future commits in this series, this manual page will expand to\ndescribe the new pseudo-merge bitmaps feature, as well as include\nexamples, relevant configuration bits, use-cases, and so on.\n\nOutside of this series, this manual page may absorb similar pieces from\nother parts of Git's documentation about packing.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/Makefile | 1 +\n Documentation/gitpacking.txt | 34 ++++++++++++++++++++++++++++++++++\n 2 files changed, 35 insertions(+)\n create mode 100644 Documentation/gitpacking.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex 3f2383a12c7..920b6248aa4 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -51,6 +51,7 @@ MAN7_TXT += gitdiffcore.txt\n MAN7_TXT += giteveryday.txt\n MAN7_TXT += gitfaq.txt\n MAN7_TXT += gitglossary.txt\n+MAN7_TXT += gitpacking.txt\n MAN7_TXT += gitnamespaces.txt\n MAN7_TXT += gitremote-helpers.txt\n MAN7_TXT += gitrevisions.txt\ndiff --git a/Documentation/gitpacking.txt b/Documentation/gitpacking.txt\nnew file mode 100644\nindex 00000000000..50e9900d845\n--- /dev/null\n+++ b/Documentation/gitpacking.txt\n@@ -0,0 +1,34 @@\n+gitpacking(7)\n+=============\n+\n+NAME\n+----\n+gitpacking - Advanced concepts related to packing in Git\n+\n+SYNOPSIS\n+--------\n+gitpacking\n+\n+DESCRIPTION\n+-----------\n+\n+This document aims to describe some advanced concepts related to packing\n+in Git.\n+\n+Many concepts are currently described scattered between manual pages of\n+various Git commands, including linkgit:git-pack-objects[1],\n+linkgit:git-repack[1], and others, as well as linkgit:gitformat-pack[5],\n+and parts of the `Documentation/technical` tree.\n+\n+There are many aspects of packing in Git that are not covered in this\n+document that instead live in the aforementioned areas. Over time, those\n+scattered bits may coalesce into this document.\n+\n+SEE ALSO\n+--------\n+linkgit:git-pack-objects[1]\n+linkgit:git-repack[1]\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495186","messageId":"528b591bd84e189d7d48e01deb8e876f74fc471b.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 08/30] Documentation/gitpacking.txt: describe pseudo-merge bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:02Z","receivedAt":"2024-05-21T19:02:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Add some details to the gitpacking(7) manual page which motivate and\ndescribe pseudo-merge bitmaps.\n\nThe exact on-disk format and many of the configuration knobs will be\ndescribed in subsequent commits.\n\nHelped-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n Documentation/gitpacking.txt | 69 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 69 insertions(+)\n\ndiff --git a/Documentation/gitpacking.txt b/Documentation/gitpacking.txt\nindex 50e9900d845..ff18077129b 100644\n--- a/Documentation/gitpacking.txt\n+++ b/Documentation/gitpacking.txt\n@@ -24,6 +24,75 @@ There are many aspects of packing in Git that are not covered in this\n document that instead live in the aforementioned areas. Over time, those\n scattered bits may coalesce into this document.\n \n+== Pseudo-merge bitmaps\n+\n+=== Background\n+\n+Reachability bitmaps are most efficient when we have on-disk stored\n+bitmaps for one or more of the starting points of a traversal. For this\n+reason, Git prefers storing bitmaps for commits at the tips of refs,\n+because traversals tend to start with those points.\n+\n+But if you have a large number of refs, it's not feasible to store a\n+bitmap for _every_ ref tip. It takes up space, and just OR-ing all of\n+those bitmaps together is expensive.\n+\n+One way we can deal with that is to create bitmaps that represent\n+_groups_ of refs. When a traversal asks about the entire group, then we\n+can use this single bitmap instead of considering each ref individually.\n+Because these bitmaps represent the set of objects which would be\n+reachable in a hypothetical merge of all of the commits, we call them\n+pseudo-merge bitmaps.\n+\n+=== Overview\n+\n+A \"pseudo-merge bitmap\" is used to refer to a pair of bitmaps, as\n+follows:\n+\n+Commit bitmap::\n+\n+ A bitmap whose set bits describe the set of commits included in the\n+ pseudo-merge's \"merge\" bitmap (as below).\n+\n+Merge bitmap::\n+\n+ A bitmap whose set bits describe the reachability closure over the set\n+ of commits in the pseudo-merge's \"commits\" bitmap (as above). An\n+ identical bitmap would be generated for an octopus merge with the same\n+ set of parents as described in the commits bitmap.\n+\n+Pseudo-merge bitmaps can accelerate bitmap traversals when all commits\n+for a given pseudo-merge are listed on either side of the traversal,\n+either directly (by explicitly asking for them as part of the `HAVES`\n+or `WANTS`) or indirectly (by encountering them during a fill-in\n+traversal).\n+\n+=== Use-cases\n+\n+For example, suppose there exists a pseudo-merge bitmap with a large\n+number of commits, all of which are listed in the `WANTS` section of\n+some bitmap traversal query. When pseudo-merge bitmaps are enabled, the\n+bitmap machinery can quickly determine there is a pseudo-merge which\n+satisfies some subset of the wanted objects on either side of the query.\n+Then, we can inflate the EWAH-compressed bitmap, and `OR` it in to the\n+resulting bitmap. By contrast, without pseudo-merge bitmaps, we would\n+have to repeat the decompression and `OR`-ing step over a potentially\n+large number of individual bitmaps, which can take proportionally more\n+time.\n+\n+Another benefit of pseudo-merges arises when there is some combination\n+of (a) a large number of references, with (b) poor bitmap coverage, and\n+(c) deep, nested trees, making fill-in traversal relatively expensive.\n+For example, suppose that there are a large enough number of tags where\n+bitmapping each of the tags individually is infeasible. Without\n+pseudo-merge bitmaps, computing the result of, say, `git rev-list\n+--use-bitmap-index --count --objects --tags` would likely require a\n+large amount of fill-in traversal. But when a large quantity of those\n+tags are stored together in a pseudo-merge bitmap, the bitmap machinery\n+can take advantage of the fact that we only care about the union of\n+objects reachable from all of those tags, and answer the query much\n+faster.\n+\n SEE ALSO\n --------\n linkgit:git-pack-objects[1]\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495187","messageId":"12f318b3d7e20ff0b6038d6be7f2534f885f3220.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 09/30] Documentation/technical: describe pseudo-merge bitmaps format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:06Z","receivedAt":"2024-05-21T19:02:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to implement pseudo-merge bitmaps over the next several commits\nby first describing the serialization format which will store the new\npseudo-merge bitmaps themselves.\n\nThis format is implemented as an optional extension within the bitmap v1\nformat, making it compatible with previous versions of Git, as well as\nthe original .bitmap implementation within JGit.\n\nThe format is described in detail in the patch contents below, but the\nhigh-level description is as follows:\n\n - An array of pseudo-merge bitmaps, each containing a pair of EWAH\n bitmaps: one describing the set of pseudo-merge \"parents\", and\n another describing the set of object(s) reachable from those\n parents.\n\n - A lookup table to determine which pseudo-merge(s) a given commit\n appears in. An optional extended lookup table follows when there is\n at least one commit which appears in multiple pseudo-merge groups.\n\n - Trailing metadata, including the number of pseudo-merge(s), number\n of unique parents, the offset within the .bitmap file for the\n pseudo-merge commit lookup table, and the size of the optional\n extension itself.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/technical/bitmap-format.txt | 132 ++++++++++++++++++++++\n 1 file changed, 132 insertions(+)\n\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex f5d200939b0..ee7775a2586 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -255,3 +255,135 @@ triplet is -\n \txor_row (4 byte integer, network byte order): ::\n \tThe position of the triplet whose bitmap is used to compress\n \tthis one, or `0xffffffff` if no such bitmap exists.\n+\n+Pseudo-merge bitmaps\n+--------------------\n+\n+If the `BITMAP_OPT_PSEUDO_MERGES` flag is set, a variable number of\n+bytes (preceding the name-hash cache, commit lookup table, and trailing\n+checksum) of the `.bitmap` file is used to store pseudo-merge bitmaps.\n+\n+For more information on what pseudo-merges are, why they are useful, and\n+how to configure them, see the information in linkgit:gitpacking[7].\n+\n+=== File format\n+\n+If enabled, pseudo-merge bitmaps are stored in an optional section at\n+the end of a `.bitmap` file. The format is as follows:\n+\n+....\n++-------------------------------------------+\n+| .bitmap File |\n++-------------------------------------------+\n+| |\n+| Pseudo-merge bitmaps (Variable Length) |\n+| +---------------------------+ |\n+| | commits_bitmap (EWAH) | |\n+| +---------------------------+ |\n+| | merge_bitmap (EWAH) | |\n+| +---------------------------+ |\n+| |\n++-------------------------------------------+\n+| |\n+| Lookup Table |\n+| +---------------------------+ |\n+| | commit_pos (4 bytes) | |\n+| +---------------------------+ |\n+| | offset (8 bytes) | |\n+| +------------+--------------+ |\n+| |\n+| Offset Cases: |\n+| ------------- |\n+| |\n+| 1. MSB Unset: single pseudo-merge bitmap |\n+| + offset to pseudo-merge bitmap |\n+| |\n+| 2. MSB Set: multiple pseudo-merges |\n+| + offset to extended lookup table |\n+| |\n++-------------------------------------------+\n+| |\n+| Extended Lookup Table (Optional) |\n+| +----+----------+----------+----------+ |\n+| | N | Offset 1 | .... | Offset N | |\n+| +----+----------+----------+----------+ |\n+| | | 8 bytes | .... | 8 bytes | |\n+| +----+----------+----------+----------+ |\n+| |\n++-------------------------------------------+\n+| |\n+| Pseudo-merge Metadata |\n+| +-----------------------------------+ |\n+| | # pseudo-merges (4 bytes) | |\n+| +-----------------------------------+ |\n+| | # commits (4 bytes) | |\n+| +-----------------------------------+ |\n+| | Lookup offset (8 bytes) | |\n+| +-----------------------------------+ |\n+| | Extension size (8 bytes) | |\n+| +-----------------------------------+ |\n+| |\n++-------------------------------------------+\n+....\n+\n+* One or more pseudo-merge bitmaps, each containing:\n+\n+ ** `commits_bitmap`, an EWAH-compressed bitmap describing the set of\n+ commits included in the this psuedo-merge.\n+\n+ ** `merge_bitmap`, an EWAH-compressed bitmap describing the union of\n+ the set of objects reachable from all commits listed in the\n+ `commits_bitmap`.\n+\n+* A lookup table, mapping pseudo-merged commits to the pseudo-merges\n+ they belong to. Entries appear in increasing order of each commit's\n+ bit position. Each entry is 12 bytes wide, and is comprised of the\n+ following:\n+\n+ ** `commit_pos`, a 4-byte unsigned value (in network byte-order)\n+ containing the bit position for this commit.\n+\n+ ** `offset`, an 8-byte unsigned value (also in network byte-order)\n+ containing either one of two possible offsets, depending on whether or\n+ not the most-significant bit is set.\n+\n+ *** If unset (i.e. `offset & ((uint64_t)1<<63) == 0`), the offset\n+\t(relative to the beginning of the `.bitmap` file) at which the\n+\tpseudo-merge bitmap for this commit can be read. This indicates\n+\tonly a single pseudo-merge bitmap contains this commit.\n+\n+ *** If set (i.e. `offset & ((uint64_t)1<<63) != 0`), the offset\n+\t(again relative to the beginning of the `.bitmap` file) at which\n+\tthe extended offset table can be located describing the set of\n+\tpseudo-merge bitmaps which contain this commit. This indicates\n+\tthat multiple pseudo-merge bitmaps contain this commit.\n+\n+* An (optional) extended lookup table (written if and only if there is\n+ at least one commit which appears in more than one pseudo-merge).\n+ There are as many entries as commits which appear in multiple\n+ pseudo-merges. Each entry contains the following:\n+\n+ ** `N`, a 4-byte unsigned value equal to the number of pseudo-merges\n+ which contain a given commit.\n+\n+ ** An array of `N` 8-byte unsigned values, each of which is\n+ interpreted as an offset (relative to the beginning of the\n+ `.bitmap` file) at which a pseudo-merge bitmap for this commit can\n+ be read. These values occur in no particular order.\n+\n+* Positions for all pseudo-merges, each stored as an 8-byte unsigned\n+ value (in network byte-order) containing the offset (relative to the\n+ beginning of the `.bitmap` file) of each consecutive pseudo-merge.\n+\n+* A 4-byte unsigned value (in network byte-order) equal to the number of\n+ pseudo-merges.\n+\n+* A 4-byte unsigned value (in network byte-order) equal to the number of\n+ unique commits which appear in any pseudo-merge.\n+\n+* An 8-byte unsigned value (in network byte-order) equal to the number\n+ of bytes between the start of the pseudo-merge section and the\n+ beginning of the lookup table.\n+\n+* An 8-byte unsigned value (in network byte-order) equal to the number\n+ of bytes in the pseudo-merge section (including this field).\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495188","messageId":"40eb6137618ad4c648eaafd720a9678d8d84c96a.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 10/30] ewah: implement `ewah_bitmap_is_subset()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:09Z","receivedAt":"2024-05-21T19:02:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In order to know whether a given pseudo-merge (comprised of a \"parents\"\nand \"objects\" bitmaps) is \"satisfied\" and can be OR'd into the bitmap\nresult, we need to be able to quickly determine whether the \"parents\"\nbitmap is a subset of the current set of objects reachable on either\nside of a traversal.\n\nImplement a helper function to prepare for that, which determines\nwhether an EWAH bitmap (the parents bitmap from the pseudo-merge) is a\nsubset of a non-EWAH bitmap (in this case, the results bitmap from\neither side of the traversal).\n\nThis function makes use of the EWAH iterator to avoid inflating any part\nof the EWAH bitmap after we determine it is not a subset of the non-EWAH\nbitmap. This \"fail-fast\" allows us to avoid a potentially large amount\nof wasted effort.\n\nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 43 +++++++++++++++++++++++++++++++++++++++++++\n ewah/ewok.h | 6 ++++++\n 2 files changed, 49 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex ac7e0af622a..d352fec54ce 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -138,6 +138,49 @@ void bitmap_or(struct bitmap *self, const struct bitmap *other)\n \t\tself->words[i] |= other->words[i];\n }\n \n+int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other)\n+{\n+\tstruct ewah_iterator it;\n+\teword_t word;\n+\tsize_t i;\n+\n+\tewah_iterator_init(&it, self);\n+\n+\tfor (i = 0; i < other->word_alloc; i++) {\n+\t\tif (!ewah_iterator_next(&word, &it)) {\n+\t\t\t/*\n+\t\t\t * If we reached the end of `self`, and haven't\n+\t\t\t * rejected `self` as a possible subset of\n+\t\t\t * `other` yet, then we are done and `self` is\n+\t\t\t * indeed a subset of `other`.\n+\t\t\t */\n+\t\t\treturn 1;\n+\t\t}\n+\t\tif (word & ~other->words[i]) {\n+\t\t\t/*\n+\t\t\t * Otherwise, compare the next two pairs of\n+\t\t\t * words. If the word from `self` has bit(s) not\n+\t\t\t * in the word from `other`, `self` is not a\n+\t\t\t * subset of `other`.\n+\t\t\t */\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * If we got to this point, there may be zero or more words\n+\t * remaining in `self`, with no remaining words left in `other`.\n+\t * If there are any bits set in the remaining word(s) in `self`,\n+\t * then `self` is not a subset of `other`.\n+\t */\n+\twhile (ewah_iterator_next(&word, &it))\n+\t\tif (word)\n+\t\t\treturn 0;\n+\n+\t/* `self` is definitely a subset of `other` */\n+\treturn 1;\n+}\n+\n void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other)\n {\n \tsize_t original_size = self->word_alloc;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex c11d76c6f33..2b6c4ac499c 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -179,7 +179,13 @@ void bitmap_unset(struct bitmap *self, size_t pos);\n int bitmap_get(struct bitmap *self, size_t pos);\n void bitmap_free(struct bitmap *self);\n int bitmap_equals(struct bitmap *self, struct bitmap *other);\n+\n+/*\n+ * Both `bitmap_is_subset()` and `ewah_bitmap_is_subset()` return 1 if the set\n+ * of bits in 'self' are a subset of the bits in 'other'. Returns 0 otherwise.\n+ */\n int bitmap_is_subset(struct bitmap *self, struct bitmap *other);\n+int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other);\n \n struct ewah_bitmap * bitmap_to_ewah(struct bitmap *bitmap);\n struct bitmap *ewah_to_bitmap(struct ewah_bitmap *ewah);\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495189","messageId":"487fb7c6e9c760f3febb1d795b9b6f408f67ea0d.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 11/30] pack-bitmap: move some initialization to `bitmap_writer_init()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:12Z","receivedAt":"2024-05-21T19:02:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pack-bitmap-writer machinery uses a oidmap (backed by khash.h) to\nmap from commits selected for bitmaps (by OID) to a bitmapped_commit\nstructure (containing the bitmap itself, among other things like its XOR\noffset, etc.)\n\nThis map was initialized at the end of `bitmap_writer_build()`. New\nentries are added in `pack-bitmap-write.c::store_selected()`, which is\ncalled by the bitmap_builder machinery (which is responsible for\ntraversing history and generating the actual bitmaps).\n\nReorganize when this field is initialized and when entries are added to\nit so that we can quickly determine whether a commit is a candidate for\npseudo-merge selection, or not (since it was already selected to receive\na bitmap, and thus storing it in a pseudo-merge would be redundant).\n\nThe changes are as follows:\n\n - Introduce a new `bitmap_writer_init()` function which initializes\n the `writer.bitmaps` field (instead of waiting until the end of\n `bitmap_writer_build()`).\n\n - Add map entries in `push_bitmapped_commit()` (which is called via\n `bitmap_writer_select_commits()`) with OID keys and NULL values to\n track whether or not we *expect* to write a bitmap for some given\n commit.\n\n - Validate that a NULL entry is found matching the given key when we\n store a selected bitmap.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 3 ++-\n midx-write.c | 2 +-\n pack-bitmap-write.c | 24 ++++++++++++++++++------\n pack-bitmap.h | 2 +-\n 4 files changed, 22 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 26a6d0d7919..6209264e60c 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1340,7 +1340,8 @@ static void write_pack_file(void)\n \t\t\t\t hash_to_hex(hash));\n \n \t\t\tif (write_bitmap_index) {\n-\t\t\t\tbitmap_writer_init(&bitmap_writer);\n+\t\t\t\tbitmap_writer_init(&bitmap_writer,\n+\t\t\t\t\t\t the_repository);\n \t\t\t\tbitmap_writer_set_checksum(&bitmap_writer, hash);\n \t\t\t\tbitmap_writer_build_type_index(&bitmap_writer,\n \t\t\t\t\t&to_pack, written_list, nr_written);\ndiff --git a/midx-write.c b/midx-write.c\nindex 7c0c08c64b2..c747d1a6af3 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -820,7 +820,7 @@ static int write_midx_bitmap(const char *midx_name,\n \tfor (i = 0; i < pdata->nr_objects; i++)\n \t\tindex[i] = &pdata->objects[i].idx;\n \n-\tbitmap_writer_init(&writer);\n+\tbitmap_writer_init(&writer, the_repository);\n \tbitmap_writer_show_progress(&writer, flags & MIDX_PROGRESS);\n \tbitmap_writer_build_type_index(&writer, pdata, index,\n \t\t\t\t pdata->nr_objects);\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 6cae670412c..d8870155831 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -27,9 +27,12 @@ struct bitmapped_commit {\n \tuint32_t commit_pos;\n };\n \n-void bitmap_writer_init(struct bitmap_writer *writer)\n+void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r)\n {\n \tmemset(writer, 0, sizeof(struct bitmap_writer));\n+\tif (writer->bitmaps)\n+\t\tBUG(\"bitmap writer already initialized\");\n+\twriter->bitmaps = kh_init_oid_map();\n }\n \n void bitmap_writer_free(struct bitmap_writer *writer)\n@@ -128,11 +131,21 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n static inline void push_bitmapped_commit(struct bitmap_writer *writer,\n \t\t\t\t\t struct commit *commit)\n {\n+\tint hash_ret;\n+\tkhiter_t hash_pos;\n+\n \tif (writer->selected_nr >= writer->selected_alloc) {\n \t\twriter->selected_alloc = (writer->selected_alloc + 32) * 2;\n \t\tREALLOC_ARRAY(writer->selected, writer->selected_alloc);\n \t}\n \n+\thash_pos = kh_put_oid_map(writer->bitmaps, commit->object.oid,\n+\t\t\t\t &hash_ret);\n+\tif (!hash_ret)\n+\t\tdie(_(\"duplicate entry when writing bitmap index: %s\"),\n+\t\t oid_to_hex(&commit->object.oid));\n+\tkh_value(writer->bitmaps, hash_pos) = NULL;\n+\n \twriter->selected[writer->selected_nr].commit = commit;\n \twriter->selected[writer->selected_nr].bitmap = NULL;\n \twriter->selected[writer->selected_nr].write_as = NULL;\n@@ -483,14 +496,14 @@ static void store_selected(struct bitmap_writer *writer,\n {\n \tstruct bitmapped_commit *stored = &writer->selected[ent->idx];\n \tkhiter_t hash_pos;\n-\tint hash_ret;\n \n \tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n \n-\thash_pos = kh_put_oid_map(writer->bitmaps, commit->object.oid, &hash_ret);\n-\tif (hash_ret == 0)\n-\t\tdie(\"Duplicate entry when writing index: %s\",\n+\thash_pos = kh_get_oid_map(writer->bitmaps, commit->object.oid);\n+\tif (hash_pos == kh_end(writer->bitmaps))\n+\t\tdie(_(\"attempted to store non-selected commit: '%s'\"),\n \t\t oid_to_hex(&commit->object.oid));\n+\n \tkh_value(writer->bitmaps, hash_pos) = stored;\n }\n \n@@ -506,7 +519,6 @@ int bitmap_writer_build(struct bitmap_writer *writer,\n \tuint32_t *mapping;\n \tint closed = 1; /* until proven otherwise */\n \n-\twriter->bitmaps = kh_init_oid_map();\n \twriter->to_pack = to_pack;\n \n \tif (writer->show_progress)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3091095f336..f87e60153dd 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -114,7 +114,7 @@ struct bitmap_writer {\n \tunsigned char pack_checksum[GIT_MAX_RAWSZ];\n };\n \n-void bitmap_writer_init(struct bitmap_writer *writer);\n+void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r);\n void bitmap_writer_show_progress(struct bitmap_writer *writer, int show);\n void bitmap_writer_set_checksum(struct bitmap_writer *writer,\n \t\t\t\tconst unsigned char *sha1);\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495190","messageId":"827732acf99406fe30e87efd679fb6631de9c130.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 12/30] pseudo-merge.ch: initial commit","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:15Z","receivedAt":"2024-05-21T19:02:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Add a new (empty) header file to contain the implementation for\nselecting, reading, and applying pseudo-merge bitmaps.\n\nFor now this header and its corresponding implementation are left\nempty, but they will evolve over the course of subsequent commit(s).\n\nSigned-off-by: Taylor Blau \n---\n Makefile | 1 +\n pseudo-merge.c | 2 ++\n pseudo-merge.h | 6 ++++++\n 3 files changed, 9 insertions(+)\n create mode 100644 pseudo-merge.c\n create mode 100644 pseudo-merge.h\n\ndiff --git a/Makefile b/Makefile\nindex 0285db56306..4705a69f57f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1105,6 +1105,7 @@ LIB_OBJS += prompt.o\n LIB_OBJS += protocol.o\n LIB_OBJS += protocol-caps.o\n LIB_OBJS += prune-packed.o\n+LIB_OBJS += pseudo-merge.o\n LIB_OBJS += quote.o\n LIB_OBJS += range-diff.o\n LIB_OBJS += reachable.o\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nnew file mode 100644\nindex 00000000000..37e037ba272\n--- /dev/null\n+++ b/pseudo-merge.c\n@@ -0,0 +1,2 @@\n+#include \"git-compat-util.h\"\n+#include \"pseudo-merge.h\"\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nnew file mode 100644\nindex 00000000000..cab8ff6960a\n--- /dev/null\n+++ b/pseudo-merge.h\n@@ -0,0 +1,6 @@\n+#ifndef PSEUDO_MERGE_H\n+#define PSEUDO_MERGE_H\n+\n+#include \"git-compat-util.h\"\n+\n+#endif\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495191","messageId":"8608dd1860fd332655b7977b30a6cb79e78b692b.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 13/30] pack-bitmap-write: support storing pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:19Z","receivedAt":"2024-05-21T19:02:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to write pseudo-merge bitmaps by annotating individual bitmapped\ncommits (which are represented by the `bitmapped_commit` structure) with\nan extra bit indicating whether or not they are a pseudo-merge.\n\nIn subsequent commits, pseudo-merge bitmaps will be generated by\nallocating a fake commit node with parents covering the full set of\ncommits represented by the pseudo-merge bitmap. These commits will be\nadded to the set of \"selected\" commits as usual, but will be written\nspecially instead of being included with the rest of the selected\ncommits.\n\nMechanically speaking, there are two parts of this change:\n\n - The bitmapped_commit struct gets a new bit indicating whether it is\n a pseudo-merge, or an ordinary commit selected for bitmaps.\n\n - A handful of changes to only write out the non-pseudo-merge commits\n when enumerating through the selected array (see the new\n `bitmap_writer_selected_nr()` function). Pseudo-merge commits appear\n after all non-pseudo-merge commits, so it is safe to enumerate\n through the selected array like so:\n\n for (i = 0; i < bitmap_writer_selected_nr(); i++)\n if (writer.selected[i].pseudo_merge)\n BUG(\"unexpected pseudo-merge\");\n\n without encountering the BUG().\n\nSigned-off-by: Taylor Blau \n---\n object.h | 2 +-\n pack-bitmap-write.c | 96 +++++++++++++++++++++++++++++----------------\n pack-bitmap.h | 3 ++\n 3 files changed, 67 insertions(+), 34 deletions(-)\n\ndiff --git a/object.h b/object.h\nindex 99b9c8f114c..e6f9e89d3c5 100644\n--- a/object.h\n+++ b/object.h\n@@ -81,7 +81,7 @@ void object_array_init(struct object_array *array);\n * reflog.c: 10--12\n * builtin/show-branch.c: 0-------------------------------------------26\n * builtin/unpack-objects.c: 2021\n- * pack-bitmap.h: 22\n+ * pack-bitmap.h: 2122\n */\n #define FLAG_BITS 28\n \ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex d8870155831..60eb1e71c98 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -25,8 +25,14 @@ struct bitmapped_commit {\n \tint flags;\n \tint xor_offset;\n \tuint32_t commit_pos;\n+\tunsigned pseudo_merge : 1;\n };\n \n+static inline int bitmap_writer_nr_selected_commits(struct bitmap_writer *writer)\n+{\n+\treturn writer->selected_nr - writer->pseudo_merges_nr;\n+}\n+\n void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r)\n {\n \tmemset(writer, 0, sizeof(struct bitmap_writer));\n@@ -129,27 +135,31 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n */\n \n static inline void push_bitmapped_commit(struct bitmap_writer *writer,\n-\t\t\t\t\t struct commit *commit)\n+\t\t\t\t\t struct commit *commit,\n+\t\t\t\t\t unsigned pseudo_merge)\n {\n-\tint hash_ret;\n-\tkhiter_t hash_pos;\n-\n \tif (writer->selected_nr >= writer->selected_alloc) {\n \t\twriter->selected_alloc = (writer->selected_alloc + 32) * 2;\n \t\tREALLOC_ARRAY(writer->selected, writer->selected_alloc);\n \t}\n \n-\thash_pos = kh_put_oid_map(writer->bitmaps, commit->object.oid,\n-\t\t\t\t &hash_ret);\n-\tif (!hash_ret)\n-\t\tdie(_(\"duplicate entry when writing bitmap index: %s\"),\n-\t\t oid_to_hex(&commit->object.oid));\n-\tkh_value(writer->bitmaps, hash_pos) = NULL;\n+\tif (!pseudo_merge) {\n+\t\tint hash_ret;\n+\t\tkhiter_t hash_pos = kh_put_oid_map(writer->bitmaps,\n+\t\t\t\t\t\t commit->object.oid,\n+\t\t\t\t\t\t &hash_ret);\n+\n+\t\tif (!hash_ret)\n+\t\t\tdie(_(\"duplicate entry when writing bitmap index: %s\"),\n+\t\t\t oid_to_hex(&commit->object.oid));\n+\t\tkh_value(writer->bitmaps, hash_pos) = NULL;\n+\t}\n \n \twriter->selected[writer->selected_nr].commit = commit;\n \twriter->selected[writer->selected_nr].bitmap = NULL;\n \twriter->selected[writer->selected_nr].write_as = NULL;\n \twriter->selected[writer->selected_nr].flags = 0;\n+\twriter->selected[writer->selected_nr].pseudo_merge = pseudo_merge;\n \n \twriter->selected_nr++;\n }\n@@ -180,16 +190,20 @@ static void compute_xor_offsets(struct bitmap_writer *writer)\n \n \twhile (next < writer->selected_nr) {\n \t\tstruct bitmapped_commit *stored = &writer->selected[next];\n-\n \t\tint best_offset = 0;\n \t\tstruct ewah_bitmap *best_bitmap = stored->bitmap;\n \t\tstruct ewah_bitmap *test_xor;\n \n+\t\tif (stored->pseudo_merge)\n+\t\t\tgoto next;\n+\n \t\tfor (i = 1; i <= MAX_XOR_OFFSET_SEARCH; ++i) {\n \t\t\tint curr = next - i;\n \n \t\t\tif (curr < 0)\n \t\t\t\tbreak;\n+\t\t\tif (writer->selected[curr].pseudo_merge)\n+\t\t\t\tcontinue;\n \n \t\t\ttest_xor = ewah_pool_new();\n \t\t\tewah_xor(writer->selected[curr].bitmap, stored->bitmap, test_xor);\n@@ -205,6 +219,7 @@ static void compute_xor_offsets(struct bitmap_writer *writer)\n \t\t\t}\n \t\t}\n \n+next:\n \t\tstored->xor_offset = best_offset;\n \t\tstored->write_as = best_bitmap;\n \n@@ -217,7 +232,8 @@ struct bb_commit {\n \tstruct bitmap *commit_mask;\n \tstruct bitmap *bitmap;\n \tunsigned selected:1,\n-\t\t maximal:1;\n+\t\t maximal:1,\n+\t\t pseudo_merge:1;\n \tunsigned idx; /* within selected array */\n };\n \n@@ -255,17 +271,18 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \trevs.first_parent_only = 1;\n \n \tfor (i = 0; i < writer->selected_nr; i++) {\n-\t\tstruct commit *c = writer->selected[i].commit;\n-\t\tstruct bb_commit *ent = bb_data_at(&bb->data, c);\n+\t\tstruct bitmapped_commit *bc = &writer->selected[i];\n+\t\tstruct bb_commit *ent = bb_data_at(&bb->data, bc->commit);\n \n \t\tent->selected = 1;\n \t\tent->maximal = 1;\n+\t\tent->pseudo_merge = bc->pseudo_merge;\n \t\tent->idx = i;\n \n \t\tent->commit_mask = bitmap_new();\n \t\tbitmap_set(ent->commit_mask, i);\n \n-\t\tadd_pending_object(&revs, &c->object, \"\");\n+\t\tadd_pending_object(&revs, &bc->commit->object, \"\");\n \t}\n \n \tif (prepare_revision_walk(&revs))\n@@ -444,8 +461,13 @@ static int fill_bitmap_commit(struct bitmap_writer *writer,\n \t\tstruct commit *c = prio_queue_get(queue);\n \n \t\tif (old_bitmap && mapping) {\n-\t\t\tstruct ewah_bitmap *old = bitmap_for_commit(old_bitmap, c);\n+\t\t\tstruct ewah_bitmap *old;\n \t\t\tstruct bitmap *remapped = bitmap_new();\n+\n+\t\t\tif (commit->object.flags & BITMAP_PSEUDO_MERGE)\n+\t\t\t\told = NULL;\n+\t\t\telse\n+\t\t\t\told = bitmap_for_commit(old_bitmap, c);\n \t\t\t/*\n \t\t\t * If this commit has an old bitmap, then translate that\n \t\t\t * bitmap and add its bits to this one. No need to walk\n@@ -464,12 +486,14 @@ static int fill_bitmap_commit(struct bitmap_writer *writer,\n \t\t * Mark ourselves and queue our tree. The commit\n \t\t * walk ensures we cover all parents.\n \t\t */\n-\t\tpos = find_object_pos(writer, &c->object.oid, &found);\n-\t\tif (!found)\n-\t\t\treturn -1;\n-\t\tbitmap_set(ent->bitmap, pos);\n-\t\tprio_queue_put(tree_queue,\n-\t\t\t repo_get_commit_tree(the_repository, c));\n+\t\tif (!(c->object.flags & BITMAP_PSEUDO_MERGE)) {\n+\t\t\tpos = find_object_pos(writer, &c->object.oid, &found);\n+\t\t\tif (!found)\n+\t\t\t\treturn -1;\n+\t\t\tbitmap_set(ent->bitmap, pos);\n+\t\t\tprio_queue_put(tree_queue,\n+\t\t\t\t repo_get_commit_tree(the_repository, c));\n+\t\t}\n \n \t\tfor (p = c->parents; p; p = p->next) {\n \t\t\tpos = find_object_pos(writer, &p->item->object.oid,\n@@ -499,6 +523,9 @@ static void store_selected(struct bitmap_writer *writer,\n \n \tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n \n+\tif (ent->pseudo_merge)\n+\t\treturn;\n+\n \thash_pos = kh_get_oid_map(writer->bitmaps, commit->object.oid);\n \tif (hash_pos == kh_end(writer->bitmaps))\n \t\tdie(_(\"attempted to store non-selected commit: '%s'\"),\n@@ -631,7 +658,7 @@ void bitmap_writer_select_commits(struct bitmap_writer *writer,\n \n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n-\t\t\tpush_bitmapped_commit(writer, indexed_commits[i]);\n+\t\t\tpush_bitmapped_commit(writer, indexed_commits[i], 0);\n \t\treturn;\n \t}\n \n@@ -664,7 +691,7 @@ void bitmap_writer_select_commits(struct bitmap_writer *writer,\n \t\t\t}\n \t\t}\n \n-\t\tpush_bitmapped_commit(writer, chosen);\n+\t\tpush_bitmapped_commit(writer, chosen, 0);\n \n \t\ti += next + 1;\n \t\tdisplay_progress(writer->progress, i);\n@@ -701,8 +728,11 @@ static void write_selected_commits_v1(struct bitmap_writer *writer,\n {\n \tint i;\n \n-\tfor (i = 0; i < writer->selected_nr; ++i) {\n+\tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); ++i) {\n \t\tstruct bitmapped_commit *stored = &writer->selected[i];\n+\t\tif (stored->pseudo_merge)\n+\t\t\tBUG(\"unexpected pseudo-merge among selected: %s\",\n+\t\t\t oid_to_hex(&stored->commit->object.oid));\n \n \t\tif (offsets)\n \t\t\toffsets[i] = hashfile_total(f);\n@@ -735,10 +765,10 @@ static void write_lookup_table(struct bitmap_writer *writer, struct hashfile *f,\n \tuint32_t i;\n \tuint32_t *table, *table_inv;\n \n-\tALLOC_ARRAY(table, writer->selected_nr);\n-\tALLOC_ARRAY(table_inv, writer->selected_nr);\n+\tALLOC_ARRAY(table, bitmap_writer_nr_selected_commits(writer));\n+\tALLOC_ARRAY(table_inv, bitmap_writer_nr_selected_commits(writer));\n \n-\tfor (i = 0; i < writer->selected_nr; i++)\n+\tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++)\n \t\ttable[i] = i;\n \n \t/*\n@@ -746,16 +776,16 @@ static void write_lookup_table(struct bitmap_writer *writer, struct hashfile *f,\n \t * bitmap corresponds to j'th bitmapped commit (among the selected\n \t * commits) in lex order of OIDs.\n \t */\n-\tQSORT_S(table, writer->selected_nr, table_cmp, writer);\n+\tQSORT_S(table, bitmap_writer_nr_selected_commits(writer), table_cmp, writer);\n \n \t/* table_inv helps us discover that relationship (i'th bitmap\n \t * to j'th commit by j = table_inv[i])\n \t */\n-\tfor (i = 0; i < writer->selected_nr; i++)\n+\tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++)\n \t\ttable_inv[table[i]] = i;\n \n \ttrace2_region_enter(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n-\tfor (i = 0; i < writer->selected_nr; i++) {\n+\tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++) {\n \t\tstruct bitmapped_commit *selected = &writer->selected[table[i]];\n \t\tuint32_t xor_offset = selected->xor_offset;\n \t\tuint32_t xor_row;\n@@ -827,7 +857,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \tmemcpy(header.magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE));\n \theader.version = htons(default_version);\n \theader.options = htons(flags | options);\n-\theader.entry_count = htonl(writer->selected_nr);\n+\theader.entry_count = htonl(bitmap_writer_nr_selected_commits(writer));\n \thashcpy(header.checksum, writer->pack_checksum);\n \n \thashwrite(f, &header, sizeof(header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz);\n@@ -839,7 +869,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n \t\tCALLOC_ARRAY(offsets, index_nr);\n \n-\tfor (i = 0; i < writer->selected_nr; i++) {\n+\tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++) {\n \t\tstruct bitmapped_commit *stored = &writer->selected[i];\n \t\tint commit_pos = oid_pos(&stored->commit->object.oid, index,\n \t\t\t\t\t index_nr, oid_access);\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex f87e60153dd..6937a0f090f 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -21,6 +21,7 @@ struct bitmap_disk_header {\n \tunsigned char checksum[GIT_MAX_RAWSZ];\n };\n \n+#define BITMAP_PSEUDO_MERGE (1u<<21)\n #define NEEDS_BITMAP (1u<<22)\n \n /*\n@@ -109,6 +110,8 @@ struct bitmap_writer {\n \tstruct bitmapped_commit *selected;\n \tunsigned int selected_nr, selected_alloc;\n \n+\tuint32_t pseudo_merges_nr;\n+\n \tstruct progress *progress;\n \tint show_progress;\n \tunsigned char pack_checksum[GIT_MAX_RAWSZ];\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495192","messageId":"99d2b6872baf5702aca74429c7c31b6fab5b1ec0.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 14/30] pack-bitmap: implement `bitmap_writer_has_bitmapped_object_id()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:22Z","receivedAt":"2024-05-21T19:02:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to implement pseudo-merge bitmap selection by implementing a\nnecessary new function, `bitmap_writer_has_bitmapped_object_id()`.\n\nThis function returns whether or not the bitmap_writer selected the\ngiven object ID for bitmapping. This will allow the pseudo-merge\nmachinery to reject candidates for pseudo-merges if they have already\nbeen selected as an ordinary bitmap tip.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 6 ++++++\n pack-bitmap.h | 2 ++\n 2 files changed, 8 insertions(+)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 60eb1e71c98..299aa8af6f5 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -130,6 +130,12 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n \t}\n }\n \n+int bitmap_writer_has_bitmapped_object_id(struct bitmap_writer *writer,\n+\t\t\t\t\t const struct object_id *oid)\n+{\n+\treturn kh_get_oid_map(writer->bitmaps, *oid) != kh_end(writer->bitmaps);\n+}\n+\n /**\n * Compute the actual bitmaps\n */\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 6937a0f090f..e175f28e0de 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -125,6 +125,8 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n \t\t\t\t struct packing_data *to_pack,\n \t\t\t\t struct pack_idx_entry **index,\n \t\t\t\t uint32_t index_nr);\n+int bitmap_writer_has_bitmapped_object_id(struct bitmap_writer *writer,\n+\t\t\t\t\t const struct object_id *oid);\n uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \t\t\t\tstruct packing_data *mapping);\n int rebuild_bitmap(const uint32_t *reposition,\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495193","messageId":"e7209c60fa546230bacd169cd28710651b19c39d.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 15/30] pack-bitmap: make `bitmap_writer_push_bitmapped_commit()` public","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:26Z","receivedAt":"2024-05-21T19:02:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pseudo-merge selection code will be added in a subsequent commit,\nand will need a way to push the allocated commit structures into the\nbitmap writer from a separate compilation unit.\n\nMake the `bitmap_writer_push_bitmapped_commit()` function part of the\npack-bitmap.h header in order to make this possible.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 9 ++++-----\n pack-bitmap.h | 2 ++\n 2 files changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 299aa8af6f5..bc19b33ad16 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -140,9 +140,8 @@ int bitmap_writer_has_bitmapped_object_id(struct bitmap_writer *writer,\n * Compute the actual bitmaps\n */\n \n-static inline void push_bitmapped_commit(struct bitmap_writer *writer,\n-\t\t\t\t\t struct commit *commit,\n-\t\t\t\t\t unsigned pseudo_merge)\n+void bitmap_writer_push_commit(struct bitmap_writer *writer,\n+\t\t\t struct commit *commit, unsigned pseudo_merge)\n {\n \tif (writer->selected_nr >= writer->selected_alloc) {\n \t\twriter->selected_alloc = (writer->selected_alloc + 32) * 2;\n@@ -664,7 +663,7 @@ void bitmap_writer_select_commits(struct bitmap_writer *writer,\n \n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n-\t\t\tpush_bitmapped_commit(writer, indexed_commits[i], 0);\n+\t\t\tbitmap_writer_push_commit(writer, indexed_commits[i], 0);\n \t\treturn;\n \t}\n \n@@ -697,7 +696,7 @@ void bitmap_writer_select_commits(struct bitmap_writer *writer,\n \t\t\t}\n \t\t}\n \n-\t\tpush_bitmapped_commit(writer, chosen, 0);\n+\t\tbitmap_writer_push_commit(writer, chosen, 0);\n \n \t\ti += next + 1;\n \t\tdisplay_progress(writer->progress, i);\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex e175f28e0de..a7e2f56c971 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -127,6 +127,8 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n \t\t\t\t uint32_t index_nr);\n int bitmap_writer_has_bitmapped_object_id(struct bitmap_writer *writer,\n \t\t\t\t\t const struct object_id *oid);\n+void bitmap_writer_push_commit(struct bitmap_writer *writer,\n+\t\t\t struct commit *commit, unsigned pseudo_merge);\n uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \t\t\t\tstruct packing_data *mapping);\n int rebuild_bitmap(const uint32_t *reposition,\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495194","messageId":"3070135eb4b9bd16117e82f1817c112c56a24b55.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 16/30] config: introduce git_config_float()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:29Z","receivedAt":"2024-05-21T19:02:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Future commits will want to parse a floating point value from\nconfiguration, but we have no way to parse such a value prior to this\npatch.\n\nThe core of the routine is implemented in git_parse_float(). Unlike\ngit_parse_unsigned() and git_parse_signed(), however, the function\nimplemented here only works on type \"float\", and not related types like\n\"double\", or \"long double\".\n\nThis is because \"double\" and \"long double\" use different functions to\nconvert from ASCII strings to floating point values (strtod() and\nstrtold(), respectively). Likewise, there is no pointer type that can\nassign to any of these values (except for \"void *\"), so the only way to\ndefine this trio of functions would be with a macro expansion that is\nparameterized over the floating point type and conversion function.\n\nThat is all doable, but likely to be overkill given our current needs,\nwhich is only to parse floats.\n\nSigned-off-by: Taylor Blau \n---\n config.c | 9 +++++++++\n config.h | 6 ++++++\n parse.c | 29 +++++++++++++++++++++++++++++\n parse.h | 1 +\n 4 files changed, 45 insertions(+)\n\ndiff --git a/config.c b/config.c\nindex 77a0fd2d80e..ee681fda34b 100644\n--- a/config.c\n+++ b/config.c\n@@ -1243,6 +1243,15 @@ ssize_t git_config_ssize_t(const char *name, const char *value,\n \treturn ret;\n }\n \n+float git_config_float(const char *name, const char *value,\n+\t\t const struct key_value_info *kvi)\n+{\n+\tfloat ret;\n+\tif (!git_parse_float(value, &ret))\n+\t\tdie_bad_number(name, value, kvi);\n+\treturn ret;\n+}\n+\n static const struct fsync_component_name {\n \tconst char *name;\n \tenum fsync_component component_bits;\ndiff --git a/config.h b/config.h\nindex f4966e37494..b0d1baba95a 100644\n--- a/config.h\n+++ b/config.h\n@@ -261,6 +261,12 @@ unsigned long git_config_ulong(const char *, const char *,\n ssize_t git_config_ssize_t(const char *, const char *,\n \t\t\t const struct key_value_info *);\n \n+/**\n+ * Identical to `git_config_int`, but for floating point values.\n+ */\n+float git_config_float(const char *, const char *,\n+\t\t const struct key_value_info *);\n+\n /**\n * Same as `git_config_bool`, except that integers are returned as-is, and\n * an `is_bool` flag is unset.\ndiff --git a/parse.c b/parse.c\nindex 42d691a0fbb..a5967e80910 100644\n--- a/parse.c\n+++ b/parse.c\n@@ -125,6 +125,35 @@ int git_parse_ssize_t(const char *value, ssize_t *ret)\n \treturn 1;\n }\n \n+int git_parse_float(const char *value, float *ret)\n+{\n+\tchar *end;\n+\tfloat val;\n+\tuintmax_t factor;\n+\n+\tif (!value || !*value) {\n+\t\terrno = EINVAL;\n+\t\treturn 0;\n+\t}\n+\n+\terrno = 0;\n+\tval = strtof(value, &end);\n+\tif (errno == ERANGE)\n+\t\treturn 0;\n+\tif (end == value) {\n+\t\terrno = EINVAL;\n+\t\treturn 0;\n+\t}\n+\tfactor = get_unit_factor(end);\n+\tif (!factor) {\n+\t\terrno = EINVAL;\n+\t\treturn 0;\n+\t}\n+\tval *= factor;\n+\t*ret = val;\n+\treturn 1;\n+}\n+\n int git_parse_maybe_bool_text(const char *value)\n {\n \tif (!value)\ndiff --git a/parse.h b/parse.h\nindex 07d2193d698..7df82c5f5b8 100644\n--- a/parse.h\n+++ b/parse.h\n@@ -6,6 +6,7 @@ int git_parse_ssize_t(const char *, ssize_t *);\n int git_parse_ulong(const char *, unsigned long *);\n int git_parse_int(const char *value, int *ret);\n int git_parse_int64(const char *value, int64_t *ret);\n+int git_parse_float(const char *value, float *ret);\n \n /**\n * Same as `git_config_bool`, except that it returns -1 on error rather\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495195","messageId":"3029473c094b3edf51828b7a1d1acfc8e959ece6.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 17/30] pseudo-merge: implement support for selecting pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:32Z","receivedAt":"2024-05-21T19:02:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Teach the new pseudo-merge machinery how to select non-bitmapped commits\nfor inclusion in different pseudo-merge group(s) based on a handful of\ncriteria.\n\nNote that the selected pseudo-merge commits aren't actually used or\nwritten anywhere yet. This will be done in the following commit.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config.txt | 2 +\n Documentation/config/bitmap-pseudo-merge.txt | 90 ++++\n Documentation/gitpacking.txt | 83 ++++\n pack-bitmap-write.c | 21 +\n pack-bitmap.h | 2 +\n pseudo-merge.c | 454 +++++++++++++++++++\n pseudo-merge.h | 94 ++++\n 7 files changed, 746 insertions(+)\n create mode 100644 Documentation/config/bitmap-pseudo-merge.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 6f649c997c0..caa34311214 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -384,6 +384,8 @@ include::config/apply.txt[]\n \n include::config/attr.txt[]\n \n+include::config/bitmap-pseudo-merge.txt[]\n+\n include::config/blame.txt[]\n \n include::config/branch.txt[]\ndiff --git a/Documentation/config/bitmap-pseudo-merge.txt b/Documentation/config/bitmap-pseudo-merge.txt\nnew file mode 100644\nindex 00000000000..d4a2023b84a\n--- /dev/null\n+++ b/Documentation/config/bitmap-pseudo-merge.txt\n@@ -0,0 +1,90 @@\n+NOTE: The configuration options in `bitmapPseudoMerge.*` are considered\n+EXPERIMENTAL and may be subject to change or be removed entirely in the\n+future.\n+\n+bitmapPseudoMerge..pattern::\n+\tRegular expression used to match reference names. Commits\n+\tpointed to by references matching this pattern (and meeting\n+\tthe below criteria, like `bitmapPseudoMerge..sampleRate`\n+\tand `bitmapPseudoMerge..threshold`) will be considered\n+\tfor inclusion in a pseudo-merge bitmap.\n++\n+Commits are grouped into pseudo-merge groups based on whether or not\n+any reference(s) that point at a given commit match the pattern, which\n+is an extended regular expression.\n++\n+Within a pseudo-merge group, commits may be further grouped into\n+sub-groups based on the capture groups in the pattern. These\n+sub-groupings are formed from the regular expressions by concatenating\n+any capture groups from the regular expression, with a '-' dash in\n+between.\n++\n+For example, if the pattern is `refs/tags/`, then all tags (provided\n+they meet the below criteria) will be considered candidates for the\n+same pseudo-merge group. However, if the pattern is instead\n+`refs/remotes/([0-9])+/tags/`, then tags from different remotes will\n+be grouped into separate pseudo-merge groups, based on the remote\n+number.\n+\n+bitmapPseudoMerge..decay::\n+\tDetermines the rate at which consecutive pseudo-merge bitmap\n+\tgroups decrease in size. Must be non-negative. This parameter\n+\tcan be thought of as `k` in the function `f(n) = C * n^-k`,\n+\twhere `f(n)` is the size of the `n`th group.\n++\n+Setting the decay rate equal to `0` will cause all groups to be the\n+same size. Setting the decay rate equal to `1` will cause the `n`th\n+group to be `1/n` the size of the initial group. Higher values of the\n+decay rate cause consecutive groups to shrink at an increasing rate.\n+The default is `1`.\n++\n+If all groups are the same size, it is possible that groups containing\n+newer commits will be able to be used less often than earlier groups,\n+since it is more likely that the references pointing at newer commits\n+will be updated more often than a reference pointing at an old commit.\n+\n+bitmapPseudoMerge..sampleRate::\n+\tDetermines the proportion of non-bitmapped commits (among\n+\treference tips) which are selected for inclusion in an\n+\tunstable pseudo-merge bitmap. Must be between `0` and `1`\n+\t(inclusive). The default is `1`.\n+\n+bitmapPseudoMerge..threshold::\n+\tDetermines the minimum age of non-bitmapped commits (among\n+\treference tips, as above) which are candidates for inclusion\n+\tin an unstable pseudo-merge bitmap. The default is\n+\t`1.week.ago`.\n+\n+bitmapPseudoMerge..maxMerges::\n+\tDetermines the maximum number of pseudo-merge commits among\n+\twhich commits may be distributed.\n++\n+For pseudo-merge groups whose pattern does not contain any capture\n+groups, this setting is applied for all commits matching the regular\n+expression. For patterns that have one or more capture groups, this\n+setting is applied for each distinct capture group.\n++\n+For example, if your capture group is `refs/tags/`, then this setting\n+will distribute all tags into a maximum of `maxMerges` pseudo-merge\n+commits. However, if your capture group is, say,\n+`refs/remotes/([0-9]+)/tags/`, then this setting will be applied to\n+each remote's set of tags individually.\n++\n+Must be non-negative. The default value is 64.\n+\n+bitmapPseudoMerge..stableThreshold::\n+\tDetermines the minimum age of commits (among reference tips,\n+\tas above, however stable commits are still considered\n+\tcandidates even when they have been covered by a bitmap) which\n+\tare candidates for a stable a pseudo-merge bitmap. The default\n+\tis `1.month.ago`.\n++\n+Setting this threshold to a smaller value (e.g., 1.week.ago) will cause\n+more stable groups to be generated (which impose a one-time generation\n+cost) but those groups will likely become stale over time. Using a\n+larger value incurs the opposite penalty (fewer stable groups which are\n+more useful).\n+\n+bitmapPseudoMerge..stableSize::\n+\tDetermines the size (in number of commits) of a stable\n+\tpsuedo-merge bitmap. The default is `512`.\ndiff --git a/Documentation/gitpacking.txt b/Documentation/gitpacking.txt\nindex ff18077129b..1ed645ff910 100644\n--- a/Documentation/gitpacking.txt\n+++ b/Documentation/gitpacking.txt\n@@ -93,6 +93,89 @@ can take advantage of the fact that we only care about the union of\n objects reachable from all of those tags, and answer the query much\n faster.\n \n+=== Configuration\n+\n+Reference tips are grouped into different pseudo-merge groups according\n+to two criteria. A reference name matches one or more of the defined\n+pseudo-merge patterns, and optionally one or more capture groups within\n+that pattern which further partition the group.\n+\n+Within a group, commits may be considered \"stable\", or \"unstable\"\n+depending on their age. These are adjusted by setting the\n+`bitmapPseudoMerge..stableThreshold` and\n+`bitmapPseudoMerge..threshold` configuration values, respectively.\n+\n+All stable commits are grouped into pseudo-merges of equal size\n+(`bitmapPseudoMerge..stableSize`). If the `stableSize`\n+configuration is set to, say, 100, then the first 100 commits (ordered\n+by committer date) which are older than the `stableThreshold` value will\n+form one group, the next 100 commits will form another group, and so on.\n+\n+Among unstable commits, the pseudo-merge machinery will attempt to\n+combine older commits into large groups as opposed to newer commits\n+which will appear in smaller groups. This is based on the heuristic that\n+references whose tip commit is older are less likely to be modified to\n+point at a different commit than a reference whose tip commit is newer.\n+\n+The size of groups is determined by a power-law decay function, and the\n+decay parameter roughly corresponds to \"k\" in `f(n) = C*n^(-k/100)`,\n+where `f(n)` describes the size of the `n`-th pseudo-merge group. The\n+sample rate controls what percentage of eligible commits are considered\n+as candidates. The threshold parameter indicates the minimum age (so as\n+to avoid including too-recent commits in a pseudo-merge group, making it\n+less likely to be valid). The \"maxMerges\" parameter sets an upper-bound\n+on the number of pseudo-merge commits an individual group\n+\n+The \"stable\"-related parameters control \"stable\" pseudo-merge groups,\n+comprised of a fixed number of commits which are older than the\n+configured \"stable threshold\" value and may be grouped together in\n+chunks of \"stableSize\" in order of age.\n+\n+The exact configuration for pseudo-merges is as follows:\n+\n+include::config/bitmap-pseudo-merge.txt[]\n+\n+=== Examples\n+\n+Suppose that you have a repository with a large number of references,\n+and you want a bare-bones configuration of pseudo-merge bitmaps that\n+will enhance bitmap coverage of the `refs/` namespace. You may start\n+wiht a configuration like so:\n+\n+ [bitmapPseudoMerge \"all\"]\n+\tpattern = \"refs/\"\n+\tthreshold = now\n+\tstableThreshold = never\n+\tsampleRate = 100\n+\tmaxMerges = 64\n+\n+This will create pseudo-merge bitmaps for all references, regardless of\n+their age, and group them into 64 pseudo-merge commits.\n+\n+If you wanted to separate tags from branches when generating\n+pseudo-merge commits, you would instead define the pattern with a\n+capture group, like so:\n+\n+ [bitmapPseudoMerge \"all\"]\n+\tpattern = \"refs/(heads/tags)/\"\n+\n+Suppose instead that you are working in a fork-network repository, with\n+each fork specified by some numeric ID, and whose refs reside in\n+`refs/virtual/NNN/` (where `NNN` is the numeric ID corresponding to some\n+fork) in the network. In this instance, you may instead write something\n+like:\n+\n+ [bitmapPseudoMerge \"all\"]\n+\tpattern = \"refs/virtual/([0-9]+)/(heads|tags)/\"\n+\tthreshold = now\n+\tstableThreshold = never\n+\tsampleRate = 100\n+\tmaxMerges = 64\n+\n+Which would generate pseudo-merge group identifiers like \"1234-heads\",\n+and \"5678-tags\" (for branches in fork \"1234\", and tags in remote \"5678\",\n+respectively).\n+\n SEE ALSO\n --------\n linkgit:git-pack-objects[1]\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex bc19b33ad16..d5884ea5e9c 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -17,6 +17,7 @@\n #include \"trace2.h\"\n #include \"tree.h\"\n #include \"tree-walk.h\"\n+#include \"pseudo-merge.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -39,11 +40,25 @@ void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r)\n \tif (writer->bitmaps)\n \t\tBUG(\"bitmap writer already initialized\");\n \twriter->bitmaps = kh_init_oid_map();\n+\twriter->pseudo_merge_commits = kh_init_oid_map();\n+\n+\tstring_list_init_dup(&writer->pseudo_merge_groups);\n+\n+\tload_pseudo_merges_from_config(&writer->pseudo_merge_groups);\n+}\n+\n+static void free_pseudo_merge_commit_idx(struct pseudo_merge_commit_idx *idx)\n+{\n+\tif (!idx)\n+\t\treturn;\n+\tfree(idx->pseudo_merge);\n+\tfree(idx);\n }\n \n void bitmap_writer_free(struct bitmap_writer *writer)\n {\n \tuint32_t i;\n+\tstruct pseudo_merge_commit_idx *idx;\n \n \tif (!writer)\n \t\treturn;\n@@ -55,6 +70,10 @@ void bitmap_writer_free(struct bitmap_writer *writer)\n \n \tkh_destroy_oid_map(writer->bitmaps);\n \n+\tkh_foreach_value(writer->pseudo_merge_commits, idx,\n+\t\t\t free_pseudo_merge_commit_idx(idx));\n+\tkh_destroy_oid_map(writer->pseudo_merge_commits);\n+\n \tfor (i = 0; i < writer->selected_nr; i++) {\n \t\tstruct bitmapped_commit *bc = &writer->selected[i];\n \t\tif (bc->write_as != bc->bitmap)\n@@ -703,6 +722,8 @@ void bitmap_writer_select_commits(struct bitmap_writer *writer,\n \t}\n \n \tstop_progress(&writer->progress);\n+\n+\tselect_pseudo_merges(writer, indexed_commits, indexed_commits_nr);\n }\n \n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex a7e2f56c971..1e730ea1e54 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -110,6 +110,8 @@ struct bitmap_writer {\n \tstruct bitmapped_commit *selected;\n \tunsigned int selected_nr, selected_alloc;\n \n+\tstruct string_list pseudo_merge_groups;\n+\tkh_oid_map_t *pseudo_merge_commits; /* oid -> pseudo merge(s) */\n \tuint32_t pseudo_merges_nr;\n \n \tstruct progress *progress;\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex 37e037ba272..4be730563eb 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -1,2 +1,456 @@\n #include \"git-compat-util.h\"\n #include \"pseudo-merge.h\"\n+#include \"date.h\"\n+#include \"oid-array.h\"\n+#include \"strbuf.h\"\n+#include \"config.h\"\n+#include \"string-list.h\"\n+#include \"refs.h\"\n+#include \"pack-bitmap.h\"\n+#include \"commit.h\"\n+#include \"alloc.h\"\n+#include \"progress.h\"\n+\n+#define DEFAULT_PSEUDO_MERGE_DECAY 1.0f\n+#define DEFAULT_PSEUDO_MERGE_MAX_MERGES 64\n+#define DEFAULT_PSEUDO_MERGE_SAMPLE_RATE 1\n+#define DEFAULT_PSEUDO_MERGE_THRESHOLD approxidate(\"1.week.ago\")\n+#define DEFAULT_PSEUDO_MERGE_STABLE_THRESHOLD approxidate(\"1.month.ago\")\n+#define DEFAULT_PSEUDO_MERGE_STABLE_SIZE 512\n+\n+static float gitexp(float base, int exp)\n+{\n+\tfloat result = 1;\n+\twhile (1) {\n+\t\tif (exp % 2)\n+\t\t\tresult *= base;\n+\t\texp >>= 1;\n+\t\tif (!exp)\n+\t\t\tbreak;\n+\t\tbase *= base;\n+\t}\n+\treturn result;\n+}\n+\n+static uint32_t pseudo_merge_group_size(const struct pseudo_merge_group *group,\n+\t\t\t\t\tconst struct pseudo_merge_matches *matches,\n+\t\t\t\t\tuint32_t i)\n+{\n+\tfloat C = 0.0f;\n+\tuint32_t n;\n+\n+\t/*\n+\t * The size of pseudo-merge groups decays according to a power series,\n+\t * which looks like:\n+\t *\n+\t * f(n) = C * n^-k\n+\t *\n+\t * , where 'n' is the n-th pseudo-merge group, 'f(n)' is its size, 'k'\n+\t * is the decay rate, and 'C' is a scaling value.\n+\t *\n+\t * The value of C depends on the number of groups, decay rate, and total\n+\t * number of commits. It is computed such that if there are M and N\n+\t * total groups and commits, respectively, that:\n+\t *\n+\t * N = f(0) + f(1) + ... f(M-1)\n+\t *\n+\t * Rearranging to isolate C, we get:\n+\t *\n+\t * N = \\sum_{n=1}^M C / n^k\n+\t *\n+\t * N / C = \\sum_{n=1}^M n^-k\n+\t *\n+\t * C = N / \\sum_{n=1}^M n^-k\n+\t *\n+\t * For example, if we have a decay rate of 'k' being equal to 1.5, 'N'\n+\t * total commits equal to 10,000, and 'M' being equal to 6 groups, then\n+\t * the (rounded) group sizes are:\n+\t *\n+\t * { 5469, 1934, 1053, 684, 489, 372 }\n+\t *\n+\t * increasing the number of total groups, say to 10, scales the group\n+\t * sizes appropriately:\n+\t *\n+\t * { 5012, 1772, 964, 626, 448, 341, 271, 221, 186, 158 }\n+\t */\n+\tfor (n = 0; n < group->max_merges; n++)\n+\t\tC += 1.0f / gitexp(n + 1, group->decay);\n+\tC = matches->unstable_nr / C;\n+\n+\treturn (uint32_t)((C / gitexp(i + 1, group->decay)) + 0.5);\n+}\n+\n+static void pseudo_merge_group_init(struct pseudo_merge_group *group)\n+{\n+\tmemset(group, 0, sizeof(struct pseudo_merge_group));\n+\n+\tstrmap_init_with_options(&group->matches, NULL, 0);\n+\n+\tgroup->decay = DEFAULT_PSEUDO_MERGE_DECAY;\n+\tgroup->max_merges = DEFAULT_PSEUDO_MERGE_MAX_MERGES;\n+\tgroup->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE;\n+\tgroup->threshold = DEFAULT_PSEUDO_MERGE_THRESHOLD;\n+\tgroup->stable_threshold = DEFAULT_PSEUDO_MERGE_STABLE_THRESHOLD;\n+\tgroup->stable_size = DEFAULT_PSEUDO_MERGE_STABLE_SIZE;\n+}\n+\n+static int pseudo_merge_config(const char *var, const char *value,\n+\t\t\t const struct config_context *ctx,\n+\t\t\t void *cb_data)\n+{\n+\tstruct string_list *list = cb_data;\n+\tstruct string_list_item *item;\n+\tstruct pseudo_merge_group *group;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *sub, *key;\n+\tsize_t sub_len;\n+\tint ret = 0;\n+\n+\tif (parse_config_key(var, \"bitmappseudomerge\", &sub, &sub_len, &key))\n+\t\tgoto done;\n+\n+\tif (!sub_len)\n+\t\tgoto done;\n+\n+\tstrbuf_add(&buf, sub, sub_len);\n+\n+\titem = string_list_lookup(list, buf.buf);\n+\tif (!item) {\n+\t\titem = string_list_insert(list, buf.buf);\n+\n+\t\titem->util = xmalloc(sizeof(struct pseudo_merge_group));\n+\t\tpseudo_merge_group_init(item->util);\n+\t}\n+\n+\tgroup = item->util;\n+\n+\tif (!strcmp(key, \"pattern\")) {\n+\t\tstruct strbuf re = STRBUF_INIT;\n+\n+\t\tfree(group->pattern);\n+\t\tif (*value != '^')\n+\t\t\tstrbuf_addch(&re, '^');\n+\t\tstrbuf_addstr(&re, value);\n+\n+\t\tgroup->pattern = xcalloc(1, sizeof(regex_t));\n+\t\tif (regcomp(group->pattern, re.buf, REG_EXTENDED))\n+\t\t\tdie(_(\"failed to load pseudo-merge regex for %s: '%s'\"),\n+\t\t\t sub, re.buf);\n+\n+\t\tstrbuf_release(&re);\n+\t} else if (!strcmp(key, \"decay\")) {\n+\t\tgroup->decay = git_config_float(var, value, ctx->kvi);\n+\t\tif (group->decay < 0) {\n+\t\t\twarning(_(\"%s must be non-negative, using default\"), var);\n+\t\t\tgroup->decay = DEFAULT_PSEUDO_MERGE_DECAY;\n+\t\t}\n+\t} else if (!strcmp(key, \"samplerate\")) {\n+\t\tgroup->sample_rate = git_config_float(var, value, ctx->kvi);\n+\t\tif (!(0 <= group->sample_rate && group->sample_rate <= 1)) {\n+\t\t\twarning(_(\"%s must be between 0 and 1, using default\"), var);\n+\t\t\tgroup->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE;\n+\t\t}\n+\t} else if (!strcmp(key, \"threshold\")) {\n+\t\tif (git_config_expiry_date(&group->threshold, var, value)) {\n+\t\t\tret = -1;\n+\t\t\tgoto done;\n+\t\t}\n+\t} else if (!strcmp(key, \"maxmerges\")) {\n+\t\tgroup->max_merges = git_config_int(var, value, ctx->kvi);\n+\t\tif (group->max_merges < 0) {\n+\t\t\twarning(_(\"%s must be non-negative, using default\"), var);\n+\t\t\tgroup->max_merges = DEFAULT_PSEUDO_MERGE_MAX_MERGES;\n+\t\t}\n+\t} else if (!strcmp(key, \"stablethreshold\")) {\n+\t\tif (git_config_expiry_date(&group->stable_threshold, var, value)) {\n+\t\t\tret = -1;\n+\t\t\tgoto done;\n+\t\t}\n+\t} else if (!strcmp(key, \"stablesize\")) {\n+\t\tgroup->stable_size = git_config_int(var, value, ctx->kvi);\n+\t\tif (group->stable_size <= 0) {\n+\t\t\twarning(_(\"%s must be positive, using default\"), var);\n+\t\t\tgroup->stable_size = DEFAULT_PSEUDO_MERGE_STABLE_SIZE;\n+\t\t}\n+\t}\n+\n+done:\n+\tstrbuf_release(&buf);\n+\n+\treturn ret;\n+}\n+\n+void load_pseudo_merges_from_config(struct string_list *list)\n+{\n+\tstruct string_list_item *item;\n+\n+\tgit_config(pseudo_merge_config, list);\n+\n+\tfor_each_string_list_item(item, list) {\n+\t\tstruct pseudo_merge_group *group = item->util;\n+\t\tif (!group->pattern)\n+\t\t\tdie(_(\"pseudo-merge group '%s' missing required pattern\"),\n+\t\t\t item->string);\n+\t\tif (group->threshold < group->stable_threshold)\n+\t\t\tdie(_(\"pseudo-merge group '%s' has unstable threshold \"\n+\t\t\t \"before stable one\"), item->string);\n+\t}\n+}\n+\n+static int find_pseudo_merge_group_for_ref(const char *refname,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t int flags UNUSED,\n+\t\t\t\t\t void *_data)\n+{\n+\tstruct bitmap_writer *writer = _data;\n+\tstruct object_id peeled;\n+\tstruct commit *c;\n+\tuint32_t i;\n+\tint has_bitmap;\n+\n+\tif (!peel_iterated_oid(oid, &peeled))\n+\t\toid = &peeled;\n+\n+\tc = lookup_commit(the_repository, oid);\n+\tif (!c)\n+\t\treturn 0;\n+\n+\thas_bitmap = bitmap_writer_has_bitmapped_object_id(writer, oid);\n+\n+\tfor (i = 0; i < writer->pseudo_merge_groups.nr; i++) {\n+\t\tstruct pseudo_merge_group *group;\n+\t\tstruct pseudo_merge_matches *matches;\n+\t\tstruct strbuf group_name = STRBUF_INIT;\n+\t\tregmatch_t captures[16];\n+\t\tsize_t j;\n+\n+\t\tgroup = writer->pseudo_merge_groups.items[i].util;\n+\t\tif (regexec(group->pattern, refname, ARRAY_SIZE(captures),\n+\t\t\t captures, 0))\n+\t\t\tcontinue;\n+\n+\t\tif (captures[ARRAY_SIZE(captures) - 1].rm_so != -1)\n+\t\t\twarning(_(\"pseudo-merge regex from config has too many capture \"\n+\t\t\t\t \"groups (max=%\"PRIuMAX\")\"),\n+\t\t\t\t(uintmax_t)ARRAY_SIZE(captures) - 2);\n+\n+\t\tfor (j = !!group->pattern->re_nsub; j < ARRAY_SIZE(captures); j++) {\n+\t\t\tregmatch_t *match = &captures[j];\n+\t\t\tif (match->rm_so == -1)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (group_name.len)\n+\t\t\t\tstrbuf_addch(&group_name, '-');\n+\n+\t\t\tstrbuf_add(&group_name, refname + match->rm_so,\n+\t\t\t\t match->rm_eo - match->rm_so);\n+\t\t}\n+\n+\t\tmatches = strmap_get(&group->matches, group_name.buf);\n+\t\tif (!matches) {\n+\t\t\tmatches = xcalloc(1, sizeof(*matches));\n+\t\t\tstrmap_put(&group->matches, strbuf_detach(&group_name, NULL),\n+\t\t\t\t matches);\n+\t\t}\n+\n+\t\tif (c->date <= group->stable_threshold) {\n+\t\t\tALLOC_GROW(matches->stable, matches->stable_nr + 1,\n+\t\t\t\t matches->stable_alloc);\n+\t\t\tmatches->stable[matches->stable_nr++] = c;\n+\t\t} else if (c->date <= group->threshold && !has_bitmap) {\n+\t\t\tALLOC_GROW(matches->unstable, matches->unstable_nr + 1,\n+\t\t\t\t matches->unstable_alloc);\n+\t\t\tmatches->unstable[matches->unstable_nr++] = c;\n+\t\t}\n+\n+\t\tstrbuf_release(&group_name);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static struct commit *push_pseudo_merge(struct pseudo_merge_group *group)\n+{\n+\tstruct commit *merge;\n+\n+\tALLOC_GROW(group->merges, group->merges_nr + 1, group->merges_alloc);\n+\n+\tmerge = alloc_commit_node(the_repository);\n+\tmerge->object.parsed = 1;\n+\tmerge->object.flags |= BITMAP_PSEUDO_MERGE;\n+\n+\tgroup->merges[group->merges_nr++] = merge;\n+\n+\treturn merge;\n+}\n+\n+static struct pseudo_merge_commit_idx *pseudo_merge_idx(kh_oid_map_t *pseudo_merge_commits,\n+\t\t\t\t\t\t\tconst struct object_id *oid)\n+\n+{\n+\tstruct pseudo_merge_commit_idx *pmc;\n+\tint hash_ret;\n+\tkhiter_t hash_pos = kh_put_oid_map(pseudo_merge_commits, *oid,\n+\t\t\t\t\t &hash_ret);\n+\n+\tif (hash_ret) {\n+\t\tCALLOC_ARRAY(pmc, 1);\n+\t\tkh_value(pseudo_merge_commits, hash_pos) = pmc;\n+\t} else {\n+\t\tpmc = kh_value(pseudo_merge_commits, hash_pos);\n+\t}\n+\n+\treturn pmc;\n+}\n+\n+#define MIN_PSEUDO_MERGE_SIZE 8\n+\n+static void select_pseudo_merges_1(struct bitmap_writer *writer,\n+\t\t\t\t struct pseudo_merge_group *group,\n+\t\t\t\t struct pseudo_merge_matches *matches)\n+{\n+\tuint32_t i, j;\n+\tuint32_t stable_merges_nr;\n+\n+\tif (!matches->stable_nr && !matches->unstable_nr)\n+\t\treturn; /* all tips in this group already have bitmaps */\n+\n+\tstable_merges_nr = matches->stable_nr / group->stable_size;\n+\tif (matches->stable_nr % group->stable_size)\n+\t\tstable_merges_nr++;\n+\n+\t/* make stable_merges_nr pseudo merges for stable commits */\n+\tfor (i = 0, j = 0; i < stable_merges_nr; i++) {\n+\t\tstruct commit *merge;\n+\t\tstruct commit_list **p;\n+\n+\t\tmerge = push_pseudo_merge(group);\n+\t\tp = &merge->parents;\n+\n+\t\t/*\n+\t\t * For each pseudo-merge created above, add parents to the\n+\t\t * allocated commit node from the stable set of commits\n+\t\t * (un-bitmapped, newer than the stable threshold).\n+\t\t */\n+\t\tdo {\n+\t\t\tstruct commit *c;\n+\t\t\tstruct pseudo_merge_commit_idx *pmc;\n+\n+\t\t\tif (j >= matches->stable_nr)\n+\t\t\t\tbreak;\n+\n+\t\t\tc = matches->stable[j++];\n+\t\t\t/*\n+\t\t\t * Here and below, make sure that we keep our mapping of\n+\t\t\t * commits -> pseudo-merge(s) which include the key'd\n+\t\t\t * commit up-to-date.\n+\t\t\t */\n+\t\t\tpmc = pseudo_merge_idx(writer->pseudo_merge_commits,\n+\t\t\t\t\t &c->object.oid);\n+\n+\t\t\tALLOC_GROW(pmc->pseudo_merge, pmc->nr + 1, pmc->alloc);\n+\n+\t\t\tpmc->pseudo_merge[pmc->nr++] = writer->pseudo_merges_nr;\n+\t\t\tp = commit_list_append(c, p);\n+\t\t} while (j % group->stable_size);\n+\n+\t\tbitmap_writer_push_commit(writer, merge, 1);\n+\t\twriter->pseudo_merges_nr++;\n+\t}\n+\n+\t/* make up to group->max_merges pseudo merges for unstable commits */\n+\tfor (i = 0, j = 0; i < group->max_merges; i++) {\n+\t\tstruct commit *merge;\n+\t\tstruct commit_list **p;\n+\t\tuint32_t size, end;\n+\n+\t\tmerge = push_pseudo_merge(group);\n+\t\tp = &merge->parents;\n+\n+\t\tsize = pseudo_merge_group_size(group, matches, i);\n+\t\tend = size < MIN_PSEUDO_MERGE_SIZE ? matches->unstable_nr : j + size;\n+\n+\t\t/*\n+\t\t * For each pseudo-merge commit created above, add parents to\n+\t\t * the allocated commit node from the unstable set of commits\n+\t\t * (newer than the stable threshold).\n+\t\t *\n+\t\t * Account for the sample rate, since not every candidate from\n+\t\t * the set of stable commits will be included as a pseudo-merge\n+\t\t * parent.\n+\t\t */\n+\t\tfor (; j < end && j < matches->unstable_nr; j++) {\n+\t\t\tstruct commit *c = matches->unstable[j];\n+\t\t\tstruct pseudo_merge_commit_idx *pmc;\n+\n+\t\t\tif (j % (uint32_t)(1.0f / group->sample_rate))\n+\t\t\t\tcontinue;\n+\n+\t\t\tpmc = pseudo_merge_idx(writer->pseudo_merge_commits,\n+\t\t\t\t\t &c->object.oid);\n+\n+\t\t\tALLOC_GROW(pmc->pseudo_merge, pmc->nr + 1, pmc->alloc);\n+\n+\t\t\tpmc->pseudo_merge[pmc->nr++] = writer->pseudo_merges_nr;\n+\t\t\tp = commit_list_append(c, p);\n+\t\t}\n+\n+\t\tbitmap_writer_push_commit(writer, merge, 1);\n+\t\twriter->pseudo_merges_nr++;\n+\t\tif (end >= matches->unstable_nr)\n+\t\t\tbreak;\n+\t}\n+}\n+\n+static int commit_date_cmp(const void *va, const void *vb)\n+{\n+\ttimestamp_t a = (*(const struct commit **)va)->date;\n+\ttimestamp_t b = (*(const struct commit **)vb)->date;\n+\n+\tif (a < b)\n+\t\treturn -1;\n+\telse if (a > b)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static void sort_pseudo_merge_matches(struct pseudo_merge_matches *matches)\n+{\n+\tQSORT(matches->stable, matches->stable_nr, commit_date_cmp);\n+\tQSORT(matches->unstable, matches->unstable_nr, commit_date_cmp);\n+}\n+\n+void select_pseudo_merges(struct bitmap_writer *writer,\n+\t\t\t struct commit **commits, size_t commits_nr)\n+{\n+\tstruct progress *progress = NULL;\n+\tuint32_t i;\n+\n+\tif (!writer->pseudo_merge_groups.nr)\n+\t\treturn;\n+\n+\tif (writer->show_progress)\n+\t\tprogress = start_progress(\"Selecting pseudo-merge commits\",\n+\t\t\t\t\t writer->pseudo_merge_groups.nr);\n+\n+\tfor_each_ref(find_pseudo_merge_group_for_ref, writer);\n+\n+\tfor (i = 0; i < writer->pseudo_merge_groups.nr; i++) {\n+\t\tstruct pseudo_merge_group *group;\n+\t\tstruct hashmap_iter iter;\n+\t\tstruct strmap_entry *e;\n+\n+\t\tgroup = writer->pseudo_merge_groups.items[i].util;\n+\t\tstrmap_for_each_entry(&group->matches, &iter, e) {\n+\t\t\tstruct pseudo_merge_matches *matches = e->value;\n+\n+\t\t\tsort_pseudo_merge_matches(matches);\n+\n+\t\t\tselect_pseudo_merges_1(writer, group, matches);\n+\t\t}\n+\n+\t\tdisplay_progress(progress, i + 1);\n+\t}\n+\n+\tstop_progress(&progress);\n+}\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nindex cab8ff6960a..cab54daf14b 100644\n--- a/pseudo-merge.h\n+++ b/pseudo-merge.h\n@@ -2,5 +2,99 @@\n #define PSEUDO_MERGE_H\n \n #include \"git-compat-util.h\"\n+#include \"strmap.h\"\n+#include \"khash.h\"\n+#include \"ewah/ewok.h\"\n+\n+struct commit;\n+struct string_list;\n+struct bitmap_index;\n+struct bitmap_writer;\n+\n+/*\n+ * A pseudo-merge group tracks the set of non-bitmapped reference tips\n+ * that match the given pattern.\n+ *\n+ * Within those matches, they are further segmented by separating\n+ * consecutive capture groups with '-' dash character capture groups\n+ * with '-' dash characters.\n+ *\n+ * Those groups are then ordered by committer date and partitioned\n+ * into individual pseudo-merge(s) according to the decay, max_merges,\n+ * sample_rate, and threshold parameters.\n+ */\n+struct pseudo_merge_group {\n+\tregex_t *pattern;\n+\n+\t/* capture group(s) -> struct pseudo_merge_matches */\n+\tstruct strmap matches;\n+\n+\t/*\n+\t * The individual pseudo-merge(s) that are generated from the\n+\t * above array of matches, partitioned according to the below\n+\t * parameters.\n+\t */\n+\tstruct commit **merges;\n+\tsize_t merges_nr;\n+\tsize_t merges_alloc;\n+\n+\t/*\n+\t * Pseudo-merge grouping parameters. See git-config(1) for\n+\t * more information.\n+\t */\n+\tfloat decay;\n+\tint max_merges;\n+\tfloat sample_rate;\n+\tint stable_size;\n+\ttimestamp_t threshold;\n+\ttimestamp_t stable_threshold;\n+};\n+\n+struct pseudo_merge_matches {\n+\tstruct commit **stable;\n+\tstruct commit **unstable;\n+\tsize_t stable_nr, stable_alloc;\n+\tsize_t unstable_nr, unstable_alloc;\n+};\n+\n+/*\n+ * Read the repository's configuration:\n+ *\n+ * - bitmapPseudoMerge..pattern\n+ * - bitmapPseudoMerge..decay\n+ * - bitmapPseudoMerge..sampleRate\n+ * - bitmapPseudoMerge..threshold\n+ * - bitmapPseudoMerge..maxMerges\n+ * - bitmapPseudoMerge..stableThreshold\n+ * - bitmapPseudoMerge..stableSize\n+ *\n+ * and populates the given `list` with pseudo-merge groups. String\n+ * entry keys are the pseudo-merge group names, and the values are\n+ * pointers to the pseudo_merge_group structure itself.\n+ */\n+void load_pseudo_merges_from_config(struct string_list *list);\n+\n+/*\n+ * A pseudo-merge commit index (pseudo_merge_commit_idx) maps a\n+ * particular (non-pseudo-merge) commit to the list of pseudo-merge(s)\n+ * it appears in.\n+ */\n+struct pseudo_merge_commit_idx {\n+\tuint32_t *pseudo_merge;\n+\tsize_t nr, alloc;\n+};\n+\n+/*\n+ * Selects pseudo-merges from a list of commits, populating the given\n+ * string_list of pseudo-merge groups.\n+ *\n+ * Populates the pseudo_merge_commits map with a commit_idx\n+ * corresponding to each commit in the list. Counts the total number\n+ * of pseudo-merges generated.\n+ *\n+ * Optionally shows a progress meter.\n+ */\n+void select_pseudo_merges(struct bitmap_writer *writer,\n+\t\t\t struct commit **commits, size_t commits_nr);\n \n #endif\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495196","messageId":"311226f65c27295aff159fa741e1e6a44ade4b8b.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 18/30] pack-bitmap-write.c: write pseudo-merge table","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:35Z","receivedAt":"2024-05-21T19:02:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the pack-bitmap writer machinery understands how to select and\nstore pseudo-merge commits, teach it how to write the new optional\npseudo-merge .bitmap extension.\n\nNo readers yet exist for this new extension to the .bitmap format. The\nfollowing commits will take any preparatory step(s) necessary before\nthen implementing the routines necessary to read this new table.\n\nIn the meantime, the new `write_pseudo_merges()` function implements\nwriting this new format as described by a previous commit in\nDocumentation/technical/bitmap-format.txt.\n\nWriting this table is fairly straightforward and consists of a few\nsub-components:\n\n - a pair of bitmaps for each pseudo-merge (one for the pseudo-merge\n \"parents\", and another for the objects reachable from those parents)\n\n - for each commit, the offset of either (a) the pseudo-merge it\n belongs to, or (b) an extended lookup table if it belongs to >1\n pseudo-merge groups\n\n - if there are any commits belonging to >1 pseudo-merge group, the\n extended lookup tables (which each consist of the number of\n pseudo-merge groups a commit appears in, and then that many 4-byte\n unsigned )\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 131 ++++++++++++++++++++++++++++++++++++++++++++\n pack-bitmap.h | 1 +\n 2 files changed, 132 insertions(+)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex d5884ea5e9c..47250398aa2 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -18,6 +18,7 @@\n #include \"tree.h\"\n #include \"tree-walk.h\"\n #include \"pseudo-merge.h\"\n+#include \"oid-array.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -771,6 +772,130 @@ static void write_selected_commits_v1(struct bitmap_writer *writer,\n \t}\n }\n \n+static void write_pseudo_merges(struct bitmap_writer *writer,\n+\t\t\t\tstruct hashfile *f)\n+{\n+\tstruct oid_array commits = OID_ARRAY_INIT;\n+\tstruct bitmap **commits_bitmap = NULL;\n+\toff_t *pseudo_merge_ofs = NULL;\n+\toff_t start, table_start, next_ext;\n+\n+\tuint32_t base = bitmap_writer_nr_selected_commits(writer);\n+\tsize_t i, j = 0;\n+\n+\tCALLOC_ARRAY(commits_bitmap, writer->pseudo_merges_nr);\n+\tCALLOC_ARRAY(pseudo_merge_ofs, writer->pseudo_merges_nr);\n+\n+\tfor (i = 0; i < writer->pseudo_merges_nr; i++) {\n+\t\tstruct bitmapped_commit *merge = &writer->selected[base + i];\n+\t\tstruct commit_list *p;\n+\n+\t\tif (!merge->pseudo_merge)\n+\t\t\tBUG(\"found non-pseudo merge commit at %\"PRIuMAX, (uintmax_t)i);\n+\n+\t\tcommits_bitmap[i] = bitmap_new();\n+\n+\t\tfor (p = merge->commit->parents; p; p = p->next)\n+\t\t\tbitmap_set(commits_bitmap[i],\n+\t\t\t\t find_object_pos(writer, &p->item->object.oid,\n+\t\t\t\t\t\t NULL));\n+\t}\n+\n+\tstart = hashfile_total(f);\n+\n+\tfor (i = 0; i < writer->pseudo_merges_nr; i++) {\n+\t\tstruct ewah_bitmap *commits_ewah = bitmap_to_ewah(commits_bitmap[i]);\n+\n+\t\tpseudo_merge_ofs[i] = hashfile_total(f);\n+\n+\t\tdump_bitmap(f, commits_ewah);\n+\t\tdump_bitmap(f, writer->selected[base+i].write_as);\n+\n+\t\tewah_free(commits_ewah);\n+\t}\n+\n+\tnext_ext = st_add(hashfile_total(f),\n+\t\t\t st_mult(kh_size(writer->pseudo_merge_commits),\n+\t\t\t\t sizeof(uint64_t)));\n+\n+\ttable_start = hashfile_total(f);\n+\n+\tcommits.alloc = kh_size(writer->pseudo_merge_commits);\n+\tCALLOC_ARRAY(commits.oid, commits.alloc);\n+\n+\tfor (i = kh_begin(writer->pseudo_merge_commits); i != kh_end(writer->pseudo_merge_commits); i++) {\n+\t\tif (!kh_exist(writer->pseudo_merge_commits, i))\n+\t\t\tcontinue;\n+\t\toid_array_append(&commits, &kh_key(writer->pseudo_merge_commits, i));\n+\t}\n+\n+\toid_array_sort(&commits);\n+\n+\t/* write lookup table (non-extended) */\n+\tfor (i = 0; i < commits.nr; i++) {\n+\t\tint hash_pos;\n+\t\tstruct pseudo_merge_commit_idx *c;\n+\n+\t\thash_pos = kh_get_oid_map(writer->pseudo_merge_commits,\n+\t\t\t\t\t commits.oid[i]);\n+\t\tif (hash_pos == kh_end(writer->pseudo_merge_commits))\n+\t\t\tBUG(\"could not find pseudo-merge commit %s\",\n+\t\t\t oid_to_hex(&commits.oid[i]));\n+\n+\t\tc = kh_value(writer->pseudo_merge_commits, hash_pos);\n+\n+\t\thashwrite_be32(f, find_object_pos(writer, &commits.oid[i],\n+\t\t\t\t\t\t NULL));\n+\t\tif (c->nr == 1)\n+\t\t\thashwrite_be64(f, pseudo_merge_ofs[c->pseudo_merge[0]]);\n+\t\telse if (c->nr > 1) {\n+\t\t\tif (next_ext & ((uint64_t)1<<63))\n+\t\t\t\tdie(_(\"too many pseudo-merges\"));\n+\t\t\thashwrite_be64(f, next_ext | ((uint64_t)1<<63));\n+\t\t\tnext_ext = st_add3(next_ext,\n+\t\t\t\t\t sizeof(uint32_t),\n+\t\t\t\t\t st_mult(c->nr, sizeof(uint64_t)));\n+\t\t} else\n+\t\t\tBUG(\"expected commit '%s' to have at least one \"\n+\t\t\t \"pseudo-merge\", oid_to_hex(&commits.oid[i]));\n+\t}\n+\n+\t/* write lookup table (extended) */\n+\tfor (i = 0; i < commits.nr; i++) {\n+\t\tint hash_pos;\n+\t\tstruct pseudo_merge_commit_idx *c;\n+\n+\t\thash_pos = kh_get_oid_map(writer->pseudo_merge_commits,\n+\t\t\t\t\t commits.oid[i]);\n+\t\tif (hash_pos == kh_end(writer->pseudo_merge_commits))\n+\t\t\tBUG(\"could not find pseudo-merge commit %s\",\n+\t\t\t oid_to_hex(&commits.oid[i]));\n+\n+\t\tc = kh_value(writer->pseudo_merge_commits, hash_pos);\n+\t\tif (c->nr == 1)\n+\t\t\tcontinue;\n+\n+\t\thashwrite_be32(f, c->nr);\n+\t\tfor (j = 0; j < c->nr; j++)\n+\t\t\thashwrite_be64(f, pseudo_merge_ofs[c->pseudo_merge[j]]);\n+\t}\n+\n+\t/* write positions for all pseudo merges */\n+\tfor (i = 0; i < writer->pseudo_merges_nr; i++)\n+\t\thashwrite_be64(f, pseudo_merge_ofs[i]);\n+\n+\thashwrite_be32(f, writer->pseudo_merges_nr);\n+\thashwrite_be32(f, kh_size(writer->pseudo_merge_commits));\n+\thashwrite_be64(f, table_start - start);\n+\thashwrite_be64(f, hashfile_total(f) - start + sizeof(uint64_t));\n+\n+\tfor (i = 0; i < writer->pseudo_merges_nr; i++)\n+\t\tbitmap_free(commits_bitmap[i]);\n+\n+\tfree(pseudo_merge_ofs);\n+\tfree(commits_bitmap);\n+}\n+\n static int table_cmp(const void *_va, const void *_vb, void *_data)\n {\n \tstruct bitmap_writer *writer = _data;\n@@ -878,6 +1003,9 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \n \tint fd = odb_mkstemp(&tmp_file, \"pack/tmp_bitmap_XXXXXX\");\n \n+\tif (writer->pseudo_merges_nr)\n+\t\toptions |= BITMAP_OPT_PSEUDO_MERGES;\n+\n \tf = hashfd(fd, tmp_file.buf);\n \n \tmemcpy(header.magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE));\n@@ -907,6 +1035,9 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \n \twrite_selected_commits_v1(writer, f, offsets);\n \n+\tif (options & BITMAP_OPT_PSEUDO_MERGES)\n+\t\twrite_pseudo_merges(writer, f);\n+\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n \t\twrite_lookup_table(writer, f, offsets);\n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 1e730ea1e54..db9ae554fa8 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -37,6 +37,7 @@ enum pack_bitmap_opts {\n \tBITMAP_OPT_FULL_DAG = 0x1,\n \tBITMAP_OPT_HASH_CACHE = 0x4,\n \tBITMAP_OPT_LOOKUP_TABLE = 0x10,\n+\tBITMAP_OPT_PSEUDO_MERGES = 0x20,\n };\n \n enum pack_bitmap_flags {\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495197","messageId":"55dd7a8023e78d187c3f71164537f49af07110bf.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 19/30] pack-bitmap: extract `read_bitmap()` function","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:38Z","receivedAt":"2024-05-21T19:02:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pack-bitmap machinery uses the `read_bitmap_1()` function to read a\nbitmap from within the mmap'd region corresponding to the .bitmap file.\nAs as side-effect of calling this function, `read_bitmap_1()` increments\nthe `index->map_pos` variable to reflect the number of bytes read.\n\nExtract the core of this routine to a separate function (that operates\nover a `const unsigned char *`, a `size_t` and a `size_t *` pointer)\ninstead of a `struct bitmap_index *` pointer.\n\nThis function (called `read_bitmap()`) is part of the pack-bitmap.h API\nso that it can be used within the upcoming portion of the implementation\nin pseduo-merge.ch.\n\nRewrite the existing function, `read_bitmap_1()`, in terms of its more\ngeneric counterpart.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 24 +++++++++++++++---------\n pack-bitmap.h | 2 ++\n 2 files changed, 17 insertions(+), 9 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 35c5ef9d3cd..3519edb896b 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -129,17 +129,13 @@ static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st)\n \treturn composed;\n }\n \n-/*\n- * Read a bitmap from the current read position on the mmaped\n- * index, and increase the read position accordingly\n- */\n-static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n+struct ewah_bitmap *read_bitmap(const unsigned char *map,\n+\t\t\t\tsize_t map_size, size_t *map_pos)\n {\n \tstruct ewah_bitmap *b = ewah_pool_new();\n \n-\tssize_t bitmap_size = ewah_read_mmap(b,\n-\t\tindex->map + index->map_pos,\n-\t\tindex->map_size - index->map_pos);\n+\tssize_t bitmap_size = ewah_read_mmap(b, map + *map_pos,\n+\t\t\t\t\t map_size - *map_pos);\n \n \tif (bitmap_size < 0) {\n \t\terror(_(\"failed to load bitmap index (corrupted?)\"));\n@@ -147,10 +143,20 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \t\treturn NULL;\n \t}\n \n-\tindex->map_pos += bitmap_size;\n+\t*map_pos += bitmap_size;\n+\n \treturn b;\n }\n \n+/*\n+ * Read a bitmap from the current read position on the mmaped\n+ * index, and increase the read position accordingly\n+ */\n+static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n+{\n+\treturn read_bitmap(index->map, index->map_size, &index->map_pos);\n+}\n+\n static uint32_t bitmap_num_objects(struct bitmap_index *index)\n {\n \tif (index->midx)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex db9ae554fa8..21aabf805ea 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -160,4 +160,6 @@ int bitmap_is_preferred_refname(struct repository *r, const char *refname);\n \n int verify_bitmap_files(struct repository *r);\n \n+struct ewah_bitmap *read_bitmap(const unsigned char *map,\n+\t\t\t\tsize_t map_size, size_t *map_pos);\n #endif\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495198","messageId":"3cc5434e44e09e8ba5f73df602bc3801112fa8c0.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 20/30] pseudo-merge: scaffolding for reads","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:41Z","receivedAt":"2024-05-21T19:02:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement scaffolding within the new pseudo-merge compilation unit\nnecessary to use the pseudo-merge API from within the pack-bitmap.c\nmachinery.\n\nThe core of this scaffolding is two-fold:\n\n - The `pseudo_merge` structure itself, which represents an individual\n pseudo-merge bitmap. It has fields for both bitmaps, as well as\n metadata about its position within the memory-mapped region, and\n a few extra bits indicating whether or not it is satisfied, and\n which bitmaps(s, if any) have been read, since they are initialized\n lazily.\n\n - The `pseudo_merge_map` structure, which holds an array of\n pseudo_merges, as well as a pointer to the memory-mapped region\n containing the pseudo-merge serialization from within a .bitmap\n file.\n\nNote that the `bitmap_index` structure is defined statically within the\npack-bitmap.o compilation unit, so we can't take in a `struct\nbitmap_index *`. Instead, wrap the primary components necessary to read\nthe pseudo-merges in this new structure to avoid exposing the\nimplementation details of the `bitmap_index` structure.\n\nSigned-off-by: Taylor Blau \n---\n pseudo-merge.c | 10 ++++++++\n pseudo-merge.h | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 75 insertions(+)\n\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex 4be730563eb..1aca70ecdfb 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -454,3 +454,13 @@ void select_pseudo_merges(struct bitmap_writer *writer,\n \n \tstop_progress(&progress);\n }\n+\n+void free_pseudo_merge_map(struct pseudo_merge_map *pm)\n+{\n+\tuint32_t i;\n+\tfor (i = 0; i < pm->nr; i++) {\n+\t\tewah_pool_free(pm->v[i].commits);\n+\t\tewah_pool_free(pm->v[i].bitmap);\n+\t}\n+\tfree(pm->v);\n+}\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nindex cab54daf14b..a3f0243062c 100644\n--- a/pseudo-merge.h\n+++ b/pseudo-merge.h\n@@ -97,4 +97,69 @@ struct pseudo_merge_commit_idx {\n void select_pseudo_merges(struct bitmap_writer *writer,\n \t\t\t struct commit **commits, size_t commits_nr);\n \n+/*\n+ * Represents a serialized view of a file containing pseudo-merge(s)\n+ * (see Documentation/technical/bitmap-format.txt for a specification\n+ * of the format).\n+ */\n+struct pseudo_merge_map {\n+\t/*\n+\t * An array of pseudo-merge(s), lazily loaded from the .bitmap\n+\t * file.\n+\t */\n+\tstruct pseudo_merge *v;\n+\tsize_t nr;\n+\tsize_t commits_nr;\n+\n+\t/*\n+\t * Pointers into a memory-mapped view of the .bitmap file:\n+\t *\n+\t * - map: the beginning of the .bitmap file\n+\t * - commits: the beginning of the pseudo-merge commit index\n+\t * - map_size: the size of the .bitmap file\n+\t */\n+\tconst unsigned char *map;\n+\tconst unsigned char *commits;\n+\n+\tsize_t map_size;\n+};\n+\n+/*\n+ * An individual pseudo-merge, storing a pair of lazily-loaded\n+ * bitmaps:\n+ *\n+ * - commits: the set of commit(s) that are part of the pseudo-merge\n+ * - bitmap: the set of object(s) reachable from the above set of\n+ * commits.\n+ *\n+ * The `at` and `bitmap_at` fields are used to store the locations of\n+ * each of the above bitmaps in the .bitmap file.\n+ */\n+struct pseudo_merge {\n+\tstruct ewah_bitmap *commits;\n+\tstruct ewah_bitmap *bitmap;\n+\n+\toff_t at;\n+\toff_t bitmap_at;\n+\n+\t/*\n+\t * `satisfied` indicates whether the given pseudo-merge has been\n+\t * used.\n+\t *\n+\t * `loaded_commits` and `loaded_bitmap` indicate whether the\n+\t * respective bitmaps have been loaded and read from the\n+\t * .bitmap file.\n+\t */\n+\tunsigned satisfied : 1,\n+\t\t loaded_commits : 1,\n+\t\t loaded_bitmap : 1;\n+};\n+\n+/*\n+ * Frees the given pseudo-merge map, releasing any memory held by (a)\n+ * parsed EWAH bitmaps, or (b) the array of pseudo-merges itself. Does\n+ * not free the memory-mapped view of the .bitmap file.\n+ */\n+void free_pseudo_merge_map(struct pseudo_merge_map *pm);\n+\n #endif\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495199","messageId":"7664f5f964867f62dbd748351bda83007958a751.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 21/30] pack-bitmap.c: read pseudo-merge extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:45Z","receivedAt":"2024-05-21T19:02:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the scaffolding for reading the pseudo-merge extension has been\nlaid, teach the pack-bitmap machinery to read the pseudo-merge extension\nwhen present.\n\nNote that pseudo-merges themselves are not yet used during traversal,\nthis step will be taken by a future commit.\n\nIn the meantime, read the table and initialize the pseudo_merge_map\nstructure introduced by a previous commit. When the pseudo-merge\nextension is present, `load_bitmap_header()` performs basic sanity\nchecks to make sure that the table is well-formed.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 39 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 39 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 3519edb896b..fc9c3e2fc43 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -20,6 +20,7 @@\n #include \"list-objects-filter-options.h\"\n #include \"midx.h\"\n #include \"config.h\"\n+#include \"pseudo-merge.h\"\n \n /*\n * An entry on the bitmap index, representing the bitmap for a given\n@@ -86,6 +87,9 @@ struct bitmap_index {\n \t */\n \tunsigned char *table_lookup;\n \n+\t/* This contains the pseudo-merge cache within 'map' (if found). */\n+\tstruct pseudo_merge_map pseudo_merges;\n+\n \t/*\n \t * Extended index.\n \t *\n@@ -205,6 +209,41 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t\t\t\tindex->table_lookup = (void *)(index_end - table_size);\n \t\t\tindex_end -= table_size;\n \t\t}\n+\n+\t\tif (flags & BITMAP_OPT_PSEUDO_MERGES) {\n+\t\t\tunsigned char *pseudo_merge_ofs;\n+\t\t\tsize_t table_size;\n+\t\t\tuint32_t i;\n+\n+\t\t\tif (sizeof(table_size) > index_end - index->map - header_size)\n+\t\t\t\treturn error(_(\"corrupted bitmap index file (too short to fit pseudo-merge table header)\"));\n+\n+\t\t\ttable_size = get_be64(index_end - 8);\n+\t\t\tif (table_size > index_end - index->map - header_size)\n+\t\t\t\treturn error(_(\"corrupted bitmap index file (too short to fit pseudo-merge table)\"));\n+\n+\t\t\tif (git_env_bool(\"GIT_TEST_USE_PSEUDO_MERGES\", 1)) {\n+\t\t\t\tconst unsigned char *ext = (index_end - table_size);\n+\n+\t\t\t\tindex->pseudo_merges.map = index->map;\n+\t\t\t\tindex->pseudo_merges.map_size = index->map_size;\n+\t\t\t\tindex->pseudo_merges.commits = ext + get_be64(index_end - 16);\n+\t\t\t\tindex->pseudo_merges.commits_nr = get_be32(index_end - 20);\n+\t\t\t\tindex->pseudo_merges.nr = get_be32(index_end - 24);\n+\n+\t\t\t\tCALLOC_ARRAY(index->pseudo_merges.v,\n+\t\t\t\t\t index->pseudo_merges.nr);\n+\n+\t\t\t\tpseudo_merge_ofs = index_end - 24 -\n+\t\t\t\t\t(index->pseudo_merges.nr * sizeof(uint64_t));\n+\t\t\t\tfor (i = 0; i < index->pseudo_merges.nr; i++) {\n+\t\t\t\t\tindex->pseudo_merges.v[i].at = get_be64(pseudo_merge_ofs);\n+\t\t\t\t\tpseudo_merge_ofs += sizeof(uint64_t);\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tindex_end -= table_size;\n+\t\t}\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495200","messageId":"8ba0a9c5402fb154bc316768a8fbb016e302a686.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 22/30] pseudo-merge: implement support for reading pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:48Z","receivedAt":"2024-05-21T19:02:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement the basic API for reading pseudo-merge bitmaps, which consists\nof four basic functions:\n\n - pseudo_merge_bitmap()\n - use_pseudo_merge()\n - apply_pseudo_merges_for_commit()\n - cascade_pseudo_merges()\n\nThese functions are all documented in pseudo-merge.h, but their rough\ndescriptions are as follows:\n\n - pseudo_merge_bitmap() reads and inflates the objects EWAH bitmap for\n a given pseudo-merge\n\n - use_pseudo_merge() does the same as pseudo_merge_bitmap(), but on\n the commits EWAH bitmap, not the objects bitmap\n\n - apply_pseudo_merges_for_commit() applies all satisfied pseudo-merge\n commits for a given result set, and cascades any yet-unsatisfied\n pseudo-merges if any were applied in the previous step\n\n - cascade_pseudo_merges() applies all pseudo-merges which are\n satisfied but have not been previously applied, repeating this\n process until no more pseudo-merges can be applied\n\nThe core of the API is the latter two functions, which are responsible\nfor applying pseudo-merges during the object traversal implemented in\nthe pack-bitmap machinery.\n\nThe other two functions (pseudo_merge_bitmap(), and use_pseudo_merge())\nare low-level ways to interact with the pseudo-merge machinery, which\nwill be useful in future commits.\n\nSigned-off-by: Taylor Blau \n---\n pseudo-merge.c | 231 +++++++++++++++++++++++++++++++++++++++++++++++++\n pseudo-merge.h | 44 ++++++++++\n 2 files changed, 275 insertions(+)\n\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex 1aca70ecdfb..0f50ac6183e 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -10,6 +10,7 @@\n #include \"commit.h\"\n #include \"alloc.h\"\n #include \"progress.h\"\n+#include \"hex.h\"\n \n #define DEFAULT_PSEUDO_MERGE_DECAY 1.0f\n #define DEFAULT_PSEUDO_MERGE_MAX_MERGES 64\n@@ -464,3 +465,233 @@ void free_pseudo_merge_map(struct pseudo_merge_map *pm)\n \t}\n \tfree(pm->v);\n }\n+\n+struct pseudo_merge_commit_ext {\n+\tuint32_t nr;\n+\tconst unsigned char *ptr;\n+};\n+\n+static int pseudo_merge_ext_at(const struct pseudo_merge_map *pm,\n+\t\t\t struct pseudo_merge_commit_ext *ext, size_t at)\n+{\n+\tif (at >= pm->map_size)\n+\t\treturn error(_(\"extended pseudo-merge read out-of-bounds \"\n+\t\t\t \"(%\"PRIuMAX\" >= %\"PRIuMAX\")\"),\n+\t\t\t (uintmax_t)at, (uintmax_t)pm->map_size);\n+\n+\text->nr = get_be32(pm->map + at);\n+\text->ptr = pm->map + at + sizeof(uint32_t);\n+\n+\treturn 0;\n+}\n+\n+struct ewah_bitmap *pseudo_merge_bitmap(const struct pseudo_merge_map *pm,\n+\t\t\t\t\tstruct pseudo_merge *merge)\n+{\n+\tif (!merge->loaded_commits)\n+\t\tBUG(\"cannot use unloaded pseudo-merge bitmap\");\n+\n+\tif (!merge->loaded_bitmap) {\n+\t\tsize_t at = merge->bitmap_at;\n+\n+\t\tmerge->bitmap = read_bitmap(pm->map, pm->map_size, &at);\n+\t\tmerge->loaded_bitmap = 1;\n+\t}\n+\n+\treturn merge->bitmap;\n+}\n+\n+struct pseudo_merge *use_pseudo_merge(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct pseudo_merge *merge)\n+{\n+\tif (!merge->loaded_commits) {\n+\t\tsize_t pos = merge->at;\n+\n+\t\tmerge->commits = read_bitmap(pm->map, pm->map_size, &pos);\n+\t\tmerge->bitmap_at = pos;\n+\t\tmerge->loaded_commits = 1;\n+\t}\n+\treturn merge;\n+}\n+\n+static struct pseudo_merge *pseudo_merge_at(const struct pseudo_merge_map *pm,\n+\t\t\t\t\t struct object_id *oid,\n+\t\t\t\t\t size_t want)\n+{\n+\tsize_t lo = 0;\n+\tsize_t hi = pm->nr;\n+\n+\twhile (lo < hi) {\n+\t\tsize_t mi = lo + (hi - lo) / 2;\n+\t\tsize_t got = pm->v[mi].at;\n+\n+\t\tif (got == want)\n+\t\t\treturn use_pseudo_merge(pm, &pm->v[mi]);\n+\t\telse if (got < want)\n+\t\t\thi = mi;\n+\t\telse\n+\t\t\tlo = mi + 1;\n+\t}\n+\n+\twarning(_(\"could not find pseudo-merge for commit %s at offset %\"PRIuMAX),\n+\t\toid_to_hex(oid), (uintmax_t)want);\n+\n+\treturn NULL;\n+}\n+\n+struct pseudo_merge_commit {\n+\tuint32_t commit_pos;\n+\tuint64_t pseudo_merge_ofs;\n+};\n+\n+#define PSEUDO_MERGE_COMMIT_RAWSZ (sizeof(uint32_t)+sizeof(uint64_t))\n+\n+static void read_pseudo_merge_commit_at(struct pseudo_merge_commit *merge,\n+\t\t\t\t\tconst unsigned char *at)\n+{\n+\tmerge->commit_pos = get_be32(at);\n+\tmerge->pseudo_merge_ofs = get_be64(at + sizeof(uint32_t));\n+}\n+\n+static int nth_pseudo_merge_ext(const struct pseudo_merge_map *pm,\n+\t\t\t\tstruct pseudo_merge_commit_ext *ext,\n+\t\t\t\tstruct pseudo_merge_commit *merge,\n+\t\t\t\tuint32_t n)\n+{\n+\tsize_t ofs;\n+\n+\tif (n >= ext->nr)\n+\t\treturn error(_(\"extended pseudo-merge lookup out-of-bounds \"\n+\t\t\t \"(%\"PRIu32\" >= %\"PRIu32\")\"), n, ext->nr);\n+\n+\tofs = get_be64(ext->ptr + st_mult(n, sizeof(uint64_t)));\n+\tif (ofs >= pm->map_size)\n+\t\treturn error(_(\"out-of-bounds read: (%\"PRIuMAX\" >= %\"PRIuMAX\")\"),\n+\t\t\t (uintmax_t)ofs, (uintmax_t)pm->map_size);\n+\n+\tread_pseudo_merge_commit_at(merge, pm->map + ofs);\n+\n+\treturn 0;\n+}\n+\n+static unsigned apply_pseudo_merge(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct pseudo_merge *merge,\n+\t\t\t\t struct bitmap *result,\n+\t\t\t\t struct bitmap *roots)\n+{\n+\tif (merge->satisfied)\n+\t\treturn 0;\n+\n+\tif (!ewah_bitmap_is_subset(merge->commits, roots ? roots : result))\n+\t\treturn 0;\n+\n+\tbitmap_or_ewah(result, pseudo_merge_bitmap(pm, merge));\n+\tif (roots)\n+\t\tbitmap_or_ewah(roots, pseudo_merge_bitmap(pm, merge));\n+\tmerge->satisfied = 1;\n+\n+\treturn 1;\n+}\n+\n+static int pseudo_merge_commit_cmp(const void *va, const void *vb)\n+{\n+\tstruct pseudo_merge_commit merge;\n+\tuint32_t key = *(uint32_t*)va;\n+\n+\tread_pseudo_merge_commit_at(&merge, vb);\n+\n+\tif (key < merge.commit_pos)\n+\t\treturn -1;\n+\tif (key > merge.commit_pos)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static struct pseudo_merge_commit *find_pseudo_merge(const struct pseudo_merge_map *pm,\n+\t\t\t\t\t\t uint32_t pos)\n+{\n+\tif (!pm->commits_nr)\n+\t\treturn NULL;\n+\n+\treturn bsearch(&pos, pm->commits, pm->commits_nr,\n+\t\t PSEUDO_MERGE_COMMIT_RAWSZ, pseudo_merge_commit_cmp);\n+}\n+\n+int apply_pseudo_merges_for_commit(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct bitmap *result,\n+\t\t\t\t struct commit *commit, uint32_t commit_pos)\n+{\n+\tstruct pseudo_merge *merge;\n+\tstruct pseudo_merge_commit *merge_commit;\n+\tint ret = 0;\n+\n+\tmerge_commit = find_pseudo_merge(pm, commit_pos);\n+\tif (!merge_commit)\n+\t\treturn 0;\n+\n+\tif (merge_commit->pseudo_merge_ofs & ((uint64_t)1<<63)) {\n+\t\tstruct pseudo_merge_commit_ext ext = { 0 };\n+\t\toff_t ofs = merge_commit->pseudo_merge_ofs & ~((uint64_t)1<<63);\n+\t\tuint32_t i;\n+\n+\t\tif (pseudo_merge_ext_at(pm, &ext, ofs) < -1) {\n+\t\t\twarning(_(\"could not read extended pseudo-merge table \"\n+\t\t\t\t \"for commit %s\"),\n+\t\t\t\toid_to_hex(&commit->object.oid));\n+\t\t\treturn ret;\n+\t\t}\n+\n+\t\tfor (i = 0; i < ext.nr; i++) {\n+\t\t\tif (nth_pseudo_merge_ext(pm, &ext, merge_commit, i) < 0)\n+\t\t\t\treturn ret;\n+\n+\t\t\tmerge = pseudo_merge_at(pm, &commit->object.oid,\n+\t\t\t\t\t\tmerge_commit->pseudo_merge_ofs);\n+\n+\t\t\tif (!merge)\n+\t\t\t\treturn ret;\n+\n+\t\t\tif (apply_pseudo_merge(pm, merge, result, NULL))\n+\t\t\t\tret++;\n+\t\t}\n+\t} else {\n+\t\tmerge = pseudo_merge_at(pm, &commit->object.oid,\n+\t\t\t\t\tmerge_commit->pseudo_merge_ofs);\n+\n+\t\tif (!merge)\n+\t\t\treturn ret;\n+\n+\t\tif (apply_pseudo_merge(pm, merge, result, NULL))\n+\t\t\tret++;\n+\t}\n+\n+\tif (ret)\n+\t\tcascade_pseudo_merges(pm, result, NULL);\n+\n+\treturn ret;\n+}\n+\n+int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n+\t\t\t struct bitmap *result,\n+\t\t\t struct bitmap *roots)\n+{\n+\tunsigned any_satisfied;\n+\tint ret = 0;\n+\n+\tdo {\n+\t\tstruct pseudo_merge *merge;\n+\t\tuint32_t i;\n+\n+\t\tany_satisfied = 0;\n+\n+\t\tfor (i = 0; i < pm->nr; i++) {\n+\t\t\tmerge = use_pseudo_merge(pm, &pm->v[i]);\n+\t\t\tif (apply_pseudo_merge(pm, merge, result, roots)) {\n+\t\t\t\tany_satisfied |= 1;\n+\t\t\t\tret++;\n+\t\t\t}\n+\t\t}\n+\t} while (any_satisfied);\n+\n+\treturn ret;\n+}\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nindex a3f0243062c..c00b622be4b 100644\n--- a/pseudo-merge.h\n+++ b/pseudo-merge.h\n@@ -162,4 +162,48 @@ struct pseudo_merge {\n */\n void free_pseudo_merge_map(struct pseudo_merge_map *pm);\n \n+/*\n+ * Loads the bitmap corresponding to the given pseudo-merge from the\n+ * map, if it has not already been loaded.\n+ */\n+struct ewah_bitmap *pseudo_merge_bitmap(const struct pseudo_merge_map *pm,\n+\t\t\t\t\tstruct pseudo_merge *merge);\n+\n+/*\n+ * Loads the pseudo-merge and its commits bitmap from the given\n+ * pseudo-merge map, if it has not already been loaded.\n+ */\n+struct pseudo_merge *use_pseudo_merge(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct pseudo_merge *merge);\n+\n+/*\n+ * Applies pseudo-merge(s) containing the given commit to the bitmap\n+ * \"result\".\n+ *\n+ * If any pseudo-merge(s) were satisfied, returns the number\n+ * satisfied, otherwise returns 0. If any were satisfied, the\n+ * remaining unsatisfied pseudo-merges are cascaded (see below).\n+ */\n+int apply_pseudo_merges_for_commit(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct bitmap *result,\n+\t\t\t\t struct commit *commit, uint32_t commit_pos);\n+\n+/*\n+ * Applies pseudo-merge(s) which are satisfied according to the\n+ * current bitmap in result (or roots, see below). If any\n+ * pseudo-merges were satisfied, repeat the process over unsatisfied\n+ * pseudo-merge commits until no more pseudo-merges are satisfied.\n+ *\n+ * Result is the bitmap to which the pseudo-merge(s) are applied.\n+ * Roots (if given) is a bitmap of the traversal tip(s) for either\n+ * side of a reachability traversal.\n+ *\n+ * Roots may given instead of a populated results bitmap at the\n+ * beginning of a traversal on either side where the reachability\n+ * closure over tips is not yet known.\n+ */\n+int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n+\t\t\t struct bitmap *result,\n+\t\t\t struct bitmap *roots);\n+\n #endif\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495201","messageId":"2c02f303b6f5629c08b92cbab06a827905c83f1d.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 23/30] ewah: implement `ewah_bitmap_popcount()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:52Z","receivedAt":"2024-05-21T19:02:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Some of the pseudo-merge test helpers (which will be introduced in the\nfollowing commit) will want to indicate the total number of commits in\nor objects reachable from a pseudo-merge.\n\nImplement a popcount() function that operates on EWAH bitmaps to quickly\ndetermine how many bits are set in each of the respective bitmaps.\n\nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 14 ++++++++++++++\n ewah/ewok.h | 1 +\n 2 files changed, 15 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex d352fec54ce..dc2ca190f12 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -212,6 +212,20 @@ size_t bitmap_popcount(struct bitmap *self)\n \treturn count;\n }\n \n+size_t ewah_bitmap_popcount(struct ewah_bitmap *self)\n+{\n+\tstruct ewah_iterator it;\n+\teword_t word;\n+\tsize_t count = 0;\n+\n+\tewah_iterator_init(&it, self);\n+\n+\twhile (ewah_iterator_next(&word, &it))\n+\t\tcount += ewah_bit_popcount64(word);\n+\n+\treturn count;\n+}\n+\n int bitmap_is_empty(struct bitmap *self)\n {\n \tsize_t i;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 2b6c4ac499c..7074a6347b7 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -195,6 +195,7 @@ void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other);\n void bitmap_or(struct bitmap *self, const struct bitmap *other);\n \n size_t bitmap_popcount(struct bitmap *self);\n+size_t ewah_bitmap_popcount(struct ewah_bitmap *self);\n int bitmap_is_empty(struct bitmap *self);\n \n #endif\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495202","messageId":"82cce72bf55a2543eb9a4045848a85d04cdb618e.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 24/30] pack-bitmap: implement test helpers for pseudo-merge","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:55Z","receivedAt":"2024-05-21T19:03:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement three new sub-commands for the \"bitmap\" test-helper:\n\n - t/helper test-tool bitmap dump-pseudo-merges\n - t/helper test-tool bitmap dump-pseudo-merge-commits \n - t/helper test-tool bitmap dump-pseudo-merge-objects \n\nThese three helpers dump the list of pseudo merges, the \"parents\" of the\nnth pseudo-merges, and the set of objects reachable from those parents,\nrespectively.\n\nThese helpers will be useful in subsequent patches when we add test\ncoverage for pseudo-merge bitmaps.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 126 +++++++++++++++++++++++++++++++++++++++++\n pack-bitmap.h | 3 +\n t/helper/test-bitmap.c | 34 ++++++++---\n 3 files changed, 156 insertions(+), 7 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex fc9c3e2fc43..c13074673af 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -2443,6 +2443,132 @@ int test_bitmap_hashes(struct repository *r)\n \treturn 0;\n }\n \n+static void bit_pos_to_object_id(struct bitmap_index *bitmap_git,\n+\t\t\t\t uint32_t bit_pos,\n+\t\t\t\t struct object_id *oid)\n+{\n+\tuint32_t index_pos;\n+\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tindex_pos = pack_pos_to_midx(bitmap_git->midx, bit_pos);\n+\telse\n+\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, bit_pos);\n+\n+\tnth_bitmap_object_oid(bitmap_git, oid, index_pos);\n+}\n+\n+int test_bitmap_pseudo_merges(struct repository *r)\n+{\n+\tstruct bitmap_index *bitmap_git;\n+\tuint32_t i;\n+\n+\tbitmap_git = prepare_bitmap_git(r);\n+\tif (!bitmap_git || !bitmap_git->pseudo_merges.nr)\n+\t\tgoto cleanup;\n+\n+\tfor (i = 0; i < bitmap_git->pseudo_merges.nr; i++) {\n+\t\tstruct pseudo_merge *merge;\n+\t\tstruct ewah_bitmap *commits_bitmap, *merge_bitmap;\n+\n+\t\tmerge = use_pseudo_merge(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t &bitmap_git->pseudo_merges.v[i]);\n+\t\tcommits_bitmap = merge->commits;\n+\t\tmerge_bitmap = pseudo_merge_bitmap(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t\t merge);\n+\n+\t\tprintf(\"at=%\"PRIuMAX\", commits=%\"PRIuMAX\", objects=%\"PRIuMAX\"\\n\",\n+\t\t (uintmax_t)merge->at,\n+\t\t (uintmax_t)ewah_bitmap_popcount(commits_bitmap),\n+\t\t (uintmax_t)ewah_bitmap_popcount(merge_bitmap));\n+\t}\n+\n+cleanup:\n+\tfree_bitmap_index(bitmap_git);\n+\treturn 0;\n+}\n+\n+static void dump_ewah_object_ids(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct ewah_bitmap *bitmap)\n+\n+{\n+\tstruct ewah_iterator it;\n+\teword_t word;\n+\tuint32_t pos = 0;\n+\n+\tewah_iterator_init(&it, bitmap);\n+\n+\twhile (ewah_iterator_next(&word, &it)) {\n+\t\tstruct object_id oid;\n+\t\tuint32_t offset;\n+\n+\t\tfor (offset = 0; offset < BITS_IN_EWORD; offset++) {\n+\t\t\tif (!(word >> offset))\n+\t\t\t\tbreak;\n+\n+\t\t\toffset += ewah_bit_ctz64(word >> offset);\n+\n+\t\t\tbit_pos_to_object_id(bitmap_git, pos + offset, &oid);\n+\t\t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n+\t\t}\n+\t\tpos += BITS_IN_EWORD;\n+\t}\n+}\n+\n+int test_bitmap_pseudo_merge_commits(struct repository *r, uint32_t n)\n+{\n+\tstruct bitmap_index *bitmap_git;\n+\tstruct pseudo_merge *merge;\n+\tint ret = 0;\n+\n+\tbitmap_git = prepare_bitmap_git(r);\n+\tif (!bitmap_git || !bitmap_git->pseudo_merges.nr)\n+\t\tgoto cleanup;\n+\n+\tif (n >= bitmap_git->pseudo_merges.nr) {\n+\t\tret = error(_(\"pseudo-merge index out of range \"\n+\t\t\t \"(%\"PRIu32\" >= %\"PRIuMAX\")\"),\n+\t\t\t n, (uintmax_t)bitmap_git->pseudo_merges.nr);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tmerge = use_pseudo_merge(&bitmap_git->pseudo_merges,\n+\t\t\t\t &bitmap_git->pseudo_merges.v[n]);\n+\tdump_ewah_object_ids(bitmap_git, merge->commits);\n+\n+cleanup:\n+\tfree_bitmap_index(bitmap_git);\n+\treturn ret;\n+}\n+\n+int test_bitmap_pseudo_merge_objects(struct repository *r, uint32_t n)\n+{\n+\tstruct bitmap_index *bitmap_git;\n+\tstruct pseudo_merge *merge;\n+\tint ret = 0;\n+\n+\tbitmap_git = prepare_bitmap_git(r);\n+\tif (!bitmap_git || !bitmap_git->pseudo_merges.nr)\n+\t\tgoto cleanup;\n+\n+\tif (n >= bitmap_git->pseudo_merges.nr) {\n+\t\tret = error(_(\"pseudo-merge index out of range \"\n+\t\t\t \"(%\"PRIu32\" >= %\"PRIuMAX\")\"),\n+\t\t\t n, (uintmax_t)bitmap_git->pseudo_merges.nr);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tmerge = use_pseudo_merge(&bitmap_git->pseudo_merges,\n+\t\t\t\t &bitmap_git->pseudo_merges.v[n]);\n+\n+\tdump_ewah_object_ids(bitmap_git,\n+\t\t\t pseudo_merge_bitmap(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t\t merge));\n+\n+cleanup:\n+\tfree_bitmap_index(bitmap_git);\n+\treturn ret;\n+}\n+\n int rebuild_bitmap(const uint32_t *reposition,\n \t\t struct ewah_bitmap *source,\n \t\t struct bitmap *dest)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 21aabf805ea..4466b5ad0fb 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -73,6 +73,9 @@ void traverse_bitmap_commit_list(struct bitmap_index *,\n void test_bitmap_walk(struct rev_info *revs);\n int test_bitmap_commits(struct repository *r);\n int test_bitmap_hashes(struct repository *r);\n+int test_bitmap_pseudo_merges(struct repository *r);\n+int test_bitmap_pseudo_merge_commits(struct repository *r, uint32_t n);\n+int test_bitmap_pseudo_merge_objects(struct repository *r, uint32_t n);\n \n #define GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL \\\n \t\"GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL\"\ndiff --git a/t/helper/test-bitmap.c b/t/helper/test-bitmap.c\nindex af43ee1cb5e..6af2b42678f 100644\n--- a/t/helper/test-bitmap.c\n+++ b/t/helper/test-bitmap.c\n@@ -13,21 +13,41 @@ static int bitmap_dump_hashes(void)\n \treturn test_bitmap_hashes(the_repository);\n }\n \n+static int bitmap_dump_pseudo_merges(void)\n+{\n+\treturn test_bitmap_pseudo_merges(the_repository);\n+}\n+\n+static int bitmap_dump_pseudo_merge_commits(uint32_t n)\n+{\n+\treturn test_bitmap_pseudo_merge_commits(the_repository, n);\n+}\n+\n+static int bitmap_dump_pseudo_merge_objects(uint32_t n)\n+{\n+\treturn test_bitmap_pseudo_merge_objects(the_repository, n);\n+}\n+\n int cmd__bitmap(int argc, const char **argv)\n {\n \tsetup_git_directory();\n \n-\tif (argc != 2)\n-\t\tgoto usage;\n-\n-\tif (!strcmp(argv[1], \"list-commits\"))\n+\tif (argc == 2 && !strcmp(argv[1], \"list-commits\"))\n \t\treturn bitmap_list_commits();\n-\tif (!strcmp(argv[1], \"dump-hashes\"))\n+\tif (argc == 2 && !strcmp(argv[1], \"dump-hashes\"))\n \t\treturn bitmap_dump_hashes();\n+\tif (argc == 2 && !strcmp(argv[1], \"dump-pseudo-merges\"))\n+\t\treturn bitmap_dump_pseudo_merges();\n+\tif (argc == 3 && !strcmp(argv[1], \"dump-pseudo-merge-commits\"))\n+\t\treturn bitmap_dump_pseudo_merge_commits(atoi(argv[2]));\n+\tif (argc == 3 && !strcmp(argv[1], \"dump-pseudo-merge-objects\"))\n+\t\treturn bitmap_dump_pseudo_merge_objects(atoi(argv[2]));\n \n-usage:\n \tusage(\"\\ttest-tool bitmap list-commits\\n\"\n-\t \"\\ttest-tool bitmap dump-hashes\");\n+\t \"\\ttest-tool bitmap dump-hashes\\n\"\n+\t \"\\ttest-tool bitmap dump-pseudo-merges\\n\"\n+\t \"\\ttest-tool bitmap dump-pseudo-merge-commits \\n\"\n+\t \"\\ttest-tool bitmap dump-pseudo-merge-objects \");\n \n \treturn -1;\n }\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495203","messageId":"890f6c4b9deb9e3bf02aa180c7ad4ced7f7b6a80.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 25/30] t/test-lib-functions.sh: support `--date` in `test_commit_bulk()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:02:59Z","receivedAt":"2024-05-21T19:03:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"One of the tests we'll want to add for pseudo-merge bitmaps needs to be\nable to generate a large number of commits at a specific date.\n\nSupport the `--date` option (with identical semantics to the `--date`\noption for `test_commit()`) within `test_commit_bulk` as a prerequisite\nfor that.\n\nSigned-off-by: Taylor Blau \n---\n t/test-lib-functions.sh | 12 +++++++++++-\n 1 file changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 862d80c9748..16fd585e34b 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -458,6 +458,7 @@ test_commit_bulk () {\n \tindir=.\n \tref=HEAD\n \tn=1\n+\tnotick=\n \tmessage='commit %s'\n \tfilename='%s.t'\n \tcontents='content %s'\n@@ -488,6 +489,12 @@ test_commit_bulk () {\n \t\t\tfilename=\"${1#--*=}-%s.t\"\n \t\t\tcontents=\"${1#--*=} %s\"\n \t\t\t;;\n+\t\t--date)\n+\t\t\tnotick=yes\n+\t\t\tGIT_COMMITTER_DATE=\"$2\"\n+\t\t\tGIT_AUTHOR_DATE=\"$2\"\n+\t\t\tshift\n+\t\t\t;;\n \t\t-*)\n \t\t\tBUG \"invalid test_commit_bulk option: $1\"\n \t\t\t;;\n@@ -507,7 +514,10 @@ test_commit_bulk () {\n \n \twhile test \"$total\" -gt 0\n \tdo\n-\t\ttest_tick &&\n+\t\tif test -z \"$notick\"\n+\t\tthen\n+\t\t\ttest_tick\n+\t\tfi &&\n \t\techo \"commit $ref\"\n \t\tprintf 'author %s <%s> %s\\n' \\\n \t\t\t\"$GIT_AUTHOR_NAME\" \\\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495204","messageId":"41691824f78818f3c70cad6d02cd7f66d12c68c3.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 26/30] pack-bitmap.c: use pseudo-merges during traversal","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:03:03Z","receivedAt":"2024-05-21T19:03:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that all of the groundwork has been laid to support reading and\nusing pseudo-merges, make use of that work in this commit by teaching\nthe pack-bitmap machinery to use pseudo-merge(s) when available during\ntraversal.\n\nThe basic operation is as follows:\n\n - When enumerating objects on either side of a reachability query,\n first see if any subset of the roots satisfies some pseudo-merge\n bitmap. If it does, apply that pseudo-merge bitmap.\n\n - If any pseudo-merge bitmap(s) were applied in the previous step, OR\n them into the result[^1]. Then repeat the process over all\n pseudo-merge bitmaps (we'll refer to this as \"cascading\"\n pseudo-merges). Once this is done, OR in the resulting bitmap.\n\n - If there is no fill-in traversal to be done, return the bitmap for\n that side of the reachability query. If there is fill-in traversal,\n then for each commit we encounter via show_commit(), check to see if\n any unsatisfied pseudo-merges containing that commit as one of its\n parents has been made satisfied by the presence of that commit.\n\n If so, OR in the object set from that pseudo-merge bitmap, and then\n cascade. If not, continue traversal.\n\nA similar implementation is present in the boundary-based bitmap\ntraversal routines.\n\n[^1]: Importantly, we cannot OR in the entire set of roots along with\n the objects reachable from whatever pseudo-merge bitmaps were\n satisfied. This may leave some dangling bits corresponding to any\n unsatisfied root(s) getting OR'd into the resulting bitmap, tricking\n other parts of the traversal into thinking we already have a\n reachability closure over those commit(s) when we do not.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 112 ++++++++++-\n t/t5333-pseudo-merge-bitmaps.sh | 323 ++++++++++++++++++++++++++++++++\n 2 files changed, 434 insertions(+), 1 deletion(-)\n create mode 100755 t/t5333-pseudo-merge-bitmaps.sh\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex c13074673af..e61058dada6 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -114,6 +114,9 @@ struct bitmap_index {\n \tunsigned int version;\n };\n \n+static int pseudo_merges_satisfied_nr;\n+static int pseudo_merges_cascades_nr;\n+\n static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st)\n {\n \tstruct ewah_bitmap *parent;\n@@ -1006,6 +1009,22 @@ static void show_commit(struct commit *commit UNUSED,\n {\n }\n \n+static unsigned apply_pseudo_merges_for_commit_1(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct bitmap *result,\n+\t\t\t\t\t\t struct commit *commit,\n+\t\t\t\t\t\t uint32_t commit_pos)\n+{\n+\tint ret;\n+\n+\tret = apply_pseudo_merges_for_commit(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t result, commit, commit_pos);\n+\n+\tif (ret)\n+\t\tpseudo_merges_satisfied_nr += ret;\n+\n+\treturn ret;\n+}\n+\n static int add_to_include_set(struct bitmap_index *bitmap_git,\n \t\t\t struct include_data *data,\n \t\t\t struct commit *commit,\n@@ -1026,6 +1045,10 @@ static int add_to_include_set(struct bitmap_index *bitmap_git,\n \t}\n \n \tbitmap_set(data->base, bitmap_pos);\n+\tif (apply_pseudo_merges_for_commit_1(bitmap_git, data->base, commit,\n+\t\t\t\t\t bitmap_pos))\n+\t\treturn 0;\n+\n \treturn 1;\n }\n \n@@ -1151,6 +1174,20 @@ static void show_boundary_object(struct object *object UNUSED,\n \tBUG(\"should not be called\");\n }\n \n+static unsigned cascade_pseudo_merges_1(struct bitmap_index *bitmap_git,\n+\t\t\t\t\tstruct bitmap *result,\n+\t\t\t\t\tstruct bitmap *roots)\n+{\n+\tint ret = cascade_pseudo_merges(&bitmap_git->pseudo_merges,\n+\t\t\t\t\tresult, roots);\n+\tif (ret) {\n+\t\tpseudo_merges_cascades_nr++;\n+\t\tpseudo_merges_satisfied_nr += ret;\n+\t}\n+\n+\treturn ret;\n+}\n+\n static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \t\t\t\t\t struct rev_info *revs,\n \t\t\t\t\t struct object_list *roots)\n@@ -1160,6 +1197,7 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \tunsigned int i;\n \tunsigned int tmp_blobs, tmp_trees, tmp_tags;\n \tint any_missing = 0;\n+\tint existing_bitmaps = 0;\n \n \tcb.bitmap_git = bitmap_git;\n \tcb.base = bitmap_new();\n@@ -1167,6 +1205,25 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \n \trevs->ignore_missing_links = 1;\n \n+\tif (bitmap_git->pseudo_merges.nr) {\n+\t\tstruct bitmap *roots_bitmap = bitmap_new();\n+\t\tstruct object_list *objects = NULL;\n+\n+\t\tfor (objects = roots; objects; objects = objects->next) {\n+\t\t\tstruct object *object = objects->item;\n+\t\t\tint pos;\n+\n+\t\t\tpos = bitmap_position(bitmap_git, &object->oid);\n+\t\t\tif (pos < 0)\n+\t\t\t\tcontinue;\n+\n+\t\t\tbitmap_set(roots_bitmap, pos);\n+\t\t}\n+\n+\t\tif (!cascade_pseudo_merges_1(bitmap_git, cb.base, roots_bitmap))\n+\t\t\tbitmap_free(roots_bitmap);\n+\t}\n+\n \t/*\n \t * OR in any existing reachability bitmaps among `roots` into\n \t * `cb.base`.\n@@ -1178,8 +1235,10 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \t\t\tcontinue;\n \n \t\tif (add_commit_to_bitmap(bitmap_git, &cb.base,\n-\t\t\t\t\t (struct commit *)object))\n+\t\t\t\t\t (struct commit *)object)) {\n+\t\t\texisting_bitmaps = 1;\n \t\t\tcontinue;\n+\t\t}\n \n \t\tany_missing = 1;\n \t}\n@@ -1187,6 +1246,9 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \tif (!any_missing)\n \t\tgoto cleanup;\n \n+\tif (existing_bitmaps)\n+\t\tcascade_pseudo_merges_1(bitmap_git, cb.base, NULL);\n+\n \ttmp_blobs = revs->blob_objects;\n \ttmp_trees = revs->tree_objects;\n \ttmp_tags = revs->blob_objects;\n@@ -1242,6 +1304,13 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \treturn cb.base;\n }\n \n+static void unsatisfy_all_pseudo_merges(struct bitmap_index *bitmap_git)\n+{\n+\tuint32_t i;\n+\tfor (i = 0; i < bitmap_git->pseudo_merges.nr; i++)\n+\t\tbitmap_git->pseudo_merges.v[i].satisfied = 0;\n+}\n+\n static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\t\t\t struct rev_info *revs,\n \t\t\t\t struct object_list *roots,\n@@ -1249,9 +1318,32 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n {\n \tstruct bitmap *base = NULL;\n \tint needs_walk = 0;\n+\tunsigned existing_bitmaps = 0;\n \n \tstruct object_list *not_mapped = NULL;\n \n+\tunsatisfy_all_pseudo_merges(bitmap_git);\n+\n+\tif (bitmap_git->pseudo_merges.nr) {\n+\t\tstruct bitmap *roots_bitmap = bitmap_new();\n+\t\tstruct object_list *objects = NULL;\n+\n+\t\tfor (objects = roots; objects; objects = objects->next) {\n+\t\t\tstruct object *object = objects->item;\n+\t\t\tint pos;\n+\n+\t\t\tpos = bitmap_position(bitmap_git, &object->oid);\n+\t\t\tif (pos < 0)\n+\t\t\t\tcontinue;\n+\n+\t\t\tbitmap_set(roots_bitmap, pos);\n+\t\t}\n+\n+\t\tbase = bitmap_new();\n+\t\tif (!cascade_pseudo_merges_1(bitmap_git, base, roots_bitmap))\n+\t\t\tbitmap_free(roots_bitmap);\n+\t}\n+\n \t/*\n \t * Go through all the roots for the walk. The ones that have bitmaps\n \t * on the bitmap index will be `or`ed together to form an initial\n@@ -1262,11 +1354,21 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t */\n \twhile (roots) {\n \t\tstruct object *object = roots->item;\n+\n \t\troots = roots->next;\n \n+\t\tif (base) {\n+\t\t\tint pos = bitmap_position(bitmap_git, &object->oid);\n+\t\t\tif (pos > 0 && bitmap_get(base, pos)) {\n+\t\t\t\tobject->flags |= SEEN;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n \t\tif (object->type == OBJ_COMMIT &&\n \t\t add_commit_to_bitmap(bitmap_git, &base, (struct commit *)object)) {\n \t\t\tobject->flags |= SEEN;\n+\t\t\texisting_bitmaps = 1;\n \t\t\tcontinue;\n \t\t}\n \n@@ -1282,6 +1384,9 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \n \troots = not_mapped;\n \n+\tif (existing_bitmaps)\n+\t\tcascade_pseudo_merges_1(bitmap_git, base, NULL);\n+\n \t/*\n \t * Let's iterate through all the roots that don't have bitmaps to\n \t * check if we can determine them to be reachable from the existing\n@@ -1866,6 +1971,11 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tobject_list_free(&wants);\n \tobject_list_free(&haves);\n \n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"pseudo_merges_satisfied\",\n+\t\t\t pseudo_merges_satisfied_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"pseudo_merges_cascades\",\n+\t\t\t pseudo_merges_cascades_nr);\n+\n \treturn bitmap_git;\n \n cleanup:\ndiff --git a/t/t5333-pseudo-merge-bitmaps.sh b/t/t5333-pseudo-merge-bitmaps.sh\nnew file mode 100755\nindex 00000000000..3a7dc7278a7\n--- /dev/null\n+++ b/t/t5333-pseudo-merge-bitmaps.sh\n@@ -0,0 +1,323 @@\n+#!/bin/sh\n+\n+test_description='pseudo-merge bitmaps'\n+\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n+. ./test-lib.sh\n+\n+test_pseudo_merges () {\n+\ttest-tool bitmap dump-pseudo-merges\n+}\n+\n+test_pseudo_merge_commits () {\n+\ttest-tool bitmap dump-pseudo-merge-commits \"$1\"\n+}\n+\n+test_pseudo_merges_satisfied () {\n+\ttest_trace2_data bitmap pseudo_merges_satisfied \"$1\"\n+}\n+\n+test_pseudo_merges_cascades () {\n+\ttest_trace2_data bitmap pseudo_merges_cascades \"$1\"\n+}\n+\n+tag_everything () {\n+\tgit rev-list --all --no-object-names >in &&\n+\tperl -lne '\n+\t\tprint \"create refs/tags/\" . $. . \" \" . $1 if /([0-9a-f]+)/\n+\t' expect &&\n+\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\n+\ttest_pseudo_merges_satisfied 0 merges &&\n+\ttest_must_be_empty merges &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'pseudo-merges accurately represent their objects' '\n+\ttest_config bitmapPseudoMerge.test.pattern \"refs/tags/\" &&\n+\ttest_config bitmapPseudoMerge.test.maxMerges 8 &&\n+\ttest_config bitmapPseudoMerge.test.stableThreshold never &&\n+\n+\tgit repack -adb &&\n+\n+\ttest_pseudo_merges >merges &&\n+\ttest_line_count = 8 merges &&\n+\n+\tfor i in $(test_seq 0 $(($(wc -l commits &&\n+\n+\t\tgit rev-list --objects --no-object-names --stdin expect.raw &&\n+\t\ttest-tool bitmap dump-pseudo-merge-objects $i >actual.raw &&\n+\n+\t\tsort -u expect &&\n+\t\tsort -u actual &&\n+\n+\t\ttest_cmp expect actual || return 1\n+\tdone\n+'\n+\n+test_expect_success 'bitmap traversal with pseudo-merges' '\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\tgit rev-list --count --all --objects >expect &&\n+\n+\ttest_pseudo_merges_satisfied 8 trace2.txt &&\n+\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\tgit rev-list --count --all --objects >expect &&\n+\n+\ttest_pseudo_merges_satisfied 8 merges &&\n+\t\ttest_line_count = 1 merges &&\n+\t\ttest_pseudo_merge_commits 0 >commits &&\n+\n+\t\ttest-tool bitmap list-commits >bitmaps &&\n+\t\tbitmaps_nr=\"$(wc -l expect &&\n+\n+\t\ttest $(cat expect) -eq $(wc -l merges &&\n+\t\ttest_line_count = 1 merges &&\n+\n+\t\ttest_pseudo_merge_commits 0 >oids &&\n+\t\tgit cat-file --batch commits &&\n+\n+\t\ttest $(wc -l in &&\n+\t\tgit update-ref --stdin merges &&\n+\t\tmerges_nr=\"$(wc -l oids &&\n+\t\t\tgit cat-file --batch commits &&\n+\n+\t\t\texpect=\"$(grep -c \"^committer.*$old +0000$\" commits)\" &&\n+\t\t\tactual=\"$(wc -l oids &&\n+\t\tgit cat-file --batch commits &&\n+\t\ttest $(wc -l err &&\n+\n+\tcat >expect <<-EOF &&\n+\tfatal: pseudo-merge group ${SQ}test${SQ} has unstable threshold before stable one\n+\tEOF\n+\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'pseudo-merge pattern with capture groups' '\n+\tgit init pseudo-merge-captures &&\n+\t(\n+\t\tcd pseudo-merge-captures &&\n+\n+\t\ttest_commit_bulk 128 &&\n+\t\ttag_everything &&\n+\n+\t\tfor r in $(test_seq 8)\n+\t\tdo\n+\t\t\ttest_commit_bulk 16 &&\n+\n+\t\t\tgit rev-list HEAD~16.. >in &&\n+\n+\t\t\tperl -lne \"print \\\"create refs/remotes/$r/tags/\\$. \\$_\\\"\" refs &&\n+\n+\t\ttest_pseudo_merges >merges &&\n+\t\tfor m in $(test_seq 0 $(($(wc -l oids &&\n+\t\t\tgrep -f oids refs |\n+\t\t\tperl -lne \"print \\$1 if /refs\\/remotes\\/([0-9]+)/\" |\n+\t\t\tsort -u || return 1\n+\t\tdone >remotes &&\n+\n+\t\ttest $(wc -l merges &&\n+\t\ttest_line_count = 2 merges &&\n+\n+\t\ttest_pseudo_merge_commits 0 >commits-0.raw &&\n+\t\ttest_pseudo_merge_commits 1 >commits-1.raw &&\n+\n+\t\tsort commits-0.raw >commits-0 &&\n+\t\tsort commits-1.raw >commits-1 &&\n+\n+\t\tcomm -12 commits-0 commits-1 >overlap &&\n+\n+\t\ttest_line_count -gt 0 overlap\n+\t)\n+'\n+\n+test_expect_success 'pseudo-merge overlap traversal' '\n+\t(\n+\t\tcd pseudo-merge-overlap &&\n+\n+\t\t: >trace2.txt &&\n+\t\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\t\tgit rev-list --count --all --objects >expect &&\n+\n+\t\ttest_pseudo_merges_satisfied 2 trace2.txt &&\n+\t\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\t\tgit rev-list --count --all --objects >expect &&\n+\n+\t\ttest_pseudo_merges_satisfied 2 \n---\n pack-bitmap.c | 26 +++++++++++++++++++++++++-\n 1 file changed, 25 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex e61058dada6..1966b3b95f1 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -116,6 +116,10 @@ struct bitmap_index {\n \n static int pseudo_merges_satisfied_nr;\n static int pseudo_merges_cascades_nr;\n+static int existing_bitmaps_hits_nr;\n+static int existing_bitmaps_misses_nr;\n+static int roots_with_bitmaps_nr;\n+static int roots_without_bitmaps_nr;\n \n static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st)\n {\n@@ -1040,10 +1044,14 @@ static int add_to_include_set(struct bitmap_index *bitmap_git,\n \n \tpartial = bitmap_for_commit(bitmap_git, commit);\n \tif (partial) {\n+\t\texisting_bitmaps_hits_nr++;\n+\n \t\tbitmap_or_ewah(data->base, partial);\n \t\treturn 0;\n \t}\n \n+\texisting_bitmaps_misses_nr++;\n+\n \tbitmap_set(data->base, bitmap_pos);\n \tif (apply_pseudo_merges_for_commit_1(bitmap_git, data->base, commit,\n \t\t\t\t\t bitmap_pos))\n@@ -1099,8 +1107,12 @@ static int add_commit_to_bitmap(struct bitmap_index *bitmap_git,\n {\n \tstruct ewah_bitmap *or_with = bitmap_for_commit(bitmap_git, commit);\n \n-\tif (!or_with)\n+\tif (!or_with) {\n+\t\texisting_bitmaps_misses_nr++;\n \t\treturn 0;\n+\t}\n+\n+\texisting_bitmaps_hits_nr++;\n \n \tif (!*base)\n \t\t*base = ewah_to_bitmap(or_with);\n@@ -1407,8 +1419,12 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\t\tobject->flags &= ~UNINTERESTING;\n \t\t\tadd_pending_object(revs, object, \"\");\n \t\t\tneeds_walk = 1;\n+\n+\t\t\troots_without_bitmaps_nr++;\n \t\t} else {\n \t\t\tobject->flags |= SEEN;\n+\n+\t\t\troots_with_bitmaps_nr++;\n \t\t}\n \t}\n \n@@ -1975,6 +1991,14 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\t pseudo_merges_satisfied_nr);\n \ttrace2_data_intmax(\"bitmap\", the_repository, \"pseudo_merges_cascades\",\n \t\t\t pseudo_merges_cascades_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"bitmap/hits\",\n+\t\t\t existing_bitmaps_hits_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"bitmap/misses\",\n+\t\t\t existing_bitmaps_misses_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"bitmap/roots_with_bitmap\",\n+\t\t\t roots_with_bitmaps_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"bitmap/roots_without_bitmap\",\n+\t\t\t roots_without_bitmaps_nr);\n \n \treturn bitmap_git;\n \n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495206","messageId":"da2fb5b4b48816eb97d2190461f7b3dfa4da917d.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 28/30] ewah: `bitmap_equals_ewah()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:03:10Z","receivedAt":"2024-05-21T19:03:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to reuse existing pseudo-merge bitmaps by implementing a\n`bitmap_equals_ewah()` helper.\n\nThis helper will be used to see if a raw bitmap (containing the set of\nparents for some pseudo-merge) is equal to any existing pseudo-merge's\ncommits bitmap (which are stored as EWAH-compressed bitmaps on disk).\n\nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 19 +++++++++++++++++++\n ewah/ewok.h | 1 +\n 2 files changed, 20 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex dc2ca190f12..55928dada86 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -261,6 +261,25 @@ int bitmap_equals(struct bitmap *self, struct bitmap *other)\n \treturn 1;\n }\n \n+int bitmap_equals_ewah(struct bitmap *self, struct ewah_bitmap *other)\n+{\n+\tstruct ewah_iterator it;\n+\teword_t word;\n+\tsize_t i = 0;\n+\n+\tewah_iterator_init(&it, other);\n+\n+\twhile (ewah_iterator_next(&word, &it))\n+\t\tif (word != (i < self->word_alloc ? self->words[i++] : 0))\n+\t\t\treturn 0;\n+\n+\tfor (; i < self->word_alloc; i++)\n+\t\tif (self->words[i])\n+\t\t\treturn 0;\n+\n+\treturn 1;\n+}\n+\n int bitmap_is_subset(struct bitmap *self, struct bitmap *other)\n {\n \tsize_t common_size, i;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 7074a6347b7..5e357e24933 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -179,6 +179,7 @@ void bitmap_unset(struct bitmap *self, size_t pos);\n int bitmap_get(struct bitmap *self, size_t pos);\n void bitmap_free(struct bitmap *self);\n int bitmap_equals(struct bitmap *self, struct bitmap *other);\n+int bitmap_equals_ewah(struct bitmap *self, struct ewah_bitmap *other);\n \n /*\n * Both `bitmap_is_subset()` and `ewah_bitmap_is_subset()` return 1 if the set\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495207","messageId":"ff21247281ff1c1bd1bb83947cae819e7fc92589.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 29/30] pseudo-merge: implement support for finding existing merges","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:03:14Z","receivedAt":"2024-05-21T19:03:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This patch implements support for reusing existing pseudo-merge commits\nwhen writing bitmaps when there is an existing pseudo-merge bitmap which\nhas exactly the same set of parents as one that we are about to write.\n\nNote that unstable pseudo-merges are likely to change between\nconsecutive repacks, and so are generally poor candidates for reuse.\nHowever, stable pseudo-merges (see the configuration option\n'bitmapPseudoMerge..stableThreshold') are by definition unlikely\nto change between runs (as they represent long-running branches).\n\nBecause there is no index from a *set* of pseudo-merge parents to a\nmatching pseudo-merge bitmap, we have to construct the bitmap\ncorresponding to the set of parents for each pending pseudo-merge commit\nand see if a matching bitmap exists.\n\nThis is technically quadratic in the number of pseudo-merges, but is OK\nin practice for a couple of reasons:\n\n - non-matching pseudo-merge bitmaps are rejected quickly as soon as\n they differ in a single bit\n\n - already-matched pseudo-merge bitmaps are discarded from subsequent\n rounds of search\n\n - the number of pseudo-merges is generally small, even for large\n repositories\n\nIn order to do this, implement (a) a function that finds a matching\npseudo-merge given some uncompressed bitset describing its parents, (b)\na function that computes the bitset of parents for a given pseudo-merge\ncommit, and (c) call that function before computing the set of reachable\nobjects for some pending pseudo-merge.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 15 ++++++--\n pack-bitmap.c | 32 +++++++++++++++++\n pack-bitmap.h | 2 ++\n pseudo-merge.c | 55 ++++++++++++++++++++++++++++\n pseudo-merge.h | 7 ++++\n t/t5333-pseudo-merge-bitmaps.sh | 64 +++++++++++++++++++++++++++++++++\n 6 files changed, 173 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 47250398aa2..6e8060f8a0b 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -19,6 +19,10 @@\n #include \"tree-walk.h\"\n #include \"pseudo-merge.h\"\n #include \"oid-array.h\"\n+#include \"config.h\"\n+#include \"alloc.h\"\n+#include \"refs.h\"\n+#include \"strmap.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -465,6 +469,7 @@ static int fill_bitmap_tree(struct bitmap_writer *writer,\n }\n \n static int reused_bitmaps_nr;\n+static int reused_pseudo_merge_bitmaps_nr;\n \n static int fill_bitmap_commit(struct bitmap_writer *writer,\n \t\t\t struct bb_commit *ent,\n@@ -490,7 +495,7 @@ static int fill_bitmap_commit(struct bitmap_writer *writer,\n \t\t\tstruct bitmap *remapped = bitmap_new();\n \n \t\t\tif (commit->object.flags & BITMAP_PSEUDO_MERGE)\n-\t\t\t\told = NULL;\n+\t\t\t\told = pseudo_merge_bitmap_for_commit(old_bitmap, c);\n \t\t\telse\n \t\t\t\told = bitmap_for_commit(old_bitmap, c);\n \t\t\t/*\n@@ -501,7 +506,10 @@ static int fill_bitmap_commit(struct bitmap_writer *writer,\n \t\t\tif (old && !rebuild_bitmap(mapping, old, remapped)) {\n \t\t\t\tbitmap_or(ent->bitmap, remapped);\n \t\t\t\tbitmap_free(remapped);\n-\t\t\t\treused_bitmaps_nr++;\n+\t\t\t\tif (commit->object.flags & BITMAP_PSEUDO_MERGE)\n+\t\t\t\t\treused_pseudo_merge_bitmaps_nr++;\n+\t\t\t\telse\n+\t\t\t\t\treused_bitmaps_nr++;\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tbitmap_free(remapped);\n@@ -631,6 +639,9 @@ int bitmap_writer_build(struct bitmap_writer *writer,\n \t\t\t the_repository);\n \ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n \t\t\t \"building_bitmaps_reused\", reused_bitmaps_nr);\n+\ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n+\t\t\t \"building_bitmaps_pseudo_merge_reused\",\n+\t\t\t reused_pseudo_merge_bitmaps_nr);\n \n \tstop_progress(&writer->progress);\n \ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 1966b3b95f1..70230e26479 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1316,6 +1316,37 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \treturn cb.base;\n }\n \n+struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct commit *commit)\n+{\n+\tstruct commit_list *p;\n+\tstruct bitmap *parents;\n+\tstruct pseudo_merge *match = NULL;\n+\n+\tif (!bitmap_git->pseudo_merges.nr)\n+\t\treturn NULL;\n+\n+\tparents = bitmap_new();\n+\n+\tfor (p = commit->parents; p; p = p->next) {\n+\t\tint pos = bitmap_position(bitmap_git, &p->item->object.oid);\n+\t\tif (pos < 0 || pos >= bitmap_num_objects(bitmap_git))\n+\t\t\tgoto done;\n+\n+\t\tbitmap_set(parents, pos);\n+\t}\n+\n+\tmatch = pseudo_merge_for_parents(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t\tparents);\n+\n+done:\n+\tbitmap_free(parents);\n+\tif (match)\n+\t\treturn pseudo_merge_bitmap(&bitmap_git->pseudo_merges, match);\n+\n+\treturn NULL;\n+}\n+\n static void unsatisfy_all_pseudo_merges(struct bitmap_index *bitmap_git)\n {\n \tuint32_t i;\n@@ -2809,6 +2840,7 @@ void free_bitmap_index(struct bitmap_index *b)\n \t\t */\n \t\tclose_midx_revindex(b->midx);\n \t}\n+\tfree_pseudo_merge_map(&b->pseudo_merges);\n \tfree(b);\n }\n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 4466b5ad0fb..1171e6d9893 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -142,6 +142,8 @@ int rebuild_bitmap(const uint32_t *reposition,\n \t\t struct bitmap *dest);\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t struct commit *commit);\n+struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct commit *commit);\n void bitmap_writer_select_commits(struct bitmap_writer *writer,\n \t\t\t\t struct commit **indexed_commits,\n \t\t\t\t unsigned int indexed_commits_nr);\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex 0f50ac6183e..36a617f64e6 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -695,3 +695,58 @@ int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n \n \treturn ret;\n }\n+\n+struct pseudo_merge *pseudo_merge_for_parents(const struct pseudo_merge_map *pm,\n+\t\t\t\t\t struct bitmap *parents)\n+{\n+\tstruct pseudo_merge *match = NULL;\n+\tsize_t i;\n+\n+\tif (!pm->nr)\n+\t\treturn NULL;\n+\n+\t/*\n+\t * NOTE: this loop is quadratic in the worst-case (where no\n+\t * matching pseudo-merge bitmaps are found), but in practice\n+\t * this is OK for a few reasons:\n+\t *\n+\t * - Rejecting pseudo-merge bitmaps that do not match the\n+\t * given commit is done quickly (i.e. `bitmap_equals_ewah()`\n+\t * returns early when we know the two bitmaps aren't equal.\n+\t *\n+\t * - Already matched pseudo-merge bitmaps (which we track with\n+\t * the `->satisfied` bit here) are skipped as potential\n+\t * candidates.\n+\t *\n+\t * - The number of pseudo-merges should be small (in the\n+\t * hundreds for most repositories).\n+\t *\n+\t * If in the future this semi-quadratic behavior does become a\n+\t * problem, another approach would be to keep track of which\n+\t * pseudo-merges are still \"viable\" after enumerating the\n+\t * pseudo-merge commit's parents:\n+\t *\n+\t * - A pseudo-merge bitmap becomes non-viable when the bit(s)\n+\t * corresponding to one or more parent(s) of the given\n+\t * commit are not set in a candidate pseudo-merge's commits\n+\t * bitmap.\n+\t *\n+\t * - After processing all bits, enumerate the remaining set of\n+\t * viable pseudo-merge bitmaps, and check that their\n+\t * popcount() matches the number of parents in the given\n+\t * commit.\n+\t */\n+\tfor (i = 0; i < pm->nr; i++) {\n+\t\tstruct pseudo_merge *candidate = use_pseudo_merge(pm, &pm->v[i]);\n+\t\tif (!candidate || candidate->satisfied)\n+\t\t\tcontinue;\n+\t\tif (!bitmap_equals_ewah(parents, candidate->commits))\n+\t\t\tcontinue;\n+\n+\t\tmatch = candidate;\n+\t\tmatch->satisfied = 1;\n+\t\tbreak;\n+\t}\n+\n+\treturn match;\n+}\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nindex c00b622be4b..62fde979015 100644\n--- a/pseudo-merge.h\n+++ b/pseudo-merge.h\n@@ -206,4 +206,11 @@ int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n \t\t\t struct bitmap *result,\n \t\t\t struct bitmap *roots);\n \n+/*\n+ * Returns a pseudo-merge which contains the exact set of commits\n+ * listed in the \"parents\" bitamp, or NULL if none could be found.\n+ */\n+struct pseudo_merge *pseudo_merge_for_parents(const struct pseudo_merge_map *pm,\n+\t\t\t\t\t struct bitmap *parents);\n+\n #endif\ndiff --git a/t/t5333-pseudo-merge-bitmaps.sh b/t/t5333-pseudo-merge-bitmaps.sh\nindex 3a7dc7278a7..7ae4b7a35b7 100755\n--- a/t/t5333-pseudo-merge-bitmaps.sh\n+++ b/t/t5333-pseudo-merge-bitmaps.sh\n@@ -22,6 +22,10 @@ test_pseudo_merges_cascades () {\n \ttest_trace2_data bitmap pseudo_merges_cascades \"$1\"\n }\n \n+test_pseudo_merges_reused () {\n+\ttest_trace2_data pack-bitmap-write building_bitmaps_pseudo_merge_reused \"$1\"\n+}\n+\n tag_everything () {\n \tgit rev-list --all --no-object-names >in &&\n \tperl -lne '\n@@ -320,4 +324,64 @@ test_expect_success 'pseudo-merge overlap stale traversal' '\n \t)\n '\n \n+test_expect_success 'pseudo-merge reuse' '\n+\tgit init pseudo-merge-reuse &&\n+\t(\n+\t\tcd pseudo-merge-reuse &&\n+\n+\t\tstable=\"1641013200\" && # 2022-01-01\n+\t\tunstable=\"1672549200\" && # 2023-01-01\n+\n+\t\tfor date in $stable $unstable\n+\t\tdo\n+\t\t\ttest_commit_bulk --date \"$date +0000\" 128 &&\n+\t\t\ttest_tick || return 1\n+\t\tdone &&\n+\n+\t\ttag_everything &&\n+\n+\t\tgit \\\n+\t\t\t-c bitmapPseudoMerge.test.pattern=\"refs/tags/\" \\\n+\t\t\t-c bitmapPseudoMerge.test.maxMerges=1 \\\n+\t\t\t-c bitmapPseudoMerge.test.threshold=now \\\n+\t\t\t-c bitmapPseudoMerge.test.stableThreshold=$(($unstable - 1)) \\\n+\t\t\t-c bitmapPseudoMerge.test.stableSize=512 \\\n+\t\t\trepack -adb &&\n+\n+\t\ttest_pseudo_merges >merges &&\n+\t\ttest_line_count = 2 merges &&\n+\n+\t\ttest_pseudo_merge_commits 0 >stable-oids.before &&\n+\t\ttest_pseudo_merge_commits 1 >unstable-oids.before &&\n+\n+\t\t: >trace2.txt &&\n+\t\tGIT_TRACE2_EVENT=$PWD/trace2.txt git \\\n+\t\t\t-c bitmapPseudoMerge.test.pattern=\"refs/tags/\" \\\n+\t\t\t-c bitmapPseudoMerge.test.maxMerges=2 \\\n+\t\t\t-c bitmapPseudoMerge.test.threshold=now \\\n+\t\t\t-c bitmapPseudoMerge.test.stableThreshold=$(($unstable - 1)) \\\n+\t\t\t-c bitmapPseudoMerge.test.stableSize=512 \\\n+\t\t\trepack -adb &&\n+\n+\t\ttest_pseudo_merges_reused 1 merges &&\n+\t\ttest_line_count = 3 merges &&\n+\n+\t\ttest_pseudo_merge_commits 0 >stable-oids.after &&\n+\t\tfor i in 1 2\n+\t\tdo\n+\t\t\ttest_pseudo_merge_commits $i || return 1\n+\t\tdone >unstable-oids.after &&\n+\n+\t\tsort -u expect &&\n+\t\tsort -u actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tsort -u expect &&\n+\t\tsort -u actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.45.1.175.gbea44add9db\n\n"},{"id":"495208","messageId":"6a6d88fa512ba344543f5f0df33d5a61e406f3db.1716318089.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"[PATCH v3 30/30] t/perf: implement performace tests for pseudo-merge bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:03:17Z","receivedAt":"2024-05-21T19:03:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement a straightforward performance test demonstrating the benefit\nof pseudo-merge bitmaps by measuring how long it takes to count\nreachable objects in a few different scenarios:\n\n - without bitmaps, to demonstrate a reasonable baseline\n - with bitmaps, but without pseudo-merges\n - with bitmaps and pseudo-merges\n\nResults from running this test on git.git are as follows:\n\n Test this tree\n -----------------------------------------------------------------------------------\n 5333.2: git rev-list --count --all --objects (no bitmaps) 3.46(3.37+0.09)\n 5333.3: git rev-list --count --all --objects (no pseudo-merges) 0.13(0.11+0.01)\n 5333.4: git rev-list --count --all --objects (with pseudo-merges) 0.12(0.11+0.01)\n\nSigned-off-by: Taylor Blau \n---\n t/perf/p5333-pseudo-merge-bitmaps.sh | 32 ++++++++++++++++++++++++++++\n 1 file changed, 32 insertions(+)\n create mode 100755 t/perf/p5333-pseudo-merge-bitmaps.sh\n\ndiff --git a/t/perf/p5333-pseudo-merge-bitmaps.sh b/t/perf/p5333-pseudo-merge-bitmaps.sh\nnew file mode 100755\nindex 00000000000..4bec409d10e\n--- /dev/null\n+++ b/t/perf/p5333-pseudo-merge-bitmaps.sh\n@@ -0,0 +1,32 @@\n+#!/bin/sh\n+\n+test_description='pseudo-merge bitmaps'\n+. ./perf-lib.sh\n+\n+test_perf_large_repo\n+\n+test_expect_success 'setup' '\n+\tgit \\\n+\t\t-c bitmapPseudoMerge.all.pattern=\"refs/\" \\\n+\t\t-c bitmapPseudoMerge.all.threshold=now \\\n+\t\t-c bitmapPseudoMerge.all.stableThreshold=never \\\n+\t\t-c bitmapPseudoMerge.all.maxMerges=64 \\\n+\t\t-c pack.writeBitmapLookupTable=true \\\n+\t\trepack -adb\n+'\n+\n+test_perf 'git rev-list --count --all --objects (no bitmaps)' '\n+\tgit rev-list --objects --all\n+'\n+\n+test_perf 'git rev-list --count --all --objects (no pseudo-merges)' '\n+\tGIT_TEST_USE_PSEDUO_MERGES=0 \\\n+\t\tgit rev-list --objects --all --use-bitmap-index\n+'\n+\n+test_perf 'git rev-list --count --all --objects (with pseudo-merges)' '\n+\tGIT_TEST_USE_PSEDUO_MERGES=1 \\\n+\t\tgit rev-list --objects --all --use-bitmap-index\n+'\n+\n+test_done\n-- \n2.45.1.175.gbea44add9db\n"},{"id":"495209","messageId":"ZkzwnJQRZqFGNBr9@nand.local","threadId":"61166","inReplyTo":"38c96fc1909162a4d9c188f55b7c708cfc1b14b9.1716318089.git.me@ttaylorr.com","subject":"Re: [PATCH v3 01/30] object.h: add flags allocated by pack-bitmap.h","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-21T19:06:04Z","receivedAt":"2024-05-21T19:06:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, May 21, 2024 at 03:01:48PM -0400, Taylor Blau wrote:\n> In commit 7cc8f971085 (pack-objects: implement bitmap writing,\n> 2013-12-21) the NEEDS_BITMAP flag was introduced into pack-bitmap.h, but\n> no object flags allocation table existed at the time.\n\nOops. When I prepared these patches for the list, I told format-patch\nthat the base was origin/master, not @{u}. The correct upstream on which\npatch 7 and onwards is based is 'tb/pack-bitmap-write-cleanups' as I\nwrote in the cover letter.\n\nJunio: please ignore this patch and apply patches 7-30 on top of your\n'tb/pack-bitmap-write-cleanups'.\n\nThanks,\nTaylor\n"},{"id":"495372","messageId":"20240523100225.GA1308330@coredump.intra.peff.net","threadId":"61166","inReplyTo":"3070135eb4b9bd16117e82f1817c112c56a24b55.1716318089.git.me@ttaylorr.com","subject":"Re: [PATCH v3 16/30] config: introduce git_config_float()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-23T10:02:25Z","receivedAt":"2024-05-23T10:02:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 21, 2024 at 03:02:29PM -0400, Taylor Blau wrote:\n\n> Future commits will want to parse a floating point value from\n> configuration, but we have no way to parse such a value prior to this\n> patch.\n> \n> The core of the routine is implemented in git_parse_float(). Unlike\n> git_parse_unsigned() and git_parse_signed(), however, the function\n> implemented here only works on type \"float\", and not related types like\n> \"double\", or \"long double\".\n> \n> This is because \"double\" and \"long double\" use different functions to\n> convert from ASCII strings to floating point values (strtod() and\n> strtold(), respectively). Likewise, there is no pointer type that can\n> assign to any of these values (except for \"void *\"), so the only way to\n> define this trio of functions would be with a macro expansion that is\n> parameterized over the floating point type and conversion function.\n\nI agree it doesn't make sense to support both. But if we have to choose\nbetween the two, should we just use \"double\"?\n\nI doubt you need the extra precision for your case, but I also doubt\nthat the speed/storage benefits of \"float\" would matter. And support for\n\"float\" in C is kind of weird. There is no \"float\" specifier for printf.\nAnd according to my copy of strtof(3), until C99 we only had strtod()!\n\n\nRegarding using non-integers at all, I do wonder how much we need them.\nWe've usually stuck to integers in other spots, even if it means a sort\nof pseudo-fixed-point (e.g., rename scores). Looking ahead, you're using\nthese for the power-series knobs. I guess it would be pretty confusing\nto try to force integers there. I dunno. Not really an objection, but I\njust wonder if it was something you considered.\n\n-Peff\n"},{"id":"495373","messageId":"20240523101248.GB1308330@coredump.intra.peff.net","threadId":"61166","inReplyTo":"3029473c094b3edf51828b7a1d1acfc8e959ece6.1716318089.git.me@ttaylorr.com","subject":"Re: [PATCH v3 17/30] pseudo-merge: implement support for selecting pseudo-merge commits","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-23T10:12:48Z","receivedAt":"2024-05-23T10:12:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 21, 2024 at 03:02:32PM -0400, Taylor Blau wrote:\n\n> diff --git a/Documentation/config/bitmap-pseudo-merge.txt b/Documentation/config/bitmap-pseudo-merge.txt\n> new file mode 100644\n> index 00000000000..d4a2023b84a\n> --- /dev/null\n> +++ b/Documentation/config/bitmap-pseudo-merge.txt\n> @@ -0,0 +1,90 @@\n> +NOTE: The configuration options in `bitmapPseudoMerge.*` are considered\n> +EXPERIMENTAL and may be subject to change or be removed entirely in the\n> +future.\n\nI'm happy to see this all marked as experimental. We really don't know\nwhat selection approach will be best yet, and it's good not to lock\nourselves in.\n\nI wasn't sure how this would format via asciidoc, since we're in the\nmiddle of a list of variables. It...kind of looks like the note goes\nunder the previous entry (for attr.tree) in the text manpage. Though\nlooking at the docbook, I _think_ it's actually outside of that, and\nit's just how the roff ends up indenting it?\n\nI don't know if it's worth spending too much time on, but maybe there's\nan easy way to do it differently (I couldn't think of one).\n\nThis paragraph might also be a good place to refer to gitpacking(7).\n\n> diff --git a/Documentation/gitpacking.txt b/Documentation/gitpacking.txt\n> index ff18077129b..1ed645ff910 100644\n> --- a/Documentation/gitpacking.txt\n> +++ b/Documentation/gitpacking.txt\n\nThanks, I think this new file and the content you added address all of\nmy documentation complaints.\n\nI think it will also be a good place to discuss bitmaps in general, but\nyou were IMHO wise to not do that in this series, which is already quite\nbig. :)\n\n-Peff\n"},{"id":"495376","messageId":"20240523104000.GC1308330@coredump.intra.peff.net","threadId":"61166","inReplyTo":"8ba0a9c5402fb154bc316768a8fbb016e302a686.1716318089.git.me@ttaylorr.com","subject":"Re: [PATCH v3 22/30] pseudo-merge: implement support for reading pseudo-merge commits","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-23T10:40:00Z","receivedAt":"2024-05-23T10:40:01Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 21, 2024 at 03:02:48PM -0400, Taylor Blau wrote:\n\n> These functions are all documented in pseudo-merge.h, but their rough\n> descriptions are as follows:\n> \n> - pseudo_merge_bitmap() reads and inflates the objects EWAH bitmap for\n> a given pseudo-merge\n> \n> - use_pseudo_merge() does the same as pseudo_merge_bitmap(), but on\n> the commits EWAH bitmap, not the objects bitmap\n> \n> - apply_pseudo_merges_for_commit() applies all satisfied pseudo-merge\n> commits for a given result set, and cascades any yet-unsatisfied\n> pseudo-merges if any were applied in the previous step\n> \n> - cascade_pseudo_merges() applies all pseudo-merges which are\n> satisfied but have not been previously applied, repeating this\n> process until no more pseudo-merges can be applied\n\nOK, so I think this commit is getting into the meat of how the new\nbitmaps will be used. Just to restate it from a high-level to make sure\nI understand, I think it is:\n\n 1. When we are traversing (or even before we traverse and just know\n our tips), we can always say \"hey, I have a commit in the bitmap;\n does this satisfy any pseudo-merges?\". Where \"satisfy\" is \"all of\n the commits pseudo-merged for that bitmap are already in our\n result\". And if so, then we can use the pseudo-merge bitmap by\n OR-ing it in.\n\n And that's apply_pseudo_merges_for_commit().\n\n 2. That \"OR\" operation may likewise open up new options, so we\n recurse. And that's the \"cascade\" function.\n\nThis commit is not yet adding the calls into this code for part (1). I\nthink there's an open question there of overhead; i.e., how expensive it\nis to check whether each pseudo-merge is satisfied. And whether it makes\nsense to just do it once at the beginning of the traversal (with the\nprovided tips) or to keep checking as we traverse (more expensive, but\nit makes it more likely to use an older pseudo-merge bitmap that's had\nnew history built on top of some of its commits).\n\nBut those calls should come later, so let's read on.\n\n> +static int pseudo_merge_ext_at(const struct pseudo_merge_map *pm,\n> +\t\t\t struct pseudo_merge_commit_ext *ext, size_t at)\n> +{\n> +\tif (at >= pm->map_size)\n> +\t\treturn error(_(\"extended pseudo-merge read out-of-bounds \"\n> +\t\t\t \"(%\"PRIuMAX\" >= %\"PRIuMAX\")\"),\n> +\t\t\t (uintmax_t)at, (uintmax_t)pm->map_size);\n> +\n> +\text->nr = get_be32(pm->map + at);\n> +\text->ptr = pm->map + at + sizeof(uint32_t);\n> +\n> +\treturn 0;\n> +}\n\nI was happy to see the boundary check here. Do we need a length check,\ntoo? We'd need at least four bytes here for the uint32_t. Does map_size\ninclude the trailing hash? If not, then it might provide a bit of slop\n(we'd read garbage, but never go outside the mmap).\n\nI guess the \">=\" in the size check implies that we have at least one\nbyte, but I don't think anything promises that we're correctly 4-byte\naligned.\n\nThe rest of the length check is here:\n\n> +struct ewah_bitmap *pseudo_merge_bitmap(const struct pseudo_merge_map *pm,\n> +\t\t\t\t\tstruct pseudo_merge *merge)\n> +{\n> +\tif (!merge->loaded_commits)\n> +\t\tBUG(\"cannot use unloaded pseudo-merge bitmap\");\n> +\n> +\tif (!merge->loaded_bitmap) {\n> +\t\tsize_t at = merge->bitmap_at;\n> +\n> +\t\tmerge->bitmap = read_bitmap(pm->map, pm->map_size, &at);\n> +\t\tmerge->loaded_bitmap = 1;\n> +\t}\n> +\n> +\treturn merge->bitmap;\n> +}\n\nWhen we call read_bitmap(), it knows where the end is, and it's\ncareful to avoid reading past it. Good.\n\n-Peff\n"},{"id":"495377","messageId":"20240523104224.GD1308330@coredump.intra.peff.net","threadId":"61166","inReplyTo":"890f6c4b9deb9e3bf02aa180c7ad4ced7f7b6a80.1716318089.git.me@ttaylorr.com","subject":"Re: [PATCH v3 25/30] t/test-lib-functions.sh: support `--date` in `test_commit_bulk()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-23T10:42:24Z","receivedAt":"2024-05-23T10:42:26Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 21, 2024 at 03:02:59PM -0400, Taylor Blau wrote:\n\n> diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\n> index 862d80c9748..16fd585e34b 100644\n> --- a/t/test-lib-functions.sh\n> +++ b/t/test-lib-functions.sh\n> @@ -458,6 +458,7 @@ test_commit_bulk () {\n> \tindir=.\n> \tref=HEAD\n> \tn=1\n> +\tnotick=\n> \tmessage='commit %s'\n> \tfilename='%s.t'\n> \tcontents='content %s'\n> @@ -488,6 +489,12 @@ test_commit_bulk () {\n> \t\t\tfilename=\"${1#--*=}-%s.t\"\n> \t\t\tcontents=\"${1#--*=} %s\"\n> \t\t\t;;\n> +\t\t--date)\n> +\t\t\tnotick=yes\n> +\t\t\tGIT_COMMITTER_DATE=\"$2\"\n> +\t\t\tGIT_AUTHOR_DATE=\"$2\"\n> +\t\t\tshift\n> +\t\t\t;;\n\nThis gives all of the bulk commits the same date. Which is kind of\nunrealistic. Conceivably you'd want to be set the starting date at some\nold spot, and then tick forward from there. It may not matter much in\npractice, though.\n\n-Peff\n"},{"id":"495378","messageId":"20240523104837.GE1308330@coredump.intra.peff.net","threadId":"61166","inReplyTo":"41691824f78818f3c70cad6d02cd7f66d12c68c3.1716318089.git.me@ttaylorr.com","subject":"Re: [PATCH v3 26/30] pack-bitmap.c: use pseudo-merges during traversal","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-23T10:48:37Z","receivedAt":"2024-05-23T10:48:38Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 21, 2024 at 03:03:03PM -0400, Taylor Blau wrote:\n\n> The basic operation is as follows:\n> \n> - When enumerating objects on either side of a reachability query,\n> first see if any subset of the roots satisfies some pseudo-merge\n> bitmap. If it does, apply that pseudo-merge bitmap.\n> \n> - If any pseudo-merge bitmap(s) were applied in the previous step, OR\n> them into the result[^1]. Then repeat the process over all\n> pseudo-merge bitmaps (we'll refer to this as \"cascading\"\n> pseudo-merges). Once this is done, OR in the resulting bitmap.\n> \n> - If there is no fill-in traversal to be done, return the bitmap for\n> that side of the reachability query. If there is fill-in traversal,\n> then for each commit we encounter via show_commit(), check to see if\n> any unsatisfied pseudo-merges containing that commit as one of its\n> parents has been made satisfied by the presence of that commit.\n> \n> If so, OR in the object set from that pseudo-merge bitmap, and then\n> cascade. If not, continue traversal.\n\nAh, OK. This is the high-level overview I was looking for in the earlier\ncommit. ;) I think it is fine here. I just hadn't gotten to it yet (and\nI think it is much better stated than what I wrote in my earlier\nresponse).\n\n> [^1]: Importantly, we cannot OR in the entire set of roots along with\n> the objects reachable from whatever pseudo-merge bitmaps were\n> satisfied. This may leave some dangling bits corresponding to any\n> unsatisfied root(s) getting OR'd into the resulting bitmap, tricking\n> other parts of the traversal into thinking we already have a\n> reachability closure over those commit(s) when we do not.\n\nI think I know how you made this realization. :)\n\nThe code itself looks as I'd expect, and the entry points are nice and\nclean.\n\n-Peff\n"},{"id":"495379","messageId":"20240523105422.GF1308330@coredump.intra.peff.net","threadId":"61166","inReplyTo":"6a6d88fa512ba344543f5f0df33d5a61e406f3db.1716318089.git.me@ttaylorr.com","subject":"Re: [PATCH v3 30/30] t/perf: implement performace tests for pseudo-merge bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-23T10:54:22Z","receivedAt":"2024-05-23T10:54:23Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 21, 2024 at 03:03:17PM -0400, Taylor Blau wrote:\n\n> Implement a straightforward performance test demonstrating the benefit\n> of pseudo-merge bitmaps by measuring how long it takes to count\n> reachable objects in a few different scenarios:\n> \n> - without bitmaps, to demonstrate a reasonable baseline\n> - with bitmaps, but without pseudo-merges\n> - with bitmaps and pseudo-merges\n> \n> Results from running this test on git.git are as follows:\n> \n> Test this tree\n> -----------------------------------------------------------------------------------\n> 5333.2: git rev-list --count --all --objects (no bitmaps) 3.46(3.37+0.09)\n> 5333.3: git rev-list --count --all --objects (no pseudo-merges) 0.13(0.11+0.01)\n> 5333.4: git rev-list --count --all --objects (with pseudo-merges) 0.12(0.11+0.01)\n\nThat's not a very exciting result. I have a feeling that your git.git is\nnot a very interesting test case. We'd want a lot of refs, and ones that\nare old and have bushy history that is not included in the more recent\nbranches. So something like a bunch of old unmerged pull request heads,\nfor example. ;) Do you have more interesting numbers for something like\nthat?\n\n-Peff\n"},{"id":"495380","messageId":"20240523110532.GA1326297@coredump.intra.peff.net","threadId":"61166","inReplyTo":"cover.1716318088.git.me@ttaylorr.com","subject":"Re: [PATCH v3 00/30] pack-bitmap: pseudo-merge reachability bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-23T11:05:32Z","receivedAt":"2024-05-23T11:05:35Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 21, 2024 at 03:01:38PM -0400, Taylor Blau wrote:\n\n> Here is another reroll my topic to introduce pseudo-merge bitmaps.\n\nOK, I got through the whole thing. I left a few small comments, but\nmostly just observations. Overall, the shape of it looks pretty good.\nThe much bigger question to me is: does it work?\n\nThe perf results you showed at the end are underwhelming, but I think\nthat's mostly because it's not an interesting repository. I think it\nwould be nice to see at least some point-in-time benchmarks on a\nsingle repository.\n\nBut much more interesting to me is how it performs in the real world in\naggregate, over time:\n\n - how often / how much do pseudo-merges speed up queries in the real\n world. Clones/fetches, but also reachability queries. Could\n connectivity checks use bitmaps with this?\n\n - how often do pseudo-merge groups get invalidated by refs changing\n (and thus we lose the speedups from above)?\n\n - what's the cost like to generate them initially?\n\n - what's the cost like for subsequent repacks? Does the selection /\n grouping algorithm do a good job of keeping the older, larger groups\n stable (so that we can reuse them verbatim)?\n\nI know you don't have those answers yet, and I know there's some\nchicken-and-egg with getting this integrated so that you can start to\nexplore that. So I mostly reviewed this with an eye towards:\n\n - does the idea make sense (I think it does, but I'm kind of biased)\n\n - are the patches going to hurt anybody who isn't using the new\n feature (I think the answer is no)\n\n - does the on-disk representation seem right, since that is hard to\n change later (I didn't see any issues)\n\n - does the implementation look clean and plausibly correct (yes, but\n what I'm getting at is that I didn't pore over all of the new code\n with a microscope. Mostly I think the proof is in the pudding that\n it provides the same correct answers more quickly).\n\nSo to my eyes it looks good to move forward and let people start playing\nwith it. The big \"experimental\" warning in the config is good. Maybe\nwe'd want want in gitpacking(7), too?\n\nI did wonder briefly what the backup plan is if there are problems with\nthe on-disk format (or worst case, if it turns out to be a dead end).\nWe've allocated a flag bit which I think we'd need to respect forever\n(though as an optional extension, it's OK to understand and ignore it).\nIf we needed a \"pseudo-merge bitmaps v2\", how would that work? I think\nwe'd have to allocate another bit in the flags field.\n\nI wonder if the start of the pseudo-merge section should have a 4-byte\nversion/flags field itself? I don't think that's something we've done\nbefore, and maybe it's overkill. I dunno. It's just a lot easier to do\nnow than later.\n\n-Peff\n"},{"id":"495413","messageId":"xmqqfru8wmzv.fsf@gitster.g","threadId":"61166","inReplyTo":"20240523104224.GD1308330@coredump.intra.peff.net","subject":"Re: [PATCH v3 25/30] t/test-lib-functions.sh: support `--date` in `test_commit_bulk()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-23T15:45:24Z","receivedAt":"2024-05-23T15:45:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Tue, May 21, 2024 at 03:02:59PM -0400, Taylor Blau wrote:\n>\n>> diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\n>> index 862d80c9748..16fd585e34b 100644\n>> --- a/t/test-lib-functions.sh\n>> +++ b/t/test-lib-functions.sh\n>> @@ -458,6 +458,7 @@ test_commit_bulk () {\n>> \tindir=.\n>> \tref=HEAD\n>> \tn=1\n>> +\tnotick=\n>> \tmessage='commit %s'\n>> \tfilename='%s.t'\n>> \tcontents='content %s'\n>> @@ -488,6 +489,12 @@ test_commit_bulk () {\n>> \t\t\tfilename=\"${1#--*=}-%s.t\"\n>> \t\t\tcontents=\"${1#--*=} %s\"\n>> \t\t\t;;\n>> +\t\t--date)\n>> +\t\t\tnotick=yes\n>> +\t\t\tGIT_COMMITTER_DATE=\"$2\"\n>> +\t\t\tGIT_AUTHOR_DATE=\"$2\"\n>> +\t\t\tshift\n>> +\t\t\t;;\n>\n> This gives all of the bulk commits the same date. Which is kind of\n> unrealistic.\n\nYeah, giving this helper function a \"--notick\" option, without\nadding this \"--date\" option, is a better design, I suspect.\n\nThe callers Taylor expected to use --date can set the _DATE\nvariables and pass \"--notick\", and callers that want the same\ntimestamp without caring which exact timestamp can just pass\n\"--notick\" without futzing with _DATE variables.\n\nThanks.\n"},{"id":"495438","messageId":"Zk+CIoMaF/fT3ejo@nand.local","threadId":"61166","inReplyTo":"20240523100225.GA1308330@coredump.intra.peff.net","subject":"Re: [PATCH v3 16/30] config: introduce git_config_float()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T17:51:30Z","receivedAt":"2024-05-23T17:51:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, May 23, 2024 at 06:02:25AM -0400, Jeff King wrote:\n> On Tue, May 21, 2024 at 03:02:29PM -0400, Taylor Blau wrote:\n>\n> > Future commits will want to parse a floating point value from\n> > configuration, but we have no way to parse such a value prior to this\n> > patch.\n> >\n> > The core of the routine is implemented in git_parse_float(). Unlike\n> > git_parse_unsigned() and git_parse_signed(), however, the function\n> > implemented here only works on type \"float\", and not related types like\n> > \"double\", or \"long double\".\n> >\n> > This is because \"double\" and \"long double\" use different functions to\n> > convert from ASCII strings to floating point values (strtod() and\n> > strtold(), respectively). Likewise, there is no pointer type that can\n> > assign to any of these values (except for \"void *\"), so the only way to\n> > define this trio of functions would be with a macro expansion that is\n> > parameterized over the floating point type and conversion function.\n>\n> I agree it doesn't make sense to support both. But if we have to choose\n> between the two, should we just use \"double\"?\n\nYeah, I share your feeling that there is no great need to support\ndouble-precision floats here, but there's also no reason to artificially\nlimit ourselves to single-precision ones, either.\n\n> Regarding using non-integers at all, I do wonder how much we need them.\n> We've usually stuck to integers in other spots, even if it means a sort\n> of pseudo-fixed-point (e.g., rename scores). Looking ahead, you're using\n> these for the power-series knobs. I guess it would be pretty confusing\n> to try to force integers there. I dunno. Not really an objection, but I\n> just wonder if it was something you considered.\n\nI had originally written the series that way, but Patrick suggested that\nI use floating point numbers instead ;-).\n\nThanks,\nTaylor\n"},{"id":"495439","messageId":"Zk+DW4uBXDSd5xAf@nand.local","threadId":"61166","inReplyTo":"20240523101248.GB1308330@coredump.intra.peff.net","subject":"Re: [PATCH v3 17/30] pseudo-merge: implement support for selecting pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T17:56:43Z","receivedAt":"2024-05-23T17:56:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, May 23, 2024 at 06:12:48AM -0400, Jeff King wrote:\n> On Tue, May 21, 2024 at 03:02:32PM -0400, Taylor Blau wrote:\n>\n> > diff --git a/Documentation/config/bitmap-pseudo-merge.txt b/Documentation/config/bitmap-pseudo-merge.txt\n> > new file mode 100644\n> > index 00000000000..d4a2023b84a\n> > --- /dev/null\n> > +++ b/Documentation/config/bitmap-pseudo-merge.txt\n> > @@ -0,0 +1,90 @@\n> > +NOTE: The configuration options in `bitmapPseudoMerge.*` are considered\n> > +EXPERIMENTAL and may be subject to change or be removed entirely in the\n> > +future.\n>\n> I'm happy to see this all marked as experimental. We really don't know\n> what selection approach will be best yet, and it's good not to lock\n> ourselves in.\n>\n> I wasn't sure how this would format via asciidoc, since we're in the\n> middle of a list of variables. It...kind of looks like the note goes\n> under the previous entry (for attr.tree) in the text manpage. Though\n> looking at the docbook, I _think_ it's actually outside of that, and\n> it's just how the roff ends up indenting it?\n\nI think that you're right that this is a roff indentation thing. Looking\nat the html version rendered by asciidoc, it places the NOTE in an\nadmonition block [1] that is separate from the previous attr.tree entry.\n\n> I don't know if it's worth spending too much time on, but maybe there's\n> an easy way to do it differently (I couldn't think of one).\n>\n> This paragraph might also be a good place to refer to gitpacking(7).\n\nGreat idea, thanks.\n\n> > diff --git a/Documentation/gitpacking.txt b/Documentation/gitpacking.txt\n> > index ff18077129b..1ed645ff910 100644\n> > --- a/Documentation/gitpacking.txt\n> > +++ b/Documentation/gitpacking.txt\n>\n> Thanks, I think this new file and the content you added address all of\n> my documentation complaints.\n\nGood :-).\n\n> I think it will also be a good place to discuss bitmaps in general, but\n> you were IMHO wise to not do that in this series, which is already quite\n> big. :)\n\nThanks. Yeah, I would like to add more about bitmaps and other advanced\npacking concepts in general to gitpacking(7), but let's do so outside of\nthis already-gigantic series ;-).\n\nThanks,\nTaylor\n\n[1]: https://docs.asciidoctor.org/asciidoc/latest/blocks/admonitions/\n"},{"id":"495443","messageId":"Zk+GV1M815zuE8aU@nand.local","threadId":"61166","inReplyTo":"20240523104000.GC1308330@coredump.intra.peff.net","subject":"Re: [PATCH v3 22/30] pseudo-merge: implement support for reading pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T18:09:27Z","receivedAt":"2024-05-23T18:09:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, May 23, 2024 at 06:40:00AM -0400, Jeff King wrote:\n> OK, so I think this commit is getting into the meat of how the new\n> bitmaps will be used. Just to restate it from a high-level to make sure\n> I understand, I think it is:\n>\n> 1. When we are traversing (or even before we traverse and just know\n> our tips), we can always say \"hey, I have a commit in the bitmap;\n> does this satisfy any pseudo-merges?\". Where \"satisfy\" is \"all of\n> the commits pseudo-merged for that bitmap are already in our\n> result\". And if so, then we can use the pseudo-merge bitmap by\n> OR-ing it in.\n>\n> And that's apply_pseudo_merges_for_commit().\n>\n> 2. That \"OR\" operation may likewise open up new options, so we\n> recurse. And that's the \"cascade\" function.\n\nExactly. I think implicit in the above is that your (2) is also a\nrecursive step, since each cascade step may open us up to new\npseudo-merges, which themselves may reach objects which satisfy other\npseudo-merges, and so on.\n\n> > +static int pseudo_merge_ext_at(const struct pseudo_merge_map *pm,\n> > +\t\t\t struct pseudo_merge_commit_ext *ext, size_t at)\n> > +{\n> > +\tif (at >= pm->map_size)\n> > +\t\treturn error(_(\"extended pseudo-merge read out-of-bounds \"\n> > +\t\t\t \"(%\"PRIuMAX\" >= %\"PRIuMAX\")\"),\n> > +\t\t\t (uintmax_t)at, (uintmax_t)pm->map_size);\n> > +\n> > +\text->nr = get_be32(pm->map + at);\n> > +\text->ptr = pm->map + at + sizeof(uint32_t);\n> > +\n> > +\treturn 0;\n> > +}\n>\n> I was happy to see the boundary check here. Do we need a length check,\n> too? We'd need at least four bytes here for the uint32_t. Does map_size\n> include the trailing hash? If not, then it might provide a bit of slop\n> (we'd read garbage, but never go outside the mmap).\n>\n> I guess the \">=\" in the size check implies that we have at least one\n> byte, but I don't think anything promises that we're correctly 4-byte\n> aligned.\n\nYeah, we could read into the trailing hash area, which would just be\ngarbage from our perspective. But I think that adding a length check is\neasy enough to do, something like:\n\n--- 8< ---\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex b539791396..7d13101149 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -478,6 +478,10 @@ static int pseudo_merge_ext_at(const struct pseudo_merge_map *pm,\n \t\treturn error(_(\"extended pseudo-merge read out-of-bounds \"\n \t\t\t \"(%\"PRIuMAX\" >= %\"PRIuMAX\")\"),\n \t\t\t (uintmax_t)at, (uintmax_t)pm->map_size);\n+\tif (at + 4 >= pm->map_size)\n+\t\treturn error(_(\"extended pseudo-merge entry is too short \"\n+\t\t\t \"(%\"PRIuMAX\" >= %\"PRIuMAX\")\"),\n+\t\t\t (uintmax_t)(at + 4), (uintmax_t)pm->map_size);\n\n \text->nr = get_be32(pm->map + at);\n \text->ptr = pm->map + at + sizeof(uint32_t);\n--- >8 ---\n\n> The rest of the length check is here:\n>\n> > +struct ewah_bitmap *pseudo_merge_bitmap(const struct pseudo_merge_map *pm,\n> > +\t\t\t\t\tstruct pseudo_merge *merge)\n> > +{\n> > +\tif (!merge->loaded_commits)\n> > +\t\tBUG(\"cannot use unloaded pseudo-merge bitmap\");\n> > +\n> > +\tif (!merge->loaded_bitmap) {\n> > +\t\tsize_t at = merge->bitmap_at;\n> > +\n> > +\t\tmerge->bitmap = read_bitmap(pm->map, pm->map_size, &at);\n> > +\t\tmerge->loaded_bitmap = 1;\n> > +\t}\n> > +\n> > +\treturn merge->bitmap;\n> > +}\n>\n> When we call read_bitmap(), it knows where the end is, and it's\n> careful to avoid reading past it. Good.\n\nYep, thanks for double checking.\n\nThanks,\nTaylor\n"},{"id":"495444","messageId":"Zk+Jh50esXx7DSbL@nand.local","threadId":"61166","inReplyTo":"xmqqfru8wmzv.fsf@gitster.g","subject":"Re: [PATCH v3 25/30] t/test-lib-functions.sh: support `--date` in `test_commit_bulk()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T18:23:03Z","receivedAt":"2024-05-23T18:23:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, May 23, 2024 at 08:45:24AM -0700, Junio C Hamano wrote:\n> Jeff King writes:\n>\n> > On Tue, May 21, 2024 at 03:02:59PM -0400, Taylor Blau wrote:\n> >\n> >> diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\n> >> index 862d80c9748..16fd585e34b 100644\n> >> --- a/t/test-lib-functions.sh\n> >> +++ b/t/test-lib-functions.sh\n> >> @@ -458,6 +458,7 @@ test_commit_bulk () {\n> >> \tindir=.\n> >> \tref=HEAD\n> >> \tn=1\n> >> +\tnotick=\n> >> \tmessage='commit %s'\n> >> \tfilename='%s.t'\n> >> \tcontents='content %s'\n> >> @@ -488,6 +489,12 @@ test_commit_bulk () {\n> >> \t\t\tfilename=\"${1#--*=}-%s.t\"\n> >> \t\t\tcontents=\"${1#--*=} %s\"\n> >> \t\t\t;;\n> >> +\t\t--date)\n> >> +\t\t\tnotick=yes\n> >> +\t\t\tGIT_COMMITTER_DATE=\"$2\"\n> >> +\t\t\tGIT_AUTHOR_DATE=\"$2\"\n> >> +\t\t\tshift\n> >> +\t\t\t;;\n> >\n> > This gives all of the bulk commits the same date. Which is kind of\n> > unrealistic.\n>\n> Yeah, giving this helper function a \"--notick\" option, without\n> adding this \"--date\" option, is a better design, I suspect.\n\nAgreed. Thanks, both.\n\nThanks,\nTaylor\n"},{"id":"495445","messageId":"Zk+JvTSU671BiouV@nand.local","threadId":"61166","inReplyTo":"20240523104837.GE1308330@coredump.intra.peff.net","subject":"Re: [PATCH v3 26/30] pack-bitmap.c: use pseudo-merges during traversal","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T18:23:57Z","receivedAt":"2024-05-23T18:24:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, May 23, 2024 at 06:48:37AM -0400, Jeff King wrote:\n> On Tue, May 21, 2024 at 03:03:03PM -0400, Taylor Blau wrote:\n>\n> > The basic operation is as follows:\n> >\n> > - When enumerating objects on either side of a reachability query,\n> > first see if any subset of the roots satisfies some pseudo-merge\n> > bitmap. If it does, apply that pseudo-merge bitmap.\n> >\n> > - If any pseudo-merge bitmap(s) were applied in the previous step, OR\n> > them into the result[^1]. Then repeat the process over all\n> > pseudo-merge bitmaps (we'll refer to this as \"cascading\"\n> > pseudo-merges). Once this is done, OR in the resulting bitmap.\n> >\n> > - If there is no fill-in traversal to be done, return the bitmap for\n> > that side of the reachability query. If there is fill-in traversal,\n> > then for each commit we encounter via show_commit(), check to see if\n> > any unsatisfied pseudo-merges containing that commit as one of its\n> > parents has been made satisfied by the presence of that commit.\n> >\n> > If so, OR in the object set from that pseudo-merge bitmap, and then\n> > cascade. If not, continue traversal.\n>\n> Ah, OK. This is the high-level overview I was looking for in the earlier\n> commit. ;) I think it is fine here. I just hadn't gotten to it yet (and\n> I think it is much better stated than what I wrote in my earlier\n> response).\n\nGood, thanks.\n\n> > [^1]: Importantly, we cannot OR in the entire set of roots along with\n> > the objects reachable from whatever pseudo-merge bitmaps were\n> > satisfied. This may leave some dangling bits corresponding to any\n> > unsatisfied root(s) getting OR'd into the resulting bitmap, tricking\n> > other parts of the traversal into thinking we already have a\n> > reachability closure over those commit(s) when we do not.\n>\n> I think I know how you made this realization. :)\n\n;-).\n\nThanks,\nTaylor\n"},{"id":"495449","messageId":"Zk+epE2S4KErPyVk@nand.local","threadId":"61166","inReplyTo":"20240523105422.GF1308330@coredump.intra.peff.net","subject":"Re: [PATCH v3 30/30] t/perf: implement performace tests for pseudo-merge bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T19:53:08Z","receivedAt":"2024-05-23T19:53:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, May 23, 2024 at 06:54:22AM -0400, Jeff King wrote:\n> On Tue, May 21, 2024 at 03:03:17PM -0400, Taylor Blau wrote:\n>\n> > Implement a straightforward performance test demonstrating the benefit\n> > of pseudo-merge bitmaps by measuring how long it takes to count\n> > reachable objects in a few different scenarios:\n> >\n> > - without bitmaps, to demonstrate a reasonable baseline\n> > - with bitmaps, but without pseudo-merges\n> > - with bitmaps and pseudo-merges\n> >\n> > Results from running this test on git.git are as follows:\n> >\n> > Test this tree\n> > -----------------------------------------------------------------------------------\n> > 5333.2: git rev-list --count --all --objects (no bitmaps) 3.46(3.37+0.09)\n> > 5333.3: git rev-list --count --all --objects (no pseudo-merges) 0.13(0.11+0.01)\n> > 5333.4: git rev-list --count --all --objects (with pseudo-merges) 0.12(0.11+0.01)\n>\n> That's not a very exciting result.\n\nI suspect some of it might have to do with:\n\n--- 8< ---\ndiff --git a/t/perf/p5333-pseudo-merge-bitmaps.sh b/t/perf/p5333-pseudo-merge-bitmaps.sh\nindex 4bec409d10..2e8b1d2635 100755\n--- a/t/perf/p5333-pseudo-merge-bitmaps.sh\n+++ b/t/perf/p5333-pseudo-merge-bitmaps.sh\n@@ -20,12 +20,12 @@ test_perf 'git rev-list --count --all --objects (no bitmaps)' '\n '\n\n test_perf 'git rev-list --count --all --objects (no pseudo-merges)' '\n-\tGIT_TEST_USE_PSEDUO_MERGES=0 \\\n+\tGIT_TEST_USE_PSEUDO_MERGES=0 \\\n \t\tgit rev-list --objects --all --use-bitmap-index\n '\n\n test_perf 'git rev-list --count --all --objects (with pseudo-merges)' '\n-\tGIT_TEST_USE_PSEDUO_MERGES=1 \\\n+\tGIT_TEST_USE_PSEUDO_MERGES=1 \\\n \t\tgit rev-list --objects --all --use-bitmap-index\n '\n--- > 8---\n\nSure enough, that shows us a little gap between the \"no pseudo-merges\"\nand \"with pseudo-merges\" case:\n\n```\nTest this tree\n-----------------------------------------------------------------------------------\n5333.2: git rev-list --count --all --objects (no bitmaps) 3.54(3.45+0.08)\n5333.3: git rev-list --count --all --objects (no pseudo-merges) 0.43(0.40+0.03)\n5333.4: git rev-list --count --all --objects (with pseudo-merges) 0.12(0.11+0.01)\n```\n\n> I have a feeling that your git.git is not a very interesting test\n> case. We'd want a lot of refs, and ones that are old and have bushy\n> history that is not included in the more recent branches. So something\n> like a bunch of old unmerged pull request heads, for example. ;) Do\n> you have more interesting numbers for something like that?\n\nIndeed, here's one for a private repository which meets that criteria:\n\n```\nTest this tree\n---------------------------------------------------------------------------------------\n5333.1: git rev-list --count --all --objects (no bitmaps) 122.29(121.31+0.97)\n5333.2: git rev-list --count --all --objects (no pseudo-merges) 21.88(21.30+0.58)\n5333.3: git rev-list --count --all --objects (with pseudo-merges) 5.05(4.77+0.28)\n```\n\nThanks,\nTaylor\n"},{"id":"495450","messageId":"Zk+hP++s/Pz+yym4@nand.local","threadId":"61166","inReplyTo":"20240523110532.GA1326297@coredump.intra.peff.net","subject":"Re: [PATCH v3 00/30] pack-bitmap: pseudo-merge reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T20:04:15Z","receivedAt":"2024-05-23T20:04:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, May 23, 2024 at 07:05:32AM -0400, Jeff King wrote:\n> I wonder if the start of the pseudo-merge section should have a 4-byte\n> version/flags field itself? I don't think that's something we've done\n> before, and maybe it's overkill. I dunno. It's just a lot easier to do\n> now than later.\n\nI think the tricky thing here would be that the extension itself is a\nvariable size, so every version would have to put the \"extension size\"\nfield in the same place.\n\nOtherwise, an older Git client which doesn't understand a future version\nof the pseudo-merge extension wouldn't know how large the extension is,\nand wouldn't be able to adjust the index_end field appropriately to skip\nover it.\n\nOf course, we could make it a convention that says \"all versions have to\nplace the extension size field at the same relative offset\", but it\nfeels weird to read some of the extension while not understanding the\nwhole thing.\n\nI'm definitely not saying that I think the specification ought to be set\nin stone forever, but I think any changes would want to be behind a new\nbitmap extension, not a version within the same extension.\n\nThanks,\nTaylor\n"},{"id":"495454","messageId":"Zk+qG4INyrqPL5SN@nand.local","threadId":"61166","inReplyTo":"20240523110532.GA1326297@coredump.intra.peff.net","subject":"Re: [PATCH v3 00/30] pack-bitmap: pseudo-merge reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T20:42:03Z","receivedAt":"2024-05-23T20:42:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, May 23, 2024 at 07:05:32AM -0400, Jeff King wrote:\n> On Tue, May 21, 2024 at 03:01:38PM -0400, Taylor Blau wrote:\n>\n> > Here is another reroll my topic to introduce pseudo-merge bitmaps.\n>\n> OK, I got through the whole thing. I left a few small comments, but\n> mostly just observations. Overall, the shape of it looks pretty good.\n> The much bigger question to me is: does it work?\n\nThanks for working through this series, I appreciate you taking a close\nlook, especially since it's on the longer end.\n\n> The perf results you showed at the end are underwhelming, but I think\n> that's mostly because it's not an interesting repository. I think it\n> would be nice to see at least some point-in-time benchmarks on a\n> single repository.\n\nI'd say it's mostly because I misspelt \"pseudo\" as \"psuedo\", but either\nway ;-).\n\n> I know you don't have those answers yet, and I know there's some\n> chicken-and-egg with getting this integrated so that you can start to\n> explore that. So I mostly reviewed this with an eye towards:\n>\n> - does the idea make sense (I think it does, but I'm kind of biased)\n>\n> - are the patches going to hurt anybody who isn't using the new\n> feature (I think the answer is no)\n>\n> - does the on-disk representation seem right, since that is hard to\n> change later (I didn't see any issues)\n>\n> - does the implementation look clean and plausibly correct (yes, but\n> what I'm getting at is that I didn't pore over all of the new code\n> with a microscope. Mostly I think the proof is in the pudding that\n> it provides the same correct answers more quickly).\n>\n> So to my eyes it looks good to move forward and let people start playing\n> with it. The big \"experimental\" warning in the config is good. Maybe\n> we'd want want in gitpacking(7), too?\n\nI think adding a matching warning in gitpacking(7) is a good idea. Your\nfeeling matches my own here that the burden of this series on those not\nusing pseudo-merge bitmaps is negligible, and that this unblocks those\nwho want to move forward with testing this out in the wild on doing so.\n\nI'll send a v4 shortly with the minor changes that I picked up from your\nlast read-through, and then I think we are good to go here.\n\nThanks,\nTaylor\n"},{"id":"495456","messageId":"cover.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1710972293.git.me@ttaylorr.com","subject":"[PATCH v4 00/24] pack-bitmap: pseudo-merge reachability bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:06Z","receivedAt":"2024-05-23T21:26:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is another reroll my topic to introduce pseudo-merge bitmaps.\n\nThe implementation is still relatively unchanged compared to last time,\nsave for the review that Peff provided on the remaining parts of this\nseries.\n\nAs usual, there is a range-diff below, but the significant changes since\nlast time are as follows:\n\n - replace git_config_float() with git_config_double() (and matching\n tweaks in the callers)\n\n - add a NOTE to gitpacking(7) reflecting that the pseudo-merge bitmaps\n feature is considered experimental\n\n - add a length check when reading extended pseudo merges via the\n pseudo_merge_ext_at() function\n\n - replace the new `--date` option for `test_commit_bulk` with a\n `--notick` option (and set the GIT_COMMITTER_DATE values\n appropriately at the callers)\n\n - fix broken performance tests due to a typo on \"pseudo\", and include\n results from a large repository.\n\nThe series is still based on 'tb/pack-bitmap-write-cleanups'.\n\nTaylor Blau (24):\n Documentation/gitpacking.txt: initial commit\n Documentation/gitpacking.txt: describe pseudo-merge bitmaps\n Documentation/technical: describe pseudo-merge bitmaps format\n ewah: implement `ewah_bitmap_is_subset()`\n pack-bitmap: move some initialization to `bitmap_writer_init()`\n pseudo-merge.ch: initial commit\n pack-bitmap-write: support storing pseudo-merge commits\n pack-bitmap: implement `bitmap_writer_has_bitmapped_object_id()`\n pack-bitmap: make `bitmap_writer_push_bitmapped_commit()` public\n config: introduce `git_config_double()`\n pseudo-merge: implement support for selecting pseudo-merge commits\n pack-bitmap-write.c: write pseudo-merge table\n pack-bitmap: extract `read_bitmap()` function\n pseudo-merge: scaffolding for reads\n pack-bitmap.c: read pseudo-merge extension\n pseudo-merge: implement support for reading pseudo-merge commits\n ewah: implement `ewah_bitmap_popcount()`\n pack-bitmap: implement test helpers for pseudo-merge\n t/test-lib-functions.sh: support `--notick` in `test_commit_bulk()`\n pack-bitmap.c: use pseudo-merges during traversal\n pack-bitmap: extra trace2 information\n ewah: `bitmap_equals_ewah()`\n pseudo-merge: implement support for finding existing merges\n t/perf: implement performance tests for pseudo-merge bitmaps\n\n Documentation/Makefile | 1 +\n Documentation/config.txt | 2 +\n Documentation/config/bitmap-pseudo-merge.txt | 91 +++\n Documentation/gitpacking.txt | 189 +++++\n Documentation/technical/bitmap-format.txt | 132 ++++\n Makefile | 1 +\n builtin/pack-objects.c | 3 +-\n config.c | 9 +\n config.h | 7 +\n ewah/bitmap.c | 76 ++\n ewah/ewok.h | 8 +\n midx-write.c | 2 +-\n object.h | 2 +-\n pack-bitmap-write.c | 274 ++++++-\n pack-bitmap.c | 359 ++++++++-\n pack-bitmap.h | 19 +-\n parse.c | 29 +\n parse.h | 1 +\n pseudo-merge.c | 756 +++++++++++++++++++\n pseudo-merge.h | 216 ++++++\n t/helper/test-bitmap.c | 34 +-\n t/perf/p5333-pseudo-merge-bitmaps.sh | 32 +\n t/t5333-pseudo-merge-bitmaps.sh | 393 ++++++++++\n t/test-lib-functions.sh | 9 +-\n 24 files changed, 2590 insertions(+), 55 deletions(-)\n create mode 100644 Documentation/config/bitmap-pseudo-merge.txt\n create mode 100644 Documentation/gitpacking.txt\n create mode 100644 pseudo-merge.c\n create mode 100644 pseudo-merge.h\n create mode 100755 t/perf/p5333-pseudo-merge-bitmaps.sh\n create mode 100755 t/t5333-pseudo-merge-bitmaps.sh\n\nRange-diff against v3:\n -: ----------- > 1: 0f20c9becf4 Documentation/gitpacking.txt: initial commit\n 1: 528b591bd84 ! 2: 48afaa74928 Documentation/gitpacking.txt: describe pseudo-merge bitmaps\n @@ Documentation/gitpacking.txt: There are many aspects of packing in Git that are\n \n +== Pseudo-merge bitmaps\n +\n ++NOTE: Pseudo-merge bitmaps are considered an experimental feature, so\n ++the configuration and many of the ideas are subject to change.\n ++\n +=== Background\n +\n +Reachability bitmaps are most efficient when we have on-disk stored\n 2: 12f318b3d7e = 3: 44046f83c1a Documentation/technical: describe pseudo-merge bitmaps format\n 3: 40eb6137618 = 4: 211d6f14128 ewah: implement `ewah_bitmap_is_subset()`\n 4: 487fb7c6e9c = 5: 650cac2dcf9 pack-bitmap: move some initialization to `bitmap_writer_init()`\n 5: 827732acf99 = 6: 6647d8832ce pseudo-merge.ch: initial commit\n 6: 8608dd1860f = 7: e8ef1ef5ee4 pack-bitmap-write: support storing pseudo-merge commits\n 7: 99d2b6872ba = 8: fe458728c8a pack-bitmap: implement `bitmap_writer_has_bitmapped_object_id()`\n 8: e7209c60fa5 = 9: 6bf372f4020 pack-bitmap: make `bitmap_writer_push_bitmapped_commit()` public\n 9: 3070135eb4b ! 10: 6c77671ae9c config: introduce git_config_float()\n @@ Metadata\n Author: Taylor Blau \n \n ## Commit message ##\n - config: introduce git_config_float()\n + config: introduce `git_config_double()`\n \n - Future commits will want to parse a floating point value from\n - configuration, but we have no way to parse such a value prior to this\n - patch.\n + Future commits will want to parse a double-precision floating point\n + value from configuration, but we have no way to parse such a value prior\n + to this patch.\n \n - The core of the routine is implemented in git_parse_float(). Unlike\n + The core of the routine is implemented in git_parse_double(). Unlike\n git_parse_unsigned() and git_parse_signed(), however, the function\n - implemented here only works on type \"float\", and not related types like\n - \"double\", or \"long double\".\n + implemented here only works on type \"double\", and not related types like\n + \"float\", or \"long double\".\n \n - This is because \"double\" and \"long double\" use different functions to\n - convert from ASCII strings to floating point values (strtod() and\n + This is because \"float\" and \"long double\" use different functions to\n + convert from ASCII strings to floating point values (strtof() and\n strtold(), respectively). Likewise, there is no pointer type that can\n assign to any of these values (except for \"void *\"), so the only way to\n define this trio of functions would be with a macro expansion that is\n parameterized over the floating point type and conversion function.\n \n That is all doable, but likely to be overkill given our current needs,\n - which is only to parse floats.\n + which is only to parse double-precision floats.\n \n Signed-off-by: Taylor Blau \n \n @@ config.c: ssize_t git_config_ssize_t(const char *name, const char *value,\n \treturn ret;\n }\n \n -+float git_config_float(const char *name, const char *value,\n -+\t\t const struct key_value_info *kvi)\n ++double git_config_double(const char *name, const char *value,\n ++\t\t\t const struct key_value_info *kvi)\n +{\n -+\tfloat ret;\n -+\tif (!git_parse_float(value, &ret))\n ++\tdouble ret;\n ++\tif (!git_parse_double(value, &ret))\n +\t\tdie_bad_number(name, value, kvi);\n +\treturn ret;\n +}\n @@ config.h: unsigned long git_config_ulong(const char *, const char *,\n \t\t\t const struct key_value_info *);\n \n +/**\n -+ * Identical to `git_config_int`, but for floating point values.\n ++ * Identically to `git_config_double`, but for double-precision floating point\n ++ * values.\n + */\n -+float git_config_float(const char *, const char *,\n -+\t\t const struct key_value_info *);\n ++double git_config_double(const char *, const char *,\n ++\t\t\t const struct key_value_info *);\n +\n /**\n * Same as `git_config_bool`, except that integers are returned as-is, and\n @@ parse.c: int git_parse_ssize_t(const char *value, ssize_t *ret)\n \treturn 1;\n }\n \n -+int git_parse_float(const char *value, float *ret)\n ++int git_parse_double(const char *value, double *ret)\n +{\n +\tchar *end;\n -+\tfloat val;\n ++\tdouble val;\n +\tuintmax_t factor;\n +\n +\tif (!value || !*value) {\n @@ parse.c: int git_parse_ssize_t(const char *value, ssize_t *ret)\n +\t}\n +\n +\terrno = 0;\n -+\tval = strtof(value, &end);\n ++\tval = strtod(value, &end);\n +\tif (errno == ERANGE)\n +\t\treturn 0;\n +\tif (end == value) {\n @@ parse.h: int git_parse_ssize_t(const char *, ssize_t *);\n int git_parse_ulong(const char *, unsigned long *);\n int git_parse_int(const char *value, int *ret);\n int git_parse_int64(const char *value, int64_t *ret);\n -+int git_parse_float(const char *value, float *ret);\n ++int git_parse_double(const char *value, double *ret);\n \n /**\n * Same as `git_config_bool`, except that it returns -1 on error rather\n10: 3029473c094 ! 11: 180072ce848 pseudo-merge: implement support for selecting pseudo-merge commits\n @@ Documentation/config/bitmap-pseudo-merge.txt (new)\n @@\n +NOTE: The configuration options in `bitmapPseudoMerge.*` are considered\n +EXPERIMENTAL and may be subject to change or be removed entirely in the\n -+future.\n ++future. For more information about the pseudo-merge bitmap feature, see\n ++the \"Pseudo-merge bitmaps\" section of linkgit:gitpacking[7].\n +\n +bitmapPseudoMerge..pattern::\n +\tRegular expression used to match reference names. Commits\n @@ pseudo-merge.c\n +#include \"alloc.h\"\n +#include \"progress.h\"\n +\n -+#define DEFAULT_PSEUDO_MERGE_DECAY 1.0f\n ++#define DEFAULT_PSEUDO_MERGE_DECAY 1.0\n +#define DEFAULT_PSEUDO_MERGE_MAX_MERGES 64\n +#define DEFAULT_PSEUDO_MERGE_SAMPLE_RATE 1\n +#define DEFAULT_PSEUDO_MERGE_THRESHOLD approxidate(\"1.week.ago\")\n +#define DEFAULT_PSEUDO_MERGE_STABLE_THRESHOLD approxidate(\"1.month.ago\")\n +#define DEFAULT_PSEUDO_MERGE_STABLE_SIZE 512\n +\n -+static float gitexp(float base, int exp)\n ++static double gitexp(double base, int exp)\n +{\n -+\tfloat result = 1;\n ++\tdouble result = 1;\n +\twhile (1) {\n +\t\tif (exp % 2)\n +\t\t\tresult *= base;\n @@ pseudo-merge.c\n +\t\t\t\t\tconst struct pseudo_merge_matches *matches,\n +\t\t\t\t\tuint32_t i)\n +{\n -+\tfloat C = 0.0f;\n ++\tdouble C = 0.0f;\n +\tuint32_t n;\n +\n +\t/*\n @@ pseudo-merge.c\n +\t * { 5012, 1772, 964, 626, 448, 341, 271, 221, 186, 158 }\n +\t */\n +\tfor (n = 0; n < group->max_merges; n++)\n -+\t\tC += 1.0f / gitexp(n + 1, group->decay);\n ++\t\tC += 1.0 / gitexp(n + 1, group->decay);\n +\tC = matches->unstable_nr / C;\n +\n +\treturn (uint32_t)((C / gitexp(i + 1, group->decay)) + 0.5);\n @@ pseudo-merge.c\n +\n +\t\tstrbuf_release(&re);\n +\t} else if (!strcmp(key, \"decay\")) {\n -+\t\tgroup->decay = git_config_float(var, value, ctx->kvi);\n ++\t\tgroup->decay = git_config_double(var, value, ctx->kvi);\n +\t\tif (group->decay < 0) {\n +\t\t\twarning(_(\"%s must be non-negative, using default\"), var);\n +\t\t\tgroup->decay = DEFAULT_PSEUDO_MERGE_DECAY;\n +\t\t}\n +\t} else if (!strcmp(key, \"samplerate\")) {\n -+\t\tgroup->sample_rate = git_config_float(var, value, ctx->kvi);\n ++\t\tgroup->sample_rate = git_config_double(var, value, ctx->kvi);\n +\t\tif (!(0 <= group->sample_rate && group->sample_rate <= 1)) {\n +\t\t\twarning(_(\"%s must be between 0 and 1, using default\"), var);\n +\t\t\tgroup->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE;\n @@ pseudo-merge.c\n +\t\t\tstruct commit *c = matches->unstable[j];\n +\t\t\tstruct pseudo_merge_commit_idx *pmc;\n +\n -+\t\t\tif (j % (uint32_t)(1.0f / group->sample_rate))\n ++\t\t\tif (j % (uint32_t)(1.0 / group->sample_rate))\n +\t\t\t\tcontinue;\n +\n +\t\t\tpmc = pseudo_merge_idx(writer->pseudo_merge_commits,\n @@ pseudo-merge.h\n +\t * Pseudo-merge grouping parameters. See git-config(1) for\n +\t * more information.\n +\t */\n -+\tfloat decay;\n ++\tdouble decay;\n +\tint max_merges;\n -+\tfloat sample_rate;\n ++\tdouble sample_rate;\n +\tint stable_size;\n +\ttimestamp_t threshold;\n +\ttimestamp_t stable_threshold;\n11: 311226f65c2 = 12: 90df19e43f5 pack-bitmap-write.c: write pseudo-merge table\n12: 55dd7a8023e = 13: c653a10f8e4 pack-bitmap: extract `read_bitmap()` function\n13: 3cc5434e44e = 14: 435ac048003 pseudo-merge: scaffolding for reads\n14: 7664f5f9648 = 15: fa7a948964c pack-bitmap.c: read pseudo-merge extension\n15: 8ba0a9c5402 ! 16: 3a72e66cb69 pseudo-merge: implement support for reading pseudo-merge commits\n @@ pseudo-merge.c\n #include \"progress.h\"\n +#include \"hex.h\"\n \n - #define DEFAULT_PSEUDO_MERGE_DECAY 1.0f\n + #define DEFAULT_PSEUDO_MERGE_DECAY 1.0\n #define DEFAULT_PSEUDO_MERGE_MAX_MERGES 64\n @@ pseudo-merge.c: void free_pseudo_merge_map(struct pseudo_merge_map *pm)\n \t}\n @@ pseudo-merge.c: void free_pseudo_merge_map(struct pseudo_merge_map *pm)\n +\t\treturn error(_(\"extended pseudo-merge read out-of-bounds \"\n +\t\t\t \"(%\"PRIuMAX\" >= %\"PRIuMAX\")\"),\n +\t\t\t (uintmax_t)at, (uintmax_t)pm->map_size);\n ++\tif (at + 4 >= pm->map_size)\n ++\t\treturn error(_(\"extended pseudo-merge entry is too short \"\n ++\t\t\t \"(%\"PRIuMAX\" >= %\"PRIuMAX\")\"),\n ++\t\t\t (uintmax_t)(at + 4), (uintmax_t)pm->map_size);\n +\n +\text->nr = get_be32(pm->map + at);\n +\text->ptr = pm->map + at + sizeof(uint32_t);\n16: 2c02f303b6f = 17: 42a836fda8a ewah: implement `ewah_bitmap_popcount()`\n17: 82cce72bf55 = 18: 06ba1a5bbfd pack-bitmap: implement test helpers for pseudo-merge\n18: 890f6c4b9de ! 19: 936f6d1b7e3 t/test-lib-functions.sh: support `--date` in `test_commit_bulk()`\n @@ Metadata\n Author: Taylor Blau \n \n ## Commit message ##\n - t/test-lib-functions.sh: support `--date` in `test_commit_bulk()`\n + t/test-lib-functions.sh: support `--notick` in `test_commit_bulk()`\n \n One of the tests we'll want to add for pseudo-merge bitmaps needs to be\n able to generate a large number of commits at a specific date.\n \n - Support the `--date` option (with identical semantics to the `--date`\n - option for `test_commit()`) within `test_commit_bulk` as a prerequisite\n - for that.\n + Support the `--notick` option (with identical semantics to the\n + `--notick` option for `test_commit()`) within `test_commit_bulk` as a\n + prerequisite for that. Callers can then set the various _DATE variables\n + themselves.\n \n Signed-off-by: Taylor Blau \n \n @@ t/test-lib-functions.sh: test_commit_bulk () {\n \t\t\tfilename=\"${1#--*=}-%s.t\"\n \t\t\tcontents=\"${1#--*=} %s\"\n \t\t\t;;\n -+\t\t--date)\n ++\t\t--notick)\n +\t\t\tnotick=yes\n -+\t\t\tGIT_COMMITTER_DATE=\"$2\"\n -+\t\t\tGIT_AUTHOR_DATE=\"$2\"\n -+\t\t\tshift\n +\t\t\t;;\n \t\t-*)\n \t\t\tBUG \"invalid test_commit_bulk option: $1\"\n19: 41691824f78 ! 20: cad38608aae pack-bitmap.c: use pseudo-merges during traversal\n @@ t/t5333-pseudo-merge-bitmaps.sh (new)\n +\t\tnew=\"1672549200\" && # 2023-01-01\n +\t\told=\"1641013200\" && # 2022-01-01\n +\n -+\t\ttest_commit_bulk --message=\"new\" --date \"$new +0000\" 128 &&\n -+\t\ttest_commit_bulk --message=\"old\" --date \"$old +0000\" 128 &&\n -+\t\ttest_tick &&\n ++\t\tGIT_COMMITTER_DATE=\"$new +0000\" &&\n ++\t\texport GIT_COMMITTER_DATE &&\n ++\t\ttest_commit_bulk --message=\"new\" --notick 128 &&\n ++\n ++\t\tGIT_COMMITTER_DATE=\"$old +0000\" &&\n ++\t\texport GIT_COMMITTER_DATE &&\n ++\t\ttest_commit_bulk --message=\"old\" --notick 128 &&\n +\n +\t\ttag_everything &&\n +\n @@ t/t5333-pseudo-merge-bitmaps.sh (new)\n +\t\tmid=\"1654059600\" && # 2022-06-01\n +\t\told=\"1641013200\" && # 2022-01-01\n +\n -+\t\ttest_commit_bulk --message=\"mid\" --date \"$mid +0000\" 128 &&\n -+\t\ttest_tick &&\n ++\t\tGIT_COMMITTER_DATE=\"$mid +0000\" &&\n ++\t\texport GIT_COMMITTER_DATE &&\n ++\t\ttest_commit_bulk --message=\"mid\" --notick 128 &&\n +\n +\t\tgit for-each-ref --format=\"delete %(refname)\" refs/tags >in &&\n +\t\tgit update-ref --stdin \n \n ## Commit message ##\n - t/perf: implement performace tests for pseudo-merge bitmaps\n + t/perf: implement performance tests for pseudo-merge bitmaps\n \n Implement a straightforward performance test demonstrating the benefit\n of pseudo-merge bitmaps by measuring how long it takes to count\n @@ Commit message\n \n Test this tree\n -----------------------------------------------------------------------------------\n - 5333.2: git rev-list --count --all --objects (no bitmaps) 3.46(3.37+0.09)\n - 5333.3: git rev-list --count --all --objects (no pseudo-merges) 0.13(0.11+0.01)\n + 5333.2: git rev-list --count --all --objects (no bitmaps) 3.54(3.45+0.08)\n + 5333.3: git rev-list --count --all --objects (no pseudo-merges) 0.43(0.40+0.03)\n 5333.4: git rev-list --count --all --objects (with pseudo-merges) 0.12(0.11+0.01)\n \n + On a private repository which is much larger, and has many spikey parts\n + of history that aren't merged into the 'master' branch, the results are\n + as follows:\n +\n + Test this tree\n + ---------------------------------------------------------------------------------------\n + 5333.1: git rev-list --count --all --objects (no bitmaps) 122.29(121.31+0.97)\n + 5333.2: git rev-list --count --all --objects (no pseudo-merges) 21.88(21.30+0.58)\n + 5333.3: git rev-list --count --all --objects (with pseudo-merges) 5.05(4.77+0.28)\n +\n Signed-off-by: Taylor Blau \n \n ## t/perf/p5333-pseudo-merge-bitmaps.sh (new) ##\n @@ t/perf/p5333-pseudo-merge-bitmaps.sh (new)\n +'\n +\n +test_perf 'git rev-list --count --all --objects (no pseudo-merges)' '\n -+\tGIT_TEST_USE_PSEDUO_MERGES=0 \\\n ++\tGIT_TEST_USE_PSEUDO_MERGES=0 \\\n +\t\tgit rev-list --objects --all --use-bitmap-index\n +'\n +\n +test_perf 'git rev-list --count --all --objects (with pseudo-merges)' '\n -+\tGIT_TEST_USE_PSEDUO_MERGES=1 \\\n ++\tGIT_TEST_USE_PSEUDO_MERGES=1 \\\n +\t\tgit rev-list --objects --all --use-bitmap-index\n +'\n +\n\nbase-commit: bf65967764f34adc2ca00d4c8195840ad3e4e127\n-- \n2.45.1.175.gcf0316ad0e9\n"},{"id":"495457","messageId":"0f20c9becf452ef7a7e931b36336ccddba0f1d13.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 01/24] Documentation/gitpacking.txt: initial commit","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:10Z","receivedAt":"2024-05-23T21:26:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a new manual page, gitpacking(7) to collect useful information\nabout advanced packing concepts in Git.\n\nIn future commits in this series, this manual page will expand to\ndescribe the new pseudo-merge bitmaps feature, as well as include\nexamples, relevant configuration bits, use-cases, and so on.\n\nOutside of this series, this manual page may absorb similar pieces from\nother parts of Git's documentation about packing.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/Makefile | 1 +\n Documentation/gitpacking.txt | 34 ++++++++++++++++++++++++++++++++++\n 2 files changed, 35 insertions(+)\n create mode 100644 Documentation/gitpacking.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex 3f2383a12c7..920b6248aa4 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -51,6 +51,7 @@ MAN7_TXT += gitdiffcore.txt\n MAN7_TXT += giteveryday.txt\n MAN7_TXT += gitfaq.txt\n MAN7_TXT += gitglossary.txt\n+MAN7_TXT += gitpacking.txt\n MAN7_TXT += gitnamespaces.txt\n MAN7_TXT += gitremote-helpers.txt\n MAN7_TXT += gitrevisions.txt\ndiff --git a/Documentation/gitpacking.txt b/Documentation/gitpacking.txt\nnew file mode 100644\nindex 00000000000..50e9900d845\n--- /dev/null\n+++ b/Documentation/gitpacking.txt\n@@ -0,0 +1,34 @@\n+gitpacking(7)\n+=============\n+\n+NAME\n+----\n+gitpacking - Advanced concepts related to packing in Git\n+\n+SYNOPSIS\n+--------\n+gitpacking\n+\n+DESCRIPTION\n+-----------\n+\n+This document aims to describe some advanced concepts related to packing\n+in Git.\n+\n+Many concepts are currently described scattered between manual pages of\n+various Git commands, including linkgit:git-pack-objects[1],\n+linkgit:git-repack[1], and others, as well as linkgit:gitformat-pack[5],\n+and parts of the `Documentation/technical` tree.\n+\n+There are many aspects of packing in Git that are not covered in this\n+document that instead live in the aforementioned areas. Over time, those\n+scattered bits may coalesce into this document.\n+\n+SEE ALSO\n+--------\n+linkgit:git-pack-objects[1]\n+linkgit:git-repack[1]\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495458","messageId":"48afaa7492815350ae17405da1a8d09eb8e97c15.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 02/24] Documentation/gitpacking.txt: describe pseudo-merge bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:13Z","receivedAt":"2024-05-23T21:26:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Add some details to the gitpacking(7) manual page which motivate and\ndescribe pseudo-merge bitmaps.\n\nThe exact on-disk format and many of the configuration knobs will be\ndescribed in subsequent commits.\n\nHelped-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n Documentation/gitpacking.txt | 72 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 72 insertions(+)\n\ndiff --git a/Documentation/gitpacking.txt b/Documentation/gitpacking.txt\nindex 50e9900d845..f24396f0173 100644\n--- a/Documentation/gitpacking.txt\n+++ b/Documentation/gitpacking.txt\n@@ -24,6 +24,78 @@ There are many aspects of packing in Git that are not covered in this\n document that instead live in the aforementioned areas. Over time, those\n scattered bits may coalesce into this document.\n \n+== Pseudo-merge bitmaps\n+\n+NOTE: Pseudo-merge bitmaps are considered an experimental feature, so\n+the configuration and many of the ideas are subject to change.\n+\n+=== Background\n+\n+Reachability bitmaps are most efficient when we have on-disk stored\n+bitmaps for one or more of the starting points of a traversal. For this\n+reason, Git prefers storing bitmaps for commits at the tips of refs,\n+because traversals tend to start with those points.\n+\n+But if you have a large number of refs, it's not feasible to store a\n+bitmap for _every_ ref tip. It takes up space, and just OR-ing all of\n+those bitmaps together is expensive.\n+\n+One way we can deal with that is to create bitmaps that represent\n+_groups_ of refs. When a traversal asks about the entire group, then we\n+can use this single bitmap instead of considering each ref individually.\n+Because these bitmaps represent the set of objects which would be\n+reachable in a hypothetical merge of all of the commits, we call them\n+pseudo-merge bitmaps.\n+\n+=== Overview\n+\n+A \"pseudo-merge bitmap\" is used to refer to a pair of bitmaps, as\n+follows:\n+\n+Commit bitmap::\n+\n+ A bitmap whose set bits describe the set of commits included in the\n+ pseudo-merge's \"merge\" bitmap (as below).\n+\n+Merge bitmap::\n+\n+ A bitmap whose set bits describe the reachability closure over the set\n+ of commits in the pseudo-merge's \"commits\" bitmap (as above). An\n+ identical bitmap would be generated for an octopus merge with the same\n+ set of parents as described in the commits bitmap.\n+\n+Pseudo-merge bitmaps can accelerate bitmap traversals when all commits\n+for a given pseudo-merge are listed on either side of the traversal,\n+either directly (by explicitly asking for them as part of the `HAVES`\n+or `WANTS`) or indirectly (by encountering them during a fill-in\n+traversal).\n+\n+=== Use-cases\n+\n+For example, suppose there exists a pseudo-merge bitmap with a large\n+number of commits, all of which are listed in the `WANTS` section of\n+some bitmap traversal query. When pseudo-merge bitmaps are enabled, the\n+bitmap machinery can quickly determine there is a pseudo-merge which\n+satisfies some subset of the wanted objects on either side of the query.\n+Then, we can inflate the EWAH-compressed bitmap, and `OR` it in to the\n+resulting bitmap. By contrast, without pseudo-merge bitmaps, we would\n+have to repeat the decompression and `OR`-ing step over a potentially\n+large number of individual bitmaps, which can take proportionally more\n+time.\n+\n+Another benefit of pseudo-merges arises when there is some combination\n+of (a) a large number of references, with (b) poor bitmap coverage, and\n+(c) deep, nested trees, making fill-in traversal relatively expensive.\n+For example, suppose that there are a large enough number of tags where\n+bitmapping each of the tags individually is infeasible. Without\n+pseudo-merge bitmaps, computing the result of, say, `git rev-list\n+--use-bitmap-index --count --objects --tags` would likely require a\n+large amount of fill-in traversal. But when a large quantity of those\n+tags are stored together in a pseudo-merge bitmap, the bitmap machinery\n+can take advantage of the fact that we only care about the union of\n+objects reachable from all of those tags, and answer the query much\n+faster.\n+\n SEE ALSO\n --------\n linkgit:git-pack-objects[1]\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495459","messageId":"44046f83c1a8a5971fd434debbf31df00658e2b8.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 03/24] Documentation/technical: describe pseudo-merge bitmaps format","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:16Z","receivedAt":"2024-05-23T21:26:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to implement pseudo-merge bitmaps over the next several commits\nby first describing the serialization format which will store the new\npseudo-merge bitmaps themselves.\n\nThis format is implemented as an optional extension within the bitmap v1\nformat, making it compatible with previous versions of Git, as well as\nthe original .bitmap implementation within JGit.\n\nThe format is described in detail in the patch contents below, but the\nhigh-level description is as follows:\n\n - An array of pseudo-merge bitmaps, each containing a pair of EWAH\n bitmaps: one describing the set of pseudo-merge \"parents\", and\n another describing the set of object(s) reachable from those\n parents.\n\n - A lookup table to determine which pseudo-merge(s) a given commit\n appears in. An optional extended lookup table follows when there is\n at least one commit which appears in multiple pseudo-merge groups.\n\n - Trailing metadata, including the number of pseudo-merge(s), number\n of unique parents, the offset within the .bitmap file for the\n pseudo-merge commit lookup table, and the size of the optional\n extension itself.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/technical/bitmap-format.txt | 132 ++++++++++++++++++++++\n 1 file changed, 132 insertions(+)\n\ndiff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt\nindex f5d200939b0..ee7775a2586 100644\n--- a/Documentation/technical/bitmap-format.txt\n+++ b/Documentation/technical/bitmap-format.txt\n@@ -255,3 +255,135 @@ triplet is -\n \txor_row (4 byte integer, network byte order): ::\n \tThe position of the triplet whose bitmap is used to compress\n \tthis one, or `0xffffffff` if no such bitmap exists.\n+\n+Pseudo-merge bitmaps\n+--------------------\n+\n+If the `BITMAP_OPT_PSEUDO_MERGES` flag is set, a variable number of\n+bytes (preceding the name-hash cache, commit lookup table, and trailing\n+checksum) of the `.bitmap` file is used to store pseudo-merge bitmaps.\n+\n+For more information on what pseudo-merges are, why they are useful, and\n+how to configure them, see the information in linkgit:gitpacking[7].\n+\n+=== File format\n+\n+If enabled, pseudo-merge bitmaps are stored in an optional section at\n+the end of a `.bitmap` file. The format is as follows:\n+\n+....\n++-------------------------------------------+\n+| .bitmap File |\n++-------------------------------------------+\n+| |\n+| Pseudo-merge bitmaps (Variable Length) |\n+| +---------------------------+ |\n+| | commits_bitmap (EWAH) | |\n+| +---------------------------+ |\n+| | merge_bitmap (EWAH) | |\n+| +---------------------------+ |\n+| |\n++-------------------------------------------+\n+| |\n+| Lookup Table |\n+| +---------------------------+ |\n+| | commit_pos (4 bytes) | |\n+| +---------------------------+ |\n+| | offset (8 bytes) | |\n+| +------------+--------------+ |\n+| |\n+| Offset Cases: |\n+| ------------- |\n+| |\n+| 1. MSB Unset: single pseudo-merge bitmap |\n+| + offset to pseudo-merge bitmap |\n+| |\n+| 2. MSB Set: multiple pseudo-merges |\n+| + offset to extended lookup table |\n+| |\n++-------------------------------------------+\n+| |\n+| Extended Lookup Table (Optional) |\n+| +----+----------+----------+----------+ |\n+| | N | Offset 1 | .... | Offset N | |\n+| +----+----------+----------+----------+ |\n+| | | 8 bytes | .... | 8 bytes | |\n+| +----+----------+----------+----------+ |\n+| |\n++-------------------------------------------+\n+| |\n+| Pseudo-merge Metadata |\n+| +-----------------------------------+ |\n+| | # pseudo-merges (4 bytes) | |\n+| +-----------------------------------+ |\n+| | # commits (4 bytes) | |\n+| +-----------------------------------+ |\n+| | Lookup offset (8 bytes) | |\n+| +-----------------------------------+ |\n+| | Extension size (8 bytes) | |\n+| +-----------------------------------+ |\n+| |\n++-------------------------------------------+\n+....\n+\n+* One or more pseudo-merge bitmaps, each containing:\n+\n+ ** `commits_bitmap`, an EWAH-compressed bitmap describing the set of\n+ commits included in the this psuedo-merge.\n+\n+ ** `merge_bitmap`, an EWAH-compressed bitmap describing the union of\n+ the set of objects reachable from all commits listed in the\n+ `commits_bitmap`.\n+\n+* A lookup table, mapping pseudo-merged commits to the pseudo-merges\n+ they belong to. Entries appear in increasing order of each commit's\n+ bit position. Each entry is 12 bytes wide, and is comprised of the\n+ following:\n+\n+ ** `commit_pos`, a 4-byte unsigned value (in network byte-order)\n+ containing the bit position for this commit.\n+\n+ ** `offset`, an 8-byte unsigned value (also in network byte-order)\n+ containing either one of two possible offsets, depending on whether or\n+ not the most-significant bit is set.\n+\n+ *** If unset (i.e. `offset & ((uint64_t)1<<63) == 0`), the offset\n+\t(relative to the beginning of the `.bitmap` file) at which the\n+\tpseudo-merge bitmap for this commit can be read. This indicates\n+\tonly a single pseudo-merge bitmap contains this commit.\n+\n+ *** If set (i.e. `offset & ((uint64_t)1<<63) != 0`), the offset\n+\t(again relative to the beginning of the `.bitmap` file) at which\n+\tthe extended offset table can be located describing the set of\n+\tpseudo-merge bitmaps which contain this commit. This indicates\n+\tthat multiple pseudo-merge bitmaps contain this commit.\n+\n+* An (optional) extended lookup table (written if and only if there is\n+ at least one commit which appears in more than one pseudo-merge).\n+ There are as many entries as commits which appear in multiple\n+ pseudo-merges. Each entry contains the following:\n+\n+ ** `N`, a 4-byte unsigned value equal to the number of pseudo-merges\n+ which contain a given commit.\n+\n+ ** An array of `N` 8-byte unsigned values, each of which is\n+ interpreted as an offset (relative to the beginning of the\n+ `.bitmap` file) at which a pseudo-merge bitmap for this commit can\n+ be read. These values occur in no particular order.\n+\n+* Positions for all pseudo-merges, each stored as an 8-byte unsigned\n+ value (in network byte-order) containing the offset (relative to the\n+ beginning of the `.bitmap` file) of each consecutive pseudo-merge.\n+\n+* A 4-byte unsigned value (in network byte-order) equal to the number of\n+ pseudo-merges.\n+\n+* A 4-byte unsigned value (in network byte-order) equal to the number of\n+ unique commits which appear in any pseudo-merge.\n+\n+* An 8-byte unsigned value (in network byte-order) equal to the number\n+ of bytes between the start of the pseudo-merge section and the\n+ beginning of the lookup table.\n+\n+* An 8-byte unsigned value (in network byte-order) equal to the number\n+ of bytes in the pseudo-merge section (including this field).\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495460","messageId":"211d6f1412874d6211f4ce92f74bb3ed88292f8e.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 04/24] ewah: implement `ewah_bitmap_is_subset()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:20Z","receivedAt":"2024-05-23T21:26:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In order to know whether a given pseudo-merge (comprised of a \"parents\"\nand \"objects\" bitmaps) is \"satisfied\" and can be OR'd into the bitmap\nresult, we need to be able to quickly determine whether the \"parents\"\nbitmap is a subset of the current set of objects reachable on either\nside of a traversal.\n\nImplement a helper function to prepare for that, which determines\nwhether an EWAH bitmap (the parents bitmap from the pseudo-merge) is a\nsubset of a non-EWAH bitmap (in this case, the results bitmap from\neither side of the traversal).\n\nThis function makes use of the EWAH iterator to avoid inflating any part\nof the EWAH bitmap after we determine it is not a subset of the non-EWAH\nbitmap. This \"fail-fast\" allows us to avoid a potentially large amount\nof wasted effort.\n\nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 43 +++++++++++++++++++++++++++++++++++++++++++\n ewah/ewok.h | 6 ++++++\n 2 files changed, 49 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex ac7e0af622a..d352fec54ce 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -138,6 +138,49 @@ void bitmap_or(struct bitmap *self, const struct bitmap *other)\n \t\tself->words[i] |= other->words[i];\n }\n \n+int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other)\n+{\n+\tstruct ewah_iterator it;\n+\teword_t word;\n+\tsize_t i;\n+\n+\tewah_iterator_init(&it, self);\n+\n+\tfor (i = 0; i < other->word_alloc; i++) {\n+\t\tif (!ewah_iterator_next(&word, &it)) {\n+\t\t\t/*\n+\t\t\t * If we reached the end of `self`, and haven't\n+\t\t\t * rejected `self` as a possible subset of\n+\t\t\t * `other` yet, then we are done and `self` is\n+\t\t\t * indeed a subset of `other`.\n+\t\t\t */\n+\t\t\treturn 1;\n+\t\t}\n+\t\tif (word & ~other->words[i]) {\n+\t\t\t/*\n+\t\t\t * Otherwise, compare the next two pairs of\n+\t\t\t * words. If the word from `self` has bit(s) not\n+\t\t\t * in the word from `other`, `self` is not a\n+\t\t\t * subset of `other`.\n+\t\t\t */\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * If we got to this point, there may be zero or more words\n+\t * remaining in `self`, with no remaining words left in `other`.\n+\t * If there are any bits set in the remaining word(s) in `self`,\n+\t * then `self` is not a subset of `other`.\n+\t */\n+\twhile (ewah_iterator_next(&word, &it))\n+\t\tif (word)\n+\t\t\treturn 0;\n+\n+\t/* `self` is definitely a subset of `other` */\n+\treturn 1;\n+}\n+\n void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other)\n {\n \tsize_t original_size = self->word_alloc;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex c11d76c6f33..2b6c4ac499c 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -179,7 +179,13 @@ void bitmap_unset(struct bitmap *self, size_t pos);\n int bitmap_get(struct bitmap *self, size_t pos);\n void bitmap_free(struct bitmap *self);\n int bitmap_equals(struct bitmap *self, struct bitmap *other);\n+\n+/*\n+ * Both `bitmap_is_subset()` and `ewah_bitmap_is_subset()` return 1 if the set\n+ * of bits in 'self' are a subset of the bits in 'other'. Returns 0 otherwise.\n+ */\n int bitmap_is_subset(struct bitmap *self, struct bitmap *other);\n+int ewah_bitmap_is_subset(struct ewah_bitmap *self, struct bitmap *other);\n \n struct ewah_bitmap * bitmap_to_ewah(struct bitmap *bitmap);\n struct bitmap *ewah_to_bitmap(struct ewah_bitmap *ewah);\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495461","messageId":"650cac2dcf920b2bb64d7a026aa83228a2b2c354.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 05/24] pack-bitmap: move some initialization to `bitmap_writer_init()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:23Z","receivedAt":"2024-05-23T21:26:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pack-bitmap-writer machinery uses a oidmap (backed by khash.h) to\nmap from commits selected for bitmaps (by OID) to a bitmapped_commit\nstructure (containing the bitmap itself, among other things like its XOR\noffset, etc.)\n\nThis map was initialized at the end of `bitmap_writer_build()`. New\nentries are added in `pack-bitmap-write.c::store_selected()`, which is\ncalled by the bitmap_builder machinery (which is responsible for\ntraversing history and generating the actual bitmaps).\n\nReorganize when this field is initialized and when entries are added to\nit so that we can quickly determine whether a commit is a candidate for\npseudo-merge selection, or not (since it was already selected to receive\na bitmap, and thus storing it in a pseudo-merge would be redundant).\n\nThe changes are as follows:\n\n - Introduce a new `bitmap_writer_init()` function which initializes\n the `writer.bitmaps` field (instead of waiting until the end of\n `bitmap_writer_build()`).\n\n - Add map entries in `push_bitmapped_commit()` (which is called via\n `bitmap_writer_select_commits()`) with OID keys and NULL values to\n track whether or not we *expect* to write a bitmap for some given\n commit.\n\n - Validate that a NULL entry is found matching the given key when we\n store a selected bitmap.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 3 ++-\n midx-write.c | 2 +-\n pack-bitmap-write.c | 24 ++++++++++++++++++------\n pack-bitmap.h | 2 +-\n 4 files changed, 22 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 26a6d0d7919..6209264e60c 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1340,7 +1340,8 @@ static void write_pack_file(void)\n \t\t\t\t hash_to_hex(hash));\n \n \t\t\tif (write_bitmap_index) {\n-\t\t\t\tbitmap_writer_init(&bitmap_writer);\n+\t\t\t\tbitmap_writer_init(&bitmap_writer,\n+\t\t\t\t\t\t the_repository);\n \t\t\t\tbitmap_writer_set_checksum(&bitmap_writer, hash);\n \t\t\t\tbitmap_writer_build_type_index(&bitmap_writer,\n \t\t\t\t\t&to_pack, written_list, nr_written);\ndiff --git a/midx-write.c b/midx-write.c\nindex 7c0c08c64b2..c747d1a6af3 100644\n--- a/midx-write.c\n+++ b/midx-write.c\n@@ -820,7 +820,7 @@ static int write_midx_bitmap(const char *midx_name,\n \tfor (i = 0; i < pdata->nr_objects; i++)\n \t\tindex[i] = &pdata->objects[i].idx;\n \n-\tbitmap_writer_init(&writer);\n+\tbitmap_writer_init(&writer, the_repository);\n \tbitmap_writer_show_progress(&writer, flags & MIDX_PROGRESS);\n \tbitmap_writer_build_type_index(&writer, pdata, index,\n \t\t\t\t pdata->nr_objects);\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 6cae670412c..d8870155831 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -27,9 +27,12 @@ struct bitmapped_commit {\n \tuint32_t commit_pos;\n };\n \n-void bitmap_writer_init(struct bitmap_writer *writer)\n+void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r)\n {\n \tmemset(writer, 0, sizeof(struct bitmap_writer));\n+\tif (writer->bitmaps)\n+\t\tBUG(\"bitmap writer already initialized\");\n+\twriter->bitmaps = kh_init_oid_map();\n }\n \n void bitmap_writer_free(struct bitmap_writer *writer)\n@@ -128,11 +131,21 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n static inline void push_bitmapped_commit(struct bitmap_writer *writer,\n \t\t\t\t\t struct commit *commit)\n {\n+\tint hash_ret;\n+\tkhiter_t hash_pos;\n+\n \tif (writer->selected_nr >= writer->selected_alloc) {\n \t\twriter->selected_alloc = (writer->selected_alloc + 32) * 2;\n \t\tREALLOC_ARRAY(writer->selected, writer->selected_alloc);\n \t}\n \n+\thash_pos = kh_put_oid_map(writer->bitmaps, commit->object.oid,\n+\t\t\t\t &hash_ret);\n+\tif (!hash_ret)\n+\t\tdie(_(\"duplicate entry when writing bitmap index: %s\"),\n+\t\t oid_to_hex(&commit->object.oid));\n+\tkh_value(writer->bitmaps, hash_pos) = NULL;\n+\n \twriter->selected[writer->selected_nr].commit = commit;\n \twriter->selected[writer->selected_nr].bitmap = NULL;\n \twriter->selected[writer->selected_nr].write_as = NULL;\n@@ -483,14 +496,14 @@ static void store_selected(struct bitmap_writer *writer,\n {\n \tstruct bitmapped_commit *stored = &writer->selected[ent->idx];\n \tkhiter_t hash_pos;\n-\tint hash_ret;\n \n \tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n \n-\thash_pos = kh_put_oid_map(writer->bitmaps, commit->object.oid, &hash_ret);\n-\tif (hash_ret == 0)\n-\t\tdie(\"Duplicate entry when writing index: %s\",\n+\thash_pos = kh_get_oid_map(writer->bitmaps, commit->object.oid);\n+\tif (hash_pos == kh_end(writer->bitmaps))\n+\t\tdie(_(\"attempted to store non-selected commit: '%s'\"),\n \t\t oid_to_hex(&commit->object.oid));\n+\n \tkh_value(writer->bitmaps, hash_pos) = stored;\n }\n \n@@ -506,7 +519,6 @@ int bitmap_writer_build(struct bitmap_writer *writer,\n \tuint32_t *mapping;\n \tint closed = 1; /* until proven otherwise */\n \n-\twriter->bitmaps = kh_init_oid_map();\n \twriter->to_pack = to_pack;\n \n \tif (writer->show_progress)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3091095f336..f87e60153dd 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -114,7 +114,7 @@ struct bitmap_writer {\n \tunsigned char pack_checksum[GIT_MAX_RAWSZ];\n };\n \n-void bitmap_writer_init(struct bitmap_writer *writer);\n+void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r);\n void bitmap_writer_show_progress(struct bitmap_writer *writer, int show);\n void bitmap_writer_set_checksum(struct bitmap_writer *writer,\n \t\t\t\tconst unsigned char *sha1);\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495462","messageId":"6647d8832ce1b70c4b46bd1191086f7e4bc19a34.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 06/24] pseudo-merge.ch: initial commit","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:26Z","receivedAt":"2024-05-23T21:26:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Add a new (empty) header file to contain the implementation for\nselecting, reading, and applying pseudo-merge bitmaps.\n\nFor now this header and its corresponding implementation are left\nempty, but they will evolve over the course of subsequent commit(s).\n\nSigned-off-by: Taylor Blau \n---\n Makefile | 1 +\n pseudo-merge.c | 2 ++\n pseudo-merge.h | 6 ++++++\n 3 files changed, 9 insertions(+)\n create mode 100644 pseudo-merge.c\n create mode 100644 pseudo-merge.h\n\ndiff --git a/Makefile b/Makefile\nindex 0285db56306..4705a69f57f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1105,6 +1105,7 @@ LIB_OBJS += prompt.o\n LIB_OBJS += protocol.o\n LIB_OBJS += protocol-caps.o\n LIB_OBJS += prune-packed.o\n+LIB_OBJS += pseudo-merge.o\n LIB_OBJS += quote.o\n LIB_OBJS += range-diff.o\n LIB_OBJS += reachable.o\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nnew file mode 100644\nindex 00000000000..37e037ba272\n--- /dev/null\n+++ b/pseudo-merge.c\n@@ -0,0 +1,2 @@\n+#include \"git-compat-util.h\"\n+#include \"pseudo-merge.h\"\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nnew file mode 100644\nindex 00000000000..cab8ff6960a\n--- /dev/null\n+++ b/pseudo-merge.h\n@@ -0,0 +1,6 @@\n+#ifndef PSEUDO_MERGE_H\n+#define PSEUDO_MERGE_H\n+\n+#include \"git-compat-util.h\"\n+\n+#endif\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495463","messageId":"e8ef1ef5ee46f9d0f5623137bf7393306c060980.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 07/24] pack-bitmap-write: support storing pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:29Z","receivedAt":"2024-05-23T21:26:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to write pseudo-merge bitmaps by annotating individual bitmapped\ncommits (which are represented by the `bitmapped_commit` structure) with\nan extra bit indicating whether or not they are a pseudo-merge.\n\nIn subsequent commits, pseudo-merge bitmaps will be generated by\nallocating a fake commit node with parents covering the full set of\ncommits represented by the pseudo-merge bitmap. These commits will be\nadded to the set of \"selected\" commits as usual, but will be written\nspecially instead of being included with the rest of the selected\ncommits.\n\nMechanically speaking, there are two parts of this change:\n\n - The bitmapped_commit struct gets a new bit indicating whether it is\n a pseudo-merge, or an ordinary commit selected for bitmaps.\n\n - A handful of changes to only write out the non-pseudo-merge commits\n when enumerating through the selected array (see the new\n `bitmap_writer_selected_nr()` function). Pseudo-merge commits appear\n after all non-pseudo-merge commits, so it is safe to enumerate\n through the selected array like so:\n\n for (i = 0; i < bitmap_writer_selected_nr(); i++)\n if (writer.selected[i].pseudo_merge)\n BUG(\"unexpected pseudo-merge\");\n\n without encountering the BUG().\n\nSigned-off-by: Taylor Blau \n---\n object.h | 2 +-\n pack-bitmap-write.c | 96 +++++++++++++++++++++++++++++----------------\n pack-bitmap.h | 3 ++\n 3 files changed, 67 insertions(+), 34 deletions(-)\n\ndiff --git a/object.h b/object.h\nindex 99b9c8f114c..e6f9e89d3c5 100644\n--- a/object.h\n+++ b/object.h\n@@ -81,7 +81,7 @@ void object_array_init(struct object_array *array);\n * reflog.c: 10--12\n * builtin/show-branch.c: 0-------------------------------------------26\n * builtin/unpack-objects.c: 2021\n- * pack-bitmap.h: 22\n+ * pack-bitmap.h: 2122\n */\n #define FLAG_BITS 28\n \ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex d8870155831..60eb1e71c98 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -25,8 +25,14 @@ struct bitmapped_commit {\n \tint flags;\n \tint xor_offset;\n \tuint32_t commit_pos;\n+\tunsigned pseudo_merge : 1;\n };\n \n+static inline int bitmap_writer_nr_selected_commits(struct bitmap_writer *writer)\n+{\n+\treturn writer->selected_nr - writer->pseudo_merges_nr;\n+}\n+\n void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r)\n {\n \tmemset(writer, 0, sizeof(struct bitmap_writer));\n@@ -129,27 +135,31 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n */\n \n static inline void push_bitmapped_commit(struct bitmap_writer *writer,\n-\t\t\t\t\t struct commit *commit)\n+\t\t\t\t\t struct commit *commit,\n+\t\t\t\t\t unsigned pseudo_merge)\n {\n-\tint hash_ret;\n-\tkhiter_t hash_pos;\n-\n \tif (writer->selected_nr >= writer->selected_alloc) {\n \t\twriter->selected_alloc = (writer->selected_alloc + 32) * 2;\n \t\tREALLOC_ARRAY(writer->selected, writer->selected_alloc);\n \t}\n \n-\thash_pos = kh_put_oid_map(writer->bitmaps, commit->object.oid,\n-\t\t\t\t &hash_ret);\n-\tif (!hash_ret)\n-\t\tdie(_(\"duplicate entry when writing bitmap index: %s\"),\n-\t\t oid_to_hex(&commit->object.oid));\n-\tkh_value(writer->bitmaps, hash_pos) = NULL;\n+\tif (!pseudo_merge) {\n+\t\tint hash_ret;\n+\t\tkhiter_t hash_pos = kh_put_oid_map(writer->bitmaps,\n+\t\t\t\t\t\t commit->object.oid,\n+\t\t\t\t\t\t &hash_ret);\n+\n+\t\tif (!hash_ret)\n+\t\t\tdie(_(\"duplicate entry when writing bitmap index: %s\"),\n+\t\t\t oid_to_hex(&commit->object.oid));\n+\t\tkh_value(writer->bitmaps, hash_pos) = NULL;\n+\t}\n \n \twriter->selected[writer->selected_nr].commit = commit;\n \twriter->selected[writer->selected_nr].bitmap = NULL;\n \twriter->selected[writer->selected_nr].write_as = NULL;\n \twriter->selected[writer->selected_nr].flags = 0;\n+\twriter->selected[writer->selected_nr].pseudo_merge = pseudo_merge;\n \n \twriter->selected_nr++;\n }\n@@ -180,16 +190,20 @@ static void compute_xor_offsets(struct bitmap_writer *writer)\n \n \twhile (next < writer->selected_nr) {\n \t\tstruct bitmapped_commit *stored = &writer->selected[next];\n-\n \t\tint best_offset = 0;\n \t\tstruct ewah_bitmap *best_bitmap = stored->bitmap;\n \t\tstruct ewah_bitmap *test_xor;\n \n+\t\tif (stored->pseudo_merge)\n+\t\t\tgoto next;\n+\n \t\tfor (i = 1; i <= MAX_XOR_OFFSET_SEARCH; ++i) {\n \t\t\tint curr = next - i;\n \n \t\t\tif (curr < 0)\n \t\t\t\tbreak;\n+\t\t\tif (writer->selected[curr].pseudo_merge)\n+\t\t\t\tcontinue;\n \n \t\t\ttest_xor = ewah_pool_new();\n \t\t\tewah_xor(writer->selected[curr].bitmap, stored->bitmap, test_xor);\n@@ -205,6 +219,7 @@ static void compute_xor_offsets(struct bitmap_writer *writer)\n \t\t\t}\n \t\t}\n \n+next:\n \t\tstored->xor_offset = best_offset;\n \t\tstored->write_as = best_bitmap;\n \n@@ -217,7 +232,8 @@ struct bb_commit {\n \tstruct bitmap *commit_mask;\n \tstruct bitmap *bitmap;\n \tunsigned selected:1,\n-\t\t maximal:1;\n+\t\t maximal:1,\n+\t\t pseudo_merge:1;\n \tunsigned idx; /* within selected array */\n };\n \n@@ -255,17 +271,18 @@ static void bitmap_builder_init(struct bitmap_builder *bb,\n \trevs.first_parent_only = 1;\n \n \tfor (i = 0; i < writer->selected_nr; i++) {\n-\t\tstruct commit *c = writer->selected[i].commit;\n-\t\tstruct bb_commit *ent = bb_data_at(&bb->data, c);\n+\t\tstruct bitmapped_commit *bc = &writer->selected[i];\n+\t\tstruct bb_commit *ent = bb_data_at(&bb->data, bc->commit);\n \n \t\tent->selected = 1;\n \t\tent->maximal = 1;\n+\t\tent->pseudo_merge = bc->pseudo_merge;\n \t\tent->idx = i;\n \n \t\tent->commit_mask = bitmap_new();\n \t\tbitmap_set(ent->commit_mask, i);\n \n-\t\tadd_pending_object(&revs, &c->object, \"\");\n+\t\tadd_pending_object(&revs, &bc->commit->object, \"\");\n \t}\n \n \tif (prepare_revision_walk(&revs))\n@@ -444,8 +461,13 @@ static int fill_bitmap_commit(struct bitmap_writer *writer,\n \t\tstruct commit *c = prio_queue_get(queue);\n \n \t\tif (old_bitmap && mapping) {\n-\t\t\tstruct ewah_bitmap *old = bitmap_for_commit(old_bitmap, c);\n+\t\t\tstruct ewah_bitmap *old;\n \t\t\tstruct bitmap *remapped = bitmap_new();\n+\n+\t\t\tif (commit->object.flags & BITMAP_PSEUDO_MERGE)\n+\t\t\t\told = NULL;\n+\t\t\telse\n+\t\t\t\told = bitmap_for_commit(old_bitmap, c);\n \t\t\t/*\n \t\t\t * If this commit has an old bitmap, then translate that\n \t\t\t * bitmap and add its bits to this one. No need to walk\n@@ -464,12 +486,14 @@ static int fill_bitmap_commit(struct bitmap_writer *writer,\n \t\t * Mark ourselves and queue our tree. The commit\n \t\t * walk ensures we cover all parents.\n \t\t */\n-\t\tpos = find_object_pos(writer, &c->object.oid, &found);\n-\t\tif (!found)\n-\t\t\treturn -1;\n-\t\tbitmap_set(ent->bitmap, pos);\n-\t\tprio_queue_put(tree_queue,\n-\t\t\t repo_get_commit_tree(the_repository, c));\n+\t\tif (!(c->object.flags & BITMAP_PSEUDO_MERGE)) {\n+\t\t\tpos = find_object_pos(writer, &c->object.oid, &found);\n+\t\t\tif (!found)\n+\t\t\t\treturn -1;\n+\t\t\tbitmap_set(ent->bitmap, pos);\n+\t\t\tprio_queue_put(tree_queue,\n+\t\t\t\t repo_get_commit_tree(the_repository, c));\n+\t\t}\n \n \t\tfor (p = c->parents; p; p = p->next) {\n \t\t\tpos = find_object_pos(writer, &p->item->object.oid,\n@@ -499,6 +523,9 @@ static void store_selected(struct bitmap_writer *writer,\n \n \tstored->bitmap = bitmap_to_ewah(ent->bitmap);\n \n+\tif (ent->pseudo_merge)\n+\t\treturn;\n+\n \thash_pos = kh_get_oid_map(writer->bitmaps, commit->object.oid);\n \tif (hash_pos == kh_end(writer->bitmaps))\n \t\tdie(_(\"attempted to store non-selected commit: '%s'\"),\n@@ -631,7 +658,7 @@ void bitmap_writer_select_commits(struct bitmap_writer *writer,\n \n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n-\t\t\tpush_bitmapped_commit(writer, indexed_commits[i]);\n+\t\t\tpush_bitmapped_commit(writer, indexed_commits[i], 0);\n \t\treturn;\n \t}\n \n@@ -664,7 +691,7 @@ void bitmap_writer_select_commits(struct bitmap_writer *writer,\n \t\t\t}\n \t\t}\n \n-\t\tpush_bitmapped_commit(writer, chosen);\n+\t\tpush_bitmapped_commit(writer, chosen, 0);\n \n \t\ti += next + 1;\n \t\tdisplay_progress(writer->progress, i);\n@@ -701,8 +728,11 @@ static void write_selected_commits_v1(struct bitmap_writer *writer,\n {\n \tint i;\n \n-\tfor (i = 0; i < writer->selected_nr; ++i) {\n+\tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); ++i) {\n \t\tstruct bitmapped_commit *stored = &writer->selected[i];\n+\t\tif (stored->pseudo_merge)\n+\t\t\tBUG(\"unexpected pseudo-merge among selected: %s\",\n+\t\t\t oid_to_hex(&stored->commit->object.oid));\n \n \t\tif (offsets)\n \t\t\toffsets[i] = hashfile_total(f);\n@@ -735,10 +765,10 @@ static void write_lookup_table(struct bitmap_writer *writer, struct hashfile *f,\n \tuint32_t i;\n \tuint32_t *table, *table_inv;\n \n-\tALLOC_ARRAY(table, writer->selected_nr);\n-\tALLOC_ARRAY(table_inv, writer->selected_nr);\n+\tALLOC_ARRAY(table, bitmap_writer_nr_selected_commits(writer));\n+\tALLOC_ARRAY(table_inv, bitmap_writer_nr_selected_commits(writer));\n \n-\tfor (i = 0; i < writer->selected_nr; i++)\n+\tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++)\n \t\ttable[i] = i;\n \n \t/*\n@@ -746,16 +776,16 @@ static void write_lookup_table(struct bitmap_writer *writer, struct hashfile *f,\n \t * bitmap corresponds to j'th bitmapped commit (among the selected\n \t * commits) in lex order of OIDs.\n \t */\n-\tQSORT_S(table, writer->selected_nr, table_cmp, writer);\n+\tQSORT_S(table, bitmap_writer_nr_selected_commits(writer), table_cmp, writer);\n \n \t/* table_inv helps us discover that relationship (i'th bitmap\n \t * to j'th commit by j = table_inv[i])\n \t */\n-\tfor (i = 0; i < writer->selected_nr; i++)\n+\tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++)\n \t\ttable_inv[table[i]] = i;\n \n \ttrace2_region_enter(\"pack-bitmap-write\", \"writing_lookup_table\", the_repository);\n-\tfor (i = 0; i < writer->selected_nr; i++) {\n+\tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++) {\n \t\tstruct bitmapped_commit *selected = &writer->selected[table[i]];\n \t\tuint32_t xor_offset = selected->xor_offset;\n \t\tuint32_t xor_row;\n@@ -827,7 +857,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \tmemcpy(header.magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE));\n \theader.version = htons(default_version);\n \theader.options = htons(flags | options);\n-\theader.entry_count = htonl(writer->selected_nr);\n+\theader.entry_count = htonl(bitmap_writer_nr_selected_commits(writer));\n \thashcpy(header.checksum, writer->pack_checksum);\n \n \thashwrite(f, &header, sizeof(header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz);\n@@ -839,7 +869,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n \t\tCALLOC_ARRAY(offsets, index_nr);\n \n-\tfor (i = 0; i < writer->selected_nr; i++) {\n+\tfor (i = 0; i < bitmap_writer_nr_selected_commits(writer); i++) {\n \t\tstruct bitmapped_commit *stored = &writer->selected[i];\n \t\tint commit_pos = oid_pos(&stored->commit->object.oid, index,\n \t\t\t\t\t index_nr, oid_access);\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex f87e60153dd..6937a0f090f 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -21,6 +21,7 @@ struct bitmap_disk_header {\n \tunsigned char checksum[GIT_MAX_RAWSZ];\n };\n \n+#define BITMAP_PSEUDO_MERGE (1u<<21)\n #define NEEDS_BITMAP (1u<<22)\n \n /*\n@@ -109,6 +110,8 @@ struct bitmap_writer {\n \tstruct bitmapped_commit *selected;\n \tunsigned int selected_nr, selected_alloc;\n \n+\tuint32_t pseudo_merges_nr;\n+\n \tstruct progress *progress;\n \tint show_progress;\n \tunsigned char pack_checksum[GIT_MAX_RAWSZ];\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495464","messageId":"fe458728c8a14a6b1fe9e86340c3ad79e7fa804e.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 08/24] pack-bitmap: implement `bitmap_writer_has_bitmapped_object_id()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:32Z","receivedAt":"2024-05-23T21:26:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to implement pseudo-merge bitmap selection by implementing a\nnecessary new function, `bitmap_writer_has_bitmapped_object_id()`.\n\nThis function returns whether or not the bitmap_writer selected the\ngiven object ID for bitmapping. This will allow the pseudo-merge\nmachinery to reject candidates for pseudo-merges if they have already\nbeen selected as an ordinary bitmap tip.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 6 ++++++\n pack-bitmap.h | 2 ++\n 2 files changed, 8 insertions(+)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 60eb1e71c98..299aa8af6f5 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -130,6 +130,12 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n \t}\n }\n \n+int bitmap_writer_has_bitmapped_object_id(struct bitmap_writer *writer,\n+\t\t\t\t\t const struct object_id *oid)\n+{\n+\treturn kh_get_oid_map(writer->bitmaps, *oid) != kh_end(writer->bitmaps);\n+}\n+\n /**\n * Compute the actual bitmaps\n */\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 6937a0f090f..e175f28e0de 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -125,6 +125,8 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n \t\t\t\t struct packing_data *to_pack,\n \t\t\t\t struct pack_idx_entry **index,\n \t\t\t\t uint32_t index_nr);\n+int bitmap_writer_has_bitmapped_object_id(struct bitmap_writer *writer,\n+\t\t\t\t\t const struct object_id *oid);\n uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \t\t\t\tstruct packing_data *mapping);\n int rebuild_bitmap(const uint32_t *reposition,\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495465","messageId":"6bf372f4020dda272ab4f69cb42333465475dd91.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 09/24] pack-bitmap: make `bitmap_writer_push_bitmapped_commit()` public","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:36Z","receivedAt":"2024-05-23T21:26:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pseudo-merge selection code will be added in a subsequent commit,\nand will need a way to push the allocated commit structures into the\nbitmap writer from a separate compilation unit.\n\nMake the `bitmap_writer_push_bitmapped_commit()` function part of the\npack-bitmap.h header in order to make this possible.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 9 ++++-----\n pack-bitmap.h | 2 ++\n 2 files changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 299aa8af6f5..bc19b33ad16 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -140,9 +140,8 @@ int bitmap_writer_has_bitmapped_object_id(struct bitmap_writer *writer,\n * Compute the actual bitmaps\n */\n \n-static inline void push_bitmapped_commit(struct bitmap_writer *writer,\n-\t\t\t\t\t struct commit *commit,\n-\t\t\t\t\t unsigned pseudo_merge)\n+void bitmap_writer_push_commit(struct bitmap_writer *writer,\n+\t\t\t struct commit *commit, unsigned pseudo_merge)\n {\n \tif (writer->selected_nr >= writer->selected_alloc) {\n \t\twriter->selected_alloc = (writer->selected_alloc + 32) * 2;\n@@ -664,7 +663,7 @@ void bitmap_writer_select_commits(struct bitmap_writer *writer,\n \n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n-\t\t\tpush_bitmapped_commit(writer, indexed_commits[i], 0);\n+\t\t\tbitmap_writer_push_commit(writer, indexed_commits[i], 0);\n \t\treturn;\n \t}\n \n@@ -697,7 +696,7 @@ void bitmap_writer_select_commits(struct bitmap_writer *writer,\n \t\t\t}\n \t\t}\n \n-\t\tpush_bitmapped_commit(writer, chosen, 0);\n+\t\tbitmap_writer_push_commit(writer, chosen, 0);\n \n \t\ti += next + 1;\n \t\tdisplay_progress(writer->progress, i);\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex e175f28e0de..a7e2f56c971 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -127,6 +127,8 @@ void bitmap_writer_build_type_index(struct bitmap_writer *writer,\n \t\t\t\t uint32_t index_nr);\n int bitmap_writer_has_bitmapped_object_id(struct bitmap_writer *writer,\n \t\t\t\t\t const struct object_id *oid);\n+void bitmap_writer_push_commit(struct bitmap_writer *writer,\n+\t\t\t struct commit *commit, unsigned pseudo_merge);\n uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git,\n \t\t\t\tstruct packing_data *mapping);\n int rebuild_bitmap(const uint32_t *reposition,\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495466","messageId":"6c77671ae9ce063d5659ab0542d4bfa3e1303995.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 10/24] config: introduce `git_config_double()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:39Z","receivedAt":"2024-05-23T21:26:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Future commits will want to parse a double-precision floating point\nvalue from configuration, but we have no way to parse such a value prior\nto this patch.\n\nThe core of the routine is implemented in git_parse_double(). Unlike\ngit_parse_unsigned() and git_parse_signed(), however, the function\nimplemented here only works on type \"double\", and not related types like\n\"float\", or \"long double\".\n\nThis is because \"float\" and \"long double\" use different functions to\nconvert from ASCII strings to floating point values (strtof() and\nstrtold(), respectively). Likewise, there is no pointer type that can\nassign to any of these values (except for \"void *\"), so the only way to\ndefine this trio of functions would be with a macro expansion that is\nparameterized over the floating point type and conversion function.\n\nThat is all doable, but likely to be overkill given our current needs,\nwhich is only to parse double-precision floats.\n\nSigned-off-by: Taylor Blau \n---\n config.c | 9 +++++++++\n config.h | 7 +++++++\n parse.c | 29 +++++++++++++++++++++++++++++\n parse.h | 1 +\n 4 files changed, 46 insertions(+)\n\ndiff --git a/config.c b/config.c\nindex 77a0fd2d80e..7df89f17275 100644\n--- a/config.c\n+++ b/config.c\n@@ -1243,6 +1243,15 @@ ssize_t git_config_ssize_t(const char *name, const char *value,\n \treturn ret;\n }\n \n+double git_config_double(const char *name, const char *value,\n+\t\t\t const struct key_value_info *kvi)\n+{\n+\tdouble ret;\n+\tif (!git_parse_double(value, &ret))\n+\t\tdie_bad_number(name, value, kvi);\n+\treturn ret;\n+}\n+\n static const struct fsync_component_name {\n \tconst char *name;\n \tenum fsync_component component_bits;\ndiff --git a/config.h b/config.h\nindex f4966e37494..f5f306f373d 100644\n--- a/config.h\n+++ b/config.h\n@@ -261,6 +261,13 @@ unsigned long git_config_ulong(const char *, const char *,\n ssize_t git_config_ssize_t(const char *, const char *,\n \t\t\t const struct key_value_info *);\n \n+/**\n+ * Identically to `git_config_double`, but for double-precision floating point\n+ * values.\n+ */\n+double git_config_double(const char *, const char *,\n+\t\t\t const struct key_value_info *);\n+\n /**\n * Same as `git_config_bool`, except that integers are returned as-is, and\n * an `is_bool` flag is unset.\ndiff --git a/parse.c b/parse.c\nindex 42d691a0fbb..7a60a4f816c 100644\n--- a/parse.c\n+++ b/parse.c\n@@ -125,6 +125,35 @@ int git_parse_ssize_t(const char *value, ssize_t *ret)\n \treturn 1;\n }\n \n+int git_parse_double(const char *value, double *ret)\n+{\n+\tchar *end;\n+\tdouble val;\n+\tuintmax_t factor;\n+\n+\tif (!value || !*value) {\n+\t\terrno = EINVAL;\n+\t\treturn 0;\n+\t}\n+\n+\terrno = 0;\n+\tval = strtod(value, &end);\n+\tif (errno == ERANGE)\n+\t\treturn 0;\n+\tif (end == value) {\n+\t\terrno = EINVAL;\n+\t\treturn 0;\n+\t}\n+\tfactor = get_unit_factor(end);\n+\tif (!factor) {\n+\t\terrno = EINVAL;\n+\t\treturn 0;\n+\t}\n+\tval *= factor;\n+\t*ret = val;\n+\treturn 1;\n+}\n+\n int git_parse_maybe_bool_text(const char *value)\n {\n \tif (!value)\ndiff --git a/parse.h b/parse.h\nindex 07d2193d698..6bb9a54d9ac 100644\n--- a/parse.h\n+++ b/parse.h\n@@ -6,6 +6,7 @@ int git_parse_ssize_t(const char *, ssize_t *);\n int git_parse_ulong(const char *, unsigned long *);\n int git_parse_int(const char *value, int *ret);\n int git_parse_int64(const char *value, int64_t *ret);\n+int git_parse_double(const char *value, double *ret);\n \n /**\n * Same as `git_config_bool`, except that it returns -1 on error rather\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495467","messageId":"180072ce84868265acfda8c1adf375e39a3b7610.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 11/24] pseudo-merge: implement support for selecting pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:42Z","receivedAt":"2024-05-23T21:26:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Teach the new pseudo-merge machinery how to select non-bitmapped commits\nfor inclusion in different pseudo-merge group(s) based on a handful of\ncriteria.\n\nNote that the selected pseudo-merge commits aren't actually used or\nwritten anywhere yet. This will be done in the following commit.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config.txt | 2 +\n Documentation/config/bitmap-pseudo-merge.txt | 91 ++++\n Documentation/gitpacking.txt | 83 ++++\n pack-bitmap-write.c | 21 +\n pack-bitmap.h | 2 +\n pseudo-merge.c | 454 +++++++++++++++++++\n pseudo-merge.h | 94 ++++\n 7 files changed, 747 insertions(+)\n create mode 100644 Documentation/config/bitmap-pseudo-merge.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 6f649c997c0..caa34311214 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -384,6 +384,8 @@ include::config/apply.txt[]\n \n include::config/attr.txt[]\n \n+include::config/bitmap-pseudo-merge.txt[]\n+\n include::config/blame.txt[]\n \n include::config/branch.txt[]\ndiff --git a/Documentation/config/bitmap-pseudo-merge.txt b/Documentation/config/bitmap-pseudo-merge.txt\nnew file mode 100644\nindex 00000000000..1f264eca99b\n--- /dev/null\n+++ b/Documentation/config/bitmap-pseudo-merge.txt\n@@ -0,0 +1,91 @@\n+NOTE: The configuration options in `bitmapPseudoMerge.*` are considered\n+EXPERIMENTAL and may be subject to change or be removed entirely in the\n+future. For more information about the pseudo-merge bitmap feature, see\n+the \"Pseudo-merge bitmaps\" section of linkgit:gitpacking[7].\n+\n+bitmapPseudoMerge..pattern::\n+\tRegular expression used to match reference names. Commits\n+\tpointed to by references matching this pattern (and meeting\n+\tthe below criteria, like `bitmapPseudoMerge..sampleRate`\n+\tand `bitmapPseudoMerge..threshold`) will be considered\n+\tfor inclusion in a pseudo-merge bitmap.\n++\n+Commits are grouped into pseudo-merge groups based on whether or not\n+any reference(s) that point at a given commit match the pattern, which\n+is an extended regular expression.\n++\n+Within a pseudo-merge group, commits may be further grouped into\n+sub-groups based on the capture groups in the pattern. These\n+sub-groupings are formed from the regular expressions by concatenating\n+any capture groups from the regular expression, with a '-' dash in\n+between.\n++\n+For example, if the pattern is `refs/tags/`, then all tags (provided\n+they meet the below criteria) will be considered candidates for the\n+same pseudo-merge group. However, if the pattern is instead\n+`refs/remotes/([0-9])+/tags/`, then tags from different remotes will\n+be grouped into separate pseudo-merge groups, based on the remote\n+number.\n+\n+bitmapPseudoMerge..decay::\n+\tDetermines the rate at which consecutive pseudo-merge bitmap\n+\tgroups decrease in size. Must be non-negative. This parameter\n+\tcan be thought of as `k` in the function `f(n) = C * n^-k`,\n+\twhere `f(n)` is the size of the `n`th group.\n++\n+Setting the decay rate equal to `0` will cause all groups to be the\n+same size. Setting the decay rate equal to `1` will cause the `n`th\n+group to be `1/n` the size of the initial group. Higher values of the\n+decay rate cause consecutive groups to shrink at an increasing rate.\n+The default is `1`.\n++\n+If all groups are the same size, it is possible that groups containing\n+newer commits will be able to be used less often than earlier groups,\n+since it is more likely that the references pointing at newer commits\n+will be updated more often than a reference pointing at an old commit.\n+\n+bitmapPseudoMerge..sampleRate::\n+\tDetermines the proportion of non-bitmapped commits (among\n+\treference tips) which are selected for inclusion in an\n+\tunstable pseudo-merge bitmap. Must be between `0` and `1`\n+\t(inclusive). The default is `1`.\n+\n+bitmapPseudoMerge..threshold::\n+\tDetermines the minimum age of non-bitmapped commits (among\n+\treference tips, as above) which are candidates for inclusion\n+\tin an unstable pseudo-merge bitmap. The default is\n+\t`1.week.ago`.\n+\n+bitmapPseudoMerge..maxMerges::\n+\tDetermines the maximum number of pseudo-merge commits among\n+\twhich commits may be distributed.\n++\n+For pseudo-merge groups whose pattern does not contain any capture\n+groups, this setting is applied for all commits matching the regular\n+expression. For patterns that have one or more capture groups, this\n+setting is applied for each distinct capture group.\n++\n+For example, if your capture group is `refs/tags/`, then this setting\n+will distribute all tags into a maximum of `maxMerges` pseudo-merge\n+commits. However, if your capture group is, say,\n+`refs/remotes/([0-9]+)/tags/`, then this setting will be applied to\n+each remote's set of tags individually.\n++\n+Must be non-negative. The default value is 64.\n+\n+bitmapPseudoMerge..stableThreshold::\n+\tDetermines the minimum age of commits (among reference tips,\n+\tas above, however stable commits are still considered\n+\tcandidates even when they have been covered by a bitmap) which\n+\tare candidates for a stable a pseudo-merge bitmap. The default\n+\tis `1.month.ago`.\n++\n+Setting this threshold to a smaller value (e.g., 1.week.ago) will cause\n+more stable groups to be generated (which impose a one-time generation\n+cost) but those groups will likely become stale over time. Using a\n+larger value incurs the opposite penalty (fewer stable groups which are\n+more useful).\n+\n+bitmapPseudoMerge..stableSize::\n+\tDetermines the size (in number of commits) of a stable\n+\tpsuedo-merge bitmap. The default is `512`.\ndiff --git a/Documentation/gitpacking.txt b/Documentation/gitpacking.txt\nindex f24396f0173..4a6fcba6f72 100644\n--- a/Documentation/gitpacking.txt\n+++ b/Documentation/gitpacking.txt\n@@ -96,6 +96,89 @@ can take advantage of the fact that we only care about the union of\n objects reachable from all of those tags, and answer the query much\n faster.\n \n+=== Configuration\n+\n+Reference tips are grouped into different pseudo-merge groups according\n+to two criteria. A reference name matches one or more of the defined\n+pseudo-merge patterns, and optionally one or more capture groups within\n+that pattern which further partition the group.\n+\n+Within a group, commits may be considered \"stable\", or \"unstable\"\n+depending on their age. These are adjusted by setting the\n+`bitmapPseudoMerge..stableThreshold` and\n+`bitmapPseudoMerge..threshold` configuration values, respectively.\n+\n+All stable commits are grouped into pseudo-merges of equal size\n+(`bitmapPseudoMerge..stableSize`). If the `stableSize`\n+configuration is set to, say, 100, then the first 100 commits (ordered\n+by committer date) which are older than the `stableThreshold` value will\n+form one group, the next 100 commits will form another group, and so on.\n+\n+Among unstable commits, the pseudo-merge machinery will attempt to\n+combine older commits into large groups as opposed to newer commits\n+which will appear in smaller groups. This is based on the heuristic that\n+references whose tip commit is older are less likely to be modified to\n+point at a different commit than a reference whose tip commit is newer.\n+\n+The size of groups is determined by a power-law decay function, and the\n+decay parameter roughly corresponds to \"k\" in `f(n) = C*n^(-k/100)`,\n+where `f(n)` describes the size of the `n`-th pseudo-merge group. The\n+sample rate controls what percentage of eligible commits are considered\n+as candidates. The threshold parameter indicates the minimum age (so as\n+to avoid including too-recent commits in a pseudo-merge group, making it\n+less likely to be valid). The \"maxMerges\" parameter sets an upper-bound\n+on the number of pseudo-merge commits an individual group\n+\n+The \"stable\"-related parameters control \"stable\" pseudo-merge groups,\n+comprised of a fixed number of commits which are older than the\n+configured \"stable threshold\" value and may be grouped together in\n+chunks of \"stableSize\" in order of age.\n+\n+The exact configuration for pseudo-merges is as follows:\n+\n+include::config/bitmap-pseudo-merge.txt[]\n+\n+=== Examples\n+\n+Suppose that you have a repository with a large number of references,\n+and you want a bare-bones configuration of pseudo-merge bitmaps that\n+will enhance bitmap coverage of the `refs/` namespace. You may start\n+wiht a configuration like so:\n+\n+ [bitmapPseudoMerge \"all\"]\n+\tpattern = \"refs/\"\n+\tthreshold = now\n+\tstableThreshold = never\n+\tsampleRate = 100\n+\tmaxMerges = 64\n+\n+This will create pseudo-merge bitmaps for all references, regardless of\n+their age, and group them into 64 pseudo-merge commits.\n+\n+If you wanted to separate tags from branches when generating\n+pseudo-merge commits, you would instead define the pattern with a\n+capture group, like so:\n+\n+ [bitmapPseudoMerge \"all\"]\n+\tpattern = \"refs/(heads/tags)/\"\n+\n+Suppose instead that you are working in a fork-network repository, with\n+each fork specified by some numeric ID, and whose refs reside in\n+`refs/virtual/NNN/` (where `NNN` is the numeric ID corresponding to some\n+fork) in the network. In this instance, you may instead write something\n+like:\n+\n+ [bitmapPseudoMerge \"all\"]\n+\tpattern = \"refs/virtual/([0-9]+)/(heads|tags)/\"\n+\tthreshold = now\n+\tstableThreshold = never\n+\tsampleRate = 100\n+\tmaxMerges = 64\n+\n+Which would generate pseudo-merge group identifiers like \"1234-heads\",\n+and \"5678-tags\" (for branches in fork \"1234\", and tags in remote \"5678\",\n+respectively).\n+\n SEE ALSO\n --------\n linkgit:git-pack-objects[1]\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex bc19b33ad16..d5884ea5e9c 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -17,6 +17,7 @@\n #include \"trace2.h\"\n #include \"tree.h\"\n #include \"tree-walk.h\"\n+#include \"pseudo-merge.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -39,11 +40,25 @@ void bitmap_writer_init(struct bitmap_writer *writer, struct repository *r)\n \tif (writer->bitmaps)\n \t\tBUG(\"bitmap writer already initialized\");\n \twriter->bitmaps = kh_init_oid_map();\n+\twriter->pseudo_merge_commits = kh_init_oid_map();\n+\n+\tstring_list_init_dup(&writer->pseudo_merge_groups);\n+\n+\tload_pseudo_merges_from_config(&writer->pseudo_merge_groups);\n+}\n+\n+static void free_pseudo_merge_commit_idx(struct pseudo_merge_commit_idx *idx)\n+{\n+\tif (!idx)\n+\t\treturn;\n+\tfree(idx->pseudo_merge);\n+\tfree(idx);\n }\n \n void bitmap_writer_free(struct bitmap_writer *writer)\n {\n \tuint32_t i;\n+\tstruct pseudo_merge_commit_idx *idx;\n \n \tif (!writer)\n \t\treturn;\n@@ -55,6 +70,10 @@ void bitmap_writer_free(struct bitmap_writer *writer)\n \n \tkh_destroy_oid_map(writer->bitmaps);\n \n+\tkh_foreach_value(writer->pseudo_merge_commits, idx,\n+\t\t\t free_pseudo_merge_commit_idx(idx));\n+\tkh_destroy_oid_map(writer->pseudo_merge_commits);\n+\n \tfor (i = 0; i < writer->selected_nr; i++) {\n \t\tstruct bitmapped_commit *bc = &writer->selected[i];\n \t\tif (bc->write_as != bc->bitmap)\n@@ -703,6 +722,8 @@ void bitmap_writer_select_commits(struct bitmap_writer *writer,\n \t}\n \n \tstop_progress(&writer->progress);\n+\n+\tselect_pseudo_merges(writer, indexed_commits, indexed_commits_nr);\n }\n \n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex a7e2f56c971..1e730ea1e54 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -110,6 +110,8 @@ struct bitmap_writer {\n \tstruct bitmapped_commit *selected;\n \tunsigned int selected_nr, selected_alloc;\n \n+\tstruct string_list pseudo_merge_groups;\n+\tkh_oid_map_t *pseudo_merge_commits; /* oid -> pseudo merge(s) */\n \tuint32_t pseudo_merges_nr;\n \n \tstruct progress *progress;\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex 37e037ba272..0f6854c753f 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -1,2 +1,456 @@\n #include \"git-compat-util.h\"\n #include \"pseudo-merge.h\"\n+#include \"date.h\"\n+#include \"oid-array.h\"\n+#include \"strbuf.h\"\n+#include \"config.h\"\n+#include \"string-list.h\"\n+#include \"refs.h\"\n+#include \"pack-bitmap.h\"\n+#include \"commit.h\"\n+#include \"alloc.h\"\n+#include \"progress.h\"\n+\n+#define DEFAULT_PSEUDO_MERGE_DECAY 1.0\n+#define DEFAULT_PSEUDO_MERGE_MAX_MERGES 64\n+#define DEFAULT_PSEUDO_MERGE_SAMPLE_RATE 1\n+#define DEFAULT_PSEUDO_MERGE_THRESHOLD approxidate(\"1.week.ago\")\n+#define DEFAULT_PSEUDO_MERGE_STABLE_THRESHOLD approxidate(\"1.month.ago\")\n+#define DEFAULT_PSEUDO_MERGE_STABLE_SIZE 512\n+\n+static double gitexp(double base, int exp)\n+{\n+\tdouble result = 1;\n+\twhile (1) {\n+\t\tif (exp % 2)\n+\t\t\tresult *= base;\n+\t\texp >>= 1;\n+\t\tif (!exp)\n+\t\t\tbreak;\n+\t\tbase *= base;\n+\t}\n+\treturn result;\n+}\n+\n+static uint32_t pseudo_merge_group_size(const struct pseudo_merge_group *group,\n+\t\t\t\t\tconst struct pseudo_merge_matches *matches,\n+\t\t\t\t\tuint32_t i)\n+{\n+\tdouble C = 0.0f;\n+\tuint32_t n;\n+\n+\t/*\n+\t * The size of pseudo-merge groups decays according to a power series,\n+\t * which looks like:\n+\t *\n+\t * f(n) = C * n^-k\n+\t *\n+\t * , where 'n' is the n-th pseudo-merge group, 'f(n)' is its size, 'k'\n+\t * is the decay rate, and 'C' is a scaling value.\n+\t *\n+\t * The value of C depends on the number of groups, decay rate, and total\n+\t * number of commits. It is computed such that if there are M and N\n+\t * total groups and commits, respectively, that:\n+\t *\n+\t * N = f(0) + f(1) + ... f(M-1)\n+\t *\n+\t * Rearranging to isolate C, we get:\n+\t *\n+\t * N = \\sum_{n=1}^M C / n^k\n+\t *\n+\t * N / C = \\sum_{n=1}^M n^-k\n+\t *\n+\t * C = N / \\sum_{n=1}^M n^-k\n+\t *\n+\t * For example, if we have a decay rate of 'k' being equal to 1.5, 'N'\n+\t * total commits equal to 10,000, and 'M' being equal to 6 groups, then\n+\t * the (rounded) group sizes are:\n+\t *\n+\t * { 5469, 1934, 1053, 684, 489, 372 }\n+\t *\n+\t * increasing the number of total groups, say to 10, scales the group\n+\t * sizes appropriately:\n+\t *\n+\t * { 5012, 1772, 964, 626, 448, 341, 271, 221, 186, 158 }\n+\t */\n+\tfor (n = 0; n < group->max_merges; n++)\n+\t\tC += 1.0 / gitexp(n + 1, group->decay);\n+\tC = matches->unstable_nr / C;\n+\n+\treturn (uint32_t)((C / gitexp(i + 1, group->decay)) + 0.5);\n+}\n+\n+static void pseudo_merge_group_init(struct pseudo_merge_group *group)\n+{\n+\tmemset(group, 0, sizeof(struct pseudo_merge_group));\n+\n+\tstrmap_init_with_options(&group->matches, NULL, 0);\n+\n+\tgroup->decay = DEFAULT_PSEUDO_MERGE_DECAY;\n+\tgroup->max_merges = DEFAULT_PSEUDO_MERGE_MAX_MERGES;\n+\tgroup->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE;\n+\tgroup->threshold = DEFAULT_PSEUDO_MERGE_THRESHOLD;\n+\tgroup->stable_threshold = DEFAULT_PSEUDO_MERGE_STABLE_THRESHOLD;\n+\tgroup->stable_size = DEFAULT_PSEUDO_MERGE_STABLE_SIZE;\n+}\n+\n+static int pseudo_merge_config(const char *var, const char *value,\n+\t\t\t const struct config_context *ctx,\n+\t\t\t void *cb_data)\n+{\n+\tstruct string_list *list = cb_data;\n+\tstruct string_list_item *item;\n+\tstruct pseudo_merge_group *group;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *sub, *key;\n+\tsize_t sub_len;\n+\tint ret = 0;\n+\n+\tif (parse_config_key(var, \"bitmappseudomerge\", &sub, &sub_len, &key))\n+\t\tgoto done;\n+\n+\tif (!sub_len)\n+\t\tgoto done;\n+\n+\tstrbuf_add(&buf, sub, sub_len);\n+\n+\titem = string_list_lookup(list, buf.buf);\n+\tif (!item) {\n+\t\titem = string_list_insert(list, buf.buf);\n+\n+\t\titem->util = xmalloc(sizeof(struct pseudo_merge_group));\n+\t\tpseudo_merge_group_init(item->util);\n+\t}\n+\n+\tgroup = item->util;\n+\n+\tif (!strcmp(key, \"pattern\")) {\n+\t\tstruct strbuf re = STRBUF_INIT;\n+\n+\t\tfree(group->pattern);\n+\t\tif (*value != '^')\n+\t\t\tstrbuf_addch(&re, '^');\n+\t\tstrbuf_addstr(&re, value);\n+\n+\t\tgroup->pattern = xcalloc(1, sizeof(regex_t));\n+\t\tif (regcomp(group->pattern, re.buf, REG_EXTENDED))\n+\t\t\tdie(_(\"failed to load pseudo-merge regex for %s: '%s'\"),\n+\t\t\t sub, re.buf);\n+\n+\t\tstrbuf_release(&re);\n+\t} else if (!strcmp(key, \"decay\")) {\n+\t\tgroup->decay = git_config_double(var, value, ctx->kvi);\n+\t\tif (group->decay < 0) {\n+\t\t\twarning(_(\"%s must be non-negative, using default\"), var);\n+\t\t\tgroup->decay = DEFAULT_PSEUDO_MERGE_DECAY;\n+\t\t}\n+\t} else if (!strcmp(key, \"samplerate\")) {\n+\t\tgroup->sample_rate = git_config_double(var, value, ctx->kvi);\n+\t\tif (!(0 <= group->sample_rate && group->sample_rate <= 1)) {\n+\t\t\twarning(_(\"%s must be between 0 and 1, using default\"), var);\n+\t\t\tgroup->sample_rate = DEFAULT_PSEUDO_MERGE_SAMPLE_RATE;\n+\t\t}\n+\t} else if (!strcmp(key, \"threshold\")) {\n+\t\tif (git_config_expiry_date(&group->threshold, var, value)) {\n+\t\t\tret = -1;\n+\t\t\tgoto done;\n+\t\t}\n+\t} else if (!strcmp(key, \"maxmerges\")) {\n+\t\tgroup->max_merges = git_config_int(var, value, ctx->kvi);\n+\t\tif (group->max_merges < 0) {\n+\t\t\twarning(_(\"%s must be non-negative, using default\"), var);\n+\t\t\tgroup->max_merges = DEFAULT_PSEUDO_MERGE_MAX_MERGES;\n+\t\t}\n+\t} else if (!strcmp(key, \"stablethreshold\")) {\n+\t\tif (git_config_expiry_date(&group->stable_threshold, var, value)) {\n+\t\t\tret = -1;\n+\t\t\tgoto done;\n+\t\t}\n+\t} else if (!strcmp(key, \"stablesize\")) {\n+\t\tgroup->stable_size = git_config_int(var, value, ctx->kvi);\n+\t\tif (group->stable_size <= 0) {\n+\t\t\twarning(_(\"%s must be positive, using default\"), var);\n+\t\t\tgroup->stable_size = DEFAULT_PSEUDO_MERGE_STABLE_SIZE;\n+\t\t}\n+\t}\n+\n+done:\n+\tstrbuf_release(&buf);\n+\n+\treturn ret;\n+}\n+\n+void load_pseudo_merges_from_config(struct string_list *list)\n+{\n+\tstruct string_list_item *item;\n+\n+\tgit_config(pseudo_merge_config, list);\n+\n+\tfor_each_string_list_item(item, list) {\n+\t\tstruct pseudo_merge_group *group = item->util;\n+\t\tif (!group->pattern)\n+\t\t\tdie(_(\"pseudo-merge group '%s' missing required pattern\"),\n+\t\t\t item->string);\n+\t\tif (group->threshold < group->stable_threshold)\n+\t\t\tdie(_(\"pseudo-merge group '%s' has unstable threshold \"\n+\t\t\t \"before stable one\"), item->string);\n+\t}\n+}\n+\n+static int find_pseudo_merge_group_for_ref(const char *refname,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t int flags UNUSED,\n+\t\t\t\t\t void *_data)\n+{\n+\tstruct bitmap_writer *writer = _data;\n+\tstruct object_id peeled;\n+\tstruct commit *c;\n+\tuint32_t i;\n+\tint has_bitmap;\n+\n+\tif (!peel_iterated_oid(oid, &peeled))\n+\t\toid = &peeled;\n+\n+\tc = lookup_commit(the_repository, oid);\n+\tif (!c)\n+\t\treturn 0;\n+\n+\thas_bitmap = bitmap_writer_has_bitmapped_object_id(writer, oid);\n+\n+\tfor (i = 0; i < writer->pseudo_merge_groups.nr; i++) {\n+\t\tstruct pseudo_merge_group *group;\n+\t\tstruct pseudo_merge_matches *matches;\n+\t\tstruct strbuf group_name = STRBUF_INIT;\n+\t\tregmatch_t captures[16];\n+\t\tsize_t j;\n+\n+\t\tgroup = writer->pseudo_merge_groups.items[i].util;\n+\t\tif (regexec(group->pattern, refname, ARRAY_SIZE(captures),\n+\t\t\t captures, 0))\n+\t\t\tcontinue;\n+\n+\t\tif (captures[ARRAY_SIZE(captures) - 1].rm_so != -1)\n+\t\t\twarning(_(\"pseudo-merge regex from config has too many capture \"\n+\t\t\t\t \"groups (max=%\"PRIuMAX\")\"),\n+\t\t\t\t(uintmax_t)ARRAY_SIZE(captures) - 2);\n+\n+\t\tfor (j = !!group->pattern->re_nsub; j < ARRAY_SIZE(captures); j++) {\n+\t\t\tregmatch_t *match = &captures[j];\n+\t\t\tif (match->rm_so == -1)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (group_name.len)\n+\t\t\t\tstrbuf_addch(&group_name, '-');\n+\n+\t\t\tstrbuf_add(&group_name, refname + match->rm_so,\n+\t\t\t\t match->rm_eo - match->rm_so);\n+\t\t}\n+\n+\t\tmatches = strmap_get(&group->matches, group_name.buf);\n+\t\tif (!matches) {\n+\t\t\tmatches = xcalloc(1, sizeof(*matches));\n+\t\t\tstrmap_put(&group->matches, strbuf_detach(&group_name, NULL),\n+\t\t\t\t matches);\n+\t\t}\n+\n+\t\tif (c->date <= group->stable_threshold) {\n+\t\t\tALLOC_GROW(matches->stable, matches->stable_nr + 1,\n+\t\t\t\t matches->stable_alloc);\n+\t\t\tmatches->stable[matches->stable_nr++] = c;\n+\t\t} else if (c->date <= group->threshold && !has_bitmap) {\n+\t\t\tALLOC_GROW(matches->unstable, matches->unstable_nr + 1,\n+\t\t\t\t matches->unstable_alloc);\n+\t\t\tmatches->unstable[matches->unstable_nr++] = c;\n+\t\t}\n+\n+\t\tstrbuf_release(&group_name);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static struct commit *push_pseudo_merge(struct pseudo_merge_group *group)\n+{\n+\tstruct commit *merge;\n+\n+\tALLOC_GROW(group->merges, group->merges_nr + 1, group->merges_alloc);\n+\n+\tmerge = alloc_commit_node(the_repository);\n+\tmerge->object.parsed = 1;\n+\tmerge->object.flags |= BITMAP_PSEUDO_MERGE;\n+\n+\tgroup->merges[group->merges_nr++] = merge;\n+\n+\treturn merge;\n+}\n+\n+static struct pseudo_merge_commit_idx *pseudo_merge_idx(kh_oid_map_t *pseudo_merge_commits,\n+\t\t\t\t\t\t\tconst struct object_id *oid)\n+\n+{\n+\tstruct pseudo_merge_commit_idx *pmc;\n+\tint hash_ret;\n+\tkhiter_t hash_pos = kh_put_oid_map(pseudo_merge_commits, *oid,\n+\t\t\t\t\t &hash_ret);\n+\n+\tif (hash_ret) {\n+\t\tCALLOC_ARRAY(pmc, 1);\n+\t\tkh_value(pseudo_merge_commits, hash_pos) = pmc;\n+\t} else {\n+\t\tpmc = kh_value(pseudo_merge_commits, hash_pos);\n+\t}\n+\n+\treturn pmc;\n+}\n+\n+#define MIN_PSEUDO_MERGE_SIZE 8\n+\n+static void select_pseudo_merges_1(struct bitmap_writer *writer,\n+\t\t\t\t struct pseudo_merge_group *group,\n+\t\t\t\t struct pseudo_merge_matches *matches)\n+{\n+\tuint32_t i, j;\n+\tuint32_t stable_merges_nr;\n+\n+\tif (!matches->stable_nr && !matches->unstable_nr)\n+\t\treturn; /* all tips in this group already have bitmaps */\n+\n+\tstable_merges_nr = matches->stable_nr / group->stable_size;\n+\tif (matches->stable_nr % group->stable_size)\n+\t\tstable_merges_nr++;\n+\n+\t/* make stable_merges_nr pseudo merges for stable commits */\n+\tfor (i = 0, j = 0; i < stable_merges_nr; i++) {\n+\t\tstruct commit *merge;\n+\t\tstruct commit_list **p;\n+\n+\t\tmerge = push_pseudo_merge(group);\n+\t\tp = &merge->parents;\n+\n+\t\t/*\n+\t\t * For each pseudo-merge created above, add parents to the\n+\t\t * allocated commit node from the stable set of commits\n+\t\t * (un-bitmapped, newer than the stable threshold).\n+\t\t */\n+\t\tdo {\n+\t\t\tstruct commit *c;\n+\t\t\tstruct pseudo_merge_commit_idx *pmc;\n+\n+\t\t\tif (j >= matches->stable_nr)\n+\t\t\t\tbreak;\n+\n+\t\t\tc = matches->stable[j++];\n+\t\t\t/*\n+\t\t\t * Here and below, make sure that we keep our mapping of\n+\t\t\t * commits -> pseudo-merge(s) which include the key'd\n+\t\t\t * commit up-to-date.\n+\t\t\t */\n+\t\t\tpmc = pseudo_merge_idx(writer->pseudo_merge_commits,\n+\t\t\t\t\t &c->object.oid);\n+\n+\t\t\tALLOC_GROW(pmc->pseudo_merge, pmc->nr + 1, pmc->alloc);\n+\n+\t\t\tpmc->pseudo_merge[pmc->nr++] = writer->pseudo_merges_nr;\n+\t\t\tp = commit_list_append(c, p);\n+\t\t} while (j % group->stable_size);\n+\n+\t\tbitmap_writer_push_commit(writer, merge, 1);\n+\t\twriter->pseudo_merges_nr++;\n+\t}\n+\n+\t/* make up to group->max_merges pseudo merges for unstable commits */\n+\tfor (i = 0, j = 0; i < group->max_merges; i++) {\n+\t\tstruct commit *merge;\n+\t\tstruct commit_list **p;\n+\t\tuint32_t size, end;\n+\n+\t\tmerge = push_pseudo_merge(group);\n+\t\tp = &merge->parents;\n+\n+\t\tsize = pseudo_merge_group_size(group, matches, i);\n+\t\tend = size < MIN_PSEUDO_MERGE_SIZE ? matches->unstable_nr : j + size;\n+\n+\t\t/*\n+\t\t * For each pseudo-merge commit created above, add parents to\n+\t\t * the allocated commit node from the unstable set of commits\n+\t\t * (newer than the stable threshold).\n+\t\t *\n+\t\t * Account for the sample rate, since not every candidate from\n+\t\t * the set of stable commits will be included as a pseudo-merge\n+\t\t * parent.\n+\t\t */\n+\t\tfor (; j < end && j < matches->unstable_nr; j++) {\n+\t\t\tstruct commit *c = matches->unstable[j];\n+\t\t\tstruct pseudo_merge_commit_idx *pmc;\n+\n+\t\t\tif (j % (uint32_t)(1.0 / group->sample_rate))\n+\t\t\t\tcontinue;\n+\n+\t\t\tpmc = pseudo_merge_idx(writer->pseudo_merge_commits,\n+\t\t\t\t\t &c->object.oid);\n+\n+\t\t\tALLOC_GROW(pmc->pseudo_merge, pmc->nr + 1, pmc->alloc);\n+\n+\t\t\tpmc->pseudo_merge[pmc->nr++] = writer->pseudo_merges_nr;\n+\t\t\tp = commit_list_append(c, p);\n+\t\t}\n+\n+\t\tbitmap_writer_push_commit(writer, merge, 1);\n+\t\twriter->pseudo_merges_nr++;\n+\t\tif (end >= matches->unstable_nr)\n+\t\t\tbreak;\n+\t}\n+}\n+\n+static int commit_date_cmp(const void *va, const void *vb)\n+{\n+\ttimestamp_t a = (*(const struct commit **)va)->date;\n+\ttimestamp_t b = (*(const struct commit **)vb)->date;\n+\n+\tif (a < b)\n+\t\treturn -1;\n+\telse if (a > b)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static void sort_pseudo_merge_matches(struct pseudo_merge_matches *matches)\n+{\n+\tQSORT(matches->stable, matches->stable_nr, commit_date_cmp);\n+\tQSORT(matches->unstable, matches->unstable_nr, commit_date_cmp);\n+}\n+\n+void select_pseudo_merges(struct bitmap_writer *writer,\n+\t\t\t struct commit **commits, size_t commits_nr)\n+{\n+\tstruct progress *progress = NULL;\n+\tuint32_t i;\n+\n+\tif (!writer->pseudo_merge_groups.nr)\n+\t\treturn;\n+\n+\tif (writer->show_progress)\n+\t\tprogress = start_progress(\"Selecting pseudo-merge commits\",\n+\t\t\t\t\t writer->pseudo_merge_groups.nr);\n+\n+\tfor_each_ref(find_pseudo_merge_group_for_ref, writer);\n+\n+\tfor (i = 0; i < writer->pseudo_merge_groups.nr; i++) {\n+\t\tstruct pseudo_merge_group *group;\n+\t\tstruct hashmap_iter iter;\n+\t\tstruct strmap_entry *e;\n+\n+\t\tgroup = writer->pseudo_merge_groups.items[i].util;\n+\t\tstrmap_for_each_entry(&group->matches, &iter, e) {\n+\t\t\tstruct pseudo_merge_matches *matches = e->value;\n+\n+\t\t\tsort_pseudo_merge_matches(matches);\n+\n+\t\t\tselect_pseudo_merges_1(writer, group, matches);\n+\t\t}\n+\n+\t\tdisplay_progress(progress, i + 1);\n+\t}\n+\n+\tstop_progress(&progress);\n+}\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nindex cab8ff6960a..f809cf42aeb 100644\n--- a/pseudo-merge.h\n+++ b/pseudo-merge.h\n@@ -2,5 +2,99 @@\n #define PSEUDO_MERGE_H\n \n #include \"git-compat-util.h\"\n+#include \"strmap.h\"\n+#include \"khash.h\"\n+#include \"ewah/ewok.h\"\n+\n+struct commit;\n+struct string_list;\n+struct bitmap_index;\n+struct bitmap_writer;\n+\n+/*\n+ * A pseudo-merge group tracks the set of non-bitmapped reference tips\n+ * that match the given pattern.\n+ *\n+ * Within those matches, they are further segmented by separating\n+ * consecutive capture groups with '-' dash character capture groups\n+ * with '-' dash characters.\n+ *\n+ * Those groups are then ordered by committer date and partitioned\n+ * into individual pseudo-merge(s) according to the decay, max_merges,\n+ * sample_rate, and threshold parameters.\n+ */\n+struct pseudo_merge_group {\n+\tregex_t *pattern;\n+\n+\t/* capture group(s) -> struct pseudo_merge_matches */\n+\tstruct strmap matches;\n+\n+\t/*\n+\t * The individual pseudo-merge(s) that are generated from the\n+\t * above array of matches, partitioned according to the below\n+\t * parameters.\n+\t */\n+\tstruct commit **merges;\n+\tsize_t merges_nr;\n+\tsize_t merges_alloc;\n+\n+\t/*\n+\t * Pseudo-merge grouping parameters. See git-config(1) for\n+\t * more information.\n+\t */\n+\tdouble decay;\n+\tint max_merges;\n+\tdouble sample_rate;\n+\tint stable_size;\n+\ttimestamp_t threshold;\n+\ttimestamp_t stable_threshold;\n+};\n+\n+struct pseudo_merge_matches {\n+\tstruct commit **stable;\n+\tstruct commit **unstable;\n+\tsize_t stable_nr, stable_alloc;\n+\tsize_t unstable_nr, unstable_alloc;\n+};\n+\n+/*\n+ * Read the repository's configuration:\n+ *\n+ * - bitmapPseudoMerge..pattern\n+ * - bitmapPseudoMerge..decay\n+ * - bitmapPseudoMerge..sampleRate\n+ * - bitmapPseudoMerge..threshold\n+ * - bitmapPseudoMerge..maxMerges\n+ * - bitmapPseudoMerge..stableThreshold\n+ * - bitmapPseudoMerge..stableSize\n+ *\n+ * and populates the given `list` with pseudo-merge groups. String\n+ * entry keys are the pseudo-merge group names, and the values are\n+ * pointers to the pseudo_merge_group structure itself.\n+ */\n+void load_pseudo_merges_from_config(struct string_list *list);\n+\n+/*\n+ * A pseudo-merge commit index (pseudo_merge_commit_idx) maps a\n+ * particular (non-pseudo-merge) commit to the list of pseudo-merge(s)\n+ * it appears in.\n+ */\n+struct pseudo_merge_commit_idx {\n+\tuint32_t *pseudo_merge;\n+\tsize_t nr, alloc;\n+};\n+\n+/*\n+ * Selects pseudo-merges from a list of commits, populating the given\n+ * string_list of pseudo-merge groups.\n+ *\n+ * Populates the pseudo_merge_commits map with a commit_idx\n+ * corresponding to each commit in the list. Counts the total number\n+ * of pseudo-merges generated.\n+ *\n+ * Optionally shows a progress meter.\n+ */\n+void select_pseudo_merges(struct bitmap_writer *writer,\n+\t\t\t struct commit **commits, size_t commits_nr);\n \n #endif\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495468","messageId":"90df19e43f54ac3950e05c5a007e47650e75fe1a.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 12/24] pack-bitmap-write.c: write pseudo-merge table","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:46Z","receivedAt":"2024-05-23T21:26:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the pack-bitmap writer machinery understands how to select and\nstore pseudo-merge commits, teach it how to write the new optional\npseudo-merge .bitmap extension.\n\nNo readers yet exist for this new extension to the .bitmap format. The\nfollowing commits will take any preparatory step(s) necessary before\nthen implementing the routines necessary to read this new table.\n\nIn the meantime, the new `write_pseudo_merges()` function implements\nwriting this new format as described by a previous commit in\nDocumentation/technical/bitmap-format.txt.\n\nWriting this table is fairly straightforward and consists of a few\nsub-components:\n\n - a pair of bitmaps for each pseudo-merge (one for the pseudo-merge\n \"parents\", and another for the objects reachable from those parents)\n\n - for each commit, the offset of either (a) the pseudo-merge it\n belongs to, or (b) an extended lookup table if it belongs to >1\n pseudo-merge groups\n\n - if there are any commits belonging to >1 pseudo-merge group, the\n extended lookup tables (which each consist of the number of\n pseudo-merge groups a commit appears in, and then that many 4-byte\n unsigned )\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 131 ++++++++++++++++++++++++++++++++++++++++++++\n pack-bitmap.h | 1 +\n 2 files changed, 132 insertions(+)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex d5884ea5e9c..47250398aa2 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -18,6 +18,7 @@\n #include \"tree.h\"\n #include \"tree-walk.h\"\n #include \"pseudo-merge.h\"\n+#include \"oid-array.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -771,6 +772,130 @@ static void write_selected_commits_v1(struct bitmap_writer *writer,\n \t}\n }\n \n+static void write_pseudo_merges(struct bitmap_writer *writer,\n+\t\t\t\tstruct hashfile *f)\n+{\n+\tstruct oid_array commits = OID_ARRAY_INIT;\n+\tstruct bitmap **commits_bitmap = NULL;\n+\toff_t *pseudo_merge_ofs = NULL;\n+\toff_t start, table_start, next_ext;\n+\n+\tuint32_t base = bitmap_writer_nr_selected_commits(writer);\n+\tsize_t i, j = 0;\n+\n+\tCALLOC_ARRAY(commits_bitmap, writer->pseudo_merges_nr);\n+\tCALLOC_ARRAY(pseudo_merge_ofs, writer->pseudo_merges_nr);\n+\n+\tfor (i = 0; i < writer->pseudo_merges_nr; i++) {\n+\t\tstruct bitmapped_commit *merge = &writer->selected[base + i];\n+\t\tstruct commit_list *p;\n+\n+\t\tif (!merge->pseudo_merge)\n+\t\t\tBUG(\"found non-pseudo merge commit at %\"PRIuMAX, (uintmax_t)i);\n+\n+\t\tcommits_bitmap[i] = bitmap_new();\n+\n+\t\tfor (p = merge->commit->parents; p; p = p->next)\n+\t\t\tbitmap_set(commits_bitmap[i],\n+\t\t\t\t find_object_pos(writer, &p->item->object.oid,\n+\t\t\t\t\t\t NULL));\n+\t}\n+\n+\tstart = hashfile_total(f);\n+\n+\tfor (i = 0; i < writer->pseudo_merges_nr; i++) {\n+\t\tstruct ewah_bitmap *commits_ewah = bitmap_to_ewah(commits_bitmap[i]);\n+\n+\t\tpseudo_merge_ofs[i] = hashfile_total(f);\n+\n+\t\tdump_bitmap(f, commits_ewah);\n+\t\tdump_bitmap(f, writer->selected[base+i].write_as);\n+\n+\t\tewah_free(commits_ewah);\n+\t}\n+\n+\tnext_ext = st_add(hashfile_total(f),\n+\t\t\t st_mult(kh_size(writer->pseudo_merge_commits),\n+\t\t\t\t sizeof(uint64_t)));\n+\n+\ttable_start = hashfile_total(f);\n+\n+\tcommits.alloc = kh_size(writer->pseudo_merge_commits);\n+\tCALLOC_ARRAY(commits.oid, commits.alloc);\n+\n+\tfor (i = kh_begin(writer->pseudo_merge_commits); i != kh_end(writer->pseudo_merge_commits); i++) {\n+\t\tif (!kh_exist(writer->pseudo_merge_commits, i))\n+\t\t\tcontinue;\n+\t\toid_array_append(&commits, &kh_key(writer->pseudo_merge_commits, i));\n+\t}\n+\n+\toid_array_sort(&commits);\n+\n+\t/* write lookup table (non-extended) */\n+\tfor (i = 0; i < commits.nr; i++) {\n+\t\tint hash_pos;\n+\t\tstruct pseudo_merge_commit_idx *c;\n+\n+\t\thash_pos = kh_get_oid_map(writer->pseudo_merge_commits,\n+\t\t\t\t\t commits.oid[i]);\n+\t\tif (hash_pos == kh_end(writer->pseudo_merge_commits))\n+\t\t\tBUG(\"could not find pseudo-merge commit %s\",\n+\t\t\t oid_to_hex(&commits.oid[i]));\n+\n+\t\tc = kh_value(writer->pseudo_merge_commits, hash_pos);\n+\n+\t\thashwrite_be32(f, find_object_pos(writer, &commits.oid[i],\n+\t\t\t\t\t\t NULL));\n+\t\tif (c->nr == 1)\n+\t\t\thashwrite_be64(f, pseudo_merge_ofs[c->pseudo_merge[0]]);\n+\t\telse if (c->nr > 1) {\n+\t\t\tif (next_ext & ((uint64_t)1<<63))\n+\t\t\t\tdie(_(\"too many pseudo-merges\"));\n+\t\t\thashwrite_be64(f, next_ext | ((uint64_t)1<<63));\n+\t\t\tnext_ext = st_add3(next_ext,\n+\t\t\t\t\t sizeof(uint32_t),\n+\t\t\t\t\t st_mult(c->nr, sizeof(uint64_t)));\n+\t\t} else\n+\t\t\tBUG(\"expected commit '%s' to have at least one \"\n+\t\t\t \"pseudo-merge\", oid_to_hex(&commits.oid[i]));\n+\t}\n+\n+\t/* write lookup table (extended) */\n+\tfor (i = 0; i < commits.nr; i++) {\n+\t\tint hash_pos;\n+\t\tstruct pseudo_merge_commit_idx *c;\n+\n+\t\thash_pos = kh_get_oid_map(writer->pseudo_merge_commits,\n+\t\t\t\t\t commits.oid[i]);\n+\t\tif (hash_pos == kh_end(writer->pseudo_merge_commits))\n+\t\t\tBUG(\"could not find pseudo-merge commit %s\",\n+\t\t\t oid_to_hex(&commits.oid[i]));\n+\n+\t\tc = kh_value(writer->pseudo_merge_commits, hash_pos);\n+\t\tif (c->nr == 1)\n+\t\t\tcontinue;\n+\n+\t\thashwrite_be32(f, c->nr);\n+\t\tfor (j = 0; j < c->nr; j++)\n+\t\t\thashwrite_be64(f, pseudo_merge_ofs[c->pseudo_merge[j]]);\n+\t}\n+\n+\t/* write positions for all pseudo merges */\n+\tfor (i = 0; i < writer->pseudo_merges_nr; i++)\n+\t\thashwrite_be64(f, pseudo_merge_ofs[i]);\n+\n+\thashwrite_be32(f, writer->pseudo_merges_nr);\n+\thashwrite_be32(f, kh_size(writer->pseudo_merge_commits));\n+\thashwrite_be64(f, table_start - start);\n+\thashwrite_be64(f, hashfile_total(f) - start + sizeof(uint64_t));\n+\n+\tfor (i = 0; i < writer->pseudo_merges_nr; i++)\n+\t\tbitmap_free(commits_bitmap[i]);\n+\n+\tfree(pseudo_merge_ofs);\n+\tfree(commits_bitmap);\n+}\n+\n static int table_cmp(const void *_va, const void *_vb, void *_data)\n {\n \tstruct bitmap_writer *writer = _data;\n@@ -878,6 +1003,9 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \n \tint fd = odb_mkstemp(&tmp_file, \"pack/tmp_bitmap_XXXXXX\");\n \n+\tif (writer->pseudo_merges_nr)\n+\t\toptions |= BITMAP_OPT_PSEUDO_MERGES;\n+\n \tf = hashfd(fd, tmp_file.buf);\n \n \tmemcpy(header.magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE));\n@@ -907,6 +1035,9 @@ void bitmap_writer_finish(struct bitmap_writer *writer,\n \n \twrite_selected_commits_v1(writer, f, offsets);\n \n+\tif (options & BITMAP_OPT_PSEUDO_MERGES)\n+\t\twrite_pseudo_merges(writer, f);\n+\n \tif (options & BITMAP_OPT_LOOKUP_TABLE)\n \t\twrite_lookup_table(writer, f, offsets);\n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 1e730ea1e54..db9ae554fa8 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -37,6 +37,7 @@ enum pack_bitmap_opts {\n \tBITMAP_OPT_FULL_DAG = 0x1,\n \tBITMAP_OPT_HASH_CACHE = 0x4,\n \tBITMAP_OPT_LOOKUP_TABLE = 0x10,\n+\tBITMAP_OPT_PSEUDO_MERGES = 0x20,\n };\n \n enum pack_bitmap_flags {\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495469","messageId":"c653a10f8e40ff33c8110c07572c3468791581d9.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 13/24] pack-bitmap: extract `read_bitmap()` function","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:49Z","receivedAt":"2024-05-23T21:26:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The pack-bitmap machinery uses the `read_bitmap_1()` function to read a\nbitmap from within the mmap'd region corresponding to the .bitmap file.\nAs as side-effect of calling this function, `read_bitmap_1()` increments\nthe `index->map_pos` variable to reflect the number of bytes read.\n\nExtract the core of this routine to a separate function (that operates\nover a `const unsigned char *`, a `size_t` and a `size_t *` pointer)\ninstead of a `struct bitmap_index *` pointer.\n\nThis function (called `read_bitmap()`) is part of the pack-bitmap.h API\nso that it can be used within the upcoming portion of the implementation\nin pseduo-merge.ch.\n\nRewrite the existing function, `read_bitmap_1()`, in terms of its more\ngeneric counterpart.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 24 +++++++++++++++---------\n pack-bitmap.h | 2 ++\n 2 files changed, 17 insertions(+), 9 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 35c5ef9d3cd..3519edb896b 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -129,17 +129,13 @@ static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st)\n \treturn composed;\n }\n \n-/*\n- * Read a bitmap from the current read position on the mmaped\n- * index, and increase the read position accordingly\n- */\n-static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n+struct ewah_bitmap *read_bitmap(const unsigned char *map,\n+\t\t\t\tsize_t map_size, size_t *map_pos)\n {\n \tstruct ewah_bitmap *b = ewah_pool_new();\n \n-\tssize_t bitmap_size = ewah_read_mmap(b,\n-\t\tindex->map + index->map_pos,\n-\t\tindex->map_size - index->map_pos);\n+\tssize_t bitmap_size = ewah_read_mmap(b, map + *map_pos,\n+\t\t\t\t\t map_size - *map_pos);\n \n \tif (bitmap_size < 0) {\n \t\terror(_(\"failed to load bitmap index (corrupted?)\"));\n@@ -147,10 +143,20 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n \t\treturn NULL;\n \t}\n \n-\tindex->map_pos += bitmap_size;\n+\t*map_pos += bitmap_size;\n+\n \treturn b;\n }\n \n+/*\n+ * Read a bitmap from the current read position on the mmaped\n+ * index, and increase the read position accordingly\n+ */\n+static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index)\n+{\n+\treturn read_bitmap(index->map, index->map_size, &index->map_pos);\n+}\n+\n static uint32_t bitmap_num_objects(struct bitmap_index *index)\n {\n \tif (index->midx)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex db9ae554fa8..21aabf805ea 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -160,4 +160,6 @@ int bitmap_is_preferred_refname(struct repository *r, const char *refname);\n \n int verify_bitmap_files(struct repository *r);\n \n+struct ewah_bitmap *read_bitmap(const unsigned char *map,\n+\t\t\t\tsize_t map_size, size_t *map_pos);\n #endif\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495470","messageId":"435ac048003d55355c8b58812d836933dcffae2b.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 14/24] pseudo-merge: scaffolding for reads","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:52Z","receivedAt":"2024-05-23T21:26:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement scaffolding within the new pseudo-merge compilation unit\nnecessary to use the pseudo-merge API from within the pack-bitmap.c\nmachinery.\n\nThe core of this scaffolding is two-fold:\n\n - The `pseudo_merge` structure itself, which represents an individual\n pseudo-merge bitmap. It has fields for both bitmaps, as well as\n metadata about its position within the memory-mapped region, and\n a few extra bits indicating whether or not it is satisfied, and\n which bitmaps(s, if any) have been read, since they are initialized\n lazily.\n\n - The `pseudo_merge_map` structure, which holds an array of\n pseudo_merges, as well as a pointer to the memory-mapped region\n containing the pseudo-merge serialization from within a .bitmap\n file.\n\nNote that the `bitmap_index` structure is defined statically within the\npack-bitmap.o compilation unit, so we can't take in a `struct\nbitmap_index *`. Instead, wrap the primary components necessary to read\nthe pseudo-merges in this new structure to avoid exposing the\nimplementation details of the `bitmap_index` structure.\n\nSigned-off-by: Taylor Blau \n---\n pseudo-merge.c | 10 ++++++++\n pseudo-merge.h | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 75 insertions(+)\n\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex 0f6854c753f..f0080d53c03 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -454,3 +454,13 @@ void select_pseudo_merges(struct bitmap_writer *writer,\n \n \tstop_progress(&progress);\n }\n+\n+void free_pseudo_merge_map(struct pseudo_merge_map *pm)\n+{\n+\tuint32_t i;\n+\tfor (i = 0; i < pm->nr; i++) {\n+\t\tewah_pool_free(pm->v[i].commits);\n+\t\tewah_pool_free(pm->v[i].bitmap);\n+\t}\n+\tfree(pm->v);\n+}\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nindex f809cf42aeb..e9216baace8 100644\n--- a/pseudo-merge.h\n+++ b/pseudo-merge.h\n@@ -97,4 +97,69 @@ struct pseudo_merge_commit_idx {\n void select_pseudo_merges(struct bitmap_writer *writer,\n \t\t\t struct commit **commits, size_t commits_nr);\n \n+/*\n+ * Represents a serialized view of a file containing pseudo-merge(s)\n+ * (see Documentation/technical/bitmap-format.txt for a specification\n+ * of the format).\n+ */\n+struct pseudo_merge_map {\n+\t/*\n+\t * An array of pseudo-merge(s), lazily loaded from the .bitmap\n+\t * file.\n+\t */\n+\tstruct pseudo_merge *v;\n+\tsize_t nr;\n+\tsize_t commits_nr;\n+\n+\t/*\n+\t * Pointers into a memory-mapped view of the .bitmap file:\n+\t *\n+\t * - map: the beginning of the .bitmap file\n+\t * - commits: the beginning of the pseudo-merge commit index\n+\t * - map_size: the size of the .bitmap file\n+\t */\n+\tconst unsigned char *map;\n+\tconst unsigned char *commits;\n+\n+\tsize_t map_size;\n+};\n+\n+/*\n+ * An individual pseudo-merge, storing a pair of lazily-loaded\n+ * bitmaps:\n+ *\n+ * - commits: the set of commit(s) that are part of the pseudo-merge\n+ * - bitmap: the set of object(s) reachable from the above set of\n+ * commits.\n+ *\n+ * The `at` and `bitmap_at` fields are used to store the locations of\n+ * each of the above bitmaps in the .bitmap file.\n+ */\n+struct pseudo_merge {\n+\tstruct ewah_bitmap *commits;\n+\tstruct ewah_bitmap *bitmap;\n+\n+\toff_t at;\n+\toff_t bitmap_at;\n+\n+\t/*\n+\t * `satisfied` indicates whether the given pseudo-merge has been\n+\t * used.\n+\t *\n+\t * `loaded_commits` and `loaded_bitmap` indicate whether the\n+\t * respective bitmaps have been loaded and read from the\n+\t * .bitmap file.\n+\t */\n+\tunsigned satisfied : 1,\n+\t\t loaded_commits : 1,\n+\t\t loaded_bitmap : 1;\n+};\n+\n+/*\n+ * Frees the given pseudo-merge map, releasing any memory held by (a)\n+ * parsed EWAH bitmaps, or (b) the array of pseudo-merges itself. Does\n+ * not free the memory-mapped view of the .bitmap file.\n+ */\n+void free_pseudo_merge_map(struct pseudo_merge_map *pm);\n+\n #endif\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495471","messageId":"fa7a948964c9d1a787ef1d9ba0d5202c099a36be.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 15/24] pack-bitmap.c: read pseudo-merge extension","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:55Z","receivedAt":"2024-05-23T21:26:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the scaffolding for reading the pseudo-merge extension has been\nlaid, teach the pack-bitmap machinery to read the pseudo-merge extension\nwhen present.\n\nNote that pseudo-merges themselves are not yet used during traversal,\nthis step will be taken by a future commit.\n\nIn the meantime, read the table and initialize the pseudo_merge_map\nstructure introduced by a previous commit. When the pseudo-merge\nextension is present, `load_bitmap_header()` performs basic sanity\nchecks to make sure that the table is well-formed.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 39 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 39 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 3519edb896b..fc9c3e2fc43 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -20,6 +20,7 @@\n #include \"list-objects-filter-options.h\"\n #include \"midx.h\"\n #include \"config.h\"\n+#include \"pseudo-merge.h\"\n \n /*\n * An entry on the bitmap index, representing the bitmap for a given\n@@ -86,6 +87,9 @@ struct bitmap_index {\n \t */\n \tunsigned char *table_lookup;\n \n+\t/* This contains the pseudo-merge cache within 'map' (if found). */\n+\tstruct pseudo_merge_map pseudo_merges;\n+\n \t/*\n \t * Extended index.\n \t *\n@@ -205,6 +209,41 @@ static int load_bitmap_header(struct bitmap_index *index)\n \t\t\t\tindex->table_lookup = (void *)(index_end - table_size);\n \t\t\tindex_end -= table_size;\n \t\t}\n+\n+\t\tif (flags & BITMAP_OPT_PSEUDO_MERGES) {\n+\t\t\tunsigned char *pseudo_merge_ofs;\n+\t\t\tsize_t table_size;\n+\t\t\tuint32_t i;\n+\n+\t\t\tif (sizeof(table_size) > index_end - index->map - header_size)\n+\t\t\t\treturn error(_(\"corrupted bitmap index file (too short to fit pseudo-merge table header)\"));\n+\n+\t\t\ttable_size = get_be64(index_end - 8);\n+\t\t\tif (table_size > index_end - index->map - header_size)\n+\t\t\t\treturn error(_(\"corrupted bitmap index file (too short to fit pseudo-merge table)\"));\n+\n+\t\t\tif (git_env_bool(\"GIT_TEST_USE_PSEUDO_MERGES\", 1)) {\n+\t\t\t\tconst unsigned char *ext = (index_end - table_size);\n+\n+\t\t\t\tindex->pseudo_merges.map = index->map;\n+\t\t\t\tindex->pseudo_merges.map_size = index->map_size;\n+\t\t\t\tindex->pseudo_merges.commits = ext + get_be64(index_end - 16);\n+\t\t\t\tindex->pseudo_merges.commits_nr = get_be32(index_end - 20);\n+\t\t\t\tindex->pseudo_merges.nr = get_be32(index_end - 24);\n+\n+\t\t\t\tCALLOC_ARRAY(index->pseudo_merges.v,\n+\t\t\t\t\t index->pseudo_merges.nr);\n+\n+\t\t\t\tpseudo_merge_ofs = index_end - 24 -\n+\t\t\t\t\t(index->pseudo_merges.nr * sizeof(uint64_t));\n+\t\t\t\tfor (i = 0; i < index->pseudo_merges.nr; i++) {\n+\t\t\t\t\tindex->pseudo_merges.v[i].at = get_be64(pseudo_merge_ofs);\n+\t\t\t\t\tpseudo_merge_ofs += sizeof(uint64_t);\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tindex_end -= table_size;\n+\t\t}\n \t}\n \n \tindex->entry_count = ntohl(header->entry_count);\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495472","messageId":"3a72e66cb6932fa73f8261043b6dcc262ebb6e31.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 16/24] pseudo-merge: implement support for reading pseudo-merge commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:26:58Z","receivedAt":"2024-05-23T21:27:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement the basic API for reading pseudo-merge bitmaps, which consists\nof four basic functions:\n\n - pseudo_merge_bitmap()\n - use_pseudo_merge()\n - apply_pseudo_merges_for_commit()\n - cascade_pseudo_merges()\n\nThese functions are all documented in pseudo-merge.h, but their rough\ndescriptions are as follows:\n\n - pseudo_merge_bitmap() reads and inflates the objects EWAH bitmap for\n a given pseudo-merge\n\n - use_pseudo_merge() does the same as pseudo_merge_bitmap(), but on\n the commits EWAH bitmap, not the objects bitmap\n\n - apply_pseudo_merges_for_commit() applies all satisfied pseudo-merge\n commits for a given result set, and cascades any yet-unsatisfied\n pseudo-merges if any were applied in the previous step\n\n - cascade_pseudo_merges() applies all pseudo-merges which are\n satisfied but have not been previously applied, repeating this\n process until no more pseudo-merges can be applied\n\nThe core of the API is the latter two functions, which are responsible\nfor applying pseudo-merges during the object traversal implemented in\nthe pack-bitmap machinery.\n\nThe other two functions (pseudo_merge_bitmap(), and use_pseudo_merge())\nare low-level ways to interact with the pseudo-merge machinery, which\nwill be useful in future commits.\n\nSigned-off-by: Taylor Blau \n---\n pseudo-merge.c | 235 +++++++++++++++++++++++++++++++++++++++++++++++++\n pseudo-merge.h | 44 +++++++++\n 2 files changed, 279 insertions(+)\n\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex f0080d53c03..7d131011497 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -10,6 +10,7 @@\n #include \"commit.h\"\n #include \"alloc.h\"\n #include \"progress.h\"\n+#include \"hex.h\"\n \n #define DEFAULT_PSEUDO_MERGE_DECAY 1.0\n #define DEFAULT_PSEUDO_MERGE_MAX_MERGES 64\n@@ -464,3 +465,237 @@ void free_pseudo_merge_map(struct pseudo_merge_map *pm)\n \t}\n \tfree(pm->v);\n }\n+\n+struct pseudo_merge_commit_ext {\n+\tuint32_t nr;\n+\tconst unsigned char *ptr;\n+};\n+\n+static int pseudo_merge_ext_at(const struct pseudo_merge_map *pm,\n+\t\t\t struct pseudo_merge_commit_ext *ext, size_t at)\n+{\n+\tif (at >= pm->map_size)\n+\t\treturn error(_(\"extended pseudo-merge read out-of-bounds \"\n+\t\t\t \"(%\"PRIuMAX\" >= %\"PRIuMAX\")\"),\n+\t\t\t (uintmax_t)at, (uintmax_t)pm->map_size);\n+\tif (at + 4 >= pm->map_size)\n+\t\treturn error(_(\"extended pseudo-merge entry is too short \"\n+\t\t\t \"(%\"PRIuMAX\" >= %\"PRIuMAX\")\"),\n+\t\t\t (uintmax_t)(at + 4), (uintmax_t)pm->map_size);\n+\n+\text->nr = get_be32(pm->map + at);\n+\text->ptr = pm->map + at + sizeof(uint32_t);\n+\n+\treturn 0;\n+}\n+\n+struct ewah_bitmap *pseudo_merge_bitmap(const struct pseudo_merge_map *pm,\n+\t\t\t\t\tstruct pseudo_merge *merge)\n+{\n+\tif (!merge->loaded_commits)\n+\t\tBUG(\"cannot use unloaded pseudo-merge bitmap\");\n+\n+\tif (!merge->loaded_bitmap) {\n+\t\tsize_t at = merge->bitmap_at;\n+\n+\t\tmerge->bitmap = read_bitmap(pm->map, pm->map_size, &at);\n+\t\tmerge->loaded_bitmap = 1;\n+\t}\n+\n+\treturn merge->bitmap;\n+}\n+\n+struct pseudo_merge *use_pseudo_merge(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct pseudo_merge *merge)\n+{\n+\tif (!merge->loaded_commits) {\n+\t\tsize_t pos = merge->at;\n+\n+\t\tmerge->commits = read_bitmap(pm->map, pm->map_size, &pos);\n+\t\tmerge->bitmap_at = pos;\n+\t\tmerge->loaded_commits = 1;\n+\t}\n+\treturn merge;\n+}\n+\n+static struct pseudo_merge *pseudo_merge_at(const struct pseudo_merge_map *pm,\n+\t\t\t\t\t struct object_id *oid,\n+\t\t\t\t\t size_t want)\n+{\n+\tsize_t lo = 0;\n+\tsize_t hi = pm->nr;\n+\n+\twhile (lo < hi) {\n+\t\tsize_t mi = lo + (hi - lo) / 2;\n+\t\tsize_t got = pm->v[mi].at;\n+\n+\t\tif (got == want)\n+\t\t\treturn use_pseudo_merge(pm, &pm->v[mi]);\n+\t\telse if (got < want)\n+\t\t\thi = mi;\n+\t\telse\n+\t\t\tlo = mi + 1;\n+\t}\n+\n+\twarning(_(\"could not find pseudo-merge for commit %s at offset %\"PRIuMAX),\n+\t\toid_to_hex(oid), (uintmax_t)want);\n+\n+\treturn NULL;\n+}\n+\n+struct pseudo_merge_commit {\n+\tuint32_t commit_pos;\n+\tuint64_t pseudo_merge_ofs;\n+};\n+\n+#define PSEUDO_MERGE_COMMIT_RAWSZ (sizeof(uint32_t)+sizeof(uint64_t))\n+\n+static void read_pseudo_merge_commit_at(struct pseudo_merge_commit *merge,\n+\t\t\t\t\tconst unsigned char *at)\n+{\n+\tmerge->commit_pos = get_be32(at);\n+\tmerge->pseudo_merge_ofs = get_be64(at + sizeof(uint32_t));\n+}\n+\n+static int nth_pseudo_merge_ext(const struct pseudo_merge_map *pm,\n+\t\t\t\tstruct pseudo_merge_commit_ext *ext,\n+\t\t\t\tstruct pseudo_merge_commit *merge,\n+\t\t\t\tuint32_t n)\n+{\n+\tsize_t ofs;\n+\n+\tif (n >= ext->nr)\n+\t\treturn error(_(\"extended pseudo-merge lookup out-of-bounds \"\n+\t\t\t \"(%\"PRIu32\" >= %\"PRIu32\")\"), n, ext->nr);\n+\n+\tofs = get_be64(ext->ptr + st_mult(n, sizeof(uint64_t)));\n+\tif (ofs >= pm->map_size)\n+\t\treturn error(_(\"out-of-bounds read: (%\"PRIuMAX\" >= %\"PRIuMAX\")\"),\n+\t\t\t (uintmax_t)ofs, (uintmax_t)pm->map_size);\n+\n+\tread_pseudo_merge_commit_at(merge, pm->map + ofs);\n+\n+\treturn 0;\n+}\n+\n+static unsigned apply_pseudo_merge(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct pseudo_merge *merge,\n+\t\t\t\t struct bitmap *result,\n+\t\t\t\t struct bitmap *roots)\n+{\n+\tif (merge->satisfied)\n+\t\treturn 0;\n+\n+\tif (!ewah_bitmap_is_subset(merge->commits, roots ? roots : result))\n+\t\treturn 0;\n+\n+\tbitmap_or_ewah(result, pseudo_merge_bitmap(pm, merge));\n+\tif (roots)\n+\t\tbitmap_or_ewah(roots, pseudo_merge_bitmap(pm, merge));\n+\tmerge->satisfied = 1;\n+\n+\treturn 1;\n+}\n+\n+static int pseudo_merge_commit_cmp(const void *va, const void *vb)\n+{\n+\tstruct pseudo_merge_commit merge;\n+\tuint32_t key = *(uint32_t*)va;\n+\n+\tread_pseudo_merge_commit_at(&merge, vb);\n+\n+\tif (key < merge.commit_pos)\n+\t\treturn -1;\n+\tif (key > merge.commit_pos)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static struct pseudo_merge_commit *find_pseudo_merge(const struct pseudo_merge_map *pm,\n+\t\t\t\t\t\t uint32_t pos)\n+{\n+\tif (!pm->commits_nr)\n+\t\treturn NULL;\n+\n+\treturn bsearch(&pos, pm->commits, pm->commits_nr,\n+\t\t PSEUDO_MERGE_COMMIT_RAWSZ, pseudo_merge_commit_cmp);\n+}\n+\n+int apply_pseudo_merges_for_commit(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct bitmap *result,\n+\t\t\t\t struct commit *commit, uint32_t commit_pos)\n+{\n+\tstruct pseudo_merge *merge;\n+\tstruct pseudo_merge_commit *merge_commit;\n+\tint ret = 0;\n+\n+\tmerge_commit = find_pseudo_merge(pm, commit_pos);\n+\tif (!merge_commit)\n+\t\treturn 0;\n+\n+\tif (merge_commit->pseudo_merge_ofs & ((uint64_t)1<<63)) {\n+\t\tstruct pseudo_merge_commit_ext ext = { 0 };\n+\t\toff_t ofs = merge_commit->pseudo_merge_ofs & ~((uint64_t)1<<63);\n+\t\tuint32_t i;\n+\n+\t\tif (pseudo_merge_ext_at(pm, &ext, ofs) < -1) {\n+\t\t\twarning(_(\"could not read extended pseudo-merge table \"\n+\t\t\t\t \"for commit %s\"),\n+\t\t\t\toid_to_hex(&commit->object.oid));\n+\t\t\treturn ret;\n+\t\t}\n+\n+\t\tfor (i = 0; i < ext.nr; i++) {\n+\t\t\tif (nth_pseudo_merge_ext(pm, &ext, merge_commit, i) < 0)\n+\t\t\t\treturn ret;\n+\n+\t\t\tmerge = pseudo_merge_at(pm, &commit->object.oid,\n+\t\t\t\t\t\tmerge_commit->pseudo_merge_ofs);\n+\n+\t\t\tif (!merge)\n+\t\t\t\treturn ret;\n+\n+\t\t\tif (apply_pseudo_merge(pm, merge, result, NULL))\n+\t\t\t\tret++;\n+\t\t}\n+\t} else {\n+\t\tmerge = pseudo_merge_at(pm, &commit->object.oid,\n+\t\t\t\t\tmerge_commit->pseudo_merge_ofs);\n+\n+\t\tif (!merge)\n+\t\t\treturn ret;\n+\n+\t\tif (apply_pseudo_merge(pm, merge, result, NULL))\n+\t\t\tret++;\n+\t}\n+\n+\tif (ret)\n+\t\tcascade_pseudo_merges(pm, result, NULL);\n+\n+\treturn ret;\n+}\n+\n+int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n+\t\t\t struct bitmap *result,\n+\t\t\t struct bitmap *roots)\n+{\n+\tunsigned any_satisfied;\n+\tint ret = 0;\n+\n+\tdo {\n+\t\tstruct pseudo_merge *merge;\n+\t\tuint32_t i;\n+\n+\t\tany_satisfied = 0;\n+\n+\t\tfor (i = 0; i < pm->nr; i++) {\n+\t\t\tmerge = use_pseudo_merge(pm, &pm->v[i]);\n+\t\t\tif (apply_pseudo_merge(pm, merge, result, roots)) {\n+\t\t\t\tany_satisfied |= 1;\n+\t\t\t\tret++;\n+\t\t\t}\n+\t\t}\n+\t} while (any_satisfied);\n+\n+\treturn ret;\n+}\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nindex e9216baace8..755edc054ae 100644\n--- a/pseudo-merge.h\n+++ b/pseudo-merge.h\n@@ -162,4 +162,48 @@ struct pseudo_merge {\n */\n void free_pseudo_merge_map(struct pseudo_merge_map *pm);\n \n+/*\n+ * Loads the bitmap corresponding to the given pseudo-merge from the\n+ * map, if it has not already been loaded.\n+ */\n+struct ewah_bitmap *pseudo_merge_bitmap(const struct pseudo_merge_map *pm,\n+\t\t\t\t\tstruct pseudo_merge *merge);\n+\n+/*\n+ * Loads the pseudo-merge and its commits bitmap from the given\n+ * pseudo-merge map, if it has not already been loaded.\n+ */\n+struct pseudo_merge *use_pseudo_merge(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct pseudo_merge *merge);\n+\n+/*\n+ * Applies pseudo-merge(s) containing the given commit to the bitmap\n+ * \"result\".\n+ *\n+ * If any pseudo-merge(s) were satisfied, returns the number\n+ * satisfied, otherwise returns 0. If any were satisfied, the\n+ * remaining unsatisfied pseudo-merges are cascaded (see below).\n+ */\n+int apply_pseudo_merges_for_commit(const struct pseudo_merge_map *pm,\n+\t\t\t\t struct bitmap *result,\n+\t\t\t\t struct commit *commit, uint32_t commit_pos);\n+\n+/*\n+ * Applies pseudo-merge(s) which are satisfied according to the\n+ * current bitmap in result (or roots, see below). If any\n+ * pseudo-merges were satisfied, repeat the process over unsatisfied\n+ * pseudo-merge commits until no more pseudo-merges are satisfied.\n+ *\n+ * Result is the bitmap to which the pseudo-merge(s) are applied.\n+ * Roots (if given) is a bitmap of the traversal tip(s) for either\n+ * side of a reachability traversal.\n+ *\n+ * Roots may given instead of a populated results bitmap at the\n+ * beginning of a traversal on either side where the reachability\n+ * closure over tips is not yet known.\n+ */\n+int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n+\t\t\t struct bitmap *result,\n+\t\t\t struct bitmap *roots);\n+\n #endif\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495473","messageId":"42a836fda8add1e0f26394660b88f0e2e3e635e7.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 17/24] ewah: implement `ewah_bitmap_popcount()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:27:02Z","receivedAt":"2024-05-23T21:27:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Some of the pseudo-merge test helpers (which will be introduced in the\nfollowing commit) will want to indicate the total number of commits in\nor objects reachable from a pseudo-merge.\n\nImplement a popcount() function that operates on EWAH bitmaps to quickly\ndetermine how many bits are set in each of the respective bitmaps.\n\nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 14 ++++++++++++++\n ewah/ewok.h | 1 +\n 2 files changed, 15 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex d352fec54ce..dc2ca190f12 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -212,6 +212,20 @@ size_t bitmap_popcount(struct bitmap *self)\n \treturn count;\n }\n \n+size_t ewah_bitmap_popcount(struct ewah_bitmap *self)\n+{\n+\tstruct ewah_iterator it;\n+\teword_t word;\n+\tsize_t count = 0;\n+\n+\tewah_iterator_init(&it, self);\n+\n+\twhile (ewah_iterator_next(&word, &it))\n+\t\tcount += ewah_bit_popcount64(word);\n+\n+\treturn count;\n+}\n+\n int bitmap_is_empty(struct bitmap *self)\n {\n \tsize_t i;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 2b6c4ac499c..7074a6347b7 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -195,6 +195,7 @@ void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other);\n void bitmap_or(struct bitmap *self, const struct bitmap *other);\n \n size_t bitmap_popcount(struct bitmap *self);\n+size_t ewah_bitmap_popcount(struct ewah_bitmap *self);\n int bitmap_is_empty(struct bitmap *self);\n \n #endif\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495474","messageId":"06ba1a5bbfd206cd47c9d8d474371042dc226031.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 18/24] pack-bitmap: implement test helpers for pseudo-merge","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:27:05Z","receivedAt":"2024-05-23T21:27:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement three new sub-commands for the \"bitmap\" test-helper:\n\n - t/helper test-tool bitmap dump-pseudo-merges\n - t/helper test-tool bitmap dump-pseudo-merge-commits \n - t/helper test-tool bitmap dump-pseudo-merge-objects \n\nThese three helpers dump the list of pseudo merges, the \"parents\" of the\nnth pseudo-merges, and the set of objects reachable from those parents,\nrespectively.\n\nThese helpers will be useful in subsequent patches when we add test\ncoverage for pseudo-merge bitmaps.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 126 +++++++++++++++++++++++++++++++++++++++++\n pack-bitmap.h | 3 +\n t/helper/test-bitmap.c | 34 ++++++++---\n 3 files changed, 156 insertions(+), 7 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex fc9c3e2fc43..c13074673af 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -2443,6 +2443,132 @@ int test_bitmap_hashes(struct repository *r)\n \treturn 0;\n }\n \n+static void bit_pos_to_object_id(struct bitmap_index *bitmap_git,\n+\t\t\t\t uint32_t bit_pos,\n+\t\t\t\t struct object_id *oid)\n+{\n+\tuint32_t index_pos;\n+\n+\tif (bitmap_is_midx(bitmap_git))\n+\t\tindex_pos = pack_pos_to_midx(bitmap_git->midx, bit_pos);\n+\telse\n+\t\tindex_pos = pack_pos_to_index(bitmap_git->pack, bit_pos);\n+\n+\tnth_bitmap_object_oid(bitmap_git, oid, index_pos);\n+}\n+\n+int test_bitmap_pseudo_merges(struct repository *r)\n+{\n+\tstruct bitmap_index *bitmap_git;\n+\tuint32_t i;\n+\n+\tbitmap_git = prepare_bitmap_git(r);\n+\tif (!bitmap_git || !bitmap_git->pseudo_merges.nr)\n+\t\tgoto cleanup;\n+\n+\tfor (i = 0; i < bitmap_git->pseudo_merges.nr; i++) {\n+\t\tstruct pseudo_merge *merge;\n+\t\tstruct ewah_bitmap *commits_bitmap, *merge_bitmap;\n+\n+\t\tmerge = use_pseudo_merge(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t &bitmap_git->pseudo_merges.v[i]);\n+\t\tcommits_bitmap = merge->commits;\n+\t\tmerge_bitmap = pseudo_merge_bitmap(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t\t merge);\n+\n+\t\tprintf(\"at=%\"PRIuMAX\", commits=%\"PRIuMAX\", objects=%\"PRIuMAX\"\\n\",\n+\t\t (uintmax_t)merge->at,\n+\t\t (uintmax_t)ewah_bitmap_popcount(commits_bitmap),\n+\t\t (uintmax_t)ewah_bitmap_popcount(merge_bitmap));\n+\t}\n+\n+cleanup:\n+\tfree_bitmap_index(bitmap_git);\n+\treturn 0;\n+}\n+\n+static void dump_ewah_object_ids(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct ewah_bitmap *bitmap)\n+\n+{\n+\tstruct ewah_iterator it;\n+\teword_t word;\n+\tuint32_t pos = 0;\n+\n+\tewah_iterator_init(&it, bitmap);\n+\n+\twhile (ewah_iterator_next(&word, &it)) {\n+\t\tstruct object_id oid;\n+\t\tuint32_t offset;\n+\n+\t\tfor (offset = 0; offset < BITS_IN_EWORD; offset++) {\n+\t\t\tif (!(word >> offset))\n+\t\t\t\tbreak;\n+\n+\t\t\toffset += ewah_bit_ctz64(word >> offset);\n+\n+\t\t\tbit_pos_to_object_id(bitmap_git, pos + offset, &oid);\n+\t\t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n+\t\t}\n+\t\tpos += BITS_IN_EWORD;\n+\t}\n+}\n+\n+int test_bitmap_pseudo_merge_commits(struct repository *r, uint32_t n)\n+{\n+\tstruct bitmap_index *bitmap_git;\n+\tstruct pseudo_merge *merge;\n+\tint ret = 0;\n+\n+\tbitmap_git = prepare_bitmap_git(r);\n+\tif (!bitmap_git || !bitmap_git->pseudo_merges.nr)\n+\t\tgoto cleanup;\n+\n+\tif (n >= bitmap_git->pseudo_merges.nr) {\n+\t\tret = error(_(\"pseudo-merge index out of range \"\n+\t\t\t \"(%\"PRIu32\" >= %\"PRIuMAX\")\"),\n+\t\t\t n, (uintmax_t)bitmap_git->pseudo_merges.nr);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tmerge = use_pseudo_merge(&bitmap_git->pseudo_merges,\n+\t\t\t\t &bitmap_git->pseudo_merges.v[n]);\n+\tdump_ewah_object_ids(bitmap_git, merge->commits);\n+\n+cleanup:\n+\tfree_bitmap_index(bitmap_git);\n+\treturn ret;\n+}\n+\n+int test_bitmap_pseudo_merge_objects(struct repository *r, uint32_t n)\n+{\n+\tstruct bitmap_index *bitmap_git;\n+\tstruct pseudo_merge *merge;\n+\tint ret = 0;\n+\n+\tbitmap_git = prepare_bitmap_git(r);\n+\tif (!bitmap_git || !bitmap_git->pseudo_merges.nr)\n+\t\tgoto cleanup;\n+\n+\tif (n >= bitmap_git->pseudo_merges.nr) {\n+\t\tret = error(_(\"pseudo-merge index out of range \"\n+\t\t\t \"(%\"PRIu32\" >= %\"PRIuMAX\")\"),\n+\t\t\t n, (uintmax_t)bitmap_git->pseudo_merges.nr);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tmerge = use_pseudo_merge(&bitmap_git->pseudo_merges,\n+\t\t\t\t &bitmap_git->pseudo_merges.v[n]);\n+\n+\tdump_ewah_object_ids(bitmap_git,\n+\t\t\t pseudo_merge_bitmap(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t\t merge));\n+\n+cleanup:\n+\tfree_bitmap_index(bitmap_git);\n+\treturn ret;\n+}\n+\n int rebuild_bitmap(const uint32_t *reposition,\n \t\t struct ewah_bitmap *source,\n \t\t struct bitmap *dest)\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 21aabf805ea..4466b5ad0fb 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -73,6 +73,9 @@ void traverse_bitmap_commit_list(struct bitmap_index *,\n void test_bitmap_walk(struct rev_info *revs);\n int test_bitmap_commits(struct repository *r);\n int test_bitmap_hashes(struct repository *r);\n+int test_bitmap_pseudo_merges(struct repository *r);\n+int test_bitmap_pseudo_merge_commits(struct repository *r, uint32_t n);\n+int test_bitmap_pseudo_merge_objects(struct repository *r, uint32_t n);\n \n #define GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL \\\n \t\"GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL\"\ndiff --git a/t/helper/test-bitmap.c b/t/helper/test-bitmap.c\nindex af43ee1cb5e..6af2b42678f 100644\n--- a/t/helper/test-bitmap.c\n+++ b/t/helper/test-bitmap.c\n@@ -13,21 +13,41 @@ static int bitmap_dump_hashes(void)\n \treturn test_bitmap_hashes(the_repository);\n }\n \n+static int bitmap_dump_pseudo_merges(void)\n+{\n+\treturn test_bitmap_pseudo_merges(the_repository);\n+}\n+\n+static int bitmap_dump_pseudo_merge_commits(uint32_t n)\n+{\n+\treturn test_bitmap_pseudo_merge_commits(the_repository, n);\n+}\n+\n+static int bitmap_dump_pseudo_merge_objects(uint32_t n)\n+{\n+\treturn test_bitmap_pseudo_merge_objects(the_repository, n);\n+}\n+\n int cmd__bitmap(int argc, const char **argv)\n {\n \tsetup_git_directory();\n \n-\tif (argc != 2)\n-\t\tgoto usage;\n-\n-\tif (!strcmp(argv[1], \"list-commits\"))\n+\tif (argc == 2 && !strcmp(argv[1], \"list-commits\"))\n \t\treturn bitmap_list_commits();\n-\tif (!strcmp(argv[1], \"dump-hashes\"))\n+\tif (argc == 2 && !strcmp(argv[1], \"dump-hashes\"))\n \t\treturn bitmap_dump_hashes();\n+\tif (argc == 2 && !strcmp(argv[1], \"dump-pseudo-merges\"))\n+\t\treturn bitmap_dump_pseudo_merges();\n+\tif (argc == 3 && !strcmp(argv[1], \"dump-pseudo-merge-commits\"))\n+\t\treturn bitmap_dump_pseudo_merge_commits(atoi(argv[2]));\n+\tif (argc == 3 && !strcmp(argv[1], \"dump-pseudo-merge-objects\"))\n+\t\treturn bitmap_dump_pseudo_merge_objects(atoi(argv[2]));\n \n-usage:\n \tusage(\"\\ttest-tool bitmap list-commits\\n\"\n-\t \"\\ttest-tool bitmap dump-hashes\");\n+\t \"\\ttest-tool bitmap dump-hashes\\n\"\n+\t \"\\ttest-tool bitmap dump-pseudo-merges\\n\"\n+\t \"\\ttest-tool bitmap dump-pseudo-merge-commits \\n\"\n+\t \"\\ttest-tool bitmap dump-pseudo-merge-objects \");\n \n \treturn -1;\n }\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495475","messageId":"936f6d1b7e392367bb87a755d014633d7171f0ab.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 19/24] t/test-lib-functions.sh: support `--notick` in `test_commit_bulk()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:27:08Z","receivedAt":"2024-05-23T21:27:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"One of the tests we'll want to add for pseudo-merge bitmaps needs to be\nable to generate a large number of commits at a specific date.\n\nSupport the `--notick` option (with identical semantics to the\n`--notick` option for `test_commit()`) within `test_commit_bulk` as a\nprerequisite for that. Callers can then set the various _DATE variables\nthemselves.\n\nSigned-off-by: Taylor Blau \n---\n t/test-lib-functions.sh | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 862d80c9748..427b375b392 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -458,6 +458,7 @@ test_commit_bulk () {\n \tindir=.\n \tref=HEAD\n \tn=1\n+\tnotick=\n \tmessage='commit %s'\n \tfilename='%s.t'\n \tcontents='content %s'\n@@ -488,6 +489,9 @@ test_commit_bulk () {\n \t\t\tfilename=\"${1#--*=}-%s.t\"\n \t\t\tcontents=\"${1#--*=} %s\"\n \t\t\t;;\n+\t\t--notick)\n+\t\t\tnotick=yes\n+\t\t\t;;\n \t\t-*)\n \t\t\tBUG \"invalid test_commit_bulk option: $1\"\n \t\t\t;;\n@@ -507,7 +511,10 @@ test_commit_bulk () {\n \n \twhile test \"$total\" -gt 0\n \tdo\n-\t\ttest_tick &&\n+\t\tif test -z \"$notick\"\n+\t\tthen\n+\t\t\ttest_tick\n+\t\tfi &&\n \t\techo \"commit $ref\"\n \t\tprintf 'author %s <%s> %s\\n' \\\n \t\t\t\"$GIT_AUTHOR_NAME\" \\\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495476","messageId":"cad38608aaec65e962d74c5398e46d7c47359f01.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 20/24] pack-bitmap.c: use pseudo-merges during traversal","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:27:11Z","receivedAt":"2024-05-23T21:27:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that all of the groundwork has been laid to support reading and\nusing pseudo-merges, make use of that work in this commit by teaching\nthe pack-bitmap machinery to use pseudo-merge(s) when available during\ntraversal.\n\nThe basic operation is as follows:\n\n - When enumerating objects on either side of a reachability query,\n first see if any subset of the roots satisfies some pseudo-merge\n bitmap. If it does, apply that pseudo-merge bitmap.\n\n - If any pseudo-merge bitmap(s) were applied in the previous step, OR\n them into the result[^1]. Then repeat the process over all\n pseudo-merge bitmaps (we'll refer to this as \"cascading\"\n pseudo-merges). Once this is done, OR in the resulting bitmap.\n\n - If there is no fill-in traversal to be done, return the bitmap for\n that side of the reachability query. If there is fill-in traversal,\n then for each commit we encounter via show_commit(), check to see if\n any unsatisfied pseudo-merges containing that commit as one of its\n parents has been made satisfied by the presence of that commit.\n\n If so, OR in the object set from that pseudo-merge bitmap, and then\n cascade. If not, continue traversal.\n\nA similar implementation is present in the boundary-based bitmap\ntraversal routines.\n\n[^1]: Importantly, we cannot OR in the entire set of roots along with\n the objects reachable from whatever pseudo-merge bitmaps were\n satisfied. This may leave some dangling bits corresponding to any\n unsatisfied root(s) getting OR'd into the resulting bitmap, tricking\n other parts of the traversal into thinking we already have a\n reachability closure over those commit(s) when we do not.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 112 ++++++++++-\n t/t5333-pseudo-merge-bitmaps.sh | 328 ++++++++++++++++++++++++++++++++\n 2 files changed, 439 insertions(+), 1 deletion(-)\n create mode 100755 t/t5333-pseudo-merge-bitmaps.sh\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex c13074673af..e61058dada6 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -114,6 +114,9 @@ struct bitmap_index {\n \tunsigned int version;\n };\n \n+static int pseudo_merges_satisfied_nr;\n+static int pseudo_merges_cascades_nr;\n+\n static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st)\n {\n \tstruct ewah_bitmap *parent;\n@@ -1006,6 +1009,22 @@ static void show_commit(struct commit *commit UNUSED,\n {\n }\n \n+static unsigned apply_pseudo_merges_for_commit_1(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct bitmap *result,\n+\t\t\t\t\t\t struct commit *commit,\n+\t\t\t\t\t\t uint32_t commit_pos)\n+{\n+\tint ret;\n+\n+\tret = apply_pseudo_merges_for_commit(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t result, commit, commit_pos);\n+\n+\tif (ret)\n+\t\tpseudo_merges_satisfied_nr += ret;\n+\n+\treturn ret;\n+}\n+\n static int add_to_include_set(struct bitmap_index *bitmap_git,\n \t\t\t struct include_data *data,\n \t\t\t struct commit *commit,\n@@ -1026,6 +1045,10 @@ static int add_to_include_set(struct bitmap_index *bitmap_git,\n \t}\n \n \tbitmap_set(data->base, bitmap_pos);\n+\tif (apply_pseudo_merges_for_commit_1(bitmap_git, data->base, commit,\n+\t\t\t\t\t bitmap_pos))\n+\t\treturn 0;\n+\n \treturn 1;\n }\n \n@@ -1151,6 +1174,20 @@ static void show_boundary_object(struct object *object UNUSED,\n \tBUG(\"should not be called\");\n }\n \n+static unsigned cascade_pseudo_merges_1(struct bitmap_index *bitmap_git,\n+\t\t\t\t\tstruct bitmap *result,\n+\t\t\t\t\tstruct bitmap *roots)\n+{\n+\tint ret = cascade_pseudo_merges(&bitmap_git->pseudo_merges,\n+\t\t\t\t\tresult, roots);\n+\tif (ret) {\n+\t\tpseudo_merges_cascades_nr++;\n+\t\tpseudo_merges_satisfied_nr += ret;\n+\t}\n+\n+\treturn ret;\n+}\n+\n static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \t\t\t\t\t struct rev_info *revs,\n \t\t\t\t\t struct object_list *roots)\n@@ -1160,6 +1197,7 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \tunsigned int i;\n \tunsigned int tmp_blobs, tmp_trees, tmp_tags;\n \tint any_missing = 0;\n+\tint existing_bitmaps = 0;\n \n \tcb.bitmap_git = bitmap_git;\n \tcb.base = bitmap_new();\n@@ -1167,6 +1205,25 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \n \trevs->ignore_missing_links = 1;\n \n+\tif (bitmap_git->pseudo_merges.nr) {\n+\t\tstruct bitmap *roots_bitmap = bitmap_new();\n+\t\tstruct object_list *objects = NULL;\n+\n+\t\tfor (objects = roots; objects; objects = objects->next) {\n+\t\t\tstruct object *object = objects->item;\n+\t\t\tint pos;\n+\n+\t\t\tpos = bitmap_position(bitmap_git, &object->oid);\n+\t\t\tif (pos < 0)\n+\t\t\t\tcontinue;\n+\n+\t\t\tbitmap_set(roots_bitmap, pos);\n+\t\t}\n+\n+\t\tif (!cascade_pseudo_merges_1(bitmap_git, cb.base, roots_bitmap))\n+\t\t\tbitmap_free(roots_bitmap);\n+\t}\n+\n \t/*\n \t * OR in any existing reachability bitmaps among `roots` into\n \t * `cb.base`.\n@@ -1178,8 +1235,10 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \t\t\tcontinue;\n \n \t\tif (add_commit_to_bitmap(bitmap_git, &cb.base,\n-\t\t\t\t\t (struct commit *)object))\n+\t\t\t\t\t (struct commit *)object)) {\n+\t\t\texisting_bitmaps = 1;\n \t\t\tcontinue;\n+\t\t}\n \n \t\tany_missing = 1;\n \t}\n@@ -1187,6 +1246,9 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \tif (!any_missing)\n \t\tgoto cleanup;\n \n+\tif (existing_bitmaps)\n+\t\tcascade_pseudo_merges_1(bitmap_git, cb.base, NULL);\n+\n \ttmp_blobs = revs->blob_objects;\n \ttmp_trees = revs->tree_objects;\n \ttmp_tags = revs->blob_objects;\n@@ -1242,6 +1304,13 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \treturn cb.base;\n }\n \n+static void unsatisfy_all_pseudo_merges(struct bitmap_index *bitmap_git)\n+{\n+\tuint32_t i;\n+\tfor (i = 0; i < bitmap_git->pseudo_merges.nr; i++)\n+\t\tbitmap_git->pseudo_merges.v[i].satisfied = 0;\n+}\n+\n static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\t\t\t struct rev_info *revs,\n \t\t\t\t struct object_list *roots,\n@@ -1249,9 +1318,32 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n {\n \tstruct bitmap *base = NULL;\n \tint needs_walk = 0;\n+\tunsigned existing_bitmaps = 0;\n \n \tstruct object_list *not_mapped = NULL;\n \n+\tunsatisfy_all_pseudo_merges(bitmap_git);\n+\n+\tif (bitmap_git->pseudo_merges.nr) {\n+\t\tstruct bitmap *roots_bitmap = bitmap_new();\n+\t\tstruct object_list *objects = NULL;\n+\n+\t\tfor (objects = roots; objects; objects = objects->next) {\n+\t\t\tstruct object *object = objects->item;\n+\t\t\tint pos;\n+\n+\t\t\tpos = bitmap_position(bitmap_git, &object->oid);\n+\t\t\tif (pos < 0)\n+\t\t\t\tcontinue;\n+\n+\t\t\tbitmap_set(roots_bitmap, pos);\n+\t\t}\n+\n+\t\tbase = bitmap_new();\n+\t\tif (!cascade_pseudo_merges_1(bitmap_git, base, roots_bitmap))\n+\t\t\tbitmap_free(roots_bitmap);\n+\t}\n+\n \t/*\n \t * Go through all the roots for the walk. The ones that have bitmaps\n \t * on the bitmap index will be `or`ed together to form an initial\n@@ -1262,11 +1354,21 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t */\n \twhile (roots) {\n \t\tstruct object *object = roots->item;\n+\n \t\troots = roots->next;\n \n+\t\tif (base) {\n+\t\t\tint pos = bitmap_position(bitmap_git, &object->oid);\n+\t\t\tif (pos > 0 && bitmap_get(base, pos)) {\n+\t\t\t\tobject->flags |= SEEN;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n \t\tif (object->type == OBJ_COMMIT &&\n \t\t add_commit_to_bitmap(bitmap_git, &base, (struct commit *)object)) {\n \t\t\tobject->flags |= SEEN;\n+\t\t\texisting_bitmaps = 1;\n \t\t\tcontinue;\n \t\t}\n \n@@ -1282,6 +1384,9 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \n \troots = not_mapped;\n \n+\tif (existing_bitmaps)\n+\t\tcascade_pseudo_merges_1(bitmap_git, base, NULL);\n+\n \t/*\n \t * Let's iterate through all the roots that don't have bitmaps to\n \t * check if we can determine them to be reachable from the existing\n@@ -1866,6 +1971,11 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tobject_list_free(&wants);\n \tobject_list_free(&haves);\n \n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"pseudo_merges_satisfied\",\n+\t\t\t pseudo_merges_satisfied_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"pseudo_merges_cascades\",\n+\t\t\t pseudo_merges_cascades_nr);\n+\n \treturn bitmap_git;\n \n cleanup:\ndiff --git a/t/t5333-pseudo-merge-bitmaps.sh b/t/t5333-pseudo-merge-bitmaps.sh\nnew file mode 100755\nindex 00000000000..4c9aebcffdc\n--- /dev/null\n+++ b/t/t5333-pseudo-merge-bitmaps.sh\n@@ -0,0 +1,328 @@\n+#!/bin/sh\n+\n+test_description='pseudo-merge bitmaps'\n+\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0\n+\n+. ./test-lib.sh\n+\n+test_pseudo_merges () {\n+\ttest-tool bitmap dump-pseudo-merges\n+}\n+\n+test_pseudo_merge_commits () {\n+\ttest-tool bitmap dump-pseudo-merge-commits \"$1\"\n+}\n+\n+test_pseudo_merges_satisfied () {\n+\ttest_trace2_data bitmap pseudo_merges_satisfied \"$1\"\n+}\n+\n+test_pseudo_merges_cascades () {\n+\ttest_trace2_data bitmap pseudo_merges_cascades \"$1\"\n+}\n+\n+tag_everything () {\n+\tgit rev-list --all --no-object-names >in &&\n+\tperl -lne '\n+\t\tprint \"create refs/tags/\" . $. . \" \" . $1 if /([0-9a-f]+)/\n+\t' expect &&\n+\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\n+\ttest_pseudo_merges_satisfied 0 merges &&\n+\ttest_must_be_empty merges &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'pseudo-merges accurately represent their objects' '\n+\ttest_config bitmapPseudoMerge.test.pattern \"refs/tags/\" &&\n+\ttest_config bitmapPseudoMerge.test.maxMerges 8 &&\n+\ttest_config bitmapPseudoMerge.test.stableThreshold never &&\n+\n+\tgit repack -adb &&\n+\n+\ttest_pseudo_merges >merges &&\n+\ttest_line_count = 8 merges &&\n+\n+\tfor i in $(test_seq 0 $(($(wc -l commits &&\n+\n+\t\tgit rev-list --objects --no-object-names --stdin expect.raw &&\n+\t\ttest-tool bitmap dump-pseudo-merge-objects $i >actual.raw &&\n+\n+\t\tsort -u expect &&\n+\t\tsort -u actual &&\n+\n+\t\ttest_cmp expect actual || return 1\n+\tdone\n+'\n+\n+test_expect_success 'bitmap traversal with pseudo-merges' '\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\tgit rev-list --count --all --objects >expect &&\n+\n+\ttest_pseudo_merges_satisfied 8 trace2.txt &&\n+\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\tgit rev-list --count --all --objects >expect &&\n+\n+\ttest_pseudo_merges_satisfied 8 merges &&\n+\t\ttest_line_count = 1 merges &&\n+\t\ttest_pseudo_merge_commits 0 >commits &&\n+\n+\t\ttest-tool bitmap list-commits >bitmaps &&\n+\t\tbitmaps_nr=\"$(wc -l expect &&\n+\n+\t\ttest $(cat expect) -eq $(wc -l merges &&\n+\t\ttest_line_count = 1 merges &&\n+\n+\t\ttest_pseudo_merge_commits 0 >oids &&\n+\t\tgit cat-file --batch commits &&\n+\n+\t\ttest $(wc -l in &&\n+\t\tgit update-ref --stdin merges &&\n+\t\tmerges_nr=\"$(wc -l oids &&\n+\t\t\tgit cat-file --batch commits &&\n+\n+\t\t\texpect=\"$(grep -c \"^committer.*$old +0000$\" commits)\" &&\n+\t\t\tactual=\"$(wc -l oids &&\n+\t\tgit cat-file --batch commits &&\n+\t\ttest $(wc -l err &&\n+\n+\tcat >expect <<-EOF &&\n+\tfatal: pseudo-merge group ${SQ}test${SQ} has unstable threshold before stable one\n+\tEOF\n+\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'pseudo-merge pattern with capture groups' '\n+\tgit init pseudo-merge-captures &&\n+\t(\n+\t\tcd pseudo-merge-captures &&\n+\n+\t\ttest_commit_bulk 128 &&\n+\t\ttag_everything &&\n+\n+\t\tfor r in $(test_seq 8)\n+\t\tdo\n+\t\t\ttest_commit_bulk 16 &&\n+\n+\t\t\tgit rev-list HEAD~16.. >in &&\n+\n+\t\t\tperl -lne \"print \\\"create refs/remotes/$r/tags/\\$. \\$_\\\"\" refs &&\n+\n+\t\ttest_pseudo_merges >merges &&\n+\t\tfor m in $(test_seq 0 $(($(wc -l oids &&\n+\t\t\tgrep -f oids refs |\n+\t\t\tperl -lne \"print \\$1 if /refs\\/remotes\\/([0-9]+)/\" |\n+\t\t\tsort -u || return 1\n+\t\tdone >remotes &&\n+\n+\t\ttest $(wc -l merges &&\n+\t\ttest_line_count = 2 merges &&\n+\n+\t\ttest_pseudo_merge_commits 0 >commits-0.raw &&\n+\t\ttest_pseudo_merge_commits 1 >commits-1.raw &&\n+\n+\t\tsort commits-0.raw >commits-0 &&\n+\t\tsort commits-1.raw >commits-1 &&\n+\n+\t\tcomm -12 commits-0 commits-1 >overlap &&\n+\n+\t\ttest_line_count -gt 0 overlap\n+\t)\n+'\n+\n+test_expect_success 'pseudo-merge overlap traversal' '\n+\t(\n+\t\tcd pseudo-merge-overlap &&\n+\n+\t\t: >trace2.txt &&\n+\t\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\t\tgit rev-list --count --all --objects >expect &&\n+\n+\t\ttest_pseudo_merges_satisfied 2 trace2.txt &&\n+\t\tGIT_TRACE2_EVENT=$PWD/trace2.txt \\\n+\t\t\tgit rev-list --count --all --objects --use-bitmap-index >actual &&\n+\t\tgit rev-list --count --all --objects >expect &&\n+\n+\t\ttest_pseudo_merges_satisfied 2 \n---\n pack-bitmap.c | 26 +++++++++++++++++++++++++-\n 1 file changed, 25 insertions(+), 1 deletion(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex e61058dada6..1966b3b95f1 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -116,6 +116,10 @@ struct bitmap_index {\n \n static int pseudo_merges_satisfied_nr;\n static int pseudo_merges_cascades_nr;\n+static int existing_bitmaps_hits_nr;\n+static int existing_bitmaps_misses_nr;\n+static int roots_with_bitmaps_nr;\n+static int roots_without_bitmaps_nr;\n \n static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st)\n {\n@@ -1040,10 +1044,14 @@ static int add_to_include_set(struct bitmap_index *bitmap_git,\n \n \tpartial = bitmap_for_commit(bitmap_git, commit);\n \tif (partial) {\n+\t\texisting_bitmaps_hits_nr++;\n+\n \t\tbitmap_or_ewah(data->base, partial);\n \t\treturn 0;\n \t}\n \n+\texisting_bitmaps_misses_nr++;\n+\n \tbitmap_set(data->base, bitmap_pos);\n \tif (apply_pseudo_merges_for_commit_1(bitmap_git, data->base, commit,\n \t\t\t\t\t bitmap_pos))\n@@ -1099,8 +1107,12 @@ static int add_commit_to_bitmap(struct bitmap_index *bitmap_git,\n {\n \tstruct ewah_bitmap *or_with = bitmap_for_commit(bitmap_git, commit);\n \n-\tif (!or_with)\n+\tif (!or_with) {\n+\t\texisting_bitmaps_misses_nr++;\n \t\treturn 0;\n+\t}\n+\n+\texisting_bitmaps_hits_nr++;\n \n \tif (!*base)\n \t\t*base = ewah_to_bitmap(or_with);\n@@ -1407,8 +1419,12 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\t\tobject->flags &= ~UNINTERESTING;\n \t\t\tadd_pending_object(revs, object, \"\");\n \t\t\tneeds_walk = 1;\n+\n+\t\t\troots_without_bitmaps_nr++;\n \t\t} else {\n \t\t\tobject->flags |= SEEN;\n+\n+\t\t\troots_with_bitmaps_nr++;\n \t\t}\n \t}\n \n@@ -1975,6 +1991,14 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\t pseudo_merges_satisfied_nr);\n \ttrace2_data_intmax(\"bitmap\", the_repository, \"pseudo_merges_cascades\",\n \t\t\t pseudo_merges_cascades_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"bitmap/hits\",\n+\t\t\t existing_bitmaps_hits_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"bitmap/misses\",\n+\t\t\t existing_bitmaps_misses_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"bitmap/roots_with_bitmap\",\n+\t\t\t roots_with_bitmaps_nr);\n+\ttrace2_data_intmax(\"bitmap\", the_repository, \"bitmap/roots_without_bitmap\",\n+\t\t\t roots_without_bitmaps_nr);\n \n \treturn bitmap_git;\n \n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495478","messageId":"625596a143229e6a9a8a36d2b0eaa480ac204809.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 22/24] ewah: `bitmap_equals_ewah()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:27:18Z","receivedAt":"2024-05-23T21:27:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare to reuse existing pseudo-merge bitmaps by implementing a\n`bitmap_equals_ewah()` helper.\n\nThis helper will be used to see if a raw bitmap (containing the set of\nparents for some pseudo-merge) is equal to any existing pseudo-merge's\ncommits bitmap (which are stored as EWAH-compressed bitmaps on disk).\n\nSigned-off-by: Taylor Blau \n---\n ewah/bitmap.c | 19 +++++++++++++++++++\n ewah/ewok.h | 1 +\n 2 files changed, 20 insertions(+)\n\ndiff --git a/ewah/bitmap.c b/ewah/bitmap.c\nindex dc2ca190f12..55928dada86 100644\n--- a/ewah/bitmap.c\n+++ b/ewah/bitmap.c\n@@ -261,6 +261,25 @@ int bitmap_equals(struct bitmap *self, struct bitmap *other)\n \treturn 1;\n }\n \n+int bitmap_equals_ewah(struct bitmap *self, struct ewah_bitmap *other)\n+{\n+\tstruct ewah_iterator it;\n+\teword_t word;\n+\tsize_t i = 0;\n+\n+\tewah_iterator_init(&it, other);\n+\n+\twhile (ewah_iterator_next(&word, &it))\n+\t\tif (word != (i < self->word_alloc ? self->words[i++] : 0))\n+\t\t\treturn 0;\n+\n+\tfor (; i < self->word_alloc; i++)\n+\t\tif (self->words[i])\n+\t\t\treturn 0;\n+\n+\treturn 1;\n+}\n+\n int bitmap_is_subset(struct bitmap *self, struct bitmap *other)\n {\n \tsize_t common_size, i;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 7074a6347b7..5e357e24933 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -179,6 +179,7 @@ void bitmap_unset(struct bitmap *self, size_t pos);\n int bitmap_get(struct bitmap *self, size_t pos);\n void bitmap_free(struct bitmap *self);\n int bitmap_equals(struct bitmap *self, struct bitmap *other);\n+int bitmap_equals_ewah(struct bitmap *self, struct ewah_bitmap *other);\n \n /*\n * Both `bitmap_is_subset()` and `ewah_bitmap_is_subset()` return 1 if the set\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495479","messageId":"fdd506d4544c4ce4cf96eab688da0def311addd3.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 23/24] pseudo-merge: implement support for finding existing merges","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:27:21Z","receivedAt":"2024-05-23T21:27:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This patch implements support for reusing existing pseudo-merge commits\nwhen writing bitmaps when there is an existing pseudo-merge bitmap which\nhas exactly the same set of parents as one that we are about to write.\n\nNote that unstable pseudo-merges are likely to change between\nconsecutive repacks, and so are generally poor candidates for reuse.\nHowever, stable pseudo-merges (see the configuration option\n'bitmapPseudoMerge..stableThreshold') are by definition unlikely\nto change between runs (as they represent long-running branches).\n\nBecause there is no index from a *set* of pseudo-merge parents to a\nmatching pseudo-merge bitmap, we have to construct the bitmap\ncorresponding to the set of parents for each pending pseudo-merge commit\nand see if a matching bitmap exists.\n\nThis is technically quadratic in the number of pseudo-merges, but is OK\nin practice for a couple of reasons:\n\n - non-matching pseudo-merge bitmaps are rejected quickly as soon as\n they differ in a single bit\n\n - already-matched pseudo-merge bitmaps are discarded from subsequent\n rounds of search\n\n - the number of pseudo-merges is generally small, even for large\n repositories\n\nIn order to do this, implement (a) a function that finds a matching\npseudo-merge given some uncompressed bitset describing its parents, (b)\na function that computes the bitset of parents for a given pseudo-merge\ncommit, and (c) call that function before computing the set of reachable\nobjects for some pending pseudo-merge.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap-write.c | 15 +++++++-\n pack-bitmap.c | 32 ++++++++++++++++\n pack-bitmap.h | 2 +\n pseudo-merge.c | 55 ++++++++++++++++++++++++++++\n pseudo-merge.h | 7 ++++\n t/t5333-pseudo-merge-bitmaps.sh | 65 +++++++++++++++++++++++++++++++++\n 6 files changed, 174 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 47250398aa2..6e8060f8a0b 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -19,6 +19,10 @@\n #include \"tree-walk.h\"\n #include \"pseudo-merge.h\"\n #include \"oid-array.h\"\n+#include \"config.h\"\n+#include \"alloc.h\"\n+#include \"refs.h\"\n+#include \"strmap.h\"\n \n struct bitmapped_commit {\n \tstruct commit *commit;\n@@ -465,6 +469,7 @@ static int fill_bitmap_tree(struct bitmap_writer *writer,\n }\n \n static int reused_bitmaps_nr;\n+static int reused_pseudo_merge_bitmaps_nr;\n \n static int fill_bitmap_commit(struct bitmap_writer *writer,\n \t\t\t struct bb_commit *ent,\n@@ -490,7 +495,7 @@ static int fill_bitmap_commit(struct bitmap_writer *writer,\n \t\t\tstruct bitmap *remapped = bitmap_new();\n \n \t\t\tif (commit->object.flags & BITMAP_PSEUDO_MERGE)\n-\t\t\t\told = NULL;\n+\t\t\t\told = pseudo_merge_bitmap_for_commit(old_bitmap, c);\n \t\t\telse\n \t\t\t\told = bitmap_for_commit(old_bitmap, c);\n \t\t\t/*\n@@ -501,7 +506,10 @@ static int fill_bitmap_commit(struct bitmap_writer *writer,\n \t\t\tif (old && !rebuild_bitmap(mapping, old, remapped)) {\n \t\t\t\tbitmap_or(ent->bitmap, remapped);\n \t\t\t\tbitmap_free(remapped);\n-\t\t\t\treused_bitmaps_nr++;\n+\t\t\t\tif (commit->object.flags & BITMAP_PSEUDO_MERGE)\n+\t\t\t\t\treused_pseudo_merge_bitmaps_nr++;\n+\t\t\t\telse\n+\t\t\t\t\treused_bitmaps_nr++;\n \t\t\t\tcontinue;\n \t\t\t}\n \t\t\tbitmap_free(remapped);\n@@ -631,6 +639,9 @@ int bitmap_writer_build(struct bitmap_writer *writer,\n \t\t\t the_repository);\n \ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n \t\t\t \"building_bitmaps_reused\", reused_bitmaps_nr);\n+\ttrace2_data_intmax(\"pack-bitmap-write\", the_repository,\n+\t\t\t \"building_bitmaps_pseudo_merge_reused\",\n+\t\t\t reused_pseudo_merge_bitmaps_nr);\n \n \tstop_progress(&writer->progress);\n \ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 1966b3b95f1..70230e26479 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1316,6 +1316,37 @@ static struct bitmap *find_boundary_objects(struct bitmap_index *bitmap_git,\n \treturn cb.base;\n }\n \n+struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct commit *commit)\n+{\n+\tstruct commit_list *p;\n+\tstruct bitmap *parents;\n+\tstruct pseudo_merge *match = NULL;\n+\n+\tif (!bitmap_git->pseudo_merges.nr)\n+\t\treturn NULL;\n+\n+\tparents = bitmap_new();\n+\n+\tfor (p = commit->parents; p; p = p->next) {\n+\t\tint pos = bitmap_position(bitmap_git, &p->item->object.oid);\n+\t\tif (pos < 0 || pos >= bitmap_num_objects(bitmap_git))\n+\t\t\tgoto done;\n+\n+\t\tbitmap_set(parents, pos);\n+\t}\n+\n+\tmatch = pseudo_merge_for_parents(&bitmap_git->pseudo_merges,\n+\t\t\t\t\t\tparents);\n+\n+done:\n+\tbitmap_free(parents);\n+\tif (match)\n+\t\treturn pseudo_merge_bitmap(&bitmap_git->pseudo_merges, match);\n+\n+\treturn NULL;\n+}\n+\n static void unsatisfy_all_pseudo_merges(struct bitmap_index *bitmap_git)\n {\n \tuint32_t i;\n@@ -2809,6 +2840,7 @@ void free_bitmap_index(struct bitmap_index *b)\n \t\t */\n \t\tclose_midx_revindex(b->midx);\n \t}\n+\tfree_pseudo_merge_map(&b->pseudo_merges);\n \tfree(b);\n }\n \ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 4466b5ad0fb..1171e6d9893 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -142,6 +142,8 @@ int rebuild_bitmap(const uint32_t *reposition,\n \t\t struct bitmap *dest);\n struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\t\t\t struct commit *commit);\n+struct ewah_bitmap *pseudo_merge_bitmap_for_commit(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t\t struct commit *commit);\n void bitmap_writer_select_commits(struct bitmap_writer *writer,\n \t\t\t\t struct commit **indexed_commits,\n \t\t\t\t unsigned int indexed_commits_nr);\ndiff --git a/pseudo-merge.c b/pseudo-merge.c\nindex 7d131011497..a117520996c 100644\n--- a/pseudo-merge.c\n+++ b/pseudo-merge.c\n@@ -699,3 +699,58 @@ int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n \n \treturn ret;\n }\n+\n+struct pseudo_merge *pseudo_merge_for_parents(const struct pseudo_merge_map *pm,\n+\t\t\t\t\t struct bitmap *parents)\n+{\n+\tstruct pseudo_merge *match = NULL;\n+\tsize_t i;\n+\n+\tif (!pm->nr)\n+\t\treturn NULL;\n+\n+\t/*\n+\t * NOTE: this loop is quadratic in the worst-case (where no\n+\t * matching pseudo-merge bitmaps are found), but in practice\n+\t * this is OK for a few reasons:\n+\t *\n+\t * - Rejecting pseudo-merge bitmaps that do not match the\n+\t * given commit is done quickly (i.e. `bitmap_equals_ewah()`\n+\t * returns early when we know the two bitmaps aren't equal.\n+\t *\n+\t * - Already matched pseudo-merge bitmaps (which we track with\n+\t * the `->satisfied` bit here) are skipped as potential\n+\t * candidates.\n+\t *\n+\t * - The number of pseudo-merges should be small (in the\n+\t * hundreds for most repositories).\n+\t *\n+\t * If in the future this semi-quadratic behavior does become a\n+\t * problem, another approach would be to keep track of which\n+\t * pseudo-merges are still \"viable\" after enumerating the\n+\t * pseudo-merge commit's parents:\n+\t *\n+\t * - A pseudo-merge bitmap becomes non-viable when the bit(s)\n+\t * corresponding to one or more parent(s) of the given\n+\t * commit are not set in a candidate pseudo-merge's commits\n+\t * bitmap.\n+\t *\n+\t * - After processing all bits, enumerate the remaining set of\n+\t * viable pseudo-merge bitmaps, and check that their\n+\t * popcount() matches the number of parents in the given\n+\t * commit.\n+\t */\n+\tfor (i = 0; i < pm->nr; i++) {\n+\t\tstruct pseudo_merge *candidate = use_pseudo_merge(pm, &pm->v[i]);\n+\t\tif (!candidate || candidate->satisfied)\n+\t\t\tcontinue;\n+\t\tif (!bitmap_equals_ewah(parents, candidate->commits))\n+\t\t\tcontinue;\n+\n+\t\tmatch = candidate;\n+\t\tmatch->satisfied = 1;\n+\t\tbreak;\n+\t}\n+\n+\treturn match;\n+}\ndiff --git a/pseudo-merge.h b/pseudo-merge.h\nindex 755edc054ae..2aca01d0566 100644\n--- a/pseudo-merge.h\n+++ b/pseudo-merge.h\n@@ -206,4 +206,11 @@ int cascade_pseudo_merges(const struct pseudo_merge_map *pm,\n \t\t\t struct bitmap *result,\n \t\t\t struct bitmap *roots);\n \n+/*\n+ * Returns a pseudo-merge which contains the exact set of commits\n+ * listed in the \"parents\" bitamp, or NULL if none could be found.\n+ */\n+struct pseudo_merge *pseudo_merge_for_parents(const struct pseudo_merge_map *pm,\n+\t\t\t\t\t struct bitmap *parents);\n+\n #endif\ndiff --git a/t/t5333-pseudo-merge-bitmaps.sh b/t/t5333-pseudo-merge-bitmaps.sh\nindex 4c9aebcffdc..f052f395a77 100755\n--- a/t/t5333-pseudo-merge-bitmaps.sh\n+++ b/t/t5333-pseudo-merge-bitmaps.sh\n@@ -22,6 +22,10 @@ test_pseudo_merges_cascades () {\n \ttest_trace2_data bitmap pseudo_merges_cascades \"$1\"\n }\n \n+test_pseudo_merges_reused () {\n+\ttest_trace2_data pack-bitmap-write building_bitmaps_pseudo_merge_reused \"$1\"\n+}\n+\n tag_everything () {\n \tgit rev-list --all --no-object-names >in &&\n \tperl -lne '\n@@ -325,4 +329,65 @@ test_expect_success 'pseudo-merge overlap stale traversal' '\n \t)\n '\n \n+test_expect_success 'pseudo-merge reuse' '\n+\tgit init pseudo-merge-reuse &&\n+\t(\n+\t\tcd pseudo-merge-reuse &&\n+\n+\t\tstable=\"1641013200\" && # 2022-01-01\n+\t\tunstable=\"1672549200\" && # 2023-01-01\n+\n+\t\tGIT_COMMITTER_DATE=\"$stable +0000\" &&\n+\t\texport GIT_COMMITTER_DATE &&\n+\t\ttest_commit_bulk --notick 128 &&\n+\t\tGIT_COMMITTER_DATE=\"$unstable +0000\" &&\n+\t\texport GIT_COMMITTER_DATE &&\n+\t\ttest_commit_bulk --notick 128 &&\n+\n+\t\ttag_everything &&\n+\n+\t\tgit \\\n+\t\t\t-c bitmapPseudoMerge.test.pattern=\"refs/tags/\" \\\n+\t\t\t-c bitmapPseudoMerge.test.maxMerges=1 \\\n+\t\t\t-c bitmapPseudoMerge.test.threshold=now \\\n+\t\t\t-c bitmapPseudoMerge.test.stableThreshold=$(($unstable - 1)) \\\n+\t\t\t-c bitmapPseudoMerge.test.stableSize=512 \\\n+\t\t\trepack -adb &&\n+\n+\t\ttest_pseudo_merges >merges &&\n+\t\ttest_line_count = 2 merges &&\n+\n+\t\ttest_pseudo_merge_commits 0 >stable-oids.before &&\n+\t\ttest_pseudo_merge_commits 1 >unstable-oids.before &&\n+\n+\t\t: >trace2.txt &&\n+\t\tGIT_TRACE2_EVENT=$PWD/trace2.txt git \\\n+\t\t\t-c bitmapPseudoMerge.test.pattern=\"refs/tags/\" \\\n+\t\t\t-c bitmapPseudoMerge.test.maxMerges=2 \\\n+\t\t\t-c bitmapPseudoMerge.test.threshold=now \\\n+\t\t\t-c bitmapPseudoMerge.test.stableThreshold=$(($unstable - 1)) \\\n+\t\t\t-c bitmapPseudoMerge.test.stableSize=512 \\\n+\t\t\trepack -adb &&\n+\n+\t\ttest_pseudo_merges_reused 1 merges &&\n+\t\ttest_line_count = 3 merges &&\n+\n+\t\ttest_pseudo_merge_commits 0 >stable-oids.after &&\n+\t\tfor i in 1 2\n+\t\tdo\n+\t\t\ttest_pseudo_merge_commits $i || return 1\n+\t\tdone >unstable-oids.after &&\n+\n+\t\tsort -u expect &&\n+\t\tsort -u actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tsort -u expect &&\n+\t\tsort -u actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.45.1.175.gcf0316ad0e9\n\n"},{"id":"495480","messageId":"cf0316ad0e9fc1ab77f67df69d3dc0abff5b80ce.1716499565.git.me@ttaylorr.com","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"[PATCH v4 24/24] t/perf: implement performance tests for pseudo-merge bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-05-23T21:27:24Z","receivedAt":"2024-05-23T21:27:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement a straightforward performance test demonstrating the benefit\nof pseudo-merge bitmaps by measuring how long it takes to count\nreachable objects in a few different scenarios:\n\n - without bitmaps, to demonstrate a reasonable baseline\n - with bitmaps, but without pseudo-merges\n - with bitmaps and pseudo-merges\n\nResults from running this test on git.git are as follows:\n\n Test this tree\n -----------------------------------------------------------------------------------\n 5333.2: git rev-list --count --all --objects (no bitmaps) 3.54(3.45+0.08)\n 5333.3: git rev-list --count --all --objects (no pseudo-merges) 0.43(0.40+0.03)\n 5333.4: git rev-list --count --all --objects (with pseudo-merges) 0.12(0.11+0.01)\n\nOn a private repository which is much larger, and has many spikey parts\nof history that aren't merged into the 'master' branch, the results are\nas follows:\n\n Test this tree\n ---------------------------------------------------------------------------------------\n 5333.1: git rev-list --count --all --objects (no bitmaps) 122.29(121.31+0.97)\n 5333.2: git rev-list --count --all --objects (no pseudo-merges) 21.88(21.30+0.58)\n 5333.3: git rev-list --count --all --objects (with pseudo-merges) 5.05(4.77+0.28)\n\nSigned-off-by: Taylor Blau \n---\n t/perf/p5333-pseudo-merge-bitmaps.sh | 32 ++++++++++++++++++++++++++++\n 1 file changed, 32 insertions(+)\n create mode 100755 t/perf/p5333-pseudo-merge-bitmaps.sh\n\ndiff --git a/t/perf/p5333-pseudo-merge-bitmaps.sh b/t/perf/p5333-pseudo-merge-bitmaps.sh\nnew file mode 100755\nindex 00000000000..2e8b1d2635e\n--- /dev/null\n+++ b/t/perf/p5333-pseudo-merge-bitmaps.sh\n@@ -0,0 +1,32 @@\n+#!/bin/sh\n+\n+test_description='pseudo-merge bitmaps'\n+. ./perf-lib.sh\n+\n+test_perf_large_repo\n+\n+test_expect_success 'setup' '\n+\tgit \\\n+\t\t-c bitmapPseudoMerge.all.pattern=\"refs/\" \\\n+\t\t-c bitmapPseudoMerge.all.threshold=now \\\n+\t\t-c bitmapPseudoMerge.all.stableThreshold=never \\\n+\t\t-c bitmapPseudoMerge.all.maxMerges=64 \\\n+\t\t-c pack.writeBitmapLookupTable=true \\\n+\t\trepack -adb\n+'\n+\n+test_perf 'git rev-list --count --all --objects (no bitmaps)' '\n+\tgit rev-list --objects --all\n+'\n+\n+test_perf 'git rev-list --count --all --objects (no pseudo-merges)' '\n+\tGIT_TEST_USE_PSEUDO_MERGES=0 \\\n+\t\tgit rev-list --objects --all --use-bitmap-index\n+'\n+\n+test_perf 'git rev-list --count --all --objects (with pseudo-merges)' '\n+\tGIT_TEST_USE_PSEUDO_MERGES=1 \\\n+\t\tgit rev-list --objects --all --use-bitmap-index\n+'\n+\n+test_done\n-- \n2.45.1.175.gcf0316ad0e9\n"},{"id":"495619","messageId":"20240525031317.GA1889620@coredump.intra.peff.net","threadId":"61166","inReplyTo":"Zk+epE2S4KErPyVk@nand.local","subject":"Re: [PATCH v3 30/30] t/perf: implement performace tests for pseudo-merge bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-25T03:13:17Z","receivedAt":"2024-05-25T03:13:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 23, 2024 at 03:53:08PM -0400, Taylor Blau wrote:\n\n> -\tGIT_TEST_USE_PSEDUO_MERGES=0 \\\n> +\tGIT_TEST_USE_PSEUDO_MERGES=0 \\\n\nWhoops.\n\n> Sure enough, that shows us a little gap between the \"no pseudo-merges\"\n> and \"with pseudo-merges\" case:\n> \n> ```\n> Test this tree\n> -----------------------------------------------------------------------------------\n> 5333.2: git rev-list --count --all --objects (no bitmaps) 3.54(3.45+0.08)\n> 5333.3: git rev-list --count --all --objects (no pseudo-merges) 0.43(0.40+0.03)\n> 5333.4: git rev-list --count --all --objects (with pseudo-merges) 0.12(0.11+0.01)\n> ```\n\nOK, that seems more like it. 300ms is nice, but there's just not that\nmuch improvement to make here.\n\nThis one is much more exciting:\n\n> ```\n> Test this tree\n> ---------------------------------------------------------------------------------------\n> 5333.1: git rev-list --count --all --objects (no bitmaps) 122.29(121.31+0.97)\n> 5333.2: git rev-list --count --all --objects (no pseudo-merges) 21.88(21.30+0.58)\n> 5333.3: git rev-list --count --all --objects (with pseudo-merges) 5.05(4.77+0.28)\n> ```\n\nVery nice improvement.\n\nI wonder what we spend the final 5s on. Maybe just book-keeping to\nassemble all the tips (and maybe even parse tip commits? I can't\nremember if we ever optimized that out). Anyway, that's all out of scope\nfor your series. Getting rid of the expensive traversal would let us\nfocus on those final bits. ;)\n\n-Peff\n"},{"id":"495620","messageId":"20240525031516.GB1889620@coredump.intra.peff.net","threadId":"61166","inReplyTo":"Zk+hP++s/Pz+yym4@nand.local","subject":"Re: [PATCH v3 00/30] pack-bitmap: pseudo-merge reachability bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-25T03:15:16Z","receivedAt":"2024-05-25T03:15:17Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 23, 2024 at 04:04:15PM -0400, Taylor Blau wrote:\n\n> On Thu, May 23, 2024 at 07:05:32AM -0400, Jeff King wrote:\n> > I wonder if the start of the pseudo-merge section should have a 4-byte\n> > version/flags field itself? I don't think that's something we've done\n> > before, and maybe it's overkill. I dunno. It's just a lot easier to do\n> > now than later.\n> \n> I think the tricky thing here would be that the extension itself is a\n> variable size, so every version would have to put the \"extension size\"\n> field in the same place.\n> \n> Otherwise, an older Git client which doesn't understand a future version\n> of the pseudo-merge extension wouldn't know how large the extension is,\n> and wouldn't be able to adjust the index_end field appropriately to skip\n> over it.\n> \n> Of course, we could make it a convention that says \"all versions have to\n> place the extension size field at the same relative offset\", but it\n> feels weird to read some of the extension while not understanding the\n> whole thing.\n\nAh, yeah, I didn't think of that. That definitely complicates things.\n\nIt certainly would be possible to have a version+size header at the\nstart. Which is...basically reinventing the chunk format. Let's not\nworry about it for now, and as a long-term thing we might consider\nmoving the bitmap format over to the chunk style.\n\n-Peff\n"},{"id":"495621","messageId":"20240525032229.GA1890223@coredump.intra.peff.net","threadId":"61166","inReplyTo":"180072ce84868265acfda8c1adf375e39a3b7610.1716499565.git.me@ttaylorr.com","subject":"Re: [PATCH v4 11/24] pseudo-merge: implement support for selecting pseudo-merge commits","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-25T03:22:29Z","receivedAt":"2024-05-25T03:22:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 23, 2024 at 05:26:42PM -0400, Taylor Blau wrote:\n\n> +static uint32_t pseudo_merge_group_size(const struct pseudo_merge_group *group,\n> +\t\t\t\t\tconst struct pseudo_merge_matches *matches,\n> +\t\t\t\t\tuint32_t i)\n> +{\n> +\tdouble C = 0.0f;\n\nThis version mostly drops the \"f\" from floating point constants, since\nthey're now doubles. But this one doesn't.\n\nI don't think it really matters in practice (the number obviously fits\nin a float, and then it ends up as a double in the variable), so it's\nreally just a style / readability question.\n\nNot worth a re-roll IMHO.\n\n-Peff\n"},{"id":"495622","messageId":"20240525032534.GB1890223@coredump.intra.peff.net","threadId":"61166","inReplyTo":"936f6d1b7e392367bb87a755d014633d7171f0ab.1716499565.git.me@ttaylorr.com","subject":"Re: [PATCH v4 19/24] t/test-lib-functions.sh: support `--notick` in `test_commit_bulk()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-25T03:25:34Z","receivedAt":"2024-05-25T03:25:35Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 23, 2024 at 05:27:08PM -0400, Taylor Blau wrote:\n\n> One of the tests we'll want to add for pseudo-merge bitmaps needs to be\n> able to generate a large number of commits at a specific date.\n> \n> Support the `--notick` option (with identical semantics to the\n> `--notick` option for `test_commit()`) within `test_commit_bulk` as a\n> prerequisite for that. Callers can then set the various _DATE variables\n> themselves.\n\nLooks good. I expected to see you add \"--date\" _also_, but it looks like\nyou just set $GIT_COMMITTER_DATE yourself. Which I think is fine for our\npurposes here.\n\n-Peff\n"},{"id":"495623","messageId":"20240525032630.GC1890223@coredump.intra.peff.net","threadId":"61166","inReplyTo":"cover.1716499565.git.me@ttaylorr.com","subject":"Re: [PATCH v4 00/24] pack-bitmap: pseudo-merge reachability bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-05-25T03:26:30Z","receivedAt":"2024-05-25T03:26:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 23, 2024 at 05:26:06PM -0400, Taylor Blau wrote:\n\n> Here is another reroll my topic to introduce pseudo-merge bitmaps.\n> \n> The implementation is still relatively unchanged compared to last time,\n> save for the review that Peff provided on the remaining parts of this\n> series.\n\nThanks, this version looks good to me!\n\n-Peff\n"}]}