{"thread":{"id":"65995","subject":"git-last-modified(1) slower than git-log(1)?","startedAt":"2026-07-14T18:36:13Z","lastAt":"2026-07-17T19:12:11Z","messageCount":8,"participants":["Gusted","Jeff King","Toon Claes"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"548157","messageId":"17f356ff-7bfb-47f5-b714-62a95cc8b821@codeberg.org","threadId":"65995","inReplyTo":null,"subject":"git-last-modified(1) slower than git-log(1)?","fromName":"Gusted","fromEmail":"gusted@codeberg.org","sentAt":"2026-07-14T18:33:59Z","receivedAt":"2026-07-14T18:36:13Z","isPatch":false,"body":"Hi,\n\nI'm working at switching Forgejo's implementation of getting the last\nmodified commits in a directory to git-last-modified(1). I'd expected\nequal or better performance than the current implementation, but have\nnot yet been able to get this and I'm a bit puzzled as to why.\n\nThe current implementation of Forgejo (inherited from Gitea) works\nroughly like this:\n1. Run `git log --name-status -c --format=commit%x00%H %P%x00\" --parents\n--no-renames -t -z $OID -- :(literal)some/path`, the output of this is\nquite complex and possible outputs more information than necessary.\n2. The output of this is piped to some code to a parser and reconstructs\nwhat commit ID last modified each file in the directory.\n3. Via `git cat-file --batch` get each unique commits information.\n\nWith git-last-modified(1) (-z --show-trees --max-depth=0) this replaces\nstep 1-2, but is slower. I've isolated the degraded performance to the\nfact that git-last-changed(1) takes more time to finish. So from my\nperspective it does not seem worth it to replace the current\nimplementation with git-last-modified(1), and I would like to know if\nI'm missing something here or if git-last-modified(1) possibly could see\na speedup?\n\nThe repository I'm currently using to evaluate the performance is\nhttps://codeberg.org/ziglang/zig\n\nReproduction steps:\n1. `git clone https://codeberg.org/ziglang/zig $(mktemp -d)`\n2. cd to tmp directory.\n3. `git commit-graph write --changed-paths`. As git-last-modified(1)\nmakes good use of the bloom filters.\n4. `hyperfine 'git last-modified -z -t --max-depth=0\n80d06578ac66bce3aa0a21e9610cdb782b9a0593 -- doc/langref/' 'git log\n--name-status -c \"--format=commit%x00%H %P%x00\" --parents --no-renames\n-t -z 80d06578ac66bce3aa0a21e9610cdb782b9a0593 -- \":(literal)doc/langref\"'`\n\nWith as output:\nBenchmark 1: git last-modified -z -t --max-depth=0\n80d06578ac66bce3aa0a21e9610cdb782b9a0593 -- doc/langref/\n Time (mean ± σ): 66.5 ms ± 0.6 ms [User: 60.6 ms, System: 5.2 ms]\n Range (min … max): 65.3 ms … 67.7 ms 44 runs\n\nBenchmark 2: git log --name-status -c \"--format=commit%x00%H %P%x00\"\n--parents --no-renames -t -z 80d06578ac66bce3aa0a21e9610cdb782b9a0593 --\n\":(literal)doc/langref\"\n Time (mean ± σ): 26.2 ms ± 1.0 ms [User: 17.3 ms, System: 8.4 ms]\n Range (min … max): 24.3 ms … 30.1 ms 110 runs\n\nSummary\n git log --name-status -c \"--format=commit%x00%H %P%x00\" --parents\n--no-renames -t -z 80d06578ac66bce3aa0a21e9610cdb782b9a0593 --\n\":(literal)doc/langref\" ran\n 2.54 ± 0.10 times faster than git last-modified -z -t --max-depth=0\n80d06578ac66bce3aa0a21e9610cdb782b9a0593 -- doc/langref/\n\nKind Regards\nGusted\n"},{"id":"548353","messageId":"20260716042808.GA1151612@coredump.intra.peff.net","threadId":"65995","inReplyTo":"17f356ff-7bfb-47f5-b714-62a95cc8b821@codeberg.org","subject":"Re: git-last-modified(1) slower than git-log(1)?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-07-16T04:28:08Z","receivedAt":"2026-07-16T04:28:10Z","isPatch":false,"body":"On Tue, Jul 14, 2026 at 08:33:59PM +0200, Gusted wrote:\n\n> The repository I'm currently using to evaluate the performance is\n> https://codeberg.org/ziglang/zig\n> \n> Reproduction steps:\n> 1. `git clone https://codeberg.org/ziglang/zig $(mktemp -d)`\n> 2. cd to tmp directory.\n> 3. `git commit-graph write --changed-paths`. As git-last-modified(1)\n> makes good use of the bloom filters.\n> 4. `hyperfine 'git last-modified -z -t --max-depth=0\n> 80d06578ac66bce3aa0a21e9610cdb782b9a0593 -- doc/langref/' 'git log\n> --name-status -c \"--format=commit%x00%H %P%x00\" --parents --no-renames\n> -t -z 80d06578ac66bce3aa0a21e9610cdb782b9a0593 -- \":(literal)doc/langref\"'`\n\nThanks for this concrete reproduction. I can see the same problem here.\nInterestingly, if we turn off changed-paths, we get very different\nresults.\n\nWithout a commit graph at all, last-modified wins (this is using the zig\nrepo and the commands above):\n\n  - log: 150ms\n  - last-modified: 79ms\n\nBut with a graph and no changed-paths, they're about equal:\n\n  - log: 61ms\n  - last-modified: 61ms\n\nAnd then with changed-paths, the log command gets much faster but\nlast-modified gets slower!\n\n  - log: 20ms\n  - last-modified: 64ms\n\nI think there's a tradeoff in the way that last-modified uses the bloom\nfilters. It makes a key for every path we're interested in, and then for\neach commit, we check each key to say \"is this in the commit's filter?\".\n\nSo if you have a subdirectory with a non-trivial number of entries (like\ndoc/langref here which has 290), but most commits don't touch that path\nat all (only 120 out of ~39k in this case), we'll spend a lot of time\nchecking each key against each filter. We save ourselves opening the\ntrees, but at the cost of 290*39k filter comparisons).\n\nWhereas in the git-log case, we make a filter key out of the single\npathspec we're given, and then check each commit against that. So we\nonly do a single filter check for each commit to narrow it down to those\n120 that matter (modulo a few filter false positives).\n\nBut I don't see any reason that last-modified couldn't _also_ do that:\npre-filter the commits with a commit matching the original pathspec, and\ndiscard most commits with a single filter check.\n\nThe hacky patch below does this, and brings my last-modified runtime\ndown to 16ms (a 4x improvement, and just a bit faster than git-log).\n\nIt tries to reuse the logic from revision.c, so it's doing the exact\nsame filtering that git-log would do. I think there are other ways to do\nit. E.g., we could make our own \"root\" bloom key that contains all of\nthe paths and pre-filter with that. But it seemed to be a little slower\nwhen I tried it (~24ms). I'd guess that the problem is that because the\nbloom filter is probabilistic, if you shove too many items into a single\nkey you'll end getting more and more false positives. So putting all 290\nentries into one key is too much, and we are better off just considering\nthe shared prefix.\n\nAnyway, here's the patch. Toon, I'm not planning to take it further\nimmediately, but you may be interested in poking at it. It probably\nneeds at least:\n\n  - some light refactoring of revision.c\n\n  - tests? We don't seem to cover last-modified with changed-paths at\n    all, and just rely on the test-vars CI job which sets\n    GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS. It did pass for me with that\n    flag, so surely I didn't introduce any bugs. :)\n\n  - more timing exploration; e.g., might it make things worse if\n    doc/langref were touched in 99% of the commits? Probably not, but it\n    might be nice to check timings against a few repo shapes and request\n    depths.\n\n  - Not all pathspecs can support bloom filters (e.g., \"*.c\" would not).\n    So in theory:\n\n       git last-modified HEAD -- \"*.c\"\n\n    could work, but wouldn't be optimized. I don't think it _does_ work\n    now, because last-modified's max-depth logic complains. So it might\n    be a non-issue.\n\n    But I think it is solvable if we really wanted. Rather than\n    traversing looking for \"*.c\", we actually expand the pathspec in the\n    tip commit to a set of literal paths, and then as we traverse we\n    look for those paths. So we could collect all of \"*.c\" and then\n    add bloom keys for the shared prefixes. I think this does get tricky\n    in the general case, though. If you have \"a/b/c\" and \"a/b/d\",\n    looking for \"a/b\" is reasonable. But what if you also have \"a/e\"?\n    Should you just have a key for \"a/\", or both \"a/b\" and \"a/e\"?\n    There are some tradeoffs between how often uninteresting things in\n    \"a/\" will give us a false positive, versus the cost of checking\n    extra keys.\n\n    So maybe an interesting area, but given that in practice most people\n    will feed a single pathspec to last-modified, it's a lot easier to\n    just use that.\n\n  - I know that last-modified was derived from GitHub's blame-tree\n    implementation (which I originally wrote, but stopped paying\n    attention to well before it learned about changed-path filters). I\n    don't know if the problem was solved separately there, but it would\n    be worth checking. +cc Taylor\n\n-Peff\n\n---\ndiff --git a/builtin/last-modified.c b/builtin/last-modified.c\nindex 5478182f2e..c07169258f 100644\n--- a/builtin/last-modified.c\n+++ b/builtin/last-modified.c\n@@ -254,6 +254,29 @@ static void pass_to_parent(struct bitmap *c,\n \tbitmap_set(p, pos);\n }\n \n+/*\n+ * revision.c already has this functionality, but it is not public\n+ * and it looks up the filter itself. But probably some refactoring\n+ * could make it available at the right level?\n+ */\n+static bool filter_contains_keyvec(const struct bloom_filter *filter,\n+\t\t\t\t   struct rev_info *rev)\n+{\n+\t/*\n+\t * If we have no keys, we must pessimistically assume a match.\n+\t */\n+\tif (!rev->bloom_keyvecs_nr)\n+\t\treturn true;\n+\n+\tfor (int i = 0; i < rev->bloom_keyvecs_nr; i++) {\n+\t\tif (bloom_filter_contains_vec(filter,\n+\t\t\t\t\t      rev->bloom_keyvecs[i],\n+\t\t\t\t\t      rev->bloom_filter_settings))\n+\t\t\treturn true;\n+\t}\n+\treturn false;\n+}\n+\n static bool maybe_changed_path(struct last_modified *lm,\n \t\t\t       struct commit *origin,\n \t\t\t       struct bitmap *active)\n@@ -272,6 +295,9 @@ static bool maybe_changed_path(struct last_modified *lm,\n \tif (!filter)\n \t\treturn true;\n \n+\tif (!filter_contains_keyvec(filter, &lm->rev))\n+\t\treturn false;\n+\n \thashmap_for_each_entry(&lm->paths, &iter, ent, hashent) {\n \t\tif (active && !bitmap_get(active, ent->diff_idx))\n \t\t\tcontinue;\n@@ -499,7 +525,22 @@ static int last_modified_init(struct last_modified *lm, struct repository *r,\n \t\treturn argc;\n \t}\n \n-\tlm->rev.bloom_filter_settings = get_bloom_filter_settings(lm->rev.repo);\n+\t/*\n+\t * Load the bloom settings, but also convert our pathspec into\n+\t * bloom_keyvecs that can be used later. This helper should\n+\t * probably be factored out, but we don't want to do it ourselves.\n+\t * There is logic about which pathspecs are allowed or not that\n+\t * we would not want to duplicate.\n+\t */\n+\tprepare_to_use_bloom_filter(&lm->rev);\n+\n+\t/*\n+\t * Even if our initial pathspecs forbid using bloom filters, we'd still\n+\t * use them for the literal paths we expand below in\n+\t * populate_paths_from_revs().\n+\t */\n+\tif (!lm->rev.bloom_filter_settings)\n+\t\tlm->rev.bloom_filter_settings = get_bloom_filter_settings(lm->rev.repo);\n \n \tif (populate_paths_from_revs(lm) < 0)\n \t\treturn -1;\ndiff --git a/revision.c b/revision.c\nindex 137a86d33b..f5b36ea2cc 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -705,7 +705,7 @@ static int convert_pathspec_to_bloom_keyvec(struct bloom_keyvec **out,\n \treturn res;\n }\n \n-static void prepare_to_use_bloom_filter(struct rev_info *revs)\n+void prepare_to_use_bloom_filter(struct rev_info *revs)\n {\n \tif (!revs->commits)\n \t\treturn;\ndiff --git a/revision.h b/revision.h\nindex 569b3fa1cb..1f761b85d0 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -576,4 +576,6 @@ int rewrite_parents(struct rev_info *revs,\n  */\n struct commit_list *get_saved_parents(struct rev_info *revs, const struct commit *commit);\n \n+void prepare_to_use_bloom_filter(struct rev_info *revs);\n+\n #endif\n"},{"id":"548389","messageId":"87v7afffpa.fsf@emacs.iotcl.com","threadId":"65995","inReplyTo":"17f356ff-7bfb-47f5-b714-62a95cc8b821@codeberg.org","subject":"Re: git-last-modified(1) slower than git-log(1)?","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2026-07-16T09:26:25Z","receivedAt":"2026-07-16T09:26:32Z","isPatch":false,"body":"Gusted <gusted@codeberg.org> writes:\n\n> Hi,\n>\n> I'm working at switching Forgejo's implementation of getting the last\n> modified commits in a directory to git-last-modified(1). I'd expected\n> equal or better performance than the current implementation, but have\n> not yet been able to get this and I'm a bit puzzled as to why.\n>\n> The current implementation of Forgejo (inherited from Gitea) works\n> roughly like this:\n> 1. Run `git log --name-status -c --format=commit%x00%H %P%x00\" --parents\n> --no-renames -t -z $OID -- :(literal)some/path`, the output of this is\n> quite complex and possible outputs more information than necessary.\n> 2. The output of this is piped to some code to a parser and reconstructs\n> what commit ID last modified each file in the directory.\n> 3. Via `git cat-file --batch` get each unique commits information.\n>\n> With git-last-modified(1) (-z --show-trees --max-depth=0) this replaces\n> step 1-2, but is slower. I've isolated the degraded performance to the\n> fact that git-last-changed(1) takes more time to finish. So from my\n> perspective it does not seem worth it to replace the current\n> implementation with git-last-modified(1), and I would like to know if\n> I'm missing something here or if git-last-modified(1) possibly could see\n> a speedup?\n>\n> The repository I'm currently using to evaluate the performance is\n> https://codeberg.org/ziglang/zig\n>\n> Reproduction steps:\n> 1. `git clone https://codeberg.org/ziglang/zig $(mktemp -d)`\n> 2. cd to tmp directory.\n> 3. `git commit-graph write --changed-paths`. As git-last-modified(1)\n> makes good use of the bloom filters.\n> 4. `hyperfine 'git last-modified -z -t --max-depth=0\n> 80d06578ac66bce3aa0a21e9610cdb782b9a0593 -- doc/langref/' 'git log\n> --name-status -c \"--format=commit%x00%H %P%x00\" --parents --no-renames\n> -t -z 80d06578ac66bce3aa0a21e9610cdb782b9a0593 -- \":(literal)doc/langref\"'`\n>\n> With as output:\n> Benchmark 1: git last-modified -z -t --max-depth=0\n> 80d06578ac66bce3aa0a21e9610cdb782b9a0593 -- doc/langref/\n>  Time (mean ± σ): 66.5 ms ± 0.6 ms [User: 60.6 ms, System: 5.2 ms]\n>  Range (min … max): 65.3 ms … 67.7 ms 44 runs\n>\n> Benchmark 2: git log --name-status -c \"--format=commit%x00%H %P%x00\"\n> --parents --no-renames -t -z 80d06578ac66bce3aa0a21e9610cdb782b9a0593 --\n> \":(literal)doc/langref\"\n>  Time (mean ± σ): 26.2 ms ± 1.0 ms [User: 17.3 ms, System: 8.4 ms]\n>  Range (min … max): 24.3 ms … 30.1 ms 110 runs\n>\n> Summary\n>  git log --name-status -c \"--format=commit%x00%H %P%x00\" --parents\n> --no-renames -t -z 80d06578ac66bce3aa0a21e9610cdb782b9a0593 --\n> \":(literal)doc/langref\" ran\n>  2.54 ± 0.10 times faster than git last-modified -z -t --max-depth=0\n> 80d06578ac66bce3aa0a21e9610cdb782b9a0593 -- doc/langref/\n\nHi Gusted,\n\nThanks for reaching out.\n\nYou're actually not the first to notice this, and I've been aware of\nthis.\n\nThe thing is, you're testing the difference on a single file. For us at\nGitLab, it wasn't very useful to optimize that use-case, because usually\nwe want to see the last commit for a bunch of files at once.\nSo the use-case for git-last-modified(1) for us has been to replace\n(pseudo code):\n\n$ FILES=$(git ls-tree $COMMIT $PATH)\n$ foreach $FILE in $FILES; do git log -1 $COMMIT -- $FILE; end\n\nGitLab is batching files 25 at once, and in my benchmarking, it was\nshown git-last-modified(1) is faster:\n\n$ git last-modified $COMMIT -- <files\n\n(I did this benchmarking in our Gitaly component to have a real-world\nexperience and you can visit the results at:\nhttps://gitlab.com/gitlab-org/gitaly/-/merge_requests/7999#note_2850505479\n)\n\nSo we left the door open for future improvement, although I never have\ngotten to it. At some point I was trying to chase down when git-log(1)\nwas doing differently, but I never figured it out.\n\nBut this email challenged me already. And with some help of AI, I\nmanaged to work on some improvements. You can expect a patch series\nsoon.\n\n(Right before sending out this mail I noticed Peff sent out some changes\nas well. I'll coordinate how to combine.)\n\n-- \nCheers,\nToon\n"},{"id":"548397","messageId":"87se5jf9f7.fsf@emacs.iotcl.com","threadId":"65995","inReplyTo":"20260716042808.GA1151612@coredump.intra.peff.net","subject":"Re: git-last-modified(1) slower than git-log(1)?","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2026-07-16T11:42:04Z","receivedAt":"2026-07-16T11:42:16Z","isPatch":false,"body":"Jeff King <peff@peff.net> writes:\n\n> On Tue, Jul 14, 2026 at 08:33:59PM +0200, Gusted wrote:\n>\n>> The repository I'm currently using to evaluate the performance is\n>> https://codeberg.org/ziglang/zig\n>> \n>> Reproduction steps:\n>> 1. `git clone https://codeberg.org/ziglang/zig $(mktemp -d)`\n>> 2. cd to tmp directory.\n>> 3. `git commit-graph write --changed-paths`. As git-last-modified(1)\n>> makes good use of the bloom filters.\n>> 4. `hyperfine 'git last-modified -z -t --max-depth=0\n>> 80d06578ac66bce3aa0a21e9610cdb782b9a0593 -- doc/langref/' 'git log\n>> --name-status -c \"--format=commit%x00%H %P%x00\" --parents --no-renames\n>> -t -z 80d06578ac66bce3aa0a21e9610cdb782b9a0593 -- \":(literal)doc/langref\"'`\n>\n> Thanks for this concrete reproduction. I can see the same problem\n> here.\n\nAs I mentioned, I was aware of that issue, but never felt the need (and\ndidn't have the deep Bloom filter knowledge) to fix it.\n\n> Interestingly, if we turn off changed-paths, we get very different\n> results.\n>\n> Without a commit graph at all, last-modified wins (this is using the zig\n> repo and the commands above):\n>\n>   - log: 150ms\n>   - last-modified: 79ms\n>\n> But with a graph and no changed-paths, they're about equal:\n>\n>   - log: 61ms\n>   - last-modified: 61ms\n>\n> And then with changed-paths, the log command gets much faster but\n> last-modified gets slower!\n>\n>   - log: 20ms\n>   - last-modified: 64ms\n>\n> I think there's a tradeoff in the way that last-modified uses the bloom\n> filters. It makes a key for every path we're interested in, and then for\n> each commit, we check each key to say \"is this in the commit's filter?\".\n>\n> So if you have a subdirectory with a non-trivial number of entries (like\n> doc/langref here which has 290), but most commits don't touch that path\n> at all (only 120 out of ~39k in this case), we'll spend a lot of time\n> checking each key against each filter. We save ourselves opening the\n> trees, but at the cost of 290*39k filter comparisons).\n>\n> Whereas in the git-log case, we make a filter key out of the single\n> pathspec we're given, and then check each commit against that. So we\n> only do a single filter check for each commit to narrow it down to those\n> 120 that matter (modulo a few filter false positives).\n\nOh, that's very useful of you to explain this. Thank you.\n\n> But I don't see any reason that last-modified couldn't _also_ do that:\n> pre-filter the commits with a commit matching the original pathspec, and\n> discard most commits with a single filter check.\n>\n> The hacky patch below does this, and brings my last-modified runtime\n> down to 16ms (a 4x improvement, and just a bit faster than git-log).\n\nFunny, I was toying around with my AI agent, and they came with a\nsimilar solution, but I'm working on a cleaner solution.\n\n> It tries to reuse the logic from revision.c, so it's doing the exact\n> same filtering that git-log would do. I think there are other ways to do\n> it. E.g., we could make our own \"root\" bloom key that contains all of\n> the paths and pre-filter with that. But it seemed to be a little slower\n> when I tried it (~24ms). I'd guess that the problem is that because the\n> bloom filter is probabilistic, if you shove too many items into a single\n> key you'll end getting more and more false positives. So putting all 290\n> entries into one key is too much, and we are better off just considering\n> the shared prefix.\n>\n> Anyway, here's the patch. Toon, I'm not planning to take it further\n> immediately, but you may be interested in poking at it. It probably\n> needs at least:\n>\n>   - some light refactoring of revision.c\n\nAgreed.\n\n>   - tests? We don't seem to cover last-modified with changed-paths at\n>     all, and just rely on the test-vars CI job which sets\n>     GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS. It did pass for me with that\n>     flag, so surely I didn't introduce any bugs. :)\n\nFair of you calling that out. Thanks for checking.\n\n>   - more timing exploration; e.g., might it make things worse if\n>     doc/langref were touched in 99% of the commits? Probably not, but it\n>     might be nice to check timings against a few repo shapes and request\n>     depths.\n\nMaybe, I tried a few things.\n\nOn gitlab-org/gitlab (our Rails monolith), there seems to be a noticable\nimprovement when running for `app/`:\n\n    Benchmark 1: old\n      Time (mean ± σ):     435.5 ms ±   9.8 ms    [User: 369.5 ms, System: 64.4 ms]\n      Range (min … max):   425.3 ms … 450.9 ms    5 runs\n     \n    Benchmark 2: new\n      Time (mean ± σ):     278.5 ms ±  32.3 ms    [User: 208.0 ms, System: 69.3 ms]\n      Range (min … max):   246.2 ms … 314.8 ms    5 runs\n     \n    Summary\n      new ran\n        1.56 ± 0.18 times faster than old\n\nThe app/ directory is touched by roughly 35% of the commits.\n\nIn gitlab-org/gitaly, I ran a benchmark on `internal/`, which is touched\nin about 58% of the commits:\n\n    Benchmark 1: old\n      Time (mean ± σ):      13.2 ms ±   1.3 ms    [User: 10.4 ms, System: 2.5 ms]\n      Range (min … max):    11.3 ms …  14.8 ms    10 runs\n     \n    Benchmark 2: new\n      Time (mean ± σ):       9.9 ms ±   0.4 ms    [User: 7.3 ms, System: 2.4 ms]\n      Range (min … max):     9.6 ms …  10.8 ms    10 runs\n     \n    Summary\n      new ran\n        1.33 ± 0.14 times faster than old\n\n(although Gitaly a lot less commits, ~23k commits vs ~530k in our Rails\nmonolith)\n\nAnd also --recursive it's faster:\n\n    Benchmark 1: old\n      Time (mean ± σ):     204.6 ms ±   4.4 ms    [User: 193.2 ms, System: 10.5 ms]\n      Range (min … max):   199.9 ms … 213.2 ms    10 runs\n     \n    Benchmark 2: new\n      Time (mean ± σ):     181.5 ms ±   2.7 ms    [User: 170.9 ms, System: 9.7 ms]\n      Range (min … max):   177.0 ms … 187.1 ms    10 runs\n     \n    Summary\n      new ran\n        1.13 ± 0.03 times faster than old\n\nPersonally I'm not too worried any use-case would be at least equally\nfast.\n\n>   - Not all pathspecs can support bloom filters (e.g., \"*.c\" would not).\n>     So in theory:\n>\n>        git last-modified HEAD -- \"*.c\"\n>\n>     could work, but wouldn't be optimized. I don't think it _does_ work\n>     now, because last-modified's max-depth logic complains. So it might\n>     be a non-issue.\n>\n>     But I think it is solvable if we really wanted. Rather than\n>     traversing looking for \"*.c\", we actually expand the pathspec in the\n>     tip commit to a set of literal paths, and then as we traverse we\n>     look for those paths. So we could collect all of \"*.c\" and then\n>     add bloom keys for the shared prefixes. I think this does get tricky\n>     in the general case, though. If you have \"a/b/c\" and \"a/b/d\",\n>     looking for \"a/b\" is reasonable. But what if you also have \"a/e\"?\n>     Should you just have a key for \"a/\", or both \"a/b\" and \"a/e\"?\n>     There are some tradeoffs between how often uninteresting things in\n>     \"a/\" will give us a false positive, versus the cost of checking\n>     extra keys.\n>\n>     So maybe an interesting area, but given that in practice most people\n>     will feed a single pathspec to last-modified, it's a lot easier to\n>     just use that.\n\nYeah, I rather not deal with that right now.\n\n>   - I know that last-modified was derived from GitHub's blame-tree\n>     implementation (which I originally wrote, but stopped paying\n>     attention to well before it learned about changed-path filters). I\n>     don't know if the problem was solved separately there, but it would\n>     be worth checking. +cc Taylor\n>\n> -Peff\n>\n> ---\n> diff --git a/builtin/last-modified.c b/builtin/last-modified.c\n> index 5478182f2e..c07169258f 100644\n> --- a/builtin/last-modified.c\n> +++ b/builtin/last-modified.c\n> @@ -254,6 +254,29 @@ static void pass_to_parent(struct bitmap *c,\n>  \tbitmap_set(p, pos);\n>  }\n>  \n> +/*\n> + * revision.c already has this functionality, but it is not public\n> + * and it looks up the filter itself. But probably some refactoring\n> + * could make it available at the right level?\n\nI assume you're talking about check_maybe_different_in_bloom_filter()?\n\nI was working on a fix to simply make it public and call it, but that's\na very valid point you're making. I'll change my plans.\n\n> + */\n> +static bool filter_contains_keyvec(const struct bloom_filter *filter,\n> +\t\t\t\t   struct rev_info *rev)\n> +{\n> ... [snip]\n\n-- \nCheers,\nToon\n"},{"id":"548466","messageId":"6491c7e6-9310-4a5e-8ba5-9d3a2e7e3312@codeberg.org","threadId":"65995","inReplyTo":"87v7afffpa.fsf@emacs.iotcl.com","subject":"Re: git-last-modified(1) slower than git-log(1)?","fromName":"Gusted","fromEmail":"gusted@codeberg.org","sentAt":"2026-07-17T00:19:49Z","receivedAt":"2026-07-17T00:19:58Z","isPatch":false,"body":"\n\nOn 7/16/26 11:26 AM, Toon Claes wrote:\n> Hi Gusted,\n> \n> Thanks for reaching out.\n> \n> You're actually not the first to notice this, and I've been aware of\n> this.\n> \n> The thing is, you're testing the difference on a single file. For us at\n> GitLab, it wasn't very useful to optimize that use-case, because usually\n> we want to see the last commit for a bunch of files at once.\n> So the use-case for git-last-modified(1) for us has been to replace\n> (pseudo code):\n> \n> $ FILES=$(git ls-tree $COMMIT $PATH)\n> $ foreach $FILE in $FILES; do git log -1 $COMMIT -- $FILE; end\n> \n> GitLab is batching files 25 at once, and in my benchmarking, it was\n> shown git-last-modified(1) is faster:\n> \n> $ git last-modified $COMMIT -- <files\n> \n> (I did this benchmarking in our Gitaly component to have a real-world\n> experience and you can visit the results at:\n> https://gitlab.com/gitlab-org/gitaly/-/merge_requests/7999#note_2850505479\n> )\n> \n> So we left the door open for future improvement, although I never have\n> gotten to it. At some point I was trying to chase down when git-log(1)\n> was doing differently, but I never figured it out.\n> \n> But this email challenged me already. And with some help of AI, I\n> managed to work on some improvements. You can expect a patch series\n> soon.\n> \n> (Right before sending out this mail I noticed Peff sent out some changes\n> as well. I'll coordinate how to combine.)\n> \n\nHi Toon and Jeff,\n\nThanks for having a look at this!\n\nThe use case for Forgejo is the same as Gitlab then, we only use it to\nget the last-modified of each entry in a directory. Looking a bit closer\nI missed that in Forgejo's code it's considered the output might not\nhave the answer of all entries, and happily calls it as many time is\nneeded on the 'remaining paths', especially when using gitlab as\nrepository this required a lot of git-log calls.\n\nLooking at the mentioned benchmark, yeah this is where Forgejo's\nimplementation with git-log would fail in terms of performance\n(seemingly even slower than doing N-1 git-log calls, by using your\nbenchmark numbers). It's somewhere in the minutes :')\n\nKind Regards\nGusted\n"},{"id":"548480","messageId":"20260717080216.GB1832790@coredump.intra.peff.net","threadId":"65995","inReplyTo":"87v7afffpa.fsf@emacs.iotcl.com","subject":"Re: git-last-modified(1) slower than git-log(1)?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-07-17T08:02:16Z","receivedAt":"2026-07-17T08:02:17Z","isPatch":false,"body":"On Thu, Jul 16, 2026 at 11:26:25AM +0200, Toon Claes wrote:\n\n> The thing is, you're testing the difference on a single file. For us at\n> GitLab, it wasn't very useful to optimize that use-case, because usually\n> we want to see the last commit for a bunch of files at once.\n> So the use-case for git-last-modified(1) for us has been to replace\n> (pseudo code):\n\nThat was my assumption at first, too, but I think the log command there\nreally is returning results for the whole subtree. You just have to\npost-process it to pick out the files from each commit.\n\n> $ FILES=$(git ls-tree $COMMIT $PATH)\n> $ foreach $FILE in $FILES; do git log -1 $COMMIT -- $FILE; end\n\nYeah, that is the most horrible way to do it. It's expensive in\nprocesses, but also in walking over the same set of history repeatedly.\n\nThe log in Gusted's example does a single walk, but it is up to the\ncaller to then interpret the walk results. That would add extra time,\nbut I think it scales independently of the time difference he's\nobserving. In his hyperfine results, last-modified is scaling with the\ntotal numbers of commits in the repo, but processing the output scales\nto the number of commits which actually touched the subtree in question.\n\nSo I think it really could perform better than last-modified, even with\nthe post-processing step (which we didn't see nor time). But we should\nbe able to do better in last-modified using similar top-level commit\nfiltering.\n\n-Peff\n"},{"id":"548481","messageId":"20260717080905.GC1832790@coredump.intra.peff.net","threadId":"65995","inReplyTo":"87se5jf9f7.fsf@emacs.iotcl.com","subject":"Re: git-last-modified(1) slower than git-log(1)?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-07-17T08:09:05Z","receivedAt":"2026-07-17T08:09:07Z","isPatch":false,"body":"On Thu, Jul 16, 2026 at 01:42:04PM +0200, Toon Claes wrote:\n\n> >   - more timing exploration; e.g., might it make things worse if\n> >     doc/langref were touched in 99% of the commits? Probably not, but it\n> >     might be nice to check timings against a few repo shapes and request\n> >     depths.\n> \n> Maybe, I tried a few things.\n\nYeah, I would be surprised to find a practical case where it makes\nthings slower. Checking one bloom key is cheap-ish, and unless the\nsubtree being queried is touched by almost every commit, it's going to\nbe a net win.\n\n> Personally I'm not too worried any use-case would be at least equally\n> fast.\n\nSo yeah, that's my gut feeling, too.\n\n> > +/*\n> > + * revision.c already has this functionality, but it is not public\n> > + * and it looks up the filter itself. But probably some refactoring\n> > + * could make it available at the right level?\n> \n> I assume you're talking about check_maybe_different_in_bloom_filter()?\n\nYeah, exactly. We already do the first half (getting the commit's\nfilter) ourselves. And then most of the rest is just trace2 accounting,\nwhich we don't necessarily need to do.  So we're left with just that one\nbloom over the keyvecs, which is fairly trivial. Mostly it felt weird to\nbe looking at the innards of rev_info, and the logic for what those\nkeyvecs means should remain in revision.c.\n\n> I was working on a fix to simply make it public and call it, but that's\n> a very valid point you're making. I'll change my plans.\n\nI was just thinking to split it into two (get the filter, and then check\nthe filter against the rev_info) and make the latter half public.\n\n-Peff\n"},{"id":"548557","messageId":"87fr1h1ldr.fsf@emacs.iotcl.com","threadId":"65995","inReplyTo":"87v7afffpa.fsf@emacs.iotcl.com","subject":"Re: git-last-modified(1) slower than git-log(1)?","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2026-07-17T19:12:00Z","receivedAt":"2026-07-17T19:12:11Z","isPatch":false,"body":"Toon Claes <toon@iotcl.com> writes:\n\n> You're actually not the first to notice this, and I've been aware of\n> this.\n\nHah, wait, ... the previous reporter was ... you[1].\n\n> The thing is, you're testing the difference on a single file.\n\nI was mixing up use-cases, and the problem you're stating here isn't\nrelated to your previous report. Now while I did sent out patches for\nthe problem in this email, the issue in the other email isn't addressed\nwith it. Unfortunately.\n\n(nerdsniping Peff :-))\n\n\nGusted <gusted@codeberg.org> writes:\n\n> The current implementation of Forgejo (inherited from Gitea) works\n> roughly like this:\n> 1. Run `git log --name-status -c --format=commit%x00%H %P%x00\" --parents\n> --no-renames -t -z $OID -- :(literal)some/path`, the output of this is\n> quite complex and possible outputs more information than necessary.\n\nThat's quite clever actually.\n\n> 2. The output of this is piped to some code to a parser and reconstructs\n> what commit ID last modified each file in the directory.\n\nMy only worry would be this could end up in a very long list of\n(duplicate) commits. But you can probably filter out data as you read in\nlines.\n\n> 3. Via `git cat-file --batch` get each unique commits information.\n\nYes we use git-cat-file(1) in batch mode too.\n\nBut, and I learned this from blog post[2] from GitHub about Git v2.55,\ngit v2.55 now has git-format-rev(1)[3]. You can pipe the output of\ngit-last-modified(1) into that and get everything you want at once\n(maybe).\n\nI haven't tried yet to integrate that into GitLab to see if it would\nbring any gains.\n\n[1]: https://lore.kernel.org/git/03f96860-29fc-42a7-a220-c3ec65eb8516@codeberg.org/\n[2]: https://github.blog/open-source/git/highlights-from-git-2-55/\n[3]: https://git-scm.com/docs/git-format-rev\n\n-- \nCheers,\nToon\n"}]}