{"thread":{"id":"58502","subject":"[PATCH] pack-bitmap: remove trace2 region from hot path","startedAt":"2022-09-23T13:00:46Z","lastAt":"2022-09-27T18:37:26Z","messageCount":10,"participants":["Derrick Stolee via GitGitGadget","Junio C Hamano","Derrick Stolee","Taylor Blau","Ævar Arnfjörð Bjarmason","Jeff Hostetler"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"463509","messageId":"pull.1365.git.1663938034607.gitgitgadget@gmail.com","threadId":"58502","inReplyTo":null,"subject":"[PATCH] pack-bitmap: remove trace2 region from hot path","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-23T13:00:34Z","receivedAt":"2022-09-23T13:00:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <derrickstolee@github.com>\n\nThe trace2 region around the call to lazy_bitmap_for_commit() in\nbitmap_for_commit() was added in 28cd730680d (pack-bitmap: prepare to\nread lookup table extension, 2022-08-14). While adding trace2 regions is\ntypically helpful for tracking performance, this method is called\npossibly thousands of times as a commit walk explores commit history\nlooking for a matching bitmap. When trace2 output is enabled, this\nregion is emitted many times and performance is throttled by that\noutput.\n\nFor now, remove these regions entirely.\n\nThis is a critical path, and it would be valuable to measure that the\ntime spent in bitmap_for_commit() does not increase when using the\ncommit lookup table. The best way to do that would be to use a mechanism\nthat sums the time spent in a region and reports a single value at the\nend of the process. This technique was introduced but not merged by [1]\nso maybe this example presents some justification to revisit that\napproach.\n\n[1] https://lore.kernel.org/git/pull.1099.v2.git.1640720202.gitgitgadget@gmail.com/\n\nSigned-off-by: Derrick Stolee <derrickstolee@github.com>\n---\n    [For 2.38.0] pack-bitmap: remove trace2 region from hot path\n    \n    I noticed this while trying to backport the Abhradeep's lookup table\n    work into GitHub's fork. Something went wrong in that process, causing\n    this region to significantly slow down. It turns out that slow down does\n    not reproduce on current 'master', which is good. I must have missed\n    something while I was backporting.\n    \n    Regardless, the use of trace2_region_enter() here should be removed or\n    replaced. For the sake of 2.38.0, this simple removal is safe enough.\n    However, to really dig into what was happening I had to construct a\n    rebase [2] of Jeff's trace2 stopwatch work, then apply changes on top\n    [3] that could replace this region with trace2_timer_*() methods.\n    \n    [2]\n    https://github.com/git/git/compare/master...derrickstolee:trace2-stopwatch\n    [3]\n    https://github.com/derrickstolee/git/compare/trace2-stopwatch...bitmap-trace2\n    \n    As a separate discussion, it might be worth revisiting that stopwatch\n    work so we have it available as a tool when doing these kinds of\n    investigations.\n    \n    Thanks, -Stolee\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1365%2Fderrickstolee%2Fbitmap-remove-trace2-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1365/derrickstolee/bitmap-remove-trace2-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1365\n\n pack-bitmap.c | 2 --\n 1 file changed, 2 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9a208abc1fd..13457dd77e5 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -830,10 +830,8 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\tif (!bitmap_git->table_lookup)\n \t\t\treturn NULL;\n \n-\t\ttrace2_region_enter(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n \t\t/* NEEDSWORK: cache misses aren't recorded */\n \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n-\t\ttrace2_region_leave(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n \t\tif (!bitmap)\n \t\t\treturn NULL;\n \t\treturn lookup_stored_bitmap(bitmap);\n\nbase-commit: 1b3d6e17fe83eb6f79ffbac2f2c61bbf1eaef5f8\n-- \ngitgitgadget\n"},{"id":"463523","messageId":"xmqqpmfmuilk.fsf@gitster.g","threadId":"58502","inReplyTo":"pull.1365.git.1663938034607.gitgitgadget@gmail.com","subject":"Re: [PATCH] pack-bitmap: remove trace2 region from hot path","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-23T17:05:27Z","receivedAt":"2022-09-23T17:05:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n\n> -\t\ttrace2_region_enter(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n>  \t\t/* NEEDSWORK: cache misses aren't recorded */\n>  \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n> -\t\ttrace2_region_leave(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n>  \t\tif (!bitmap)\n>  \t\t\treturn NULL;\n>  \t\treturn lookup_stored_bitmap(bitmap);\n\nAs \"git blame\" is bad in finding a piece of code that no longer\nexists, it may make sense to leave a comment around here why we do\nnot have a trace2 region around this call, perhaps?\n\n+\t/* do not add trace2_region around here in the hot path */\n\tbitmap = lazy_bitmap_for_commit(...);\n\nor something?\n\nThanks.\n"},{"id":"463525","messageId":"xmqqedw2uhnj.fsf@gitster.g","threadId":"58502","inReplyTo":"xmqqpmfmuilk.fsf@gitster.g","subject":"Re: [PATCH] pack-bitmap: remove trace2 region from hot path","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-23T17:25:52Z","receivedAt":"2022-09-23T17:26:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> \"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n>\n>\n>> -\t\ttrace2_region_enter(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n>>  \t\t/* NEEDSWORK: cache misses aren't recorded */\n>>  \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n>> -\t\ttrace2_region_leave(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n>>  \t\tif (!bitmap)\n>>  \t\t\treturn NULL;\n>>  \t\treturn lookup_stored_bitmap(bitmap);\n>\n> As \"git blame\" is bad in finding a piece of code that no longer\n> exists, it may make sense to leave a comment around here why we do\n> not have a trace2 region around this call, perhaps?\n>\n> +\t/* do not add trace2_region around here in the hot path */\n> \tbitmap = lazy_bitmap_for_commit(...);\n>\n> or something?\n>\n> Thanks.\n\nAlso, we'd need to disable t5310.150 that expects to see\nreading_lookup_table label in the trace.\n\n"},{"id":"463526","messageId":"xmqq7d1uuh5f.fsf@gitster.g","threadId":"58502","inReplyTo":"pull.1365.git.1663938034607.gitgitgadget@gmail.com","subject":"Re: [PATCH] pack-bitmap: remove trace2 region from hot path","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-23T17:36:44Z","receivedAt":"2022-09-23T17:38:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n>     I noticed this while trying to backport the Abhradeep's lookup table\n>     work into GitHub's fork. Something went wrong in that process, causing\n>     this region to significantly slow down. It turns out that slow down does\n>     not reproduce on current 'master', which is good. I must have missed\n>     something while I was backporting.\n>     \n>     Regardless, the use of trace2_region_enter() here should be removed or\n>     replaced. For the sake of 2.38.0, this simple removal is safe enough.\n\nWell, not doing this removal is even safer, if we know that it is\nnot hurting in the -rc code.  It would be better than breaking our\ntests without knowing ;-)  I am not upset that the patch broke the\ntest, by the way.  Compared to things silently breaking without\nsurfacing, a loud breakage in tests is a much easier thing to\nhandle.\n\nMy test run with the attached just finished, so that's what I am\ngoing to queue tentatively on top.\n\nThanks.\n\n pack-bitmap.c           | 1 +\n t/t5310-pack-bitmaps.sh | 2 +-\n 2 files changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git c/pack-bitmap.c w/pack-bitmap.c\nindex 13457dd77e..a2bbbbd811 100644\n--- c/pack-bitmap.c\n+++ w/pack-bitmap.c\n@@ -830,6 +830,7 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\tif (!bitmap_git->table_lookup)\n \t\t\treturn NULL;\n \n+\t\t/* this is a fairly hot codepath - no trace2_region please */\n \t\t/* NEEDSWORK: cache misses aren't recorded */\n \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n \t\tif (!bitmap)\ndiff --git c/t/t5310-pack-bitmaps.sh w/t/t5310-pack-bitmaps.sh\nindex 7e50f8e765..2b1efc923b 100755\n--- c/t/t5310-pack-bitmaps.sh\n+++ w/t/t5310-pack-bitmaps.sh\n@@ -455,7 +455,7 @@ test_expect_success 'verify writing bitmap lookup table when enabled' '\n \tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace2\n '\n \n-test_expect_success 'lookup table is actually used to traverse objects' '\n+: test_expect_success 'lookup table is actually used to traverse objects' '\n \tgit repack -adb &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace3\" \\\n \t\tgit rev-list --use-bitmap-index --count --all &&\n"},{"id":"463527","messageId":"edf25c5d-06f4-0b91-6773-f4fe7705c2f8@github.com","threadId":"58502","inReplyTo":"xmqq7d1uuh5f.fsf@gitster.g","subject":"Re: [PATCH] pack-bitmap: remove trace2 region from hot path","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-23T18:07:46Z","receivedAt":"2022-09-23T18:09:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/23/2022 1:36 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n> \n>>     I noticed this while trying to backport the Abhradeep's lookup table\n>>     work into GitHub's fork. Something went wrong in that process, causing\n>>     this region to significantly slow down. It turns out that slow down does\n>>     not reproduce on current 'master', which is good. I must have missed\n>>     something while I was backporting.\n>>     \n>>     Regardless, the use of trace2_region_enter() here should be removed or\n>>     replaced. For the sake of 2.38.0, this simple removal is safe enough.\n> \n> Well, not doing this removal is even safer, if we know that it is\n> not hurting in the -rc code.  It would be better than breaking our\n> tests without knowing ;-)  I am not upset that the patch broke the\n> test, by the way.  Compared to things silently breaking without\n> surfacing, a loud breakage in tests is a much easier thing to\n> handle.\n> \n> My test run with the attached just finished, so that's what I am\n> going to queue tentatively on top.\n\nWhoops! I'm very sorry for not noticing that there was a test checking\nfor this trace. Your modification makes sense and I'll try to do a more\ncareful replacement of that test, if possible.\n\nThanks,\n-Stolee\n"},{"id":"463545","messageId":"Yy4GVT8VZ4dOyGDy@nand.local","threadId":"58502","inReplyTo":"xmqq7d1uuh5f.fsf@gitster.g","subject":"Re: [PATCH] pack-bitmap: remove trace2 region from hot path","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-09-23T19:17:41Z","receivedAt":"2022-09-23T19:17:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Sep 23, 2022 at 10:36:44AM -0700, Junio C Hamano wrote:\n> diff --git c/t/t5310-pack-bitmaps.sh w/t/t5310-pack-bitmaps.sh\n> index 7e50f8e765..2b1efc923b 100755\n> --- c/t/t5310-pack-bitmaps.sh\n> +++ w/t/t5310-pack-bitmaps.sh\n> @@ -455,7 +455,7 @@ test_expect_success 'verify writing bitmap lookup table when enabled' '\n>  \tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace2\n>  '\n>\n> -test_expect_success 'lookup table is actually used to traverse objects' '\n> +: test_expect_success 'lookup table is actually used to traverse objects' '\n>  \tgit repack -adb &&\n>  \tGIT_TRACE2_EVENT=\"$(pwd)/trace3\" \\\n>  \t\tgit rev-list --use-bitmap-index --count --all &&\n\nYeah, I would advocate for dropping this test entirely. That trace2\nmarker is the only indication that is so obvious that we're using the\nlookup table to read bitmaps.\n\nPerhaps the pack-bitmap helper could be taught to dump which extensions\nit sees, and then write a test based on that? Although that isn't really\ntesting anything new, either, since we sometimes *notice* the extension\nbut don't actually read anything based on it (e.g., when disabled).\n\nSo I'd be content just dropping this.\n\nThanks,\nTaylor\n"},{"id":"463623","messageId":"pull.1365.v2.git.1664198277250.gitgitgadget@gmail.com","threadId":"58502","inReplyTo":"pull.1365.git.1663938034607.gitgitgadget@gmail.com","subject":"[PATCH v2] pack-bitmap: remove trace2 region from hot path","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-26T13:17:57Z","receivedAt":"2022-09-26T14:52:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <derrickstolee@github.com>\n\nThe trace2 region around the call to lazy_bitmap_for_commit() in\nbitmap_for_commit() was added in 28cd730680d (pack-bitmap: prepare to\nread lookup table extension, 2022-08-14). While adding trace2 regions is\ntypically helpful for tracking performance, this method is called\npossibly thousands of times as a commit walk explores commit history\nlooking for a matching bitmap. When trace2 output is enabled, this\nregion is emitted many times and performance is throttled by that\noutput.\n\nFor now, remove these regions entirely.\n\nThis is a critical path, and it would be valuable to measure that the\ntime spent in bitmap_for_commit() does not increase when using the\ncommit lookup table. The best way to do that would be to use a mechanism\nthat sums the time spent in a region and reports a single value at the\nend of the process. This technique was introduced but not merged by [1]\nso maybe this example presents some justification to revisit that\napproach.\n\n[1] https://lore.kernel.org/git/pull.1099.v2.git.1640720202.gitgitgadget@gmail.com/\n\nTo help with the 'git blame' output in this region, add a comment that\nwarns against adding a trace2 region. Delete a test from t5310 that used\nthat trace output to check that this lookup optimization was activated.\nTo create this kind of test again in the future, the stopwatch traces\nmentioned earlier could be used as a signal that we activated this code\npath.\n\nHelpedy-by: Junio C Hamano <gitster@pobox.com>\nSigned-off-by: Derrick Stolee <derrickstolee@github.com>\n---\n    [For 2.38.0] pack-bitmap: remove trace2 region from hot path\n    \n    I noticed this while trying to backport the Abhradeep's lookup table\n    work into GitHub's fork. Something went wrong in that process, causing\n    this region to significantly slow down. It turns out that slow down does\n    not reproduce on current 'master', which is good. I must have missed\n    something while I was backporting.\n    \n    Regardless, the use of trace2_region_enter() here should be removed or\n    replaced. For the sake of 2.38.0, this simple removal is safe enough.\n    However, to really dig into what was happening I had to construct a\n    rebase [2] of Jeff's trace2 stopwatch work, then apply changes on top\n    [3] that could replace this region with trace2_timer_*() methods.\n    \n    [2]\n    https://github.com/git/git/compare/master...derrickstolee:trace2-stopwatch\n    [3]\n    https://github.com/derrickstolee/git/compare/trace2-stopwatch...bitmap-trace2\n    \n    As a separate discussion, it might be worth revisiting that stopwatch\n    work so we have it available as a tool when doing these kinds of\n    investigations.\n    \n    \n    Updates in v2\n    =============\n    \n     * I removed the test that was checking for trace2 regions. In [3], I\n       played around with using trace2_counter() to replace that check, so\n       we could reinstate the test without the performance drop from using\n       repeated trace2_region() calls.\n     * I squashed in Junio's comment change.\n    \n    Thanks, -Stolee\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1365%2Fderrickstolee%2Fbitmap-remove-trace2-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1365/derrickstolee/bitmap-remove-trace2-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1365\n\nRange-diff vs v1:\n\n 1:  987827eaf90 ! 1:  8ef63c9e32f pack-bitmap: remove trace2 region from hot path\n     @@ Commit message\n      \n          [1] https://lore.kernel.org/git/pull.1099.v2.git.1640720202.gitgitgadget@gmail.com/\n      \n     +    To help with the 'git blame' output in this region, add a comment that\n     +    warns against adding a trace2 region. Delete a test from t5310 that used\n     +    that trace output to check that this lookup optimization was activated.\n     +    To create this kind of test again in the future, the stopwatch traces\n     +    mentioned earlier could be used as a signal that we activated this code\n     +    path.\n     +\n     +    Helpedy-by: Junio C Hamano <gitster@pobox.com>\n          Signed-off-by: Derrick Stolee <derrickstolee@github.com>\n      \n       ## pack-bitmap.c ##\n     @@ pack-bitmap.c: struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap\n       \t\t\treturn NULL;\n       \n      -\t\ttrace2_region_enter(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n     ++\t\t/* this is a fairly hot codepath - no trace2_region please */\n       \t\t/* NEEDSWORK: cache misses aren't recorded */\n       \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n      -\t\ttrace2_region_leave(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n       \t\tif (!bitmap)\n       \t\t\treturn NULL;\n       \t\treturn lookup_stored_bitmap(bitmap);\n     +\n     + ## t/t5310-pack-bitmaps.sh ##\n     +@@ t/t5310-pack-bitmaps.sh: test_expect_success 'verify writing bitmap lookup table when enabled' '\n     + \tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace2\n     + '\n     + \n     +-test_expect_success 'lookup table is actually used to traverse objects' '\n     +-\tgit repack -adb &&\n     +-\tGIT_TRACE2_EVENT=\"$(pwd)/trace3\" \\\n     +-\t\tgit rev-list --use-bitmap-index --count --all &&\n     +-\tgrep \"\\\"label\\\":\\\"reading_lookup_table\\\"\" trace3\n     +-'\n     +-\n     + test_expect_success 'truncated bitmap fails gracefully (lookup table)' '\n     + \ttest_config pack.writebitmaphashcache false &&\n     + \tgit repack -adb &&\n\n\n pack-bitmap.c           | 3 +--\n t/t5310-pack-bitmaps.sh | 7 -------\n 2 files changed, 1 insertion(+), 9 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9a208abc1fd..a2bbbbd811d 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -830,10 +830,9 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n \t\tif (!bitmap_git->table_lookup)\n \t\t\treturn NULL;\n \n-\t\ttrace2_region_enter(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n+\t\t/* this is a fairly hot codepath - no trace2_region please */\n \t\t/* NEEDSWORK: cache misses aren't recorded */\n \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n-\t\ttrace2_region_leave(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n \t\tif (!bitmap)\n \t\t\treturn NULL;\n \t\treturn lookup_stored_bitmap(bitmap);\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 7e50f8e7653..6d693eef82f 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -455,13 +455,6 @@ test_expect_success 'verify writing bitmap lookup table when enabled' '\n \tgrep \"\\\"label\\\":\\\"writing_lookup_table\\\"\" trace2\n '\n \n-test_expect_success 'lookup table is actually used to traverse objects' '\n-\tgit repack -adb &&\n-\tGIT_TRACE2_EVENT=\"$(pwd)/trace3\" \\\n-\t\tgit rev-list --use-bitmap-index --count --all &&\n-\tgrep \"\\\"label\\\":\\\"reading_lookup_table\\\"\" trace3\n-'\n-\n test_expect_success 'truncated bitmap fails gracefully (lookup table)' '\n \ttest_config pack.writebitmaphashcache false &&\n \tgit repack -adb &&\n\nbase-commit: 1b3d6e17fe83eb6f79ffbac2f2c61bbf1eaef5f8\n-- \ngitgitgadget\n"},{"id":"463632","messageId":"220926.868rm618ds.gmgdl@evledraar.gmail.com","threadId":"58502","inReplyTo":"pull.1365.v2.git.1664198277250.gitgitgadget@gmail.com","subject":"Re: [PATCH v2] pack-bitmap: remove trace2 region from hot path","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-09-26T15:01:24Z","receivedAt":"2022-09-26T16:19:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Sep 26 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee <derrickstolee@github.com>\n>\n> The trace2 region around the call to lazy_bitmap_for_commit() in\n> bitmap_for_commit() was added in 28cd730680d (pack-bitmap: prepare to\n> read lookup table extension, 2022-08-14). While adding trace2 regions is\n> typically helpful for tracking performance, this method is called\n> possibly thousands of times as a commit walk explores commit history\n> looking for a matching bitmap. When trace2 output is enabled, this\n> region is emitted many times and performance is throttled by that\n> output.\n>\n> For now, remove these regions entirely.\n>\n> This is a critical path, and it would be valuable to measure that the\n> time spent in bitmap_for_commit() does not increase when using the\n> commit lookup table. The best way to do that would be to use a mechanism\n> that sums the time spent in a region and reports a single value at the\n> end of the process. This technique was introduced but not merged by [1]\n> so maybe this example presents some justification to revisit that\n> approach.\n\nJust getting rid of this seems like a good thing for now.\n\nBut aside: Yes, one way to mitigate this rather than removing the\ntracing would be to make it really fast.\n\nBut just skimming pack-bitmap.c do we really need trace2 at the\ngranularity of a single commit? Looking at who calls bitmap_for_commit()\nwouldn't something like this sketch-out be much more useful?:\n\t\n\tdiff --git a/pack-bitmap.c b/pack-bitmap.c\n\tindex 9d5205055a5..439aec220c7 100644\n\t--- a/pack-bitmap.c\n\t+++ b/pack-bitmap.c\n\t@@ -830,10 +830,8 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git,\n\t \t\tif (!bitmap_git->table_lookup)\n\t \t\t\treturn NULL;\n\t \n\t-\t\ttrace2_region_enter(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n\t \t\t/* NEEDSWORK: cache misses aren't recorded */\n\t \t\tbitmap = lazy_bitmap_for_commit(bitmap_git, commit);\n\t-\t\ttrace2_region_leave(\"pack-bitmap\", \"reading_lookup_table\", the_repository);\n\t \t\tif (!bitmap)\n\t \t\t\treturn NULL;\n\t \t\treturn lookup_stored_bitmap(bitmap);\n\t@@ -1042,6 +1040,7 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n\t \t * The ones without bitmaps in the index will be stored in the\n\t \t * `not_mapped_list` for further processing.\n\t \t */\n\t+\t/* begin trace2 find roots? */\n\t \twhile (roots) {\n\t \t\tstruct object *object = roots->item;\n\t \t\troots = roots->next;\n\t@@ -1055,6 +1054,7 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n\t \t\tobject_list_insert(object, &not_mapped);\n\t \t}\n\t \n\t+\t/* end trace2 find roots? */\n\t \t/*\n\t \t * Best case scenario: We found bitmaps for all the roots,\n\t \t * so the resulting `or` bitmap has the full reachability analysis\n\t@@ -1100,7 +1100,7 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n\t \t\tincdata.base = base;\n\t \t\tincdata.seen = seen;\n\t \n\t-\t\trevs->include_check = should_include;\n\t+\t\trevs->include_check = should_include; // Will call bitmap_for_commit()\n\t \t\trevs->include_check_obj = should_include_obj;\n\t \t\trevs->include_check_data = &incdata;\n\t \n\t@@ -1110,9 +1110,11 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n\t \t\tshow_data.bitmap_git = bitmap_git;\n\t \t\tshow_data.base = base;\n\t \n\t+\t\t/* begin trace2 rev list? */\n\t \t\ttraverse_commit_list(revs,\n\t \t\t\t\t     show_commit, show_object,\n\t \t\t\t\t     &show_data);\n\t+\t\t/* end trace2 rev list? */\n\t \n\t \t\trevs->include_check = NULL;\n\t \t\trevs->include_check_obj = NULL;\n\t\n\nThis is *not* the same as stricking the tracing into\nbitmap_for_commit(), but do we really need the tracing that far inside\nour loop?\n\n"},{"id":"463649","messageId":"e1aab606-fdde-e10f-ed00-2af67bbfec27@github.com","threadId":"58502","inReplyTo":"220926.868rm618ds.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2] pack-bitmap: remove trace2 region from hot path","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-26T17:31:47Z","receivedAt":"2022-09-26T17:55:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/26/2022 11:01 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Sep 26 2022, Derrick Stolee via GitGitGadget wrote:\n> \n>> This is a critical path, and it would be valuable to measure that the\n>> time spent in bitmap_for_commit() does not increase when using the\n>> commit lookup table. The best way to do that would be to use a mechanism\n>> that sums the time spent in a region and reports a single value at the\n>> end of the process. This technique was introduced but not merged by [1]\n>> so maybe this example presents some justification to revisit that\n>> approach.\n> \n> Just getting rid of this seems like a good thing for now.\n> \n> But aside: Yes, one way to mitigate this rather than removing the\n> tracing would be to make it really fast.\n> \n> But just skimming pack-bitmap.c do we really need trace2 at the\n> granularity of a single commit? Looking at who calls bitmap_for_commit()\n> wouldn't something like this sketch-out be much more useful?:\n\nThe point of it being where it was to check that we hit the path\ncustom to the commit lookup extension, hence the test that is removed.\n\nYou are proposing a different region entirely, meant to measure the\ncost of the entire walk. Maybe that's valuable, but not critical to\nthis topic.\n\nIf you're proposing the high-level traces as an alternative to my\nprototype using the trace2 timers, then I'll point out that the timer\napproach allows us to determine how much time is being spent checking\nfor bitmaps versus walking commits, which the regions you provided don't.\n\nThanks,\n-Stolee\n"},{"id":"463749","messageId":"4c4954d8-82ea-8674-542e-865c514a7ae5@jeffhostetler.com","threadId":"58502","inReplyTo":"pull.1365.git.1663938034607.gitgitgadget@gmail.com","subject":"Re: [PATCH] pack-bitmap: remove trace2 region from hot path","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-09-27T18:37:19Z","receivedAt":"2022-09-27T18:37:26Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 9/23/22 9:00 AM, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <derrickstolee@github.com>\n> \n> The trace2 region around the call to lazy_bitmap_for_commit() in\n> bitmap_for_commit() was added in 28cd730680d (pack-bitmap: prepare to\n> read lookup table extension, 2022-08-14). While adding trace2 regions is\n> typically helpful for tracking performance, this method is called\n> possibly thousands of times as a commit walk explores commit history\n> looking for a matching bitmap. When trace2 output is enabled, this\n> region is emitted many times and performance is throttled by that\n> output.\n> \n> For now, remove these regions entirely.\n> \n> This is a critical path, and it would be valuable to measure that the\n> time spent in bitmap_for_commit() does not increase when using the\n> commit lookup table. The best way to do that would be to use a mechanism\n> that sums the time spent in a region and reports a single value at the\n> end of the process. This technique was introduced but not merged by [1]\n> so maybe this example presents some justification to revisit that\n> approach.\n> \n> [1] https://lore.kernel.org/git/pull.1099.v2.git.1640720202.gitgitgadget@gmail.com/\n\nI'll dust off my \"timers and counters\" series, address the\nvarious issues that were raised back in December, and resubmit.\n\nThanks for the reminder.\nJeff\n\n"}]}