{"thread":{"id":"66503","subject":"[PATCH] SubmittingPatches: avoid add-failing-test-then-fix anti-pattern","startedAt":"2026-10-11T15:32:06Z","lastAt":"2026-10-12T14:34:33Z","messageCount":6,"participants":["Jon Simons","Kristoffer Haugsbakk","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"554716","messageId":"20261011153206.72386-1-jon@jonsimons.org","threadId":"66503","inReplyTo":null,"subject":"[PATCH] SubmittingPatches: avoid add-failing-test-then-fix anti-pattern","fromName":"Jon Simons","fromEmail":"jon@jonsimons.org","sentAt":"2026-10-11T15:32:06Z","receivedAt":"2026-10-11T15:32:06Z","isPatch":true,"sender":{"key":"jon@jonsimons.org","avatar":"https://avatars.githubusercontent.com/u/118440?v=4"},"body":"Document the preference to avoid the 'add a failing test and fix it in a\nsubsequent commit' anti-pattern.  Although such a sequence is fine for\nlocal development, reviewers instead prefer a single commit that\nintroduces the test together with its code change.\n\nThe goal of this addition is to capture the preference in writing, with\nhopes of optimizing future review flows by reducing back-and-forths to\nthe list.\n\nSigned-off-by: Jon Simons <jon@jonsimons.org>\n---\n Documentation/SubmittingPatches | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/Documentation/SubmittingPatches b/Documentation/SubmittingPatches\nindex c60855f706..27b53d2b31 100644\n--- a/Documentation/SubmittingPatches\n+++ b/Documentation/SubmittingPatches\n@@ -271,6 +271,16 @@ fixed by accident to avoid regression.  Also, try merging your work to\n that are still in flight may have unexpected interactions with what\n you are trying to do in your topic.\n \n+When fixing a bug, you may find it useful to first write a test using\n+`test_expect_failure` to confirm that the test exposes the bug.\n+However, unless there is a strong reason not to, submit the test as\n+`test_expect_success` in the same commit as your code change that\n+fixes the bug.  A separate commit that only changes `test_expect_failure`\n+to `test_expect_success` can leave the behavior being tested out of the\n+diff context, making the fix harder to review.  Reviewers who want to\n+verify that your test reproduces the bug can temporarily undo the code\n+changes while keeping the test.\n+\n Pushing to a fork of https://github.com/git/git will use their CI\n integration to test your changes on Linux, Mac and Windows. See the\n <<GHCI,GitHub CI>> section for details.\n-- \n2.55.0\n\n\n"},{"id":"554719","messageId":"b0d0ba54-8a46-4c1a-8d64-8b2ac07b051f@app.fastmail.com","threadId":"66503","inReplyTo":"20261011153206.72386-1-jon@jonsimons.org","subject":"Re: [PATCH] SubmittingPatches: avoid add-failing-test-then-fix anti-pattern","fromName":"Kristoffer Haugsbakk","fromEmail":"kristofferhaugsbakk@fastmail.com","sentAt":"2026-10-11T16:21:55Z","receivedAt":"2026-10-11T16:21:55Z","isPatch":true,"sender":{"key":"kristofferhaugsbakk@fastmail.com","avatar":null},"body":"On Sun, Oct 11, 2026, at 17:32, Jon Simons wrote:\n> Document the preference to avoid the 'add a failing test and fix it in a\n> subsequent commit' anti-pattern.  Although such a sequence is fine for\n> local development, reviewers instead prefer a single commit that\n> introduces the test together with its code change.\n\nI have skimmed this document and indeed there does not seem to be an\nexisting mention of this point. And this is a useful point to mention.\n\nI think the commit message should link to a few examples of this being\nstated. Like one of the ones you seemed to have found in your previous\nmessage.\n\n>\n> The goal of this addition is to capture the preference in writing, with\n> hopes of optimizing future review flows by reducing back-and-forths to\n> the list.\n\nSure, that’s always the case for additions to this file. ;)\n\n>\n> Signed-off-by: Jon Simons <jon@jonsimons.org>\n> ---\n>  Documentation/SubmittingPatches | 10 ++++++++++\n>  1 file changed, 10 insertions(+)\n>\n> diff --git a/Documentation/SubmittingPatches\n> b/Documentation/SubmittingPatches\n> index c60855f706..27b53d2b31 100644\n> --- a/Documentation/SubmittingPatches\n> +++ b/Documentation/SubmittingPatches\n> @@ -271,6 +271,16 @@ fixed by accident to avoid regression.  Also, try\n> merging your work to\n>  that are still in flight may have unexpected interactions with what\n>  you are trying to do in your topic.\n>\n> +When fixing a bug, you may find it useful to first write a test using\n> +`test_expect_failure` to confirm that the test exposes the bug.\n> +However, unless there is a strong reason not to, submit the test as\n> +`test_expect_success` in the same commit as your code change that\n> +fixes the bug.  A separate commit that only changes `test_expect_failure`\n> +to `test_expect_success` can leave the behavior being tested out of the\n> +diff context, making the fix harder to review.  Reviewers who want to\n> +verify that your test reproduces the bug can temporarily undo the code\n> +changes while keeping the test.\n\nThis is well-written.\n\nTis comes under `[[tests]]`, so it is appropriately placed. It also\ncomes after `[[separate-commits]]`, and it sort of functions as a\ncounterweight to making *too many* separate commits.\n\n> +\n>  Pushing to a fork of https://github.com/git/git will use their CI\n>  integration to test your changes on Linux, Mac and Windows. See the\n>  <<GHCI,GitHub CI>> section for details.\n> --\n> 2.55.0\n\n"},{"id":"554730","messageId":"xmqq5wz7dcpx.fsf@gitster.g","threadId":"66503","inReplyTo":"20261011153206.72386-1-jon@jonsimons.org","subject":"Re: [PATCH] SubmittingPatches: avoid add-failing-test-then-fix anti-pattern","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-10-11T23:44:26Z","receivedAt":"2026-10-11T23:44:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jon Simons <jon@jonsimons.org> writes:\n\n> +When fixing a bug, you may find it useful to first write a test using\n> +`test_expect_failure` to confirm that the test exposes the bug.\n> +However, unless there is a strong reason not to, submit the test as\n> +`test_expect_success` in the same commit as your code change that\n> +fixes the bug.  A separate commit that only changes `test_expect_failure`\n> +to `test_expect_success` can leave the behavior being tested out of the\n> +diff context, making the fix harder to review.  Reviewers who want to\n> +verify that your test reproduces the bug can temporarily undo the code\n> +changes while keeping the test.\n\nThis indeed matches what contributors have been told to do.\n\nBut do we even need to say the first two and a half lines?  One\nmight find it convenient not to write any tests, or to be satisfied\nwith manual testing, or to send test_expect_failure() separately\nand early.  None of these is what we recommend.\n\nAs you described, adding a test_expect_failure() test with a large\nbody alone in one patch, and fixing the code in another patch would\nshow a change from test_expect_failure() to test_expect_success()\nwithout showing much of the body of the test in the second patch,\nmaking the review unnecessarily cumbersome.  If the test is trivial\nand small, it may fit within the diff context and avoid such a\nproblem, but then there is no need to add such a small test as a\nfailing one in a separate patch in the first place, unless the goal\nis to inflate the commit count.\n\nSo, unless we can think of concrete cases that qualify as a\n\"strong reason\", I do not think we want to write everything before\nand up to \"a strong reason not to\" in your text.\n\nHow about a short and sweet sentence like this instead?\n\n    A bugfix patch should include tests in test_expect_success()\n    blocks that demonstrate the breakage when the code change that\n    implements the fix is omitted.\n\n\n"},{"id":"554737","messageId":"4978013d-36c4-4441-a93c-68f1f35c06d0@jonsimons.org","threadId":"66503","inReplyTo":"xmqq5wz7dcpx.fsf@gitster.g","subject":"Re: [PATCH] SubmittingPatches: avoid add-failing-test-then-fix anti-pattern","fromName":"Jon Simons","fromEmail":"jon@jonsimons.org","sentAt":"2026-10-12T01:46:48Z","receivedAt":"2026-10-12T01:46:48Z","isPatch":true,"sender":{"key":"jon@jonsimons.org","avatar":"https://avatars.githubusercontent.com/u/118440?v=4"},"body":"On 10/11/26 7:44 PM, Junio C Hamano wrote:\n> Jon Simons <jon@jonsimons.org> writes:\n> \n>> +When fixing a bug, you may find it useful to first write a test using\n>> +`test_expect_failure` to confirm that the test exposes the bug.\n>> +However, unless there is a strong reason not to, submit the test as\n>> +`test_expect_success` in the same commit as your code change that\n>> +fixes the bug.  A separate commit that only changes `test_expect_failure`\n>> +to `test_expect_success` can leave the behavior being tested out of the\n>> +diff context, making the fix harder to review.  Reviewers who want to\n>> +verify that your test reproduces the bug can temporarily undo the code\n>> +changes while keeping the test.\n> \n> This indeed matches what contributors have been told to do.\n> \n> But do we even need to say the first two and a half lines?  One\n> might find it convenient not to write any tests, or to be satisfied\n> with manual testing, or to send test_expect_failure() separately\n> and early.  None of these is what we recommend.\n> \n> As you described, adding a test_expect_failure() test with a large\n> body alone in one patch, and fixing the code in another patch would\n> show a change from test_expect_failure() to test_expect_success()\n> without showing much of the body of the test in the second patch,\n> making the review unnecessarily cumbersome.  If the test is trivial\n> and small, it may fit within the diff context and avoid such a\n> problem, but then there is no need to add such a small test as a\n> failing one in a separate patch in the first place, unless the goal\n> is to inflate the commit count.\n> \n> So, unless we can think of concrete cases that qualify as a\n> \"strong reason\", I do not think we want to write everything before\n> and up to \"a strong reason not to\" in your text.\n\nAck, I think this makes sense.  I do not have a list of concrete cases\nthat would fit a \"strong reason\" and be applicable in the general sense.\n\n> How about a short and sweet sentence like this instead?\n> \n>     A bugfix patch should include tests in test_expect_success()\n>     blocks that demonstrate the breakage when the code change that\n>     implements the fix is omitted.\n\nMuch crisper, I will apply this to a v2.\n\n\nThanks,\n-Jon\n\n"},{"id":"554739","messageId":"20261012030029.12200-1-jon@jonsimons.org","threadId":"66503","inReplyTo":"20261011153206.72386-1-jon@jonsimons.org","subject":"[PATCH v2] SubmittingPatches: avoid add-failing-test-then-fix anti-pattern","fromName":"Jon Simons","fromEmail":"jon@jonsimons.org","sentAt":"2026-10-12T03:00:29Z","receivedAt":"2026-10-12T03:00:29Z","isPatch":true,"sender":{"key":"jon@jonsimons.org","avatar":"https://avatars.githubusercontent.com/u/118440?v=4"},"body":"Document the preference to avoid the 'add a failing test and fix it in\na subsequent commit' anti-pattern.  Although such a sequence is fine\nfor local development, reviewers instead prefer a single commit that\nintroduces the test together with its code change [1].\n\nThe goal of this addition is to capture the preference in writing, with\nhopes of optimizing future review flows by reducing back-and-forths to\nthe list.\n\n[1] A couple examples of feedback, with rationale for avoiding\n    the pattern, from August 2025 here:\n\n    - https://lore.kernel.org/git/xmqqqzxpzo8b.fsf@gitster.g/\n    - https://lore.kernel.org/git/xmqqfrdk3aqy.fsf@gitster.g/\n\nSigned-off-by: Jon Simons <jon@jonsimons.org>\n---\n Documentation/SubmittingPatches | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/Documentation/SubmittingPatches b/Documentation/SubmittingPatches\nindex c60855f706..3238a33932 100644\n--- a/Documentation/SubmittingPatches\n+++ b/Documentation/SubmittingPatches\n@@ -271,6 +271,15 @@ fixed by accident to avoid regression.  Also, try merging your work to\n that are still in flight may have unexpected interactions with what\n you are trying to do in your topic.\n \n+A bugfix patch should include tests in test_expect_success() blocks that\n+demonstrate the breakage when the code change that implements the fix is\n+omitted.  Submit tests as `test_expect_success` in the same commit as\n+your code change that fixes the bug.  A separate commit that only changes\n+`test_expect_failure` to `test_expect_success` can leave the behavior\n+being tested out of the diff context, making the fix harder to review.\n+Reviewers who want to verify that your test reproduces the bug can\n+temporarily undo the code changes while keeping the test.\n+\n Pushing to a fork of https://github.com/git/git will use their CI\n integration to test your changes on Linux, Mac and Windows. See the\n <<GHCI,GitHub CI>> section for details.\n-- \n2.55.0\n\n\n"},{"id":"554759","messageId":"xmqq4ierasxy.fsf@gitster.g","threadId":"66503","inReplyTo":"20261012030029.12200-1-jon@jonsimons.org","subject":"Re: [PATCH v2] SubmittingPatches: avoid add-failing-test-then-fix anti-pattern","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-10-12T14:34:33Z","receivedAt":"2026-10-12T14:34:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jon Simons <jon@jonsimons.org> writes:\n\n> +A bugfix patch should include tests in test_expect_success() blocks that\n> +demonstrate the breakage when the code change that implements the fix is\n> +omitted.  Submit tests as `test_expect_success` in the same commit as\n> +your code change that fixes the bug.  A separate commit that only changes\n> +`test_expect_failure` to `test_expect_success` can leave the behavior\n> +being tested out of the diff context, making the fix harder to review.\n> +Reviewers who want to verify that your test reproduces the bug can\n> +temporarily undo the code changes while keeping the test.\n> +\n>  Pushing to a fork of https://github.com/git/git will use their CI\n>  integration to test your changes on Linux, Mac and Windows. See the\n>  <<GHCI,GitHub CI>> section for details.\n\nDoesn't the second sentence \"Submit tests ...\" merely repeat what\nthe first sentence already says without adding much value?  After\nre-reading them three times, I think \"the same commit\" is the only\nnew point.  I would condense the first two sentences down to\n\n    A bugfix patch should add, in the same commit, code change and\n    test_expect_success() blocks that demonstrate the breakage when\n    the fix is omitted.\n\nor something, if I were writing this from scratch.\n\nBy the way, I consider that the SubmittingPatches document is\nprimarily about giving readers a set of rules they should follow,\nand not about teaching the reasoning behind a rule to such a deep\nlevel, unless the rule requires judgment on when or when not to\napply it.  We agreed we do not need \"unless there is a strong reason\nnot to\", which means there is no judgment needed, so I was not\nexpecting to see the latter half of the paragraph.\n\nHaving said all that, i will queue this version as-is because we are\nquickly reaching the point of diminishing returns.  Thanks.\n\n"}]}