{"thread":{"id":"44824","subject":"[ANNOUNCE] git-test: run automated tests against a range of Git commits","startedAt":"2017-01-06T15:52:27Z","lastAt":"2017-01-12T05:04:14Z","messageCount":5,"participants":["Michael Haggerty","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"308857","messageId":"1341c01a-aca7-699c-c53a-28d048614bfe@alum.mit.edu","threadId":"44824","inReplyTo":null,"subject":"[ANNOUNCE] git-test: run automated tests against a range of Git commits","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2017-01-06T15:52:16Z","receivedAt":"2017-01-06T15:52:27Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"I just released ⁠⁠⁠⁠`git test⁠⁠⁠⁠`, a script for running automated\ntests across a range of Git commits and keeping track of the results in\ngit notes:\n\n    https://github.com/mhagger/git-test\n\nThis is a script that I've been using in one form or another for years\nand I find it really handy [1].\n\n`git-test` is meant for people who want *all* of their commits (not just\nthe branch tip) to pass their automated tests.\n\nTl;dr: How to use `git test`\n\n1.  Define the test you want to run. The string can be any shell\n    command:\n\n        git test add \"make -j8 && make -j16 test\"\n\n2.  Create a separate linked worktree in which to run your tests:\n\n        git worktree add --detach ../test HEAD\n\n3.  Create a terminal window and `cd` to the directory containing\n    the testing worktree, and run the test against all of the commits\n    on your branch:\n\n        cd ../test\n        git test range master..mybranch\n\n    If any of the commits are broken, `git tree` will display the\n    error then stop with that commit checked out.\n\n4.  As you work, whenever you want to test new commits, go to the\n    testing terminal window and run the same command again:\n\n        git test range master..mybranch\n\n    `git test` is smart enough to remember which commits (actually,\n    trees) have already been tested and only run the test against\n    commits that have been changed or added. And since the tests are\n    run in a different worktree, you can continue working in your\n    main working directory while the tests run.\n\nIt is also possible to define more than one test suite in a given\nrepository, retry tests, etc. Type `git test help` or read the `README`\nfile [2] for more information.\n\n`git test` stores the test results in git notes (under\n`refs/notes/test/<name>`), linked to the commit's tree SHA-1. This means\nthat test results remain valid even across some kinds of commit\nrewriting, like changes to commit metadata or squashing adjacent\ncommits, and a subset of results even remains valid if a commit is split\nor if some commits earlier in a patch series are reordered.\n\nI don't plan to turn this into a gigantic project or anything, but I\nfind this script really useful so I wanted to put it out in the world.\nFeedback and/or pull requests welcome!\n\nMichael\n\n[1] The name sucks, I know :-/\n[2] https://github.com/mhagger/git-test/blob/master/README.md\n\n"},{"id":"308918","messageId":"20170107071832.2rucap3rskzmkgq4@sigill.intra.peff.net","threadId":"44824","inReplyTo":"1341c01a-aca7-699c-c53a-28d048614bfe@alum.mit.edu","subject":"Re: [ANNOUNCE] git-test: run automated tests against a range of Git commits","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-01-07T07:18:32Z","receivedAt":"2017-01-07T07:18:40Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 06, 2017 at 04:52:16PM +0100, Michael Haggerty wrote:\n\n> I just released ⁠⁠⁠⁠`git test⁠⁠⁠⁠`, a script for running automated\n> tests across a range of Git commits and keeping track of the results in\n> git notes:\n> \n>     https://github.com/mhagger/git-test\n> \n> This is a script that I've been using in one form or another for years\n> and I find it really handy [1].\n\nNeat. I usually \"git rebase -x 'make -j8 test' @{u}\" after finishing a\ntopic to make sure the intermediate steps are good. But it would be neat\nto have this running continuously in the background to alert me to\nproblems sooner (and the key thing there is that it remembers\nalready-run tests, so it should be safe to basically for new commits\nevery 10 seconds or so).\n\nI did hit a few interesting cases trying out \"git test\". So here's a\nnarrative, and you can pick out where there may be room for improvement\nin the tool, and where I'm just being dumb. :)\n\nI tried it out first on a topic I finished earlier today, which has 3\ncommits. So I did:\n\n  $ git test add 'make -j8 test'\n  $ git test range @{u}..HEAD\n\nIt barfed on the first commit, because the script expects \"git co\" to\nwork, but I don't have that alias. No big deal (and I already submitted\na PR to fix it).\n\nSo then I reinvoked it like:\n\n  $ git test range @{u}..HEAD\n\nand it actually ran some tests. Yay.\n\nAnd then of course I wanted to prove to myself how cool the notes\nfeature is, so I ran it again. It didn't run any tests this time. Yay\nagain. But there were a few surprises:\n\n  $ git test range @{u}..HEAD\n  setup_test default\n  Using test default; command: make -j8 test\n  Old status: bad\n  Tree 9fcdbd5c78^{tree} is already known to be bad!\n  Old status: good\n  Tree c22f4f6624^{tree} is already known to be good.\n  Old status: good\n  Tree 19e2e62e5e^{tree} is already known to be good.\n  Already on 'jk/wait-for-child-cleanup'\n  Your branch is ahead of 'origin/master' by 3 commits.\n\n  ALL TESTS SUCCESSFUL\n\nMy initial run with \"git co\" had left the first commit marked as \"bad\".\nThat's not _too_ surprising, since it did indeed fail. I think it's\nprobably a bug to record a failure note, though, if checking out fails.\nIt's not necessarily an immutable property of the tree. In my case, it\nwas obviously dependent on a change in the git-test code, but that's\nhopefully a fairly uncommon occurrence. But there are other reasons for\ngit-checkout to complain. For instance, imagine topic \"foo\" creates file\n\"foo.c\". If I do:\n\n  $ echo content >foo.c\n  $ git test foo@{u}..foo\n\nthen checkout will complain about overwriting the untracked \"foo.c\".\nIt's reasonable to abort the operation there, but probably not to write\na permanent failure note. The problem wasn't in \"foo\", but in my local\ntree.\n\nThe second thing that surprised me was \"ALL TESTS SUCCESSFUL\", when\nclearly one of them was known-bad. :)\n\nSo at this point I knew I needed to re-run the test. Looks like there's\na \"--force\" option. Let's try it. There's no need to re-run the other\ntwo, so let's just give it one commit:\n\n  $ git test range -f HEAD~2\n  ...\n  Object 95649d6cf9ec68f05d1dc57ec1b989b8d263a7ae^{tree} has no note\n  Object e1970ce43abfbf625bce68516857e910748e5965^{tree} has no note\n  Object 368f99d57e8ed17243f2e164431449d48bfca2fb^{tree} has no note\n  Object ceede59ea90cebad52ba9c8263fef3fb6ef17593^{tree} has no note\n  Object dfe070511c652f2b8e1bf6540f238c9ca9ba41d3^{tree} has no note\n  Object 902d960b382a0cd424618ff4e1316da40e4be2f6^{tree} has no note\n  ...\n\nThis started spewing out many lines like the one above, until I hit ^C.\nYikes!\n\nThinking something was wrong with the \"-f\" option, I tried it without:\n\n  $ git test range -f HEAD~2\n  ...\n  commit e83c5163316f89bfbde7d9ab23ca2e25604af290 (origin/initial)\n  Author: Linus Torvalds <torvalds@ppc970.osdl.org>\n  Date:   Thu Apr 7 15:13:13 2005 -0700\n\n      Initial revision of \"git\", the information manager from hell\n\nOops. Now I see the problem. I was expecting the arguments to be\nrebase-like. But they're really rev-list like. So in both cases it\nwanted to test all the way up from the root commits to HEAD~2.  The \"-f\"\nversion never got to testing a commit because it was so busy trying to\ndelete the old notes.\n\nI see the symmetry and simplicity in allowing the user to specify a full\nrange. But it also seems like it's easy to make a mistake that's going\nto want to test a lot of commits. I wonder if it should complain when\nthere's no lower bound to the commit range. Or alternatively, if there's\na single positive reference, treat it as a lower bound, with HEAD as the\nupper bound (which is vaguely rebase-like).\n\nA few other observations about the note deletion:\n\n  - The \"has no note\" message should perhaps be suppressed. We're just\n    trying to overwrite the value if there is one (alternatively,\n    instead of removing it, just overwrite it, so the old note stays\n    until we get a result one way or the other).\n\n  - It was sufficiently slow that it looks like we invoke \"git notes\n    remove\" once per commit. It would be a lot more efficient to batch\n    them (not just in terms of process startup, but because you're going\n    to write a _ton_ of intermediate notes trees).\n\n    Of course none of that matters if you don't do something stupid like\n    trying to \"git test\" 45,000 commits. :)\n\nSo OK. Now I know what's going on. And I can get what I want with:\n\n  $ git test range -f HEAD~3..HEAD~2\n\nwhich works, and now all of my tests are correctly marked. Of course\nthat's a lot to type. It would be easier as:\n\n  $ git test range -f -1 HEAD~2\n  usage: git-test [-h] {add,range,help} ...\n  git-test: error: unrecognized arguments: HEAD~2\n\nbut the tool doesn't seem to like passing through the rev-list argument.\n\nIt would be even easier if I could just repeat my range and only re-test\nthe \"bad\" commits. It was then that I decided to actually read the rest\nof \"git test help range\" and see that you already wrote such an option,\ncleverly hidden under the name \"--retest\".\n\nAnd one final nit. I notice there is also a \"--keep-going\" option. Which\nmade me surprised that we bothered to test HEAD~1 and HEAD, when we knew\nthat HEAD~2 was bogus. I suspect this is related to the \"ALL TESTS\nSUCCESSFUL\" issue. Presumably cached test results are not treated as\n\"bad\" in general, which seems funny to me.\n\nSo those were all little cosmetic things. The other big thing I wanted\nto see was what it's like to fix a bug deep in a topic. So I used \"git\nrebase -i\" to inset a compile error into the first commit of my 3-patch\nseries. And then I tested it:\n\n  $ git test add -t compile 'make -j8'\n  $ git test range -t compile HEAD~3..\n\nAs predicted, it stopped at the first commit and told me it was buggy.\nBut I'm dumped onto a detached HEAD, and I'm on my own to actually get\nthe working tree to a state where I can test and fix on my actual\nbranch.\n\nThat's a nice outcome of the \"git rebase -x make\" approach. When you hit\na bug, you can fix it, \"git commit --amend\", and \"git rebase\n--continue\". The downside is that it takes ownership of the branch while\nyou're doing it. So the whole concept of \"run this in the background\nwith a separate worktree\" breaks down completely.\n\nI think it should be possible to script the next steps, though.\nSomething like like \"git test fix foo\", which would:\n\n  - expand the range of foo@{u}..foo to get the list of commits\n\n  - see which ones were marked as broken\n\n  - kick off an interactive rebase, but override GIT_EDITOR to mark any\n    broken ones as \"edit\" instead of \"pick\"\n\nThat lets you separate the act of testing from the act of fixing. You\ncan let the tester run continuously in the background, and only stop to\nfix when you're at an appropriate point in your work.\n\nAnyway. Seems like a neat tool. I may play around with it and see if I\ncan fit it into my workflow.\n\n-Peff\n"},{"id":"308959","messageId":"ce6f98a4-1fb7-aa4b-2efb-78d8f49397a7@alum.mit.edu","threadId":"44824","inReplyTo":"20170107071832.2rucap3rskzmkgq4@sigill.intra.peff.net","subject":"Re: [ANNOUNCE] git-test: run automated tests against a range of Git commits","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2017-01-08T09:52:25Z","receivedAt":"2017-01-08T09:56:16Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 01/07/2017 08:18 AM, Jeff King wrote:\n> On Fri, Jan 06, 2017 at 04:52:16PM +0100, Michael Haggerty wrote:\n> \n>> I just released ⁠⁠⁠⁠`git test⁠⁠⁠⁠`, a script for running automated\n>> tests across a range of Git commits and keeping track of the results in\n>> git notes:\n>>\n>>     https://github.com/mhagger/git-test\n>>\n>> This is a script that I've been using in one form or another for years\n>> and I find it really handy [1].\n> \n> Neat. I usually \"git rebase -x 'make -j8 test' @{u}\" after finishing a\n> topic to make sure the intermediate steps are good. But it would be neat\n> to have this running continuously in the background to alert me to\n> problems sooner (and the key thing there is that it remembers\n> already-run tests, so it should be safe to basically for new commits\n> every 10 seconds or so).\n> \n> I did hit a few interesting cases trying out \"git test\". So here's a\n> narrative, and you can pick out where there may be room for improvement\n> in the tool, and where I'm just being dumb. :)\n> \n> I tried it out first on a topic I finished earlier today, which has 3\n> commits. So I did:\n> \n>   $ git test add 'make -j8 test'\n>   $ git test range @{u}..HEAD\n> \n> It barfed on the first commit, because the script expects \"git co\" to\n> work, but I don't have that alias. No big deal (and I already submitted\n> a PR to fix it).\n\nI make the same mistake in most of my scripts :-/ Thanks for the PR; I\nmerged it.\n\n> So then I reinvoked it like:\n> \n>   $ git test range @{u}..HEAD\n> \n> and it actually ran some tests. Yay.\n> \n> And then of course I wanted to prove to myself how cool the notes\n> feature is, so I ran it again. It didn't run any tests this time. Yay\n> again. But there were a few surprises:\n> \n>   $ git test range @{u}..HEAD\n>   setup_test default\n>   Using test default; command: make -j8 test\n>   Old status: bad\n>   Tree 9fcdbd5c78^{tree} is already known to be bad!\n>   Old status: good\n>   Tree c22f4f6624^{tree} is already known to be good.\n>   Old status: good\n>   Tree 19e2e62e5e^{tree} is already known to be good.\n>   Already on 'jk/wait-for-child-cleanup'\n>   Your branch is ahead of 'origin/master' by 3 commits.\n> \n>   ALL TESTS SUCCESSFUL\n>\n> My initial run with \"git co\" had left the first commit marked as \"bad\".\n> That's not _too_ surprising, since it did indeed fail. I think it's\n> probably a bug to record a failure note, though, if checking out fails.\n> It's not necessarily an immutable property of the tree.\n\nI definitely agree. This was an oversight which I just fixed.\n\n> [...]\n> \n> The second thing that surprised me was \"ALL TESTS SUCCESSFUL\", when\n> clearly one of them was known-bad. :)\n\nReplayed results weren't being treated internally as failures. That's\nfixed, too.\n\n> So at this point I knew I needed to re-run the test. Looks like there's\n> a \"--force\" option. Let's try it. There's no need to re-run the other\n> two, so let's just give it one commit:\n> \n>   $ git test range -f HEAD~2\n>   ...\n>   Object 95649d6cf9ec68f05d1dc57ec1b989b8d263a7ae^{tree} has no note\n>   Object e1970ce43abfbf625bce68516857e910748e5965^{tree} has no note\n>   Object 368f99d57e8ed17243f2e164431449d48bfca2fb^{tree} has no note\n>   Object ceede59ea90cebad52ba9c8263fef3fb6ef17593^{tree} has no note\n>   Object dfe070511c652f2b8e1bf6540f238c9ca9ba41d3^{tree} has no note\n>   Object 902d960b382a0cd424618ff4e1316da40e4be2f6^{tree} has no note\n>   ...\n> \n> This started spewing out many lines like the one above, until I hit ^C.\n> Yikes!\n> \n> [...]\n> I see the symmetry and simplicity in allowing the user to specify a full\n> range. But it also seems like it's easy to make a mistake that's going\n> to want to test a lot of commits. I wonder if it should complain when\n> there's no lower bound to the commit range. Or alternatively, if there's\n> a single positive reference, treat it as a lower bound, with HEAD as the\n> upper bound (which is vaguely rebase-like).\n\nI see how this might be unexpected, and it's definitely inconvenient at\nsome times (like when you want to test a single commit). I thought it\nwould be nice to allow arbitrary `rev-list` expressions (albeit\ncurrently only a single word), but I think that you are right that other\nsemantics would be more convenient.\n\nI'm thinking of maybe\n\n* If an argument matches `*..*`, pass it to `rev-list` (like now).\n\n* Otherwise, treat each argument as a single commit/tree (i.e., pass it\nto `rev-parse`).\n\n* If no argument is specified, test `@{u}..` (assuming that an\n  upstream is configured). Though actually, this won't be as\n  convenient as it sounds, because (a) `git test` is often run\n  in a separate worktree, and (2) on errors, it currently leaves the\n  repository with a detached `HEAD`.\n\n* Support a `--stdin` option, to read a list of commits/trees to test\n  from standard input. By this mechanism, users could use arbitrary\n  `rev-list` commands to choose what to test.\n\n> A few other observations about the note deletion:\n> \n>   - The \"has no note\" message should perhaps be suppressed. We're just\n>     trying to overwrite the value if there is one (alternatively,\n>     instead of removing it, just overwrite it, so the old note stays\n>     until we get a result one way or the other).\n\nYes. That's a one-time problem that I haven't seen in a long time. The\nscript is overly chatty in general.\n\nAside: It would be nice if `git notes` had a subcommand to initialize a\nnote reference with an empty tree. (I know how to do it longhand, but\nit's awkward and it should be possible to do it via the `notes` interface.)\n\nI think ideally `git notes add` would look for pre-existing notes, and:\n\n* If none are found, create an empty notes reference.\n\n* If pre-existing notes are found and there was no existing test with\n  that name, probably just leave the old notes in place.\n\n* If pre-existing notes are found and there was already a test with\n  that name but a different command, perhaps insist that the user\n  decide explicitly whether to forget the old results or continue using\n  them. This might help users avoid the mistake of re-using old results\n  even if they change the manner of testing.\n\n>   - It was sufficiently slow that it looks like we invoke \"git notes\n>     remove\" once per commit. It would be a lot more efficient to batch\n>     them (not just in terms of process startup, but because you're going\n>     to write a _ton_ of intermediate notes trees).\n> \n>     Of course none of that matters if you don't do something stupid like\n>     trying to \"git test\" 45,000 commits. :)\n\nYeah, I've never experienced that problem myself :-P But I see that\nnotes supports `git notes remove --ignore-missing --stdin`, so that will\nbe easy to implement.\n\n> [...]\n> It would be even easier if I could just repeat my range and only re-test\n> the \"bad\" commits. It was then that I decided to actually read the rest\n> of \"git test help range\" and see that you already wrote such an option,\n> cleverly hidden under the name \"--retest\".\n\nI think you were being ironic, but if not, would this have been easier\nto find under another name?\n\n> And one final nit. I notice there is also a \"--keep-going\" option. Which\n> made me surprised that we bothered to test HEAD~1 and HEAD, when we knew\n> that HEAD~2 was bogus. I suspect this is related to the \"ALL TESTS\n> SUCCESSFUL\" issue.\n\nYes, that's part of the same bug from above.\n\n> So those were all little cosmetic things. The other big thing I wanted\n> to see was what it's like to fix a bug deep in a topic. So I used \"git\n> rebase -i\" to inset a compile error into the first commit of my 3-patch\n> series. And then I tested it:\n> \n>   $ git test add -t compile 'make -j8'\n>   $ git test range -t compile HEAD~3..\n> \n> As predicted, it stopped at the first commit and told me it was buggy.\n> But I'm dumped onto a detached HEAD, and I'm on my own to actually get\n> the working tree to a state where I can test and fix on my actual\n> branch.\n\nYeah, this is awkward, not only because many people don't know what to\nmake of detached HEAD, but also because it makes it awkward in general\nto use `git test` in your main working directory. I didn't model this\nbehavior on `git rebase --interactive`'s `edit` command, because I\nrarely use that. But I can see how they would fit together pretty well\nfor people who like that workflow.\n\nI've considered that rather than leave you in a detached HEAD state,\nmaybe `git test` should always restore your old branch. But it seems\nlike it would more often be useful to be in the directory with the\nbroken commit checked out and any test results, coredumps, etc intact.\n\nI would definitely like to implement a `git test reset` command that\nreturns you to your initial branch (like `git bisect reset`).\n\nI like your idea of a `git test fix` command:\n\n> [...]\n> I think it should be possible to script the next steps, though.\n> Something like like \"git test fix foo\", which would:\n> \n>   - expand the range of foo@{u}..foo to get the list of commits\n> \n>   - see which ones were marked as broken\n> \n>   - kick off an interactive rebase, but override GIT_EDITOR to mark any\n>     broken ones as \"edit\" instead of \"pick\"\n> \n> That lets you separate the act of testing from the act of fixing. You\n> can let the tester run continuously in the background, and only stop to\n> fix when you're at an appropriate point in your work.\n\nI think you would usually only want to mark only the *first* broken\ncommit as \"edit\", because often errors cascade to descendant commits.\n\nThanks for all the great feedback!\n\nMichael\n\n"},{"id":"309132","messageId":"20170110093502.wbfgof55gdo6mtov@sigill.intra.peff.net","threadId":"44824","inReplyTo":"ce6f98a4-1fb7-aa4b-2efb-78d8f49397a7@alum.mit.edu","subject":"Re: [ANNOUNCE] git-test: run automated tests against a range of Git commits","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-01-10T09:35:02Z","receivedAt":"2017-01-10T09:35:13Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jan 08, 2017 at 10:52:25AM +0100, Michael Haggerty wrote:\n\n> > I see the symmetry and simplicity in allowing the user to specify a full\n> > range. But it also seems like it's easy to make a mistake that's going\n> > to want to test a lot of commits. I wonder if it should complain when\n> > there's no lower bound to the commit range. Or alternatively, if there's\n> > a single positive reference, treat it as a lower bound, with HEAD as the\n> > upper bound (which is vaguely rebase-like).\n> \n> I see how this might be unexpected, and it's definitely inconvenient at\n> some times (like when you want to test a single commit). I thought it\n> would be nice to allow arbitrary `rev-list` expressions (albeit\n> currently only a single word), but I think that you are right that other\n> semantics would be more convenient.\n> \n> I'm thinking of maybe\n> \n> * If an argument matches `*..*`, pass it to `rev-list` (like now).\n> \n> * Otherwise, treat each argument as a single commit/tree (i.e., pass it\n> to `rev-parse`).\n> \n> * If no argument is specified, test `@{u}..` (assuming that an\n>   upstream is configured). Though actually, this won't be as\n>   convenient as it sounds, because (a) `git test` is often run\n>   in a separate worktree, and (2) on errors, it currently leaves the\n>   repository with a detached `HEAD`.\n> \n> * Support a `--stdin` option, to read a list of commits/trees to test\n>   from standard input. By this mechanism, users could use arbitrary\n>   `rev-list` commands to choose what to test.\n\nThat seems quite reasonable to me. The rebase semantics of \"a single\nargument is my upstream\" is convenient, but I think it also confuses\npeople. Yours seem very simple to explain.\n\nIt doesn't allow complex stuff like \"git test ^foo ^bar HEAD\", but I\ndon't think \"git rev-list ^foo ^bar HEAD | git test --stdin\" is really\nso bad for complicated cases like that (if anybody would even ever want\nsuch a thing).\n\n> Aside: It would be nice if `git notes` had a subcommand to initialize a\n> note reference with an empty tree. (I know how to do it longhand, but\n> it's awkward and it should be possible to do it via the `notes` interface.)\n> \n> I think ideally `git notes add` would look for pre-existing notes, and:\n> \n> * If none are found, create an empty notes reference.\n> \n> * If pre-existing notes are found and there was no existing test with\n>   that name, probably just leave the old notes in place.\n> \n> * If pre-existing notes are found and there was already a test with\n>   that name but a different command, perhaps insist that the user\n>   decide explicitly whether to forget the old results or continue using\n>   them. This might help users avoid the mistake of re-using old results\n>   even if they change the manner of testing.\n\nI'm not quite sure what you mean here. By \"test\" and \"command\", do you\nmean the test name that is used in the notes ref, and the command that\nit is defined as?\n\nIn the notes-cache.c subsystem, the commit message stores a validity\ntoken which must match in order to use the cache. You could do something\nsimilar here (store the executed command in the commit message, and\ninvalidate the cache the user has changed the command). The notes-cache\nstuff isn't available outside of the C code, though. You could either\nexpose it, or just do something similar longhand.\n\nThinking about it, though, I did notice that the tree sha1 is not the\nonly input to the cache. Things like config.mak (not to mention the\nsystem itself) contribute to the test results. So no system will ever be\nperfect, and it seems like just making an easy way to force a retest (or\njust invalidate the whole cache) would be sufficient.\n\nBut maybe I totally missed what you were trying to say here.\n\n> > It would be even easier if I could just repeat my range and only re-test\n> > the \"bad\" commits. It was then that I decided to actually read the rest\n> > of \"git test help range\" and see that you already wrote such an option,\n> > cleverly hidden under the name \"--retest\".\n> \n> I think you were being ironic, but if not, would this have been easier\n> to find under another name?\n\nSorry, the knob on my sarcasm module must have accidentally been knocked\ndown from \"so thick it's obvious even by email\".\n\nYes, I did just mean that I was being blind. You not only had added the\noption already, but gave it the exact name I was about to propose it\nunder. :)\n\n> Yeah, this is awkward, not only because many people don't know what to\n> make of detached HEAD, but also because it makes it awkward in general\n> to use `git test` in your main working directory. I didn't model this\n> behavior on `git rebase --interactive`'s `edit` command, because I\n> rarely use that. But I can see how they would fit together pretty well\n> for people who like that workflow.\n\nYeah, after sleeping on it, I think it's best if \"git test\" remains\nseparate from that. It's primary function is to run the test, possibly\nserving up a cached answer. So it would be perfectly reasonable to do:\n\n  git rebase -x 'git test range HEAD'\n\nto accomplish the interactive testing (though perhaps just \"git test\"\nwould be a nice synonym for that).\n\nAnd then \"jump to a thing that I know is broken\" becomes a separate\naction, whether you are using \"git test\" or not. I wonder if we could\nhave:\n\n  git rebase -e HEAD~2\n\nto do an interactive rebase, with \"edit\" for HEAD~2. I feel like\nsomebody even proposed that at one point, but I don't think it got\nmerged.\n\nAnd then \"git test fix\" basically becomes:\n\n  git rebase -e \"$(git test first-broken)\"\n\nthough I think you'd still want a shorthand for that.\n\n> > I think it should be possible to script the next steps, though.\n> > Something like like \"git test fix foo\", which would:\n> > \n> >   - expand the range of foo@{u}..foo to get the list of commits\n> > \n> >   - see which ones were marked as broken\n> > \n> >   - kick off an interactive rebase, but override GIT_EDITOR to mark any\n> >     broken ones as \"edit\" instead of \"pick\"\n> > \n> > That lets you separate the act of testing from the act of fixing. You\n> > can let the tester run continuously in the background, and only stop to\n> > fix when you're at an appropriate point in your work.\n> \n> I think you would usually only want to mark only the *first* broken\n> commit as \"edit\", because often errors cascade to descendant commits.\n\nYeah, I had a similar thought. OTOH, if you didn't say \"--keep-going\",\nyou'd only have one breakage either way. I doubt it matters that much in\npractice, and either behavior would probably be fine until somebody\ncomes up with a good use for the multi-commit case.\n\n-Peff\n"},{"id":"309248","messageId":"51fff12e-8bac-41fb-c93a-df504ae25942@alum.mit.edu","threadId":"44824","inReplyTo":"20170110093502.wbfgof55gdo6mtov@sigill.intra.peff.net","subject":"Re: [ANNOUNCE] git-test: run automated tests against a range of Git commits","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2017-01-12T05:02:53Z","receivedAt":"2017-01-12T05:04:14Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 01/10/2017 10:35 AM, Jeff King wrote:\n> On Sun, Jan 08, 2017 at 10:52:25AM +0100, Michael Haggerty wrote:\n>> [...]\n\nSince my last email, I have implemented a bunch of what we discussed\n[1]. Because of the new semantics, I also *renamed the main command* from\n\n    git test range [...]\n\nto\n\n    git test run [...]\n\n>> I'm thinking of maybe\n>>\n>> * If an argument matches `*..*`, pass it to `rev-list` (like now).\n\nThis was already implemented.\n\n>> * Otherwise, treat each argument as a single commit/tree (i.e., pass it\n>> to `rev-parse`).\n\nThis is now implemented, too. Plus, it is now allowed to specify\nmultiple commits and/or ranges in a single invocation.\n\n>> * If no argument is specified, test `@{u}..` (assuming that an\n>>   upstream is configured). Though actually, this won't be as\n>>   convenient as it sounds, because (a) `git test` is often run\n>>   in a separate worktree, and (2) on errors, it currently leaves the\n>>   repository with a detached `HEAD`.\n\nI decided that if no argument is specified, it makes more sense to test\nHEAD if the working tree is clean, and the contents of the working tree\notherwise. In the latter case no results are stored to notes, but it is\nstill useful to be able to run a preconfigured test quickly while\nworking. These are both implemented.\n\n>> * Support a `--stdin` option, to read a list of commits/trees to test\n>>   from standard input. By this mechanism, users could use arbitrary\n>>   `rev-list` commands to choose what to test.\n\nThis is now implemented, too.\n\n> [...]\n>> I think ideally `git notes add` would look for pre-existing notes, and:\n>>\n>> * If none are found, create an empty notes reference.\n>>\n>> * If pre-existing notes are found and there was no existing test with\n>>   that name, probably just leave the old notes in place.\n>>\n>> * If pre-existing notes are found and there was already a test with\n>>   that name but a different command, perhaps insist that the user\n>>   decide explicitly whether to forget the old results or continue using\n>>   them. This might help users avoid the mistake of re-using old results\n>>   even if they change the manner of testing.\n> \n> I'm not quite sure what you mean here. By \"test\" and \"command\", do you\n> mean the test name that is used in the notes ref, and the command that\n> it is defined as?\n\nBy \"test\", I mean a test as configured in `git-test` and referred to by\nits name (e.g., in `--test=name`). Currently the only configuration for\na test is `test.<name>.command`, but I structured the namespace that way\nto leave room to add more configuration in the future.\n\n> In the notes-cache.c subsystem, the commit message stores a validity\n> token which must match in order to use the cache. You could do something\n> similar here (store the executed command in the commit message, and\n> invalidate the cache the user has changed the command). The notes-cache\n> stuff isn't available outside of the C code, though. You could either\n> expose it, or just do something similar longhand.\n> \n> Thinking about it, though, I did notice that the tree sha1 is not the\n> only input to the cache. Things like config.mak (not to mention the\n> system itself) contribute to the test results. So no system will ever be\n> perfect, and it seems like just making an easy way to force a retest (or\n> just invalidate the whole cache) would be sufficient.\n\nIt's true that the tree and the test command are not the only inputs to\nthe testing machinery. I wasn't hoping to build a watertight system to\nprevent accidental use of old results when `config.mak` or something\nelse in the environment has changed. I was only trying to give the user\na reminder that if they change the command, it's a good time to consider\nwhether the old results have to be invalidated.\n\nI suppose if we wanted to make this system more watertight, we could let\nthe test configuration specify the names of other files on which its\nresults depend; for example,\n\n    test.default.command = make -j16 test\n    test.default.auxiliaryInput = config.mak\n\nand include some kind of hash of the auxiliary inputs in a validity\ntoken. But that feels like overkill.\n\n> [...]\n>> Yeah, this is awkward, not only because many people don't know what to\n>> make of detached HEAD, but also because it makes it awkward in general\n>> to use `git test` in your main working directory. I didn't model this\n>> behavior on `git rebase --interactive`'s `edit` command, because I\n>> rarely use that. But I can see how they would fit together pretty well\n>> for people who like that workflow.\n> \n> Yeah, after sleeping on it, I think it's best if \"git test\" remains\n> separate from that. It's primary function is to run the test, possibly\n> serving up a cached answer. So it would be perfectly reasonable to do:\n> \n>   git rebase -x 'git test range HEAD'\n> \n> to accomplish the interactive testing (though perhaps just \"git test\"\n> would be a nice synonym for that).\n\nThat invocation can now be written `git test run`, which is a bit\nshorter. There could be a special case that `git test` is shorthand for\n`git test run`, but it quickly gets ambiguous if the user wants to start\nadding any `run` arguments.\n\n> And then \"jump to a thing that I know is broken\" becomes a separate\n> action, whether you are using \"git test\" or not. I wonder if we could\n> have:\n> \n>   git rebase -e HEAD~2\n> \n> to do an interactive rebase, with \"edit\" for HEAD~2. I feel like\n> somebody even proposed that at one point, but I don't think it got\n> merged.\n\nThat sounds handy. (The GIT_EDITOR thing is pretty hacky.)\n\n> And then \"git test fix\" basically becomes:\n> \n>   git rebase -e \"$(git test first-broken)\"\n> \n> though I think you'd still want a shorthand for that.\n\nThere could be a `git test fix --rebase <range>` that does this. Plus my\npreference, `git test fix --branch=fix <range>`, which creates a branch\nnamed `fix` at the first broken commit and checks it out. (If the fix is\nnontrivial, my next step is usually to `git imerge rebase` the original\nbranch onto the `fix` branch.)\n\n>>> I think it should be possible to script the next steps, though.\n>>> Something like like \"git test fix foo\", which would: [...]\n>>\n>> I think you would usually only want to mark only the *first* broken\n>> commit as \"edit\", because often errors cascade to descendant commits.\n> \n> Yeah, I had a similar thought. OTOH, if you didn't say \"--keep-going\",\n> you'd only have one breakage either way. [...]\n\nTrue enough.\n\nThanks for all your feedback!\n\nMichael\n\n[1] https://github.com/mhagger/git-test\n\n"}]}