{"thread":{"id":"45100","subject":"Continuous Testing of Git on Windows","startedAt":"2017-02-10T12:33:25Z","lastAt":"2017-02-18T11:49:32Z","messageCount":19,"participants":["Johannes Schindelin","Junio C Hamano","Christian Couder","Philip Oakley"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"311264","messageId":"alpine.DEB.2.20.1702101241210.3496@virtualbox","threadId":"45100","inReplyTo":null,"subject":"Continuous Testing of Git on Windows","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-02-10T12:24:20Z","receivedAt":"2017-02-10T12:33:25Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi team,\n\nat the GitMerge conference, I heard a couple of times \"I do not bother\nreporting bugs in `pu` or `next` on MacOSX and Windows anymore, only Linux\nseems to be truly supported\" or some variation thereof.\n\nThis is a strong indicator that we have some room for improvement here.\n\nActive readers of the Git mailing list will not be surprised to read that\nI think we have to react to build/test failures quicker, that it is not\nenough to declare it okay for those integration branches to fail the test\nsuite or even to fail to build[*1*].\n\nIn that vein, I worked quite a bit on \"Continuous Integration\", or more\nappropriately: Continuous Testing. That is, I created an ensemble of jobs\nthat build & test the four integration tests of upstream Git (`maint`,\n`master`, `next` and `pu`) to verify the *basic* validity of their\nrespective current revisions.\n\nMost CI integrations these days require custom configuration files to be\ncommitted, and certain knobs to be twisted on GitHub (which I cannot turn\nbecause I have no special privileges on git/git). After struggling with\nmaking it work *somehow* anyway (even trying to get in touch with Travis,\nbut they have not bothered to reply to my requests in over a year...), I\ndecided to go with the Visual Studio Team Services (or short, VSTS; it\ndoes come in handy that it is developed by distant colleagues of mine, so\nthey *have* to reply to my requests) where the CI configuration can be\nseparated from the source code.\n\nThe entire setup is a little bit more complex than your grandfather's CI\nsetup: it has to orchestrate five separate Git repositories, two of them\ngenerated and updated from live 32/64-bit Git for Windows SDKs, using a\ncustom pool of build agents due to high resource demands, using three\nseparate Git for Windows installations to support 32/64-bit as well as\nupdating git.exe via `git pull`[*2*].\n\nThere is currently only one downside to that setup: the ability to have\npublicly accessible build logs on VSTS is still in development.\n\nThis is not *such* a big downside: if the MacOSX/Linux CI based on\nTravis[*3*] is any indicator, few people, if any, give a flying,\nfornicating fly about public build logs.\n\nHowever, we should strive to improve our software development practices,\nand one such well-known Best Practice is to use Continuous Testing more\neffectively, i.e. *not* to ignore it.\n\nThat is why I taught the Git for Windows CI job that tests the four\nupstream Git integration branches to *also* bisect test breakages and then\nupload comments to the identified commit on GitHub. See an example here\n(the identified breakage seems to have disappeared in the meantime):\n\nhttps://github.com/git/git/commit/5a12b3d76973#commitcomment-20802488\n\nThe code that generates this comment can be seen here:\n\nhttps://github.com/git-for-windows/build-extra/blob/50c392c7d107/please.sh#L1648-L1665\n\nSo here is hoping to a quicker turnaround from breakage to fix in the\nfuture!\n\nCiao,\nJohannes\n\nP.S.: I realize that these commit comments may *still* be ignored, but I\nsimply was not yet ready to annoy everybody by having automated mails sent\nout.\n\nFootnote *1*: It would be kind of okay if, say, `pu` would simply pick up\n*all* patches so that they would not be forgotten. But that is not the\ncase. Even worse: it was stated recently that the expectation is that the\n*submitters* of patches find bugs in their code, that the patches should\nessentially be bug-free by the time they were submitted. This reasoning\nfalls flat on its face considering the very real failures, of course,\ndemonstrating our dear need for Continuous Testing.\n\nFootnote *2*: I will describe the entire setup, including links to the\ninvolved repositories, in a separate mail at a later stage.\n\nFootnote *3*: Look at https://travis-ci.org/git/git/builds, and be happy\nif you have a red/green deficiency so you cannot see all that red.\n"},{"id":"311475","messageId":"xmqq60kdbqmy.fsf@gitster.mtv.corp.google.com","threadId":"45100","inReplyTo":"alpine.DEB.2.20.1702101241210.3496@virtualbox","subject":"Re: Continuous Testing of Git on Windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-13T23:46:45Z","receivedAt":"2017-02-13T23:46:57Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> That is why I taught the Git for Windows CI job that tests the four\n> upstream Git integration branches to *also* bisect test breakages and then\n> upload comments to the identified commit on GitHub\n\nGood. I do not think it is useful to try 'pu' as an aggregate and\nexpect it to always build and work [*1*], but your \"bisect and\npinpoint\" approach makes it useful to identify individual topic that\nbrings in a breakage. I wouldn't be surprised if original submitter\nand I were the only two people who actually compiled the patches on\na topic in isolation while a topic is in 'pu', and chances are that\nthese two people didn't try their builds on Windows. A CI like this\none will help the coverage to stop premature topics from advancing\nto 'pu' without getting any Windows exposure.\n\nThanks.\n\n\n[Footnote]\n\n*1* The reason why topics not in 'next' but in 'pu', especially the\n ones merged near the tip of 'pu', exist in 'pu' are because they\n are interesting enough and could be polished to become eligible\n for 'next' but known to be premature for 'next' yet. They are\n there primarily to give human contributors an easier way to\n download them as a whole and help polish them. And I have to be\n selective when I queue things on 'pu'; it is not like I have\n infinite amount of time to pick up any cruft that is sent to the\n list.\n"},{"id":"311578","messageId":"alpine.DEB.2.20.1702142150220.3496@virtualbox","threadId":"45100","inReplyTo":"xmqq60kdbqmy.fsf@gitster.mtv.corp.google.com","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-02-14T20:55:28Z","receivedAt":"2017-02-14T20:55:43Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 13 Feb 2017, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n> \n> > That is why I taught the Git for Windows CI job that tests the four\n> > upstream Git integration branches to *also* bisect test breakages and\n> > then upload comments to the identified commit on GitHub\n> \n> Good. I do not think it is useful to try 'pu' as an aggregate and\n> expect it to always build and work [*1*], but your \"bisect and\n> pinpoint\" approach makes it useful to identify individual topic that\n> brings in a breakage.\n\nSadly the many different merge bases[*1*] between `next` and `pu` (which\nare the obvious good/bad points for bisecting automatically) bring my\nbuild agents to its knees. I may have to disable the bisecting feature as\na consequence.\n\nCiao,\nJohannes\n\nFootnote *1*: There are currently 21, some of which stemming back from a\nyear ago. For bisecting, they all have to be tested individually, putting\na major dent into bisect's otherwise speedy process.\n"},{"id":"311579","messageId":"xmqqd1ek8oqo.fsf@gitster.mtv.corp.google.com","threadId":"45100","inReplyTo":"alpine.DEB.2.20.1702142150220.3496@virtualbox","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-14T21:08:15Z","receivedAt":"2017-02-14T21:08:22Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> On Mon, 13 Feb 2017, Junio C Hamano wrote:\n>\n>> Johannes Schindelin writes:\n>> \n>> > That is why I taught the Git for Windows CI job that tests the four\n>> > upstream Git integration branches to *also* bisect test breakages and\n>> > then upload comments to the identified commit on GitHub\n>> \n>> Good. I do not think it is useful to try 'pu' as an aggregate and\n>> expect it to always build and work [*1*], but your \"bisect and\n>> pinpoint\" approach makes it useful to identify individual topic that\n>> brings in a breakage.\n>\n> Sadly the many different merge bases[*1*] between `next` and `pu` (which\n> are the obvious good/bad points for bisecting automatically) bring my\n> build agents to its knees. I may have to disable the bisecting feature as\n> a consequence.\n\nProbably a less resource intensive approach is to find the tips of\nthe topics not in 'next' but in 'pu' and test them. That would give\nyou which topic(s) are problematic, which is a better starting point\nthan \"Oh, 'pu' does not build\". After identifying which branch is\nproblematic, bisection of individual topic would be of more manageable\nsize.\n\n $ git log --first-parent --oneline 'pu^{/^### match next}..pu'\n\nwill you the merges of topics left outside 'next'. I often reorder\nto make the ones that look more OK than others closer to the bottom,\nand if the breakages caused by them are caught earlier than they hit\n'next', that would be ideal.\n\nThis is one of these times I wish \"git bisect --first-parent\" were\navailable. The above \"log\" gives me 27 merges right now, which\nshould be bisectable within 5 rounds to identify a single broken\ntopic (if there is only one breakage, that is).\n\n\n\n"},{"id":"311603","messageId":"CAP8UFD1+AgBVqSh=wHteM3uKO+55ZqqD4cHzBUfN0KTPXyvutQ@mail.gmail.com","threadId":"45100","inReplyTo":"xmqqd1ek8oqo.fsf@gitster.mtv.corp.google.com","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-14T23:00:56Z","receivedAt":"2017-02-14T23:01:05Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Feb 14, 2017 at 10:08 PM, Junio C Hamano wrote:\n> Johannes Schindelin writes:\n>\n>> On Mon, 13 Feb 2017, Junio C Hamano wrote:\n>>\n>>> Johannes Schindelin writes:\n>>>\n>>> > That is why I taught the Git for Windows CI job that tests the four\n>>> > upstream Git integration branches to *also* bisect test breakages and\n>>> > then upload comments to the identified commit on GitHub\n>>>\n>>> Good. I do not think it is useful to try 'pu' as an aggregate and\n>>> expect it to always build and work [*1*], but your \"bisect and\n>>> pinpoint\" approach makes it useful to identify individual topic that\n>>> brings in a breakage.\n>>\n>> Sadly the many different merge bases[*1*] between `next` and `pu` (which\n>> are the obvious good/bad points for bisecting automatically) bring my\n>> build agents to its knees. I may have to disable the bisecting feature as\n>> a consequence.\n\nYeah, this is a bug in the bisect algorithm. Fixing it is in the GSoC\n2017 Ideas.\n\n> Probably a less resource intensive approach is to find the tips of\n> the topics not in 'next' but in 'pu' and test them. That would give\n> you which topic(s) are problematic, which is a better starting point\n> than \"Oh, 'pu' does not build\". After identifying which branch is\n> problematic, bisection of individual topic would be of more manageable\n> size.\n\nIt is still probably more resource intensive than it couls be.\n\n[...]\n\n> This is one of these times I wish \"git bisect --first-parent\" were\n> available.\n\nImplementing \"git bisect --first-parent\" is also part of the GSoC 2017 Ideas.\n\nBy the way it should not be very difficult as a patch to do this and\nmore was proposed a long time ago:\n\nhttps://public-inbox.org/git/4D3CDDF9.6080405@intel.com/\n\n> The above \"log\" gives me 27 merges right now, which\n> should be bisectable within 5 rounds to identify a single broken\n> topic (if there is only one breakage, that is).\n"},{"id":"311604","messageId":"xmqq1sv074h7.fsf@gitster.mtv.corp.google.com","threadId":"45100","inReplyTo":"CAP8UFD1+AgBVqSh=wHteM3uKO+55ZqqD4cHzBUfN0KTPXyvutQ@mail.gmail.com","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-14T23:11:16Z","receivedAt":"2017-02-14T23:11:24Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> By the way it should not be very difficult as a patch to do this and\n> more was proposed a long time ago:\n>\n> https://public-inbox.org/git/4D3CDDF9.6080405@intel.com/\n\nThanks for a link. The one I found most interesting in the thread\nis by Avery [*1*], where he explains why \"first-parent\" bisection\nmakes sense in \"many people develop topics of their own, and they\nare aggregated into an integration branch\" environment:\n\n Basically, we push/fetch *all* the branches from *everybody* into a\n single repo, and build all of them as frequently as we can. If you\n think about it, if you have all the branches that someone might have\n pulled/merged from, then you don't have to think of the git history\n as a whole complicated DAG; you can just think of it as a whole\n bunch of separate chunks of linear history. Moreover, as long as\n people are careful to only pull from a branch when that branch is\n passing all tests - which you can easily see by looking at the\n gitbuilder console - then playing inside each of these chunks of\n linear history can help you figure out where particular bugs were\n introduced during \"messy\" branches.\n\n It also allows you a nice separation of concerns. The owner of the\n mainline branch (the \"integration manager\" person) only really cares\n about which branch they merged that caused a problem, because that\n person doesn't want to fix bugs, he/she simply wants to know who\n owns the failing branch, so that person can fix *their* bug and\n their branch will merge without breaking things.\n\n[Reference]\n\n*1* https://public-inbox.org/git/AANLkTinwbm9gcZhGeQCbOEPov0_xV7uJyQvC7J13qO15@mail.gmail.com/\n"},{"id":"311605","messageId":"E2C1B7A8FBF94C8CB1C9C5754D882800@PhilipOakley","threadId":"45100","inReplyTo":"CAP8UFD1+AgBVqSh=wHteM3uKO+55ZqqD4cHzBUfN0KTPXyvutQ@mail.gmail.com","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-02-14T23:27:07Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Christian Couder\" \n> On Tue, Feb 14, 2017 at 10:08 PM, Junio C Hamano \n> wrote:\n>> Johannes Schindelin writes:\n>>\n>>> On Mon, 13 Feb 2017, Junio C Hamano wrote:\n>>>\n>>>> Johannes Schindelin writes:\n>>>>\n>>>> > That is why I taught the Git for Windows CI job that tests the four\n>>>> > upstream Git integration branches to *also* bisect test breakages and\n>>>> > then upload comments to the identified commit on GitHub\n>>>>\n>>>> Good. I do not think it is useful to try 'pu' as an aggregate and\n>>>> expect it to always build and work [*1*], but your \"bisect and\n>>>> pinpoint\" approach makes it useful to identify individual topic that\n>>>> brings in a breakage.\n>>>\n>>> Sadly the many different merge bases[*1*] between `next` and `pu` (which\n>>> are the obvious good/bad points for bisecting automatically) bring my\n>>> build agents to its knees. I may have to disable the bisecting feature \n>>> as\n>>> a consequence.\n>\n> Yeah, this is a bug in the bisect algorithm. Fixing it is in the GSoC\n> 2017 Ideas.\n\nThere are also a few ideas at the SO answers: \nhttp://stackoverflow.com/a/5652323/717355\n\n>\n>> Probably a less resource intensive approach is to find the tips of\n>> the topics not in 'next' but in 'pu' and test them. That would give\n>> you which topic(s) are problematic, which is a better starting point\n>> than \"Oh, 'pu' does not build\". After identifying which branch is\n>> problematic, bisection of individual topic would be of more manageable\n>> size.\n>\n> It is still probably more resource intensive than it couls be.\n>\n> [...]\n>\n>> This is one of these times I wish \"git bisect --first-parent\" were\n>> available.\n>\n> Implementing \"git bisect --first-parent\" is also part of the GSoC 2017 \n> Ideas.\n>\n> By the way it should not be very difficult as a patch to do this and\n> more was proposed a long time ago:\n>\n> https://public-inbox.org/git/4D3CDDF9.6080405@intel.com/\n>\n>> The above \"log\" gives me 27 merges right now, which\n>> should be bisectable within 5 rounds to identify a single broken\n>> topic (if there is only one breakage, that is).\n>\n--\nPhilip \n\n"},{"id":"311607","messageId":"xmqqshng5osz.fsf@gitster.mtv.corp.google.com","threadId":"45100","inReplyTo":"E2C1B7A8FBF94C8CB1C9C5754D882800@PhilipOakley","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-14T23:35:08Z","receivedAt":"2017-02-14T23:35:15Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Philip Oakley\" writes:\n\n> There are also a few ideas at the SO answers:\n> http://stackoverflow.com/a/5652323/717355\n\nI vaguely recall that I saw somebody said the same \"mark tips of\ntopics as good\" on the list and answered with why it does not quite\nwork, though.\n"},{"id":"311648","messageId":"alpine.DEB.2.20.1702151506250.3496@virtualbox","threadId":"45100","inReplyTo":"CAP8UFD1+AgBVqSh=wHteM3uKO+55ZqqD4cHzBUfN0KTPXyvutQ@mail.gmail.com","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-02-15T14:07:06Z","receivedAt":"2017-02-15T14:07:58Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Christian,\n\nOn Wed, 15 Feb 2017, Christian Couder wrote:\n\n> On Tue, Feb 14, 2017 at 10:08 PM, Junio C Hamano \n> wrote:\n> \n> > Probably a less resource intensive approach is to find the tips of the\n> > topics not in 'next' but in 'pu' and test them. That would give you\n> > which topic(s) are problematic, which is a better starting point than\n> > \"Oh, 'pu' does not build\". After identifying which branch is\n> > problematic, bisection of individual topic would be of more manageable\n> > size.\n> \n> It is still probably more resource intensive than it couls be.\n\nIndeed.\n\nCiao,\nDscho\n"},{"id":"311649","messageId":"alpine.DEB.2.20.1702151509251.3496@virtualbox","threadId":"45100","inReplyTo":"E2C1B7A8FBF94C8CB1C9C5754D882800@PhilipOakley","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-02-15T14:22:42Z","receivedAt":"2017-02-15T14:23:07Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Philip,\n\nOn Tue, 14 Feb 2017, Philip Oakley wrote:\n\n> From: \"Christian Couder\" \n> > On Tue, Feb 14, 2017 at 10:08 PM, Junio C Hamano \n> > wrote:\n> > > Johannes Schindelin writes:\n> > >\n> > > > On Mon, 13 Feb 2017, Junio C Hamano wrote:\n> > > >\n> > > > > Johannes Schindelin writes:\n> > > > >\n> > > > > > That is why I taught the Git for Windows CI job that tests the\n> > > > > > four upstream Git integration branches to *also* bisect test\n> > > > > > breakages and then upload comments to the identified commit on\n> > > > > > GitHub\n> > > > >\n> > > > > Good. I do not think it is useful to try 'pu' as an aggregate\n> > > > > and expect it to always build and work [*1*], but your \"bisect\n> > > > > and pinpoint\" approach makes it useful to identify individual\n> > > > > topic that brings in a breakage.\n> > > >\n> > > > Sadly the many different merge bases[*1*] between `next` and `pu`\n> > > > (which are the obvious good/bad points for bisecting\n> > > > automatically) bring my build agents to its knees. I may have to\n> > > > disable the bisecting feature as a consequence.\n> >\n> > Yeah, this is a bug in the bisect algorithm. Fixing it is in the GSoC\n> > 2017 Ideas.\n> \n> There are also a few ideas at the SO answers:\n> http://stackoverflow.com/a/5652323/717355\n\nThanks for that link!\n\nHowever, my main aim was not to get distracted into yet another corner of\nGit that needs to be fixed (I am on enough of those projects already).\n\nI was merely surprised (and not in a good way) that a plenty ordinary\nbisect between `next` and `pu` all of a sudden tested a *one year old*\ncommit whether it was good or not.\n\nAnd I doubt that the strategy to mark all second parents of all merge\ncommits in pu..next as \"good\" would work well, as the merge bases *still*\nwould have to be tested.\n\nI guess what I have to resort to is this: if I know that `next` tests\nfine, and that `pu` fails, I shall mark all merge bases as \"good\". I am\nsure this has its own set of pitfalls, undoubtedly costing me more time on\nthat front...\n\nBut at least my cursory analysis of this idea seems to make sense: I use\n`next` essentially as a catch-all to verify that the breakage has entered\n`pu`, but not yet `next`. This reasoning makes sense, given that we know\nthe waterfall topology of pu/next/master/maint: patches enter from left to\nright, i.e. anything that entered `pu` may later enter `next`, but not\nvice versa.\n\nCiao,\nDscho\n"},{"id":"311671","messageId":"7EA15219331242ABB08B9A9AA9F08CBE@PhilipOakley","threadId":"45100","inReplyTo":"xmqqshng5osz.fsf@gitster.mtv.corp.google.com","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-02-15T21:06:22Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Junio C Hamano\" \n> \"Philip Oakley\" writes:\n>\n>> There are also a few ideas at the SO answers:\n>> http://stackoverflow.com/a/5652323/717355\n>\n> I vaguely recall that I saw somebody said the same \"mark tips of\n> topics as good\" on the list and answered with why it does not quite\n> work, though.\n>\nI think you may mean \nhttps://public-inbox.org/git/7v8vyam5la.fsf@alter.siamese.dyndns.org/\n\nI think we are thinking of opposite abstractions.\n\nFor regular bisect, the assumption (to a first order) is that there is a \nsingle point of infection of a single persistent bug with a well defined \ntest, and that the goal is to find the point of first infection, as all \nother incidents of the bug are in successor commits, which are all infected. \nThe fail-fix-break again sequence you mentioned in that thread is to my mind \na red herring as it contradicts the normal bisection assumptions (but see \nbelow).\n\nIn the next..pu case the abstraction is in the other direction, we have \npotentially multiple points of infection (from feature branches), and a \nbroad test (the whole test suite). In this case I believe we would like to \ninvestigate initially the --first-parent line with a classic bisect for the \nfirst point of failure (obviously including feature branch merges). This \nwould identify which feature merge, or regular commit, created the first \nbreakage.\n\nOnce the first point of failure has been identified, for the next..pu case, \neach of the post-fail second parents of merge commits _could_ then also be \nchecked (which is a linear search, not a bisection), to identify any \nadditional feature branches that need attention. This second stage search \nwould probably be an option, but if the merging sequence onto pu is \ngenerally from good to bad, then the search is likely to be short. At least \nfor a CI system this 2nd stage could provide useful feedback to the authors \nof their mistakes...\n\nI haven't looked back at the actual patches in that thread, so they may not \nhave followed my expectation of the --multi-bug (TM) search algorithm.\n--\n\nPhilip\n\n\n"},{"id":"311675","messageId":"xmqqvasb2li8.fsf@gitster.mtv.corp.google.com","threadId":"45100","inReplyTo":"7EA15219331242ABB08B9A9AA9F08CBE@PhilipOakley","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-15T21:26:55Z","receivedAt":"2017-02-15T21:27:03Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Philip Oakley\" writes:\n\n> In the next..pu case the abstraction is in the other direction, we\n> have potentially multiple points of infection (from feature branches),\n> and a broad test (the whole test suite). In this case I believe we\n> would like to investigate initially the --first-parent line with a\n> classic bisect for the first point of failure (obviously including\n> feature branch merges). This would identify which feature merge, or\n> regular commit, created the first breakage.\n\nIf you are going first-parent, you would limit the bisection to a\nsingle-strand-of-pearls, and I agree that it is a good strategy to\nfind which topic branch merge broke the tip of 'pu'.\n\nIf we assume that there is no funny interaction among topics that\ncancel a breakage brought in by one topic with another breakage by\nanother topic, then no matter how many broken topics there are, I\nagree that we would get to the first broken topic.\n\nA good thing that comes once we assume that topics are more-or-less\nindependent is that we could rebuild 'pu' minus the broken topic\nidentified by the above procedure and repeat it to find other broken\ntopics, still using the --first-parent bisection, because master..pu\nis a linear sequence of merges of individual topics.\n\n\n"},{"id":"311686","messageId":"4E11838A459C46928A7B817E2F67BA08@PhilipOakley","threadId":"45100","inReplyTo":"xmqqshng5osz.fsf@gitster.mtv.corp.google.com","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-02-15T22:19:40Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Junio C Hamano\" \n> \"Philip Oakley\" writes:\n>\n>> There are also a few ideas at the SO answers:\n>> http://stackoverflow.com/a/5652323/717355\n>\n> I vaguely recall that I saw somebody said the same \"mark tips of\n> topics as good\" on the list and answered with why it does not quite\n> work, though.\n>\nI think you may mean\nhttps://public-inbox.org/git/7v8vyam5la.fsf@alter.siamese.dyndns.org/\n\nI think we are thinking of opposite abstractions.\n\nFor regular bisect, the assumption (to a first order) is that there is a\nsingle point of infection of a single persistent bug with a well defined\ntest, and that the goal is to find the point of first infection, as all\nother incidents of the bug are in successor commits, which are all infected.\nThe fail-fix-break again sequence you mentioned in that thread is to my mind\na red herring as it contradicts the normal bisection assumptions (but see\nbelow).\n\nIn the next..pu case the abstraction is in the other direction, we have\npotentially multiple points of infection (from feature branches), and a\nbroad test (the whole test suite). In this case I believe we would like to\ninvestigate initially the --first-parent line with a classic bisect for the\nfirst point of failure (obviously including feature branch merges). This\nwould identify which feature merge, or regular commit, created the first\nbreakage.\n\nOnce the first point of failure has been identified, for the next..pu case,\neach of the post-fail second parents of merge commits _could_ then also be\nchecked (which is a linear search, not a bisection), to identify any\nadditional feature branches that need attention. This second stage search\nwould probably be an option, but if the merging sequence onto pu is\ngenerally from good to bad, then the search is likely to be short. At least\nfor a CI system this 2nd stage could provide useful feedback to the authors\nof their mistakes...\n\nI haven't looked back at the actual patches in that thread, so they may not\nhave followed my expectation of the --multi-bug (TM) search algorithm.\n--\n\nPhilip\n\n\n"},{"id":"311687","messageId":"74FD91647E574D02BFDF4AD5D8EF52C6@PhilipOakley","threadId":"45100","inReplyTo":"xmqqshng5osz.fsf@gitster.mtv.corp.google.com","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-02-15T22:19:45Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Junio C Hamano\" \n> \"Philip Oakley\" writes:\n>\n>> There are also a few ideas at the SO answers:\n>> http://stackoverflow.com/a/5652323/717355\n>\n> I vaguely recall that I saw somebody said the same \"mark tips of\n> topics as good\" on the list and answered with why it does not quite\n> work, though.\n>\nI think you may mean\nhttps://public-inbox.org/git/7v8vyam5la.fsf@alter.siamese.dyndns.org/\n\nI think we are thinking of opposite abstractions.\n\nFor regular bisect, the assumption (to a first order) is that there is a\nsingle point of infection of a single persistent bug with a well defined\ntest, and that the goal is to find the point of first infection, as all\nother incidents of the bug are in successor commits, which are all infected.\nThe fail-fix-break again sequence you mentioned in that thread is to my mind\na red herring as it contradicts the normal bisection assumptions (but see\nbelow).\n\nIn the next..pu case the abstraction is in the other direction, we have\npotentially multiple points of infection (from feature branches), and a\nbroad test (the whole test suite). In this case I believe we would like to\ninvestigate initially the --first-parent line with a classic bisect for the\nfirst point of failure (obviously including feature branch merges). This\nwould identify which feature merge, or regular commit, created the first\nbreakage.\n\nOnce the first point of failure has been identified, for the next..pu case,\neach of the post-fail second parents of merge commits _could_ then also be\nchecked (which is a linear search, not a bisection), to identify any\nadditional feature branches that need attention. This second stage search\nwould probably be an option, but if the merging sequence onto pu is\ngenerally from good to bad, then the search is likely to be short. At least\nfor a CI system this 2nd stage could provide useful feedback to the authors\nof their mistakes...\n\nI haven't looked back at the actual patches in that thread, so they may not\nhave followed my expectation of the --multi-bug (TM) search algorithm.\n--\n\nPhilip\n\n\n"},{"id":"311701","messageId":"72ADDC07AFEF4BD88EADCDAF75DFB2CB@PhilipOakley","threadId":"45100","inReplyTo":"xmqqvasb2li8.fsf@gitster.mtv.corp.google.com","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-02-15T23:33:36Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"[sorry for the repeated emails - I'd prepared it off line, and then suffered \na number of auto send actions]\nFrom: \"Junio C Hamano\" \n> \"Philip Oakley\" writes:\n>\n>> In the next..pu case the abstraction is in the other direction, we\n>> have potentially multiple points of infection (from feature branches),\n>> and a broad test (the whole test suite). In this case I believe we\n>> would like to investigate initially the --first-parent line with a\n>> classic bisect for the first point of failure (obviously including\n>> feature branch merges). This would identify which feature merge, or\n>> regular commit, created the first breakage.\n>\n> If you are going first-parent, you would limit the bisection to a\n> single-strand-of-pearls, and I agree that it is a good strategy to\n> find which topic branch merge broke the tip of 'pu'.\n>\n> If we assume that there is no funny interaction among topics that\n> cancel a breakage brought in by one topic with another breakage by\n> another topic, then no matter how many broken topics there are, I\n> agree that we would get to the first broken topic.\n>\n\n> A good thing that comes once we assume that topics are more-or-less\n> independent is that we could rebuild 'pu' minus the broken topic\n> identified by the above procedure and repeat it to find other broken\n> topics, still using the --first-parent bisection, because master..pu\n> is a linear sequence of merges of individual topics.\n>\n\nFor an integrator, or especially a CI tool, simply checking the second \nparents of each topic merge (post fail) should at least indicate if the \nbasics of the feature actually passed the tests, though it doesn't check for \ninteraction issues. This could give direct author feedback!\n--\nPhilip \n\n"},{"id":"311708","messageId":"77C7E23E18774409AA1818B12C844985@PhilipOakley","threadId":"45100","inReplyTo":"alpine.DEB.2.20.1702151509251.3496@virtualbox","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-02-15T23:57:41Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Johannes Schindelin\" \n> Hi Philip,\n>\n> On Tue, 14 Feb 2017, Philip Oakley wrote:\n>\n>> From: \"Christian Couder\" \n>> > On Tue, Feb 14, 2017 at 10:08 PM, Junio C Hamano \n>> > wrote:\n>> > > Johannes Schindelin writes:\n>> > >\n>> > > > On Mon, 13 Feb 2017, Junio C Hamano wrote:\n>> > > >\n>> > > > > Johannes Schindelin writes:\n>> > > > >\n>> > > > > > That is why I taught the Git for Windows CI job that tests the\n>> > > > > > four upstream Git integration branches to *also* bisect test\n>> > > > > > breakages and then upload comments to the identified commit on\n>> > > > > > GitHub\n>> > > > >\n>> > > > > Good. I do not think it is useful to try 'pu' as an aggregate\n>> > > > > and expect it to always build and work [*1*], but your \"bisect\n>> > > > > and pinpoint\" approach makes it useful to identify individual\n>> > > > > topic that brings in a breakage.\n>> > > >\n>> > > > Sadly the many different merge bases[*1*] between `next` and `pu`\n>> > > > (which are the obvious good/bad points for bisecting\n>> > > > automatically) bring my build agents to its knees. I may have to\n>> > > > disable the bisecting feature as a consequence.\n>> >\n>> > Yeah, this is a bug in the bisect algorithm. Fixing it is in the GSoC\n>> > 2017 Ideas.\n>>\n>> There are also a few ideas at the SO answers:\n>> http://stackoverflow.com/a/5652323/717355\n>\n> Thanks for that link!\n>\n> However, my main aim was not to get distracted into yet another corner of\n> Git that needs to be fixed (I am on enough of those projects already).\n>\n> I was merely surprised (and not in a good way) that a plenty ordinary\n> bisect between `next` and `pu` all of a sudden tested a *one year old*\n> commit whether it was good or not.\n>\n> And I doubt that the strategy to mark all second parents of all merge\n> commits in pu..next as \"good\" would work well, as the merge bases *still*\n> would have to be tested.\n\nI was expecting that if all the second parents were marked as good, then \nthere would be no merge bases, as there shouldn't be a forked graph, just \nthe linear string of pearls - if bisect doesn't do that then there's an \nfailed optimisation to be had.\n\nI don't see anything in the `git bisect --help` page that would indicate \nthat the merges themselves are omitted from the bisection.\n\n>\n> I guess what I have to resort to is this: if I know that `next` tests\n> fine, and that `pu` fails, I shall mark all merge bases as \"good\". I am\n> sure this has its own set of pitfalls, undoubtedly costing me more time on\n> that front...\n>\n> But at least my cursory analysis of this idea seems to make sense: I use\n> `next` essentially as a catch-all to verify that the breakage has entered\n> `pu`, but not yet `next`. This reasoning makes sense, given that we know\n> the waterfall topology of pu/next/master/maint: patches enter from left to\n> right, i.e. anything that entered `pu` may later enter `next`, but not\n> vice versa.\n\nIt may even be worth 'splitting' the pu branch sequence into the existing pu \n(with merges from series that are selected as reasonable), and then a pr \nbranch (public review?) on top of that holding the rest of the series that \nhave been submitted, so that the CI can do a full test on the tips of them \nto support those devs with limited test capability.\n\n\n>\n> Ciao,\n> Dscho\n> \n\n"},{"id":"311710","messageId":"xmqq60kb0ywi.fsf@gitster.mtv.corp.google.com","threadId":"45100","inReplyTo":"77C7E23E18774409AA1818B12C844985@PhilipOakley","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-16T00:20:29Z","receivedAt":"2017-02-16T00:20:42Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Philip Oakley\" writes:\n\n> It may even be worth 'splitting' the pu branch sequence into the\n> existing pu (with merges from series that are selected as reasonable),\n> and then a pr branch (public review?) on top of that holding the rest\n> of the series that have been submitted, so that the CI can do a full\n> test on the tips of them to support those devs with limited test\n> capability.\n\nI won't stop you from publishing such a pr branch yourself.\n\nFor patches whose merit is not clear because the problem they try to\nsolve is under-explained, whose solution is ill-designed, etc., IOW,\nwith issues that makes me judge that they are not interesting enough\nfor 'pu', it is not worth my time to deal with whitespace brekages\nin them to make them not even apply, to figure out what base the\npatches are meant to apply to, or to resolve conflicts caused by\nthem with topics already in flight, etc.\n\n\n\n\n\n"},{"id":"311729","messageId":"xmqqpoiizzpx.fsf@gitster.mtv.corp.google.com","threadId":"45100","inReplyTo":"72ADDC07AFEF4BD88EADCDAF75DFB2CB@PhilipOakley","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-16T01:33:30Z","receivedAt":"2017-02-16T01:33:42Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Philip Oakley\" writes:\n\n> For an integrator, or especially a CI tool, simply checking the second\n> parents of each topic merge (post fail) should at least indicate if\n> the basics of the feature actually passed the tests, though it doesn't\n> check for interaction issues. This could give direct author feedback!\n\nI think that is essentially what Avery advocated for in the old\nthread Christian gave us earlier in the thread, and I agree that is\na useful way to help contributors.\n\nWhile we are on the topic of testing efficiently and effectively,\nthere are a few other things worth taking advantage of.\n\nOne obvious thing is that not all topic branches get updated every\nday, even though the tip of 'pu' will be different every day (or on\ndays when I run the integration cycle more than once, after every\npushout), because we constantly acquire new topics or updated ones.\nThe tips of topics that are merged in 'pu^{/^### match next}..pu'\ncan be enumerated by following the first-parent chain, and they can\nbe tested all one-by-one, excluding the ones that have been tested\nalready in the previous runs. When I counted these merges the other\nday I had 27 of them, and as of this writing I have 20 of them\n(because some topics were merged to 'next' in the meantime). Among\nthem, only 3 are new. Everything else would have already been\ntested if such a test was done daily. So from that point of view,\ntesting them all could be less expensive than bisecting; in order to\nbisect a first-parent chain of 20 merges, you'd need to test 5 or\nso.\n\nAnother thing that may help to prevent breakages from seeping into\n'next' is that \"pu^{/^### match next}..pu\" are rebuilt every day,\nand the order of topics that are merged are updated. The topics\nthat are closer to be merged to 'next' are moved down, so that\ntesting the merge result would give us closer result to what would\ntesting 'next' in the near future would give us [*1*]. So paying\ncloser attention to the merges (not tips of topics) above the commit\nmarked as \"pu^{/^### match next}\" when testing would catch potential\nbreakage about to happen in 'next' due to unexpected interactions of\ntopics when merged together.\n\nThere is another point in the history that may be worth paying\ncloser attention, which is the tip of 'jch'. This always is ahead\nof \"pu^{/^### match next}\", and almost always has a handful more\ntopics, many of which are considered to be merged to 'next' in near\nfuture. The branch is meant to be at least stable enough for my\npersonal use (hence its name) in helping me run everyday integration\ncycles.\n\n\n[Footnote]\n\n*1* Suppose there are three topics A, B and C that are not yet in\n 'next', and 'pu' merges them in that order. Further suppose\n that C is a lot better cooked than others. Merging C directly\n to 'next' however can expose a hidden issue that changes C\n introduces alone is not sufficient and depends on A or B to be\n present. To avoid such a surprise after merging C to 'next',\n when 'pu' is rebuilt, I try to reorder them so that C is merged\n first on top of an equivalent of 'next', and then A and B. For\n the same reason, topics in 'master..pu^{/^### match next}' are\n also reordered so that ones that are planned to be merged to\n 'master' soon comes at the bottom.\n\n One natural consequence of this is that 'pu' is rebuilt directly\n on top of 'master' and 'next' does not fast-forwared to 'pu'.\n This arrangement also helps to spot mismerges to 'next' and\n avoid the same mismerge to affect 'master' when topics are\n merged to it.\n"},{"id":"312033","messageId":"D4A69FDAB8BD4D0080108611B61E5EF8@PhilipOakley","threadId":"45100","inReplyTo":"xmqq60kb0ywi.fsf@gitster.mtv.corp.google.com","subject":"Re: [git-for-windows] Re: Continuous Testing of Git on Windows","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-02-18T11:49:32Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Junio C Hamano\" \nSent: Thursday, February 16, 2017 12:20 AM\n> \"Philip Oakley\" writes:\n>\n>> It may even be worth 'splitting' the pu branch sequence into the\n>> existing pu (with merges from series that are selected as reasonable),\n>> and then a pr branch (public review?) on top of that holding the rest\n>> of the series that have been submitted, so that the CI can do a full\n>> test on the tips of them to support those devs with limited test\n>> capability.\n>\n> I won't stop you from publishing such a pr branch yourself.\n\nBut would others see it?... (rhetorical, better thoughts below))\n\n>\n> For patches whose merit is not clear because the problem they try to\n> solve is under-explained, whose solution is ill-designed, etc., IOW,\n> with issues that makes me judge that they are not interesting enough\n> for 'pu',\n\nIt is reasonble that that a project's integrator is able to make these \ndecisions. For some projects they may have a layered approach of descisions \nwhich does allow the gradations for the submitted feature series.\n\nSome of this does fall into dscho's differentiation between those patch \nseries that should pass the CI (continuous integration) testing, and those \nthat are there for CT (continuous testing) feedback. This could either be an \nextra branch marking the transition, or a named commit similar to the \n\"pu^{/^### match next}\", etc. In some ways it is similar to my 'pr' \nsuggestion, without the inclusion of the 'all and sundry' series.\n\nFor for integrators who are willing/want to recieve any/all contributions \nfor public view (usually those for projects of a more lenient and less \ncritical variety), then even the CT grouping could then have those \nadditional pr submissions. For Git, you provide that that 'voice of reason' \nfor gatekeeping the pu branch.\n\n\n> it is not worth my time to deal with whitespace brekages\n> in them to make them not even apply, to figure out what base the\n> patches are meant to apply to, or to resolve conflicts caused by\n> them with topics already in flight, etc.\n>\nIf a centralised CT service was available to the project, maybe via the \nGitHub PR process (which isn't curently used by the project) then is may be \na way of allowing the 'all and sundry' contributors to get their ideas upto \na basic level before even bothering yourself (because PRs do not trouble \nyou).\n\nIt may need an extra gatekeeper between the passing patches (PRs) and auto \nsubmission (the Heroku script thingy) which could flood the list with with \ninane changes - one only has to look at the \nhttp://stackoverflow.com/questions/tagged/git stream to see that.\n\nAt least if there was a break point within pu that allowed differentiation \nbetween the series that should fit a CI view, and those that are still at \nthe CT stage, then that may help.\n\nThanks\n\nPhilip.\n\n"}]}