{"thread":{"id":"42210","subject":"avoid duplicate patches from git log ?","startedAt":"2016-05-03T20:11:55Z","lastAt":"2016-05-04T19:15:51Z","messageCount":7,"participants":["Philip Oakley","Jeff King","Junio C Hamano","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"285372","messageId":"712E44CFAD1A41A982AEF1540C1F9F80@PhilipOakley","threadId":"42210","inReplyTo":null,"subject":"avoid duplicate patches from git log ?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2016-05-03T20:11:55Z","receivedAt":"2016-05-03T20:11:55Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"I was trying to search the Git for Windows (G4W) history for commits that \ntouched MSVC.\n\nI've used 'git log -SMSVC --pretty='tformat:%h (%s, \n%ad)' --date=short --reverse' to get a nice list of those commits.\n\nHowever, as the G4W project (https://github.com/git-for-windows/git/) \nfollows the main git repo and its releases, it needs to rebase it's fixup \npatches, while retaining their original series, so has repeated copies of \nthose fix patches on the second parent path (a technique Dscho called \nrebasing merges).\n\nfor example:\n> bf1a7ff (MinGW: disable CRT command line globbing, 2011-01-07)\n> a05e9a8 (MinGW: disable CRT command line globbing, 2011-01-07)\n> 45cfa35 (MinGW: disable CRT command line globbing, 2011-01-07)\n> 1d35390 (MinGW: disable CRT command line globbing, 2011-01-07)\n> 022e029 (MinGW: disable CRT command line globbing, 2011-01-07)\n\n\nHow can I filter out all the duplicate patches which are identical other \nthan the commit date?\n\nThe --left --right and --cherry don't appear to do what I'd expect/hope. Any \nsuggestions?\n--\nPhilip\n"},{"id":"285386","messageId":"20160503220039.GB25133@sigill.intra.peff.net","threadId":"42210","inReplyTo":"712E44CFAD1A41A982AEF1540C1F9F80@PhilipOakley","subject":"Re: avoid duplicate patches from git log ?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-05-03T22:00:39Z","receivedAt":"2016-05-03T22:00:39Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 03, 2016 at 09:11:55PM +0100, Philip Oakley wrote:\n\n> However, as the G4W project (https://github.com/git-for-windows/git/)\n> follows the main git repo and its releases, it needs to rebase it's fixup\n> patches, while retaining their original series, so has repeated copies of\n> those fix patches on the second parent path (a technique Dscho called\n> rebasing merges).\n> \n> for example:\n> > bf1a7ff (MinGW: disable CRT command line globbing, 2011-01-07)\n> > a05e9a8 (MinGW: disable CRT command line globbing, 2011-01-07)\n> > 45cfa35 (MinGW: disable CRT command line globbing, 2011-01-07)\n> > 1d35390 (MinGW: disable CRT command line globbing, 2011-01-07)\n> > 022e029 (MinGW: disable CRT command line globbing, 2011-01-07)\n> \n> \n> How can I filter out all the duplicate patches which are identical other\n> than the commit date?\n> \n> The --left --right and --cherry don't appear to do what I'd expect/hope. Any\n> suggestions?\n\nI don't think there's a good way right now. The option that suppresses\ncommits is --cherry-pick, but it wants there to be a \"left\" and \"right\"\nfrom a symmetric difference, and to cull duplicates from the various\nsides.\n\nI think you really just want to keep a running list of all of the\ncommits you've seen and cull any duplicates. I guess you'd want this as\npart of the history simplification step, so that whole uninteresting\nside-branches are culled.\n\nThe obvious choice for matching two commits is patch-id, though it can\nbe expensive to generate. There have been patches playing around with\ncaching in the past, but nothing merged. For your purposes, I suspect\nmatching an \"(author, authordate, subject)\" tuple would be sufficient\nand fast.\n\n-Peff\n"},{"id":"285388","messageId":"xmqqbn4mlrk2.fsf@gitster.mtv.corp.google.com","threadId":"42210","inReplyTo":"20160503220039.GB25133@sigill.intra.peff.net","subject":"Re: avoid duplicate patches from git log ?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-05-03T22:12:13Z","receivedAt":"2016-05-03T22:12:13Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> On Tue, May 03, 2016 at 09:11:55PM +0100, Philip Oakley wrote:\n>\n>> However, as the G4W project (https://github.com/git-for-windows/git/)\n>> follows the main git repo and its releases, it needs to rebase it's fixup\n>> patches, while retaining their original series, so has repeated copies of\n>> those fix patches on the second parent path (a technique Dscho called\n>> rebasing merges).\n>> \n>> for example:\n>> > bf1a7ff (MinGW: disable CRT command line globbing, 2011-01-07)\n>> > a05e9a8 (MinGW: disable CRT command line globbing, 2011-01-07)\n>> > 45cfa35 (MinGW: disable CRT command line globbing, 2011-01-07)\n>> > 1d35390 (MinGW: disable CRT command line globbing, 2011-01-07)\n>> > 022e029 (MinGW: disable CRT command line globbing, 2011-01-07)\n>> \n>> \n>> How can I filter out all the duplicate patches which are identical other\n>> than the commit date?\n>> \n>> The --left --right and --cherry don't appear to do what I'd expect/hope. Any\n>> suggestions?\n>\n> I don't think there's a good way right now. The option that suppresses\n> commits is --cherry-pick, but it wants there to be a \"left\" and \"right\"\n> from a symmetric difference, and to cull duplicates from the various\n> sides.\n>\n> I think you really just want to keep a running list of all of the\n> commits you've seen and cull any duplicates. I guess you'd want this as\n> part of the history simplification step, so that whole uninteresting\n> side-branches are culled.\n>\n> The obvious choice for matching two commits is patch-id, though it can\n> be expensive to generate. There have been patches playing around with\n> caching in the past, but nothing merged. For your purposes, I suspect\n> matching an \"(author, authordate, subject)\" tuple would be sufficient\n> and fast.\n\nWhat would be really interesting is what should happen when the side\n\"rebase merge\" branch that is supposed to be irrelevant for the\npurpose of explaining the overall history does not become empty\nafter such filtering operation.  The merge commit itself may claim\nthat both branches are equivalent, but in reality it may turn out\nthat the merge failed to reflect the effect of some other changes in\nthe history of the side branch in the result--which would be a\nticking time-bomb for future mismerges waiting to happen.\n"},{"id":"285392","messageId":"E735FEEE2908434F874CAD1874307E38@PhilipOakley","threadId":"42210","inReplyTo":"xmqqbn4mlrk2.fsf@gitster.mtv.corp.google.com","subject":"Re: avoid duplicate patches from git log ?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2016-05-03T22:36:32Z","receivedAt":"2016-05-03T22:36:32Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Junio C Hamano\" <gitster@pobox.com>\n> Jeff King <peff@peff.net> writes:\n>\n>> On Tue, May 03, 2016 at 09:11:55PM +0100, Philip Oakley wrote:\n>>\n>>> However, as the G4W project (https://github.com/git-for-windows/git/)\n>>> follows the main git repo and its releases, it needs to rebase it's \n>>> fixup\n>>> patches, while retaining their original series, so has repeated copies \n>>> of\n>>> those fix patches on the second parent path (a technique Dscho called\n>>> rebasing merges).\n>>>\n>>> for example:\n>>> > bf1a7ff (MinGW: disable CRT command line globbing, 2011-01-07)\n>>> > a05e9a8 (MinGW: disable CRT command line globbing, 2011-01-07)\n>>> > 45cfa35 (MinGW: disable CRT command line globbing, 2011-01-07)\n>>> > 1d35390 (MinGW: disable CRT command line globbing, 2011-01-07)\n>>> > 022e029 (MinGW: disable CRT command line globbing, 2011-01-07)\n>>>\n>>>\n>>> How can I filter out all the duplicate patches which are identical other\n>>> than the commit date?\n>>>\n>>> The --left --right and --cherry don't appear to do what I'd expect/hope. \n>>> Any\n>>> suggestions?\n>>\n>> I don't think there's a good way right now. The option that suppresses\n>> commits is --cherry-pick, but it wants there to be a \"left\" and \"right\"\n>> from a symmetric difference, and to cull duplicates from the various\n>> sides.\n>>\n>> I think you really just want to keep a running list of all of the\n>> commits you've seen and cull any duplicates. I guess you'd want this as\n>> part of the history simplification step, so that whole uninteresting\n>> side-branches are culled.\n>>\n>> The obvious choice for matching two commits is patch-id, though it can\n>> be expensive to generate. There have been patches playing around with\n>> caching in the past, but nothing merged. For your purposes, I suspect\n>> matching an \"(author, authordate, subject)\" tuple would be sufficient\n>> and fast.\n>\n> What would be really interesting is what should happen when the side\n> \"rebase merge\" branch that is supposed to be irrelevant for the\n> purpose of explaining the overall history does not become empty\n> after such filtering operation.  The merge commit itself may claim\n> that both branches are equivalent, but in reality it may turn out\n> that the merge failed to reflect the effect of some other changes in\n> the history of the side branch in the result--which would be a\n> ticking time-bomb for future mismerges waiting to happen.\n\nI think that's a misunderstanding of the development process for an \"on top \nof\" project, where the upstream would not be expected to take all the fixups \nfor that project's customers.\n\nThe releases of the project do need to be retained in the history, but \nbecause of the \"on top of\" policy, the prior release becomes a second parent \nto a \"theirs\" merge commit of the upstream (and subsequent rebase on top of \nthat).\n\nThus when seaching history the first parent route would have the fastest \ntransition to the upstream, but the full history would still have all the \nreleases on it.\n\nIt may be that Peff's suggestion is a workable heuristic for a rebase flow \nwhere one could eliminate those duplicates quite easily. I just had a \nfeeling that there was already something that did the patch-id thing for \nduplicate removals, but obviously I had that wrong.\n\n--\nPhilip \n"},{"id":"285461","messageId":"alpine.DEB.2.20.1605041329040.9313@virtualbox","threadId":"42210","inReplyTo":"712E44CFAD1A41A982AEF1540C1F9F80@PhilipOakley","subject":"Re: avoid duplicate patches from git log ?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-05-04T11:44:17Z","receivedAt":"2016-05-04T11:44:17Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Philip,\n\nOn Tue, 3 May 2016, Philip Oakley wrote:\n\n> I was trying to search the Git for Windows (G4W) history for commits that\n> touched MSVC.\n> \n> I've used 'git log -SMSVC --pretty='tformat:%h (%s, %ad)' --date=short\n> --reverse' to get a nice list of those commits.\n> \n> However, as the G4W project (https://github.com/git-for-windows/git/)\n> follows the main git repo and its releases, it needs to rebase it's\n> fixup patches, while retaining their original series, so has repeated\n> copies of those fix patches on the second parent path (a technique Dscho\n> called rebasing merges).\n\nActually, I no longer use rebasing merges, but instead merging rebases.\nThe difference is a little subtle:\n\nRebasing merge:\n\n- upstream ----- rebased-A - rebased-B - rebasing-merge (-s ours)\n                                       /\n- old-A - old-B -----------------------\n\nMerging rebase:\n\n- upstream ----- merging-rebase (-s ours) - rebased-A - rebased-B\n               /\n- old-A - old-B\n\nOf course both diagrams are drastically simplified, as I do not only\nrebase mere patches, but topic branches, including merge structure\n(currently 44 IIRC), to make it easier to break out the topic branches for\neasy submission later.\n\nIt turned out that the rebasing-merge strategy actually made things\nharder, as it was not quite as easy to figure out *which* commits needed\nrebasing (remember, new commits were added after rebasing-merges).\n\nWith merging-rebases, at least, one knows that the tree of the merge\ncommit starting the whole shebang is identical to upstream's tree.  All of\nthe patches that come on top of that merge commit need to be rebased the\nnext time round, including new patches.\n\nFittingly, the commit message of said merge commit begins with the\nfollowing text: \"Start the merging-rebase ...\"\n\nBTW The reason for this rather unwieldy setup is that we historically had\na hard time getting our patches into upstream Git (there was some degree\nof resistance in the past, and also a quite limiting lack of time on my\npart).\n\n> for example:\n> > bf1a7ff (MinGW: disable CRT command line globbing, 2011-01-07)\n> > a05e9a8 (MinGW: disable CRT command line globbing, 2011-01-07)\n> > 45cfa35 (MinGW: disable CRT command line globbing, 2011-01-07)\n> > 1d35390 (MinGW: disable CRT command line globbing, 2011-01-07)\n> > 022e029 (MinGW: disable CRT command line globbing, 2011-01-07)\n> \n> \n> How can I filter out all the duplicate patches which are identical other\n> than the commit date?\n\nI would go about it in a completely different manner. Remember that the\nmerging rebase starts with a merge that integrates the previous history,\nbut also resets the tree to upstream's. Therefore all the commits merged\nat the start of the merging-rebase are the ones in which you are *not*\ninterested.\n\nIn other words, this command-line will yield the output you desire:\n\ngit log -SMSVC --pretty='tformat:%h (%s, %ad)' --date=short \\\n\tHEAD^{/Start.the.merging-rebase}^2..\n\nCiao,\nDscho\n"},{"id":"285464","messageId":"alpine.DEB.2.20.1605041347170.9313@virtualbox","threadId":"42210","inReplyTo":"E735FEEE2908434F874CAD1874307E38@PhilipOakley","subject":"Re: avoid duplicate patches from git log ?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-05-04T11:58:57Z","receivedAt":"2016-05-04T11:58:57Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Philip,\n\nOn Tue, 3 May 2016, Philip Oakley wrote:\n\n> From: \"Junio C Hamano\" <gitster@pobox.com>\n> > Jeff King <peff@peff.net> writes:\n> >\n> > > On Tue, May 03, 2016 at 09:11:55PM +0100, Philip Oakley wrote:\n> > >\n> > > > However, as the G4W project (https://github.com/git-for-windows/git/)\n> > > > follows the main git repo and its releases, it needs to rebase it's\n> > > > fixup\n> > > > patches, while retaining their original series, so has repeated copies\n> > > > of\n> > > > those fix patches on the second parent path (a technique Dscho called\n> > > > rebasing merges).\n> > > >\n> > > > for example:\n> > > > > bf1a7ff (MinGW: disable CRT command line globbing, 2011-01-07)\n> > > > > a05e9a8 (MinGW: disable CRT command line globbing, 2011-01-07)\n> > > > > 45cfa35 (MinGW: disable CRT command line globbing, 2011-01-07)\n> > > > > 1d35390 (MinGW: disable CRT command line globbing, 2011-01-07)\n> > > > > 022e029 (MinGW: disable CRT command line globbing, 2011-01-07)\n> > > >\n> > > >\n> > > > How can I filter out all the duplicate patches which are identical other\n> > > > than the commit date?\n> > > >\n> > > > The --left --right and --cherry don't appear to do what I'd expect/hope.\n> > > > Any\n> > > > suggestions?\n> > >\n> > > I don't think there's a good way right now. The option that suppresses\n> > > commits is --cherry-pick, but it wants there to be a \"left\" and \"right\"\n> > > from a symmetric difference, and to cull duplicates from the various\n> > > sides.\n> > >\n> > > I think you really just want to keep a running list of all of the\n> > > commits you've seen and cull any duplicates. I guess you'd want this as\n> > > part of the history simplification step, so that whole uninteresting\n> > > side-branches are culled.\n> > >\n> > > The obvious choice for matching two commits is patch-id, though it can\n> > > be expensive to generate. There have been patches playing around with\n> > > caching in the past, but nothing merged. For your purposes, I suspect\n> > > matching an \"(author, authordate, subject)\" tuple would be sufficient\n> > > and fast.\n> >\n> > What would be really interesting is what should happen when the side\n> > \"rebase merge\" branch that is supposed to be irrelevant for the\n> > purpose of explaining the overall history does not become empty\n> > after such filtering operation.  The merge commit itself may claim\n> > that both branches are equivalent, but in reality it may turn out\n> > that the merge failed to reflect the effect of some other changes in\n> > the history of the side branch in the result--which would be a\n> > ticking time-bomb for future mismerges waiting to happen.\n> \n> I think that's a misunderstanding of the development process for an \"on\n> top of\" project, where the upstream would not be expected to take all\n> the fixups for that project's customers.\n\nExactly. The merging-rebase technique only makes sense when the entire set\nof changes is rebased.\n\nPlease note that I do drop some patches from time to time, so what Junio\nfears is actually not a time bomb, but rather the intended benefit.\n\nThe *real* advantage of the merging-rebase technique is that contributors\ncan easily call `git rebase origin/master` *even after* origin was\n\"rebased\". Because it was both rebased, and not rebased.\n\n> The releases of the project do need to be retained in the history, but\n> because of the \"on top of\" policy, the prior release becomes a second\n> parent to a \"theirs\" merge commit of the upstream (and subsequent rebase\n> on top of that).\n\nThis is a secondary concern, true. But we could easily tag the releases\nand then continue `master` with a rebased version, i.e. `master` would\nusually not fast-forward from tagged commits.\n\nBut it would make contributing much harder than it already is.\n\n> It may be that Peff's suggestion is a workable heuristic for a rebase\n> flow where one could eliminate those duplicates quite easily. I just had\n> a feeling that there was already something that did the patch-id thing\n> for duplicate removals, but obviously I had that wrong.\n\nOh, we do have that logic, it is the --cherry option of the rev-list\nmachinery. It's just that due to the merging-rebase technique, you cannot\nhave your desired commits on one side and the undesired ones on the other\nside of the \"...\".\n\nUnless...\n\nUnless you play games with the grafts. You *could* pretend that the \"Start\nthe merging-rebase\" commit had only its first parent, using a graft. Then\n\"git log --cherry --right-only SECOND_PARENT...HEAD\" (where SECOND_PARENT\nwould be the culled second parent of that merge commit) would have the\nintended result.\n\nYou should not forget to remove the graft afterwards, though. (You *might*\nbe able to finagle something more temporary by using `git replace`,\ndunno, still finicky.)\n\nCiao,\nDscho\n"},{"id":"285494","messageId":"xmqqinythbx4.fsf@gitster.mtv.corp.google.com","threadId":"42210","inReplyTo":"alpine.DEB.2.20.1605041347170.9313@virtualbox","subject":"Re: avoid duplicate patches from git log ?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-05-04T19:15:51Z","receivedAt":"2016-05-04T19:15:51Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> Please note that I do drop some patches from time to time, so what Junio\n> fears is actually not a time bomb, but rather the intended benefit.\n\nOK.  As long as all the dropped patches are intentional, by\ndefinition nothing is lost ;-)\n"}]}