{"thread":{"id":"46056","subject":"mergetool: what to do about deleting precious files?","startedAt":"2017-05-27T10:03:16Z","lastAt":"2017-05-31T00:02:46Z","messageCount":8,"participants":["Philip Oakley","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"320866","messageId":"8CEDDC40C7D24404976929CF3E7AF6FC@PhilipOakley","threadId":"46056","inReplyTo":null,"subject":"mergetool: what to do about deleting precious files?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2017-05-27T10:03:09Z","receivedAt":"2017-05-27T10:03:16Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"The git book [1] and a few blog posts [2] show how to preserve files which\nare in the current branch against changes that are on the branch being\nmerged in.\n\ne.g. (from [2])\n\necho ' merge=ours' >> .gitattributes && # commit\ngit config --global merge.ours.driver true\n\n(test) $ git checkout demo\n(demo) $ git merge -\n# contents are not merged and the original retained.\n\n\n\nHowever what is not covered (at least in the documentation ) is the case\nwhere the file to be ignored is not present on the current branch, but is\npresent on the branch to be merged in.\n\nNormal expectations would be that in such a case the new file from the\nsecond parent branch would be added to the current branch.\n\nIs there an easy way to use the current mergedriver mechanism to make this\nnew file from the second parent 'go away' - does one have to delete output\npath (%P) or what? (or is this just a documenation issue?) [3]\n\n\nInspired by SO Q&A\nhttps://stackoverflow.com/questions/44131926/git-merge-remove-unnecessary-file-folder/44138354#44138354\n\"\ngit merge\nI'm trying to merge my dev branch to master, but I don't want to merge some\nof the files/folders to master branch, because it is not needed now on\nMaster.\"\n\n\nAny suggestions for a simple merge driver to avoid merging and commiting\nsuch precious files from the second parent at merge time?\n--\nPhilip\n\n[1]\nhttp://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes#Merge-Strategies\n[2]\nhttps://medium.com/@porteneuve/how-to-make-git-preserve-specific-files-while-merging-18c92343826b\n[3]\nhttps://git-scm.com/docs/gitattributes #Defining a custom merge driver:\n\"The merge driver can learn the pathname in which the merged result will be\nstored via placeholder %P.\"\n\n"},{"id":"320877","messageId":"xmqq37bpbx5t.fsf@gitster.mtv.corp.google.com","threadId":"46056","inReplyTo":"8CEDDC40C7D24404976929CF3E7AF6FC@PhilipOakley","subject":"Re: mergetool: what to do about deleting precious files?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-28T01:14:06Z","receivedAt":"2017-05-28T01:14:14Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Philip Oakley\" writes:\n\n> The git book [1] and a few blog posts [2] show how to preserve files which\n> are in the current branch against changes that are on the branch being\n> merged in.\n>\n> e.g. (from [2])\n>\n> echo ' merge=ours' >> .gitattributes && # commit\n> git config --global merge.ours.driver true\n>\n> (test) $ git checkout demo\n> (demo) $ git merge -\n> # contents are not merged and the original retained.\n>\n>\n>\n> However what is not covered (at least in the documentation ) is the case\n> where the file to be ignored is not present on the current branch, but is\n> present on the branch to be merged in.\n\nHmph. Per-path 'ours' and 'theirs' kick in only after we decide to\nperform the content level three-way merge. I wonder what would (not\n\"should\", but \"would with the current code\") happen, with the same\nattribute setting, if the file being merged were not changed by ours\nbut modified by the side branch? I suspect that we'd take the change\nmade by the side branch.\n\n> Normal expectations would be that in such a case the new file from the\n> second parent branch would be added to the current branch.\n\nSo I do not think this is not limited to \"new file\". Anything that\na tree-level three-way merge would resolve cleanly without having to\nconsult the content-level three-way merge will complete without\nconsulting the merge.ours.driver; per-file content-level three-way\nmerge driver (which is what merge= mechanism lets you\nspecify via the attributes mechanism) is not something you would\nwant to use for this kind of thing. It is purely for resolving the\nactual content-level conflicts.\n\n"},{"id":"320881","messageId":"D689F4F80DA24C78B5E0FB025B9548E6@PhilipOakley","threadId":"46056","inReplyTo":"xmqq37bpbx5t.fsf@gitster.mtv.corp.google.com","subject":"Re: mergetool: what to do about deleting precious files?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2017-05-28T10:24:43Z","receivedAt":"2017-05-28T10:25:03Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Junio C Hamano\" \n> \"Philip Oakley\" writes:\n>\n>> The git book [1] and a few blog posts [2] show how to preserve files \n>> which\n>> are in the current branch against changes that are on the branch being\n>> merged in.\n>>\n>> e.g. (from [2])\n>>\n>> echo ' merge=ours' >> .gitattributes && # commit\n>> git config --global merge.ours.driver true\n>>\n>> (test) $ git checkout demo\n>> (demo) $ git merge -\n>> # contents are not merged and the original retained.\n>>\n>>\n>>\n>> However what is not covered (at least in the documentation ) is the case\n>> where the file to be ignored is not present on the current branch, but is\n>> present on the branch to be merged in.\n>\n> Hmph. Per-path 'ours' and 'theirs' kick in only after we decide to\n> perform the content level three-way merge. I wonder what would (not\n> \"should\", but \"would with the current code\") happen, with the same\n> attribute setting, if the file being merged were not changed by ours\n> but modified by the side branch? I suspect that we'd take the change\n> made by the side branch.\n\nHere the 'ours' strategy is defined by the user's config file merge driver \nlist.\n\nI'd understood it that once it was decided there was a merge to be performed \n(the repective blob oid's in the repo/index are different) that the problem \nof merging is then handed off to the declared merge driver.\n\n>\n>> Normal expectations would be that in such a case the new file from the\n>> second parent branch would be added to the current branch.\n>\n\nThe git-scm and blog posts suggest that the original is left in place at the \n%P path, the merge driver run, and its return values used to decide if the \nuser has to go and resolve conflicts. By setting the driver to 'true', the \nresult is then said to be that the current 'blob' (i.e. file) is accepted \nunchanged (in %P), so anything from the second parent blob was completely \nignored.\n\nHowever if we have the addition of a new file, I can't tell from the docs \nwhat should happen? Is this still a merge such that the merge driver is \ncalled, or is the added file accepted without recourse to its .gitattributes \nsetting (surely that would be a bug).\n\nThen assuming we have reached an external driver, and it wants to not add \nthat very file that was added in the second parent branch, what does the %P \npath point to (/dev/null?) - in particular, shouldn't the docs say? (I've \nnot tested, and one test is not proof)\n\nIt maybe that the user wants a merge driver that says \"If I ever see a \nsecret key or password, then remove the whole file\", which (removing the \nfile from the merge) is a currently undocumented process (if even possible).\n\n> So I do not think this is not limited to \"new file\". Anything that\n> a tree-level three-way merge would resolve cleanly without having to\n> consult the content-level three-way merge will complete without\n> consulting the merge.ours.driver; per-file content-level three-way\n> merge driver (which is what merge= mechanism lets you\n> specify via the attributes mechanism) is not something you would\n> want to use for this kind of thing. It is purely for resolving the\n> actual content-level conflicts.\n>\nThat (that Git knows best) sounds just wrong. If the user has set a file \nattribute strategy, why would we ignore it? We already have different \ninternal strategies anyway, so how do we even know that the potential merge \nwas conflict free if we have haven't checked its attribute type. Maybe I'm \nmissing something.\n--\nPhilip \n\n"},{"id":"320888","messageId":"xmqqpoet9lms.fsf@gitster.mtv.corp.google.com","threadId":"46056","inReplyTo":"D689F4F80DA24C78B5E0FB025B9548E6@PhilipOakley","subject":"Re: mergetool: what to do about deleting precious files?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-28T13:06:03Z","receivedAt":"2017-05-28T13:06:10Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Philip Oakley\" writes:\n\n>> So I do not think this is not limited to \"new file\". Anything that\n>> a tree-level three-way merge would resolve cleanly without having to\n>> consult the content-level three-way merge will complete without\n>> consulting the merge.ours.driver; per-file content-level three-way\n>> merge driver (which is what merge= mechanism lets you\n>> specify via the attributes mechanism) is not something you would\n>> want to use for this kind of thing. It is purely for resolving the\n>> actual content-level conflicts.\n>>\n> That (that Git knows best) sounds just wrong.\n\nDon't twist my words. I never said Git knows best. \n\nThe user-level merge driver is a mechanism to affect conflict level\nthree-way merges. The interface to the content level three-way\nmerge driver feeds three versions of blobs and the driver is\nexpected to give a merged result. The interface as designed is\nincapable of passing \"here is the common ancestor\", \"our side is\nmissing\" and \"their side is this content\".\n\nSo if we want a mechanism that can affect the outcome of tree-level\nthree-way merge, we need a _new_ mechanism. The existing merge\ndrivers that are written by end users (at least the ones written\ncorrectly to the spec, anyway) are not expecting to be called with\n\"in our tree, there is no blob here\", and trying to piggyback on it\nwill break existing users.\n"},{"id":"320955","messageId":"DE497A69E68043DABF9C3A7D9198B277@PhilipOakley","threadId":"46056","inReplyTo":"xmqqpoet9lms.fsf@gitster.mtv.corp.google.com","subject":"Re: mergetool: what to do about deleting precious files?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2017-05-29T12:57:08Z","receivedAt":"2017-05-29T12:57:29Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Junio C Hamano\" \n> \"Philip Oakley\" writes:\n>\n>>> So I do not think this is not limited to \"new file\". Anything that\n>>> a tree-level three-way merge would resolve cleanly without having to\n>>> consult the content-level three-way merge will complete without\n>>> consulting the merge.ours.driver; per-file content-level three-way\n>>> merge driver (which is what merge= mechanism lets you\n>>> specify via the attributes mechanism) is not something you would\n>>> want to use for this kind of thing. It is purely for resolving the\n>>> actual content-level conflicts.\n>>>\n>> That (that Git knows best) sounds just wrong.\n>\n> Don't twist my words. I never said Git knows best.\n\nThe part I was responding to was \"would resolve cleanly without having to\nconsult the content-level three-way merge will complete without\nconsulting the merge.ours.driver\".\n\nIt was that lack of consultation (by git) of the putative merge-driver that \nwas being noted.\n\nThe general misunderstanding, as I now see it, is the (false) expectation \nthat a merge-driver would do the whole merge process.\n\nIt took a bit of digging through the documentation for me to find out just \nwhat the merge process appears to be. I'm sure that it obvious to those who \nhave worked with git from the beginning and the previous patch flow process, \nbut the merge process wasn't obvious to me, and various blogs and SO Q&A on \nthe issue suggest the same for many others.\n\nIf I now understand correctly, the merge process flow is:\n\n* canonicalise content (eol, smudge-clean, $id, renormalise, etc)\n* diff the content (internal, or GIT_EXTERNAL_DIFF)\n* apply the diff\n* if conflicts, only then use merge-driver/tool\n\nWould that be a correct interpretation?\n\n\n\n>\n> The user-level merge driver is a mechanism to affect conflict level\n> three-way merges. The interface to the content level three-way\n> merge driver feeds three versions of blobs and the driver is\n> expected to give a merged result. The interface as designed is\n> incapable of passing \"here is the common ancestor\", \"our side is\n> missing\" and \"their side is this content\".\n>\n> So if we want a mechanism that can affect the outcome of tree-level\n> three-way merge, we need a _new_ mechanism. The existing merge\n> drivers that are written by end users (at least the ones written\n> correctly to the spec, anyway) are not expecting to be called with\n> \"in our tree, there is no blob here\", and trying to piggyback on it\n> will break existing users.\n\nIs an alternative to use the GIT_EXTERNAL_DIFF to create a nul diff, so no \nchanges are applied (precious/sensitive file is left behind)? This would \nhave no conflicts and no requirement for a merge-conflict driver.\n\n--\n\nPhilip \n\n"},{"id":"320978","messageId":"xmqq1sr7f9nb.fsf@gitster.mtv.corp.google.com","threadId":"46056","inReplyTo":"DE497A69E68043DABF9C3A7D9198B277@PhilipOakley","subject":"Re: mergetool: what to do about deleting precious files?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-30T00:52:56Z","receivedAt":"2017-05-30T00:53:03Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Philip Oakley\" writes:\n\n> If I now understand correctly, the merge process flow is:\n>\n> * canonicalise content (eol, smudge-clean, $id, renormalise, etc)\n> * diff the content (internal, or GIT_EXTERNAL_DIFF)\n> * apply the diff\n> * if conflicts, only then use merge-driver/tool\n>\n> Would that be a correct interpretation?\n\nNot quite. There are a lot more going on before any of those steps:\n\n * Find the common ancestor commit (which could be many).\n\n * Walk the three trees (the common ancestor's, ours and theirs) in\n parallel, noticing what happened to each path. Depending on what\n happened to the path in each branch, the merge may or may not\n \"conflict\" (e.g. when both sides added exactly the same contents\n to the same path, they are not counted as conflicting. when we\n removed while they modified, they show as conflicting).\n\n * For paths that are conflicting, feed the canonicalized content of\n the versions from common, ours and theirs to the file-level merge\n driver. The builtin file-level merge driver takes two xdiff (one\n between ancestor and ours, the other between ancestore and\n theirs) and reconciles them to produce the result. But that is\n irrelevant in the context of \"custom merge driver\"; the builtin\n one is skipped altogether and the custom contents merge driver\n the user specified via the attributes is used instead.\n\nNotice that the second step above has no customization knobs. Any\npath the second step deems not to conflict is \"merged cleanly\"\nwithout even triggering the \"oops, ours and theirs did conflicting\nchanges, to the content; let's see how the final content should look\nlike\" (aka the third step). This is *not* because \"Git knows the\nbest\"; it is merely that nobody felt the need for a mechanism to\nallow customizing the second step.\n\nAnd that is why I said you need a new customization mechanism if you\nwant to affect the outcome of the scenario that started this thread.\n"},{"id":"321090","messageId":"E0D5AC60DB0F48DEBF865597910D9531@PhilipOakley","threadId":"46056","inReplyTo":"xmqq1sr7f9nb.fsf@gitster.mtv.corp.google.com","subject":"Re: mergetool: what to do about deleting precious files?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2017-05-30T23:04:46Z","receivedAt":"2017-05-30T23:05:13Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Junio C Hamano\" \n\nThanks for the replies. Let's see if I've got it...\n\n> \"Philip Oakley\" writes:\n>\n>> If I now understand correctly, the merge process flow is:\n>>\n>> * canonicalise content (eol, smudge-clean, $id, renormalise, etc)\n>> * diff the content (internal, or GIT_EXTERNAL_DIFF)\n>> * apply the diff\n>> * if conflicts, only then use merge-driver/tool\n>>\n>> Would that be a correct interpretation?\n>\n> Not quite. There are a lot more going on before any of those steps:\n>\n> * Find the common ancestor commit (which could be many).\n\nIIUC Git selects one of them, rather than all if there are many (which then \nmay not be the optimum)\n\n>\n> * Walk the three trees (the common ancestor's, ours and theirs) in\n> parallel, noticing what happened to each path. Depending on what\n> happened to the path in each branch, the merge may or may not\n> \"conflict\" (e.g. when both sides added exactly the same contents\n> to the same path, they are not counted as conflicting. when we\n> removed while they modified, they show as conflicting).\n\nI'm assuming here that this is the sha-oid comparison, and then checking the \ntree/blob names that match them. (the top tree not having a name). So here \n\"conflict free\" is that the sha-oids match.\n\nAlso, I thnk this is saying that added or removed trees or blobs are in some \nsense are 'conflict free' (though still subject to rename/move detection \netc). An added file/blob would be conflict free for merging into it's tree, \nyes?\n\nIIUC, the comparison is therefore using the in-repo sha-oids; \nunless --renormalise was given which will do a smudge-clean washing cycle \nand recomute fresh canonical sha-oids for the comparison (rather than doing \nit later).\n\n>\n> * For paths that are conflicting, feed the canonicalized content of\n> the versions from common, ours and theirs to the file-level merge\n> driver.\n\nSo this is where any .gitattibutes settings come in, or is the merge driver \nafter the diff step? (which could also be a user diff?)\n\n> The builtin file-level merge driver takes two xdiff (one\n> between ancestor and ours, the other between ancestore and\n> theirs) and reconciles them to produce the result. But that is\n> irrelevant in the context of \"custom merge driver\"; the builtin\n> one is skipped altogether and the custom contents merge driver\n> the user specified via the attributes is used instead.\n>\n> Notice that the second step above has no customization knobs. Any\n> path the second step deems not to conflict is \"merged cleanly\"\n> without even triggering the \"oops, ours and theirs did conflicting\n> changes, to the content; let's see how the final content should look\n> like\" (aka the third step). This is *not* because \"Git knows the\n> best\"; it is merely that nobody felt the need for a mechanism to\n> allow customizing the second step.\n>\n> And that is why I said you need a new customization mechanism if you\n> want to affect the outcome of the scenario that started this thread.\n\nOk, I think I can see how I was confused between the \"tree merge\" (oid \nconflict detection) and the more usual (to users) \"file merge\" (line by \nline, etc.). I wasn't sure where to find that as someone relatively new to \nGit.\n\nThanks for the explanations.\n--\nPhilip \n\n\n---\nThis email has been checked for viruses by AVG.\nhttp://www.avg.com\n\n"},{"id":"321100","messageId":"xmqqbmq9ao6c.fsf@gitster.mtv.corp.google.com","threadId":"46056","inReplyTo":"E0D5AC60DB0F48DEBF865597910D9531@PhilipOakley","subject":"Re: mergetool: what to do about deleting precious files?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-31T00:02:35Z","receivedAt":"2017-05-31T00:02:46Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Philip Oakley\" writes:\n\n> From: \"Junio C Hamano\" \n>\n> Thanks for the replies. Let's see if I've got it...\n>\n>> \"Philip Oakley\" writes:\n>>\n>>> If I now understand correctly, the merge process flow is:\n>>>\n>>> * canonicalise content (eol, smudge-clean, $id, renormalise, etc)\n>>> * diff the content (internal, or GIT_EXTERNAL_DIFF)\n>>> * apply the diff\n>>> * if conflicts, only then use merge-driver/tool\n>>>\n>>> Would that be a correct interpretation?\n>>\n>> Not quite. There are a lot more going on before any of those steps:\n>>\n>> * Find the common ancestor commit (which could be many).\n>\n> IIUC Git selects one of them, rather than all if there are many (which\n> then may not be the optimum)\n\nNot quite. The interface to \"git merge-$backend\" can take more than\none and \"git merge\" frontend does pass them to the backend. How\nthey are used depends on the backend. The \"resolve\" one tries to\nuse all of them at once; the \"recursive\" one tries merge across them\nto come up with a tree to be used as a single \"virtual common\nancestor\". But details does not matter for the purpose of analysing\nthe case that triggered this discussion.\n\n>>\n>> * Walk the three trees (the common ancestor's, ours and theirs) in\n>> parallel, noticing what happened to each path. Depending on what\n>> happened to the path in each branch, the merge may or may not\n>> \"conflict\" (e.g. when both sides added exactly the same contents\n>> to the same path, they are not counted as conflicting. when we\n>> removed while they modified, they show as conflicting).\n>\n> I'm assuming here that this is the sha-oid comparison, and then\n> checking the tree/blob names that match them. (the top tree not having\n> a name). So here \"conflict free\" is that the sha-oids match.\n>\n> Also, I thnk this is saying that added or removed trees or blobs are\n> in some sense are 'conflict free' (though still subject to rename/move\n> detection etc). An added file/blob would be conflict free for merging\n> into it's tree, yes?\n\nAfter \"recursive\" figures out the renames, an addition that still\nremains (i.e. not matched up with a deletion elsewhere) would be a\ncandidate to be added silently (except that D/F conflict can still\nbe diagnosed).\n\n>> * For paths that are conflicting, feed the canonicalized content of\n>> the versions from common, ours and theirs to the file-level merge\n>> driver.\n>\n> So this is where any .gitattibutes settings come in, or is the merge\n> driver after the diff step? (which could also be a user diff?)\n\nI think you answered this yourself in your \"Ok, I think I can see\nhow I was confused...\" paragraph.\n"}]}