{"thread":{"id":"33164","subject":"Rename conflicts in the index","startedAt":"2013-03-13T14:08:49Z","lastAt":"2013-04-03T00:57:50Z","messageCount":16,"participants":["Edward Thomson","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"211214","messageId":"A54CE3E330039942B33B670D971F857403995D93@TK5EX14MBXC252.redmond.corp.microsoft.com","threadId":"33164","inReplyTo":null,"subject":"Rename conflicts in the index","fromName":"Edward Thomson","fromEmail":"ethomson@microsoft.com","sentAt":"2013-03-13T14:08:49Z","receivedAt":"2013-03-13T14:08:49Z","isPatch":false,"sender":{"key":"ethomson@microsoft.com","avatar":"https://gravatar.com/avatar/c80dc8bacdf4afcf27334f21b70093a25ac5559d576357ea13592bf438113950?d=mp&s=160"},"body":"When a rename conflict occurs, the information about the conflict is\nwritten to stdout and the index is updated as if the conflict were a\nsimpler conflict that did not involve renames. This doesn't give a lot of\ninformation to users after the fact - a status of \"added in ours\" does not\nprovide a lot of details about what occurred.\n\nIn reimplementations, we face a similar challenge. Unfortunately, we can't\nsimply print the output to the console, we would like to provide this\ninformation back to the caller by some mechanism. My preference would be to\nreturn this information in the index - so that the results of \"unpack trees\"\n(if you will) was a single data structure (the index) and so that we could\npersist this information to disk. I've been considering the idea of an\nextension that contains more detailed data about conflicts, but I certainly\nwouldn't want to proceed without discussing this here further.\n\nI would propose that we store the data about the file in conflict as it\noccurred through the renames. For example, in a rename 1->2 conflict where\nA was renamed to both B and C, you would have a single conflict entry\ncontaining the data for A, B and C. This would allow us to provide more\ndetailed information to the user - and allow them to (say) choose a single\nname to proceed with.\n\nIs this something that has value to core git as well? Alternately, is\nthere something particularly stupid about this proposal?\n\nThanks-\n-ed\n"},{"id":"211241","messageId":"7va9q72n1u.fsf@alter.siamese.dyndns.org","threadId":"33164","inReplyTo":"A54CE3E330039942B33B670D971F857403995D93@TK5EX14MBXC252.redmond.corp.microsoft.com","subject":"Re: Rename conflicts in the index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-03-13T18:05:01Z","receivedAt":"2013-03-13T18:05:01Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Edward Thomson writes:\n\n> I would propose that we store the data about the file in conflict as it\n> occurred through the renames. For example, in a rename 1->2 conflict where\n> A was renamed to both B and C, you would have a single conflict entry\n> containing the data for A, B and C. This would allow us to provide more\n> detailed information to the user - and allow them to (say) choose a single\n> name to proceed with.\n>\n> Is this something that has value to core git as well? Alternately, is\n> there something particularly stupid about this proposal?\n\nI do not offhand see anything particularly stupid; a new optional\nindex extension section CACHE_EXT_RENAME_CONFLICT might be a good\naddition.\n\nIs \"one side moves A to B while the other side moves it to C\" the\nonly case, or is it just an example? Off the top of my head, \"one\nside moves A to x while the other side moves B to x/y\" would also be\nsomething we would want to know. I am sure there are other cases\nthat need to be considered.\n\nI do not think we can discuss the design at the concrete level until\nthe proposal spells out to cover all interesting cases in order for\nimplementations to agree on the common semantics.\n"},{"id":"211252","messageId":"A54CE3E330039942B33B670D971F85740399654B@TK5EX14MBXC252.redmond.corp.microsoft.com","threadId":"33164","inReplyTo":"7va9q72n1u.fsf@alter.siamese.dyndns.org","subject":"RE: Rename conflicts in the index","fromName":"Edward Thomson","fromEmail":"ethomson@microsoft.com","sentAt":"2013-03-13T20:44:36Z","receivedAt":"2013-03-13T20:44:36Z","isPatch":false,"sender":{"key":"ethomson@microsoft.com","avatar":"https://gravatar.com/avatar/c80dc8bacdf4afcf27334f21b70093a25ac5559d576357ea13592bf438113950?d=mp&s=160"},"body":"Junio C Hamano [mailto:gitster@pobox.com] wrote:\n> I do not offhand see anything particularly stupid; a new optional index extension\n> section CACHE_EXT_RENAME_CONFLICT might be a good addition.\n> \n> Is \"one side moves A to B while the other side moves it to C\" the only case, or is\n> it just an example? Off the top of my head, \"one side moves A to x while the\n> other side moves B to x/y\" would also be something we would want to know. I\n> am sure there are other cases that need to be considered.\n\nYes, that was just an example. Certainly I was intending that all conflicts\nthat arose from renames would end up here since one can't really reason\nwhy the merge tool created a conflict by looking at the index alone - even\nknowing the merge tool's similarity algorithms, this would be awfully\nexpensive to piece back together, even if the index did contain non-zero\nstage entries for all the items that were involved in the conflicts.\n\nThat said, my rather naive initial thought was that we could repeat *all*\nconflicts in this area. This would give tools that knew how to understand\nthis the ability to go to a single place for conflict data, rather than\nproducing some merge of high-stage entries that comprise non-rename\nconflicts and data from the rename conflict area for rename conflicts.\n\nThanks-\n-ed\n"},{"id":"212303","messageId":"A54CE3E330039942B33B670D971F857403A0AA7A@TK5EX14MBXC253.redmond.corp.microsoft.com","threadId":"33164","inReplyTo":"7va9q72n1u.fsf@alter.siamese.dyndns.org","subject":"RE: Rename conflicts in the index","fromName":"Edward Thomson","fromEmail":"ethomson@microsoft.com","sentAt":"2013-03-26T18:30:29Z","receivedAt":"2013-03-26T18:30:29Z","isPatch":false,"sender":{"key":"ethomson@microsoft.com","avatar":"https://gravatar.com/avatar/c80dc8bacdf4afcf27334f21b70093a25ac5559d576357ea13592bf438113950?d=mp&s=160"},"body":"Junio C Hamano [mailto:gitster@pobox.com] wrote:\n> Edward Thomson writes:\n> \n> > I would propose that we store the data about the file in conflict as\n> > it occurred through the renames. For example, in a rename 1->2\n> > conflict where A was renamed to both B and C, you would have a single\n> > conflict entry containing the data for A, B and C. This would allow\n> > us to provide more detailed information to the user - and allow them\n> > to (say) choose a single name to proceed with.\n> >\n> > Is this something that has value to core git as well? Alternately, is\n> > there something particularly stupid about this proposal?\n> \n> I do not offhand see anything particularly stupid; a new optional index extension\n> section CACHE_EXT_RENAME_CONFLICT might be a good addition.\n> \n> Is \"one side moves A to B while the other side moves it to C\" the only case, or is\n> it just an example? Off the top of my head, \"one side moves A to x while the\n> other side moves B to x/y\" would also be something we would want to know. I\n> am sure there are other cases that need to be considered.\n> \n> I do not think we can discuss the design at the concrete level until the proposal\n> spells out to cover all interesting cases in order for implementations to agree on\n> the common semantics.\n\nSorry about the delay here: besides getting busy with some other things,\nI wanted both a complete writeup and to have taken a pass at a test\nimplementation this in libgit2 to make sure seemed like a reasonably sensible\napproach.\n\nI would propose a new extension, 'CONF', to handle conflict data, differing\nfrom the stage >0 entries in the index in that this extension tracks the\nconflicting file across names if the underlying merge engine has support\nfor renames.\n\nI made an attempt to keep the entry data similar to other entries in the\nindex. I would propose that entries in the conflict are as follows:\n\nFlags\n Four octets that describe the conflict. Data includes:\n\n 0x01 HAS_ANCESTOR\n There is a file in the common ancestor branch that contributes\n to this conflict. Its data will follow.\n 0x02 HAS_OURS\n There is a file in \"our\" branch that contributes to this conflict.\n Its data will follow.\n 0x04 HAS_THEIRS\n There is a file in \"their\" branch that contributes to this conflict.\n Its data will follow.\n\n 0x08 NAME_CONFLICT_OURS\n This item has a path in \"our\" branch that overlaps a different\n item in \"their\" branch. (Eg, this conflict represents the \"our\"\n side of a rename/add conflict.)\n 0x10 NAME_CONFLICT_THEIRS\n This item has a path in \"their\" branch that overlaps a different\n item in \"our\" branch. (Eg, this conflict represents the \"theirs\"\n side of a rename/add conflict.)\n\n 0x20 DF_CONFLICT_FILE\n This is the file involved in a directory/file conflict.\n 0x40 DF_CONFLICT_CHILD\n This is a child of a directory involved in a directory/file conflict.\n\n Other bits are reserved.\n\nConflict Sides\n The data about one side of a conflict will contain:\n mode (ASCII string representation of octal, null-terminated)\n path (null terminated)\n sha1 (raw bytes)\n\nThe conflict sides will be written in this order:\n Ancestor (if HAS_ANCESTOR is set)\n Ours (if HAS_OURS is set)\n Theirs (if HAS_THEIRS is set)\n\nI would propose that this not simply track rename conflicts, but all\nconflicts. Having a single canonical location is preferable - if the index\ncontains a CONF section (and the client supports it), it would use that.\nOtherwise, the client would look at stage >0 entries.\n\nI would propose that another extension, 'RSVD', track these conflicts once\nthey are resolved. The format would be the same - when a conflict is\nresolved from the CONF the entry will be placed as-is in the RSVD.\n\nExamples are not an exhaustive list, but should help elucidate the name\nand d/f conflicts:\n\nNormal edit / edit conflict, where A is edited in ours and theirs:\n\n Conflict one:\n Flags = HAS_ANCESTOR|HAS_OURS|HAS_THEIRS\n Entry 1 = A [Ancestor]\n Entry 2 = B [Ancestor]\n Entry 3 = C [Ancestor]\n\nRename / add conflict, where A is renamed to B in ours and B is added in\ntheirs:\n\n Conflict one:\n Flags = HAS_ANCESTOR|HAS_OURS|NAME_CONFLICT_OURS\n Entry 1 = A [Ancestor]\n Entry 2 = B [Ours]\n Entry 3 = A [Theirs]\n Conflict two:\n Flags = HAS_THEIRS|NAME_CONFLICT_THEIRS\n Entry 1 = File B [Theirs]\n\nD/F conflict, where some file A is deleted in theirs, and a directory\nA is created with file child:\n\n Conflict one:\n Flags = HAS_ANCESTOR|HAS_OURS|HAS_THEIRS|DF_CONFLICT_FILE\n Entry 1 = A [Ancestor]\n Entry 2 = A [Ours]\n Conflict two:\n Flags = HAS_THEIRS|DF_CONFLICT_CHILD\n Entry 1 = A/child [Theirs]\n\nThanks for your input on this.\n\n-ed\n"},{"id":"212317","messageId":"7vboa6t14w.fsf@alter.siamese.dyndns.org","threadId":"33164","inReplyTo":"A54CE3E330039942B33B670D971F857403A0AA7A@TK5EX14MBXC253.redmond.corp.microsoft.com","subject":"Re: Rename conflicts in the index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-03-26T19:24:47Z","receivedAt":"2013-03-26T19:24:47Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Edward Thomson writes:\n\n> I would propose a new extension, 'CONF', to handle conflict data, differing\n> from the stage >0 entries in the index in that this extension tracks the\n> conflicting file across names if the underlying merge engine has support\n> for renames.\n>\n> I made an attempt to keep the entry data similar to other entries in the\n> index. I would propose that entries in the conflict are as follows:\n>\n> Flags\n> Four octets that describe the conflict. Data includes:\n>\n> 0x01 HAS_ANCESTOR\n> There is a file in the common ancestor branch that contributes\n> to this conflict. Its data will follow.\n> 0x02 HAS_OURS\n> There is a file in \"our\" branch that contributes to this conflict.\n> Its data will follow.\n> 0x04 HAS_THEIRS\n> There is a file in \"their\" branch that contributes to this conflict.\n> Its data will follow.\n>\n> 0x08 NAME_CONFLICT_OURS\n> This item has a path in \"our\" branch that overlaps a different\n> item in \"their\" branch. (Eg, this conflict represents the \"our\"\n> side of a rename/add conflict.)\n> 0x10 NAME_CONFLICT_THEIRS\n> This item has a path in \"their\" branch that overlaps a different\n> item in \"our\" branch. (Eg, this conflict represents the \"theirs\"\n> side of a rename/add conflict.)\n>\n> 0x20 DF_CONFLICT_FILE\n> This is the file involved in a directory/file conflict.\n> 0x40 DF_CONFLICT_CHILD\n> This is a child of a directory involved in a directory/file conflict.\n>\n> Other bits are reserved.\n>\n> Conflict Sides\n> The data about one side of a conflict will contain:\n> mode (ASCII string representation of octal, null-terminated)\n> path (null terminated)\n> sha1 (raw bytes)\n>\n> The conflict sides will be written in this order:\n> Ancestor (if HAS_ANCESTOR is set)\n> Ours (if HAS_OURS is set)\n> Theirs (if HAS_THEIRS is set)\n\nPuzzled. Most of the above, except NAME_CONFLICT_{OURS,THEIRS}\nbits, look totally pointless duplication.\n\nWhen you are working with Git, you have to be prepared to read from\nthe datafile like the index that other people (and your previous\nversion) created, and you also have to make sure you do not make\nwhat you write out unusable by other people without a good reason.\n\nSo your tool needs code to see higher stage entries in the main\nindex to find for the conflicted paths even without the\nindex extension anyway, and if your tool does also perform merges,\nyou would need to strive for writing the main index with conflicted\nentries and implementations that do not yet understand your\nextension can keep operating. For some types of extensions, the\nlatter may be hard (and that is why I stopped at \"you would need to\nstrive for\", and not \"you must\"), but for the one under discussion,\nI do not think it is the case (by the way \"CONF\" sounds as if it is\nsome sort of configuration data).\n\nIf you are starting a brand new system from scratch, keeping only\nthe resolved entries in the main index and having a separate section\nfor conflicts might be also a valid design choice, but you do not\nlive in that world if you are discussing the design on this mailing\nlist.\n\n> I would propose that this not simply track rename conflicts, but all\n> conflicts.\n\nThat is a no starter.\n"},{"id":"212331","messageId":"A54CE3E330039942B33B670D971F857403A0AC1E@TK5EX14MBXC253.redmond.corp.microsoft.com","threadId":"33164","inReplyTo":"7vboa6t14w.fsf@alter.siamese.dyndns.org","subject":"RE: Rename conflicts in the index","fromName":"Edward Thomson","fromEmail":"ethomson@microsoft.com","sentAt":"2013-03-26T20:40:37Z","receivedAt":"2013-03-26T20:40:37Z","isPatch":false,"sender":{"key":"ethomson@microsoft.com","avatar":"https://gravatar.com/avatar/c80dc8bacdf4afcf27334f21b70093a25ac5559d576357ea13592bf438113950?d=mp&s=160"},"body":"\nJunio C Hamano [mailto:gitster@pobox.com] wrote:\n> Edward Thomson writes:\n> > I would propose a new extension, 'CONF', to handle conflict data,\n> > differing from the stage >0 entries in the index in that this\n> > extension tracks the conflicting file across names if the underlying\n> > merge engine has support for renames.\n> >\n> > I made an attempt to keep the entry data similar to other entries in\n> > the index. I would propose that entries in the conflict are as follows:\n> >\n> > Flags\n> > Four octets that describe the conflict. Data includes:\n> >\n> > 0x01 HAS_ANCESTOR\n> > There is a file in the common ancestor branch that contributes\n> > to this conflict. Its data will follow.\n> > 0x02 HAS_OURS\n> > There is a file in \"our\" branch that contributes to this conflict.\n> > Its data will follow.\n> > 0x04 HAS_THEIRS\n> > There is a file in \"their\" branch that contributes to this conflict.\n> > Its data will follow.\n> >\n> > 0x08 NAME_CONFLICT_OURS\n> > This item has a path in \"our\" branch that overlaps a different\n> > item in \"their\" branch. (Eg, this conflict represents the \"our\"\n> > side of a rename/add conflict.)\n> > 0x10 NAME_CONFLICT_THEIRS\n> > This item has a path in \"their\" branch that overlaps a different\n> > item in \"our\" branch. (Eg, this conflict represents the \"theirs\"\n> > side of a rename/add conflict.)\n> >\n> > 0x20 DF_CONFLICT_FILE\n> > This is the file involved in a directory/file conflict.\n> > 0x40 DF_CONFLICT_CHILD\n> > This is a child of a directory involved in a directory/file conflict.\n> >\n> > Other bits are reserved.\n> >\n> > Conflict Sides\n> > The data about one side of a conflict will contain:\n> > mode (ASCII string representation of octal, null-terminated)\n> > path (null terminated)\n> > sha1 (raw bytes)\n> >\n> > The conflict sides will be written in this order:\n> > Ancestor (if HAS_ANCESTOR is set)\n> > Ours (if HAS_OURS is set)\n> > Theirs (if HAS_THEIRS is set)\n> \n> Puzzled. Most of the above, except NAME_CONFLICT_{OURS,THEIRS} bits, look\n> totally pointless duplication.\n\nObviously HAS_ANCESTOR / HAS_OURS / HAS_THEIRS is to indicate to a reader\nwhether there is data to be read or not. Similar to how a mode of 0\nin the REUC indicates that the rest of the record should not be read.)\n\n> When you are working with Git, you have to be prepared to read from the\n> datafile like the index that other people (and your previous\n> version) created, and you also have to make sure you do not make what you\n> write out unusable by other people without a good reason.\n\nI'm acutely aware that you need to be able to read an index that other\npeople created - that's the problem at hand. git does not produce an\nindex that allows anyone (including itself) to reason about rename\nconflicts. It doesn't even bother to write high-stage conflict entries\nin many instances, so you can have an instance where git tells you that\na conflict occurred but one of those files is staged anyway, the other\nis just dirty in the workdir and you can commit immediately thereafter.\n\nWhile obviously it's possible to handle this situation (is file A\nin conflict? Look in the rename conflict extension. Not there? Okay,\nlook in the index.) That's not exactly elegant. My goal here was\nto have a single source for conflicts.\n\n> (by the way \"CONF\" sounds as if it is some sort of configuration data).\n\nThere's only four letters, and not everything's as easy as \"TREE\". REUC,\nfor example, sounds like a donkey, though I suppose it depends on the\nlanguage in question.\n\n-ed\n"},{"id":"212405","messageId":"A54CE3E330039942B33B670D971F857403A0BA54@TK5EX14MBXC253.redmond.corp.microsoft.com","threadId":"33164","inReplyTo":"7vboa6t14w.fsf@alter.siamese.dyndns.org","subject":"RE: Rename conflicts in the index","fromName":"Edward Thomson","fromEmail":"ethomson@microsoft.com","sentAt":"2013-03-27T17:03:40Z","receivedAt":"2013-03-27T17:03:40Z","isPatch":false,"sender":{"key":"ethomson@microsoft.com","avatar":"https://gravatar.com/avatar/c80dc8bacdf4afcf27334f21b70093a25ac5559d576357ea13592bf438113950?d=mp&s=160"},"body":"Junio C Hamano [mailto:gitster@pobox.com] wrote:\n> Edward Thomson writes:\n> > I would propose that this not simply track rename conflicts, but all\n> > conflicts.\n> \n> That is a no starter.\n\nSo. Can you explain to me why this would be a non starter? Can you suggest\nsome alternate strategy here?\n\nMaybe there's something I'm fundamentally misunderstanding. It seems that\nat present, git will:\n\n1. Detect rename conflicts when performing a merge (at least,\n git-merge-recursive will, which is the default.)\n\n2. If the rename itself caused a conflict (eg, renamed in one side, added in\n the other) then the merge cannot succeed.\n\n3. The resultant index is written as if renames were not detected, which\n means - at best - records the files that went in to the name conflict\n and git status reports an \"added in ours\" conflict, which is a pretty\n disappointing conflict. Often, though, many of the files will not\n exist at higher stage entries, since without rename detection, they\n would have not been conflicts. At worst, one side is staged, there are\n no conflicts in the index and the user can commit (and thus lose the\n other side.)\n\nThus it's not like we could add some extension that merely records the names\nthat produced the rename conflicts and point them at the higher stage entries\nin the index. That would require that they actually be in the index.\n\nThanks-\n-ed\n"},{"id":"212416","messageId":"7vk3osn3vs.fsf@alter.siamese.dyndns.org","threadId":"33164","inReplyTo":"A54CE3E330039942B33B670D971F857403A0BA54@TK5EX14MBXC253.redmond.corp.microsoft.com","subject":"Re: Rename conflicts in the index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-03-27T17:34:15Z","receivedAt":"2013-03-27T17:34:15Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Edward Thomson writes:\n\n> Junio C Hamano [mailto:gitster@pobox.com] wrote:\n>> Edward Thomson writes:\n>> > I would propose that this not simply track rename conflicts, but all\n>> > conflicts.\n>> \n>> That is a no starter.\n>\n> So. Can you explain to me why this would be a non starter?\n\nAt least two, IIRC. One is the consequence of the other.\n\nWe do not gratuitously break existing implementations. If no\nconflict is stored as higher-stage index entries in an index that\nhas your index extension, no existing implementation can read a\nconflicted index written by your implementation and have users\nresolve conflicts.\n\nWhen a path originally at A is moved to B on only one branch, and\nthere are content-level conflicts between the changes made by one\nbranch (while going from A to B) and by the other branch (while\nkeeping it at A), we would end up having three stages for path B\nwithout any trace of path A. I do not offhand know how much it\nhelps to learn A in such a situation in the real life, but we are\nindeed losing information, and I do not have any problem with an\nextension that records in the index the fact that in the two (of the\nthree) commits involved in the merge, the path was at A.\n\nBut people have been successfully using existing versions of Git\nwithout that information to merge branches with renames, and\nresolving the content-level conflicts. Your tool that\n_additionally_ records \"This path that currently has three stages\nfor B was at A in the common ancestor (i.e. stage #1) and that\nbranch (either stage #2 or stage #3)\" does not _have_ _to_ break\nthese users by removing the three stages for B from the main index.\n\nAlso we do not duplicate information unnecessarily. Nowhere in the\nabove \"we have been losing the fact that two of the three had the\ncontents we have at path B in the resulting unmerged index at path\nA, and that information might be useful as well\", there is a reason\nto write another copy of mode or SHA-1 for any of the three variants.\n\nAs I said, you do not live in the world where you are writing\nsomething like Git from scratch. Perhaps you do, but then the\nresult will not be Git and we wouldn't be discussing that system on\nthis mailing list.\n"},{"id":"212430","messageId":"A54CE3E330039942B33B670D971F857403A0BB93@TK5EX14MBXC253.redmond.corp.microsoft.com","threadId":"33164","inReplyTo":"7vk3osn3vs.fsf@alter.siamese.dyndns.org","subject":"RE: Rename conflicts in the index","fromName":"Edward Thomson","fromEmail":"ethomson@microsoft.com","sentAt":"2013-03-27T18:53:07Z","receivedAt":"2013-03-27T18:53:07Z","isPatch":false,"sender":{"key":"ethomson@microsoft.com","avatar":"https://gravatar.com/avatar/c80dc8bacdf4afcf27334f21b70093a25ac5559d576357ea13592bf438113950?d=mp&s=160"},"body":"Junio C Hamano [mailto:gitster@pobox.com] wrote:\n> We do not gratuitously break existing implementations. If no conflict is stored\n> as higher-stage index entries in an index that has your index extension, no\n> existing implementation can read a conflicted index written by your\n> implementation and have users resolve conflicts.\n\nI'm not suggesting that anybody stop writing >0 stage entries.\n\n> When a path originally at A is moved to B on only one branch, and there are\n> content-level conflicts between the changes made by one branch (while going\n> from A to B) and by the other branch (while keeping it at A), we would end up\n> having three stages for path B without any trace of path A. I do not offhand\n> know how much it helps to learn A in such a situation in the real life, but we are\n> indeed losing information, and I do not have any problem with an extension that\n> records in the index the fact that in the two (of the\n> three) commits involved in the merge, the path was at A.\n\nWhat you've described is true only for a certain class of rename conflicts,\nfor example the rename/edit conflict you've described above.\n\nIt's also true if you were to rename some item 'a' to 'b' in both branches.\nBut when 'b' is sufficiently dissimilar to become a rewrite, then I end up\nwith a rename of a->b on one side and deleting a and adding b on the other.\nThe result is a mysterious \"added by us\" conflict:\n\n100644 e2dd530c9f31550a2b0c90773ccde056929d6d66 2 b\n\nWorse yet is if I don't do the rename in my side, but I just add a new b so\nthat in theirs I've renamed a to b and in mine I have both a and b. When I\ndo the merge, I'm told I have conflicts, except that I don't:\n\n100644 08d4f831774aed5d4c6cb496affefd4020dce40c 0 b\n\nThe other branch's b is long gone and exists only as a dirty file in the\nworkdir.\n\n> But people have been successfully using existing versions of Git without that\n> information to merge branches with renames, and resolving the content-level\n> conflicts.\n\nBut you aren't afforded the option to resolve content-level conflicts if you\ndon't know where the conflict came from. For example, in a rename 1->2\nconflict, we dutifully detect that a was renamed to both b and c and fail,\nbut that fact is never given to the index. This conflict could be fed into\na merge tool or, better, automerged, with the user only needing to pick a\npath:\n\n100644 421c9102b8562ad227ba773ab1cf6bbed7b7496d 1 a\n100644 421c9102b8562ad227ba773ab1cf6bbed7b7496d 3 b\n100644 421c9102b8562ad227ba773ab1cf6bbed7b7496d 2 c\n\nI hate to sound like a broken record here, but without some more data in the\nindex - anywhere, really - any tool that doesn't have the luxury of emitting\ndata about what happened to stdout certainly can't infer anything about what\nhappened in the merge.\n\n-ed\n"},{"id":"212437","messageId":"7v38vgmxty.fsf@alter.siamese.dyndns.org","threadId":"33164","inReplyTo":"A54CE3E330039942B33B670D971F857403A0BB93@TK5EX14MBXC253.redmond.corp.microsoft.com","subject":"Re: Rename conflicts in the index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-03-27T19:44:57Z","receivedAt":"2013-03-27T19:44:57Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Edward Thomson writes:\n\n> Junio C Hamano [mailto:gitster@pobox.com] wrote:\n>> We do not gratuitously break existing implementations. If no conflict is stored\n>> as higher-stage index entries in an index that has your index extension, no\n>> existing implementation can read a conflicted index written by your\n>> implementation and have users resolve conflicts.\n>\n> I'm not suggesting that anybody stop writing >0 stage entries.\n\nAh, OK, then I misread your original message. You said\n\n> Having a single canonical location is preferable - if the index\n> contains a CONF section (and the client supports it), it would use\n> that. Otherwise, the client would look at stage >0 entries.\n\nwhich I read as \"an index with this extension would not have higher\nstage entries, an index without the extension records higher stage\nentries\". As long as the format will be backward compatible to\nallow existing users use existing tools to deal with cases the\nexisting tools can handle, then that is OK. I didn't get that\nimpression which is where my \"non starter\" came from.\n\n> What you've described is true only for a certain class of rename conflicts,\n> for example the rename/edit conflict you've described above.\n\nAs you asked me to explain why it was a non starter, I only\nillustrated with a \"renamed trivially, with content level conflict\"\nexample that shows why dropping higher-stage entries in the main\nindex would not be acceptable. The previous message did not even\nmean to cover any cases the *new* feature you have in mind is trying\nto address. Again, if it hurts existing users handling cases\nexisting tools used to handle, that makes it a non starter.\n\nHow new feature is designed, and extension is added to help that new\nfeature, is a different matter. My original \"That's a non starter\"\nmessage didn't even go that far.\n\nIn any case, the principle of \"always record the state 'merge'\nstopped to ask for help as higher stage entries to give existing\ntools and users a chance to manually resolve, and augment with\noptional extension to record additional information that might help,\nbut do not gratiutously waste bytes on redundant information\" would\napply to other exotic cases you would want to tackle with the new\nfeature, I would think.\n\nIf one branch moves path A in the original to path B and the other\none moved it to path C, for example, we can record it in different\nways, even in the main index.\n\n * Path A may have only stage #1, while path B and C has only stage\n #2 and stage #3 (the user would have to notice these three\n correspond to each other, and resolve manually).\n\n You would want to annotate \"B at stage #2 seems to have been at A\n in the original\" (similarly for C#3) if you choose to do so.\n\n * You can choose to favor \"our\" choice, and have path B with three\n stages (if we guessed wrong and the user wants to move it to C,\n the user can resolve and then \"git mv\" the path).\n\n You would want to annotate \"the other side wanted to have B at\n stage #3 at C\" in that case.\n\n * Or you may want to have in the main index both B and C (but not\n A) with all three stages (the user would have to choose which\n one survives, but discarding the other side with \"git rm\" would\n be easy).\n\n You would want to annotate the origin of the stage #1 for path B\n and C (these were originally at A), stage #2 for B (the other\n branch wants to have it at C), stage #3 for C (we want to have it\n at B).\n\nThere may be other ways, and I do not offhand know what the current\nmerge-recursive implementation does, but both of the latter two\nsound equally usable and reasonable ways, even without the\nannotation. And with your annotation that records different paths,\nthe conflict may become even easier to resolve.\n\nI still do not need to duplicate in the extensions to\ndo the above, or do I?\n\nIf the original path A was removed and a new path B was added, with\ncontents that are modified from A beyond recognition, at the merge\ntime you wouldn't know where B it came from or where A went, so\nannotating A at stage #1 to say \"it went to B\" is a nonsense. If\nyou have algorithm to do so [*1*], you would be better off detecting\nit as a rename.\n\n\n[Footnote]\n\n*1* Instead of a three-way merge that inspects only the endpoints,\n you might get a better rename trail if you looked at the\n histories of both branches. It would be a lot more expensive\n than the simple three-way, but burning CPU cycles is better than\n burning human neurons.\n"},{"id":"212800","messageId":"A54CE3E330039942B33B670D971F857403A0E20C@TK5EX14MBXC253.redmond.corp.microsoft.com","threadId":"33164","inReplyTo":"7v38vgmxty.fsf@alter.siamese.dyndns.org","subject":"RE: Rename conflicts in the index","fromName":"Edward Thomson","fromEmail":"ethomson@microsoft.com","sentAt":"2013-04-01T16:14:52Z","receivedAt":"2013-04-01T16:14:52Z","isPatch":false,"sender":{"key":"ethomson@microsoft.com","avatar":"https://gravatar.com/avatar/c80dc8bacdf4afcf27334f21b70093a25ac5559d576357ea13592bf438113950?d=mp&s=160"},"body":"Junio C Hamano [mailto:gister@pobox.com] wrote:\n> As long as\n> the format will be backward compatible to allow existing users use existing tools\n> to deal with cases the existing tools can handle, then that is OK. I didn't get that\n> impression which is where my \"non starter\" came from.\n\nI see now. Thank you for the clarification. I apologize if I was not clear\nabout this; indeed, the duplication of data in my proposed extension was\nspecifically to avoid any compatibility problems amongst clients.\n\nIn particular, when we have a rename in ours, edit in theirs conflict, we\nstore the conflict at the new (ours) path. If, for example, I rename a->b\nin my branch and merge a branch that edits a:\n\nmode hash 1 b\nmode hash 2 b\nmode hash 3 b\n\nThis prohibits us from storing anything else in the theirs side at that\npath, so if I were to have added b in their branch in addition to modifying\nb, I cannot record it.\n\nI was assuming that any change to this behavior would be a breaking one,\nwhich is where the new section came from.\n\n> * Path A may have only stage #1, while path B and C has only stage\n> #2 and stage #3 (the user would have to notice these three\n> correspond to each other, and resolve manually).\n> \n> You would want to annotate \"B at stage #2 seems to have been at A\n> in the original\" (similarly for C#3) if you choose to do so.\n\nIf we're going to make changes to the way conflicts are recorded in the\nmain index, then I would prefer this approach. It is unambiguous and all\ndata about all sides are recorded, including the names that items had in\ntheir respective branches.\n\nI would think that this might be a burden on current tools, however.\nNow if I rename a->b my just my branch, my conflict will be recorded as:\n\nmode hash 1 a\nmode hash 2 b\nmode hash 3 a\n\nAnd current git-status will not look at any rename annotations to know\nhow to report this.\n\nHowever, maybe this is not as big a problem as I'm concerned it would be.\n\n> * You can choose to favor \"our\" choice, and have path B with three\n> stages (if we guessed wrong and the user wants to move it to C,\n> the user can resolve and then \"git mv\" the path).\n\nI think this approach suffers from the drawback that the current approach\nhas, wherein this conflicts when they had path B, also, as noted above.\n\nI think that if you were to put both B and C with all three stages, this\nwould be problematic for the same reason.\n\n> *1* Instead of a three-way merge that inspects only the endpoints,\n> you might get a better rename trail if you looked at the\n> histories of both branches. It would be a lot more expensive\n> than the simple three-way, but burning CPU cycles is better than\n> burning human neurons.\n\nFor the record, I like this approach very much. It's not something that\nlibgit2 will be able to tackle in the near future; we're in a sort of\nwalk-before-you-can-run situation with merge at the moment, as you can\nprobably see. But any improvement that avoids burning neurons is a\nvaluable one.\n\nThanks-\n\n-ed\n"},{"id":"212992","messageId":"A54CE3E330039942B33B670D971F857403A0F540@TK5EX14MBXC253.redmond.corp.microsoft.com","threadId":"33164","inReplyTo":"A54CE3E330039942B33B670D971F857403A0E20C@TK5EX14MBXC253.redmond.corp.microsoft.com","subject":"RE: Rename conflicts in the index","fromName":"Edward Thomson","fromEmail":"ethomson@microsoft.com","sentAt":"2013-04-02T20:55:37Z","receivedAt":"2013-04-02T20:55:37Z","isPatch":false,"sender":{"key":"ethomson@microsoft.com","avatar":"https://gravatar.com/avatar/c80dc8bacdf4afcf27334f21b70093a25ac5559d576357ea13592bf438113950?d=mp&s=160"},"body":"Edward Thomson [ethomson@microsoft.com] wrote:\n> Junio C Hamano [mailto:gister@pobox.com] wrote:\n> > * Path A may have only stage #1, while path B and C has only stage\n> > #2 and stage #3 (the user would have to notice these three\n> > correspond to each other, and resolve manually).\n> >\n> > You would want to annotate \"B at stage #2 seems to have been at A\n> > in the original\" (similarly for C#3) if you choose to do so.\n> \n> If we're going to make changes to the way conflicts are recorded in the main\n> index, then I would prefer this approach. It is unambiguous and all data about\n> all sides are recorded, including the names that items had in their respective\n> branches.\n\nJunio, did you have additional thoughts on this?\n\nWhat would you like from me to proceed? If the aforementioned seems\nreasonable, I can update Documentation/technical/index-format.txt and\nwe can iron out the details in that fashion?\n\nThanks-\n-ed\n"},{"id":"212995","messageId":"7vr4isaath.fsf@alter.siamese.dyndns.org","threadId":"33164","inReplyTo":"A54CE3E330039942B33B670D971F857403A0F540@TK5EX14MBXC253.redmond.corp.microsoft.com","subject":"Re: Rename conflicts in the index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-02T21:20:58Z","receivedAt":"2013-04-02T21:20:58Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Edward Thomson writes:\n\n> Edward Thomson [ethomson@microsoft.com] wrote:\n>> Junio C Hamano [mailto:gister@pobox.com] wrote:\n>> > * Path A may have only stage #1, while path B and C has only stage\n>> > #2 and stage #3 (the user would have to notice these three\n>> > correspond to each other, and resolve manually).\n>> >\n>> > You would want to annotate \"B at stage #2 seems to have been at A\n>> > in the original\" (similarly for C#3) if you choose to do so.\n>> \n>> If we're going to make changes to the way conflicts are recorded in the main\n>> index, then I would prefer this approach. It is unambiguous and all data about\n>> all sides are recorded, including the names that items had in their respective\n>> branches.\n>\n> Junio, did you have additional thoughts on this?\n\nNot at this moment.\n\nI think we have covered the principles (do not unnecessarily\nduplicate information, do not break existing implementations\nunnecessarily, etc.) already, and we know how we want to record \"one\nside renamed A to B, the other side renamed A to C\" case, but I do\nnot think the discussion covered all cases yet.\n"},{"id":"212996","messageId":"A54CE3E330039942B33B670D971F857403A0F593@TK5EX14MBXC253.redmond.corp.microsoft.com","threadId":"33164","inReplyTo":"7vr4isaath.fsf@alter.siamese.dyndns.org","subject":"RE: Rename conflicts in the index","fromName":"Edward Thomson","fromEmail":"ethomson@microsoft.com","sentAt":"2013-04-02T21:29:30Z","receivedAt":"2013-04-02T21:29:30Z","isPatch":false,"sender":{"key":"ethomson@microsoft.com","avatar":"https://gravatar.com/avatar/c80dc8bacdf4afcf27334f21b70093a25ac5559d576357ea13592bf438113950?d=mp&s=160"},"body":"Junio C Hamano [mailto:gitster@pobox.com] wrote:\n> Edward Thomson writes:\n> > Junio, did you have additional thoughts on this?\n> \n> Not at this moment.\n> \n> I think we have covered the principles (do not unnecessarily duplicate\n> information, do not break existing implementations unnecessarily, etc.) already,\n> and we know how we want to record \"one side renamed A to B, the other side\n> renamed A to C\" case, but I do not think the discussion covered all cases yet.\n\nSorry, I'm not sure what you're asking for - would you just like some more\nexamples of what this looks like with aforementioned exotic conflict types?\nOr are you looking for something more strict - BNF format, for example?\n\n-ed\n"},{"id":"213013","messageId":"7vsj3877u8.fsf@alter.siamese.dyndns.org","threadId":"33164","inReplyTo":"7vr4isaath.fsf@alter.siamese.dyndns.org","subject":"Re: Rename conflicts in the index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-03T00:53:35Z","receivedAt":"2013-04-03T00:53:35Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Edward Thomson writes:\n\n> Junio C Hamano [mailto:gitster@pobox.com] wrote:\n>> Edward Thomson writes:\n>> > Junio, did you have additional thoughts on this?\n>> \n>> Not at this moment.\n>> \n>> I think we have covered the principles (do not unnecessarily duplicate\n>> information, do not break existing implementations unnecessarily, etc.) already,\n>> and we know how we want to record \"one side renamed A to B, the other side\n>> renamed A to C\" case, but I do not think the discussion covered all cases yet.\n>\n> Sorry, I'm not sure what you're asking for - would you just like some more\n> examples of what this looks like with aforementioned exotic conflict types?\n> Or are you looking for something more strict - BNF format, for example?\n\nEhh, I wasn't asking for anything ;-)\n\nYou asked if I had any additional thoughts, I answered there is\nnothing at this moment based on what I saw so far. It is not my\nimmediate itch to update the index with more rename information, but\nit is yours, so I would imagine you would know what cases you would\nwant to improve the end user experience better than I do ;-).\n\nIf I were solving the issue, I would probably proceed like this:\n\n * Start from a rough sketch of what extra information I would want\n to store in the new index extension section.\n\n * Teach read-cache.c to read from the new extension and keep it in\n an in-core data structure, and read from the in-core data\n structure and seriealize it to write to the extension section.\n\n * Perhaps enhance \"update-index\" so that it can read textual\n representation of the contents of the new extension section, turn\n it into the in-core representation, so that it can write it out\n to the index file, as a debugging/development aid.\n\n * Teach read-cache.c to read from the new extension and keep it in\n an in-core data structure.\n\n * Teach wt-status.c to read from that in-core data structure and\n improve the presentation of the cases I care about using that\n information. Use the \"update-index\" development aid to prepare\n various cases you care about.\n\n - If the kind of information that is stored in the new extension\n turns out to be insufficient, go back to the beginning and\n iterate.\n\n - If the use the in-core data structure here turns out to be\n awkward, go back one step and iterate.\n\n - As I cover one more case, I would add a test to the test suite\n so that we would know what cases are covered and what the\n expected end-user presentation should be.\n\n * Once the result of the above covers all the cases I care about,\n then update merge-recursive.c to prepare the in-core data\n structure to be written out as the extension section.\n\nAs I iterate, the rough sketch will hopefully cover all the cases I\ncare about and I'll be ready to write them down as an update to the\ndocument somewhere in Documentation/technical/api-*.\n\nThanks.\n"},{"id":"213014","messageId":"7vobdw77n5.fsf@alter.siamese.dyndns.org","threadId":"33164","inReplyTo":"7vsj3877u8.fsf@alter.siamese.dyndns.org","subject":"Re: Rename conflicts in the index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-03T00:57:50Z","receivedAt":"2013-04-03T00:57:50Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> If I were solving the issue, I would probably proceed like this:\n>\n> * Start from a rough sketch of what extra information I would want\n> to store in the new index extension section.\n>\n> * Teach read-cache.c to read from the new extension and keep it in\n> an in-core data structure, and read from the in-core data\n> structure and seriealize it to write to the extension section.\n>\n> * Perhaps enhance \"update-index\" so that it can read textual\n> representation of the contents of the new extension section, turn\n> it into the in-core representation, so that it can write it out\n> to the index file, as a debugging/development aid.\n>\n> * Teach read-cache.c to read from the new extension and keep it in\n> an in-core data structure.\n\nSorry, this is a dup of the second one. Please ignore.\n\nAlso, all \"you\" in this section should read \"I\" (because this is a\ndescription of \"If I were solving it\").\n\n> * Teach wt-status.c to read from that in-core data structure and\n> improve the presentation of the cases I care about using that\n> information. Use the \"update-index\" development aid to prepare\n> various cases you care about.\n>\n> - If the kind of information that is stored in the new extension\n> turns out to be insufficient, go back to the beginning and\n> iterate.\n>\n> - If the use the in-core data structure here turns out to be\n> awkward, go back one step and iterate.\n>\n> - As I cover one more case, I would add a test to the test suite\n> so that we would know what cases are covered and what the\n> expected end-user presentation should be.\n>\n> * Once the result of the above covers all the cases I care about,\n> then update merge-recursive.c to prepare the in-core data\n> structure to be written out as the extension section.\n>\n> As I iterate, the rough sketch will hopefully cover all the cases I\n> care about and I'll be ready to write them down as an update to the\n> document somewhere in Documentation/technical/api-*.\n>\n> Thanks.\n"}]}