{"thread":{"id":"59883","subject":"[Bug?] Information around newlines lost in merge","startedAt":"2023-06-20T09:22:53Z","lastAt":"2023-06-21T11:41:42Z","messageCount":3,"participants":["Karthik Nayak","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"478537","messageId":"CAOLa=ZRumrpfG8FxQQG=Q8tGvxEapMvOx6SZyRPh0GSpn5iuUg@mail.gmail.com","threadId":"59883","inReplyTo":null,"subject":"[Bug?] Information around newlines lost in merge","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2023-06-20T09:22:23Z","receivedAt":"2023-06-20T09:22:53Z","isPatch":false,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"When merging two files which contain newlines at the end, the blob\ncreated (with conflicts) is the\nsame as two files created without newlines at the end.\n\nWe can demonstrate the same using this simple script which uses\n`git-merge-tree(1)`. The script\ncreates two files in master, newBranch and then merges the tree. The\ncreated files seem to have\nthe same content, eventhough the source files differed by newlines.\n\n#!/usr/bin/env bash\n\ngit init\ngit commit --allow-empty -m \"base commit\"\n\ngit branch newBranch\n\necho -ne \"\\nA\\n\" > newline\necho -ne \"\\nA\" > half\ngit add .\ngit commit -m \"master commit\"\n\ngit checkout newBranch\n\necho -ne \"\\nB\\n\" > newline\necho -ne \"\\nB\" > half\ngit add .\ngit commit -m \"branch commit\"\n\ntreeOID=$(git merge-tree master newBranch | head -n1)\nhalfOID=$(git cat-file -p $treeOID | grep half | awk '{print $3}')\nnewlineOID=$(git cat-file -p $treeOID | grep newline | awk '{print $3}')\n\nif [[ $halfOID == $newlineOID ]]; then\n exit 1\nelse\n exit 0\nfi\n\nIs this expected behavior? Shouldn't the two merged files (newline,\nhalf) differ? i.e., shouldn't the file (newline) with newlines at the\nend\ncontain an extra newline compared to the file without the newline? (half).\n\nIf this is expected behavior, what would be the best way to\ndifferentiate the two? This is not a bug introduced, but rather the\nbehavior since,\nthe start, which makes me think that I'm missing something (verified\nvia git bisect on latest git).\n\nThe context of this is that it is hard to programmatically resolve\nconflicts without loosing information, since we don't know if there\nshould\nbe a newline suffix in the file or not.\n"},{"id":"478570","messageId":"xmqqh6r2ni4i.fsf@gitster.g","threadId":"59883","inReplyTo":"CAOLa=ZRumrpfG8FxQQG=Q8tGvxEapMvOx6SZyRPh0GSpn5iuUg@mail.gmail.com","subject":"Re: [Bug?] Information around newlines lost in merge","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-20T17:44:29Z","receivedAt":"2023-06-20T17:44:43Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> When merging two files which contain newlines at the end, the blob\n> created (with conflicts) is the same as two files created without\n> newlines at the end.\n>\n> If this is expected behavior, what would be the best way to\n> differentiate the two? This is not a bug introduced, but rather the\n> behavior since,\n> the start, which makes me think that I'm missing something (verified\n> via git bisect on latest git).\n\nStrictly speaking, I suspect that the behaviour was different before\nwe introduced in-core 3-way merges of two blobs---back then we ran\nthe \"merge\" program (from the RCS suite).\n\nIf we start from an empty file and have two sides add different\nincomplete lines (i.e. your \"half\" example, but without the leading\nblank line), i.e.\n\n\t$ >O\n\t$ M=\"with a single line added by side %s (without terminating LF)\"\n\t$ printf \"$M\" A >A\n\t$ printf \"$M\" B >B\n\nThe original \"git merge\" that used the external \"merge\" program\nwould have produced this:\n\n\t$ merge -p B O A 2>E\n <<<<<<< B\n with a single line added by side B (without terminating LF).=======\n with a single line added by side A (without terminating LF).>>>>>>> A\n\t$ cat E\n\tmerge: warning: conflicts during merge\n\nThat is, the output would be a mess that cannot even be machine\nparsed. It probably has changed in a slightly improved way when we\nswitched to our own internal 3-way merge of two blobs, exposed as\nthe \"git merge-file\", which gives you:\n\n $ git merge-file -p A O B\n <<<<<<< A\n with a single line added by side A (without terminating LF).\n ||||||| O\n =======\n with a single line added by side B (without terminating LF).\n >>>>>>> B\n\nAnd as you found out, if we added terminating LF to A and/or B, the\noutput would be the same. You could argue that the result is at\nleast machine parseable, instead of the output that is more faithful\nto the input (which we've seen above, in the output from \"merge\").\n\nAs \"7 repeated marker characters followed by a random label string\"\nthe merge machinery inserts cannot be relied on if you are building\na truly automated conflict resolver, lack of this one bit of\ninformation each from both sides may be the least of your problems,\nbut what it means at the same time is that you _could_ propose an\naugmented output format, perhaps like this one:\n\n $ git merge-file -p A O B\n <<<<<<< A\n with a single line added by side A (without terminating LF).\n\t\\No newline at end of file\n ||||||| O\n =======\n with a single line added by side B (without terminating LF).\n\t\\No newline at end of file\n >>>>>>> B\n\nIt has exactly the same problem we already have as these conflict\nsection separator lines in that lines that exactly would look like\nthese extra lines _could_ exist in the payload, so it is not\ncreating a new problem, but people may have built and are happy\nenough with their incomplete automation that relies on the faulty\nassumption that the merged payload would never contain lines that\nare mistaken as conflict section separator lines, and such an\naugmented output format _will_ be a breaking change to them.\n\nSo, I dunno.\n"},{"id":"478633","messageId":"CAOLa=ZTBkzc78RpAdeQO2yN_itDbmKvkQ91RQjENwD9Y8uwR+w@mail.gmail.com","threadId":"59883","inReplyTo":"xmqqh6r2ni4i.fsf@gitster.g","subject":"Re: [Bug?] Information around newlines lost in merge","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2023-06-21T11:41:10Z","receivedAt":"2023-06-21T11:41:42Z","isPatch":false,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"On Tue, Jun 20, 2023 at 7:44 PM Junio C Hamano wrote:\n>\n> $ git merge-file -p A O B\n> <<<<<<< A\n> with a single line added by side A (without terminating LF).\n> \\No newline at end of file\n> ||||||| O\n> =======\n> with a single line added by side B (without terminating LF).\n> \\No newline at end of file\n> >>>>>>> B\n>\n> It has exactly the same problem we already have as these conflict\n> section separator lines in that lines that exactly would look like\n> these extra lines _could_ exist in the payload, so it is not\n> creating a new problem, but people may have built and are happy\n> enough with their incomplete automation that relies on the faulty\n> assumption that the merged payload would never contain lines that\n> are mistaken as conflict section separator lines, and such an\n> augmented output format _will_ be a breaking change to them.\n>\n> So, I dunno.\n\nThank you for taking the time and responding. I'm wondering if there is merit\nin modifying the merge algo.\n\nPerhaps something where files merged with terminating LF would retain it.\nSo merging \"A\\n\" and \"B\\n\" would produce\n\"<<<<<<< master\\nA=======\\nB\\n>>>>>>> newBranch\\n\\n\", whereas\nfiles being merged without a terminating LF wouldn't, so merging \"A\"\nand \"B\" would\nproduce \"<<<<<<< master\\nA=======\\nB\\n>>>>>>> newBranch\\n\". Which\nwould make it easier to parse.\n\nBut overall, I get what you're saying. I will drop it here as expected behavior!\n"}]}