{"thread":{"id":"39124","subject":"[BUG] Git does not convert CRLF=>LF on files with \\r not before \\n","startedAt":"2015-04-21T13:51:34Z","lastAt":"2015-04-22T17:42:53Z","messageCount":5,"participants":["Alexandre Garnier","Junio C Hamano","Torsten Bögershausen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"259736","messageId":"CAFFOgCUTxnbL7vJpf1Hw39CJL_p2raDZ2a3DehhYhdbkVyi2fw@mail.gmail.com","threadId":"39124","inReplyTo":null,"subject":"[BUG] Git does not convert CRLF=>LF on files with \\r not before \\n","fromName":"Alexandre Garnier","fromEmail":"zigarn@gmail.com","sentAt":"2015-04-21T13:51:34Z","receivedAt":"2015-04-21T13:51:34Z","isPatch":false,"sender":{"key":"zigarn@gmail.com","avatar":"https://gravatar.com/avatar/a7916de623e2296b522dd0275b7e64e6d8eb8f6875c9b61c1a3f84498d43c750?d=mp&s=160"},"body":"Here is a test:\n\ngit init -q crlf-test\ncd crlf-test\necho '*       text=auto' > .gitattributes\ngit add .gitattributes\ngit commit -q -m \"Normalize EOL\"\necho -ne 'some content\\r\\nother \\rcontent with CR\\r\\ncontent\\r\\nagain\ncontent with\\r\\r\\n' > inline-cr.txt\necho \"Working directory content:\"\ncat -A inline-cr.txt\necho\ngit add inline-cr.txt\necho \"Indexed content:\"\ngit show :inline-cr.txt | cat -A\n\nResult\n------\nFile content:\nsome content^M$\nother ^Mcontent with CR^M$\ncontent^M$\nagain content with^M^M$\n\nIndexed content:\nsome content^M$\nother ^Mcontent with CR^M$\ncontent^M$\nagain content with^M^M$\n\nExpected result\n---------------\nFile content:\nsome content^M$\nother ^Mcontent with CR^M$\ncontent^M$\nagain content with^M^M$\n\nIndexed content:\nsome content$\nother ^Mcontent with CR$\ncontent$\nagain content with^M$\n# or even 'again content with$' for this last line\n\nIf you remove the \\r that are not at the end of the lines, EOL are\nconverted as expected:\nFile content:\nsome content^M$\nother content with CR^M$\ncontent^M$\nagain content with^M$\n\nIndexed content:\nsome content$\nother content with CR$\ncontent$\nagain content with$\n\n-- \nAlex\n"},{"id":"259747","messageId":"xmqq1tjdidle.fsf@gitster.dls.corp.google.com","threadId":"39124","inReplyTo":"CAFFOgCUTxnbL7vJpf1Hw39CJL_p2raDZ2a3DehhYhdbkVyi2fw@mail.gmail.com","subject":"Re: [BUG] Git does not convert CRLF=>LF on files with \\r not before \\n","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-04-21T17:41:01Z","receivedAt":"2015-04-21T17:41:01Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Alexandre Garnier <zigarn@gmail.com> writes:\n\n> echo '*       text=auto' > .gitattributes\n> git add .gitattributes\n> git commit -q -m \"Normalize EOL\"\n> echo -ne 'some content\\r\\nother \\rcontent with CR\\r\\ncontent\\r\\nagain\n\nWith text=auto, the user instructs us to guess, and we expect either\nLF or CRLF line-terminated files that is *TEXT*.  A lone CR in the\nmiddle of the line would mean we cannot reliably guess---it may be\nLF terminated file with CRs sprinkled inside text, some of which\nhappen to be at the end of the line, or it may be CRLF terminated\nfile with CRs sprinkled in.  We try to preserve the user input by\nnot munging when we are not sure.\n\nYou are seeing the designed and intended behaviour.\n\nBut it would be a bug if the same thing happens when the user\nexplicitly tells us that the file has CRLF line endings, and I\nsuspect we have that bug, which may want to be corrected.\n\nI've Cc'ed various people who worked on convert.c around line\nendings.  I recall we saw a few other discussion threads on\ntext=auto and eol settings.  Stakeholders may want to have a unified\ndiscussion to first list the issues in the current implementation\nand come up with fixes for them.\n\nThanks.\n"},{"id":"259756","messageId":"5536A4CB.9050000@web.de","threadId":"39124","inReplyTo":"CAFFOgCUTxnbL7vJpf1Hw39CJL_p2raDZ2a3DehhYhdbkVyi2fw@mail.gmail.com","subject":"Re: [BUG] Git does not convert CRLF=>LF on files with \\r not before \\n","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2015-04-21T19:28:11Z","receivedAt":"2015-04-21T19:28:11Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2015-04-21 15.51, Alexandre Garnier wrote:\n> Here is a test:\n> \n> git init -q crlf-test\n> cd crlf-test\n> echo '*       text=auto' > .gitattributes\n> git add .gitattributes\n> git commit -q -m \"Normalize EOL\"\n> echo -ne 'some content\\r\\nother \\rcontent with CR\\r\\ncontent\\r\\nagain\n> content with\\r\\r\\n' > inline-cr.txt\n> echo \"Working directory content:\"\n> cat -A inline-cr.txt\n> echo\n> git add inline-cr.txt\n> echo \"Indexed content:\"\n> git show :inline-cr.txt | cat -A\n> \n> Result\n> ------\n> File content:\n> some content^M$\n> other ^Mcontent with CR^M$\n> content^M$\n> again content with^M^M$\n> \n> Indexed content:\n> some content^M$\n> other ^Mcontent with CR^M$\n> content^M$\n> again content with^M^M$\n> \n> Expected result\n> ---------------\n> File content:\n> some content^M$\n> other ^Mcontent with CR^M$\n> content^M$\n> again content with^M^M$\n> \n> Indexed content:\n> some content$\n> other ^Mcontent with CR$\n> content$\n> again content with^M$\n> # or even 'again content with$' for this last line\n> \n> If you remove the \\r that are not at the end of the lines, EOL are\n> converted as expected:\n> File content:\n> some content^M$\n> other content with CR^M$\n> content^M$\n> again content with^M$\n> \n> Indexed content:\n> some content$\n> other content with CR$\n> content$\n> again content with$\n> \n\nFirst of all, thanks for the info.\n\nThe current implementation of Git does an auto-detection\nif a file is text or binary.\n\nFor a file which is \"suspected to be text\", it is expected to have either LF or CRLF as\nline endings, but a \"bare CR\" make Git wonder:\nShould this still be treated as a text file ?\nIf yes, should the CR be kept as is, or should it be converted into LF (or CRLF) ?\n\nThe current implementation may simply be explained by the fact that nobody has so far asked \nto treat this file as \"text\", so the implementation assumes it to be binary.\n\n(Which makes the code a little bit easier, at the time it was written)\n\nSo the status of today is that you can force Git to let the CR as is,\nwhen you specify that the file is \"text\".\n\nIs there a real life problem behind it ?\nAnd what should happen to the CRs ?\n"},{"id":"259797","messageId":"CAFFOgCVkpweSukCW4La737P1uu1hqX8WKr9wF1cLV8ciPu8tuQ@mail.gmail.com","threadId":"39124","inReplyTo":"5536A4CB.9050000@web.de","subject":"Re: [BUG] Git does not convert CRLF=>LF on files with \\r not before \\n","fromName":"Alexandre Garnier","fromEmail":"zigarn@gmail.com","sentAt":"2015-04-22T13:06:38Z","receivedAt":"2015-04-22T13:06:38Z","isPatch":false,"sender":{"key":"zigarn@gmail.com","avatar":"https://gravatar.com/avatar/a7916de623e2296b522dd0275b7e64e6d8eb8f6875c9b61c1a3f84498d43c750?d=mp&s=160"},"body":"Indeed, when changing the gitattributes for '* text', the replacement is OK.\nThanks for all the explanations.\n\nAt first, my use case was some source files (imported from another\nVCS) with CR in different contexts:\n - lines ending with CRCRLF\n - all content in LF or CRLF but some CR that should be EOL...\n - CR in the middle of the line for no reason!\nFor all this, I will fix the files during import.\n\nBut when digging I found some shell or awk scripts with CR as a valid\nchar in search/replacement string. I know that the EOL should not be\nCRLF in this case, but I don't know if this situation could happen in\nDOS batch files or PowerShell scripts with CRLF EOL.\n\n2015-04-21 21:28 GMT+02:00 Torsten Bögershausen <tboegi@web.de>:\n> On 2015-04-21 15.51, Alexandre Garnier wrote:\n>> Here is a test:\n>>\n>> git init -q crlf-test\n>> cd crlf-test\n>> echo '*       text=auto' > .gitattributes\n>> git add .gitattributes\n>> git commit -q -m \"Normalize EOL\"\n>> echo -ne 'some content\\r\\nother \\rcontent with CR\\r\\ncontent\\r\\nagain\n>> content with\\r\\r\\n' > inline-cr.txt\n>> echo \"Working directory content:\"\n>> cat -A inline-cr.txt\n>> echo\n>> git add inline-cr.txt\n>> echo \"Indexed content:\"\n>> git show :inline-cr.txt | cat -A\n>>\n>> Result\n>> ------\n>> File content:\n>> some content^M$\n>> other ^Mcontent with CR^M$\n>> content^M$\n>> again content with^M^M$\n>>\n>> Indexed content:\n>> some content^M$\n>> other ^Mcontent with CR^M$\n>> content^M$\n>> again content with^M^M$\n>>\n>> Expected result\n>> ---------------\n>> File content:\n>> some content^M$\n>> other ^Mcontent with CR^M$\n>> content^M$\n>> again content with^M^M$\n>>\n>> Indexed content:\n>> some content$\n>> other ^Mcontent with CR$\n>> content$\n>> again content with^M$\n>> # or even 'again content with$' for this last line\n>>\n>> If you remove the \\r that are not at the end of the lines, EOL are\n>> converted as expected:\n>> File content:\n>> some content^M$\n>> other content with CR^M$\n>> content^M$\n>> again content with^M$\n>>\n>> Indexed content:\n>> some content$\n>> other content with CR$\n>> content$\n>> again content with$\n>>\n>\n> First of all, thanks for the info.\n>\n> The current implementation of Git does an auto-detection\n> if a file is text or binary.\n>\n> For a file which is \"suspected to be text\", it is expected to have either LF or CRLF as\n> line endings, but a \"bare CR\" make Git wonder:\n> Should this still be treated as a text file ?\n> If yes, should the CR be kept as is, or should it be converted into LF (or CRLF) ?\n>\n> The current implementation may simply be explained by the fact that nobody has so far asked\n> to treat this file as \"text\", so the implementation assumes it to be binary.\n>\n> (Which makes the code a little bit easier, at the time it was written)\n>\n> So the status of today is that you can force Git to let the CR as is,\n> when you specify that the file is \"text\".\n>\n> Is there a real life problem behind it ?\n> And what should happen to the CRs ?\n>\n>\n>\n>\n>\n"},{"id":"259806","messageId":"xmqq618of49u.fsf@gitster.dls.corp.google.com","threadId":"39124","inReplyTo":"CAFFOgCVkpweSukCW4La737P1uu1hqX8WKr9wF1cLV8ciPu8tuQ@mail.gmail.com","subject":"Re: [BUG] Git does not convert CRLF=>LF on files with \\r not before \\n","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-04-22T17:42:53Z","receivedAt":"2015-04-22T17:42:53Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Alexandre Garnier <zigarn@gmail.com> writes:\n\n> Indeed, when changing the gitattributes for '* text', the replacement is OK.\n\nOK.  Earlier I said:\n\n>> But it would be a bug if the same thing happens when the user\n>> explicitly tells us that the file has CRLF line endings, and I\n>> suspect we have that bug, which may want to be corrected.\n\nbut you are saying that my suspicion is incorrect and we do not have\nsuch a bug.\n\nThanks for digging further.\n"}]}