{"thread":{"id":"29184","subject":"[BUG] attribute \"eol\" with \"crlf\"","startedAt":"2011-12-16T17:44:21Z","lastAt":"2011-12-17T19:48:24Z","messageCount":15,"participants":["Ralf Thielow","Junio C Hamano","Matthieu Moy","Adam Borowski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"181307","messageId":"CAN0XMO+OOdTJ+aNMSc2G3RVc7Wfypr4+7dU3US9GVAmMiSJ7cg@mail.gmail.com","threadId":"29184","inReplyTo":null,"subject":"[BUG] attribute \"eol\" with \"crlf\"","fromName":"Ralf Thielow","fromEmail":"ralf.thielow@googlemail.com","sentAt":"2011-12-16T17:44:21Z","receivedAt":"2011-12-16T17:44:21Z","isPatch":false,"sender":{"key":"ralf.thielow@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1275832?v=4"},"body":"There's a bug in git-1.7.8 if you use the attribute \"eol\" with \"crlf\".\n\nSteps to reproduce:\n- add and commit a text file which uses 0d0a for line breaks\n7465 7374 0d0a 0d0a 7465 7374 0d0a test....test..\n- add \".gitattributes\" with \"*.txt eol=crlf\"\n- change a line in the file\n- execute \"git checkout [file]\"\n\nThe result is:\n7465 7374 0d0d 0a0d 0d0a 7465 7374 0d0d test......test..\n\n0d0a was replaced by 0d0d0a.\n"},{"id":"181310","messageId":"7vr504ieco.fsf@alter.siamese.dyndns.org","threadId":"29184","inReplyTo":"CAN0XMO+OOdTJ+aNMSc2G3RVc7Wfypr4+7dU3US9GVAmMiSJ7cg@mail.gmail.com","subject":"Re: [BUG] attribute \"eol\" with \"crlf\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-16T18:03:03Z","receivedAt":"2011-12-16T18:03:03Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Can you bisect?\n"},{"id":"181312","messageId":"vpqr504wf70.fsf@bauges.imag.fr","threadId":"29184","inReplyTo":"CAN0XMO+OOdTJ+aNMSc2G3RVc7Wfypr4+7dU3US9GVAmMiSJ7cg@mail.gmail.com","subject":"Re: [BUG] attribute \"eol\" with \"crlf\"","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-12-16T18:21:07Z","receivedAt":"2011-12-16T18:21:07Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Ralf Thielow writes:\n\n> There's a bug in git-1.7.8 if you use the attribute \"eol\" with \"crlf\".\n>\n> Steps to reproduce:\n> - add and commit a text file which uses 0d0a for line breaks\n> 7465 7374 0d0a 0d0a 7465 7374 0d0a test....test..\n> - add \".gitattributes\" with \"*.txt eol=crlf\"\n> - change a line in the file\n> - execute \"git checkout [file]\"\n>\n> The result is:\n> 7465 7374 0d0d 0a0d 0d0a 7465 7374 0d0d test......test..\n\nIt seems to me to be the expected behavior. You committed a file whose\nline endings are not normalized to LF in the repository, and asked for a\nconversion LF -> CRLF on checkout, which Git did.\n\nGit can't know exactly the moment when you edit .gitattributes, so it\ncan't do the conversion at the time you add the eol=crlf attribute. It\ndoes it on checkout.\n\n> 0d0a was replaced by 0d0d0a.\n\nI'd say 0a (LF) was replaced by 0d0a (CRLF).\n\nWhat behavior would you have expected?\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"181314","messageId":"CAN0XMOJFCwORt_VaddgeeCNp3S-nm8DxYDPDyPCsVngRhuEP6A@mail.gmail.com","threadId":"29184","inReplyTo":"vpqr504wf70.fsf@bauges.imag.fr","subject":"Re: [BUG] attribute \"eol\" with \"crlf\"","fromName":"Ralf Thielow","fromEmail":"ralf.thielow@googlemail.com","sentAt":"2011-12-16T18:28:22Z","receivedAt":"2011-12-16T18:28:22Z","isPatch":false,"sender":{"key":"ralf.thielow@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1275832?v=4"},"body":"> What behavior would you have expected?\n\nI've expected that git doesn't change the line endings\nbecause it's already CRLF.\n"},{"id":"181317","messageId":"CAN0XMOLNF5D5m-2vBw+oA2qeD0Y9D=_zR+MLpR=f8_O7F1BDtw@mail.gmail.com","threadId":"29184","inReplyTo":"7vr504ieco.fsf@alter.siamese.dyndns.org","subject":"Re: [BUG] attribute \"eol\" with \"crlf\"","fromName":"Ralf Thielow","fromEmail":"ralf.thielow@googlemail.com","sentAt":"2011-12-16T18:49:29Z","receivedAt":"2011-12-16T18:49:29Z","isPatch":false,"sender":{"key":"ralf.thielow@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1275832?v=4"},"body":"> Can you bisect?\n\ne322ee38ad8d655f5a32b3482ae9ce813b73e4bc\n"},{"id":"181330","messageId":"20111216200955.GA8499@angband.pl","threadId":"29184","inReplyTo":"CAN0XMOJFCwORt_VaddgeeCNp3S-nm8DxYDPDyPCsVngRhuEP6A@mail.gmail.com","subject":"Re: [BUG] attribute \"eol\" with \"crlf\"","fromName":"Adam Borowski","fromEmail":"kilobyte@angband.pl","sentAt":"2011-12-16T20:09:55Z","receivedAt":"2011-12-16T20:09:55Z","isPatch":false,"sender":{"key":"kilobyte@angband.pl","avatar":"https://avatars.githubusercontent.com/u/48801?v=4"},"body":"On Fri, Dec 16, 2011 at 07:28:22PM +0100, Ralf Thielow wrote:\n> > What behavior would you have expected?\n> \n> I've expected that git doesn't change the line endings\n> because it's already CRLF.\n\nAnd how exactly can it change the file back the way it was?\ns/\\n/\\r\\n/g is roundtrippable, s/\\r?\\n/\\r\\n/g is not.\n\n-- \n1KB\t\t// Yo momma uses IPv4!\n"},{"id":"181335","messageId":"CAN0XMOK9P_25M0bVjYn9gZ+ONi7yKEZX+83X_mwJok+P5TEVoQ@mail.gmail.com","threadId":"29184","inReplyTo":"20111216200955.GA8499@angband.pl","subject":"Re: [BUG] attribute \"eol\" with \"crlf\"","fromName":"Ralf Thielow","fromEmail":"ralf.thielow@googlemail.com","sentAt":"2011-12-16T20:53:16Z","receivedAt":"2011-12-16T20:53:16Z","isPatch":false,"sender":{"key":"ralf.thielow@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1275832?v=4"},"body":"> And how exactly can it change the file back the way it was?\n\nIt shouldn't need to change the file back. That's actually the bug. :/\n"},{"id":"181337","messageId":"7vmxasgqlm.fsf@alter.siamese.dyndns.org","threadId":"29184","inReplyTo":"vpqr504wf70.fsf@bauges.imag.fr","subject":"Re: [BUG] attribute \"eol\" with \"crlf\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-16T21:21:25Z","receivedAt":"2011-12-16T21:21:25Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthieu Moy writes:\n\n> Ralf Thielow writes:\n>\n>> There's a bug in git-1.7.8 if you use the attribute \"eol\" with \"crlf\".\n>>\n>> Steps to reproduce:\n>> - add and commit a text file which uses 0d0a for line breaks\n>> 7465 7374 0d0a 0d0a 7465 7374 0d0a test....test..\n>> - add \".gitattributes\" with \"*.txt eol=crlf\"\n>> - change a line in the file\n>> - execute \"git checkout [file]\"\n>>\n>> The result is:\n>> 7465 7374 0d0d 0a0d 0d0a 7465 7374 0d0d test......test..\n>\n> It seems to me to be the expected behavior. You committed a file whose\n> line endings are not normalized to LF in the repository, and asked for a\n> conversion LF -> CRLF on checkout, which Git did.\n>\n> Git can't know exactly the moment when you edit .gitattributes, so it\n> can't do the conversion at the time you add the eol=crlf attribute. It\n> does it on checkout.\n>\n>> 0d0a was replaced by 0d0d0a.\n>\n> I'd say 0a (LF) was replaced by 0d0a (CRLF).\n>\n> What behavior would you have expected?\n\nThe sequence adds \"test\\r\\n\" file without .gitattributes to have the\nrepository record that exact byte sequence for the file. But then later\ngoes around and says \"This file wants to express the end of line with CRLF\non the filesystem, so please replace LF in the repository representation\nto CRLF when checking out, and replace CRLF in the working tree to LF when\nchecking in\".\n\nSo it is not surprising that \"\\r\\n\" coming from the repository is replaced\nto \"\\r\\r\\n\" when checked out. As far as the repository data is concerned,\nthat line has a funny byte with value \"\\r\" at the end, immediately before\nthe line terminator \"\\n\".\n\nWhat you said is _technically_ correct in that sense.\n\nHowever, I think the CRLF filter used to have a hack to strip \"\\r\" if the\nrepository data records \"\\r\" at the end of line. This was intended to help\npeople who checked in such a broken text file (if it is a text file, then\nraw ascii CR does not have a place in it in the repository representation)\nand it was a useful hack to help people recover from such mistakes to\nstart the project from DOS-only world (with CRLF in the repository data)\nand migrate to cross platform world (with LF in the repository data, CRLF\nin the DOS working tree). I suspect that the streaming filter conversion\nmay not have the same hack in it.\n"},{"id":"181341","messageId":"CAN0XMOL674Hw_LctTC+8NNqA84Of6dMjdKT0SU+DWMG7EYShYQ@mail.gmail.com","threadId":"29184","inReplyTo":"7vmxasgqlm.fsf@alter.siamese.dyndns.org","subject":"Re: [BUG] attribute \"eol\" with \"crlf\"","fromName":"Ralf Thielow","fromEmail":"ralf.thielow@googlemail.com","sentAt":"2011-12-16T22:05:27Z","receivedAt":"2011-12-16T22:05:27Z","isPatch":false,"sender":{"key":"ralf.thielow@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1275832?v=4"},"body":"So i have to commit \".gitattributes\" and everything is fine for me after!?\n\n> The sequence adds \"test\\r\\n\" file without .gitattributes to have the\n> repository record that exact byte sequence for the file. But then later\n> goes around and says \"This file wants to express the end of line with CRLF\n> on the filesystem, so please replace LF in the repository representation\n> to CRLF when checking out, and replace CRLF in the working tree to LF when\n> checking in\".\n>\n> So it is not surprising that \"\\r\\n\" coming from the repository is replaced\n> to \"\\r\\r\\n\" when checked out. As far as the repository data is concerned,\n> that line has a funny byte with value \"\\r\" at the end, immediately before\n> the line terminator \"\\n\".\n>\n> What you said is _technically_ correct in that sense.\n>\n> However, I think the CRLF filter used to have a hack to strip \"\\r\" if the\n> repository data records \"\\r\" at the end of line. This was intended to help\n> people who checked in such a broken text file (if it is a text file, then\n> raw ascii CR does not have a place in it in the repository representation)\n> and it was a useful hack to help people recover from such mistakes to\n> start the project from DOS-only world (with CRLF in the repository data)\n> and migrate to cross platform world (with LF in the repository data, CRLF\n> in the DOS working tree).  I suspect that the streaming filter conversion\n> may not have the same hack in it.\n"},{"id":"181342","messageId":"7vehw4go0v.fsf@alter.siamese.dyndns.org","threadId":"29184","inReplyTo":"CAN0XMOL674Hw_LctTC+8NNqA84Of6dMjdKT0SU+DWMG7EYShYQ@mail.gmail.com","subject":"Re: [BUG] attribute \"eol\" with \"crlf\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-16T22:17:04Z","receivedAt":"2011-12-16T22:17:04Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ralf Thielow writes:\n\n> So i have to commit \".gitattributes\" and everything is fine for me after!?\n\nNo. Sorry if I was unclear, but I do not see which part was unclear in\nwhat I wrote, so...\n\n>> The sequence adds \"test\\r\\n\" file without .gitattributes to have the\n>> repository record that exact byte sequence for the file. But then later\n>> goes around and says \"This file wants to express the end of line with CRLF\n>> on the filesystem, so please replace LF in the repository representation\n>> to CRLF when checking out, and replace CRLF in the working tree to LF when\n>> checking in\".\n>>\n>> So it is not surprising that \"\\r\\n\" coming from the repository is replaced\n>> to \"\\r\\r\\n\" when checked out. As far as the repository data is concerned,\n>> that line has a funny byte with value \"\\r\" at the end, immediately before\n>> the line terminator \"\\n\".\n>>\n>> What you said is _technically_ correct in that sense.\n>>\n>> However, I think the CRLF filter used to have a hack to strip \"\\r\" if the\n>> repository data records \"\\r\" at the end of line. This was intended to help\n>> people who checked in such a broken text file (if it is a text file, then\n>> raw ascii CR does not have a place in it in the repository representation)\n>> and it was a useful hack to help people recover from such mistakes to\n>> start the project from DOS-only world (with CRLF in the repository data)\n>> and migrate to cross platform world (with LF in the repository data, CRLF\n>> in the DOS working tree).  I suspect that the streaming filter conversion\n>> may not have the same hack in it.\n"},{"id":"181345","messageId":"CAN0XMOKts7UR6eSYWA9-xj-YCpprvhbqwfdbq4U6Hfrn0nUONQ@mail.gmail.com","threadId":"29184","inReplyTo":"7vehw4go0v.fsf@alter.siamese.dyndns.org","subject":"Re: [BUG] attribute \"eol\" with \"crlf\"","fromName":"Ralf Thielow","fromEmail":"ralf.thielow@googlemail.com","sentAt":"2011-12-16T22:36:07Z","receivedAt":"2011-12-16T22:36:07Z","isPatch":false,"sender":{"key":"ralf.thielow@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1275832?v=4"},"body":"Basicly I want to force the line endings of the files on my\nproject. :/\n\n2011/12/16 Junio C Hamano :\n> Ralf Thielow writes:\n>\n>> So i have to commit \".gitattributes\" and everything is fine for me after!?\n>\n> No.  Sorry if I was unclear, but I do not see which part was unclear in\n> what I wrote, so...\n>\n>>> The sequence adds \"test\\r\\n\" file without .gitattributes to have the\n>>> repository record that exact byte sequence for the file. But then later\n>>> goes around and says \"This file wants to express the end of line with CRLF\n>>> on the filesystem, so please replace LF in the repository representation\n>>> to CRLF when checking out, and replace CRLF in the working tree to LF when\n>>> checking in\".\n>>>\n>>> So it is not surprising that \"\\r\\n\" coming from the repository is replaced\n>>> to \"\\r\\r\\n\" when checked out. As far as the repository data is concerned,\n>>> that line has a funny byte with value \"\\r\" at the end, immediately before\n>>> the line terminator \"\\n\".\n>>>\n>>> What you said is _technically_ correct in that sense.\n>>>\n>>> However, I think the CRLF filter used to have a hack to strip \"\\r\" if the\n>>> repository data records \"\\r\" at the end of line. This was intended to help\n>>> people who checked in such a broken text file (if it is a text file, then\n>>> raw ascii CR does not have a place in it in the repository representation)\n>>> and it was a useful hack to help people recover from such mistakes to\n>>> start the project from DOS-only world (with CRLF in the repository data)\n>>> and migrate to cross platform world (with LF in the repository data, CRLF\n>>> in the DOS working tree).  I suspect that the streaming filter conversion\n>>> may not have the same hack in it.\n"},{"id":"181351","messageId":"7v39ckgkwq.fsf@alter.siamese.dyndns.org","threadId":"29184","inReplyTo":"CAN0XMOKts7UR6eSYWA9-xj-YCpprvhbqwfdbq4U6Hfrn0nUONQ@mail.gmail.com","subject":"Re: [BUG] attribute \"eol\" with \"crlf\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-16T23:24:21Z","receivedAt":"2011-12-16T23:24:21Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ralf Thielow writes:\n\n> Basicly I want to force the line endings of the files on my\n> project. :/\n\nYou need to fix the data you have recorded with CRLF in the repository if\nyou are using eol=crlf, which means \"This file wants to ...\" (see below; I\ndo not want to type the same thing again).\n\n> 2011/12/16 Junio C Hamano :\n>> Ralf Thielow writes:\n>>\n>>> So i have to commit \".gitattributes\" and everything is fine for me after!?\n>>\n>> No.  Sorry if I was unclear, but I do not see which part was unclear in\n>> what I wrote, so...\n>>\n>>>> The sequence adds \"test\\r\\n\" file without .gitattributes to have the\n>>>> repository record that exact byte sequence for the file. But then later\n>>>> goes around and says \"This file wants to express the end of line with CRLF\n>>>> on the filesystem, so please replace LF in the repository representation\n>>>> to CRLF when checking out, and replace CRLF in the working tree to LF when\n>>>> checking in\".\n>>>>\n>>>> So it is not surprising that \"\\r\\n\" coming from the repository is replaced\n>>>> to \"\\r\\r\\n\" when checked out. As far as the repository data is concerned,\n>>>> that line has a funny byte with value \"\\r\" at the end, immediately before\n>>>> the line terminator \"\\n\".\n>>>>\n>>>> What you said is _technically_ correct in that sense.\n>>>> ...\n"},{"id":"181352","messageId":"7vr504f5v5.fsf@alter.siamese.dyndns.org","threadId":"29184","inReplyTo":"7vmxasgqlm.fsf@alter.siamese.dyndns.org","subject":"Re: [BUG] attribute \"eol\" with \"crlf\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-16T23:34:38Z","receivedAt":"2011-12-16T23:34:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> ...\n> What you said is _technically_ correct in that sense.\n>\n> However, I think the CRLF filter used to have a hack to strip \"\\r\" if the\n> repository data records \"\\r\" at the end of line. This was intended to help\n> people who checked in such a broken text file (if it is a text file, then\n> raw ascii CR does not have a place in it in the repository representation)\n> and it was a useful hack to help people recover from such mistakes to\n> start the project from DOS-only world (with CRLF in the repository data)\n> and migrate to cross platform world (with LF in the repository data, CRLF\n> in the DOS working tree). I suspect that the streaming filter conversion\n> may not have the same hack in it.\n\nPerhaps something like this, but I do not use CRLF myself, so it probably\nneeds to be checked by extra sets of eyes.\n\nThanks.\n\n-- >8 --\nSubject: lf_to_crlf_filter(): resurrect CRLF->CRLF hack\n\nThe non-streaming version of the filter counts CRLF and LF in the whole\nbuffer, and returns without doing anything when they match (i.e. what is\nrecorded in the object store already uses CRLF). This was done to help\npeople who added files from the DOS world before realizing they want to go\ncross platform and adding .gitattributes to tell Git that they only want\nCRLF in their working tree.\n\nThe streaming version of the filter does not want to read the whole thing\nbefore starting to work, as that defeats the whole point of streaming. So\nwe instead check what byte follows CR whenever we see one, and add CR\nbefore LF only when the LF does not immediately follow CR already to keep\nCRLF as is.\n\nSigned-off-by: Junio C Hamano \n---\n convert.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++----------\n 1 files changed, 50 insertions(+), 10 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex c028275..8daf4e4 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -879,7 +879,8 @@ int is_null_stream_filter(struct stream_filter *filter)\n \n struct lf_to_crlf_filter {\n \tstruct stream_filter filter;\n-\tunsigned want_lf:1;\n+\tunsigned has_held:1;\n+\tchar held;\n };\n \n static int lf_to_crlf_filter_fn(struct stream_filter *filter,\n@@ -889,10 +890,14 @@ static int lf_to_crlf_filter_fn(struct stream_filter *filter,\n \tsize_t count, o = 0;\n \tstruct lf_to_crlf_filter *lf_to_crlf = (struct lf_to_crlf_filter *)filter;\n \n-\t/* Output a pending LF if we need to */\n-\tif (lf_to_crlf->want_lf) {\n-\t\toutput[o++] = '\\n';\n-\t\tlf_to_crlf->want_lf = 0;\n+\t/*\n+\t * We may be holding onto the CR to see if it is followed by a\n+\t * LF, in which case we would need to go to the main loop.\n+\t * Otherwise, just emit it to the output stream.\n+\t */\n+\tif (lf_to_crlf->has_held && (lf_to_crlf->held != '\\r' || !input)) {\n+\t\toutput[o++] = lf_to_crlf->held;\n+\t\tlf_to_crlf->has_held = 0;\n \t}\n \n \t/* We are told to drain */\n@@ -902,22 +907,57 @@ static int lf_to_crlf_filter_fn(struct stream_filter *filter,\n \t}\n \n \tcount = *isize_p;\n-\tif (count) {\n+\tif (count || lf_to_crlf->has_held) {\n \t\tsize_t i;\n+\t\tint was_cr = 0;\n+\n+\t\tif (lf_to_crlf->has_held) {\n+\t\t\twas_cr = 1;\n+\t\t\tlf_to_crlf->has_held = 0;\n+\t\t}\n+\n \t\tfor (i = 0; o < *osize_p && i < count; i++) {\n \t\t\tchar ch = input[i];\n+\n \t\t\tif (ch == '\\n') {\n \t\t\t\toutput[o++] = '\\r';\n-\t\t\t\tif (o >= *osize_p) {\n-\t\t\t\t\tlf_to_crlf->want_lf = 1;\n-\t\t\t\t\tcontinue; /* We need to increase i */\n-\t\t\t\t}\n+\t\t\t} else if (was_cr) {\n+\t\t\t\t/*\n+\t\t\t\t * Previous round saw CR and it is not followed\n+\t\t\t\t * by a LF; emit the CR before processing the\n+\t\t\t\t * current character.\n+\t\t\t\t */\n+\t\t\t\toutput[o++] = '\\r';\n \t\t\t}\n+\n+\t\t\t/*\n+\t\t\t * We may have consumed the last output slot,\n+\t\t\t * in which case we need to break out of this\n+\t\t\t * loop; hold the current character before\n+\t\t\t * returning.\n+\t\t\t */\n+\t\t\tif (*osize_p <= o) {\n+\t\t\t\tlf_to_crlf->has_held = 1;\n+\t\t\t\tlf_to_crlf->held = ch;\n+\t\t\t\tcontinue; /* break but increment i */\n+\t\t\t}\n+\n+\t\t\tif (ch == '\\r') {\n+\t\t\t\twas_cr = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\twas_cr = 0;\n \t\t\toutput[o++] = ch;\n \t\t}\n \n \t\t*osize_p -= o;\n \t\t*isize_p -= i;\n+\n+\t\tif (!lf_to_crlf->has_held && was_cr) {\n+\t\t\tlf_to_crlf->has_held = 1;\n+\t\t\tlf_to_crlf->held = '\\r';\n+\t\t}\n \t}\n \treturn 0;\n }\n"},{"id":"181403","messageId":"CAN0XMOK0=uxRHcsUmbOE_UrkUcqmRFk-OYnY7kOZkZcWxWOycQ@mail.gmail.com","threadId":"29184","inReplyTo":"7vr504f5v5.fsf@alter.siamese.dyndns.org","subject":"Re: [BUG] attribute \"eol\" with \"crlf\"","fromName":"Ralf Thielow","fromEmail":"ralf.thielow@googlemail.com","sentAt":"2011-12-17T18:04:14Z","receivedAt":"2011-12-17T18:04:14Z","isPatch":false,"sender":{"key":"ralf.thielow@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1275832?v=4"},"body":"Works fine for me. Thanks\n\n> Junio C Hamano writes:\n>\n>> ...\n>> What you said is _technically_ correct in that sense.\n>>\n>> However, I think the CRLF filter used to have a hack to strip \"\\r\" if the\n>> repository data records \"\\r\" at the end of line. This was intended to help\n>> people who checked in such a broken text file (if it is a text file, then\n>> raw ascii CR does not have a place in it in the repository representation)\n>> and it was a useful hack to help people recover from such mistakes to\n>> start the project from DOS-only world (with CRLF in the repository data)\n>> and migrate to cross platform world (with LF in the repository data, CRLF\n>> in the DOS working tree).  I suspect that the streaming filter conversion\n>> may not have the same hack in it.\n>\n> Perhaps something like this, but I do not use CRLF myself, so it probably\n> needs to be checked by extra sets of eyes.\n>\n> Thanks.\n>\n> -- >8 --\n> Subject: lf_to_crlf_filter(): resurrect CRLF->CRLF hack\n>\n> The non-streaming version of the filter counts CRLF and LF in the whole\n> buffer, and returns without doing anything when they match (i.e. what is\n> recorded in the object store already uses CRLF). This was done to help\n> people who added files from the DOS world before realizing they want to go\n> cross platform and adding .gitattributes to tell Git that they only want\n> CRLF in their working tree.\n>\n> The streaming version of the filter does not want to read the whole thing\n> before starting to work, as that defeats the whole point of streaming. So\n> we instead check what byte follows CR whenever we see one, and add CR\n> before LF only when the LF does not immediately follow CR already to keep\n> CRLF as is.\n>\n> Signed-off-by: Junio C Hamano \n> ---\n>  convert.c |   60 ++++++++++++++++++++++++++++++++++++++++++++++++++----------\n>  1 files changed, 50 insertions(+), 10 deletions(-)\n>\n> diff --git a/convert.c b/convert.c\n> index c028275..8daf4e4 100644\n> --- a/convert.c\n> +++ b/convert.c\n> @@ -879,7 +879,8 @@ int is_null_stream_filter(struct stream_filter *filter)\n>\n>  struct lf_to_crlf_filter {\n>        struct stream_filter filter;\n> -       unsigned want_lf:1;\n> +       unsigned has_held:1;\n> +       char held;\n>  };\n>\n>  static int lf_to_crlf_filter_fn(struct stream_filter *filter,\n> @@ -889,10 +890,14 @@ static int lf_to_crlf_filter_fn(struct stream_filter *filter,\n>        size_t count, o = 0;\n>        struct lf_to_crlf_filter *lf_to_crlf = (struct lf_to_crlf_filter *)filter;\n>\n> -       /* Output a pending LF if we need to */\n> -       if (lf_to_crlf->want_lf) {\n> -               output[o++] = '\\n';\n> -               lf_to_crlf->want_lf = 0;\n> +       /*\n> +        * We may be holding onto the CR to see if it is followed by a\n> +        * LF, in which case we would need to go to the main loop.\n> +        * Otherwise, just emit it to the output stream.\n> +        */\n> +       if (lf_to_crlf->has_held && (lf_to_crlf->held != '\\r' || !input)) {\n> +               output[o++] = lf_to_crlf->held;\n> +               lf_to_crlf->has_held = 0;\n>        }\n>\n>        /* We are told to drain */\n> @@ -902,22 +907,57 @@ static int lf_to_crlf_filter_fn(struct stream_filter *filter,\n>        }\n>\n>        count = *isize_p;\n> -       if (count) {\n> +       if (count || lf_to_crlf->has_held) {\n>                size_t i;\n> +               int was_cr = 0;\n> +\n> +               if (lf_to_crlf->has_held) {\n> +                       was_cr = 1;\n> +                       lf_to_crlf->has_held = 0;\n> +               }\n> +\n>                for (i = 0; o < *osize_p && i < count; i++) {\n>                        char ch = input[i];\n> +\n>                        if (ch == '\\n') {\n>                                output[o++] = '\\r';\n> -                               if (o >= *osize_p) {\n> -                                       lf_to_crlf->want_lf = 1;\n> -                                       continue; /* We need to increase i */\n> -                               }\n> +                       } else if (was_cr) {\n> +                               /*\n> +                                * Previous round saw CR and it is not followed\n> +                                * by a LF; emit the CR before processing the\n> +                                * current character.\n> +                                */\n> +                               output[o++] = '\\r';\n>                        }\n> +\n> +                       /*\n> +                        * We may have consumed the last output slot,\n> +                        * in which case we need to break out of this\n> +                        * loop; hold the current character before\n> +                        * returning.\n> +                        */\n> +                       if (*osize_p <= o) {\n> +                               lf_to_crlf->has_held = 1;\n> +                               lf_to_crlf->held = ch;\n> +                               continue; /* break but increment i */\n> +                       }\n> +\n> +                       if (ch == '\\r') {\n> +                               was_cr = 1;\n> +                               continue;\n> +                       }\n> +\n> +                       was_cr = 0;\n>                        output[o++] = ch;\n>                }\n>\n>                *osize_p -= o;\n>                *isize_p -= i;\n> +\n> +               if (!lf_to_crlf->has_held && was_cr) {\n> +                       lf_to_crlf->has_held = 1;\n> +                       lf_to_crlf->held = '\\r';\n> +               }\n>        }\n>        return 0;\n>  }\n"},{"id":"181420","messageId":"7vehw3dlo7.fsf@alter.siamese.dyndns.org","threadId":"29184","inReplyTo":"CAN0XMOK0=uxRHcsUmbOE_UrkUcqmRFk-OYnY7kOZkZcWxWOycQ@mail.gmail.com","subject":"Re: [BUG] attribute \"eol\" with \"crlf\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-17T19:48:24Z","receivedAt":"2011-12-17T19:48:24Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ralf Thielow writes:\n\n>> Perhaps something like this, but I do not use CRLF myself, so it probably\n>> needs to be checked by extra sets of eyes.\n>>\n>> Thanks.\n>>\n>\n> Works fine for me. Thanks\n\nOk; thanks. Will queue.\n"}]}