{"thread":{"id":"23779","subject":"[PATCH v2] autocrlf: Make it work also for un-normalized repositories","startedAt":"2010-05-11T22:37:57Z","lastAt":"2010-05-23T09:30:40Z","messageCount":5,"participants":["Finn Arne Gangstad","Dmitry Potapov","Eyvind Bernhardsen","Hnerik Grubbström","Clemens Buchacher"],"isPatch":true,"patchVersion":2,"patchTotal":null},"messages":[{"id":"141492","messageId":"20100511223757.GB16974@pvv.org","threadId":"23779","inReplyTo":null,"subject":"[PATCH v2] autocrlf: Make it work also for un-normalized repositories","fromName":"Finn Arne Gangstad","fromEmail":"finnag@pvv.org","sentAt":"2010-05-11T22:37:57Z","receivedAt":"2010-05-11T22:37:57Z","isPatch":true,"sender":{"key":"finnag@pvv.org","avatar":"https://gravatar.com/avatar/b421ddd58c3f0f93aa473e17b98bb8d53c221fef741746bc8cb59fae4ec6d95e?d=mp&s=160"},"body":"Previously, autocrlf would only work well for normalized\nrepositories. Any text files that contained CRLF in the repository\nwould cause problems, and would be modified when handled with\ncore.autocrlf set.\n\nChange autocrlf to not do any conversions to files that in the\nrepository already contain a CR. git with autocrlf set will never\ncreate such a file, or change a LF only file to contain CRs, so the\n(new) assumption is that if a file contains a CR, it is intentional,\nand autocrlf should not change that.\n\nThe following sequence should now always be a NOP even with autocrlf\nset (assuming a clean working directory):\n\ngit checkout <something>\ntouch *\ngit add -A .    (will add nothing)\ngit commit      (nothing to commit)\n\nPreviously this would break for any text file containing a CR.\n\nSome of you may have been folowing Eyvind's excellent thread about\ntrying to make end-of-line translation in git a bit smoother.\n\nI decided to attack the problem from a different angle: Is it possible\nto make autocrlf behave non-destructively for all the previous problem cases?\n\nStealing the problem from Eyvind's initial mail (paraphrased and\nsummarized a bit):\n\n1. Setting autocrlf globally is a pain since autocrlf does not work well\n   with CRLF in the repo\n2. Setting it in individual repos is hard since you do it \"too late\"\n   (the clone will get it wrong)\n3. If someone checks in a file with CRLF later, you get into problems again\n4. If a repository once has contained CRLF, you can't tell autocrlf\n   at which commit everything is sane again\n5. autocrlf does needless work if you know that all your users want\n   the same EOL style.\n\nI belive that this patch makes autocrlf a safe (and good) default\nsetting for Windows, and this solves problems 1-4 (it solves 2 by being\nset by default, which is early enough for clone).\n\nI implemented it by looking for CR charactes in the index, and\naborting any conversion attempt if this is found.\n\nSigned-off-by: Finn Arne Gangstad <finag@pvv.org>\n---\n\nChanges since v1:\n\nOnly check for CRs in the index if we're in CRLF_GUESS mode, if any\ncrlf attributes are set, we assume the attributes are set correctly\nand do not scan for CRs.\n\nAdded tests. Note that I only ran these on Linux, so I'm not 100% sure\nthey are written portably.\n\nI have talked a bit to Eyvind about his eol-patch, so this is\nrewritten slightly to be easier to merge with his changes.\n\n\n convert.c       |   49 +++++++++++++++++++++++++++++++++++++++++++++++++\n t/t0020-crlf.sh |   54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 103 insertions(+), 0 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 4f8fcb7..3458bca 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -120,6 +120,43 @@ static void check_safe_crlf(const char *path, int action,\n \t}\n }\n \n+static int has_cr_in_index(const char *path)\n+{\n+\tint pos, len;\n+\tunsigned long sz;\n+\tenum object_type type;\n+\tvoid *data;\n+\tint has_cr;\n+\tstruct index_state *istate = &the_index;\n+\n+\tlen = strlen(path);\n+\tpos = index_name_pos(istate, path, len);\n+\tif (pos < 0) {\n+\t\t/*\n+\t\t * We might be in the middle of a merge, in which\n+\t\t * case we would read stage #2 (ours).\n+\t\t */\n+\t\tint i;\n+\t\tfor (i = -pos - 1;\n+\t\t     (pos < 0 && i < istate->cache_nr &&\n+\t\t      !strcmp(istate->cache[i]->name, path));\n+\t\t     i++)\n+\t\t\tif (ce_stage(istate->cache[i]) == 2)\n+\t\t\t\tpos = i;\n+\t}\n+\tif (pos < 0)\n+\t\treturn 0;\n+\tdata = read_sha1_file(istate->cache[pos]->sha1, &type, &sz);\n+\tif (!data || type != OBJ_BLOB) {\n+\t\tfree(data);\n+\t\treturn 0;\n+\t}\n+\n+\thas_cr = memchr(data, '\\r', sz) != NULL;\n+\tfree(data);\n+\treturn has_cr;\n+}\n+\n static int crlf_to_git(const char *path, const char *src, size_t len,\n                        struct strbuf *buf, int action, enum safe_crlf checksafe)\n {\n@@ -145,6 +182,13 @@ static int crlf_to_git(const char *path, const char *src, size_t len,\n \t\t */\n \t\tif (is_binary(len, &stats))\n \t\t\treturn 0;\n+\n+\t\t/*\n+\t\t * If the file in the index has any CR in it, do not convert.\n+\t\t * This is the new safer autocrlf handling.\n+\t\t */\n+\t\tif (has_cr_in_index(path))\n+\t\t\treturn 0;\n \t}\n \n \tcheck_safe_crlf(path, action, &stats, checksafe);\n@@ -203,6 +247,11 @@ static int crlf_to_worktree(const char *path, const char *src, size_t len,\n \t\treturn 0;\n \n \tif (action == CRLF_GUESS) {\n+\t\t/* If we have any CR or CRLF line endings, we do not touch it */\n+\t\t/* This is the new safer autocrlf-handling */\n+\t\tif (stats.cr > 0 || stats.crlf > 0)\n+\t\t\treturn 0;\n+\t\t\n \t\t/* If we have any bare CR characters, we're not going to touch it */\n \t\tif (stats.cr != stats.crlf)\n \t\t\treturn 0;\ndiff --git a/t/t0020-crlf.sh b/t/t0020-crlf.sh\nindex c3e7e32..f425302 100755\n--- a/t/t0020-crlf.sh\n+++ b/t/t0020-crlf.sh\n@@ -453,5 +453,59 @@ test_expect_success 'invalid .gitattributes (must not crash)' '\n \tgit diff\n \n '\n+# Some more tests here to add new autocrlf functionality.\n+# We want to have a known state here, so start a bit from scratch\n+\n+test_expect_success 'setting up for new autocrlf tests' '\n+\n+\tgit config core.autocrlf false &&\n+\tgit config core.safecrlf false &&\n+\trm -rf .????* * &&\n+ \tfor w in I am all LF; do echo $w; done >alllf &&\n+\tfor w in Oh here is CRLFQ in text; do echo $w; done | q_to_cr >mixed &&\n+\tfor w in I am all CRLF; do echo $w; done | append_cr >allcrlf &&\n+\tgit add -A . &&\n+\tgit commit -m \"alllf, allcrlf and mixed only\" &&\n+\tgit tag -a -m \"message\" autocrlf-checkpoint\n+'\n+\n+test_expect_success 'setting autocrlf gives empty diff' '\n+\n+\tgit config core.autocrlf true && \n+\ttouch * &&\n+\tgit diff --exit-code\n+'\n+\n+test_expect_success 'files are clean after checkout' '\n+\trm * &&\n+\tgit checkout -f &&\n+\tgit diff --exit-code\n+'\n+\n+cr_to_Q_no_NL () {\n+    tr '\\015' Q | tr -d '\\012'\n+}\n+\n+test_expect_success 'LF only file gets CRLF with autocrlf' '\n+\t[ $(cr_to_Q_no_NL < alllf) = \"IQamQallQLFQ\" ]\n+'\n+\n+test_expect_success 'Mixed file is still mixed with autocrlf' '\n+\t[ $(cr_to_Q_no_NL < mixed) = \"OhhereisCRLFQintext\" ]\n+'\n+\n+test_expect_success 'CRLF only file has CRLF with autocrlf' '\n+\t[ $(cr_to_Q_no_NL < allcrlf) = \"IQamQallQCRLFQ\" ]\n+'\n+\n+test_expect_success 'New CRLF file gets LF in repo' '\n+\ttr -d \"\\015\" < alllf | append_cr > alllf2 &&\n+\tgit add alllf2 &&\n+\tgit commit -m \"alllf2 added\" &&\n+\tgit config core.autocrlf false &&\n+\trm * &&\n+\tgit checkout -f &&\n+\ttest_cmp alllf alllf2 \n+'\n \n test_done\n-- \n1.7.1.1.g653e8\n"},{"id":"141506","messageId":"20100512061646.GL14069@dpotapov.dyndns.org","threadId":"23779","inReplyTo":"20100511223757.GB16974@pvv.org","subject":"Re: [PATCH v2] autocrlf: Make it work also for un-normalized repositories","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2010-05-12T06:16:47Z","receivedAt":"2010-05-12T06:16:47Z","isPatch":true,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Wed, May 12, 2010 at 12:37:57AM +0200, Finn Arne Gangstad wrote:\n> @@ -203,6 +247,11 @@ static int crlf_to_worktree(const char *path, const char *src, size_t len,\n>  \t\treturn 0;\n>  \n>  \tif (action == CRLF_GUESS) {\n> +\t\t/* If we have any CR or CRLF line endings, we do not touch it */\n> +\t\t/* This is the new safer autocrlf-handling */\n> +\t\tif (stats.cr > 0 || stats.crlf > 0)\n> +\t\t\treturn 0;\n> +\t\t\n>  \t\t/* If we have any bare CR characters, we're not going to touch it */\n>  \t\tif (stats.cr != stats.crlf)\n>  \t\t\treturn 0;\n\nIf there is no CR then there is no CRLF and certainly no bare CR\ncharacters. So, all above checks can be replaced with one:\n\n\t\tif (stats.cr > 0)\n\t\t\treturn 0;\n\nOther than that, I really like your patch.\n\n\nThanks,\nDmitry\n"},{"id":"141508","messageId":"75EA5FF3-4F82-4261-AF2A-D6E7182EECE5@gmail.com","threadId":"23779","inReplyTo":"20100512061646.GL14069@dpotapov.dyndns.org","subject":"Re: [PATCH v2] autocrlf: Make it work also for un-normalized repositories","fromName":"Eyvind Bernhardsen","fromEmail":"eyvind.bernhardsen@gmail.com","sentAt":"2010-05-12T06:33:22Z","receivedAt":"2010-05-12T06:33:22Z","isPatch":true,"sender":{"key":"eyvind.bernhardsen@gmail.com","avatar":"https://avatars.githubusercontent.com/u/106762?v=4"},"body":"On 12. mai 2010, at 08.16, Dmitry Potapov <dpotapov@gmail.com> wrote:\n\n> On Wed, May 12, 2010 at 12:37:57AM +0200, Finn Arne Gangstad wrote:\n>> @@ -203,6 +247,11 @@ static int crlf_to_worktree(const char *path,  \n>> const char *src, size_t len,\n>>        return 0;\n>>\n>>    if (action == CRLF_GUESS) {\n>> +        /* If we have any CR or CRLF line endings, we do not touch  \n>> it */\n>> +        /* This is the new safer autocrlf-handling */\n>> +        if (stats.cr > 0 || stats.crlf > 0)\n>> +            return 0;\n>> +\n>>        /* If we have any bare CR characters, we're not going to  \n>> touch it */\n>>        if (stats.cr != stats.crlf)\n>>            return 0;\n>\n> If there is no CR then there is no CRLF and certainly no bare CR\n> characters. So, all above checks can be replaced with one:\n>\n>        if (stats.cr > 0)\n>            return 0;\n>\n> Other than that, I really like your patch.\n\nKeeping the tests separate helps merging with my patch. The idea is to  \nnot do the \"safe autocrlf\" test when crlf=auto, so in that case the CR  \ntest is still needed.\n-- \nEyvind\n"},{"id":"141523","messageId":"loom.20100512T112040-607@post.gmane.org","threadId":"23779","inReplyTo":"20100511223757.GB16974@pvv.org","subject":"Re: [PATCH v2] autocrlf: Make it work also for un-normalized repositories","fromName":"Hnerik Grubbström","fromEmail":"grubba@grubba.org","sentAt":"2010-05-12T09:26:06Z","receivedAt":"2010-05-12T09:26:06Z","isPatch":true,"sender":{"key":"grubba@grubba.org","avatar":"https://avatars.githubusercontent.com/u/1169458?v=4"},"body":"Finn Arne Gangstad <finnag <at> pvv.org> writes:\n\n> Previously, autocrlf would only work well for normalized\n> repositories. Any text files that contained CRLF in the repository\n> would cause problems, and would be modified when handled with\n> core.autocrlf set.\n> \n> The following sequence should now always be a NOP even with autocrlf\n> set (assuming a clean working directory):\n> \n> git checkout <something>\n> touch *\n> git add -A .    (will add nothing)\n> git commit      (nothing to commit)\n\nPlease note that the above problem is solved for the general case with the\n\"[PATCH v3 0/5] Patches to avoid reporting conversion changes.\" patch set.\n\n--\n/grubba\n"},{"id":"142139","messageId":"20100523093040.GA15602@localhost","threadId":"23779","inReplyTo":"20100511223757.GB16974@pvv.org","subject":"Re: [PATCH v2] autocrlf: Make it work also for un-normalized repositories","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2010-05-23T09:30:40Z","receivedAt":"2010-05-23T09:30:40Z","isPatch":true,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"On Wed, May 12, 2010 at 12:37:57AM +0200, Finn Arne Gangstad wrote:\n\n> Change autocrlf to not do any conversions to files that in the\n> repository already contain a CR. git with autocrlf set will never\n> create such a file, or change a LF only file to contain CRs, so the\n> (new) assumption is that if a file contains a CR, it is intentional,\n> and autocrlf should not change that.\n\nI think this is a good change. But it only covers the part where we\ntranslate CR -> LF when staging changes. With Eyvind's patches, if\nI understand correctly, it will be possible to convert files to\nhave LF line endings. Such files will be translated from LF -> CR\nwhen adding changes.\n\nSo if the file already has LF line endings, will this cause the\nsame problem the other way around?\n\nClemens\n"}]}