{"thread":{"id":"46891","subject":"Line ending normalization doesn't work as expected","startedAt":"2017-10-03T15:00:11Z","lastAt":"2018-02-16T17:19:44Z","messageCount":34,"participants":["Robert Dailey","Torsten Bögershausen","Junio C Hamano","Jonathan Nieder","tboegi@web.de","Eric Sunshine"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"329576","messageId":"CAHd499B5hM9ixnsnwWwB2uyDT10dRQpN473m5QjxH9raHtFXiw@mail.gmail.com","threadId":"46891","inReplyTo":null,"subject":"Line ending normalization doesn't work as expected","fromName":"Robert Dailey","fromEmail":"rcdailey.lists@gmail.com","sentAt":"2017-10-03T15:00:01Z","receivedAt":"2017-10-03T15:00:11Z","isPatch":false,"sender":{"key":"rcdailey.lists@gmail.com","avatar":null},"body":"I'm on Windows using Git for Windows v2.13.1. Following github's\nrecommended process for fixing line endings after adding a\n`.gitattributes` file[1], I run the following:\n\n$ rm .git/index && git reset\n\nOnce I run `git status`, I see that no files have changed. Note that I\nknow for a fact in my repository, files were committed using CRLF line\nendings (the files in question are C# code files, and no\n.gitattributes was present at the time).\n\nI also tried this:\n\n$ git rm -r --cached . && git reset --hard\n\nHowever, again `git status` shows no working copy modifications. The\none thing that *did* work (and I tried this on accident actually) is:\n\n$ git rm -r --cached . && git add .\n\nThis properly showed all files in my index with line ending\nmodifications (I ran `git diff --cached -R` to be sure; the output\nshows `^M` at the end of each line in the diff in this case). Note\nthat my global git config has `core.autocrlf` set to `false`, but I\nalso tried the top 2 commands above with it set to `true` but it made\nno difference.\n\nSo my question is: Why do the top 2 commands not work, but the third\none does? To me this all feels like magic / nondeterministic, so I'm\nhoping someone here knows what is going on and can explain the logic\nof it. Also if this is a git config issue, I'm not sure what it could\nbe. Note my `.gitattributes` just has this in it:\n\n* text=auto\n\nThanks in advance.\n\n\n[1]: https://help.github.com/articles/dealing-with-line-endings/\n"},{"id":"329579","messageId":"958c9b03-062c-0bea-3b25-939a36659f32@web.de","threadId":"46891","inReplyTo":"CAHd499B5hM9ixnsnwWwB2uyDT10dRQpN473m5QjxH9raHtFXiw@mail.gmail.com","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-10-03T16:26:28Z","receivedAt":"2017-10-03T16:26:36Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2017-10-03 17:00, Robert Dailey wrote:\n> I'm on Windows using Git for Windows v2.13.1. Following github's\n> recommended process for fixing line endings after adding a\n> `.gitattributes` file[1], I run the following:\n> \n> $ rm .git/index && git reset\n> \n> Once I run `git status`, I see that no files have changed. Note that I\n> know for a fact in my repository, files were committed using CRLF line\n> endings (the files in question are C# code files, and no\n> .gitattributes was present at the time).\n> \n> I also tried this:\n> \n> $ git rm -r --cached . && git reset --hard\n> \n> However, again `git status` shows no working copy modifications. The\n> one thing that *did* work (and I tried this on accident actually) is:\n> \n> $ git rm -r --cached . && git add .\n> \n> This properly showed all files in my index with line ending\n> modifications (I ran `git diff --cached -R` to be sure; the output\n> shows `^M` at the end of each line in the diff in this case). Note\n> that my global git config has `core.autocrlf` set to `false`, but I\n> also tried the top 2 commands above with it set to `true` but it made\n> no difference.\n> \n> So my question is: Why do the top 2 commands not work, but the third\n> one does? To me this all feels like magic / nondeterministic, so I'm\n> hoping someone here knows what is going on and can explain the logic\n> of it. Also if this is a git config issue, I'm not sure what it could\n> be. Note my `.gitattributes` just has this in it:\n\nThe short version is, that the instructions on Github are outdated.\nThis is the official procedure (since 2016, Git v2.12 or so)\nBut it should work even with older version of Git.\n\n$ echo \"* text=auto\" >.gitattributes\n$ git read-tree --empty # Clean index, force re-scan of working directory\n$ git add .\n$ git status # Show files that will be normalized\n$ git commit -m \"Introduce end-of-line normalization\"\n\n\nCould you open an issue on Github ?\n(Or can someone @github fix this ?)\n\n> \n> * text=auto\n> \n> Thanks in advance.\n> \n> \n> [1]: https://help.github.com/articles/dealing-with-line-endings/\n> \n\n"},{"id":"329583","messageId":"CAHd499AD5Kgbp-vxXTnEPkb-Mb5oEeXhaRO5kGniDdqmXwd2QQ@mail.gmail.com","threadId":"46891","inReplyTo":"958c9b03-062c-0bea-3b25-939a36659f32@web.de","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Robert Dailey","fromEmail":"rcdailey.lists@gmail.com","sentAt":"2017-10-03T17:23:51Z","receivedAt":"2017-10-03T17:23:57Z","isPatch":false,"sender":{"key":"rcdailey.lists@gmail.com","avatar":null},"body":"On Tue, Oct 3, 2017 at 11:26 AM, Torsten Bögershausen wrote:\n> The short version is, that the instructions on Github are outdated.\n> This is the official procedure (since 2016, Git v2.12 or so)\n> But it should work even with older version of Git.\n>\n> $ echo \"* text=auto\" >.gitattributes\n> $ git read-tree --empty # Clean index, force re-scan of working directory\n> $ git add .\n> $ git status # Show files that will be normalized\n> $ git commit -m \"Introduce end-of-line normalization\"\n\nIs the way I did it that worked also a valid solution? Or did it only\nwork accidentally? Again the command I ran that worked is:\n\n$ git rm -r --cached . && git add .\n"},{"id":"329585","messageId":"88c57b88-ef2c-d7db-15e2-12791d4cb201@web.de","threadId":"46891","inReplyTo":"CAHd499AD5Kgbp-vxXTnEPkb-Mb5oEeXhaRO5kGniDdqmXwd2QQ@mail.gmail.com","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-10-03T19:19:55Z","receivedAt":"2017-10-03T19:20:03Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2017-10-03 19:23, Robert Dailey wrote:\n> On Tue, Oct 3, 2017 at 11:26 AM, Torsten Bögershausen wrote:\n>> The short version is, that the instructions on Github are outdated.\n>> This is the official procedure (since 2016, Git v2.12 or so)\n>> But it should work even with older version of Git.\n>>\n>> $ echo \"* text=auto\" >.gitattributes\n>> $ git read-tree --empty # Clean index, force re-scan of working directory\n>> $ git add .\n>> $ git status # Show files that will be normalized\n>> $ git commit -m \"Introduce end-of-line normalization\"\n> \n> Is the way I did it that worked also a valid solution? Or did it only\n> work accidentally? Again the command I ran that worked is:\n> \n> $ git rm -r --cached . && git add .\n\n(Both should work)\n\nTo be honest, from the documentation, I can't figure out the difference between\n$ git read-tree --empty\nand\n$ git rm -r --cached .\n\nDoes anybody remember the discussion, why we ended up with read-tree ?\n"},{"id":"329631","messageId":"xmqq8tgrhdt0.fsf@gitster.mtv.corp.google.com","threadId":"46891","inReplyTo":"88c57b88-ef2c-d7db-15e2-12791d4cb201@web.de","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-04T02:00:59Z","receivedAt":"2017-10-04T02:01:07Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen writes:\n\n>> $ git rm -r --cached . && git add .\n>\n> (Both should work)\n>\n> To be honest, from the documentation, I can't figure out the difference between\n> $ git read-tree --empty\n> and\n> $ git rm -r --cached .\n>\n> Does anybody remember the discussion, why we ended up with read-tree ?\n\nWe used to use neither, and considered it fine to \"rm .git/index\" if\nyou wanted to empty the on-disk index file in the old world. In the\nmodern world, folks want you to avoid touching filesystem directly\nand instead want you to use Git tools, and the above are two obvious\nways to do so.\n\n\"git read-tree\" (without any parameter, i.e. \"read these 0 trees and\npopulate the index with it\") and its modern and preferred synonym\n\"git read-tree --empty\" (i.e. \"I am giving 0 trees and I know the\nsole effect of this command is to empty the index.\") are more direct\nways to express \"I want the index emptied\" between the two.\n\nThe other one, \"git rm -r --cached .\", in the end gives you the same\nstate because it tells Git to \"iterate over all the entries in the\nindex, find the ones that match pathspec '.', and remove them from\nthe index.\". It is not wrong per-se, but conceptually it is a bit\nroundabout way to say that \"I want the index emptied\", I would\nthink.\n\nI wouldn't be surprised if the \"rm -r --cached .\" were a lot slower,\ndue to the overhead of having to do the pathspec filtering that ends\nup to be a no-op, but there shouldn't be a difference in the end\nresult.\n"},{"id":"329693","messageId":"CAHd499BCMOcur6NRbKfwd81zpnyzoiVCY54d=UmGpAhbD7AVTA@mail.gmail.com","threadId":"46891","inReplyTo":"xmqq8tgrhdt0.fsf@gitster.mtv.corp.google.com","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Robert Dailey","fromEmail":"rcdailey.lists@gmail.com","sentAt":"2017-10-04T16:26:55Z","receivedAt":"2017-10-04T16:27:15Z","isPatch":false,"sender":{"key":"rcdailey.lists@gmail.com","avatar":null},"body":"On Tue, Oct 3, 2017 at 9:00 PM, Junio C Hamano wrote:\n> Torsten Bögershausen writes:\n>\n>>> $ git rm -r --cached . && git add .\n>>\n>> (Both should work)\n>>\n>> To be honest, from the documentation, I can't figure out the difference between\n>> $ git read-tree --empty\n>> and\n>> $ git rm -r --cached .\n>>\n>> Does anybody remember the discussion, why we ended up with read-tree ?\n>\n> We used to use neither, and considered it fine to \"rm .git/index\" if\n> you wanted to empty the on-disk index file in the old world. In the\n> modern world, folks want you to avoid touching filesystem directly\n> and instead want you to use Git tools, and the above are two obvious\n> ways to do so.\n>\n> \"git read-tree\" (without any parameter, i.e. \"read these 0 trees and\n> populate the index with it\") and its modern and preferred synonym\n> \"git read-tree --empty\" (i.e. \"I am giving 0 trees and I know the\n> sole effect of this command is to empty the index.\") are more direct\n> ways to express \"I want the index emptied\" between the two.\n>\n> The other one, \"git rm -r --cached .\", in the end gives you the same\n> state because it tells Git to \"iterate over all the entries in the\n> index, find the ones that match pathspec '.', and remove them from\n> the index.\". It is not wrong per-se, but conceptually it is a bit\n> roundabout way to say that \"I want the index emptied\", I would\n> think.\n>\n> I wouldn't be surprised if the \"rm -r --cached .\" were a lot slower,\n> due to the overhead of having to do the pathspec filtering that ends\n> up to be a no-op, but there shouldn't be a difference in the end\n> result.\n\nYou guys are obviously worlds ahead of me on the internals of things,\nbut from my perspective I like to avoid the \"plumbing\" commands as\nmuch as I can. Even if I used them, if I have to tell the rest of my\nteam \"this is the way to do it\", they're going to give me dirty looks\nif I ask them to run things like this that make no sense to them.\nThat's the argument I have to deal with when it comes to Git's\nusability within the team I manage. So based on this, I'd favor the\n`git rm -r --cached` approach because this is the more common result\nyou see in Google, and also makes a little more sense from a high\nlevel of thought perspective. However, this is just my personal\nopinion. `read-tree --empty` is far less self explanatory IMHO.\n\nAlso let's not forget the second part of the command chain that\nresults in the different behavior. In one case, I use `git add` which\nresults in proper line ending normalization. In the other case, I do\n`git reset --hard` which does *NOT* result in the line endings\nnormalized (`git status` shows no results). In both cases, I'm still\ndoing `git rm -r --cached`, so I am doubtful that is the root cause\nfor the line ending normalization piece. I'm still trying to\nunderstand why both give different results (root cause) and also get\nan understanding of what the correct (modern) solution is for line\nending normalization (not necessarily which is the right way to\nclear/delete the index, which is really AFAIK just a means to this end\nand an implementation detail of sorts for this specific task).\n"},{"id":"329694","messageId":"20171004165947.GN19555@aiede.mtv.corp.google.com","threadId":"46891","inReplyTo":"CAHd499BCMOcur6NRbKfwd81zpnyzoiVCY54d=UmGpAhbD7AVTA@mail.gmail.com","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-04T16:59:47Z","receivedAt":"2017-10-04T16:59:55Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Robert,\n\nRobert Dailey wrote:\n\n> You guys are obviously worlds ahead of me on the internals of things,\n> but from my perspective I like to avoid the \"plumbing\" commands as\n> much as I can.\n\nI suspect what we are dancing around is the need for some command like\n\n\tgit checkout --renormalize .\n\nwhich would shorten the sequence to\n\n\tgit checkout --renormalize .\n\tgit status; # Show files that will be normalized\n\tgit commit; # Commit the result\n\nWhat do you think? Would you be interested in writing a patch for it?\n(\"No\" is as always an acceptable answer.)\n\nThanks,\nJonathan\n"},{"id":"329700","messageId":"CAHd499DWbwBFxPUB6KFQ7+PbFQjdaRc6MwZRTj6dx+Wz9WHC6A@mail.gmail.com","threadId":"46891","inReplyTo":"20171004165947.GN19555@aiede.mtv.corp.google.com","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Robert Dailey","fromEmail":"rcdailey.lists@gmail.com","sentAt":"2017-10-04T18:03:48Z","receivedAt":"2017-10-04T18:03:54Z","isPatch":false,"sender":{"key":"rcdailey.lists@gmail.com","avatar":null},"body":"On Wed, Oct 4, 2017 at 11:59 AM, Jonathan Nieder wrote:\n> Hi Robert,\n>\n> Robert Dailey wrote:\n>\n>> You guys are obviously worlds ahead of me on the internals of things,\n>> but from my perspective I like to avoid the \"plumbing\" commands as\n>> much as I can.\n>\n> I suspect what we are dancing around is the need for some command like\n>\n> git checkout --renormalize .\n>\n> which would shorten the sequence to\n>\n> git checkout --renormalize .\n> git status; # Show files that will be normalized\n> git commit; # Commit the result\n>\n> What do you think? Would you be interested in writing a patch for it?\n> (\"No\" is as always an acceptable answer.)\n\nI wish I could, but ultimately I'd probably not be able to do it. I\nrarely have time to do recreational coding outside of work these days.\n\nThat aside, for now I want to know the proper & recommended method to\nrenormalize line endings using existing commands. Additionally I also\nam interested in knowing why only 1 of the 3 solutions I tried (In my\nOP) worked but the others didn't. My short term goal is just to get\neducated a bit. There's so much conflicting and variable information\non this topic on Google. It makes it difficult to find the one true\npath, especially since Git evolves and improves over time and\ninformation usually becomes stale.\n"},{"id":"329707","messageId":"20171004211734.GA25379@tor.lan","threadId":"46891","inReplyTo":"CAHd499BCMOcur6NRbKfwd81zpnyzoiVCY54d=UmGpAhbD7AVTA@mail.gmail.com","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-10-04T21:17:34Z","receivedAt":"2017-10-04T21:17:46Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Wed, Oct 04, 2017 at 11:26:55AM -0500, Robert Dailey wrote:\n> On Tue, Oct 3, 2017 at 9:00 PM, Junio C Hamano wrote:\n> > Torsten Bögershausen writes:\n> >\n> >>> $ git rm -r --cached . && git add .\n> >>\n> >> (Both should work)\n> >>\n> >> To be honest, from the documentation, I can't figure out the difference between\n> >> $ git read-tree --empty\n> >> and\n> >> $ git rm -r --cached .\n> >>\n> >> Does anybody remember the discussion, why we ended up with read-tree ?\n> >\n> > We used to use neither, and considered it fine to \"rm .git/index\" if\n> > you wanted to empty the on-disk index file in the old world. In the\n> > modern world, folks want you to avoid touching filesystem directly\n> > and instead want you to use Git tools, and the above are two obvious\n> > ways to do so.\n> >\n> > \"git read-tree\" (without any parameter, i.e. \"read these 0 trees and\n> > populate the index with it\") and its modern and preferred synonym\n> > \"git read-tree --empty\" (i.e. \"I am giving 0 trees and I know the\n> > sole effect of this command is to empty the index.\") are more direct\n> > ways to express \"I want the index emptied\" between the two.\n> >\n> > The other one, \"git rm -r --cached .\", in the end gives you the same\n> > state because it tells Git to \"iterate over all the entries in the\n> > index, find the ones that match pathspec '.', and remove them from\n> > the index.\". It is not wrong per-se, but conceptually it is a bit\n> > roundabout way to say that \"I want the index emptied\", I would\n> > think.\n> >\n> > I wouldn't be surprised if the \"rm -r --cached .\" were a lot slower,\n> > due to the overhead of having to do the pathspec filtering that ends\n> > up to be a no-op, but there shouldn't be a difference in the end\n> > result.\n> \n> You guys are obviously worlds ahead of me on the internals of things,\n> but from my perspective I like to avoid the \"plumbing\" commands as\n> much as I can. Even if I used them, if I have to tell the rest of my\n> team \"this is the way to do it\", they're going to give me dirty looks\n> if I ask them to run things like this that make no sense to them.\n> That's the argument I have to deal with when it comes to Git's\n> usability within the team I manage. So based on this, I'd favor the\n> `git rm -r --cached` approach because this is the more common result\n> you see in Google, and also makes a little more sense from a high\n> level of thought perspective. However, this is just my personal\n> opinion. `read-tree --empty` is far less self explanatory IMHO.\n> \n> Also let's not forget the second part of the command chain that\n> results in the different behavior. In one case, I use `git add` which\n> results in proper line ending normalization. In the other case, I do\n> `git reset --hard` which does *NOT* result in the line endings\n> normalized (`git status` shows no results). In both cases, I'm still\n> doing `git rm -r --cached`, so I am doubtful that is the root cause\n> for the line ending normalization piece. I'm still trying to\n> understand why both give different results (root cause) and also get\n> an understanding of what the correct (modern) solution is for line\n> ending normalization (not necessarily which is the right way to\n> clear/delete the index, which is really AFAIK just a means to this end\n> and an implementation detail of sorts for this specific task).\n\nHopefully I am able to give a useful answer.\n\n\"git reset --hard\" works like a hammer\nand may destroy work that has been done,\nin our case the cleaning of the index,\nwhich is needed for normalization since Git 2.10 (or so)\n\nBack to the question:\nOne solution, which you can tell your team, is this one:\n$ git rm -r --cached . && git add .\n\nAnd as Junio pointed out, this may be slower than needed.\nAnd we don't want \"slow\" solutions in the official documentation ;-)\n\nWhatever you find on search engines may get stale after a while,\nso that we appreciate direct questions here.\n\n(And I will open an issue on Github the next days)\n\nThe background is that the CRLF handling in Git changed over the years,\nand one effect is that \"git reset\" is not \"allowed\" any more.\n\nFor the interested reader:\nhttps://github.com/git-for-windows/git/issues/954\n\n"},{"id":"329712","messageId":"xmqqbmlm9y94.fsf@gitster.mtv.corp.google.com","threadId":"46891","inReplyTo":"20171004165947.GN19555@aiede.mtv.corp.google.com","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-05T01:31:03Z","receivedAt":"2017-10-05T01:31:11Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> I suspect what we are dancing around is the need for some command like\n>\n> \tgit checkout --renormalize .\n>\n> which would shorten the sequence to\n>\n> \tgit checkout --renormalize .\n> \tgit status; # Show files that will be normalized\n> \tgit commit; # Commit the result\n>\n> What do you think? Would you be interested in writing a patch for it?\n> (\"No\" is as always an acceptable answer.)\n\nI actually think what is being requested is the opposite, i.e. \"the\nobject registered in the index have wrong line endings, and the\nsafe-crlf is getting in the way to prevent me from correcting by\nhashing the working tree contents again to register contents with\ncorrected line endings, even with 'git add .'\".\n\nSo I would understand if your suggestion were for\n\n\tgit checkin --renormalize .\n\nbut not \"git checkout\". And it probably is more familiar to lay\npeople if we spelled that as \"git add --renormalize .\" ;-)\n\n\n"},{"id":"329713","messageId":"xmqq7ewa9xw6.fsf@gitster.mtv.corp.google.com","threadId":"46891","inReplyTo":"20171004211734.GA25379@tor.lan","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-05T01:38:49Z","receivedAt":"2017-10-05T01:38:56Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen writes:\n\n> One solution, which you can tell your team, is this one:\n> $ git rm -r --cached . && git add .\n\nBoth this and its \"git read-tree --empty\" cousin share a grave\nissue. The \"git add .\" step would mean that before doing these\ncommands, your working tree must be truly clean, i.e. the paths\nin the filesystem known to the index must match what is in the\nindex (modulo the line-ending gotcha you are trying to correct), \n*AND* there must be *NO* untracked paths you do not want to add\nin the working tree.\n\nThat is a reason why we should solve it differently. Perhaps adding\na new option \"git add --rehash\" to tell Git \"Hey, you may think some\npaths in the index and in the working tree are identical and no need\nto re-register, but you are WRONG. For each path in the index,\nremove it and then register the object by hashing the contents from\nthe filesystem afresh!\" would be the best way to go. That will not\npick up untracked paths left in the filesystem, and does not limit\nour solution to the \"eol normalization is screwey\" issue by not\ncalling the option \"renormalize\" or any other words that imply \"why\"\nwe are hashing again anew.\n\n"},{"id":"329715","messageId":"20171005014651.GT19555@aiede.mtv.corp.google.com","threadId":"46891","inReplyTo":"xmqqbmlm9y94.fsf@gitster.mtv.corp.google.com","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-05T01:46:51Z","receivedAt":"2017-10-05T01:46:59Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Junio C Hamano wrote:\n> Jonathan Nieder writes:\n\n>> \tgit checkout --renormalize .\n>> \tgit status; # Show files that will be normalized\n>> \tgit commit; # Commit the result\n>>\n>> What do you think? Would you be interested in writing a patch for it?\n>> (\"No\" is as always an acceptable answer.)\n>\n> I actually think what is being requested is the opposite, i.e. \"the\n> object registered in the index have wrong line endings, and the\n> safe-crlf is getting in the way to prevent me from correcting by\n> hashing the working tree contents again to register contents with\n> corrected line endings, even with 'git add .'\".\n>\n> So I would understand if your suggestion were for\n>\n> \tgit checkin --renormalize .\n>\n> but not \"git checkout\". And it probably is more familiar to lay\n> people if we spelled that as \"git add --renormalize .\" ;-)\n\nGood catch. You understood correctly --- \"git add --renormalize\" is\nthe feature that I think is being hinted at here.\n\nThanks,\nJonathan\n"},{"id":"329720","messageId":"xmqqo9pm8e4l.fsf@gitster.mtv.corp.google.com","threadId":"46891","inReplyTo":"xmqq7ewa9xw6.fsf@gitster.mtv.corp.google.com","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-05T03:31:06Z","receivedAt":"2017-10-05T03:31:14Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Both this and its \"git read-tree --empty\" cousin share a grave\n> issue. The \"git add .\" step would mean that before doing these\n> commands, your working tree must be truly clean, i.e. the paths\n> in the filesystem known to the index must match what is in the\n> index (modulo the line-ending gotcha you are trying to correct), \n> *AND* there must be *NO* untracked paths you do not want to add\n> in the working tree.\n>\n> That is a reason why we should solve it differently. Perhaps adding\n> a new option \"git add --rehash\" to tell Git \"Hey, you may think some\n> paths in the index and in the working tree are identical and no need\n> to re-register, but you are WRONG. For each path in the index,\n> remove it and then register the object by hashing the contents from\n> the filesystem afresh!\" would be the best way to go.\n\nHere is just to illustrate the direction I was heading to in the\nabove. This is not even compile tested and I won't guarantee what\ncorner cases there are, though.\n\nIn a true production code, we shouldn't be using string-list with\ntwo loops, but I just didn't want to spend more braincycles worrying\nabout removing from the list and then adding to it, both inside a\nsingle loop that iterates over it in a mere illustration patch.\n\nThe second loop uses a simple \"remove then add\", but I think it\nshould rather be a \"mark ce that it will _never_ match anything on\nthe working tree\" followed by \"add_file_to_cache()\". Currently we\ndo not have the \"mark ce that it never matches\" operation that lets\nus bypass the comparison with the current cache entry (with safecrlf\nthing that interferes), but we can afford to use a (in-core only)\nbit in the ce flags word to represent this and plumb it through.\nThat way, we will still preserve the executable bit from the\noriginal entry, hopefully ;-)\n\n\n builtin/add.c | 42 +++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 41 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 5d5773d5cd..264f84dbe7 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -26,6 +26,7 @@ static const char * const builtin_add_usage[] = {\n };\n static int patch_interactive, add_interactive, edit_interactive;\n static int take_worktree_changes;\n+static int rehash;\n \n struct update_callback_data {\n \tint flags;\n@@ -121,6 +122,41 @@ int add_files_to_cache(const char *prefix,\n \treturn !!data.add_errors;\n }\n \n+static int rehash_tracked_files(const char *prefix, const struct pathspec *pathspec,\n+\t\t\t\tint flags)\n+{\n+\tstruct string_list paths = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *path;\n+\tint i, retval = 0;\n+\n+\tfor (i = 0; i < active_nr; i++) {\n+\t\tstruct cache_entry *ce = active_cache[i];\n+\n+\t\tif (ce_stage(ce))\n+\t\t\tcontinue; /* do not touch unmerged paths */\n+\t\tif (!S_ISREG(ce->ce_mode) && !S_ISLNK(ce->ce_mode))\n+\t\t\tcontinue; /* do not touch non blobs */\n+\t\tif (pathspec && !ce_path_match(ce, pathspec, NULL))\n+\t\t\tcontinue;\n+\t\tstring_list_append(&paths, ce->name);\n+\t}\n+\n+\tfor_each_string_list_item(path, &paths) {\n+\t\t/*\n+\t\t * Having a blob contaminated with CR will trigger the\n+\t\t * safe-crlf kludge, avoidance of which is the primary\n+\t\t * thing this helper function exists. Remove it and\n+\t\t * then re-add it. Note that this may lose executable\n+\t\t * bit on a filesystem that lacks it.\n+\t\t */\n+\t\tremove_file_from_cache(path->string);\n+\t\tadd_file_to_cache(path->string, flags);\n+\t}\n+\n+\tstring_list_clear(&paths, 0);\n+\treturn retval;\n+}\n+\n static char *prune_directory(struct dir_struct *dir, struct pathspec *pathspec, int prefix)\n {\n \tchar *seen;\n@@ -274,6 +310,7 @@ static struct option builtin_add_options[] = {\n \tOPT_BOOL('e', \"edit\", &edit_interactive, N_(\"edit current diff and apply\")),\n \tOPT__FORCE(&ignored_too, N_(\"allow adding otherwise ignored files\")),\n \tOPT_BOOL('u', \"update\", &take_worktree_changes, N_(\"update tracked files\")),\n+\tOPT_BOOL(0, \"rehash\", &rehash, N_(\"really update tracked files\")),\n \tOPT_BOOL('N', \"intent-to-add\", &intent_to_add, N_(\"record only the fact that the path will be added later\")),\n \tOPT_BOOL('A', \"all\", &addremove_explicit, N_(\"add changes from all tracked and untracked files\")),\n \t{ OPTION_CALLBACK, 0, \"ignore-removal\", &addremove_explicit,\n@@ -498,7 +535,10 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \n \tplug_bulk_checkin();\n \n-\texit_status |= add_files_to_cache(prefix, &pathspec, flags);\n+\tif (rehash)\n+\t\texit_status |= rehash_tracked_files(prefix, &pathspec, flags);\n+\telse\n+\t\texit_status |= add_files_to_cache(prefix, &pathspec, flags);\n \n \tif (add_new_files)\n \t\texit_status |= add_files(&dir, flags);\n"},{"id":"329816","messageId":"20171005214248.GA31593@tor.lan","threadId":"46891","inReplyTo":"xmqqo9pm8e4l.fsf@gitster.mtv.corp.google.com","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-10-05T21:42:48Z","receivedAt":"2017-10-05T21:42:59Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":" \n> builtin/add.c | 42 +++++++++++++++++++++++++++++++++++++++++-\n> 1 file changed, 41 insertions(+), 1 deletion(-)\n> \n> diff --git a/builtin/add.c b/builtin/add.c\n> index 5d5773d5cd..264f84dbe7 100644\n> --- a/builtin/add.c\n> +++ b/builtin/add.c\n> @@ -26,6 +26,7 @@ static const char * const builtin_add_usage[] = {\n> };\n> static int patch_interactive, add_interactive, edit_interactive;\n> static int take_worktree_changes;\n> +static int rehash;\n> \n> struct update_callback_data {\n> \tint flags;\n> @@ -121,6 +122,41 @@ int add_files_to_cache(const char *prefix,\n> \treturn !!data.add_errors;\n> }\n> \n> +static int rehash_tracked_files(const char *prefix, const struct pathspec *pathspec,\n> +\t\t\t\tint flags)\n> +{\n> +\tstruct string_list paths = STRING_LIST_INIT_DUP;\n> +\tstruct string_list_item *path;\n> +\tint i, retval = 0;\n> +\n> +\tfor (i = 0; i < active_nr; i++) {\n> +\t\tstruct cache_entry *ce = active_cache[i];\n> +\n> +\t\tif (ce_stage(ce))\n> +\t\t\tcontinue; /* do not touch unmerged paths */\n> +\t\tif (!S_ISREG(ce->ce_mode) && !S_ISLNK(ce->ce_mode))\n> +\t\t\tcontinue; /* do not touch non blobs */\n> +\t\tif (pathspec && !ce_path_match(ce, pathspec, NULL))\n> +\t\t\tcontinue;\n> +\t\tstring_list_append(&paths, ce->name);\n> +\t}\n> +\n> +\tfor_each_string_list_item(path, &paths) {\n> +\t\t/*\n> +\t\t * Having a blob contaminated with CR will trigger the\n> +\t\t * safe-crlf kludge, avoidance of which is the primary\n> +\t\t * thing this helper function exists. Remove it and\n> +\t\t * then re-add it. Note that this may lose executable\n> +\t\t * bit on a filesystem that lacks it.\n> +\t\t */\n> +\t\tremove_file_from_cache(path->string);\n> +\t\tadd_file_to_cache(path->string, flags);\n> +\t}\n> +\n> +\tstring_list_clear(&paths, 0);\n> +\treturn retval;\n> +}\n> +\n> static char *prune_directory(struct dir_struct *dir, struct pathspec *pathspec, int prefix)\n> {\n> \tchar *seen;\n> @@ -274,6 +310,7 @@ static struct option builtin_add_options[] = {\n> \tOPT_BOOL('e', \"edit\", &edit_interactive, N_(\"edit current diff and apply\")),\n> \tOPT__FORCE(&ignored_too, N_(\"allow adding otherwise ignored files\")),\n> \tOPT_BOOL('u', \"update\", &take_worktree_changes, N_(\"update tracked files\")),\n> +\tOPT_BOOL(0, \"rehash\", &rehash, N_(\"really update tracked files\")),\n> \tOPT_BOOL('N', \"intent-to-add\", &intent_to_add, N_(\"record only the fact that the path will be added later\")),\n> \tOPT_BOOL('A', \"all\", &addremove_explicit, N_(\"add changes from all tracked and untracked files\")),\n> \t{ OPTION_CALLBACK, 0, \"ignore-removal\", &addremove_explicit,\n> @@ -498,7 +535,10 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n> \n> \tplug_bulk_checkin();\n> \n> -\texit_status |= add_files_to_cache(prefix, &pathspec, flags);\n> +\tif (rehash)\n> +\t\texit_status |= rehash_tracked_files(prefix, &pathspec, flags);\n> +\telse\n> +\t\texit_status |= add_files_to_cache(prefix, &pathspec, flags);\n> \n> \tif (add_new_files)\n> \t\texit_status |= add_files(&dir, flags);\n\nThat looks like a nice one.\nBefore we put this into stone:\nDoes it make sense to say \"renormalize\" instead of \"rehash\" ?\n(That term does exist already for merge.\n And rehash is more a technical term, rather then a user-point-of-view explanation)\n \n"},{"id":"329825","messageId":"xmqqa8156rok.fsf@gitster.mtv.corp.google.com","threadId":"46891","inReplyTo":"20171005214248.GA31593@tor.lan","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-06T00:33:31Z","receivedAt":"2017-10-06T00:33:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen writes:\n\n> Before we put this into stone:\n> Does it make sense to say \"renormalize\" instead of \"rehash\" ?\n> (That term does exist already for merge.\n> And rehash is more a technical term, rather then a user-point-of-view explanation)\n\nI do not mind \"renormalize\" at all.\n\nAs to the toy patch, I think it needs to (at least by default) turn\noff the add_new_files codepath, and be allowed to work without any\npathspec (in which case all tracked paths should be renormalized).\n\nAnd we really shouldn't do the \"rm && add\", which would not work\nwell on platforms where filesystem without executing-bit support is\nprevalent. This new feature is primarily needed on platforms where\nCRLF line endings are used, and unfortunately these two sets of\nplatforms overlap quite a bit X-<.\n\n\n"},{"id":"329909","messageId":"20171006175815.GA31970@tor.lan","threadId":"46891","inReplyTo":"xmqqa8156rok.fsf@gitster.mtv.corp.google.com","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-10-06T17:58:15Z","receivedAt":"2017-10-06T17:58:29Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Fri, Oct 06, 2017 at 09:33:31AM +0900, Junio C Hamano wrote:\n> Torsten Bögershausen writes:\n> \n> > Before we put this into stone:\n> > Does it make sense to say \"renormalize\" instead of \"rehash\" ?\n> > (That term does exist already for merge.\n> > And rehash is more a technical term, rather then a user-point-of-view explanation)\n> \n> I do not mind \"renormalize\" at all.\n> \n> As to the toy patch, I think it needs to (at least by default) turn\n> off the add_new_files codepath, and be allowed to work without any\n> pathspec (in which case all tracked paths should be renormalized).\n> \n\nOK, then I will pick up your patch in a couple of days/weeks, and push it further then\n(Documentation, test cases, other ?)\n\n\n"},{"id":"330459","messageId":"20171016164907.4201-1-tboegi@web.de","threadId":"46891","inReplyTo":"xmqqo9pm8e4l.fsf@gitster.mtv.corp.google.com","subject":"[PATCH v1 1/1] Introduce git add --renormalize .","fromName":"","fromEmail":"tboegi@web.de","sentAt":"2017-10-16T16:49:07Z","receivedAt":"2017-10-16T16:49:19Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"From: Torsten Bögershausen \n\nMake it safer to normalize the line endings in a repository:\nFiles that had been commited with CRLF will be commited with LF.\n(Unless core.autorclf and .gitattributes specify that Git\n should not do line ending conversions)\n\nThe old way to normalize a repo was like this:\n # Make sure that there are not untracked files\n $ echo \"* text=auto\" >.gitattributes\n $ git read-tree --empty\n $ git add .\n $ git commit -m \"Introduce end-of-line normalization\"\n\nThe new method is one step shorter, more intuitive and does not\nadd untracked files:\n $ echo \"* text=auto\" >.gitattributes\n $ git add --renormalize .\n $ git commit -m \"Introduce end-of-line normalization\"\n\nNote that \"git add --renormalize \" is the short form for\n\"git add -u --renormalize \".\n\nSigned-off-by: Torsten Bögershausen \n---\n Documentation/git-add.txt | 8 +++++++-\n Documentation/gitattributes.txt | 3 +--\n builtin/add.c | 27 +++++++++++++++++++++++++--\n cache.h | 1 +\n convert.c | 1 +\n environment.c | 1 +\n read-cache.c | 24 ++++++++++++++----------\n t/t0025-crlf-renormalize.sh | 30 ++++++++++++++++++++++++++++++\n 8 files changed, 80 insertions(+), 15 deletions(-)\n create mode 100755 t/t0025-crlf-renormalize.sh\n\ndiff --git a/Documentation/git-add.txt b/Documentation/git-add.txt\nindex f4169fb1ec..b6e431903d 100644\n--- a/Documentation/git-add.txt\n+++ b/Documentation/git-add.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n [verse]\n 'git add' [--verbose | -v] [--dry-run | -n] [--force | -f] [--interactive | -i] [--patch | -p]\n \t [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]]\n-\t [--intent-to-add | -N] [--refresh] [--ignore-errors] [--ignore-missing]\n+\t [--intent-to-add | -N] [--refresh] [--ignore-errors] [--ignore-missing] [--renormalize]\n \t [--chmod=(+|-)x] [--] [...]\n \n DESCRIPTION\n@@ -172,6 +172,12 @@ for \"git add --no-all ...\", i.e. ignored removed files.\n \twarning (e.g., if you are manually performing operations on\n \tsubmodules).\n \n+--renormalize::\n+\tNormalizes the line endings from CRLF to LF of tracked files.\n+\tThis applies to files which are either \"text\" or \"text=auto\"\n+\tin .gitattributes (or core.autocrlf is true or input)\n+ --renormalize implies -u\n+\n --chmod=(+|-)x::\n \tOverride the executable bit of the added files. The executable\n \tbit is only changed in the index, the files on disk are left\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex 4c68bc19d5..071dec2bc4 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -232,8 +232,7 @@ From a clean working directory:\n \n -------------------------------------------------\n $ echo \"* text=auto\" >.gitattributes\n-$ git read-tree --empty # Clean index, force re-scan of working directory\n-$ git add .\n+$ git add --renormalize .\n $ git status # Show files that will be normalized\n $ git commit -m \"Introduce end-of-line normalization\"\n -------------------------------------------------\ndiff --git a/builtin/add.c b/builtin/add.c\nindex a648cf4c56..ee8e756fdc 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -123,6 +123,25 @@ int add_files_to_cache(const char *prefix,\n \treturn !!data.add_errors;\n }\n \n+static int renormalize_tracked_files(const struct pathspec *pathspec, int flags)\n+{\n+\tint i, retval = 0;\n+\n+\tfor (i = 0; i < active_nr; i++) {\n+\t\tstruct cache_entry *ce = active_cache[i];\n+\n+\t\tif (ce_stage(ce))\n+\t\t\tcontinue; /* do not touch unmerged paths */\n+\t\tif (!S_ISREG(ce->ce_mode) && !S_ISLNK(ce->ce_mode))\n+\t\t\tcontinue; /* do not touch non blobs */\n+\t\tif (pathspec && !ce_path_match(ce, pathspec, NULL))\n+\t\t\tcontinue;\n+\t\tretval |= add_file_to_cache(ce->name, flags);\n+\t}\n+\n+\treturn retval;\n+}\n+\n static char *prune_directory(struct dir_struct *dir, struct pathspec *pathspec, int prefix)\n {\n \tchar *seen;\n@@ -276,6 +295,7 @@ static struct option builtin_add_options[] = {\n \tOPT_BOOL('e', \"edit\", &edit_interactive, N_(\"edit current diff and apply\")),\n \tOPT__FORCE(&ignored_too, N_(\"allow adding otherwise ignored files\")),\n \tOPT_BOOL('u', \"update\", &take_worktree_changes, N_(\"update tracked files\")),\n+\tOPT_BOOL(0, \"renormalize\", &add_renormalize, N_(\"renormalize EOL of tracked files (implies -u)\")),\n \tOPT_BOOL('N', \"intent-to-add\", &intent_to_add, N_(\"record only the fact that the path will be added later\")),\n \tOPT_BOOL('A', \"all\", &addremove_explicit, N_(\"add changes from all tracked and untracked files\")),\n \t{ OPTION_CALLBACK, 0, \"ignore-removal\", &addremove_explicit,\n@@ -406,7 +426,7 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \t\t\t chmod_arg[1] != 'x' || chmod_arg[2]))\n \t\tdie(_(\"--chmod param '%s' must be either -x or +x\"), chmod_arg);\n \n-\tadd_new_files = !take_worktree_changes && !refresh_only;\n+\tadd_new_files = !take_worktree_changes && !refresh_only && !add_renormalize;\n \trequire_pathspec = !(take_worktree_changes || (0 < addremove_explicit));\n \n \thold_locked_index(&lock_file, LOCK_DIE_ON_ERROR);\n@@ -500,7 +520,10 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \n \tplug_bulk_checkin();\n \n-\texit_status |= add_files_to_cache(prefix, &pathspec, flags);\n+\tif (add_renormalize)\n+\t\texit_status |= renormalize_tracked_files(&pathspec, flags);\n+\telse\n+\t\texit_status |= add_files_to_cache(prefix, &pathspec, flags);\n \n \tif (add_new_files)\n \t\texit_status |= add_files(&dir, flags);\ndiff --git a/cache.h b/cache.h\nindex a916bc79e3..d4e8f86696 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -489,6 +489,7 @@ extern const char *get_git_namespace(void);\n extern const char *strip_namespace(const char *namespaced_ref);\n extern const char *get_super_prefix(void);\n extern const char *get_git_work_tree(void);\n+extern int add_renormalize;\n \n /*\n * Return true if the given path is a git directory; note that this _just_\ndiff --git a/convert.c b/convert.c\nindex a09935cb81..70a37b0942 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -290,6 +290,7 @@ static int crlf_to_git(const struct index_state *istate,\n \t\t * cherry-pick.\n \t\t */\n \t\tif ((checksafe != SAFE_CRLF_RENORMALIZE) &&\n+\t\t !add_renormalize &&\n \t\t has_cr_in_index(istate, path))\n \t\t\tconvert_crlf_into_lf = 0;\n \t}\ndiff --git a/environment.c b/environment.c\nindex f1f934b6fd..16fbce1123 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -87,6 +87,7 @@ int auto_comment_line_char;\n /* Parallel index stat data preload? */\n int core_preload_index = 1;\n \n+int add_renormalize;\n /*\n * This is a hack for test programs like test-dump-untracked-cache to\n * ensure that they do not modify the untracked cache when reading it.\ndiff --git a/read-cache.c b/read-cache.c\nindex b211c57af6..3388c5053f 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -630,7 +630,7 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n {\n \tint size, namelen, was_same;\n \tmode_t st_mode = st->st_mode;\n-\tstruct cache_entry *ce, *alias;\n+\tstruct cache_entry *ce, *alias = NULL;\n \tunsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY;\n \tint verbose = flags & (ADD_CACHE_VERBOSE | ADD_CACHE_PRETEND);\n \tint pretend = flags & ADD_CACHE_PRETEND;\n@@ -677,16 +677,20 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \tif (ignore_case) {\n \t\tadjust_dirname_case(istate, ce->name);\n \t}\n+\tif (!add_renormalize) {\n+\t\talias = index_file_exists(istate, ce->name,\n+\t\t\t\t\t ce_namelen(ce), ignore_case);\n+\t\tif (alias &&\n+\t\t !ce_stage(alias) &&\n+\t\t !ie_match_stat(istate, alias, st, ce_option)) {\n+\t\t\t/* Nothing changed, really */\n+\t\t\tif (!S_ISGITLINK(alias->ce_mode))\n+\t\t\t\tce_mark_uptodate(alias);\n+\t\t\talias->ce_flags |= CE_ADDED;\n \n-\talias = index_file_exists(istate, ce->name, ce_namelen(ce), ignore_case);\n-\tif (alias && !ce_stage(alias) && !ie_match_stat(istate, alias, st, ce_option)) {\n-\t\t/* Nothing changed, really */\n-\t\tif (!S_ISGITLINK(alias->ce_mode))\n-\t\t\tce_mark_uptodate(alias);\n-\t\talias->ce_flags |= CE_ADDED;\n-\n-\t\tfree(ce);\n-\t\treturn 0;\n+\t\t\tfree(ce);\n+\t\t\treturn 0;\n+\t\t}\n \t}\n \tif (!intent_only) {\n \t\tif (index_path(&ce->oid, path, st, HASH_WRITE_OBJECT)) {\ndiff --git a/t/t0025-crlf-renormalize.sh b/t/t0025-crlf-renormalize.sh\nnew file mode 100755\nindex 0000000000..fb1ed631d2\n--- /dev/null\n+++ b/t/t0025-crlf-renormalize.sh\n@@ -0,0 +1,30 @@\n+#!/bin/sh\n+\n+test_description='CRLF renormalization'\n+\n+. ./test-lib.sh\n+\n+test_expect_success setup '\n+\tgit config core.autocrlf false &&\n+\tprintf \"LINEONE\\nLINETWO\\nLINETHREE\" >LF.txt &&\n+\tprintf \"LINEONE\\r\\nLINETWO\\r\\nLINETHREE\" >CRLF.txt &&\n+\tprintf \"LINEONE\\r\\nLINETWO\\nLINETHREE\" >CRLF_mix_LF.txt &&\n+\tgit add . &&\n+\tgit commit -m initial\n+'\n+\n+test_expect_success 'renormalize CRLF in repo' '\n+\techo \"*.txt text=auto\" >.gitattributes &&\n+\tgit add --renormalize \"*.txt\" &&\n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.14.1.729.g59c0ea183a\n\n"},{"id":"330462","messageId":"xmqqshejq9mn.fsf@gitster.mtv.corp.google.com","threadId":"46891","inReplyTo":"20171016164907.4201-1-tboegi@web.de","subject":"Re: [PATCH v1 1/1] Introduce git add --renormalize .","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-16T17:34:56Z","receivedAt":"2017-10-16T17:35:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"tboegi@web.de writes:\n\n> From: Torsten Bögershausen \n>\n> Make it safer to normalize the line endings in a repository:\n> Files that had been commited with CRLF will be commited with LF.\n> (Unless core.autorclf and .gitattributes specify that Git\n> should not do line ending conversions)\n\nA few issues I saw after a quick read:\n\n - The log message tells us old and new ways, but does not make it\n clear why users are encouraged to use the new way at all. You\n didn't make the implementation of \"add --renormalize\" to just\n start \"git add\" without calling read_cache() and letting all\n files added new to the index (which is how the old way worked) to\n give a sugarcoated equivalent of the old way for a reason, and\n that should be desribed in the log.\n\n - An ugly global variable is introduced instead of passing\n necessary information through the callchain properly, but the\n title does not say PATCH/RFC.\n\n - The documentation makes it sound as if this new feature is _only_\n about CRLF vs LF. SHouldn't this equally apply after the user\n changes .gitattributes settings that govern the \"clean\" side of\n the filter and makes what is in the index \"unclean\"?\n\nThe second point is a showstopper from maintainability's point of\nview, but none of the above should be insurmojntable.\n\nThanks.\n"},{"id":"331362","messageId":"20171030162913.23164-1-tboegi@web.de","threadId":"46891","inReplyTo":"xmqqshejq9mn.fsf@gitster.mtv.corp.google.com","subject":"[PATCH v2 1/1] Introduce git add --renormalize .","fromName":"","fromEmail":"tboegi@web.de","sentAt":"2017-10-30T16:29:13Z","receivedAt":"2017-10-30T16:29:23Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"From: Torsten Bögershausen \n\nMake it safer to normalize the line endings in a repository:\nFiles that had been commited with CRLF will be commited with LF.\n\nThe old way to normalize a repo was like this:\n # Make sure that there are not untracked files\n $ echo \"* text=auto\" >.gitattributes\n $ git read-tree --empty\n $ git add .\n $ git commit -m \"Introduce end-of-line normalization\"\n\nThe user must make sure that there are no untracked files,\notherwise they would have been added and tracked from now on.\n\nThe new \"add ..renormalize\" does not add untracked files:\n $ echo \"* text=auto\" >.gitattributes\n $ git add --renormalize .\n $ git commit -m \"Introduce end-of-line normalization\"\n\nNote that \"git add --renormalize \" is the short form for\n\"git add -u --renormalize \".\n\nWhile add it, document that the same renormalization may be needed,\nwhenever a clean filter is added or changed.\n\nHelped-By: Junio C Hamano \nSigned-off-by: Torsten Bögershausen \n---\n\nSecond version:\n- Removed the global flag\n- Make clearer that the clean filters may need renormalization\n- commit message improved\n\nDocumentation/git-add.txt | 8 +++++++-\n Documentation/gitattributes.txt | 6 ++++--\n builtin/add.c | 28 ++++++++++++++++++++++++++--\n cache.h | 1 +\n read-cache.c | 30 +++++++++++++++++++-----------\n sha1_file.c | 16 ++++++++++++++--\n t/t0025-crlf-renormalize.sh | 30 ++++++++++++++++++++++++++++++\n 7 files changed, 101 insertions(+), 18 deletions(-)\n create mode 100755 t/t0025-crlf-renormalize.sh\n\ndiff --git a/Documentation/git-add.txt b/Documentation/git-add.txt\nindex b700beaff5..09a08ce4c1 100644\n--- a/Documentation/git-add.txt\n+++ b/Documentation/git-add.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n [verse]\n 'git add' [--verbose | -v] [--dry-run | -n] [--force | -f] [--interactive | -i] [--patch | -p]\n \t [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]]\n-\t [--intent-to-add | -N] [--refresh] [--ignore-errors] [--ignore-missing]\n+\t [--intent-to-add | -N] [--refresh] [--ignore-errors] [--ignore-missing] [--renormalize]\n \t [--chmod=(+|-)x] [--] [...]\n \n DESCRIPTION\n@@ -175,6 +175,12 @@ for \"git add --no-all ...\", i.e. ignored removed files.\n \twarning (e.g., if you are manually performing operations on\n \tsubmodules).\n \n+--renormalize::\n+\tNormalizes the line endings from CRLF to LF of tracked files.\n+\tThis applies to files which are either \"text\" or \"text=auto\"\n+\tin .gitattributes (or core.autocrlf is true or input)\n+\t--renormalize implies -u\n+\n --chmod=(+|-)x::\n \tOverride the executable bit of the added files. The executable\n \tbit is only changed in the index, the files on disk are left\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex 4c68bc19d5..30687de81a 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -232,8 +232,7 @@ From a clean working directory:\n \n -------------------------------------------------\n $ echo \"* text=auto\" >.gitattributes\n-$ git read-tree --empty # Clean index, force re-scan of working directory\n-$ git add .\n+$ git add --renormalize .\n $ git status # Show files that will be normalized\n $ git commit -m \"Introduce end-of-line normalization\"\n -------------------------------------------------\n@@ -328,6 +327,9 @@ You can declare that a filter turns a content that by itself is unusable\n into a usable content by setting the filter..required configuration\n variable to `true`.\n \n+Note: Whenever the clean filter is changed, the repo should be renormalized:\n+$ git add --renormalize .\n+\n For example, in .gitattributes, you would assign the `filter`\n attribute for paths.\n \ndiff --git a/builtin/add.c b/builtin/add.c\nindex a648cf4c56..c42b50f857 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -26,6 +26,7 @@ static const char * const builtin_add_usage[] = {\n };\n static int patch_interactive, add_interactive, edit_interactive;\n static int take_worktree_changes;\n+static int add_renormalize;\n \n struct update_callback_data {\n \tint flags;\n@@ -123,6 +124,25 @@ int add_files_to_cache(const char *prefix,\n \treturn !!data.add_errors;\n }\n \n+static int renormalize_tracked_files(const struct pathspec *pathspec, int flags)\n+{\n+\tint i, retval = 0;\n+\n+\tfor (i = 0; i < active_nr; i++) {\n+\t\tstruct cache_entry *ce = active_cache[i];\n+\n+\t\tif (ce_stage(ce))\n+\t\t\tcontinue; /* do not touch unmerged paths */\n+\t\tif (!S_ISREG(ce->ce_mode) && !S_ISLNK(ce->ce_mode))\n+\t\t\tcontinue; /* do not touch non blobs */\n+\t\tif (pathspec && !ce_path_match(ce, pathspec, NULL))\n+\t\t\tcontinue;\n+\t\tretval |= add_file_to_cache(ce->name, flags | HASH_RENORMALIZE);\n+\t}\n+\n+\treturn retval;\n+}\n+\n static char *prune_directory(struct dir_struct *dir, struct pathspec *pathspec, int prefix)\n {\n \tchar *seen;\n@@ -276,6 +296,7 @@ static struct option builtin_add_options[] = {\n \tOPT_BOOL('e', \"edit\", &edit_interactive, N_(\"edit current diff and apply\")),\n \tOPT__FORCE(&ignored_too, N_(\"allow adding otherwise ignored files\")),\n \tOPT_BOOL('u', \"update\", &take_worktree_changes, N_(\"update tracked files\")),\n+\tOPT_BOOL(0, \"renormalize\", &add_renormalize, N_(\"renormalize EOL of tracked files (implies -u)\")),\n \tOPT_BOOL('N', \"intent-to-add\", &intent_to_add, N_(\"record only the fact that the path will be added later\")),\n \tOPT_BOOL('A', \"all\", &addremove_explicit, N_(\"add changes from all tracked and untracked files\")),\n \t{ OPTION_CALLBACK, 0, \"ignore-removal\", &addremove_explicit,\n@@ -406,7 +427,7 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \t\t\t chmod_arg[1] != 'x' || chmod_arg[2]))\n \t\tdie(_(\"--chmod param '%s' must be either -x or +x\"), chmod_arg);\n \n-\tadd_new_files = !take_worktree_changes && !refresh_only;\n+\tadd_new_files = !take_worktree_changes && !refresh_only && !add_renormalize;\n \trequire_pathspec = !(take_worktree_changes || (0 < addremove_explicit));\n \n \thold_locked_index(&lock_file, LOCK_DIE_ON_ERROR);\n@@ -500,7 +521,10 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \n \tplug_bulk_checkin();\n \n-\texit_status |= add_files_to_cache(prefix, &pathspec, flags);\n+\tif (add_renormalize)\n+\t\texit_status |= renormalize_tracked_files(&pathspec, flags);\n+\telse\n+\t\texit_status |= add_files_to_cache(prefix, &pathspec, flags);\n \n \tif (add_new_files)\n \t\texit_status |= add_files(&dir, flags);\ndiff --git a/cache.h b/cache.h\nindex 6440e2bf21..3e63a74113 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -686,6 +686,7 @@ extern int ie_modified(const struct index_state *, const struct cache_entry *, s\n \n #define HASH_WRITE_OBJECT 1\n #define HASH_FORMAT_CHECK 2\n+#define HASH_RENORMALIZE 4\n extern int index_fd(struct object_id *oid, int fd, struct stat *st, enum object_type type, const char *path, unsigned flags);\n extern int index_path(struct object_id *oid, const char *path, struct stat *st, unsigned flags);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 65f4fe8375..8acd3fcb93 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -631,13 +631,17 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n {\n \tint size, namelen, was_same;\n \tmode_t st_mode = st->st_mode;\n-\tstruct cache_entry *ce, *alias;\n+\tstruct cache_entry *ce, *alias = NULL;\n \tunsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY;\n \tint verbose = flags & (ADD_CACHE_VERBOSE | ADD_CACHE_PRETEND);\n \tint pretend = flags & ADD_CACHE_PRETEND;\n \tint intent_only = flags & ADD_CACHE_INTENT;\n \tint add_option = (ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE|\n \t\t\t (intent_only ? ADD_CACHE_NEW_ONLY : 0));\n+\tint newflags = HASH_WRITE_OBJECT;\n+\n+\tif (flags & HASH_RENORMALIZE)\n+\t\tnewflags |= HASH_RENORMALIZE;\n \n \tif (!S_ISREG(st_mode) && !S_ISLNK(st_mode) && !S_ISDIR(st_mode))\n \t\treturn error(\"%s: can only add regular files, symbolic links or git-directories\", path);\n@@ -678,19 +682,23 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \tif (ignore_case) {\n \t\tadjust_dirname_case(istate, ce->name);\n \t}\n+\tif (!(flags & HASH_RENORMALIZE)) {\n+\t\talias = index_file_exists(istate, ce->name,\n+\t\t\t\t\t ce_namelen(ce), ignore_case);\n+\t\tif (alias &&\n+\t\t !ce_stage(alias) &&\n+\t\t !ie_match_stat(istate, alias, st, ce_option)) {\n+\t\t\t/* Nothing changed, really */\n+\t\t\tif (!S_ISGITLINK(alias->ce_mode))\n+\t\t\t\tce_mark_uptodate(alias);\n+\t\t\talias->ce_flags |= CE_ADDED;\n \n-\talias = index_file_exists(istate, ce->name, ce_namelen(ce), ignore_case);\n-\tif (alias && !ce_stage(alias) && !ie_match_stat(istate, alias, st, ce_option)) {\n-\t\t/* Nothing changed, really */\n-\t\tif (!S_ISGITLINK(alias->ce_mode))\n-\t\t\tce_mark_uptodate(alias);\n-\t\talias->ce_flags |= CE_ADDED;\n-\n-\t\tfree(ce);\n-\t\treturn 0;\n+\t\t\tfree(ce);\n+\t\t\treturn 0;\n+\t\t}\n \t}\n \tif (!intent_only) {\n-\t\tif (index_path(&ce->oid, path, st, HASH_WRITE_OBJECT)) {\n+\t\tif (index_path(&ce->oid, path, st, newflags)) {\n \t\t\tfree(ce);\n \t\t\treturn error(\"unable to index file %s\", path);\n \t\t}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 10c3a0083d..15abb184c2 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -74,6 +74,18 @@ static struct cached_object *find_cached_object(const unsigned char *sha1)\n \treturn NULL;\n }\n \n+\n+static enum safe_crlf get_safe_crlf(unsigned flags)\n+{\n+\tif (flags & HASH_RENORMALIZE)\n+\t\treturn SAFE_CRLF_RENORMALIZE;\n+\telse if (flags & HASH_WRITE_OBJECT)\n+\t\treturn safe_crlf;\n+\telse\n+\t\treturn SAFE_CRLF_FALSE;\n+}\n+\n+\n int mkdir_in_gitdir(const char *path)\n {\n \tif (mkdir(path, 0777)) {\n@@ -1680,7 +1692,7 @@ static int index_mem(unsigned char *sha1, void *buf, size_t size,\n \tif ((type == OBJ_BLOB) && path) {\n \t\tstruct strbuf nbuf = STRBUF_INIT;\n \t\tif (convert_to_git(&the_index, path, buf, size, &nbuf,\n-\t\t\t\t write_object ? safe_crlf : SAFE_CRLF_FALSE)) {\n+\t\t\t\t get_safe_crlf(flags))) {\n \t\t\tbuf = strbuf_detach(&nbuf, &size);\n \t\t\tre_allocated = 1;\n \t\t}\n@@ -1714,7 +1726,7 @@ static int index_stream_convert_blob(unsigned char *sha1, int fd,\n \tassert(would_convert_to_git_filter_fd(path));\n \n \tconvert_to_git_filter_fd(&the_index, path, fd, &sbuf,\n-\t\t\t\t write_object ? safe_crlf : SAFE_CRLF_FALSE);\n+\t\t\t\t get_safe_crlf(flags));\n \n \tif (write_object)\n \t\tret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\ndiff --git a/t/t0025-crlf-renormalize.sh b/t/t0025-crlf-renormalize.sh\nnew file mode 100755\nindex 0000000000..fb1ed631d2\n--- /dev/null\n+++ b/t/t0025-crlf-renormalize.sh\n@@ -0,0 +1,30 @@\n+#!/bin/sh\n+\n+test_description='CRLF renormalization'\n+\n+. ./test-lib.sh\n+\n+test_expect_success setup '\n+\tgit config core.autocrlf false &&\n+\tprintf \"LINEONE\\nLINETWO\\nLINETHREE\" >LF.txt &&\n+\tprintf \"LINEONE\\r\\nLINETWO\\r\\nLINETHREE\" >CRLF.txt &&\n+\tprintf \"LINEONE\\r\\nLINETWO\\nLINETHREE\" >CRLF_mix_LF.txt &&\n+\tgit add . &&\n+\tgit commit -m initial\n+'\n+\n+test_expect_success 'renormalize CRLF in repo' '\n+\techo \"*.txt text=auto\" >.gitattributes &&\n+\tgit add --renormalize \"*.txt\" &&\n+cat >expect <actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.14.1.729.g59c0ea183a\n\n"},{"id":"331974","messageId":"xmqqvaimeixm.fsf@gitster.mtv.corp.google.com","threadId":"46891","inReplyTo":"20171030162913.23164-1-tboegi@web.de","subject":"Re: [PATCH v2 1/1] Introduce git add --renormalize .","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-07T05:50:29Z","receivedAt":"2017-11-07T05:50:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"tboegi@web.de writes:\n\n> From: Torsten Bögershausen \n>\n> Make it safer to normalize the line endings in a repository:\n> Files that had been commited with CRLF will be commited with LF.\n> ...\n> Helped-By: Junio C Hamano \n> Signed-off-by: Torsten Bögershausen \n> ---\n\nNicely explained.\n\n> @@ -175,6 +175,12 @@ for \"git add --no-all ...\", i.e. ignored removed files.\n> \twarning (e.g., if you are manually performing operations on\n> \tsubmodules).\n> \n> +--renormalize::\n> +\tNormalizes the line endings from CRLF to LF of tracked files.\n> +\tThis applies to files which are either \"text\" or \"text=auto\"\n> +\tin .gitattributes (or core.autocrlf is true or input)\n> +\t--renormalize implies -u\n> +\n\nOK.\n\n> +static int renormalize_tracked_files(const struct pathspec *pathspec, int flags)\n> +{\n> +\tint i, retval = 0;\n> +\n> +\tfor (i = 0; i < active_nr; i++) {\n> +\t\tstruct cache_entry *ce = active_cache[i];\n> +\n> +\t\tif (ce_stage(ce))\n> +\t\t\tcontinue; /* do not touch unmerged paths */\n> +\t\tif (!S_ISREG(ce->ce_mode) && !S_ISLNK(ce->ce_mode))\n> +\t\t\tcontinue; /* do not touch non blobs */\n> +\t\tif (pathspec && !ce_path_match(ce, pathspec, NULL))\n> +\t\t\tcontinue;\n> +\t\tretval |= add_file_to_cache(ce->name, flags | HASH_RENORMALIZE);\n\nWe are removing the entry and then adding the same entry under the\nsame name, and iteration over the active_cache[] from 0 through\nactive_nr should be safe, I guess.\n\n> ...\n> -\tadd_new_files = !take_worktree_changes && !refresh_only;\n> +\tadd_new_files = !take_worktree_changes && !refresh_only && !add_renormalize;\n\nIf renormalize is given, we will *not* take new files, good.\n\n> @@ -500,7 +521,10 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n> \n> \tplug_bulk_checkin();\n> \n> -\texit_status |= add_files_to_cache(prefix, &pathspec, flags);\n> +\tif (add_renormalize)\n> +\t\texit_status |= renormalize_tracked_files(&pathspec, flags);\n> +\telse\n> +\t\texit_status |= add_files_to_cache(prefix, &pathspec, flags);\n> \n> \tif (add_new_files)\n> \t\texit_status |= add_files(&dir, flags);\n\nOK.\n\n> ...\n> \tint add_option = (ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE|\n> \t\t\t (intent_only ? ADD_CACHE_NEW_ONLY : 0));\n> +\tint newflags = HASH_WRITE_OBJECT;\n> +\n> +\tif (flags & HASH_RENORMALIZE)\n> +\t\tnewflags |= HASH_RENORMALIZE;\n> ...\n> @@ -678,19 +682,23 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n> \tif (ignore_case) {\n> \t\tadjust_dirname_case(istate, ce->name);\n> \t}\n> +\tif (!(flags & HASH_RENORMALIZE)) {\n> +\t\talias = index_file_exists(istate, ce->name,\n> +\t\t\t\t\t ce_namelen(ce), ignore_case);\n> +\t\tif (alias &&\n> +\t\t !ce_stage(alias) &&\n> +\t\t !ie_match_stat(istate, alias, st, ce_option)) {\n> +\t\t\t/* Nothing changed, really */\n> +\t\t\tif (!S_ISGITLINK(alias->ce_mode))\n> +\t\t\t\tce_mark_uptodate(alias);\n> +\t\t\talias->ce_flags |= CE_ADDED;\n\nOK, so RENORMALIZE option forces the code to skip the \"does the path\nexist already? maybe we can do without adding it?\" check.\n\n> \tif (!intent_only) {\n> -\t\tif (index_path(&ce->oid, path, st, HASH_WRITE_OBJECT)) {\n> +\t\tif (index_path(&ce->oid, path, st, newflags)) {\n\nAnd then we do hash it. We later do add_index_entry() on this thing\nand we have OK_TO_REPLACE bit in the add_option, so we are good to go.\n\n> diff --git a/sha1_file.c b/sha1_file.c\n> index 10c3a0083d..15abb184c2 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -74,6 +74,18 @@ static struct cached_object *find_cached_object(const unsigned char *sha1)\n> \treturn NULL;\n> }\n> \n> +\n> +static enum safe_crlf get_safe_crlf(unsigned flags)\n> +{\n> +\tif (flags & HASH_RENORMALIZE)\n> +\t\treturn SAFE_CRLF_RENORMALIZE;\n> +\telse if (flags & HASH_WRITE_OBJECT)\n> +\t\treturn safe_crlf;\n> +\telse\n> +\t\treturn SAFE_CRLF_FALSE;\n> +}\n> +\n> +\n> int mkdir_in_gitdir(const char *path)\n> {\n> \tif (mkdir(path, 0777)) {\n> @@ -1680,7 +1692,7 @@ static int index_mem(unsigned char *sha1, void *buf, size_t size,\n> \tif ((type == OBJ_BLOB) && path) {\n> \t\tstruct strbuf nbuf = STRBUF_INIT;\n> \t\tif (convert_to_git(&the_index, path, buf, size, &nbuf,\n> -\t\t\t\t write_object ? safe_crlf : SAFE_CRLF_FALSE)) {\n> +\t\t\t\t get_safe_crlf(flags))) {\n> \t\t\tbuf = strbuf_detach(&nbuf, &size);\n> \t\t\tre_allocated = 1;\n> \t\t}\n> @@ -1714,7 +1726,7 @@ static int index_stream_convert_blob(unsigned char *sha1, int fd,\n> \tassert(would_convert_to_git_filter_fd(path));\n> \n> \tconvert_to_git_filter_fd(&the_index, path, fd, &sbuf,\n> -\t\t\t\t write_object ? safe_crlf : SAFE_CRLF_FALSE);\n> +\t\t\t\t get_safe_crlf(flags));\n> \n> \tif (write_object)\n> \t\tret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\n\nOK. We used to force CRLF_FALSE when we are not writing it out; now\nwe have three choices, and a new helper helps us isolating the logic\nto make that choice.\n\n> diff --git a/t/t0025-crlf-renormalize.sh b/t/t0025-crlf-renormalize.sh\n> new file mode 100755\n> index 0000000000..fb1ed631d2\n> --- /dev/null\n> +++ b/t/t0025-crlf-renormalize.sh\n> @@ -0,0 +1,30 @@\n> +#!/bin/sh\n> +\n> +test_description='CRLF renormalization'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success setup '\n> +\tgit config core.autocrlf false &&\n> +\tprintf \"LINEONE\\nLINETWO\\nLINETHREE\" >LF.txt &&\n> +\tprintf \"LINEONE\\r\\nLINETWO\\r\\nLINETHREE\" >CRLF.txt &&\n> +\tprintf \"LINEONE\\r\\nLINETWO\\nLINETHREE\" >CRLF_mix_LF.txt &&\n\nDid you mean to make all these files end with an incomplete line? I\nthink it does not hurt but it is misleading---the reader would try\nto see if the incomplete lines are significant and necessary part of\nthe test, which is not, and would end up wasting time, no?\n\n> +\tgit add . &&\n> +\tgit commit -m initial\n> +'\n> +\n> +test_expect_success 'renormalize CRLF in repo' '\n> +\techo \"*.txt text=auto\" >.gitattributes &&\n> +\tgit add --renormalize \"*.txt\" &&\n> +cat >expect < +i/lf w/crlf attr/text=auto CRLF.txt\n> +i/lf w/lf attr/text=auto LF.txt\n> +i/lf w/mixed attr/text=auto CRLF_mix_LF.txt\n> +EOF\n\nPerhaps use the <<-\\EOF pattern?\n\nI'd suggest squashing this in (or I can do so myself if there is no\nother change needed).\n\nThanks. Looks mostly good.\n\n t/t0025-crlf-renormalize.sh | 16 ++++++++--------\n 1 file changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/t/t0025-crlf-renormalize.sh b/t/t0025-crlf-renormalize.sh\nindex fb1ed631d2..ea4f82ea27 100755\n--- a/t/t0025-crlf-renormalize.sh\n+++ b/t/t0025-crlf-renormalize.sh\n@@ -6,9 +6,9 @@ test_description='CRLF renormalization'\n \n test_expect_success setup '\n \tgit config core.autocrlf false &&\n-\tprintf \"LINEONE\\nLINETWO\\nLINETHREE\" >LF.txt &&\n-\tprintf \"LINEONE\\r\\nLINETWO\\r\\nLINETHREE\" >CRLF.txt &&\n-\tprintf \"LINEONE\\r\\nLINETWO\\nLINETHREE\" >CRLF_mix_LF.txt &&\n+\tprintf \"LINEONE\\nLINETWO\\nLINETHREE\\n\" >LF.txt &&\n+\tprintf \"LINEONE\\r\\nLINETWO\\r\\nLINETHREE\\r\\n\" >CRLF.txt &&\n+\tprintf \"LINEONE\\r\\nLINETWO\\nLINETHREE\\n\" >CRLF_mix_LF.txt &&\n \tgit add . &&\n \tgit commit -m initial\n '\n@@ -16,11 +16,11 @@ test_expect_success setup '\n test_expect_success 'renormalize CRLF in repo' '\n \techo \"*.txt text=auto\" >.gitattributes &&\n \tgit add --renormalize \"*.txt\" &&\n-cat >expect <expect <<-\\EOF &&\n+\ti/lf w/crlf attr/text=auto CRLF.txt\n+\ti/lf w/lf attr/text=auto LF.txt\n+\ti/lf w/mixed attr/text=auto CRLF_mix_LF.txt\n+\tEOF\n \tgit ls-files --eol |\n \tsed -e \"s/\t/ /g\" -e \"s/ */ /g\" |\n \tsort >actual &&\n"},{"id":"332017","messageId":"20171107172659.GA11119@tor.lan","threadId":"46891","inReplyTo":"xmqqvaimeixm.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 1/1] Introduce git add --renormalize .","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-11-07T17:26:59Z","receivedAt":"2017-11-07T17:27:11Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"[snip]\n> \n> > @@ -175,6 +175,12 @@ for \"git add --no-all ...\", i.e. ignored removed files.\n> > \twarning (e.g., if you are manually performing operations on\n> > \tsubmodules).\n> > \n> > +--renormalize::\n> > +\tNormalizes the line endings from CRLF to LF of tracked files.\n> > +\tThis applies to files which are either \"text\" or \"text=auto\"\n> > +\tin .gitattributes (or core.autocrlf is true or input)\n> > +\t--renormalize implies -u\n> > +\n> \n> OK.\n\nI think the fact, that clean filters are re-run, and re-evaluated\nin case they are changed, should be made more clear here.\nI don't know how to explain it better that CRLF conversion and/or filters are\nre-applied, this is an attempt:\n\n\n--renormalize::\n\tNormalizes the line endings from CRLF to LF of tracked files,\n\tif the .gitattributes or core.autocrlf say so.\n\tAdditionally the clean and ident filters, if any, are re-run.\n\t--renormalize implies -u\n\n\n\n\n\n> \n> > +static int renormalize_tracked_files(const struct pathspec *pathspec, int flags)\n> > +{\n> > +\tint i, retval = 0;\n> > +\n> > +\tfor (i = 0; i < active_nr; i++) {\n> > +\t\tstruct cache_entry *ce = active_cache[i];\n> > +\n> > +\t\tif (ce_stage(ce))\n> > +\t\t\tcontinue; /* do not touch unmerged paths */\n> > +\t\tif (!S_ISREG(ce->ce_mode) && !S_ISLNK(ce->ce_mode))\n> > +\t\t\tcontinue; /* do not touch non blobs */\n> > +\t\tif (pathspec && !ce_path_match(ce, pathspec, NULL))\n> > +\t\t\tcontinue;\n> > +\t\tretval |= add_file_to_cache(ce->name, flags | HASH_RENORMALIZE);\n> \n> We are removing the entry and then adding the same entry under the\n> same name, and iteration over the active_cache[] from 0 through\n> active_nr should be safe, I guess.\n> \n> > ...\n> > -\tadd_new_files = !take_worktree_changes && !refresh_only;\n> > +\tadd_new_files = !take_worktree_changes && !refresh_only && !add_renormalize;\n> \n> If renormalize is given, we will *not* take new files, good.\n> \n> > @@ -500,7 +521,10 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n> > \n> > \tplug_bulk_checkin();\n> > \n> > -\texit_status |= add_files_to_cache(prefix, &pathspec, flags);\n> > +\tif (add_renormalize)\n> > +\t\texit_status |= renormalize_tracked_files(&pathspec, flags);\n> > +\telse\n> > +\t\texit_status |= add_files_to_cache(prefix, &pathspec, flags);\n> > \n> > \tif (add_new_files)\n> > \t\texit_status |= add_files(&dir, flags);\n> \n> OK.\n> \n> > ...\n> > \tint add_option = (ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE|\n> > \t\t\t (intent_only ? ADD_CACHE_NEW_ONLY : 0));\n> > +\tint newflags = HASH_WRITE_OBJECT;\n> > +\n> > +\tif (flags & HASH_RENORMALIZE)\n> > +\t\tnewflags |= HASH_RENORMALIZE;\n> > ...\n> > @@ -678,19 +682,23 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n> > \tif (ignore_case) {\n> > \t\tadjust_dirname_case(istate, ce->name);\n> > \t}\n> > +\tif (!(flags & HASH_RENORMALIZE)) {\n> > +\t\talias = index_file_exists(istate, ce->name,\n> > +\t\t\t\t\t ce_namelen(ce), ignore_case);\n> > +\t\tif (alias &&\n> > +\t\t !ce_stage(alias) &&\n> > +\t\t !ie_match_stat(istate, alias, st, ce_option)) {\n> > +\t\t\t/* Nothing changed, really */\n> > +\t\t\tif (!S_ISGITLINK(alias->ce_mode))\n> > +\t\t\t\tce_mark_uptodate(alias);\n> > +\t\t\talias->ce_flags |= CE_ADDED;\n> \n> OK, so RENORMALIZE option forces the code to skip the \"does the path\n> exist already? maybe we can do without adding it?\" check.\n> \n> > \tif (!intent_only) {\n> > -\t\tif (index_path(&ce->oid, path, st, HASH_WRITE_OBJECT)) {\n> > +\t\tif (index_path(&ce->oid, path, st, newflags)) {\n> \n> And then we do hash it. We later do add_index_entry() on this thing\n> and we have OK_TO_REPLACE bit in the add_option, so we are good to go.\n> \n> > diff --git a/sha1_file.c b/sha1_file.c\n> > index 10c3a0083d..15abb184c2 100644\n> > --- a/sha1_file.c\n> > +++ b/sha1_file.c\n> > @@ -74,6 +74,18 @@ static struct cached_object *find_cached_object(const unsigned char *sha1)\n> > \treturn NULL;\n> > }\n> > \n> > +\n> > +static enum safe_crlf get_safe_crlf(unsigned flags)\n> > +{\n> > +\tif (flags & HASH_RENORMALIZE)\n> > +\t\treturn SAFE_CRLF_RENORMALIZE;\n> > +\telse if (flags & HASH_WRITE_OBJECT)\n> > +\t\treturn safe_crlf;\n> > +\telse\n> > +\t\treturn SAFE_CRLF_FALSE;\n> > +}\n> > +\n> > +\n> > int mkdir_in_gitdir(const char *path)\n> > {\n> > \tif (mkdir(path, 0777)) {\n> > @@ -1680,7 +1692,7 @@ static int index_mem(unsigned char *sha1, void *buf, size_t size,\n> > \tif ((type == OBJ_BLOB) && path) {\n> > \t\tstruct strbuf nbuf = STRBUF_INIT;\n> > \t\tif (convert_to_git(&the_index, path, buf, size, &nbuf,\n> > -\t\t\t\t write_object ? safe_crlf : SAFE_CRLF_FALSE)) {\n> > +\t\t\t\t get_safe_crlf(flags))) {\n> > \t\t\tbuf = strbuf_detach(&nbuf, &size);\n> > \t\t\tre_allocated = 1;\n> > \t\t}\n> > @@ -1714,7 +1726,7 @@ static int index_stream_convert_blob(unsigned char *sha1, int fd,\n> > \tassert(would_convert_to_git_filter_fd(path));\n> > \n> > \tconvert_to_git_filter_fd(&the_index, path, fd, &sbuf,\n> > -\t\t\t\t write_object ? safe_crlf : SAFE_CRLF_FALSE);\n> > +\t\t\t\t get_safe_crlf(flags));\n> > \n> > \tif (write_object)\n> > \t\tret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\n> \n> OK. We used to force CRLF_FALSE when we are not writing it out; now\n> we have three choices, and a new helper helps us isolating the logic\n> to make that choice.\n> \n> > diff --git a/t/t0025-crlf-renormalize.sh b/t/t0025-crlf-renormalize.sh\n> > new file mode 100755\n> > index 0000000000..fb1ed631d2\n> > --- /dev/null\n> > +++ b/t/t0025-crlf-renormalize.sh\n> > @@ -0,0 +1,30 @@\n> > +#!/bin/sh\n> > +\n> > +test_description='CRLF renormalization'\n> > +\n> > +. ./test-lib.sh\n> > +\n> > +test_expect_success setup '\n> > +\tgit config core.autocrlf false &&\n> > +\tprintf \"LINEONE\\nLINETWO\\nLINETHREE\" >LF.txt &&\n> > +\tprintf \"LINEONE\\r\\nLINETWO\\r\\nLINETHREE\" >CRLF.txt &&\n> > +\tprintf \"LINEONE\\r\\nLINETWO\\nLINETHREE\" >CRLF_mix_LF.txt &&\n> \n> Did you mean to make all these files end with an incomplete line? I\n> think it does not hurt but it is misleading---the reader would try\n> to see if the incomplete lines are significant and necessary part of\n> the test, which is not, and would end up wasting time, no?\n> \n> > +\tgit add . &&\n> > +\tgit commit -m initial\n> > +'\n> > +\n> > +test_expect_success 'renormalize CRLF in repo' '\n> > +\techo \"*.txt text=auto\" >.gitattributes &&\n> > +\tgit add --renormalize \"*.txt\" &&\n> > +cat >expect < > +i/lf w/crlf attr/text=auto CRLF.txt\n> > +i/lf w/lf attr/text=auto LF.txt\n> > +i/lf w/mixed attr/text=auto CRLF_mix_LF.txt\n> > +EOF\n> \n> Perhaps use the <<-\\EOF pattern?\n> \n> I'd suggest squashing this in (or I can do so myself if there is no\n> other change needed).\n> \n> Thanks. Looks mostly good.\n\nThanks for review and proposed fixes and help.\n\n[snip the TC. Adding line endings is good)\n"},{"id":"332046","messageId":"xmqqlgjhobb4.fsf@gitster.mtv.corp.google.com","threadId":"46891","inReplyTo":"20171107172659.GA11119@tor.lan","subject":"Re: [PATCH v2 1/1] Introduce git add --renormalize .","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-08T00:37:19Z","receivedAt":"2017-11-08T00:37:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen writes:\n\n>> > +--renormalize::\n>> > +\tNormalizes the line endings from CRLF to LF of tracked files.\n>> > +\tThis applies to files which are either \"text\" or \"text=auto\"\n>> > +\tin .gitattributes (or core.autocrlf is true or input)\n>> > +\t--renormalize implies -u\n>> > +\n>> \n>> OK.\n>\n> I think the fact, that clean filters are re-run, and re-evaluated\n> in case they are changed, should be made more clear here.\n> I don't know how to explain it better that CRLF conversion and/or filters are\n> re-applied, this is an attempt:\n>\n>\n> --renormalize::\n> \tNormalizes the line endings from CRLF to LF of tracked files,\n> \tif the .gitattributes or core.autocrlf say so.\n> \tAdditionally the clean and ident filters, if any, are re-run.\n> \t--renormalize implies -u\n\nThat is certainly better. Do we have an end-user facing phrase to\ncollectively call everything the \"convert_to_git()\" processing does?\nWhen I talk casually about it, I'd call it the \"clean\" process (as\nopposed to the \"smudge\" process) as a term that includes all the\nthings that Git does to massage contents in the working tree to\nin-repository representation.\n\nIf we had such a term in Documentation/glossary-contents.txt, we\ncould even say\n\n\tAdd contents of all paths to the index by freshly applying\n\tthe \"clean\" process, even to the ones Git may think are\n\tunmodified in the working tree since they were added the\n\tlast time (based on the file timestamps etc.). This is\n\toften useful after updating settings like `core.autocrlf` in\n\tthe `.git/config` file and the `text` attributes in the\n\t`.gitattributes` file to correct the index entries that\n\trecords lines with CRLF to use LF instead, or changing what\n\tthe `clean` filter does. This option implies `-u`.\n\nThe point is to express that the CRLF/LF is a consequence (even\nthough it may be the most prominent one from end-users' point of\nview) of a larger processing.\n\n> [snip the TC. Adding line endings is good)\n\nWhat is TC in this context?\n"},{"id":"332126","messageId":"20171109184700.GA27635@tor.lan","threadId":"46891","inReplyTo":"xmqqlgjhobb4.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 1/1] Introduce git add --renormalize .","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-11-09T18:47:01Z","receivedAt":"2017-11-09T18:47:14Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"[]\n> \n> If we had such a term in Documentation/glossary-contents.txt, we\n> could even say\n> \n> \tAdd contents of all paths to the index by freshly applying\n> \tthe \"clean\" process, even to the ones Git may think are\n> \tunmodified in the working tree since they were added the\n> \tlast time (based on the file timestamps etc.). This is\n> \toften useful after updating settings like `core.autocrlf` in\n> \tthe `.git/config` file and the `text` attributes in the\n> \t`.gitattributes` file to correct the index entries that\n> \trecords lines with CRLF to use LF instead, or changing what\n> \tthe `clean` filter does. This option implies `-u`.\n> \n> The point is to express that the CRLF/LF is a consequence (even\n> though it may be the most prominent one from end-users' point of\n> view) of a larger processing.\n\nHere is a somwhat shorter description:\n\nApply the \"clean\" process freshly to all tracked files.\nThis is useful after changing `core.autocrlf` or the `text`\nattributes in the `.gitattributes` file because\nGit may not consider these files as changed.\nCorrect the files that had been commited with CRLF,\nthey will from now on have LF instead.\nRe-run what the `clean` filter does.\nThis option implies `-u`.\n\n\n> \n> > [snip the TC. Adding line endings is good)\n> \n> What is TC in this context?\n\nSorry for confusion: TC means test case.\n"},{"id":"332136","messageId":"xmqq1sl7j83x.fsf@gitster.mtv.corp.google.com","threadId":"46891","inReplyTo":"20171109184700.GA27635@tor.lan","subject":"Re: [PATCH v2 1/1] Introduce git add --renormalize .","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-10T00:22:10Z","receivedAt":"2017-11-10T00:22:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen writes:\n\n> Here is a somwhat shorter description:\n>\n> Apply the \"clean\" process freshly to all tracked files.\n> This is useful after changing `core.autocrlf` or the `text`\n> attributes in the `.gitattributes` file because\n> Git may not consider these files as changed.\n\nI think it is OK to omit .git/config for brevity (I am assuming that\nyour justification is because you thought it was obvious it is a\nconfiguration variable); but then it is equally obvious (if not\nmore) that `text` attribute comes from .gitattributes (notice we do\nnot mention core.autocrlf is a configuration variable in the above,\nbut we do say `text` is an attribute) so it can also be omitted for\nbrevity.\n\n> Correct the files that had been commited with CRLF,\n> they will from now on have LF instead.\n\nReading this as a single sentence immediately after the above\nparagraph leaves me feel confused. First of all, this would not\nhappen unless the user corrects core.autocrlf/text like described\nabove. In fact, updating these settings is done as in order to do\nthat correction. So I'd say it should not be split.\n\n> Re-run what the `clean` filter does.\n\nThis again looks out of place just like the previous sentence. In\nfact, provided if \"the clean process\" is understood by the end user,\nthis is redundant.\n\n> This option implies `-u`.\n\nTaking these altogether, perhaps\n\n Apply the \"clean\" process freshly to all tracked files to\n forcibly add them again to the index. This is useful after\n changing `core.autocrlf` configuration or the `text` attribute\n in order to correct files added with wrong CRLF/LF line endings.\n This option implies `-u`.\n\nThanks.\n"},{"id":"332359","messageId":"20171112200826.GA2677@tor.lan","threadId":"46891","inReplyTo":"xmqq1sl7j83x.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 1/1] Introduce git add --renormalize .","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-11-12T20:08:26Z","receivedAt":"2017-11-12T20:08:37Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Fri, Nov 10, 2017 at 09:22:10AM +0900, Junio C Hamano wrote:\n\n> \n> Taking these altogether, perhaps\n> \n> Apply the \"clean\" process freshly to all tracked files to\n> forcibly add them again to the index. This is useful after\n> changing `core.autocrlf` configuration or the `text` attribute\n> in order to correct files added with wrong CRLF/LF line endings.\n> This option implies `-u`.\n> \n> Thanks.\n\nOK with the text - I can send a new version the next days.\n"},{"id":"332671","messageId":"20171116163828.14937-1-tboegi@web.de","threadId":"46891","inReplyTo":"xmqqshejq9mn.fsf@gitster.mtv.corp.google.com","subject":"[PATCH v3 1/1] Introduce git add --renormalize .","fromName":"","fromEmail":"tboegi@web.de","sentAt":"2017-11-16T16:38:28Z","receivedAt":"2017-11-16T16:38:38Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"From: Torsten Bögershausen \n\nMake it safer to normalize the line endings in a repository:\nFiles that had been commited with CRLF will be commited with LF.\n\nThe old way to normalize a repo was like this:\n # Make sure that there are not untracked files\n $ echo \"* text=auto\" >.gitattributes\n $ git read-tree --empty\n $ git add .\n $ git commit -m \"Introduce end-of-line normalization\"\n\nThe user must make sure that there are no untracked files,\notherwise they would have been added and tracked from now on.\n\nThe new \"add ..renormalize\" does not add untracked files:\n $ echo \"* text=auto\" >.gitattributes\n $ git add --renormalize .\n $ git commit -m \"Introduce end-of-line normalization\"\n\nNote that \"git add --renormalize \" is the short form for\n\"git add -u --renormalize \".\n\nWhile add it, document that the same renormalization may be needed,\nwhenever a clean filter is added or changed.\n\nHelped-By: Junio C Hamano \nSigned-off-by: Torsten Bögershausen \n---\n\nChanges since V2:\n Add line endings in t0025\n Use the <<-\\EOF pattern\n Improve the documentation for \"git add --renormalize\"\n \n\nDocumentation/git-add.txt | 9 ++++++++-\n Documentation/gitattributes.txt | 6 ++++--\n builtin/add.c | 28 ++++++++++++++++++++++++++--\n cache.h | 1 +\n read-cache.c | 30 +++++++++++++++++++-----------\n sha1_file.c | 16 ++++++++++++++--\n t/t0025-crlf-renormalize.sh | 30 ++++++++++++++++++++++++++++++\n 7 files changed, 102 insertions(+), 18 deletions(-)\n create mode 100755 t/t0025-crlf-renormalize.sh\n\ndiff --git a/Documentation/git-add.txt b/Documentation/git-add.txt\nindex b700beaff5..d50fa339dc 100644\n--- a/Documentation/git-add.txt\n+++ b/Documentation/git-add.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n [verse]\n 'git add' [--verbose | -v] [--dry-run | -n] [--force | -f] [--interactive | -i] [--patch | -p]\n \t [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]]\n-\t [--intent-to-add | -N] [--refresh] [--ignore-errors] [--ignore-missing]\n+\t [--intent-to-add | -N] [--refresh] [--ignore-errors] [--ignore-missing] [--renormalize]\n \t [--chmod=(+|-)x] [--] [...]\n \n DESCRIPTION\n@@ -175,6 +175,13 @@ for \"git add --no-all ...\", i.e. ignored removed files.\n \twarning (e.g., if you are manually performing operations on\n \tsubmodules).\n \n+--renormalize::\n+\tApply the \"clean\" process freshly to all tracked files to\n+\tforcibly add them again to the index. This is useful after\n+\tchanging `core.autocrlf` configuration or the `text` attribute\n+\tin order to correct files added with wrong CRLF/LF line endings.\n+\tThis option implies `-u`.\n+\n --chmod=(+|-)x::\n \tOverride the executable bit of the added files. The executable\n \tbit is only changed in the index, the files on disk are left\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex 4c68bc19d5..30687de81a 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -232,8 +232,7 @@ From a clean working directory:\n \n -------------------------------------------------\n $ echo \"* text=auto\" >.gitattributes\n-$ git read-tree --empty # Clean index, force re-scan of working directory\n-$ git add .\n+$ git add --renormalize .\n $ git status # Show files that will be normalized\n $ git commit -m \"Introduce end-of-line normalization\"\n -------------------------------------------------\n@@ -328,6 +327,9 @@ You can declare that a filter turns a content that by itself is unusable\n into a usable content by setting the filter..required configuration\n variable to `true`.\n \n+Note: Whenever the clean filter is changed, the repo should be renormalized:\n+$ git add --renormalize .\n+\n For example, in .gitattributes, you would assign the `filter`\n attribute for paths.\n \ndiff --git a/builtin/add.c b/builtin/add.c\nindex a648cf4c56..c42b50f857 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -26,6 +26,7 @@ static const char * const builtin_add_usage[] = {\n };\n static int patch_interactive, add_interactive, edit_interactive;\n static int take_worktree_changes;\n+static int add_renormalize;\n \n struct update_callback_data {\n \tint flags;\n@@ -123,6 +124,25 @@ int add_files_to_cache(const char *prefix,\n \treturn !!data.add_errors;\n }\n \n+static int renormalize_tracked_files(const struct pathspec *pathspec, int flags)\n+{\n+\tint i, retval = 0;\n+\n+\tfor (i = 0; i < active_nr; i++) {\n+\t\tstruct cache_entry *ce = active_cache[i];\n+\n+\t\tif (ce_stage(ce))\n+\t\t\tcontinue; /* do not touch unmerged paths */\n+\t\tif (!S_ISREG(ce->ce_mode) && !S_ISLNK(ce->ce_mode))\n+\t\t\tcontinue; /* do not touch non blobs */\n+\t\tif (pathspec && !ce_path_match(ce, pathspec, NULL))\n+\t\t\tcontinue;\n+\t\tretval |= add_file_to_cache(ce->name, flags | HASH_RENORMALIZE);\n+\t}\n+\n+\treturn retval;\n+}\n+\n static char *prune_directory(struct dir_struct *dir, struct pathspec *pathspec, int prefix)\n {\n \tchar *seen;\n@@ -276,6 +296,7 @@ static struct option builtin_add_options[] = {\n \tOPT_BOOL('e', \"edit\", &edit_interactive, N_(\"edit current diff and apply\")),\n \tOPT__FORCE(&ignored_too, N_(\"allow adding otherwise ignored files\")),\n \tOPT_BOOL('u', \"update\", &take_worktree_changes, N_(\"update tracked files\")),\n+\tOPT_BOOL(0, \"renormalize\", &add_renormalize, N_(\"renormalize EOL of tracked files (implies -u)\")),\n \tOPT_BOOL('N', \"intent-to-add\", &intent_to_add, N_(\"record only the fact that the path will be added later\")),\n \tOPT_BOOL('A', \"all\", &addremove_explicit, N_(\"add changes from all tracked and untracked files\")),\n \t{ OPTION_CALLBACK, 0, \"ignore-removal\", &addremove_explicit,\n@@ -406,7 +427,7 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \t\t\t chmod_arg[1] != 'x' || chmod_arg[2]))\n \t\tdie(_(\"--chmod param '%s' must be either -x or +x\"), chmod_arg);\n \n-\tadd_new_files = !take_worktree_changes && !refresh_only;\n+\tadd_new_files = !take_worktree_changes && !refresh_only && !add_renormalize;\n \trequire_pathspec = !(take_worktree_changes || (0 < addremove_explicit));\n \n \thold_locked_index(&lock_file, LOCK_DIE_ON_ERROR);\n@@ -500,7 +521,10 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \n \tplug_bulk_checkin();\n \n-\texit_status |= add_files_to_cache(prefix, &pathspec, flags);\n+\tif (add_renormalize)\n+\t\texit_status |= renormalize_tracked_files(&pathspec, flags);\n+\telse\n+\t\texit_status |= add_files_to_cache(prefix, &pathspec, flags);\n \n \tif (add_new_files)\n \t\texit_status |= add_files(&dir, flags);\ndiff --git a/cache.h b/cache.h\nindex 6440e2bf21..3e63a74113 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -686,6 +686,7 @@ extern int ie_modified(const struct index_state *, const struct cache_entry *, s\n \n #define HASH_WRITE_OBJECT 1\n #define HASH_FORMAT_CHECK 2\n+#define HASH_RENORMALIZE 4\n extern int index_fd(struct object_id *oid, int fd, struct stat *st, enum object_type type, const char *path, unsigned flags);\n extern int index_path(struct object_id *oid, const char *path, struct stat *st, unsigned flags);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 65f4fe8375..8acd3fcb93 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -631,13 +631,17 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n {\n \tint size, namelen, was_same;\n \tmode_t st_mode = st->st_mode;\n-\tstruct cache_entry *ce, *alias;\n+\tstruct cache_entry *ce, *alias = NULL;\n \tunsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY;\n \tint verbose = flags & (ADD_CACHE_VERBOSE | ADD_CACHE_PRETEND);\n \tint pretend = flags & ADD_CACHE_PRETEND;\n \tint intent_only = flags & ADD_CACHE_INTENT;\n \tint add_option = (ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE|\n \t\t\t (intent_only ? ADD_CACHE_NEW_ONLY : 0));\n+\tint newflags = HASH_WRITE_OBJECT;\n+\n+\tif (flags & HASH_RENORMALIZE)\n+\t\tnewflags |= HASH_RENORMALIZE;\n \n \tif (!S_ISREG(st_mode) && !S_ISLNK(st_mode) && !S_ISDIR(st_mode))\n \t\treturn error(\"%s: can only add regular files, symbolic links or git-directories\", path);\n@@ -678,19 +682,23 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \tif (ignore_case) {\n \t\tadjust_dirname_case(istate, ce->name);\n \t}\n+\tif (!(flags & HASH_RENORMALIZE)) {\n+\t\talias = index_file_exists(istate, ce->name,\n+\t\t\t\t\t ce_namelen(ce), ignore_case);\n+\t\tif (alias &&\n+\t\t !ce_stage(alias) &&\n+\t\t !ie_match_stat(istate, alias, st, ce_option)) {\n+\t\t\t/* Nothing changed, really */\n+\t\t\tif (!S_ISGITLINK(alias->ce_mode))\n+\t\t\t\tce_mark_uptodate(alias);\n+\t\t\talias->ce_flags |= CE_ADDED;\n \n-\talias = index_file_exists(istate, ce->name, ce_namelen(ce), ignore_case);\n-\tif (alias && !ce_stage(alias) && !ie_match_stat(istate, alias, st, ce_option)) {\n-\t\t/* Nothing changed, really */\n-\t\tif (!S_ISGITLINK(alias->ce_mode))\n-\t\t\tce_mark_uptodate(alias);\n-\t\talias->ce_flags |= CE_ADDED;\n-\n-\t\tfree(ce);\n-\t\treturn 0;\n+\t\t\tfree(ce);\n+\t\t\treturn 0;\n+\t\t}\n \t}\n \tif (!intent_only) {\n-\t\tif (index_path(&ce->oid, path, st, HASH_WRITE_OBJECT)) {\n+\t\tif (index_path(&ce->oid, path, st, newflags)) {\n \t\t\tfree(ce);\n \t\t\treturn error(\"unable to index file %s\", path);\n \t\t}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 10c3a0083d..15abb184c2 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -74,6 +74,18 @@ static struct cached_object *find_cached_object(const unsigned char *sha1)\n \treturn NULL;\n }\n \n+\n+static enum safe_crlf get_safe_crlf(unsigned flags)\n+{\n+\tif (flags & HASH_RENORMALIZE)\n+\t\treturn SAFE_CRLF_RENORMALIZE;\n+\telse if (flags & HASH_WRITE_OBJECT)\n+\t\treturn safe_crlf;\n+\telse\n+\t\treturn SAFE_CRLF_FALSE;\n+}\n+\n+\n int mkdir_in_gitdir(const char *path)\n {\n \tif (mkdir(path, 0777)) {\n@@ -1680,7 +1692,7 @@ static int index_mem(unsigned char *sha1, void *buf, size_t size,\n \tif ((type == OBJ_BLOB) && path) {\n \t\tstruct strbuf nbuf = STRBUF_INIT;\n \t\tif (convert_to_git(&the_index, path, buf, size, &nbuf,\n-\t\t\t\t write_object ? safe_crlf : SAFE_CRLF_FALSE)) {\n+\t\t\t\t get_safe_crlf(flags))) {\n \t\t\tbuf = strbuf_detach(&nbuf, &size);\n \t\t\tre_allocated = 1;\n \t\t}\n@@ -1714,7 +1726,7 @@ static int index_stream_convert_blob(unsigned char *sha1, int fd,\n \tassert(would_convert_to_git_filter_fd(path));\n \n \tconvert_to_git_filter_fd(&the_index, path, fd, &sbuf,\n-\t\t\t\t write_object ? safe_crlf : SAFE_CRLF_FALSE);\n+\t\t\t\t get_safe_crlf(flags));\n \n \tif (write_object)\n \t\tret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\ndiff --git a/t/t0025-crlf-renormalize.sh b/t/t0025-crlf-renormalize.sh\nnew file mode 100755\nindex 0000000000..50d8f3a0f3\n--- /dev/null\n+++ b/t/t0025-crlf-renormalize.sh\n@@ -0,0 +1,30 @@\n+#!/bin/sh\n+\n+test_description='CRLF renormalization'\n+\n+. ./test-lib.sh\n+\n+test_expect_success setup '\n+\tgit config core.autocrlf false &&\n+\tprintf \"LINEONE\\nLINETWO\\nLINETHREE\\n\" >LF.txt &&\n+\tprintf \"LINEONE\\r\\nLINETWO\\r\\nLINETHREE\\r\\n\" >CRLF.txt &&\n+\tprintf \"LINEONE\\r\\nLINETWO\\nLINETHREE\\n\" >CRLF_mix_LF.txt &&\n+\tgit add . &&\n+\tgit commit -m initial\n+'\n+\n+test_expect_success 'renormalize CRLF in repo' '\n+\techo \"*.txt text=auto\" >.gitattributes &&\n+\tgit add --renormalize \"*.txt\" &&\n+cat >expect <<-\\EOF &&\n+\ti/lf w/crlf attr/text=auto CRLF.txt\n+\ti/lf w/lf attr/text=auto LF.txt\n+\ti/lf w/mixed attr/text=auto CRLF_mix_LF.txt\n+EOF\n+\tgit ls-files --eol |\n+\tsed -e \"s/\t/ /g\" -e \"s/ */ /g\" |\n+\tsort >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.15.0\n\n"},{"id":"332733","messageId":"xmqqtvxtu26y.fsf@gitster.mtv.corp.google.com","threadId":"46891","inReplyTo":"20171116163828.14937-1-tboegi@web.de","subject":"Re: [PATCH v3 1/1] Introduce git add --renormalize .","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-17T01:24:53Z","receivedAt":"2017-11-17T01:25:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"tboegi@web.de writes:\n\nI'll retitle this to\n\n\tSubject: add: introduce \"--renormalize\"\n\nand will queue with s/$old/$new/ that you'll see below.\n\n> From: Torsten Bögershausen \n>\n> Make it safer to normalize the line endings in a repository:\n\ns/:$/./;\n\n> Files that had been commited with CRLF will be commited with LF.\n>\n> The old way to normalize a repo was like this:\n> # Make sure that there are not untracked files\n> $ echo \"* text=auto\" >.gitattributes\n> $ git read-tree --empty\n> $ git add .\n> $ git commit -m \"Introduce end-of-line normalization\"\n>\n> The user must make sure that there are no untracked files,\n> otherwise they would have been added and tracked from now on.\n>\n> The new \"add ..renormalize\" does not add untracked files:\n\ns/\\.\\./--/;\n\n> $ echo \"* text=auto\" >.gitattributes\n> $ git add --renormalize .\n> $ git commit -m \"Introduce end-of-line normalization\"\n>\n> Note that \"git add --renormalize \" is the short form for\n> \"git add -u --renormalize \".\n>\n> While add it, document that the same renormalization may be needed,\n\ns/add it/at it/;\n\n> whenever a clean filter is added or changed.\n>\n> Helped-By: Junio C Hamano \n> Signed-off-by: Torsten Bögershausen \n> ---\n\nThanks.\n\n>\n> Changes since V2:\n> Add line endings in t0025\n> Use the <<-\\EOF pattern\n> Improve the documentation for \"git add --renormalize\"\n> \n>\n> Documentation/git-add.txt | 9 ++++++++-\n> Documentation/gitattributes.txt | 6 ++++--\n> builtin/add.c | 28 ++++++++++++++++++++++++++--\n> cache.h | 1 +\n> read-cache.c | 30 +++++++++++++++++++-----------\n> sha1_file.c | 16 ++++++++++++++--\n> t/t0025-crlf-renormalize.sh | 30 ++++++++++++++++++++++++++++++\n> 7 files changed, 102 insertions(+), 18 deletions(-)\n> create mode 100755 t/t0025-crlf-renormalize.sh\n>\n> diff --git a/Documentation/git-add.txt b/Documentation/git-add.txt\n> index b700beaff5..d50fa339dc 100644\n> --- a/Documentation/git-add.txt\n> +++ b/Documentation/git-add.txt\n> @@ -10,7 +10,7 @@ SYNOPSIS\n> [verse]\n> 'git add' [--verbose | -v] [--dry-run | -n] [--force | -f] [--interactive | -i] [--patch | -p]\n> \t [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]]\n> -\t [--intent-to-add | -N] [--refresh] [--ignore-errors] [--ignore-missing]\n> +\t [--intent-to-add | -N] [--refresh] [--ignore-errors] [--ignore-missing] [--renormalize]\n> \t [--chmod=(+|-)x] [--] [...]\n> \n> DESCRIPTION\n> @@ -175,6 +175,13 @@ for \"git add --no-all ...\", i.e. ignored removed files.\n> \twarning (e.g., if you are manually performing operations on\n> \tsubmodules).\n> \n> +--renormalize::\n> +\tApply the \"clean\" process freshly to all tracked files to\n> +\tforcibly add them again to the index. This is useful after\n> +\tchanging `core.autocrlf` configuration or the `text` attribute\n> +\tin order to correct files added with wrong CRLF/LF line endings.\n> +\tThis option implies `-u`.\n> +\n> --chmod=(+|-)x::\n> \tOverride the executable bit of the added files. The executable\n> \tbit is only changed in the index, the files on disk are left\n> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n> index 4c68bc19d5..30687de81a 100644\n> --- a/Documentation/gitattributes.txt\n> +++ b/Documentation/gitattributes.txt\n> @@ -232,8 +232,7 @@ From a clean working directory:\n> \n> -------------------------------------------------\n> $ echo \"* text=auto\" >.gitattributes\n> -$ git read-tree --empty # Clean index, force re-scan of working directory\n> -$ git add .\n> +$ git add --renormalize .\n> $ git status # Show files that will be normalized\n> $ git commit -m \"Introduce end-of-line normalization\"\n> -------------------------------------------------\n> @@ -328,6 +327,9 @@ You can declare that a filter turns a content that by itself is unusable\n> into a usable content by setting the filter..required configuration\n> variable to `true`.\n> \n> +Note: Whenever the clean filter is changed, the repo should be renormalized:\n> +$ git add --renormalize .\n> +\n> For example, in .gitattributes, you would assign the `filter`\n> attribute for paths.\n> \n> diff --git a/builtin/add.c b/builtin/add.c\n> index a648cf4c56..c42b50f857 100644\n> --- a/builtin/add.c\n> +++ b/builtin/add.c\n> @@ -26,6 +26,7 @@ static const char * const builtin_add_usage[] = {\n> };\n> static int patch_interactive, add_interactive, edit_interactive;\n> static int take_worktree_changes;\n> +static int add_renormalize;\n> \n> struct update_callback_data {\n> \tint flags;\n> @@ -123,6 +124,25 @@ int add_files_to_cache(const char *prefix,\n> \treturn !!data.add_errors;\n> }\n> \n> +static int renormalize_tracked_files(const struct pathspec *pathspec, int flags)\n> +{\n> +\tint i, retval = 0;\n> +\n> +\tfor (i = 0; i < active_nr; i++) {\n> +\t\tstruct cache_entry *ce = active_cache[i];\n> +\n> +\t\tif (ce_stage(ce))\n> +\t\t\tcontinue; /* do not touch unmerged paths */\n> +\t\tif (!S_ISREG(ce->ce_mode) && !S_ISLNK(ce->ce_mode))\n> +\t\t\tcontinue; /* do not touch non blobs */\n> +\t\tif (pathspec && !ce_path_match(ce, pathspec, NULL))\n> +\t\t\tcontinue;\n> +\t\tretval |= add_file_to_cache(ce->name, flags | HASH_RENORMALIZE);\n> +\t}\n> +\n> +\treturn retval;\n> +}\n> +\n> static char *prune_directory(struct dir_struct *dir, struct pathspec *pathspec, int prefix)\n> {\n> \tchar *seen;\n> @@ -276,6 +296,7 @@ static struct option builtin_add_options[] = {\n> \tOPT_BOOL('e', \"edit\", &edit_interactive, N_(\"edit current diff and apply\")),\n> \tOPT__FORCE(&ignored_too, N_(\"allow adding otherwise ignored files\")),\n> \tOPT_BOOL('u', \"update\", &take_worktree_changes, N_(\"update tracked files\")),\n> +\tOPT_BOOL(0, \"renormalize\", &add_renormalize, N_(\"renormalize EOL of tracked files (implies -u)\")),\n> \tOPT_BOOL('N', \"intent-to-add\", &intent_to_add, N_(\"record only the fact that the path will be added later\")),\n> \tOPT_BOOL('A', \"all\", &addremove_explicit, N_(\"add changes from all tracked and untracked files\")),\n> \t{ OPTION_CALLBACK, 0, \"ignore-removal\", &addremove_explicit,\n> @@ -406,7 +427,7 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n> \t\t\t chmod_arg[1] != 'x' || chmod_arg[2]))\n> \t\tdie(_(\"--chmod param '%s' must be either -x or +x\"), chmod_arg);\n> \n> -\tadd_new_files = !take_worktree_changes && !refresh_only;\n> +\tadd_new_files = !take_worktree_changes && !refresh_only && !add_renormalize;\n> \trequire_pathspec = !(take_worktree_changes || (0 < addremove_explicit));\n> \n> \thold_locked_index(&lock_file, LOCK_DIE_ON_ERROR);\n> @@ -500,7 +521,10 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n> \n> \tplug_bulk_checkin();\n> \n> -\texit_status |= add_files_to_cache(prefix, &pathspec, flags);\n> +\tif (add_renormalize)\n> +\t\texit_status |= renormalize_tracked_files(&pathspec, flags);\n> +\telse\n> +\t\texit_status |= add_files_to_cache(prefix, &pathspec, flags);\n> \n> \tif (add_new_files)\n> \t\texit_status |= add_files(&dir, flags);\n> diff --git a/cache.h b/cache.h\n> index 6440e2bf21..3e63a74113 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -686,6 +686,7 @@ extern int ie_modified(const struct index_state *, const struct cache_entry *, s\n> \n> #define HASH_WRITE_OBJECT 1\n> #define HASH_FORMAT_CHECK 2\n> +#define HASH_RENORMALIZE 4\n> extern int index_fd(struct object_id *oid, int fd, struct stat *st, enum object_type type, const char *path, unsigned flags);\n> extern int index_path(struct object_id *oid, const char *path, struct stat *st, unsigned flags);\n> \n> diff --git a/read-cache.c b/read-cache.c\n> index 65f4fe8375..8acd3fcb93 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -631,13 +631,17 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n> {\n> \tint size, namelen, was_same;\n> \tmode_t st_mode = st->st_mode;\n> -\tstruct cache_entry *ce, *alias;\n> +\tstruct cache_entry *ce, *alias = NULL;\n> \tunsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY;\n> \tint verbose = flags & (ADD_CACHE_VERBOSE | ADD_CACHE_PRETEND);\n> \tint pretend = flags & ADD_CACHE_PRETEND;\n> \tint intent_only = flags & ADD_CACHE_INTENT;\n> \tint add_option = (ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE|\n> \t\t\t (intent_only ? ADD_CACHE_NEW_ONLY : 0));\n> +\tint newflags = HASH_WRITE_OBJECT;\n> +\n> +\tif (flags & HASH_RENORMALIZE)\n> +\t\tnewflags |= HASH_RENORMALIZE;\n> \n> \tif (!S_ISREG(st_mode) && !S_ISLNK(st_mode) && !S_ISDIR(st_mode))\n> \t\treturn error(\"%s: can only add regular files, symbolic links or git-directories\", path);\n> @@ -678,19 +682,23 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n> \tif (ignore_case) {\n> \t\tadjust_dirname_case(istate, ce->name);\n> \t}\n> +\tif (!(flags & HASH_RENORMALIZE)) {\n> +\t\talias = index_file_exists(istate, ce->name,\n> +\t\t\t\t\t ce_namelen(ce), ignore_case);\n> +\t\tif (alias &&\n> +\t\t !ce_stage(alias) &&\n> +\t\t !ie_match_stat(istate, alias, st, ce_option)) {\n> +\t\t\t/* Nothing changed, really */\n> +\t\t\tif (!S_ISGITLINK(alias->ce_mode))\n> +\t\t\t\tce_mark_uptodate(alias);\n> +\t\t\talias->ce_flags |= CE_ADDED;\n> \n> -\talias = index_file_exists(istate, ce->name, ce_namelen(ce), ignore_case);\n> -\tif (alias && !ce_stage(alias) && !ie_match_stat(istate, alias, st, ce_option)) {\n> -\t\t/* Nothing changed, really */\n> -\t\tif (!S_ISGITLINK(alias->ce_mode))\n> -\t\t\tce_mark_uptodate(alias);\n> -\t\talias->ce_flags |= CE_ADDED;\n> -\n> -\t\tfree(ce);\n> -\t\treturn 0;\n> +\t\t\tfree(ce);\n> +\t\t\treturn 0;\n> +\t\t}\n> \t}\n> \tif (!intent_only) {\n> -\t\tif (index_path(&ce->oid, path, st, HASH_WRITE_OBJECT)) {\n> +\t\tif (index_path(&ce->oid, path, st, newflags)) {\n> \t\t\tfree(ce);\n> \t\t\treturn error(\"unable to index file %s\", path);\n> \t\t}\n> diff --git a/sha1_file.c b/sha1_file.c\n> index 10c3a0083d..15abb184c2 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -74,6 +74,18 @@ static struct cached_object *find_cached_object(const unsigned char *sha1)\n> \treturn NULL;\n> }\n> \n> +\n> +static enum safe_crlf get_safe_crlf(unsigned flags)\n> +{\n> +\tif (flags & HASH_RENORMALIZE)\n> +\t\treturn SAFE_CRLF_RENORMALIZE;\n> +\telse if (flags & HASH_WRITE_OBJECT)\n> +\t\treturn safe_crlf;\n> +\telse\n> +\t\treturn SAFE_CRLF_FALSE;\n> +}\n> +\n> +\n> int mkdir_in_gitdir(const char *path)\n> {\n> \tif (mkdir(path, 0777)) {\n> @@ -1680,7 +1692,7 @@ static int index_mem(unsigned char *sha1, void *buf, size_t size,\n> \tif ((type == OBJ_BLOB) && path) {\n> \t\tstruct strbuf nbuf = STRBUF_INIT;\n> \t\tif (convert_to_git(&the_index, path, buf, size, &nbuf,\n> -\t\t\t\t write_object ? safe_crlf : SAFE_CRLF_FALSE)) {\n> +\t\t\t\t get_safe_crlf(flags))) {\n> \t\t\tbuf = strbuf_detach(&nbuf, &size);\n> \t\t\tre_allocated = 1;\n> \t\t}\n> @@ -1714,7 +1726,7 @@ static int index_stream_convert_blob(unsigned char *sha1, int fd,\n> \tassert(would_convert_to_git_filter_fd(path));\n> \n> \tconvert_to_git_filter_fd(&the_index, path, fd, &sbuf,\n> -\t\t\t\t write_object ? safe_crlf : SAFE_CRLF_FALSE);\n> +\t\t\t\t get_safe_crlf(flags));\n> \n> \tif (write_object)\n> \t\tret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\n> diff --git a/t/t0025-crlf-renormalize.sh b/t/t0025-crlf-renormalize.sh\n> new file mode 100755\n> index 0000000000..50d8f3a0f3\n> --- /dev/null\n> +++ b/t/t0025-crlf-renormalize.sh\n> @@ -0,0 +1,30 @@\n> +#!/bin/sh\n> +\n> +test_description='CRLF renormalization'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success setup '\n> +\tgit config core.autocrlf false &&\n> +\tprintf \"LINEONE\\nLINETWO\\nLINETHREE\\n\" >LF.txt &&\n> +\tprintf \"LINEONE\\r\\nLINETWO\\r\\nLINETHREE\\r\\n\" >CRLF.txt &&\n> +\tprintf \"LINEONE\\r\\nLINETWO\\nLINETHREE\\n\" >CRLF_mix_LF.txt &&\n> +\tgit add . &&\n> +\tgit commit -m initial\n> +'\n> +\n> +test_expect_success 'renormalize CRLF in repo' '\n> +\techo \"*.txt text=auto\" >.gitattributes &&\n> +\tgit add --renormalize \"*.txt\" &&\n> +cat >expect <<-\\EOF &&\n> +\ti/lf w/crlf attr/text=auto CRLF.txt\n> +\ti/lf w/lf attr/text=auto LF.txt\n> +\ti/lf w/mixed attr/text=auto CRLF_mix_LF.txt\n> +EOF\n> +\tgit ls-files --eol |\n> +\tsed -e \"s/\t/ /g\" -e \"s/ */ /g\" |\n> +\tsort >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_done\n"},{"id":"332771","messageId":"CAPig+cQocmJyoALeQeCRQPNsRgvJ5PSe=u2LN8Ec6aC86D4iQw@mail.gmail.com","threadId":"46891","inReplyTo":"20171116163828.14937-1-tboegi@web.de","subject":"Re: [PATCH v3 1/1] Introduce git add --renormalize .","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-11-17T20:44:11Z","receivedAt":"2017-11-17T20:47:03Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Nov 16, 2017 at 11:38 AM, wrote:\n> Make it safer to normalize the line endings in a repository:\n> Files that had been commited with CRLF will be commited with LF.\n>\n> The old way to normalize a repo was like this:\n> # Make sure that there are not untracked files\n> $ echo \"* text=auto\" >.gitattributes\n> $ git read-tree --empty\n> $ git add .\n> $ git commit -m \"Introduce end-of-line normalization\"\n>\n> The user must make sure that there are no untracked files,\n> otherwise they would have been added and tracked from now on.\n>\n> The new \"add ..renormalize\" does not add untracked files:\n> $ echo \"* text=auto\" >.gitattributes\n> $ git add --renormalize .\n> $ git commit -m \"Introduce end-of-line normalization\"\n>\n> Note that \"git add --renormalize \" is the short form for\n> \"git add -u --renormalize \".\n>\n> While add it, document that the same renormalization may be needed,\n> whenever a clean filter is added or changed.\n\nForgive me for chiming in so late, but as a newcomer to this topic,\nthe high-level choice made by this patch feels a bit questionable. I\nunderstand that, for people familiar with the \"old way\" of normalizing\nfiles, git-add might seems like the right place to house this\nfunctionality (and perhaps that's true from an implementation angle?),\nbut as one coming to this topic with no existing bias about\nimplementation or the \"old way\", git-add feels like an odd choice.\nThis sort of normalization (emptying the index, potentially modifying\nfiles, repopulating the index) seems too high-level for git-add.\n\nI _could_ understand if this functionality lived in, say, a new\ncommand git-attr:\n\n SYNOPSIS\n\n git attr renormalize [--no-commit | [-m ]] pathname...\n git attr check [-a | --all | attr…] [--] pathname…\n git attr check --stdin [-z] [-a | --all | attr…]\n\n DESCRIPTION\n\n 'git attr renormalize'\n\n Apply the \"clean\" process ... and commit the results. This is\n useful after changing `core.autocrlf` ... etc. With '-m', uses\n as the commit message, else launches the editor. Use\n '--no-commit' to skip the automatic commit.\n\nThe 'git attr check' command subsumes the role of existing\ngit-check-attr. One could envision git-attr growing additional\nsubcommands to edit .gitattributes, much like git-config edits\n.gitconfig.\n\n(I have since read the thread in which Junio's suggested[1] that\ngit-add could house this functionality, but it still feels too\nhigh-level.)\n\nMore below...\n\n> Signed-off-by: Torsten Bögershausen \n> ---\n> diff --git a/Documentation/git-add.txt b/Documentation/git-add.txt\n> @@ -175,6 +175,13 @@ for \"git add --no-all ...\", i.e. ignored removed files.\n> +--renormalize::\n> + Apply the \"clean\" process freshly to all tracked files to\n\nThis is the only time \"clean\" appears in git-add documentation. Every\nnewcomer to git learns about git-add very early on, but \"clean\nprocess\" is a fairly advanced topic, unlikely to be on a newcomer's\nradar. The term \"renormalize\" also feels out of place in git-add\ndocumentation. If I was a newcomer reading git-add documentation, I\nthink I'd be left pretty well clueless by this description. At the\nvery least, perhaps add links to git-attributes and 'core.autocflf'\nconfiguration.\n\n> + forcibly add them again to the index. This is useful after\n> + changing `core.autocrlf` configuration or the `text` attribute\n> + in order to correct files added with wrong CRLF/LF line endings.\n> + This option implies `-u`.\n\n[1]: https://public-inbox.org/git/xmqqbmlm9y94.fsf@gitster.mtv.corp.google.com/\n"},{"id":"332785","messageId":"xmqqd14gqrwe.fsf@gitster.mtv.corp.google.com","threadId":"46891","inReplyTo":"CAPig+cQocmJyoALeQeCRQPNsRgvJ5PSe=u2LN8Ec6aC86D4iQw@mail.gmail.com","subject":"Re: [PATCH v3 1/1] Introduce git add --renormalize .","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-18T01:47:45Z","receivedAt":"2017-11-18T01:47:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Sunshine writes:\n\n> I _could_ understand if this functionality lived in, say, a new\n> command git-attr:\n> ...\n> (I have since read the thread in which Junio's suggested[1] that\n> git-add could house this functionality, but it still feels too\n> high-level.)\n\nIf this were part of a hypothetical \"git attr\" command, I would have\na hard time understanding it. I would view the \"attr\" as attributes\nattached to each path, telling various things about the path, one of\nwhich may be how contents are handled between Git and the file on\nthe filesystem. In other words, \"attr\" is _not_ contents; it is a\nset of attributes attached to path that house the contents.\n\nWhat we are correcting here at this point in the expected usecase is\nnot an attribute. We are correting the contents, and between the\ntwo potential sources of \"the right version\" of the contents, the\nuser tells that the filesystem is the right one, so \"add\" is used\nto correct mistakes in the index based on what is in the filesystem\n(if we were correcting the other way, like Jonathan said in the\nmessage in the thread you read, \"checkout\" would be used to correct\nthe filesystem version using what is in the index).\n\n\n"},{"id":"339492","messageId":"CAHd499CSe6d1SBaDYNRD9HXaWRJ9QYzmOCaBWFrV52UNRMVqhA@mail.gmail.com","threadId":"46891","inReplyTo":"xmqq8tgrhdt0.fsf@gitster.mtv.corp.google.com","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Robert Dailey","fromEmail":"rcdailey.lists@gmail.com","sentAt":"2018-02-15T15:24:40Z","receivedAt":"2018-02-15T18:33:46Z","isPatch":false,"sender":{"key":"rcdailey.lists@gmail.com","avatar":null},"body":"On Tue, Oct 3, 2017 at 9:00 PM, Junio C Hamano wrote:\n> Torsten Bögershausen writes:\n>\n>>> $ git rm -r --cached . && git add .\n>>\n>> (Both should work)\n>>\n>> To be honest, from the documentation, I can't figure out the difference between\n>> $ git read-tree --empty\n>> and\n>> $ git rm -r --cached .\n>>\n>> Does anybody remember the discussion, why we ended up with read-tree ?\n>\n> We used to use neither, and considered it fine to \"rm .git/index\" if\n> you wanted to empty the on-disk index file in the old world. In the\n> modern world, folks want you to avoid touching filesystem directly\n> and instead want you to use Git tools, and the above are two obvious\n> ways to do so.\n>\n> \"git read-tree\" (without any parameter, i.e. \"read these 0 trees and\n> populate the index with it\") and its modern and preferred synonym\n> \"git read-tree --empty\" (i.e. \"I am giving 0 trees and I know the\n> sole effect of this command is to empty the index.\") are more direct\n> ways to express \"I want the index emptied\" between the two.\n>\n> The other one, \"git rm -r --cached .\", in the end gives you the same\n> state because it tells Git to \"iterate over all the entries in the\n> index, find the ones that match pathspec '.', and remove them from\n> the index.\". It is not wrong per-se, but conceptually it is a bit\n> roundabout way to say that \"I want the index emptied\", I would\n> think.\n>\n> I wouldn't be surprised if the \"rm -r --cached .\" were a lot slower,\n> due to the overhead of having to do the pathspec filtering that ends\n> up to be a no-op, but there shouldn't be a difference in the end\n> result.\n\nSorry to bring this old thread back to life, but I did notice that\nthis causes file modes to reset back to 644 (from 755) on Windows\nversion of Git. Is there a way to `$ git read-tree --empty && git add\n.` without mucking with file permissions?\n"},{"id":"339495","messageId":"xmqqvaey83my.fsf@gitster-ct.c.googlers.com","threadId":"46891","inReplyTo":"CAHd499CSe6d1SBaDYNRD9HXaWRJ9QYzmOCaBWFrV52UNRMVqhA@mail.gmail.com","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-15T19:16:37Z","receivedAt":"2018-02-15T19:16:46Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Robert Dailey writes:\n\n> On Tue, Oct 3, 2017 at 9:00 PM, Junio C Hamano wrote:\n>> Torsten Bögershausen writes:\n>>\n>>>> $ git rm -r --cached . && git add .\n>>>\n>>> (Both should work)\n>>>\n>>> To be honest, from the documentation, I can't figure out the difference between\n>>> $ git read-tree --empty\n>>> and\n>>> $ git rm -r --cached .\n>>>\n>>> Does anybody remember the discussion, why we ended up with read-tree ?\n>> ...\n>\n> Sorry to bring this old thread back to life, but I did notice that\n> this causes file modes to reset back to 644 (from 755) on Windows\n> version of Git. Is there a way to `$ git read-tree --empty && git add\n> .` without mucking with file permissions?\n\nI think the message you are referring to is a tangent that discusses\nhow it was done in the old world, with issues that come from the\nfact that with such an approach the paths are first removed from the\nindex and then added afresh to the index, which can lose cases and\nexecutable bits when working on a filesystem that does not retain\nenough information.\n\nThe way in the new world is to use \"add --renormalize\" which was\nadded at 9472935d (\"add: introduce \"--renormalize\"\", 2017-11-16), I\nthink.\n\n"},{"id":"339510","messageId":"CAHd499C0+L5TgdaL_gBSfr54Mhx+8pGZun1ex8-YTkhUEi+k_w@mail.gmail.com","threadId":"46891","inReplyTo":"xmqqvaey83my.fsf@gitster-ct.c.googlers.com","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Robert Dailey","fromEmail":"rcdailey.lists@gmail.com","sentAt":"2018-02-15T21:47:01Z","receivedAt":"2018-02-15T21:47:08Z","isPatch":false,"sender":{"key":"rcdailey.lists@gmail.com","avatar":null},"body":"On Thu, Feb 15, 2018 at 1:16 PM, Junio C Hamano wrote:\n> I think the message you are referring to is a tangent that discusses\n> how it was done in the old world, with issues that come from the\n> fact that with such an approach the paths are first removed from the\n> index and then added afresh to the index, which can lose cases and\n> executable bits when working on a filesystem that does not retain\n> enough information.\n>\n> The way in the new world is to use \"add --renormalize\" which was\n> added at 9472935d (\"add: introduce \"--renormalize\"\", 2017-11-16), I\n> think.\n\nOh I didn't realize someone actually did it. If so, that's awesome.\nThanks Junio!\n"},{"id":"339539","messageId":"20180216163422.GA2659@tor.lan","threadId":"46891","inReplyTo":"CAHd499CSe6d1SBaDYNRD9HXaWRJ9QYzmOCaBWFrV52UNRMVqhA@mail.gmail.com","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2018-02-16T16:34:22Z","receivedAt":"2018-02-16T16:34:33Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Thu, Feb 15, 2018 at 09:24:40AM -0600, Robert Dailey wrote:\n> On Tue, Oct 3, 2017 at 9:00 PM, Junio C Hamano wrote:\n\n[]\n> \n> Sorry to bring this old thread back to life, but I did notice that\n> this causes file modes to reset back to 644 (from 755) on Windows\n> version of Git. Is there a way to `$ git read-tree --empty && git add\n> .` without mucking with file permissions?\n\nNo problem with the delay, under the time we had the chance to improve Git:\n\n>Git 2.16 Release Notes\n>======================\n>[]\n>* \"git add --renormalize .\" is a new and safer way to record the fact\n> that you are correcting the end-of-line convention and other\n> \"convert_to_git()\" glitches in the in-repository data.\n\nCould you upgrade to Git 2.16.1 (or higher, just take the latest)\nand try with\ngit add --renormalize .\n?\n"},{"id":"339542","messageId":"CAHd499BrJ9egFvkPSQdFXL6=3qq9FrtiLV0Nz6tPbf-WT-P1ag@mail.gmail.com","threadId":"46891","inReplyTo":"20180216163422.GA2659@tor.lan","subject":"Re: Line ending normalization doesn't work as expected","fromName":"Robert Dailey","fromEmail":"rcdailey.lists@gmail.com","sentAt":"2018-02-16T17:19:35Z","receivedAt":"2018-02-16T17:19:44Z","isPatch":false,"sender":{"key":"rcdailey.lists@gmail.com","avatar":null},"body":"On Fri, Feb 16, 2018 at 10:34 AM, Torsten Bögershausen wrote:\n> On Thu, Feb 15, 2018 at 09:24:40AM -0600, Robert Dailey wrote:\n>> On Tue, Oct 3, 2017 at 9:00 PM, Junio C Hamano wrote:\n>\n> []\n>>\n>> Sorry to bring this old thread back to life, but I did notice that\n>> this causes file modes to reset back to 644 (from 755) on Windows\n>> version of Git. Is there a way to `$ git read-tree --empty && git add\n>> .` without mucking with file permissions?\n>\n> No problem with the delay, under the time we had the chance to improve Git:\n>\n>>Git 2.16 Release Notes\n>>======================\n>>[]\n>>* \"git add --renormalize .\" is a new and safer way to record the fact\n>> that you are correcting the end-of-line convention and other\n>> \"convert_to_git()\" glitches in the in-repository data.\n>\n> Could you upgrade to Git 2.16.1 (or higher, just take the latest)\n> and try with\n> git add --renormalize .\n> ?\n\nThanks for the response. Unfortunately I've deliberately been stuck on\nv2.13 because of [a bug in Git for Windows][1] that hasn't yet been\nresolved (it's a bug *somewhere*, not sure if it's git related or\nnot). Curly braces in aliases are being stripped which makes newer\nreleases unusable for me. I'll try upgrading on a different machine\nand see if renormalize works for the case of binary files with file\nmodes set to 755.\n\n[1]: https://github.com/git-for-windows/git/issues/1220\n"}]}