{"thread":{"id":"34245","subject":"git rebase fail with CRLF conversion","startedAt":"2013-06-21T13:41:50Z","lastAt":"2013-06-21T14:29:54Z","messageCount":5,"participants":["Yann Droneaud"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"221594","messageId":"fb20a7d711fdd218f58f1f2090b1c515@meuh.org","threadId":"34245","inReplyTo":null,"subject":"git rebase fail with CRLF conversion","fromName":"Yann Droneaud","fromEmail":"ydroneaud@opteya.com","sentAt":"2013-06-21T13:41:50Z","receivedAt":"2013-06-21T13:41:50Z","isPatch":false,"sender":{"key":"ydroneaud@opteya.com","avatar":"https://avatars.githubusercontent.com/u/881377?v=4"},"body":"Hi (again),\n\nFollowing my previous email \"Tracking vendor release with Git\" [1][2],\nand the advice from Git users/developers, I'm trying to use \n.gitattributes\nto handle CRLF/LF conversion.\n\nI'm following advices from:\n\n  - \"Dealing with line endings\"\n    https://help.github.com/articles/dealing-with-line-endings\n\n  - \"Tim's Blog, Mind the End of Your Line\"\n    http://timclem.wordpress.com/2012/03/01/mind-the-end-of-your-line/\n\n  - \"7.2 Customizing Git - Git Attributes\"\n    http://git-scm.com/book/ch7-2.html\n\n  - \"gitattributes(5) Manual Page\"\n    https://www.kernel.org/pub/software/scm/git/docs/gitattributes.html\n\nWhile testing the behavor of Git regarding CRLF handling,\nI've found git rebase fail when trying to apply a commit\nthat's going to fix/convert CRLF to LF.\n\nI'm using the following script:\n\nA file with CRLF end of line (EOL) \"test\" is added and commited,\nthen a file .gitattributes is added and commited.\nThe .gitattributes has a single line \"test text\" to ask git to do\nEOL conversion.\nAfter adding this .gitattributes file, Git start complaining about\nthe line ending to file \"test\". The file is fixed and commited.\n\nThen this branch is rebased on top of another parent.\nAnd here come the failures:\n\n\nMost of the time git rebase failed with the following error:\n\n  ...\n  First, rewinding head to replay your work on top of it...\n  Applying: Commit raw\n  Applying: Added git attributes\n  Applying: Normalization\n  Using index info to reconstruct a base tree...\n  Falling back to patching base and 3-way merge...\n  error: Your local changes to the following files would be overwritten \nby merge:\n        test\n  Please, commit your changes or stash them before you can merge.\n  Aborting\n  Failed to merge in the changes.\n  Patch failed at 0003 Normalization\n  ...\n\n\nBut sometime it failed with another error:\n\nwith core.safecrlf = warn:\n\n  ...\n  First, rewinding head to replay your work on top of it...\n  Applying: Commit raw\n  Applying: Added git attributes\n  warning: CRLF will be replaced by LF in test.\n  The file will have its original line endings in your working \ndirectory.\n  Applying: Normalization\n  fatal: unrecognized input\n  Repository lacks necessary blobs to fall back on 3-way merge.\n  Cannot fall back to three-way merge.\n  Patch failed at 0003 Normalization\n  ...\n\nwith core.safecrlf = true:\n\n  ...\n  First, rewinding head to replay your work on top of it...\n  Applying: Commit raw\n  Applying: Added git attributes\n  fatal: CRLF would be replaced by LF in test.\n  Applying: Normalization\n  fatal: unrecognized input\n  Repository lacks necessary blobs to fall back on 3-way merge.\n  Cannot fall back to three-way merge.\n  Patch failed at 0003 Normalization\n  ...\n\n\nIn thoses two latter cases, running \"git add\" does not fail with a \nfatal error: it does nothing.\nI need to run \"touch test\" to make \"git add\" fail with error \"fatal: \nCRLF would be replaced by LF in test.\n\n\nI believe \"git rebase\" should not fail here, but more, it must not fail \nin a different fashion randomly.\n\nPlease find in reply to this email:\n  - a shell script to demonstrate the behavor\n  - a patch to t0020-crlf.sh to add two rebase tests.\n\n1. <1370970410-7935-1-git-send-email-ydroneaud@opteya.com>\n2. <http://thread.gmane.org/gmane.comp.version-control.git/227466>\n    <http://marc.info/?l=git&m=137097069115462&w=2>\n\n\nRegards\n\n-- \nYann Droneaud\nOPTEYA\n"},{"id":"221595","messageId":"6ae09e227e559a024d3ffa13570e81a2@meuh.org","threadId":"34245","inReplyTo":"fb20a7d711fdd218f58f1f2090b1c515@meuh.org","subject":"Re: git rebase fail with CRLF conversion","fromName":"Yann Droneaud","fromEmail":"ydroneaud@opteya.com","sentAt":"2013-06-21T13:51:53Z","receivedAt":"2013-06-21T13:51:53Z","isPatch":false,"sender":{"key":"ydroneaud@opteya.com","avatar":"https://avatars.githubusercontent.com/u/881377?v=4"},"body":"Hi,\n\nLe 21.06.2013 15:41, Yann Droneaud a écrit :\n>\n>\n> I believe \"git rebase\" should not fail here, but more, it must not\n> fail in a different fashion randomly.\n>\n> Please find in reply to this email:\n>  - a shell script to demonstrate the behavor\n\n\nPlease find a shell script to test git rebase with .gitattributes text \nflag\nset to enable End-Of-Line (EOL) conversion from CRLF to LF, with \ncore.safecrlf set to warn\nand to true.\n\nRegards.\n\n-- \nYann Droneaud\nOPTEYA\n\n\n----------------------------------8<----------------------------------\n#! /bin/sh\n\nset -e\n\nLC_ALL=C\nexport LC_ALL\n\n#GIT_AUTHOR_DATE=`date -R`\nGIT_AUTHOR_DATE=\"2001-01-01 00:00:00.+00:00\"\nGIT_COMMITTER_DATE=\"$GIT_AUTHOR_DATE\"\n\nexport GIT_AUTHOR_DATE\nexport GIT_COMMITTER_DATE\n\n# global settings\nset +e\nglobal_core_eol=`git config --global core.eol`\nglobal_core_safecrlf=`git config --global core.safecrlf`\nglobal_core_autocrlf=`git config --global core.autocrlf`\nset -e\n\n# archive\nif test -d work; then\n     dir=`mktemp -d work.XXXXX`\n     rmdir $dir\n     mv work $dir\nfi\n\nmkdir work\n\necho \"****** Testing in work *******\"\n\ncd work\n\n# prepare\ngit init\ngit commit --allow-empty -m \"empty root commit\"\n\n# local settings, might be different from the global settings\nset +e\nlocal_core_eol=`git config core.eol`\nlocal_core_safecrlf=`git config core.safecrlf`\nlocal_core_autocrlf=`git config core.autocrlf`\nset -e\n\necho \"global core.eol       = $global_core_eol\"\necho \"global core.safecrlf  = $global_core_safecrlf\"\necho \"global core.autocrlf  = $global_core_autocrlf\"\necho \"local core.eol        = $local_core_eol\"\necho \"local core.safecrlf   = $local_core_safecrlf\"\necho \"local core.autocrlf   = $local_core_autocrlf\"\n\n# set default configuration\ngit config --local core.eol native\ngit config --local core.safecrl warn\ngit config --local core.autocrlf false\n\necho \"current core.eol      = `git config core.eol`\"\necho \"current core.safecrlf = `git config core.safecrlf`\"\necho \"current core.autocrlf = `git config core.autocrlf`\"\n\nCRLF=\"\\r\\n\"\nCR=\"\\r\"\nLF=\"\\n\"\n\n# TODO detect line ending on the current environment\nif true ; then\n     EOL=$LF\nelse\n     EOL=$CRLF\nfi\n\necho \"Create work branches\"\ngit branch import-raw master\ngit branch import-eol master\n\necho \"Create a branch to be used as new root later\"\ngit checkout import-eol\ngit commit --allow-empty -m \"an empty commit\"\n\nfile_type=\n\ncreate_raw_file ()\n{\n     # Want to test mixed EOL:\n     # printf \"Hello World 1${CRLF}Hello World 2${CRLF}${CR}Hello World \n3${CRLF}Hello World 4\" > test\n     printf \"Hello World 1${CRLF}Hello World 2${CRLF}Hello World \n3${CRLF}Hello World 4\" > test\n     file_type=\"`file test`\"\n     echo \"::: $file_type\"\n\n}\n\ncheck_file ()\n{\n     local t=\"`file test`\"\n     if test \"x$file_type\" != \"x$t\" ; then\n\tfile_type=\"$t\"\n\techo \"::: $file_type\"\n     fi\n}\n\ncreate_git_attributes ()\n{\n     printf \"test text${EOL}\" > .gitattributes\n}\n\n#\n# First test:\n#\n# import raw, add .gitattributes after, normalize, rebase\n#\necho \"===== BEGIN: first test =====\"\ngit checkout import-raw\ngit reset --hard master\ncreate_raw_file\ngit add test && check_file\n\ngit commit -m \"Commit raw\" && check_file\n\ncreate_git_attributes  && check_file\ngit add .gitattributes && check_file\ngit commit -m \"Added git attributes\" && check_file\n\necho \"--- First kind of 'normalization'\"\n#\n# trick from https://help.github.com/articles/dealing-with-line-endings\n#\ngit rm --cached test\ngit reset --hard && check_file\ngit add test && check_file\ngit commit -m \"Normalization\"\ncheck_file\ngit tag norm1\n\necho \"--- Second kind of 'normalization'\"\ngit reset --hard HEAD^\ncheck_file\ncreate_raw_file\nrm test\ngit checkout test  && check_file\ngit add test && check_file\ngit commit -m \"Normalization\"\ncheck_file\ngit tag norm2\n\necho \"--- Third kind of 'normalization'\"\ngit reset --hard HEAD^\ncheck_file\ncreate_raw_file\ndos2unix test  && check_file\ngit add test  && check_file\ngit commit -m \"Normalization\"\ncheck_file\ngit tag norm3\n\necho \"--- differences ? ---\"\ngit diff norm1 norm2\ngit diff norm1 norm3\ngit diff norm2 norm1\ngit diff norm2 norm3\ngit diff norm3 norm1\ngit diff norm3 norm2\n\necho \"--- rebase, should failed ---\"\ngit rebase import-eol || {\n     echo \"--- Expected failure to rebase on another branch ---\"\n     check_file\n     git status\n     git diff -w --stat\n     git diff\n\n     # just adding the file again and continue ...\n     git add test && check_file\n     git rebase --continue || { echo \"### failed to continue rebase\" ; \nexit 1 ; }\n     check_file\n}\n\necho \"--- rebase complete ---\"\n\ncheck_file\n\necho \"===== END: first test =====\"\n\n\n#\n# again, with safecrlf set\n#\necho \"===== BEGIN: second test =====\"\ngit config core.safecrlf true\n\ngit checkout import-raw\ngit reset --hard master\ncreate_raw_file\ngit add test && check_file\n\ngit commit -m \"Commit raw\" && check_file\n\ncreate_git_attributes  && check_file\ngit add .gitattributes && check_file\ngit commit -m \"Added git attributes\" && check_file\n\necho \"--- First kind of 'normalization', git add should failed\"\ngit rm --cached test\ngit reset --hard && check_file\ngit add test && { echo \"### git add must failed !\" ; exit 1 ; }\n\necho \"--- Second kind of 'normalization', git add should failed\"\nrm test\ngit checkout test && check_file\ngit add test && { echo \"### git add must failed !\" ; exit 1 ; }\n\necho \"--- Third kind of 'normalization'\"\ndos2unix test && check_file\ngit add test && check_file\ngit commit -m \"Normalization\"\ncheck_file\n\necho \"--- rebase, should failed ---\"\ngit rebase import-eol || {\n     echo \"--- Expected failure to rebase on another branch ---\"\n     check_file\n     git status\n\n     set +e\n     git diff -w --stat || echo \"!? git diff return an error !?\"\n     git diff || echo \"!? git diff return an error !?\"\n     set -e\n\n     git add test && { echo \"### git add must failed !\" ; exit 1 ; }\n\n     dos2unix test && check_file\n     git add test && check_file\n     git status\n     git rebase --continue || { echo \"### failed to continue rebase\" ; \nexit 1 ; }\n     check_file\n}\n\necho \"--- rebase complete ---\"\n\ncheck_file\n\necho \"===== END: second test =====\"\n\n#\n# the way around:\n# first .gitattributes, then the file with CRLF (with safecrlf unset)\n#\necho \"===== BEGIN: third test =====\"\ngit config core.safecrlf false\ngit checkout import-raw\ngit reset --hard master\n\ncreate_git_attributes\ngit add .gitattributes\ngit commit -m \"Added git attributes\"\n\ncreate_raw_file\ngit add test && check_file\ngit commit -m \"Commit raw\" && check_file\n\necho \"--- First kind of 'normalization', git commit should failed\"\ngit rm --cached test\ngit reset --hard && check_file\ngit add test && check_file\ngit commit -m \"Normalization\" && { echo \"### git commit should failed \nhere !\" ; exit 1 ; }\ncheck_file\n\necho \"--- Second kind of 'normalization', git commit should failed\"\nrm test\ngit checkout test && check_file\ngit add test && check_file\ngit commit -m \"Normalization\" && { echo \"### git commit should failed \nhere !\" ; exit 1 ; }\ncheck_file\n\necho \"--- Third kind of 'normalization', git commit should failed\"\ndos2unix test && check_file\ngit add test && check_file\ngit commit -m \"Normalization\" && { echo \"### git commit should failed \nhere !\" ; exit 1 ; }\ncheck_file\n\necho \"--- rebase, should failed ---\"\ngit rebase import-eol || {\n     echo \"--- Expected failure to rebase on another branch ---\"\n     check_file\n     git status\n     git diff -w --stat\n     git diff\n\n     # just adding the file again and continue ...\n     git add test && check_file\n     git rebase --continue || { echo \"### failed to continue rebase\" ; \nexit 1 ; }\n     check_file\n}\n\necho \"--- rebase complete ---\"\n\ncheck_file\n\necho \"===== END: third test =====\"\n\n\n#\n# the way around\n# first .gitattributes, then the file with CRLF (with safecrlf set to \ntrue)\n#\necho \"===== BEGIN: fourth test =====\"\ngit config core.safecrlf true\ngit checkout import-raw\ngit reset --hard master\n\ncreate_git_attributes\ngit add .gitattributes\ngit commit -m \"Added git attributes\"\n\ncreate_raw_file\n# with safecrlf set to true, it's impossible to add the file with CRLF.\ngit add test && { echo \"### git add should failed here !\" ; exit 1 ; }\ndos2unix test && check_file\ngit add test && check_file\ngit commit -m \"Commit raw\" && check_file\n\necho \"--- rebase, should failed ---\"\ngit rebase import-eol || {\n     echo \"--- Expected failure to rebase on another branch ---\"\n     check_file\n     git status\n     git diff -w --stat\n     git diff\n\n     # just adding the file again and continue ...\n     git add test && check_file\n     git rebase --continue || { echo \"### failed to continue rebase\" ; \nexit 1 ; }\n     check_file\n}\n\necho \"--- rebase complete ---\"\n\ncheck_file\n\necho \"===== END: fourth test =====\"\n\nexit 0\n----------------------------------8<----------------------------------\n"},{"id":"221596","messageId":"1371824124-4242-1-git-send-email-ydroneaud@opteya.com","threadId":"34245","inReplyTo":"fb20a7d711fdd218f58f1f2090b1c515@meuh.org","subject":"[PATCH] t0020-crlf: test rebase with text conversion and safecrlf","fromName":"Yann Droneaud","fromEmail":"ydroneaud@opteya.com","sentAt":"2013-06-21T14:15:24Z","receivedAt":"2013-06-21T14:15:24Z","isPatch":true,"sender":{"key":"ydroneaud@opteya.com","avatar":"https://avatars.githubusercontent.com/u/881377?v=4"},"body":"This patch add two tests to reproduce the problems described\nin thread \"git rebase fail with CRLF conversion\"\n <fb20a7d711fdd218f58f1f2090b1c515@meuh.org>\n <http://thread.gmane.org/gmane.comp.version-control.git/228613>\n <http://marc.info/?l=git&m=137182211414404&w=2>\n\n- Add and commit a file with CRLF,\n- Add and commit a .gitattributes with text flag for the file,\n- Convert CRLF to LF in file, commit the change,\n- Rebase the whole on top of another parent\n  The rebase currently failed.\n\nThis scenario is repeated twice, first with core.safecrlf set to warn,\na second time with core.safecrlf set to true.\n\ngit diff will also fail when core.safecrlf is set to true\nand the file is not already converted to LF.\n\nSigned-off-by: Yann Droneaud <ydroneaud@opteya.com>\n---\n t/t0020-crlf.sh | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 52 insertions(+)\n\ndiff --git a/t/t0020-crlf.sh b/t/t0020-crlf.sh\nindex 1a8f44c..12147ba 100755\n--- a/t/t0020-crlf.sh\n+++ b/t/t0020-crlf.sh\n@@ -506,4 +506,56 @@ test_expect_success 'New CRLF file gets LF in repo' '\n \ttest_cmp alllf alllf2\n '\n \n+test_expect_success 'Prepare rebase test' '\n+\ttest_config core.autocrlf false &&\n+\ttest_config core.safecrlf warn &&\n+\ttest_config core.eol native &&\n+\tgit branch rebase-initial &&\n+\tgit checkout -b rebase-other rebase-initial &&\n+\tgit commit --allow-empty -m \"Another commit\" &&\n+\tgit checkout rebase-initial\n+'\n+\n+# git rebase will failed to apply the \"normalization\" commit\n+test_expect_success 'Rebase with core.safecrlf set to warn (default)' '\n+\ttest_config core.autocrlf false &&\n+\ttest_config core.safecrlf warn &&\n+\ttest_config core.eol native &&\n+\tgit checkout -b rebase-warn rebase-initial &&\n+\tfor w in A couple of lines ; do echo $w ; done | append_cr >rebase-warn.txt &&\n+\tgit add rebase-warn.txt && git commit -m \"Added raw text\" &&\n+\techo rebase-warn.txt text >> .gitattributes &&\n+\tgit add .gitattributes && git commit -m \"Added .gitattributes\" &&\n+\tgit rm --cached rebase-warn.txt && git reset --hard &&\n+\tgit add rebase-warn.txt && git commit -m \"Normalized\" &&\n+\tgit rebase rebase-other || {\n+\t\tgit status && git diff &&\n+\t\tgit add rebase-warn.txt && git rebase --continue ;\n+\t}\n+'\n+\n+#\n+# git rebase will failed to apply the \"normalization\" commit\n+# additionnaly\n+# git diff and git add will failed when core.safecrlf is set to true\n+#\n+test_expect_success 'Rebase with core.safecrlf set to true' '\n+\ttest_config core.autocrlf false &&\n+\ttest_config core.safecrlf true &&\n+\ttest_config core.eol native &&\n+\tgit checkout -b rebase-true rebase-initial &&\n+\tfor w in Another couple of lines ; do echo $w ; done | append_cr >rebase-true.txt &&\n+\tgit add rebase-true.txt && git commit -m \"Added raw text\" &&\n+\techo rebase-true.txt text >> .gitattributes &&\n+\tgit add .gitattributes && git commit -m \"Added .gitattributes\" &&\n+\tgit diff || git add rebase-true.txt ||\n+\tfor w in Another couple of lines ; do echo $w ; done >rebase-true.txt &&\n+\tgit add rebase-true.txt && git commit -m \"Normalized\" &&\n+\tgit rebase rebase-other || {\n+\t\tgit status && git diff || git add rebase-true.txt ||\n+\t\tfor w in Another couple of lines ; do echo $w ; done >rebase-true.txt &&\n+\t\tgit add rebase-true.txt && git rebase --continue ;\n+\t}\n+'\n+\n test_done\n-- \n1.8.2.1\n"},{"id":"221598","messageId":"41e212cf1702cd60114f522f96627d20@meuh.org","threadId":"34245","inReplyTo":"1371824124-4242-1-git-send-email-ydroneaud@opteya.com","subject":"Re: [PATCH] t0020-crlf: test rebase with text conversion and safecrlf","fromName":"Yann Droneaud","fromEmail":"ydroneaud@opteya.com","sentAt":"2013-06-21T14:24:57Z","receivedAt":"2013-06-21T14:24:57Z","isPatch":true,"sender":{"key":"ydroneaud@opteya.com","avatar":"https://avatars.githubusercontent.com/u/881377?v=4"},"body":"Le 21.06.2013 16:15, Yann Droneaud a écrit :\n> This patch add two tests to reproduce the problems described\n> in thread \"git rebase fail with CRLF conversion\"\n>  <fb20a7d711fdd218f58f1f2090b1c515@meuh.org>\n>  <http://thread.gmane.org/gmane.comp.version-control.git/228613>\n>  <http://marc.info/?l=git&m=137182211414404&w=2>\n>\n> - Add and commit a file with CRLF,\n> - Add and commit a .gitattributes with text flag for the file,\n> - Convert CRLF to LF in file, commit the change,\n> - Rebase the whole on top of another parent\n>   The rebase currently failed.\n>\n> This scenario is repeated twice, first with core.safecrlf set to \n> warn,\n> a second time with core.safecrlf set to true.\n>\n\nNote: those two tests might failed as described in message\n\n\"git rebase fail with CRLF conversion\"\n  <fb20a7d711fdd218f58f1f2090b1c515@meuh.org>\n  <http://thread.gmane.org/gmane.comp.version-control.git/228613>\n  <http://marc.info/?l=git&m=137182211414404&w=2>\n\nI haven't found a testcase to reproduce the same exact failure at will.\n\nSo this patch should probably not used asis until the two kind of \nerrors can be reproduced.\n\nAdditionnaly, It should also be tested on a CRLF system (Windows).\n\n> git diff will also fail when core.safecrlf is set to true\n> and the file is not already converted to LF.\n>\n\nIt might be a test on its own.\n\nRegards\n\n-- \nYann Droneaud\nOPTEYA\n"},{"id":"221599","messageId":"05ece7ca52df8ac8cd082cab4926487a@meuh.org","threadId":"34245","inReplyTo":"fb20a7d711fdd218f58f1f2090b1c515@meuh.org","subject":"Re: git rebase fail with CRLF conversion","fromName":"Yann Droneaud","fromEmail":"ydroneaud@opteya.com","sentAt":"2013-06-21T14:29:54Z","receivedAt":"2013-06-21T14:29:54Z","isPatch":false,"sender":{"key":"ydroneaud@opteya.com","avatar":"https://avatars.githubusercontent.com/u/881377?v=4"},"body":"Le 21.06.2013 15:41, Yann Droneaud a écrit :\n\n>\n> In thoses two latter cases, running \"git add\" does not fail with a\n> fatal error: it does nothing.\n> I need to run \"touch test\" to make \"git add\" fail with error \"fatal:\n> CRLF would be replaced by LF in test.\n>\n\nWhile searching on the Internet, I've found other people complaining of \na similar error\n\nhttp://stackoverflow.com/questions/5074136/git-rebase-fails-your-local-changes-to-the-following-files-would-be-overwritte\n\nThe workaround, \"git config --global core.trustctime false\", might \nsuggest there's a race somewhere while rebasing.\n\nBTW, I forgot to describe my environnment:\n\n- GNU/Linux\n- Fedora 18 x86_64\n- git version 1.8.1.4\n- git version 1.8.3.1.448.gfb7dfaa\n\n>\n> I believe \"git rebase\" should not fail here, but more, it must not\n> fail in a different fashion randomly.\n>\n\nRegards.\n\n-- \nYann Droneaud\nOPTEYA\n"}]}