{"thread":{"id":"62419","subject":"Inconsistent handling of corrupt patches based on line endings","startedAt":"2024-10-28T16:58:06Z","lastAt":"2024-10-29T16:32:27Z","messageCount":6,"participants":["Peregi Tamás","Taylor Blau","Elijah Newren","René Scharfe","Junio C Hamano","Torsten Bögershausen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"506233","messageId":"CAExmRTbMCayh+vHW3g=2p=FMb=VKXomEbn7_0t+kK-eF79JnEA@mail.gmail.com","threadId":"62419","inReplyTo":null,"subject":"Inconsistent handling of corrupt patches based on line endings","fromName":"Peregi Tamás","fromEmail":"petamas@gmail.com","sentAt":"2024-10-28T16:57:54Z","receivedAt":"2024-10-28T16:58:06Z","isPatch":false,"sender":{"key":"petamas@gmail.com","avatar":null},"body":"Hi all,\n\nI might've found an inconsistency in how git-apply treats corrupt patches\n(representing empty context lines with completely empty lines instead of\nlines containing a single space - usually a result of a \"trim trailing\nwhitespace\" editor setting) based on whether the patch file uses\nWindows-style line endings (CRLF) or Unix-style line endings (LF only).\n\nRegardless of whether I use LF or CRLF in original.txt, \"git apply --check\n--ignore-whitespace\" will happily apply intact-win.patch (uncorrupted patch\nfile with CRLF), intact-unix.patch (uncorrupted patch file with LF) and\ncorrupt-unix.patch (corrupted patch file with LF), but reports \"error:\ncorrupt patch at line 6\" for corrupt-win.patch (corrupted patch file with\nCRLF).\n\nMy intuition tells me git-apply should either fail for all corrupt patch\nfiles (regardless of EOL) or it should apply the corrupt patch to work\naround the issue caused by misconfigured text editors (regardless of EOL).\n\nAm I missing something, or is this a bug/inconsistency in\ngit-apply's behaviour?\n\nThanks in advance: Tamás Peregi\n\n**What did you do before the bug happened? (Steps to reproduce your issue)**\n\n1. Rename either original-win.txt or original-unix.txt to original.txt\n2. Run the following commands:\n2.1. git apply --check --ignore-whitespace intact-unix.patch\n2.2. git apply --check --ignore-whitespace intact-win.patch\n2.3. git apply --check --ignore-whitespace corrupt-unix.patch\n2.4. git apply --check --ignore-whitespace corrupt-win.patch\n\n**What did you expect to happen? (Expected behavior)**\n\nI'd expect one of the following:\n- All 4 commands succeed.\n- 2.1 and 2.2 succeeds, 2.3 and 2.4 both fail with \"error: corrupt patch at\nline 6\"\n\nIn other words: corrupted patches should either be accepted or not,\nregardless of what EOL markers are used in the patch file.\n\n**What happened instead? (Actual behavior)**\n\nOnly 2.4, i.e. applying corrupt-win.patch has failed.\n\n**What's different between what you expected and what actually happened?**\n\n2.3 and 2.4 should have the same result.\n\n**Anything else you want to add:**\n\nWhile the example seems contrived, it has been encountered in the wild when\nI was copying a port from https://github.com/microsoft/vcpkg (set up with\nautocrlf=false and all files treated as binaries in .gitattributes) to my\ncompany's repository (set up with autocrlf=true and patch files treated as\ntext). The patches and application worked in Microsoft's repo, then worked\non my machine (Windows) in our repor, but failed in CI on Windows, and\nstarted failing on my machine after switching to a different branch and\nswitching back due to \"git checkout\" converting the newline characters.\n\nI think these kind of corrupt patch files should fail to apply\nconsistently, but being able to apply them is also acceptable. However, the\ncurrent inconsistent behaviour (apply if patch is LF, but fail if patch is\nCRLF) is confusing, and causes \"works on my machine\" issues.\n\n\n[System Info]\ngit version:\ngit version 2.46.0.windows.1\ncpu: x86_64\nbuilt from commit: 2e6a859ffc0471f60f79c1256f766042b0d5d17d\nsizeof-long: 4\nsizeof-size_t: 8\nshell-path: D:/git-sdk-64-build-installers/usr/bin/sh\nfeature: fsmonitor--daemon\nlibcurl: 8.9.0\nOpenSSL: OpenSSL 3.2.2 4 Jun 2024\nzlib: 1.3.1\nuname: Windows 10.0 19045\ncompiler info: gnuc: 14.1\nlibc info: no libc information available\n$SHELL (typically, interactive shell): C:\\Program Files\\Git\\usr\\bin\\bash.exe\n\n\na\n\nb\nc\nd\ne\n\nf\n\n\ndiff --git a/original.txt b/original.txt\nindex e7582b7..940b5a6 100644\n--- a/original.txt\n+++ b/original.txt\n@@ -2,6 +2,7 @@ a\n \n b\n c\n+NEW LINE\n d\n e\n \n\n\ndiff --git a/original.txt b/original.txt\nindex e7582b7..940b5a6 100644\n--- a/original.txt\n+++ b/original.txt\n@@ -2,6 +2,7 @@ a\n \n b\n c\n+NEW LINE\n d\n e\n \n\n\ndiff --git a/original.txt b/original.txt\nindex e7582b7..940b5a6 100644\n--- a/original.txt\n+++ b/original.txt\n@@ -2,6 +2,7 @@ a\n\n b\n c\n+NEW LINE\n d\n e\n\n\n\ndiff --git a/original.txt b/original.txt\nindex e7582b7..940b5a6 100644\n--- a/original.txt\n+++ b/original.txt\n@@ -2,6 +2,7 @@ a\n\n b\n c\n+NEW LINE\n d\n e\n\n\n\na\n\nb\nc\nd\ne\n\nf\n"},{"id":"506240","messageId":"Zx/TwhuG4Ni7mf6F@nand.local","threadId":"62419","inReplyTo":"CAExmRTbMCayh+vHW3g=2p=FMb=VKXomEbn7_0t+kK-eF79JnEA@mail.gmail.com","subject":"Re: Inconsistent handling of corrupt patches based on line endings","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-10-28T18:11:14Z","receivedAt":"2024-10-28T18:11:17Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Oct 28, 2024 at 05:57:54PM +0100, Peregi Tamás wrote:\n> Hi all,\n>\n> I might've found an inconsistency in how git-apply treats corrupt patches\n> (representing empty context lines with completely empty lines instead of\n> lines containing a single space - usually a result of a \"trim trailing\n> whitespace\" editor setting) based on whether the patch file uses\n> Windows-style line endings (CRLF) or Unix-style line endings (LF only).\n\nLet's see if any recent apply.c folks have thoughts...:\n\n$ git shortlog -nse --no-merges --since=3.years.ago.. -- apply.c\n 21 Elijah Newren \n 12 Patrick Steinhardt \n 7 Ævar Arnfjörð Bjarmason \n 5 Junio C Hamano \n 4 Jeff King \n 4 René Scharfe \n\nThanks,\nTaylor\n"},{"id":"506241","messageId":"CABPp-BFB+eG7WVJFm3EBfgKzuspAYn9KpDHodpgKuFmWefARBw@mail.gmail.com","threadId":"62419","inReplyTo":"Zx/TwhuG4Ni7mf6F@nand.local","subject":"Re: Inconsistent handling of corrupt patches based on line endings","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2024-10-28T18:39:47Z","receivedAt":"2024-10-28T18:39:59Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Oct 28, 2024 at 11:11 AM Taylor Blau wrote:\n>\n> On Mon, Oct 28, 2024 at 05:57:54PM +0100, Peregi Tamás wrote:\n> > Hi all,\n> >\n> > I might've found an inconsistency in how git-apply treats corrupt patches\n> > (representing empty context lines with completely empty lines instead of\n> > lines containing a single space - usually a result of a \"trim trailing\n> > whitespace\" editor setting) based on whether the patch file uses\n> > Windows-style line endings (CRLF) or Unix-style line endings (LF only).\n>\n> Let's see if any recent apply.c folks have thoughts...:\n>\n> $ git shortlog -nse --no-merges --since=3.years.ago.. -- apply.c\n> 21 Elijah Newren \n> 12 Patrick Steinhardt \n> 7 Ævar Arnfjörð Bjarmason \n> 5 Junio C Hamano \n> 4 Jeff King \n> 4 René Scharfe \n\n...and thus we learn why no one wanted to clean up the header files in\ngit.git before me. ;-)\n\nI believe this behavior was caused by:\n\n$ git log -1 b507b465f7831612b9d9fc643e3e5218b64e5bfa\ncommit b507b465f7831612b9d9fc643e3e5218b64e5bfa\nAuthor: Linus Torvalds \nDate: Thu Oct 19 19:26:08 2006 -0700\n\n git-apply: prepare for upcoming GNU diff -u format change.\n\n The latest GNU diff from CVS emits an empty line to express\n an empty context line, instead of more traditional \"single\n white space followed by a newline\". Do not get broken by it.\n\n Signed-off-by: Linus Torvalds \n Signed-off-by: Junio C Hamano \n\nThat code special-cased a line containing '\\n' but not a line\ncontaining only '\\r\\n'.\n\nAs to whether that's correct, personally I'd rather only special case\nworkaround important existing clients. Back in 2006, working with GNU\ndiff was incredibly important, and I'd say is still important today.\nI can see Peregi's comment that this make line ending slightly\ninconsistent, but I feel like the blank-line handling is a workaround\nfor an existing client we want to interoperate with and absent a\nsimilar important client with mis-behaving '\\r\\n'-only lines, I\nwouldn't be interested in adding support for it. But that's just my\noff-the-cuff feeling and I don't feel strongly about it. Further, all\nbut one of my contributions above were mere header changes, so if\nothers have other opinions, they should probably be weighted more\nheavily than mine on this topic.\n"},{"id":"506265","messageId":"d74232a4-126d-4bd8-8f60-a3fac80492d5@web.de","threadId":"62419","inReplyTo":"CABPp-BFB+eG7WVJFm3EBfgKzuspAYn9KpDHodpgKuFmWefARBw@mail.gmail.com","subject":"Re: Inconsistent handling of corrupt patches based on line endings","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2024-10-28T22:07:19Z","receivedAt":"2024-10-28T22:07:35Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 28.10.24 um 19:39 schrieb Elijah Newren:\n> On Mon, Oct 28, 2024 at 11:11 AM Taylor Blau wrote:\n>>\n>> On Mon, Oct 28, 2024 at 05:57:54PM +0100, Peregi Tamás wrote:\n>>> Hi all,\n>>>\n>>> I might've found an inconsistency in how git-apply treats corrupt patches\n>>> (representing empty context lines with completely empty lines instead of\n>>> lines containing a single space - usually a result of a \"trim trailing\n>>> whitespace\" editor setting) based on whether the patch file uses\n>>> Windows-style line endings (CRLF) or Unix-style line endings (LF only).\n>\n> I believe this behavior was caused by:\n>\n> $ git log -1 b507b465f7831612b9d9fc643e3e5218b64e5bfa\n> commit b507b465f7831612b9d9fc643e3e5218b64e5bfa\n> Author: Linus Torvalds \n> Date: Thu Oct 19 19:26:08 2006 -0700\n>\n> git-apply: prepare for upcoming GNU diff -u format change.\n>\n> The latest GNU diff from CVS emits an empty line to express\n> an empty context line, instead of more traditional \"single\n> white space followed by a newline\". Do not get broken by it.\n>\n> Signed-off-by: Linus Torvalds \n> Signed-off-by: Junio C Hamano \n>\n> That code special-cased a line containing '\\n' but not a line\n> containing only '\\r\\n'.\n>\n> As to whether that's correct, personally I'd rather only special case\n> workaround important existing clients. Back in 2006, working with GNU\n> diff was incredibly important, and I'd say is still important today.\n> I can see Peregi's comment that this make line ending slightly\n> inconsistent, but I feel like the blank-line handling is a workaround\n> for an existing client we want to interoperate with and absent a\n> similar important client with mis-behaving '\\r\\n'-only lines, I\n> wouldn't be interested in adding support for it. But that's just my\n> off-the-cuff feeling and I don't feel strongly about it. Further, all\n> but one of my contributions above were mere header changes, so if\n> others have other opinions, they should probably be weighted more\n> heavily than mine on this topic.\n\nWhat would the patch(1) do?\n\nThe first column is the exit code of the subsequent command (0: success,\n1: one or more rejected lines, 2: failure):\n\n0 patch -p1 --dry-run original-unix.txt \ncharacter.\"\"\", it might be unfair to call it \"GNU's eccentricity\".\n\nBut I found that the POSIX text is fairly clear that the OP's\nCRLF-only line does not even qualify as an context line that is\ntotally empty.\nSo perhaps there is nothing to see here, and not just we and patch(1),\nbut GNU diff is doing the right thing?\n\n\n2024年10月29日(火) 7:07 René Scharfe :\n>\n> Am 28.10.24 um 19:39 schrieb Elijah Newren:\n> > On Mon, Oct 28, 2024 at 11:11 AM Taylor Blau wrote:\n> >>\n> >> On Mon, Oct 28, 2024 at 05:57:54PM +0100, Peregi Tamás wrote:\n> >>> Hi all,\n> >>>\n> >>> I might've found an inconsistency in how git-apply treats corrupt patches\n> >>> (representing empty context lines with completely empty lines instead of\n> >>> lines containing a single space - usually a result of a \"trim trailing\n> >>> whitespace\" editor setting) based on whether the patch file uses\n> >>> Windows-style line endings (CRLF) or Unix-style line endings (LF only).\n> >\n> > I believe this behavior was caused by:\n> >\n> > $ git log -1 b507b465f7831612b9d9fc643e3e5218b64e5bfa\n> > commit b507b465f7831612b9d9fc643e3e5218b64e5bfa\n> > Author: Linus Torvalds \n> > Date: Thu Oct 19 19:26:08 2006 -0700\n> >\n> > git-apply: prepare for upcoming GNU diff -u format change.\n> >\n> > The latest GNU diff from CVS emits an empty line to express\n> > an empty context line, instead of more traditional \"single\n> > white space followed by a newline\". Do not get broken by it.\n> >\n> > Signed-off-by: Linus Torvalds \n> > Signed-off-by: Junio C Hamano \n> >\n> > That code special-cased a line containing '\\n' but not a line\n> > containing only '\\r\\n'.\n> >\n> > As to whether that's correct, personally I'd rather only special case\n> > workaround important existing clients. Back in 2006, working with GNU\n> > diff was incredibly important, and I'd say is still important today.\n> > I can see Peregi's comment that this make line ending slightly\n> > inconsistent, but I feel like the blank-line handling is a workaround\n> > for an existing client we want to interoperate with and absent a\n> > similar important client with mis-behaving '\\r\\n'-only lines, I\n> > wouldn't be interested in adding support for it. But that's just my\n> > off-the-cuff feeling and I don't feel strongly about it. Further, all\n> > but one of my contributions above were mere header changes, so if\n> > others have other opinions, they should probably be weighted more\n> > heavily than mine on this topic.\n>\n> What would the patch(1) do?\n>\n> The first column is the exit code of the subsequent command (0: success,\n> 1: one or more rejected lines, 2: failure):\n>\n> 0 patch -p1 --dry-run original-unix.txt 0 patch -p1 --dry-run original-unix.txt 0 patch -p1 --dry-run original-win.txt 0 patch --ignore-whitespace -p1 --dry-run original-unix.txt 0 patch --ignore-whitespace -p1 --dry-run original-unix.txt 0 patch --ignore-whitespace -p1 --dry-run original-unix.txt 0 patch --ignore-whitespace -p1 --dry-run original-win.txt 0 patch --ignore-whitespace -p1 --dry-run original-win.txt 0 patch --ignore-whitespace -p1 --dry-run original-win.txt 1 patch -p1 --dry-run original-unix.txt 1 patch -p1 --dry-run original-win.txt 1 patch -p1 --dry-run original-win.txt 2 patch -p1 --dry-run original-unix.txt 2 patch -p1 --dry-run original-win.txt 2 patch --ignore-whitespace -p1 --dry-run original-unix.txt 2 patch --ignore-whitespace -p1 --dry-run original-win.txt \n> So basically the same as git apply?\n>\n>\n> What does current GNU diff do?\n>\n> diff -u <(printf 'a\\n\\n') <(printf 'b\\n\\n') | tail -1 | od -a\n> 0000000 sp nl\n> 0000002\n> diff -u --suppress-blank-empty <(printf 'a\\n\\n') <(printf 'b\\n\\n') | tail -1 | od -a\n> 0000000 nl\n> 0000001\n> diff -u <(printf 'a\\r\\n\\r\\n') <(printf 'b\\r\\n\\r\\n') | tail -1 | od -a\n> 0000000 sp cr nl\n> 0000003\n> diff -u --suppress-blank-empty <(printf 'a\\r\\n\\r\\n') <(printf 'b\\r\\n\\r\\n') | tail -1 | od -a\n> 0000000 sp cr nl\n> 0000003\n>\n> So it omits the space only if --suppress-blank-empty is given and the\n> blank line ends with \\n, not with \\r\\n.\n>\n>\n> Anyway, I agree with Elijah: The targeted support for GNU diff's\n> eccentricity is fine, and we're in good company with that. We could\n> remove it, since it doesn't seem to be the default (anymore?), but I\n> don't see much of a benefit. We could add support for blank context\n> lines that end in CRLF if there's a notable source of that kind of\n> that deviation from the original format.\n>\n> René\n>\n>\n"},{"id":"506291","messageId":"20241029163218.GA30259@tb-raspi4","threadId":"62419","inReplyTo":"CAExmRTbMCayh+vHW3g=2p=FMb=VKXomEbn7_0t+kK-eF79JnEA@mail.gmail.com","subject":"Re: Inconsistent handling of corrupt patches based on line endings","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2024-10-29T16:32:18Z","receivedAt":"2024-10-29T16:32:27Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Mon, Oct 28, 2024 at 05:57:54PM +0100, Peregi Tamás wrote:\n> Hi all,\n[]\n\nThanks for an extensive bug (?) report.\nI can not answer, if this a bug or working as designed.\n\nHowever, please allow a few words:\n>\n> While the example seems contrived, it has been encountered in the wild when\n> I was copying a port from https://github.com/microsoft/vcpkg (set up with\n> autocrlf=false and all files treated as binaries in .gitattributes) to my\n[]\n\nThat very repo is not in a shape I would like to see it.\nBasically, Git is designed to work with line endings using LF and nothing else.\nYou can configure Git to see CRLF in the working tree, if needed.\nBut all the diff machinery, and git-apply, is build on this assumption.\n\nIf I look into the repo, this very commit\n===========================\n commit ebdb41039450383ec7f41bd5f589cc46b7fd6a59\n Author: ...\n Date: Mon Feb 26 18:18:05 2018 -0800\n\n [everything] Use -text to ensure consistent files across machines.\n\n If you experience trouble, you can use the following to renormalize your local working directory:\n\n git add --renormalize .\n git reset .\n git checkout .\n===========================\nis probably made with good intentions.\nHowever, `git add --renormalize .` will not do anything at all.\nAll files are marked as \"-text\", so there is nothing to normalize.\nThe commit message here is simply wrong.\n\nSomething like this would have made more sense:\n===========================\n* text=auto eol=crlf\n\n# Declare files that will always have LF line endings on checkout.\n*.patch text eol=lf\nscripts/ci.baseline.txt text eol=lf\n\nports/** -linguist-detectable\n===========================\n(But I didn't check each and every file).\n\nAfter changing the .gitattributes file, run\ngit add --renormalize .\ngit add .\nwarning: in the working copy of 'crlf', LF will be replaced by CRLF the next time Git touches it\ngit commit -m \"all files with LF in the repo\"\n\ngit ls-files | xargs rm\ngit checkout .gitattributes\ngit checkout .\n\nAfter this, we can fine-tune:\ngit ls-files --eol | egrep -v \"json|cmake|patch\"\nshows that we can add lines like\n*.exe binary\nand may be\nscripts/vcpkg_completion* text eol=lf\nto .gitattributes\n\nDoes this makes sense ?\n\n"}]}