{"thread":{"id":"53846","subject":"Force git diff to create a binary patch?","startedAt":"2020-07-13T04:58:37Z","lastAt":"2020-07-14T15:35:59Z","messageCount":7,"participants":["Jason Xu","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"401472","messageId":"CAGPh-qNrjQ5xwbPcfJdBqrqOJQsVpa2hy-XHwN-8rhPPZe4Vpg@mail.gmail.com","threadId":"53846","inReplyTo":null,"subject":"Force git diff to create a binary patch?","fromName":"Jason Xu","fromEmail":"jasonx98@gmail.com","sentAt":"2020-07-13T04:58:23Z","receivedAt":"2020-07-13T04:58:37Z","isPatch":false,"sender":{"key":"jasonx98@gmail.com","avatar":null},"body":"Hello everyone,\n\nHere are two small files with non-ASCII data, yet git diff considers\nthem text instead of binary.\n\n echo -n -e '\\x01\\xff\\xdf' > bin1\n echo -n -e '\\x01\\xdf\\xff' > bin2\n git diff --binary bin1 bin2\n\nResult:\n\n diff --git a/bin1 b/bin2\n index 802dc8e..c39b638 100644\n --- a/bin1\n +++ b/bin2\n @@ -1 +1 @@\n -^A\n \\ No newline at end of file\n +^A\n \\ No newline at end of file\n\nI think `--binary` for `git diff` should force the creation of git\nbinary patches if there are any non-printable-ASCII characters, since\nmy understanding is that `--binary` is for safe encoding for email.\nOtherwise `-a` can be used.\n\nOriginal post: https://stackoverflow.com/questions/62858327/how-can-i-force-git-diff-to-create-a-git-binary-patch\n\nThanks\n"},{"id":"401487","messageId":"xmqqwo37o4db.fsf@gitster.c.googlers.com","threadId":"53846","inReplyTo":"CAGPh-qNrjQ5xwbPcfJdBqrqOJQsVpa2hy-XHwN-8rhPPZe4Vpg@mail.gmail.com","subject":"Re: Force git diff to create a binary patch?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-13T14:51:12Z","receivedAt":"2020-07-13T14:51:20Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jason Xu writes:\n\n> Here are two small files with non-ASCII data, yet git diff considers\n> them text instead of binary.\n\nBecause it is based on heuristics, without a NUL in that short file,\nthe file may not be judged to be binary.\n\n echo '* -diff' >.git/info/attributes\n\nwould tell Git that they need to be treated as binary junk not\nsuited for textual diff (you'd need to tweak the pattern to your\nneed).\n\n\n\n"},{"id":"401532","messageId":"CAGPh-qPyTNidqT=K-U8iYtG3udFZVFBhE1RyA1AZ1qEqN_NnZw@mail.gmail.com","threadId":"53846","inReplyTo":"20200713223906.GH8360@camp.crustytoothpaste.net","subject":"Re: Force git diff to create a binary patch?","fromName":"Jason Xu","fromEmail":"jasonx98@gmail.com","sentAt":"2020-07-14T04:09:41Z","receivedAt":"2020-07-14T04:09:56Z","isPatch":false,"sender":{"key":"jasonx98@gmail.com","avatar":null},"body":"So then it would not be possible to produce a diff that mixed binary\nand text diffs?\n\nIf `--binary` is for email patches, then it should still be easy to\ndetect non-email-safe characters (probably anything that isn't a\nprintable ASCII character).\n\nAlso if git's binary detection is really just checking for a NUL byte,\nthen it could be improved. Example: checking if bytes form any common\nencoding, like UTF-8, and if not then marking it as a binary. Or more\nadvanced `file` like heuristics. I don't know if this is worth the\neffort, or if others think the current binary detection is fine (it\nlikely works fine for actual executables, but not short binary data\nlike my example).\n\nOn Mon, Jul 13, 2020, 6:39 PM brian m. carlson\n wrote:\n>\n> On 2020-07-13 at 04:58:23, Jason Xu wrote:\n> > Hello everyone,\n> >\n> > Here are two small files with non-ASCII data, yet git diff considers\n> > them text instead of binary.\n> >\n> > echo -n -e '\\x01\\xff\\xdf' > bin1\n> > echo -n -e '\\x01\\xdf\\xff' > bin2\n> > git diff --binary bin1 bin2\n> >\n> > Result:\n> >\n> > diff --git a/bin1 b/bin2\n> > index 802dc8e..c39b638 100644\n> > --- a/bin1\n> > +++ b/bin2\n> > @@ -1 +1 @@\n> > -^A\n> > \\ No newline at end of file\n> > +^A\n> > \\ No newline at end of file\n> >\n> > I think `--binary` for `git diff` should force the creation of git\n> > binary patches if there are any non-printable-ASCII characters, since\n> > my understanding is that `--binary` is for safe encoding for email.\n> > Otherwise `-a` can be used.\n>\n> I was the one that asked you to post this here. While Junio's comment\n> explains _why_ this happens, I think that --binary should, well, produce\n> a binary patch, regardless of whether the file is text or binary. After\n> all, --text does the opposite.\n>\n> The description for the --binary option reads as follows:\n>\n> In addition to --full-index, output a binary diff that can be applied\n> with git-apply. Implies --patch.\n>\n> So we need to fix either the documentation or the code.\n>\n> I looked into what it takes to fix the code to do this; it's fairly\n> straightforward, but it does cause some testsuite failures which assume\n> the current behavior and will likely involve a small series. So if\n> other folks agree, I'm happy to pick this up in the next couple of weeks\n> and add support for it that would hopefully hit Git 2.29.\n> --\n> brian m. carlson: Houston, Texas, US\n"},{"id":"401536","messageId":"xmqq5zaqn29t.fsf@gitster.c.googlers.com","threadId":"53846","inReplyTo":"CAGPh-qPyTNidqT=K-U8iYtG3udFZVFBhE1RyA1AZ1qEqN_NnZw@mail.gmail.com","subject":"Re: Force git diff to create a binary patch?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-14T04:34:06Z","receivedAt":"2020-07-14T04:34:10Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jason Xu writes:\n\n>> The description for the --binary option reads as follows:\n>>\n>> In addition to --full-index, output a binary diff that can be applied\n>> with git-apply. Implies --patch.\n>>\n>> So we need to fix either the documentation or the code.\n>>\n>> I looked into what it takes to fix the code to do this; it's fairly\n>> straightforward, but it does cause some testsuite failures which assume\n>> the current behavior and will likely involve a small series. So if\n>> other folks agree, I'm happy to pick this up in the next couple of weeks\n>> and add support for it that would hopefully hit Git 2.29.\n\nI am not convinced. \n\nThe \"--binary\" option was invented as a way to tell Git to produce\nsomething that can be applied, where Git stopped at saying \"binary\nfiles differ\". So a commit that touches two paths, one text and the\nother binary, used to produce a textual patch for one and a useless\n\"binary files differ\" for the other in \"git show\". Such a commit\ncan be made more useful with \"git show --binary\" to tell the former\nto still produce textual and readable patch while showing the xdelta\nbased \"binary patch\" Git invented.\n\nSo, no, I am less convinced \"--binary\" that forces \"all paths are\nbinary, so show binary patch\" is a good idea.\n\nAnd viewed with the knowledge of that history,\n\n>> In addition to --full-index, output a binary diff that can be applied\n>> with git-apply. Implies --patch.\n\nthis description is correct---the choice is not between \"showing a\nbinary diff and showing a useless textual diff\"; the choice is\nbetween showing \"binary files differ\" and appliable \"binary patch\".\n\n"},{"id":"401535","messageId":"xmqq4kqan290.fsf@gitster.c.googlers.com","threadId":"53846","inReplyTo":"CAGPh-qPyTNidqT=K-U8iYtG3udFZVFBhE1RyA1AZ1qEqN_NnZw@mail.gmail.com","subject":"Re: Force git diff to create a binary patch?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-14T04:34:35Z","receivedAt":"2020-07-14T04:34:42Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jason Xu writes:\n\n>> The description for the --binary option reads as follows:\n>>\n>> In addition to --full-index, output a binary diff that can be applied\n>> with git-apply. Implies --patch.\n>>\n>> So we need to fix either the documentation or the code.\n>>\n>> I looked into what it takes to fix the code to do this; it's fairly\n>> straightforward, but it does cause some testsuite failures which assume\n>> the current behavior and will likely involve a small series. So if\n>> other folks agree, I'm happy to pick this up in the next couple of weeks\n>> and add support for it that would hopefully hit Git 2.29.\n\nI am not convinced. \n\nThe \"--binary\" option was invented as a way to tell Git to produce\nsomething that can be applied, where Git stopped at saying \"binary\nfiles differ\". So a commit that touches two paths, one text and the\nother binary, used to produce a textual patch for one and a useless\n\"binary files differ\" for the other in \"git show\". Such a commit\ncan be made more useful with \"git show --binary\" to tell the former\nto still produce textual and readable patch while showing the xdelta\nbased \"binary patch\" Git invented.\n\nSo, no, I am less convinced \"--binary\" that forces \"all paths are\nbinary, so show binary patch\" is a good idea.\n\nAnd viewed with the knowledge of that history,\n\n>> In addition to --full-index, output a binary diff that can be applied\n>> with git-apply. Implies --patch.\n\nthis description is correct---the choice is not between \"showing a\nbinary diff and showing a useless textual diff\"; the choice is\nbetween showing \"binary files differ\" and appliable \"binary patch\".\n"},{"id":"401537","messageId":"CAGPh-qPH+gGYHCy+rQJvzwr-Vp71UJxwr+w-G9GaV_o40K8P-Q@mail.gmail.com","threadId":"53846","inReplyTo":"xmqq4kqan290.fsf@gitster.c.googlers.com","subject":"Re: Force git diff to create a binary patch?","fromName":"Jason Xu","fromEmail":"jasonx98@gmail.com","sentAt":"2020-07-14T04:56:07Z","receivedAt":"2020-07-14T04:56:21Z","isPatch":false,"sender":{"key":"jasonx98@gmail.com","avatar":null},"body":"> The \"--binary\" option was invented as a way to tell Git to produce something that can be applied, where Git stopped at saying \"binary files differ\".\n\nDoesn't `--text` already do that? Albeit with whitespace warnings.\n\n> So, no, I am less convinced \"--binary\" that forces \"all paths are binary, so show binary patch\" is a good idea.\n\nI'm not sure exactly what this means, but if one binary and one text\nfile changed, then there should be a binary diff and a text diff, like\nthere is currently.\nThat's why I propose better binary file detection, instead of (what I\nunderstand to be) \"make all patches in a patch file GIT binary\npatches, regardless if a file is text\". But there is merit to that if\nyou don't care about readability in a \"fully binary patch\", or you\ninsist on not mixing text and binary in patches.\n\nThe terminology is confusing because within a file produced by diffing\ntwo directories, there are multiple diff sections for each file\ndiffed. So while the documentation says \"output a binary diff\", to be\nclearer it should say \"output a binary diff only for binary files\".\n\nOn Tue, Jul 14, 2020 at 12:34 AM Junio C Hamano wrote:\n>\n> Jason Xu writes:\n>\n> >> The description for the --binary option reads as follows:\n> >>\n> >> In addition to --full-index, output a binary diff that can be applied\n> >> with git-apply. Implies --patch.\n> >>\n> >> So we need to fix either the documentation or the code.\n> >>\n> >> I looked into what it takes to fix the code to do this; it's fairly\n> >> straightforward, but it does cause some testsuite failures which assume\n> >> the current behavior and will likely involve a small series. So if\n> >> other folks agree, I'm happy to pick this up in the next couple of weeks\n> >> and add support for it that would hopefully hit Git 2.29.\n>\n> I am not convinced.\n>\n> The \"--binary\" option was invented as a way to tell Git to produce\n> something that can be applied, where Git stopped at saying \"binary\n> files differ\". So a commit that touches two paths, one text and the\n> other binary, used to produce a textual patch for one and a useless\n> \"binary files differ\" for the other in \"git show\". Such a commit\n> can be made more useful with \"git show --binary\" to tell the former\n> to still produce textual and readable patch while showing the xdelta\n> based \"binary patch\" Git invented.\n>\n> So, no, I am less convinced \"--binary\" that forces \"all paths are\n> binary, so show binary patch\" is a good idea.\n>\n> And viewed with the knowledge of that history,\n>\n> >> In addition to --full-index, output a binary diff that can be applied\n> >> with git-apply. Implies --patch.\n>\n> this description is correct---the choice is not between \"showing a\n> binary diff and showing a useless textual diff\"; the choice is\n> between showing \"binary files differ\" and appliable \"binary patch\".\n"},{"id":"401552","messageId":"xmqqv9iqkt2d.fsf@gitster.c.googlers.com","threadId":"53846","inReplyTo":"CAGPh-qPH+gGYHCy+rQJvzwr-Vp71UJxwr+w-G9GaV_o40K8P-Q@mail.gmail.com","subject":"Re: Force git diff to create a binary patch?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-14T15:35:54Z","receivedAt":"2020-07-14T15:35:59Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jason Xu writes:\n\n>> The \"--binary\" option was invented as a way to tell Git to\n>> produce something that can be applied, where Git stopped at\n>> saying \"binary files differ\".\n>\n> Doesn't `--text` already do that? Albeit with whitespace warnings.\n\nI do not think so. --text is about forcing everything to be treated\ntext, so you'll not see binary patches but the patch-looking thing\nwith binary garbage you wrote in the message that started this\nthread. IOW, that is the opposite of what you want. And I agree\nthat such a \"diff -a\" output is prone to corruption during transfer\n(e.g. over e-mail) and a way to tell exactly which paths should be\nshown as binary patches is a good thing to have.\n\n> That's why I propose better binary file detection, instead of (what I\n> understand to be) \"make all patches in a patch file GIT binary\n> patches, regardless if a file is text\".\n\nOh, I agree that \"treat everything as binary and produce binary\npatch for all paths\" is a nonsense option no sane person would want\nto use. The users are better off exchanging bundles at that point,\nas such a binary-only patches are unreadable anyway.\n\nThere is no \"better detection\"; the only thing you could get is\n\"better heuristics\" and there will always be a limit to what\nheuristics can do for you. \n\nAnd that is why you were given gitattributes very early in the\ndiscussion ;-) That is the mechanism to tell exactly which paths\nshould be treated as binary.\n"}]}