{"thread":{"id":"56280","subject":"git clone corrupts file.","startedAt":"2021-08-13T18:55:10Z","lastAt":"2021-08-16T22:26:52Z","messageCount":13,"participants":["Russell, Scott","brian m. carlson","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"432699","messageId":"BN6PR15MB14264C9A96F4B5F6B01FA7FACBFA9@BN6PR15MB1426.namprd15.prod.outlook.com","threadId":"56280","inReplyTo":"BN6PR15MB14261C40E614CC11416388B4CBFA9@BN6PR15MB1426.namprd15.prod.outlook.com","subject":"git clone corrupts file.","fromName":"Russell, Scott","fromEmail":"scott.russell2@ncr.com","sentAt":"2021-08-13T18:54:43Z","receivedAt":"2021-08-13T18:55:10Z","isPatch":false,"sender":{"key":"scott.russell2@ncr.com","avatar":null},"body":"What did you do before the bug happened? git clone\nWhat did you expect to happen? file cloned matches github copy\n\nWhat happened instead? file corrupted, does not match github copy  see example\nWhat's different between what you expected and what actually happened?  corruption\n\n\n[System Info]\n\ngit version:\ngit version 2.31.1.windows.1\n\ncpu: x86_64\n\nsizeof-long: 4\n\nsizeof-size_t: 8\n\nshell-path: /bin/sh\nfeature: \nfsmonitor--daemon\nuname: Windows 10.0 17134 \n\ncompiler info: gnuc: 10.2\n\nlibc info: no libc information available\n\n$SHELL (typically, interactive shell): \n\n[Enabled Hooks]\nnot run from a git repository - no hooks to show\n\nFile from git. \n\n਍⼀⼀ 䴀椀挀爀漀猀漀昀琀 嘀椀猀甀愀氀 䌀⬀⬀ 最攀渀攀爀愀琀攀搀 椀渀挀氀甀搀攀 昀椀氀攀⸀ഀഀ\n// Used by CamTest.rc\n਍⼀⼀ഀഀ\n#define IDM_ABOUTBOX                    0x0010\n਍⌀搀攀昀椀渀攀 䤀䐀䐀开䄀䈀伀唀吀䈀伀堀                    ㄀　　ഀഀ\n\nFile in github.  \n\n//{{NO_DEPENDENCIES}}\n// Microsoft Visual C++ generated include file.\n// Used by CamTest.rc\n//\n\n\nThanks, \n\nScott Russell\nStaff SW Engineer \nNCR Corporation \nPhone: +17706237512\nmailto:Scott.Russell2@ncr.com  |  http://www.ncr.com/\n       \n\n"},{"id":"432718","messageId":"YRbya0UO2+PvOjL5@camp.crustytoothpaste.net","threadId":"56280","inReplyTo":"BN6PR15MB14264C9A96F4B5F6B01FA7FACBFA9@BN6PR15MB1426.namprd15.prod.outlook.com","subject":"Re: git clone corrupts file.","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-08-13T22:30:03Z","receivedAt":"2021-08-13T22:30:10Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-08-13 at 18:54:43, Russell, Scott wrote:\n> File from git.\n> \n> ਍⼀⼀ 䴀椀挀爀漀猀漀昀琀 嘀椀猀甀愀氀 䌀⬀⬀ 最攀渀攀爀愀琀攀搀 椀渀挀氀甀搀攀 昀椀氀攀⸀ഀഀ\n> // Used by CamTest.rc\n> ਍⼀⼀ഀഀ\n> #define IDM_ABOUTBOX                    0x0010\n> ਍⌀搀攀昀椀渀攀 䤀䐀䐀开䄀䈀伀唀吀䈀伀堀                    ㄀　　ഀഀ\n> \n> File in github. \n> \n> //{{NO_DEPENDENCIES}}\n> // Microsoft Visual C++ generated include file.\n> // Used by CamTest.rc\n> //\n\nWe're probably going to need a little more information about this. My\nguess as to what's happening here is that the editor you're using to\nview the file is set to read files as UTF-16, but the repository has\nthem stored in UTF-8, or (less likely) vice versa.\n\nCan you tell us what editor or other tool you're using to view the file\nand what settings it's using for text encoding? Can you tell us about\nany working-tree-encoding declarations in your .gitattributes? You can\nuse \"git check-attr -a PATH\" to see more information about that.\n\nWhat code page are you using on your system? Are you using PowerShell,\nCMD, or Git Bash? If you're using Git Bash, what are your locale\nsettings?\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"432810","messageId":"BN6PR15MB1426E99386269CCBDA888D51CBFD9@BN6PR15MB1426.namprd15.prod.outlook.com","threadId":"56280","inReplyTo":"YRbya0UO2+PvOjL5@camp.crustytoothpaste.net","subject":"RE: git clone corrupts file.","fromName":"Russell, Scott","fromEmail":"scott.russell2@ncr.com","sentAt":"2021-08-16T15:24:28Z","receivedAt":"2021-08-16T16:27:41Z","isPatch":false,"sender":{"key":"scott.russell2@ncr.com","avatar":null},"body":"Brian, \n\nThanks for your interest in this issue. The issue has been determined to have 2 factors. \n\n1. The files corrupted are in Unicode. Though the .h file mentioned certainly doesn't have to be Unicode, it can be ANSI, we have other files that must be Unicode. We use Unicode in quite a number of our text files.\n2. Git appears to corrupt the file by making line endings changes. \n a. Github has the correct file. It views correct there. When downloaded as a binary or text from Github in a browser, it is not corrupted. \n b. Git seems to change line endings as if the file were ANSI or single byte encoding, not Unicode. \n c. Git has the setting git config core.autocrlf false. But apparently, it is not being observed. \n d. The .gitconfig file has the [core] section with the entry autocrlf = false following the section. \n e. There is a .gitattributes file in the repo. \n f. Entries in .gitattributes specified by type are specified for the affected files. \n *.h text eol=crlf\n *.ini text eol=crlf\n\nIf you look at the 1st line of the binary view of the original file, it looks like this:\n\nFF FE 2F 00 2F 00 7B 00 7B 00 4E 00 4F 00 5F 00\n44 00 45 00 50 00 45 00 4E 00 44 00 45 00 4E 00 \n43 00 49 00 45 00 53 00 7D 00 7D 00 0D 00 0A 00 \tNote - Unicode CR LF 0D 00 0A 00 \n\n2nd line \n2F 00 2F 00 20 00 4D 00 69 00 63 00 72 00 6F 00 etc. \n\nIf you look at the git file, it looks very similar. \nHowever, git has put a non Unicode CF LF into the end of line. \nPlus an extra NULL. This extra NULL throws the 2 byte Unicode encoding off. It corrupts the line. On the next line, the extra NULL lines up the 2 byte encoding, so that line appears uncorrupted. \nYou can see that in my original email below. Every other line is not readable. \n\nFF FE 2F 00 2F 00 7B 00 7B 00 4E 00 4F 00 5F 00\n44 00 45 00 50 00 45 00 4E 00 44 00 45 00 4E 00 \n43 00 49 00 45 00 53 00 7D 00 7D 00 0D 00 0D 0A0 \tNote - Unicode CR LF 0D 00 0A 00 \n\n2nd line \n00 2F 00 2F 00 20 00 4D 00 69 00 63 00 72 00 6F etc. \n\nI would like git to observe the autocrlf false as directed. \n\nIt's important that we retain 2 byte Unicode file encoding in many of our files. And that git not add single byte CR LF into our 2 byte files. \nWe can't convert the files to other encoding for convenience of git. \n\nThanks, \n\nScott Russell\nStaff SW Engineer \nNCR Corporation \nPhone: +17706237512\nScott.Russell2@ncr.com | ncr.com\n \n\n-----Original Message-----\nFrom: brian m. carlson \nSent: Friday, August 13, 2021 6:30 PM\nTo: Russell, Scott \nCc: git@vger.kernel.org\nSubject: Re: git clone corrupts file.\n\n*External Message* - Use caution before opening links or attachments\n\nOn 2021-08-13 at 18:54:43, Russell, Scott wrote:\n> File from git.\n> \n> ਍⼀⼀ 䴀椀挀爀漀猀漀昀琀 嘀椀猀甀愀氀 䌀⬀⬀ 最攀渀攀爀愀琀攀搀 椀渀挀氀甀搀攀 昀椀氀攀⸀ഀഀ\n> // Used by CamTest.rc\n> ਍⼀⼀ഀഀ\n> #define IDM_ABOUTBOX                    0x0010\n> ਍⌀搀攀昀椀渀攀 䤀䐀䐀开䄀䈀伀唀吀䈀伀堀                    ㄀　　ഀഀ\n> \n> File in github.\n> \n> //{{NO_DEPENDENCIES}}\n> // Microsoft Visual C++ generated include file.\n> // Used by CamTest.rc\n> //\n\nWe're probably going to need a little more information about this. My guess as to what's happening here is that the editor you're using to view the file is set to read files as UTF-16, but the repository has them stored in UTF-8, or (less likely) vice versa.\n\nCan you tell us what editor or other tool you're using to view the file and what settings it's using for text encoding? Can you tell us about any working-tree-encoding declarations in your .gitattributes? You can use \"git check-attr -a PATH\" to see more information about that.\n\nWhat code page are you using on your system? Are you using PowerShell, CMD, or Git Bash? If you're using Git Bash, what are your locale settings?\n--\nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"432814","messageId":"YRqYD+SszvhhySwl@coredump.intra.peff.net","threadId":"56280","inReplyTo":"BN6PR15MB1426E99386269CCBDA888D51CBFD9@BN6PR15MB1426.namprd15.prod.outlook.com","subject":"Re: git clone corrupts file.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-16T16:53:35Z","receivedAt":"2021-08-16T16:53:38Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 16, 2021 at 03:24:28PM +0000, Russell, Scott wrote:\n\n> 1. The files corrupted are in Unicode. Though the .h file mentioned\n> certainly doesn't have to be Unicode, it can be ANSI, we have\n> other files that must be Unicode. We use Unicode in quite a\n> number of our text files.\n\nBy Unicode, I'll assume you mean UTF-16, since your example below\nappears to have a BOM marker at the beginning (FF FE).\n\nUnlike UTF-8, UTF-16 is not a superset of ASCII, and thus can't be\ntreated as \"text\" by Git (e.g., the line ending byte is no longer just\nhex \"0A\", but \"00 0A\").\n\n> f. Entries in .gitattributes specified by type are specified for the affected files. \n> *.h text eol=crlf\n> *.ini text eol=crlf\n\nSo this is your problem. The \"text\" attribute is telling Git to treat\nthe file as text (which will handle any ASCII-superset encoding like\nUTF-8, ISO8859-1, etc, but not others like UTF-16, UTF-32, EUC-JP, etc).\n\nDepending on what's in your repo and what you want to have happen,\nyou'll want to:\n\n - remove that attribute, if all of your \".h\" files are UTF-16\n\n - if only some are UTF-16, you'll need to provide patterns that\n distinguish between the two types by giving them different\n attributes (e.g., \"-text\" should override for specific files)\n\n - you can stop there if you don't need line-ending conversion for\n UTF-16 files (and there may be little point; Git will treat them as\n binary for the purposes of diffing, so there is little point in\n matching the canonical in-repo endings)\n\n - if you do want to do line ending conversion (or any other\n modifications on them), you can do so with a custom clean/smudge\n filter (see the \"filter\" attribute in \"git help attributes\")\n\n> I would like git to observe the autocrlf false as directed.\n\nHopefully the above explains it, but just to be clear, this isn't\nautocrlf kicking in, but rather the \"text\" and \"eol\" attributes you've\nspecified.\n\n> We can't convert the files to other encoding for convenience of git.\n\nIf you're happy enough not being able to get meaningful text diffs for\nthese files from Git, then the above should make your problem go away.\n\nBut an alternative workflow, if you really want UTF-16 in the working\ntree, is to convert between UTF-8 and UTF-16 as the files go in and out\no the working tree. There's no built-in support for that, but you could\ndo it with a custom clean/smudge filter. That would let Git store UTF-8\ninternally, do diffs, etc.\n\nOne lighter alternative to that is to actually store UTF-16 in the\nrepository as you are now, but provide a textconv filter (see diff\nattributes in \"git help attributes\") to convert it to UTF-8 on the fly\nwhen showing a diff. You won't be able to apply such a diff, but they're\nuseful for human eyes.\n\n-Peff\n"},{"id":"432823","messageId":"BN6PR15MB1426D70E338F6B2A988565A9CBFD9@BN6PR15MB1426.namprd15.prod.outlook.com","threadId":"56280","inReplyTo":"YRqYD+SszvhhySwl@coredump.intra.peff.net","subject":"RE: git clone corrupts file.","fromName":"Russell, Scott","fromEmail":"scott.russell2@ncr.com","sentAt":"2021-08-16T17:39:12Z","receivedAt":"2021-08-16T18:27:33Z","isPatch":false,"sender":{"key":"scott.russell2@ncr.com","avatar":null},"body":"Jeff, \n\nThanks for your reply. \nWe don't want any EOL handling of any file. That's why we specify autocrlf false. \n\nWe would like git to not any cr lf conversion on any file. Whether they be ANSI or Unicode. They had the right endings when we checked them in. \nWe don't want them adjusted.\n\nDoes removing the eol = cr lf fix that? \n\n\tYou said: UTF-16 ... can't be treated as \"text\" by Git.\n\nWe can't make any changes to the files to suit git. We just need git to store and retrieve files as committed. \n\nWill removing the \n\n\teol=cr lf \n\nfrom the line \n\n*.ini text \n\nfrom the attributes file stop the issue? \n\nIf not, does .gitattributes allow a path? Such that we could say \n\n\\config\\Language Specific\\* type - If these are Unicode, what would we say here. Can it not be text? Then binary? \n*.ini\ttext \n\n\nThanks, \n\nScott Russell\nStaff SW Engineer \nNCR Corporation \nPhone: +17706237512\nScott.Russell2@ncr.com | ncr.com\n \n\n-----Original Message-----\nFrom: Jeff King \nSent: Monday, August 16, 2021 12:54 PM\nTo: Russell, Scott \nCc: brian m. carlson ; git@vger.kernel.org\nSubject: Re: git clone corrupts file.\n\n*External Message* - Use caution before opening links or attachments\n\nOn Mon, Aug 16, 2021 at 03:24:28PM +0000, Russell, Scott wrote:\n\n> 1. The files corrupted are in Unicode. Though the .h file mentioned\n> certainly doesn't have to be Unicode, it can be ANSI, we have\n> other files that must be Unicode. We use Unicode in quite a\n> number of our text files.\n\nBy Unicode, I'll assume you mean UTF-16, since your example below appears to have a BOM marker at the beginning (FF FE).\n\nUnlike UTF-8, UTF-16 is not a superset of ASCII, and thus can't be treated as \"text\" by Git (e.g., the line ending byte is no longer just hex \"0A\", but \"00 0A\").\n\n> f. Entries in .gitattributes specified by type are specified for the affected files. \n> *.h text eol=crlf\n> *.ini text eol=crlf\n\nSo this is your problem. The \"text\" attribute is telling Git to treat the file as text (which will handle any ASCII-superset encoding like UTF-8, ISO8859-1, etc, but not others like UTF-16, UTF-32, EUC-JP, etc).\n\nDepending on what's in your repo and what you want to have happen, you'll want to:\n\n - remove that attribute, if all of your \".h\" files are UTF-16\n\n - if only some are UTF-16, you'll need to provide patterns that\n distinguish between the two types by giving them different\n attributes (e.g., \"-text\" should override for specific files)\n\n - you can stop there if you don't need line-ending conversion for\n UTF-16 files (and there may be little point; Git will treat them as\n binary for the purposes of diffing, so there is little point in\n matching the canonical in-repo endings)\n\n - if you do want to do line ending conversion (or any other\n modifications on them), you can do so with a custom clean/smudge\n filter (see the \"filter\" attribute in \"git help attributes\")\n\n> I would like git to observe the autocrlf false as directed.\n\nHopefully the above explains it, but just to be clear, this isn't autocrlf kicking in, but rather the \"text\" and \"eol\" attributes you've specified.\n\n> We can't convert the files to other encoding for convenience of git.\n\nIf you're happy enough not being able to get meaningful text diffs for these files from Git, then the above should make your problem go away.\n\nBut an alternative workflow, if you really want UTF-16 in the working tree, is to convert between UTF-8 and UTF-16 as the files go in and out o the working tree. There's no built-in support for that, but you could do it with a custom clean/smudge filter. That would let Git store UTF-8 internally, do diffs, etc.\n\nOne lighter alternative to that is to actually store UTF-16 in the repository as you are now, but provide a textconv filter (see diff attributes in \"git help attributes\") to convert it to UTF-8 on the fly when showing a diff. You won't be able to apply such a diff, but they're useful for human eyes.\n\n-Peff\n"},{"id":"432827","messageId":"YRqzM1D6w4bDYjKY@coredump.intra.peff.net","threadId":"56280","inReplyTo":"BN6PR15MB1426D70E338F6B2A988565A9CBFD9@BN6PR15MB1426.namprd15.prod.outlook.com","subject":"Re: git clone corrupts file.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-16T18:49:23Z","receivedAt":"2021-08-16T18:49:26Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 16, 2021 at 05:39:12PM +0000, Russell, Scott wrote:\n\n> We don't want any EOL handling of any file. That's why we specify autocrlf false.\n\nRight, but it's not the whole story. autocrlf is an older and broader\nmechanism for doing line-ending conversion. From its documentation in\n\"git help config\":\n\n core.autocrlf\n Setting this variable to \"true\" is the same as setting the text\n attribute to \"auto\" on all files and core.eol to \"crlf\".[...]\n\nYou obviously don't want that, but you _also_ don't want to set the text\nand eol attributes on individual paths, either.\n\n> We would like git to not any cr lf conversion on any file. Whether\n> they be ANSI or Unicode. They had the right endings when we checked\n> them in.\n> We don't want them adjusted.\n> \n> Does removing the eol = cr lf fix that?\n\nThat might be sufficient. You may also need to drop \"text\", as well.\nOtherwise core.eol will kick in and do conversions. (Sorry, I don't use\nWindows and it has been a long time since I looked into these options,\nso you may have to do some experimenting).\n\n> \tYou said: UTF-16 ... can't be treated as \"text\" by Git.\n> \n> We can't make any changes to the files to suit git. We just need git to store and retrieve files as committed.\n\nRight. That's what it does by default (if you don't set any .gitattributes).\n\nWhat I mean by \"can't be treated as text\" is that Git will not correctly\nimplement text features like CRLF conversion, nor diffs, for such an\nencoding. It is effectively a binary file from Git's perspective.\n\n> Will removing the \n> \n> \teol=cr lf \n> \n> from the line \n> \n> *.ini text \n> \n> from the attributes file stop the issue?\n> \n> If not, does .gitattributes allow a path? Such that we could say \n> \n> \\config\\Language Specific\\* type - If these are Unicode, what would we say here. Can it not be text? Then binary? \n> *.ini\ttext\n\nIf you simply drop the attributes entirely, Git will use its\nauto-detection to determine whether a file is binary, which looks for\nNULs (and UTF-16 files are generally full of them). So I suspect that\nwould do it. You can also provide the \"-text\" attribute to override that\nand make sure no line-ending conversion is done.\n\nIf you want to override a specific file, then yes, you can provide paths\n(I don't recall offhand whether we allow backslashes in the patterns;\nyou may need to use forward slashes). You can also put the pattern \"*\"\nin the \"config/Language Specific/.gitattributes\" to have it apply only\nto that directory (and ones below it).\n\nThe patterns are the same as those in .gitignore files; see the section\n\"PATTERN FORMAT\" in \"git help ignore\".\n\n-Peff\n"},{"id":"432828","messageId":"YRqzmC5ubd0TEWL/@coredump.intra.peff.net","threadId":"56280","inReplyTo":"YRqYD+SszvhhySwl@coredump.intra.peff.net","subject":"Re: git clone corrupts file.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-08-16T18:51:04Z","receivedAt":"2021-08-16T18:51:07Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 16, 2021 at 12:53:36PM -0400, Jeff King wrote:\n\n> But an alternative workflow, if you really want UTF-16 in the working\n> tree, is to convert between UTF-8 and UTF-16 as the files go in and out\n> o the working tree. There's no built-in support for that, but you could\n> do it with a custom clean/smudge filter. That would let Git store UTF-8\n> internally, do diffs, etc.\n\nOh, by the way, I totally forgot that we added an internal version of\nthis, which is easier to configure and much more efficient. See the\n\"working-tree-encoding\" attribute in \"git help attributes\".\n\nJust in case you do want to go that route.\n\n-Peff\n"},{"id":"432829","messageId":"BN6PR15MB14261548DE31763B34CEEDD5CBFD9@BN6PR15MB1426.namprd15.prod.outlook.com","threadId":"56280","inReplyTo":"YRqzM1D6w4bDYjKY@coredump.intra.peff.net","subject":"RE: git clone corrupts file.","fromName":"Russell, Scott","fromEmail":"scott.russell2@ncr.com","sentAt":"2021-08-16T18:52:30Z","receivedAt":"2021-08-16T18:52:49Z","isPatch":false,"sender":{"key":"scott.russell2@ncr.com","avatar":null},"body":"Jeff, \n\nThanks for your response. I will try these suggestions. I suspect I can come to some solution. \n\n\nThanks, \n\nScott Russell\nStaff SW Engineer \nNCR Corporation \nPhone: +17706237512\nScott.Russell2@ncr.com | ncr.com\n \n\n-----Original Message-----\nFrom: Jeff King \nSent: Monday, August 16, 2021 2:49 PM\nTo: Russell, Scott \nCc: brian m. carlson ; git@vger.kernel.org\nSubject: Re: git clone corrupts file.\n\n*External Message* - Use caution before opening links or attachments\n\nOn Mon, Aug 16, 2021 at 05:39:12PM +0000, Russell, Scott wrote:\n\n> We don't want any EOL handling of any file. That's why we specify autocrlf false.\n\nRight, but it's not the whole story. autocrlf is an older and broader mechanism for doing line-ending conversion. From its documentation in \"git help config\":\n\n core.autocrlf\n Setting this variable to \"true\" is the same as setting the text\n attribute to \"auto\" on all files and core.eol to \"crlf\".[...]\n\nYou obviously don't want that, but you _also_ don't want to set the text and eol attributes on individual paths, either.\n\n> We would like git to not any cr lf conversion on any file. Whether\n> they be ANSI or Unicode. They had the right endings when we checked\n> them in.\n> We don't want them adjusted.\n> \n> Does removing the eol = cr lf fix that?\n\nThat might be sufficient. You may also need to drop \"text\", as well.\nOtherwise core.eol will kick in and do conversions. (Sorry, I don't use Windows and it has been a long time since I looked into these options, so you may have to do some experimenting).\n\n> \tYou said: UTF-16 ... can't be treated as \"text\" by Git.\n> \n> We can't make any changes to the files to suit git. We just need git to store and retrieve files as committed.\n\nRight. That's what it does by default (if you don't set any .gitattributes).\n\nWhat I mean by \"can't be treated as text\" is that Git will not correctly implement text features like CRLF conversion, nor diffs, for such an encoding. It is effectively a binary file from Git's perspective.\n\n> Will removing the\n> \n> \teol=cr lf\n> \n> from the line\n> \n> *.ini text\n> \n> from the attributes file stop the issue?\n> \n> If not, does .gitattributes allow a path? Such that we could say\n> \n> \\config\\Language Specific\\* type - If these are Unicode, what would we say here. Can it not be text? Then binary? \n> *.ini\ttext\n\nIf you simply drop the attributes entirely, Git will use its auto-detection to determine whether a file is binary, which looks for NULs (and UTF-16 files are generally full of them). So I suspect that would do it. You can also provide the \"-text\" attribute to override that and make sure no line-ending conversion is done.\n\nIf you want to override a specific file, then yes, you can provide paths (I don't recall offhand whether we allow backslashes in the patterns; you may need to use forward slashes). You can also put the pattern \"*\"\nin the \"config/Language Specific/.gitattributes\" to have it apply only to that directory (and ones below it).\n\nThe patterns are the same as those in .gitignore files; see the section \"PATTERN FORMAT\" in \"git help ignore\".\n\n-Peff\n"},{"id":"432831","messageId":"BN6PR15MB14268392DBC2F0C210EBE803CBFD9@BN6PR15MB1426.namprd15.prod.outlook.com","threadId":"56280","inReplyTo":"YRqzmC5ubd0TEWL/@coredump.intra.peff.net","subject":"RE: git clone corrupts file.","fromName":"Russell, Scott","fromEmail":"scott.russell2@ncr.com","sentAt":"2021-08-16T18:53:14Z","receivedAt":"2021-08-16T18:53:26Z","isPatch":false,"sender":{"key":"scott.russell2@ncr.com","avatar":null},"body":"Okay, thanks. I will look for that. \n\n\nThanks, \n\nScott Russell\nStaff SW Engineer \nNCR Corporation \nPhone: +17706237512\nScott.Russell2@ncr.com | ncr.com\n \n\n-----Original Message-----\nFrom: Jeff King \nSent: Monday, August 16, 2021 2:51 PM\nTo: Russell, Scott \nCc: brian m. carlson ; git@vger.kernel.org\nSubject: Re: git clone corrupts file.\n\n*External Message* - Use caution before opening links or attachments\n\nOn Mon, Aug 16, 2021 at 12:53:36PM -0400, Jeff King wrote:\n\n> But an alternative workflow, if you really want UTF-16 in the working \n> tree, is to convert between UTF-8 and UTF-16 as the files go in and \n> out o the working tree. There's no built-in support for that, but you \n> could do it with a custom clean/smudge filter. That would let Git \n> store UTF-8 internally, do diffs, etc.\n\nOh, by the way, I totally forgot that we added an internal version of this, which is easier to configure and much more efficient. See the \"working-tree-encoding\" attribute in \"git help attributes\".\n\nJust in case you do want to go that route.\n\n-Peff\n"},{"id":"432880","messageId":"YRrdq5VAp3o35+Fb@camp.crustytoothpaste.net","threadId":"56280","inReplyTo":"YRqzmC5ubd0TEWL/@coredump.intra.peff.net","subject":"Re: git clone corrupts file.","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-08-16T21:50:35Z","receivedAt":"2021-08-16T21:51:12Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-08-16 at 18:51:04, Jeff King wrote:\n> On Mon, Aug 16, 2021 at 12:53:36PM -0400, Jeff King wrote:\n> \n> > But an alternative workflow, if you really want UTF-16 in the working\n> > tree, is to convert between UTF-8 and UTF-16 as the files go in and out\n> > o the working tree. There's no built-in support for that, but you could\n> > do it with a custom clean/smudge filter. That would let Git store UTF-8\n> > internally, do diffs, etc.\n> \n> Oh, by the way, I totally forgot that we added an internal version of\n> this, which is easier to configure and much more efficient. See the\n> \"working-tree-encoding\" attribute in \"git help attributes\".\n> \n> Just in case you do want to go that route.\n\nThe specific information you need is located in the Git FAQ[0], but\nroughly, you would probably want something like this:\n\n*.h text lf=crlf working-tree-encoding=UTF-16LE-BOM\n\nThat means that when checked out, the file will be in the format that\nlegacy Windows programs prefer (CRLF with little-endian UTF-16 with a\nBOM), but will be stored internally in Git with LF and UTF-8. That will\nmake things like git diff work much better, but still permit things to\nbe in the working tree as you wish.\n\nIf you really don't want those to be modified at all, then you'd want to\nwrite this:\n\n*.h -text\n\nHowever, Git will consider these files to be binary, since they are, and\ngit diff won't work on them without a textconv filter.\n\n[0] https://git-scm.com/docs/gitfaq#windows-text-binary\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"432885","messageId":"BN6PR15MB1426DAA212EC5A9E89E19176CBFD9@BN6PR15MB1426.namprd15.prod.outlook.com","threadId":"56280","inReplyTo":"YRrdq5VAp3o35+Fb@camp.crustytoothpaste.net","subject":"RE: git clone corrupts file.","fromName":"Russell, Scott","fromEmail":"scott.russell2@ncr.com","sentAt":"2021-08-16T22:04:20Z","receivedAt":"2021-08-16T22:04:42Z","isPatch":false,"sender":{"key":"scott.russell2@ncr.com","avatar":null},"body":"Thanks Brian, \n\nI appreciate the guidance. All our .h files can call be converted to ANSI. I don't know why we seemed to have just one saved as Unicode. \nBut it was a wakeup, and led to discovery of other files not correct. \n\nUpon reading the help on .gitattributes, I was reminded that Windows Visual Studio can save some .rc files as Unicode. \nI think that most all are ANSI but that leaves the possible result that any one saved as Unicode could unexpectedly fail compiling due to the conversion. \n\nWe have a mix of *.ini files which are a mix of mostly ANSI and more than a few others are Unicode. \nI don't know how to handle a mixture.\n\nPerhaps I will have to specify \n\n*.ini -text. \n\nUnless, does .gitattributes allow paths to be specified? In effect use the \n\nPath/path/path/* text lf=crlf working-tree-encoding=UTF-16LE-BOM\n\nAnd otherwise, \n*.ini text \t- these would be ansi if not in path/path/path \n\nThanks, \n\nScott Russell\nStaff SW Engineer \nNCR Corporation \nPhone: +17706237512\nScott.Russell2@ncr.com | ncr.com\n \n\n-----Original Message-----\nFrom: brian m. carlson \nSent: Monday, August 16, 2021 5:51 PM\nTo: Jeff King \nCc: Russell, Scott ; git@vger.kernel.org\nSubject: Re: git clone corrupts file.\n\n*External Message* - Use caution before opening links or attachments\n\nOn 2021-08-16 at 18:51:04, Jeff King wrote:\n> On Mon, Aug 16, 2021 at 12:53:36PM -0400, Jeff King wrote:\n> \n> > But an alternative workflow, if you really want UTF-16 in the \n> > working tree, is to convert between UTF-8 and UTF-16 as the files go \n> > in and out o the working tree. There's no built-in support for that, \n> > but you could do it with a custom clean/smudge filter. That would \n> > let Git store UTF-8 internally, do diffs, etc.\n> \n> Oh, by the way, I totally forgot that we added an internal version of \n> this, which is easier to configure and much more efficient. See the \n> \"working-tree-encoding\" attribute in \"git help attributes\".\n> \n> Just in case you do want to go that route.\n\nThe specific information you need is located in the Git FAQ[0], but roughly, you would probably want something like this:\n\n*.h text lf=crlf working-tree-encoding=UTF-16LE-BOM\n\nThat means that when checked out, the file will be in the format that legacy Windows programs prefer (CRLF with little-endian UTF-16 with a BOM), but will be stored internally in Git with LF and UTF-8. That will make things like git diff work much better, but still permit things to be in the working tree as you wish.\n\nIf you really don't want those to be modified at all, then you'd want to write this:\n\n*.h -text\n\nHowever, Git will consider these files to be binary, since they are, and git diff won't work on them without a textconv filter.\n\n[0] https://git-scm.com/docs/gitfaq#windows-text-binary\n--\nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"432886","messageId":"YRrkgQDRVT+O+4Ih@camp.crustytoothpaste.net","threadId":"56280","inReplyTo":"BN6PR15MB1426DAA212EC5A9E89E19176CBFD9@BN6PR15MB1426.namprd15.prod.outlook.com","subject":"Re: git clone corrupts file.","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-08-16T22:19:45Z","receivedAt":"2021-08-16T22:19:53Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-08-16 at 22:04:20, Russell, Scott wrote:\n> Thanks Brian,\n> \n> I appreciate the guidance. All our .h files can call be converted to ANSI. I don't know why we seemed to have just one saved as Unicode.\n> But it was a wakeup, and led to discovery of other files not correct.\n> \n> Upon reading the help on .gitattributes, I was reminded that Windows Visual Studio can save some .rc files as Unicode.\n> I think that most all are ANSI but that leaves the possible result that any one saved as Unicode could unexpectedly fail compiling due to the conversion.\n\nI do want to specify a distinction here. You're referring to \"Unicode\"\nand \"ANSI\", which traditionally mean, on Windows, little-endian UTF-16\nwith BOM and Windows-1252. You do not generally want Windows-1252, or\nthe encoding on which it's based, ISO-8859-1. Those are obsolete and\nhave been for well over a decade. It's unfortunate that many Windows\nprograms continue to use these terms, because neither \"Unicode\" nor\n\"ANSI\" describe an actual character set according to IANA.\n\nWhat is going to work best here is UTF-8 without a BOM. Most Windows\nprograms can handle that these days, but some still don't. If you try\nto save things as \"ANSI\" without a working-tree-encoding and they aren't\ncompletely ASCII files, then you will end up with some weird diff output\nat the very least.\n\nIf the files are completely ASCII, then no working-tree-encoding is\nnecessary, because ASCII is a subset of UTF-8.\n\n> We have a mix of *.ini files which are a mix of mostly ANSI and more than a few others are Unicode.\n> I don't know how to handle a mixture.\n> \n> Perhaps I will have to specify\n> \n> *.ini -text.\n> \n> Unless, does .gitattributes allow paths to be specified? In effect use the\n> \n> Path/path/path/* text lf=crlf working-tree-encoding=UTF-16LE-BOM\n\nYes, this syntax is allowed. See the gitattributes(5) manual page for\nwhat's allowed. You can even do this:\n\ndir/sub/path/*.ini text eol=crlf working-tree-encoding=UTF-16LE-BOM\n\nOne thing I forgot to mention is that after modifying your\n.gitattributes file, you'll want to run \"git add --renormalize .\" and\nthen commit both the .gitattributes file and any changes. Otherwise,\nyou may end up with files that don't end up converted the way that you\nwant.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"432889","messageId":"BN6PR15MB1426BCE62EFCDFE2F095C404CBFD9@BN6PR15MB1426.namprd15.prod.outlook.com","threadId":"56280","inReplyTo":"YRrkgQDRVT+O+4Ih@camp.crustytoothpaste.net","subject":"RE: git clone corrupts file.","fromName":"Russell, Scott","fromEmail":"scott.russell2@ncr.com","sentAt":"2021-08-16T22:26:36Z","receivedAt":"2021-08-16T22:26:52Z","isPatch":false,"sender":{"key":"scott.russell2@ncr.com","avatar":null},"body":"Ok, thanks for all the help. \nI think with the path in .gitattributes It will be fine. \n\ndir/sub/path/*.ini text eol=crlf working-tree-encoding=UTF-16LE-BOM\n\nI will give those a try and see how it works out. And especially thanks for the help advice on add -renormalize. I would never have done that. \n\n\nThanks, \n\nScott Russell\nStaff SW Engineer \nNCR Corporation \nPhone: +17706237512\nScott.Russell2@ncr.com | ncr.com\n \n\n-----Original Message-----\nFrom: brian m. carlson \nSent: Monday, August 16, 2021 6:20 PM\nTo: Russell, Scott \nCc: Jeff King ; git@vger.kernel.org\nSubject: Re: git clone corrupts file.\n\n*External Message* - Use caution before opening links or attachments\n\nOn 2021-08-16 at 22:04:20, Russell, Scott wrote:\n> Thanks Brian,\n> \n> I appreciate the guidance. All our .h files can call be converted to ANSI. I don't know why we seemed to have just one saved as Unicode.\n> But it was a wakeup, and led to discovery of other files not correct.\n> \n> Upon reading the help on .gitattributes, I was reminded that Windows Visual Studio can save some .rc files as Unicode.\n> I think that most all are ANSI but that leaves the possible result that any one saved as Unicode could unexpectedly fail compiling due to the conversion.\n\nI do want to specify a distinction here. You're referring to \"Unicode\"\nand \"ANSI\", which traditionally mean, on Windows, little-endian UTF-16 with BOM and Windows-1252. You do not generally want Windows-1252, or the encoding on which it's based, ISO-8859-1. Those are obsolete and have been for well over a decade. It's unfortunate that many Windows programs continue to use these terms, because neither \"Unicode\" nor \"ANSI\" describe an actual character set according to IANA.\n\nWhat is going to work best here is UTF-8 without a BOM. Most Windows programs can handle that these days, but some still don't. If you try to save things as \"ANSI\" without a working-tree-encoding and they aren't completely ASCII files, then you will end up with some weird diff output at the very least.\n\nIf the files are completely ASCII, then no working-tree-encoding is necessary, because ASCII is a subset of UTF-8.\n\n> We have a mix of *.ini files which are a mix of mostly ANSI and more than a few others are Unicode.\n> I don't know how to handle a mixture.\n> \n> Perhaps I will have to specify\n> \n> *.ini -text.\n> \n> Unless, does .gitattributes allow paths to be specified? In effect \n> use the\n> \n> Path/path/path/* text lf=crlf working-tree-encoding=UTF-16LE-BOM\n\nYes, this syntax is allowed. See the gitattributes(5) manual page for what's allowed. You can even do this:\n\ndir/sub/path/*.ini text eol=crlf working-tree-encoding=UTF-16LE-BOM\n\nOne thing I forgot to mention is that after modifying your .gitattributes file, you'll want to run \"git add --renormalize .\" and then commit both the .gitattributes file and any changes. Otherwise, you may end up with files that don't end up converted the way that you want.\n--\nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"}]}