Re: [RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?
- From
Drew Northup <drew.northup@maine.edu>
- Date
- Oct 22, 2010, 18:06 UTC
- Message-ID
- <1287770805.819.7.camel@drew-northup.unet.maine.edu>
- In-Reply-To
- <m31v7iktkn.fsf@localhost.localdomain>
On Fri, 2010-10-22 at 10:48 -0700, Jakub Narebski wrote:
> Drew Northup <drew.northup@maine.edu> writes:
Show 13 quoted lines
> > Well I shall plumb the documentation again.... just in case. I'm not
> > holding my breath that it will do what I (and frankly a fair number of
> > other people) want. We just want version control that treats text like
> > text. FULL STOP. Why isn't UTF-16 text???????
>
> If you are asking why Git detects files with text in UTF-16 / USC-2 as
> binary, it is because Git (re)uses the same heuristic that e.g. GNU
> diff (and probably also -T file test in Perl), and one of heuristics
> is that if file contains NUL ("\0") character, then it is most
> porbably binary (because legacy C programs for text would have
> troubles with NUL characters).
>
> That probably doesn't help you any...I did find that already. I still have not decided that correct place to shoehorn in Unicode detection, but I'll be sure to do that before I bother anybody else with it. I already wrote code to detect (reasonably) valid UTF-16 (if it isn't obviously valid then I'll just as soon deal with it as binary data, so as to avoid a foot-shooting exercise). My main motivation here has been to get some feedback as I write stuff so as to not waste a lot of time during writing something that could be done better. (As opposed to not done at all, which is the feeling I'm getting from a few people around here...)
-- -Drew Northup N1XIM AKA RvnPhnx on OPN ________________________________________________ "As opposed to vegetable or mineral error?" -John Pescatore, SANS NewsBites Vol. 12 Num. 59