{"thread":{"id":"25189","subject":"Unicode text files (UTF-16 in particular, but also UTF-32)","startedAt":"2010-09-21T19:26:28Z","lastAt":"2010-09-21T19:26:28Z","messageCount":1,"participants":["Drew Northup"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"151237","messageId":"1285097188.3802.57.camel@drew-northup.unet.maine.edu","threadId":"25189","inReplyTo":null,"subject":"Unicode text files (UTF-16 in particular, but also UTF-32)","fromName":"Drew Northup","fromEmail":"drew.northup@maine.edu","sentAt":"2010-09-21T19:26:28Z","receivedAt":"2010-09-21T19:26:28Z","isPatch":false,"sender":{"key":"drew.northup@maine.edu","avatar":"https://avatars.githubusercontent.com/u/18331571?v=4"},"body":"(With reference to \nMessage-ID: 00E0B9AC-2A2E-4F95-9B35-F3F63EBC3CF3 () gmail ! com )\n\"Yeah, the problem is finding someone who needs the feature _and_ is\nable/willing to implement it.\"\n\nWell, \nI'd like to take a stab at it. I've been thinking about how to go about\nit for a little over two weeks, and I'm pretty sure I've changed my mind\nmore times than there are days in the equivalent time period.\n\nCurrent idea foci:\n\n* UTF-8 BOM can go to hell: I'm not going to extra effort to figure out\nwhat do to with UTF-8 as it seems to be working already for the most\npart. I am not planning on forcing anything to unnaturally generate a\nBOM when converting to UTF-8. (It sure doesn't look like iconv() does!\nAt least not by default...) I would like to make sure that all UTF-8 I\ndeal with is valid--and it looks like that might have been done already.\n\n* UTF-16/UTF-32 detection: Detect VALID UTF-16/UTF-32 containing files.\nFiles that fail this test are treated as \"not detected\"--so they are\nwhatever we currently detect them as being (binary, more likely than\nnot) or say they are otherwise. My current inclination is to implement\nthis in a separate source file & header to be included by code such as\nconvert.c for use. Then the changes needed to convert.c will be smaller\nand all of it will be more sensibly contained.\n\n* The commit message is already UTF-8 and this shouldn't change, so I'm\nnot going to mess with it. However, the utilities that display these and\nthe diff lines as well shouldn't need to be completely re-worked if we\ndon't need to. Therefore, I propose using UTF-8 for this (diff, etc.)\neven if the in-content encoding is different. This does raise the\ninvalid UTF-8 not converting to UTF-16/UTF-32 without loss, but if we're\ngenerating invalid UTF-8 we have other problems to begin with.\n\n* Storage of UTF-16/UTF-32 content: Do we want to treat this as all\nother texts (raw content), normalizing to LF line ends and such\n(requires changes to how we handle LF/CRLF conversions)? Should we store\nit as UTF-8 and convert back to the source encoding on export from the\nobject database (metadata would be required to do this correctly)?\n\nIt seems to me as if there are a few things to hash out before getting\nbeyond the point of accurate unicode [file] content text detection.\n\nComments, suggestions, and flames (so long as BBQ sauce is provided I\nsuppose) welcome.\n\n-- \n-Drew Northup\n________________________________________________\n\"As opposed to vegetable or mineral error?\"\n-John Pescatore, SANS NewsBites Vol. 12 Num. 59\n"}]}