Re: Encoding problem on OSX?

İsmail Dönmez wrote:

I just read BUGS in the sed distribution. Strangely enough the above seems to be correct behavior:

  Another common localization-related problem happens if your input stream
  includes invalid multibyte sequences.  POSIX mandates that such
  sequences are _not_ matched by `.', so that `s/.*//' will not clear
  pattern space as you would expect.  In fact, there is no way to clear
  sed's buffers in the middle of the script in most multibyte locales
  (including UTF-8 locales).  For this reason, GNU sed provides a `z'
  command (for `zap') as an extension.

However there is still a sed bug as far as I can tell, since in the test suite, LC_ALL is set to C, and using the C locale is the suggested workaround in the GNU sed docs. This explains where my first suggested diagnostic messed up: presumably

 printf 'Th\360\235\204\236s\n' | LC_ALL=C sed "s/.*//"

would print

 <treble clef>s

and

 printf 'Th\370\235\204\236s\n' | sed "s/.*//"

would print

 ????s

with your copy of sed 4.2.1.

Well, I learned something new today. Still thinking over how to fix this in the test suite. Thanks again.