From: Jonathan Nieder <jrnieder@gmail.com>
Date: Wed, 11 Aug 2010 09:23:11 GMT
Subject: Re: Encoding problem on OSX?
Message-ID: <20100811092311.GA16901@burratino>
In-Reply-To: <AANLkTi=sBNmMoZmqO_-AFNO6bVqtJmXbdXYUf7-bKars@mail.gmail.com>

İsmail Dönmez wrote:

> Downgrading my sed to v 4.1.5 fixed the issue. Thanks for your help!

I just read BUGS in the sed distribution.  Strangely enough the above seems to
be correct behavior:

  Another common localization-related problem happens if your input stream
  includes invalid multibyte sequences.  POSIX mandates that such
  sequences are _not_ matched by `.', so that `s/.*//' will not clear
  pattern space as you would expect.  In fact, there is no way to clear
  sed's buffers in the middle of the script in most multibyte locales
  (including UTF-8 locales).  For this reason, GNU sed provides a `z'
  command (for `zap') as an extension.

However there is still a sed bug as far as I can tell, since in the
test suite, LC_ALL is set to C, and using the C locale is the
suggested workaround in the GNU sed docs.  This explains where my
first suggested diagnostic messed up: presumably

 printf 'Th\360\235\204\236s\n' | LC_ALL=C sed "s/.*//"

would print

 <treble clef>s

and

 printf 'Th\370\235\204\236s\n' | sed "s/.*//"

would print

 ????s

with your copy of sed 4.2.1.

Well, I learned something new today.  Still thinking over how to fix
this in the test suite.  Thanks again.