patcht8005: use more portable character encoding names

Some platforms do not have an extensive list of alternate names for character encodings.

On IRIX 6.5 cp1251 is unknown, but WIN1251 (which seems to be a non-standard name) is known. On Solaris 10, the opposite is true. Solaris also knows cp1251 as WINDOWS-1251, but this too is not recognized on IRIX. I could not find a name that both platforms recognized for this character set.

Solaris 7 does not know about shift-jis, but does know SJIS. It also does not know that utf-8 and UTF-8 refer to the same encoding.

With the above in mind, and the impression that upper-case code names are more portable than lower-case, the following conversions were performed:

      utf-8 --> UTF-8
     cp1251 --> CP1251
  shift-jis --> SJIS

The UTF-8 and SJIS conversions allow 2 of the 3 elements of test 4 to succeed on Solaris 7, but alas, my Solaris 7 does not have CP1251.

Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil>
---

Here's another patch in an attempt to make the character encoding names more portable. I hesitate to send it since it does not actually allow any tests to pass which were previously failing for me. At least the utf-8 to UTF-8 shouldn't cause any harm. The shift-jis to SJIS seems to be commonly supported? at least IRIX 6.5 and Solaris as old as 7 recognize it, but the evidence is only empirical. Modern Solaris, and of course Linux recognize either.

-brandon

 t/t8005-blame-i18n.sh |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)

Brandon Casey <casey@nrlssc.navy.mil> writes:

Perhaps the test can be changed from using cp1251 (whatever that is) to something more common (e.g. 8859-1)?

There was another more large-ish patch to the tests around this area recently from you. I liked them but somehow failed to queue any of them yet, sorry.

Junio C Hamano wrote:

Good idea.

You've got them in 'next'. Did you forget? :) or don't you consider that queued?

-brandon

2009/5/22 Junio C Hamano <gitster@pobox.com>:

cp1251 (win-1251) is Microsoft's Windows Russian

Some platforms do not have an extensive list of alternate names for character encodings.

Solaris 7 does not know about shift-jis, but does know SJIS. It also does not know that utf-8 and UTF-8 refer to the same encoding.

With the above in mind, the following conversions were performed:

      utf-8 --> UTF-8
  shift-jis --> SJIS

Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil>
---
 t/t8005-blame-i18n.sh |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

On IRIX 6.5 CP1251 is unknown, but WIN1251 (which seems to be a non-standard name) is known. On Solaris 10, the opposite is true. Solaris also knows CP1251 as WINDOWS-1251, but this too is not recognized on IRIX. I could not find a name that both platforms recognized for this character set.

An alternative character set which covers the same alphabet seems to be the ISO8859-5 character set. Both platforms support this character set, so use it instead.

This allows t8005.4 to pass on Solaris 7, and part of the test to pass on IRIX. (My IRIX can't convert SJIS to UTF-8 :(

Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil>
---

This should still provide the same breadth of testing: a russian code set, a japanese code set, and unicode. Wikipedia says that ISO8859-5 is not as commonly used as either CP1251 or KOI8-R, but it is available on these old platforms. You'd think that if Solaris 7 has it _and_ IRIX has it, then everything else would too, but still feel free to drop this anyway. Neither of those platforms, or Solaris 10 (for me) can convert between any of the russian and japanese encodings, so t8005.[23] still fail, and I doubt there is any hope.

-brandon

 t/t8005-blame-i18n.sh |   26 +++++++++++++-------------
 t/t8005/cp1251.txt    |    2 --
 t/t8005/iso8859-5.txt |    2 ++
 3 files changed, 15 insertions(+), 15 deletions(-)
 delete mode 100644 t/t8005/cp1251.txt
 create mode 100644 t/t8005/iso8859-5.txt

Brandon Casey <casey@nrlssc.navy.mil> writes:

Thanks.

My earlier experience with Sun is that their iconv cannot convert between different encodings of Japanese but can use UTF-8 as a pivot to convert, e.g. SJIS -> UTF-8 -> ISO-2022. Perhaps there is an workaround, but I dunno.

By the way, with the attached patch, I seem to be able to compile everything more or less cleanly on OpenSolaris (200811).

In git-compat-util.h, we do

    #define _XOPEN_SOURCE 600
    #define _XOPEN_SOURCE_EXTENDED 1

and their /usr/include/sys/feature_tests.h has this nice table:

    Feature Test Macro				     Specification
    ------------------------------------------------  -------------
    _XOPEN_SOURCE                                         XPG3
    _XOPEN_SOURCE && _XOPEN_VERSION = 4                   XPG4
    _XOPEN_SOURCE && _XOPEN_SOURCE_EXTENDED = 1           XPG4v2
    _XOPEN_SOURCE = 500                                   XPG5
    _XOPEN_SOURCE = 600  (or POSIX_C_SOURCE=200112L)      XPG6

Later in the same header, compilation with -c99 is made to fail if _XPG6 is not set, like this:

    #if defined(_STDC_C99) && (defined(__XOPEN_OR_POSIX) && !defined(_XPG6))
    #error "Compiler or options invalid for pre-UNIX 03 X/Open applications \
            and pre-2001 POSIX applications"
    #elif ...

The problem is that they check things in an order that is inconvenient for us. When they see _XOPEN_SOURCE_EXTENDED, they declare that we are XPG4v2 regardless of the value of _XOPEN_SOURCE.

It also appears that we do not need OLD_ICONV anymore, as our Makefile suggests for older releases.

It would be nice if people with older Solaris can check if I am not breaking things for them...

 Makefile          |    4 +++-
 git-compat-util.h |    2 ++
 2 files changed, 5 insertions(+), 1 deletions(-)

On Fri, May 22, 2009 at 06:06:06PM -0700, Junio C Hamano wrote:

Git builds fine on my Solaris 8 test box with this patch. It also builds fine if I drop OLD_ICONV, so it may be that we can unconditionally do so (but I don't have any older boxen to test with).

With current 'next' and NO_PERL set, I can pass every test except:

  t3900 - still problems in the eucJP test. I haven't looked closely,
          but my understanding is that this might need extra language
          packs installed (I know virtually nothing about Solaris
          administration and the box is not mine).

  t8005 - git produces incorrect (or at least not expected) results for
          the iso8859-5 to sjis conversion. It all looks like control
          characters to me, so I'm not sure how to diagnose (and it may
          just be an installation issue again).

  t4116,t5000,t5001 - Solaris tar doesn't like pax headers that
                      git-archive generates

  t7700 - seems to be some problem invoking touch. This might be easily
          fixable, so I'll look into it.

-Peff

On Mon, May 25, 2009 at 05:20:27AM -0400, Jeff King wrote:

Never mind about this one. It is broken with /usr/ucb/touch (because it uses "-r"), but I usually do my auto-builds with /usr/bin/touch (I just noticed it because I was testing by hand).

-Peff

Jeff King wrote:

I will.

It also appears from looking at /usr/include/iconv.h that Solaris 10 does not need it as long as _XPG6 is defined since that file has:

  #ifdef _XPG6
  extern size_t   iconv(iconv_t, char **_RESTRICT_KYWD,
                  size_t *_RESTRICT_KYWD, char **_RESTRICT_KYWD,
                  size_t *_RESTRICT_KYWD);
  #else
  extern size_t   iconv(iconv_t, const char **_RESTRICT_KYWD,
                  size_t *_RESTRICT_KYWD, char **_RESTRICT_KYWD,
                  size_t *_RESTRICT_KYWD);
  #endif

I haven't tested, but I will.  I think Solaris 7 needs it though.

t3900.2[23] ?

Yes, Solaris 7 and 10 fail for me on those tests. Neither can convert between eucJP and ISO-2022-JP. The other tests pass because it can convert from either of those to UTF-8 and vice versa.

The iconv libraries seem to be named according to the conversions that they can perform. So in /usr/lib/iconv/ you'll see:

   UTF-8%eucJP.so
   eucJP%UTF-8.so
   ISO-2022-JP%UTF-8.so
   UTF-8%ISO-2022-JP.so
   etc.

but you won't see (at least I don't)

   eucJP%ISO-2022-JP.so
   ISO-2022-JP.so%eucJP

t8005.[23]

Same reason as above, no conversion between ISO8859-5 and SJIS. t8005.4 passes since conversion for both of those to UTF-8 is installed.

Same here.

On Solaris 7, I additionally must skip:

   t5100.[56] t5100.1[026-9] t5100.2[0-6]

This is because iconv on Solaris 7 doesn't understand charset 'us-ascii'. If I change 'charset=us-ascii' to 'charset=646' everywhere in t5100/sample.mbox and t5100/rfc2047-samples.mbox, then all of t5100 passes. It seems '646' is the only name by which Solaris 7 recognizes ISO 646 aka ascii. Other platforms do not recognize '646'.

Also, if you want to test with the Korn shell (you'll need a couple minor tweaks to the test lib), t6030.1[23] must be skipped. It seems even Solaris 10 has what I consider a bug in ksh which affects how traps are handled. A trap set in one function, will not be called in the second sub-function.

i.e.

   f1()
     sets a trap on EXIT
     calls f2()
       calls f3()
         calls exit

   The trap set in f1() will _not_ be executed.

I have only observed this behavior on Solaris. git-bisect sets such a trap, but I think it is being rewritten in c, so hopefully this issue will go away soon.

-brandon

On Tue, May 26, 2009 at 09:56:26AM -0500, Brandon Casey wrote:

Hmph. On Solaris 8, there is no such conditional; iconv is defined similar to the second version you quoted above:

  extern size_t iconv(iconv_t, const char **, size_t *, char **, size_t *);

So it would seem we do need OLD_ICONV there. But it compiles just fine without any warnings.

Yes.

Ah, I didn't know about that. Thanks. I also have the ones for UTF-8 conversion, but not for direct eucJP to ISO-2022-JP conversion.

OK, makes sense.

Those work just fine for me on Solaris 8.

I do all of my testing with bash, having given up on Solaris /bin/sh. In fact, I have found that bash 2.05 works, but bash 2.03 does _not_ (it doesn't like sourcing the files with non-ascii characters in t8005.

-Peff

Jeff King wrote:

Oh, and have the recent changes that are now in next _allowed_ any new tests to pass that were previously failing?

-brandon

On Tue, May 26, 2009 at 09:57:41AM -0500, Brandon Casey wrote:

Just one. It had been a while since I re-checked the skipped tests, so most of what I found I could now pass was unrelated. But I believe your recent patches fixed t3901 for me. On current 'next', it passes, but on current 'maint', it says something like "cannot convert UTF-8 to utf-8".

-Peff

Jeff King <peff@peff.net> writes:

I think I know about this one.

    From:	Junio C Hamano <gitster@pobox.com>
    Subject: [PATCH] t3900: ISO-2022-JP has more than one popular variants
    Date:	Tue, 12 May 2009 02:29:10 -0700
    Message-ID: <7vljp28yah.fsf@alter.siamese.dyndns.org>

   I had to "pkg install lang-support-japanese" to pass this test.

I haven't looked at this myself, but I've seen this consistently fail.

I think I force it to use gtar.