Git 2.26.2 and failed self tests on OS X

Hi Everyone,

I'm building Git 2.26.2 from sources on a Mac-mini with OS X 10.12.6 and a PowerMac with OS X 10.5. Some of the t3902-quoted.sh tests are failing.

The Mac-Mini uses Bash 3.2.57(1)-release. The PowerMac uses 3.2.17(1)-release.

Jeff

*** t3902-quoted.sh *** not ok 1 - setup # # # mkdir "$FN" && # for_each_name "echo initial >\"\$name\"" && # git add . && # git commit -q -m Initial && # # for_each_name "echo second >\"\$name\"" && # git commit -a -m Second && # # for_each_name "echo modified >\"\$name\"" # # ok 2 - setup expected files not ok 3 - check fully quoted output from ls-files # # # git ls-files >current && test_cmp expect.quoted current # # not ok 4 - check fully quoted output from diff-files # # # git diff --name-only >current && # test_cmp expect.quoted current # # not ok 5 - check fully quoted output from diff-index # # # git diff --name-only HEAD >current && # test_cmp expect.quoted current # # not ok 6 - check fully quoted output from diff-tree # # # git diff --name-only HEAD^ HEAD >current && # test_cmp expect.quoted current # # not ok 7 - check fully quoted output from ls-tree # # # git ls-tree --name-only -r HEAD >current && # test_cmp expect.quoted current # # ok 8 - setting core.quotepath not ok 9 - check fully quoted output from ls-files # # # git ls-files >current && test_cmp expect.raw current # # not ok 10 - check fully quoted output from diff-files # # # git diff --name-only >current && # test_cmp expect.raw current # # not ok 11 - check fully quoted output from diff-index # # # git diff --name-only HEAD >current && # test_cmp expect.raw current # # not ok 12 - check fully quoted output from diff-tree # # # git diff --name-only HEAD^ HEAD >current && # test_cmp expect.raw current # # not ok 13 - check fully quoted output from ls-tree # # # git ls-tree --name-only -r HEAD >current && # test_cmp expect.raw current # # # failed 11 among 13 test(s) 1..13 make[2]: *** [t3902-quoted.sh] Error 1 make[1]: *** [test] Error 2 make: *** [test] Error 2

I am unable to reproduce these failures on OS X 10.13.6.

On Mon, May 04, 2020 at 03:03:01AM -0400, Jeffrey Walton wrote:

if the problem is the bash version but you can build/install a newer one that doesn't have this problem then you can get the test to use it by pointing to it with TEST_SHELL_PATH as in :

  $ make TEST_SHELL_PATH=/usr/local/bin/bash test

Carlo

On Mon, May 4, 2020 at 12:52 PM Carlo Marcelo Arenas Belón <carenas@gmail.com> wrote:

My OS X 10.13.6 has Bash 3.2.57, as well, and I'm unable to reproduce these failures, so it seems fairly unlikely that the problem Jeffrey is seeing isn't tied to Bash.

On Mon, May 04, 2020 at 01:18:06PM -0400, Eric Sunshine wrote:

Should I read this as ".. seems fairly likely that the problem Jeffrey is seeing is tied to bash" ?

I can't reproduce it either, so my reasoning would rather be "The problem is probably not caused by the shipped bash version"

It could be something in the environemt, that confuses our tests here.

Does ./t3902-quoted.sh -v -d give anything useful ?

On Mon, May 4, 2020 at 3:08 PM Torsten Bögershausen <tboegi@web.de> wrote:

Yes, applying s/unlikely/likely/ to my statement is the correct interpretation. (Last-second editing strikes again.)

On Mon, May 4, 2020 at 3:08 PM Torsten Bögershausen <tboegi@web.de> wrote:

Yes, it looks like the command reveals a couple of problems.

I'll have to look into my iconv recipe. It has not changed in a while, so something else may be responsible for the failure. https://github.com/noloader/Build-Scripts/blob/master/build-iconv.sh.

I'm not sure what to do with the HEAD problems.

$ ./t3902-quoted.sh -v -d Initialized empty Git repository in /Users/jwalton/Build-Scripts/git-2.26.2/t/trash directory.t3902-quoted/.git/ expecting success of 3902.1 'setup':

        mkdir "$FN" &&
        for_each_name "echo initial >\"\$name\"" &&
        git add . &&
        git commit -q -m Initial &&

        for_each_name "echo second >\"\$name\"" &&
        git commit -a -m Second &&

        for_each_name "echo modified >\"\$name\""

fatal: iconv_open(UTF-8,UTF-8-MAC) failed, but needed:
    precomposed unicode is not supported.
    If you want to use decomposed unicode, run
    "git config core.precomposeunicode false"

not ok 1 - setup # # # mkdir "$FN" && # for_each_name "echo initial >\"\$name\"" && # git add . && # git commit -q -m Initial && # # for_each_name "echo second >\"\$name\"" && # git commit -a -m Second && # # for_each_name "echo modified >\"\$name\"" # #

expecting success of 3902.2 'setup expected files': cat >expect.quoted <<\EOF && Name "Name and a\nLF" "Name and an\tHT" "Name\"" With SP in it "\346\277\261\351\207\216\t\347\264\224" "\346\277\261\351\207\216\n\347\264\224" "\346\277\261\351\207\216 \347\264\224" "\346\277\261\351\207\216\"\347\264\224" "\346\277\261\351\207\216/file" "\346\277\261\351\207\216\347\264\224" EOF

cat >expect.raw <<\EOF Name "Name and a\nLF" "Name and an\tHT" "Name\"" With SP in it "濱野\t純" "濱野\n純" 濱野 純 "濱野\"純" 濱野/file 濱野純 EOF

ok 2 - setup expected files

expecting success of 3902.3 'check fully quoted output from ls-files':

        git ls-files >current && test_cmp expect.quoted current

--- expect.quoted 2020-05-04 19:14:39.000000000 +0000 +++ current 2020-05-04 19:14:39.000000000 +0000

On Mon, May 04, 2020 at 03:20:11PM -0400, Jeffrey Walton wrote:

I am quite sure, that the trouble starts here. It seems as if you are building iconv yourself ? And that iconv does not include "UTF-8-MAC", which is a speciality for Mac.

Are there any special reasons to build your own iconv ? You are the first one repprting this kind of issue, thanks for reporting it.

On Mon, May 4, 2020 at 4:19 PM Torsten Bögershausen <tboegi@web.de> wrote:

Oh, thanks. I was not aware.

I'll open a bug report with the iconv folks letting them know configure has some gaps on OS X.

Yes, iconv bugs on older platforms.

The only way I've found to avoid intermittent problems on older platforms is using the latest iconv (which is pretty stable nowadays), and avoid the glibc iconv.

Jeff

On Mon, May 4, 2020 at 4:30 PM Jeffrey Walton <noloader@gmail.com> wrote:

Thanks again Torsten.

I am not seeing an option to enable the encoding in iConv. Would you happen to know what is needed?

Thanks again.

$ cd libiconv-1.16 $ ./configure --help `configure' configures libiconv 1.16 to adapt to many kinds of systems.

Usage: ./configure [OPTION]... [VAR=VALUE]...

To assign environment variables (e.g., CC, CFLAGS...), specify them as VAR=VALUE. See below for descriptions of some of the useful variables.

Defaults for the options are specified in brackets.

Configuration:
  -h, --help              display this help and exit
      --help=short        display options specific to this package
      --help=recursive    display the short help of all the included packages
  -V, --version           display version information and exit
  -q, --quiet, --silent   do not print `checking ...' messages
      --cache-file=FILE   cache test results in FILE [disabled]
  -C, --config-cache      alias for `--cache-file=config.cache'
  -n, --no-create         do not create output files
      --srcdir=DIR        find the sources in DIR [configure dir or `..']

Installation directories:
  --prefix=PREFIX         install architecture-independent files in PREFIX
                          [/usr/local]
  --exec-prefix=EPREFIX   install architecture-dependent files in EPREFIX
                          [PREFIX]

By default, `make install' will install all the files in `/usr/local/bin', `/usr/local/lib' etc. You can specify an installation prefix other than `/usr/local' using `--prefix', for instance `--prefix=$HOME'.

For better control, use the options below.

Fine tuning of the installation directories:
  --bindir=DIR            user executables [EPREFIX/bin]
  --sbindir=DIR           system admin executables [EPREFIX/sbin]
  --libexecdir=DIR        program executables [EPREFIX/libexec]
  --sysconfdir=DIR        read-only single-machine data [PREFIX/etc]
  --sharedstatedir=DIR    modifiable architecture-independent data [PREFIX/com]
  --localstatedir=DIR     modifiable single-machine data [PREFIX/var]
  --libdir=DIR            object code libraries [EPREFIX/lib]
  --includedir=DIR        C header files [PREFIX/include]
  --oldincludedir=DIR     C header files for non-gcc [/usr/include]
  --datarootdir=DIR       read-only arch.-independent data root [PREFIX/share]
  --datadir=DIR           read-only architecture-independent data [DATAROOTDIR]
  --infodir=DIR           info documentation [DATAROOTDIR/info]
  --localedir=DIR         locale-dependent data [DATAROOTDIR/locale]
  --mandir=DIR            man documentation [DATAROOTDIR/man]
  --docdir=DIR            documentation root [DATAROOTDIR/doc/libiconv]
  --htmldir=DIR           html documentation [DOCDIR]
  --dvidir=DIR            dvi documentation [DOCDIR]
  --pdfdir=DIR            pdf documentation [DOCDIR]
  --psdir=DIR             ps documentation [DOCDIR]

Program names:
  --program-prefix=PREFIX            prepend PREFIX to installed program names
  --program-suffix=SUFFIX            append SUFFIX to installed program names
  --program-transform-name=PROGRAM   run sed PROGRAM on installed program names

System types:
  --build=BUILD     configure for building on BUILD [guessed]
  --host=HOST       cross-compile to build programs to run on HOST [BUILD]

Optional Features:
  --disable-option-checking  ignore unrecognized --enable/--with options
  --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)
  --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]
  --enable-silent-rules   less verbose build output (undo: "make V=1")
  --disable-silent-rules  verbose build output (undo: "make V=0")
  --enable-dependency-tracking
                          do not reject slow dependency extractors
  --disable-dependency-tracking
                          speeds up one-time build
  --disable-largefile     omit support for large files
  --enable-static[=PKGS]  build static libraries [default=no]
  --enable-shared[=PKGS]  build shared libraries [default=yes]
  --enable-fast-install[=PKGS]
                          optimize for fast installation [default=yes]
  --disable-libtool-lock  avoid locking (might break parallel builds)
  --enable-relocatable    install a package that can be moved in the file
                          system
  --enable-extra-encodings
                          add support for a few rarely used encodings
  --disable-rpath         do not hardcode runtime library paths
  --disable-nls           do not use Native Language Support

Optional Packages:
  --with-PACKAGE[=ARG]    use PACKAGE [ARG=yes]
  --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
  --with-pic[=PKGS]       try to use only PIC/non-PIC objects [default=use
                          both]
  --with-aix-soname=aix|svr4|both
                          shared library versioning (aka "SONAME") variant to
                          provide on AIX, [default=aix].
  --with-gnu-ld           assume the C compiler uses GNU ld [default=no]
  --with-sysroot[=DIR]    Search for dependent libraries within DIR (or the
                          compiler's sysroot if not specified).
  --with-gnu-ld           assume the C compiler uses GNU ld [default=no]
  --with-libiconv-prefix[=DIR]  search for libiconv in DIR/include and DIR/lib
  --without-libiconv-prefix     don't search for libiconv in
includedir and libdir
  --with-libintl-prefix[=DIR]  search for libintl in DIR/include and DIR/lib
  --without-libintl-prefix     don't search for libintl in includedir and libdir

Some influential environment variables:
  CC          C compiler command
  CFLAGS      C compiler flags
  LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a
              nonstandard directory <lib dir>
  LIBS        libraries to pass to the linker, e.g. -l<library>
  CPPFLAGS    (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if
              you have headers in a nonstandard directory <include dir>
  CPP         C preprocessor
  LT_SYS_LIBRARY_PATH
              User-defined run-time library search path.

Use these variables to override the choices made by `configure' or to help it to find libraries and programs with nonstandard names/locations.

On Mon, May 04, 2020 at 05:39:50PM -0400, Jeffrey Walton wrote:

Hej Jeffrey, I am not an iconv expert - and don't intend to become one. The UTF-8-MAC conversion does probably only exist in the code for iconv shipped on Mac OS - so it could happen that this code is not open source. There may be ways to "extract it" : Either that information is part of the unicode specification, and there may be tables that can be converted into c-code that fits into the iconv code base. I don't know. Or you can use the iconv on MacOs itself to generate such a table. That is all somewhat behind my horizon.

For Git (under MacOS) we need UTF-8-MAC to compensate for HPS, (or HPF+) so that all filenames with non-ASCII codepoints are stored in the repo in precomposed unicode. This is what Linux and Windoes and Unix is using, this commit may give more information about the what and why:

  commit 76759c7dff53e8c84e975b88cb8245587c14c7ba
  Author: Torsten Bögershausen <tboegi@web.de>
  Date:   Sun Jul 8 15:50:25 2012 +0200
    git on Mac OS and precomposed unicode

By the way, does ./t3910-mac-os-precompose.sh work on your system ?

So far I am not aware about any problems with iconv from MacOs being reported here on the list. Which problems/bugs did you found in the iconv shipped with MacOs, especially together with Git ?

[snip]

Hi everyone,

I also came across this error a few months ago with the conda-forge build of Git [1], which for the moment builds against their own build of (GNU) iconv.

Yes, this encoding only exists in the Apple fork of iconv, which is open-source (probably because of the original license); tarballs can be found at [2]. However in typical Apple fashion the versioning of their iconv fork is kind of undecipherable and hard to relate to the upstream iconv versioning.

Also, I searched the iconv mailing list [3] when I came across this error and the consensus seemed to be that they do not intend to add this encoding upstream.

Cheers,

Philippe.

[1] https://github.com/conda-forge/git-feedstock/issues/50 [2] https://opensource.apple.com/tarballs/libiconv/ [3] https://lists.gnu.org/archive/cgi-bin/namazu.cgi?query=utf-8-mac&submit=Search%21&idxname=bug-gnu-libiconv&max=20&result=normal&sort=score

On Tue, May 5, 2020 at 8:26 AM Philippe Blain <philippe.blain@me.com> wrote:

Thanks Phillipe.

I found two sources for libiconv with UTF-8-Mac. First is a GitHub at https://github.com/fumiyas/libiconv-utf8mac. The second is the opensource.apple.com link you provided at [2].

The GitHub claims to be layered on top of libiconv 1.16, but I was not able to get a working tarball from it. It looks like Apple's libiconv-59.tar.gz is the answer to this problem.

Thanks again for the help.

Jeff

On Wed, May 6, 2020 at 4:36 PM Jeffrey Walton <noloader@gmail.com> wrote:

I had to punt on Apple's port. libiconv-59.tar.gz is libiconv-1.11. It is too old to be useful.

libiconv-utf8mac is based on libiconv-1.16 and worked fine. It took my patches after I figured out the complete procedure. On the upside, libiconv-utf8mac works on Linux and Solaris too. So there is support for UTF-8-Mac if needed.

Jeff

On Tue, May 5, 2020 at 12:10 AM Torsten Bögershausen <tboegi@web.de> wrote:

Thanks again Torsten .

So there are several problems with libiconv. First, sometimes it is too old and carries bugs with it. For example, sed and awk fail to build and pass their self tests because they expect a minimum version of libiconv.

Second, there are some modern bugs that needs fixing. They were reported after libiconv 1.16 was released. It needs patches.

Third, libiconv cannot get through testing with UBsan. It causes a number of packages to fail to configure or pass their self tests. It needs patches.

Fourth, libiconv cannot get through testing with Asan and Valgrind. It leaks resources like a sieve, and leaves memory and file handles open. It causes a number of packages to fail to configure or pass their self tests. It needs more patches.

Due to all the problems with libiconv I just patch it and build it everywhere I need it.

The other option is to wait for the maintainers to fix libiconv. Based on some reading about the UTF-8-Mac problem I don't think that will happen. And they are not interested in fixing the resource leaks. GNU has such low coding standards...

Jeff