[PATCH 0/3] use size_t for a few more object and buffer sizes

On 64-bit Windows, "long" is still 32 bits wide. Whenever a size_t travels through an "unsigned long", anything above 4 GiB is silently lost. Johannes's js/objects-larger-than-4gb-on-windows work fixed the read side of the object database, and Peff's jk/xdiff-size_t did the same for the diff and merge machinery. This series picks up three places that were left narrow.

 - odb: the write API still takes the object length as unsigned long.
   A large file that is converted before being stored (for example
   with core.autocrlf, which Git for Windows enables by default) can
   reach odb_write_object() with a size_t length above 4 GiB. The
   blob is then written truncated, with no error, and the index
   happily records it.

 - diff: diff_words_fill() grows its buffer with a signed "long"
   allocation size. Its output can be twice its input, and on LLP64
   the growth computation overflows past about 700MiB, which is well
   within what xdiff accepts. In practice Git dies on a huge
   allocation request.

 - convert: the helpers that decide whether a file is binary, has
   CRLF line endings or "$Id$" keywords take an unsigned long length.
   For files over 4GiB they only look at a prefix. In the same area,
   read_blob_data_from_index() narrows its result with
   cast_size_t_to_ulong(), so "git add" with core.autocrlf dies when
   the staged version is that large.

There are no new tests. Each case needs files of several GiB and a 32-bit "long", and the commit messages explain how to reproduce them.

How these were found --------------------

I wanted a list of all the places where a 64-bit size is narrowed on Windows, rather than waiting for the next bug report. Linux compilers cannot see the problem, because there "long" and "size_t" have the same width. So I analysed the tree as a Windows target instead.

The first step is a compilation database from a real MinGW configuration. Forcing --target onto a database generated on Linux does not work well: Linux-only defines leak in, regex.h is missing from MinGW, and the precompiled header is rejected for the wrong target. A Meson cross build avoids all of that:

  $ cat mingw64.cross
  [binaries]
  c       = 'x86_64-w64-mingw32-gcc'
  ar      = 'x86_64-w64-mingw32-ar'
  strip   = 'x86_64-w64-mingw32-strip'
  windres = 'x86_64-w64-mingw32-windres'

  [host_machine]
  system     = 'windows'
  cpu_family = 'x86_64'
  cpu        = 'x86_64'
  endian     = 'little'

  $ meson setup build-win --cross-file mingw64.cross \
	-Db_pch=false -Dsane_tool_path=/usr/bin \
	-Dcurl=disabled -Dexpat=disabled -Dpcre2=disabled \
	-Dgettext=disabled -Diconv=disabled -Drust=disabled
  $ meson compile -C build-win

Two of these options are not obvious. Without -Dsane_tool_path, setup stops because meson.build adds "C:/Program Files/Git/bin" to the program search path, which is not absolute on a Linux host. And -Db_pch=false is needed because clang cannot read the precompiled header that GCC produces. The build only has to get far enough to generate the headers; link errors do not matter here.

The second step runs clang-tidy over that database with -Wshorten-64-to-32, for the x86_64-w64-mingw32 target, skipping the bundled zlib and OpenSSL. One detail cost me a while: with "-checks=-*", clang-tidy drops compiler warnings unless clang-diagnostic-shorten-64-to-32 is enabled explicitly. I wrapped all this in a small script, which also refuses to report "no findings" when some translation units failed to parse:

  https://github.com/devzero2000/toyscript/blob/main/check_llp64_portability.sh

It is not meant for inclusion. I mention it so that the numbers below can be reproduced, and in case someone finds it useful.

What the scan shows -------------------

All 569 translation units were analysed, none failed. There are 1432
64-to-32 truncation warnings, but most of them are not specific to
Windows: about 1200 narrow a size_t to an int, as they do on Linux.
Only 237 have "long" or "unsigned long" as their destination.

Most of those 237 sit on API boundaries that are known and that commit c6a4629e32 names explicitly: diff, blame, textconv and fast-import. Others are in tree walking, object parsing and the pack tools, where an oversized object makes Git die rather than misbehave. I would like to convert them in later small series, one area at a time, in the same style as jk/xdiff-size_t.

Johannes, the first patch continues your write-side work. I hope it does not step on a follow-up you already have in progress.

Elia Pinto (3):
  odb: use size_t for the length of objects to write
  diff: use size_t for the allocation size of word-diff buffers
  convert: use size_t for buffer sizes

 attr.c                |  2 +-
 convert.c             | 14 +++++++-------
 diff.c                |  4 ++--
 odb.c                 |  2 +-
 odb.h                 |  4 ++--
 odb/source-inmemory.c |  2 +-
 odb/source-loose.c    |  2 +-
 odb/source.h          |  2 +-
 read-cache-ll.h       |  2 +-
 read-cache.c          |  4 ++--
 10 files changed, 19 insertions(+), 19 deletions(-)

base-commit: 47fdf2208c5b94c8cc01c106b8f7c85efb91c926 -- 2.53.0