git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: js/drop-mingw-test-cmp, was Re: What's cooking in git.git (Dec 2022, #03; Sun, 11)

From
Junio C Hamano <gitster@pobox.com>
Date
Dec 27, 2022, 12:52 UTC
Message-ID
<xmqqilhxt3dt.fsf@gitster.g>
In-Reply-To
<2a5433ae-ec48-0c24-c18e-9a415d29e590@web.de>
René Scharfe <l.s.r@web.de> writes:
Show 33 quoted lines
> Logged the sizes of files handed to test_cmp (on macOS).  19170 calls,
> median size 42 bytes, average size 617 bytes.  2307 calls with empty
> files, 1093 of which in t1092 alone.  The two biggest files in t1050,
> 2000000 and 2500000 bytes.  t9300 in third place with 180333, one
> magnitude smaller.
>
> t1050 at 8a4e8f6a67 (The second batch, 2022-12-26) on Windows:
>
> Benchmark 1: sh.exe t1050-large.sh
>   Time (mean ± σ):     18.312 s ±  0.069 s    [User: 0.000 s, System: 0.003 s]
>   Range (min … max):   18.218 s … 18.422 s    10 runs
>
> ... and with the patch:
>
> Benchmark 1: sh.exe t1050-large.sh
>   Time (mean ± σ):      5.709 s ±  0.046 s    [User: 0.000 s, System: 0.003 s]
>   Range (min … max):    5.647 s …  5.787 s    10 runs
>
> So it works as advertised for big files, but calling an external
> program 19000 times takes time as well, which explain the longer
> overall test suite duration.
>
> If we use test_cmp_bin for the two biggest comparisons we get the
> same speedup:
>
> Benchmark 1: sh.exe t1050-large.sh
>   Time (mean ± σ):      5.719 s ±  0.089 s    [User: 0.000 s, System: 0.006 s]
>   Range (min … max):    5.659 s …  5.960 s    10 runs
>
> Is this safe?  The files consist of X's and Y's at the point of
> comparison, so they aren't typical binary files, but they don't
> have line endings at all or any user-readable content, so I think
> treating them as blobs is appropriate.
Nice analysis.

If we can use the platform "diff -u" (i.e. we somehow find that it is possible to stop ignoring crlf vs lf difference), then it should give us similarly good performance for large files (but the cost to spawn the tool 19000 times would also be comparable), but we are not there yet, I presume.

Previous: René Scharfe
Message 11 of 11 in “What's cooking in git.git (Dec 2022, #03; Sun, 11)”
  1. Junio C HamanoDec 11, 2022
  2. js/drop-mingw-test-cmp, was Re: What's cooking in git.git (Dec 2022, #03; Sun, 11)Johannes Schindelin, Dec 12, 2022
  3. Junio C HamanoDec 12, 2022
  4. Johannes SchindelinDec 20, 2022
  5. Junio C HamanoDec 21, 2022
  6. Johannes SixtDec 24, 2022
  7. René ScharfeDec 24, 2022
  8. Junio C HamanoDec 25, 2022
  9. Johannes SixtDec 25, 2022
  10. René ScharfeDec 26, 2022
  11. Junio C HamanoDec 27, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.