git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Surprising use of memory and time when repacking mozilla's gecko repository

From
EWEric Wong <e@80x24.org>
Date
Jul 4, 2019, 12:04 UTC
Message-ID
<20190704120411.2dlzdlsooob4dc7k@dcvr>
In-Reply-To
<20190704100530.smn4rpiekwtfylhz@glandium.org>
Mike Hommey <mh@glandium.org> wrote:
> I'm puzzled by the fact writing objects is so much faster with 1 thread.
I/O contention in the multi-threaded cases?

"public-inbox-index" (reading from git, writing to Xapian+SQLite) on a dev machine got slow because core count exceeded what SATA could handle and had to cap the default Xapian shard count to 3 by default for v2 inboxes.

As for memory use, does git use the producer/consumer pattern for malloc+free so free happens in a different thread from the malloc?

Because current versions of glibc gets pathological with that case (but I have a long-term fix for it):

https://public-inbox.org/libc-alpha/20180731084936.g4yw6wnvt677miti@dcvr/T/ Only locklessinc malloc seemed OK in the IPC department (But I haven't kept up with jemalloc, tcmalloc developments in years)

Previous: Mike HommeyNext: Mike Hommey
Message 2 of 11 in “Surprising use of memory and time when repacking mozilla's gecko repository”
  1. Mike HommeyJul 4, 2019
  2. Eric WongJul 4, 2019
  3. Mike HommeyJul 4, 2019
  4. Jeff KingJul 5, 2019
  5. Mike HommeyJul 5, 2019
  6. Jakub NarebskiJul 5, 2019
  7. Mike HommeyJul 5, 2019
  8. Mike HommeyJul 5, 2019
  9. Jeff KingJul 5, 2019
  10. Mike HommeyJul 5, 2019
  11. Mike HommeyJul 5, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.