git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Unpredictable peak memory usage when using `git log` command

From
Patrick Steinhardt <ps@pks.im>
Date
Sep 2, 2024, 13:08 UTC
Message-ID
<ZtW4vD3j87oVkWKC@tanuki>
In-Reply-To
<20240830210607.GB1038751@coredump.intra.peff.net>
On Fri, Aug 30, 2024 at 05:06:07PM -0400, Jeff King wrote:
Show 23 quoted lines
> On Fri, Aug 30, 2024 at 03:20:15PM +0300, Yuri Karnilaev wrote:
> 
> > 2. Processing commits in batches:
> > ```
> > /usr/bin/time -l -h -p git log --ignore-missing --pretty=format:%H%x02%P%x02%aN%x02%aE%x02%at%x00 -n 1000 --skip=1000000 --numstat > 1.txt
> > ```
> > [...]
> > Operating System: Mac OS 14.6.1 (23G93)
> > Git Version: 2.39.3 (Apple Git-146)
> 
> I sent a patch which I think should make things better for you, but I
> wanted to mention two things in a more general way:
> 
>   1. You should really consider building a commit-graph file with "git
>      commit-graph write --reachable". That will reduce the memory usage
>      for this case, but also improve the CPU quite a bit (we won't have
>      to open those million skipped commits to chase their parent
>      pointers).
> 
>      I haven't kept up with the defaults for writing graph files. I
>      thought gc.writeCommitGraph defaults to "true" these days, though
>      that wouldn't help in a freshly cloned repository (arguably we
>      should write the commit graph on clone?).

It does default to true indeed. There is also an option to write commit graph on fetch via "fetch.writeCommitGraph", but that setting is set to false by default. To the best of my knowledge there is no option to generate on a clone, but I agree that it would be sensible to have such a thing.

Patrick
Previous: Yuri Karnilaev
Message 6 of 6 in “Unpredictable peak memory usage when using `git log` command”
  1. Yuri KarnilaevAug 30, 2024
  2. revision: free commit buffers for skipped commitsJeff King, Aug 30, 2024
  3. Junio C HamanoAug 30, 2024
  4. Jeff KingAug 30, 2024
  5. Yuri KarnilaevAug 31, 2024
  6. Patrick SteinhardtSep 2, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.