git/list[1] front-page[2] threads[3] people[4] search[5] about
 

git rev-list | git cherry-pick --stdin is leaky

From
Stephen Boyd <sboyd@codeaurora.org>
Date
Apr 30, 2013, 00:11 UTC
Message-ID
<517F0C18.8060703@codeaurora.org>
(resending since the attachment seems to make vger sad)
Hi,

I'm running git rev-list | git cherry-pick --stdin on a range of about 300 commits. Eventually the chery-pick dies with:

    error: cannot fork() for commit: Cannot allocate memory

Running valgrind shows me that the tree traversal code is leaking gigabytes of memory (particularly unpack_callback). Since cherry-pick is a very long running process all these allocations are never freed and eventually I run out of memory. The worst offender and summary is:

==7986== 938,956,692 (929,961,582 direct, 8,995,110 indirect) bytes in 7,765,439 blocks are definitely lost in loss record 257 of 257 ==7986== at 0x4C267CC: calloc (vg_replace_malloc.c:467) ==7986== by 0x4FAF57: xcalloc (wrapper.c:119) ==7986== by 0x4F5281: unpack_callback (unpack-trees.c:539) ==7986== by 0x4F40E5: traverse_trees (tree-walk.c:407) ==7986== by 0x4F586C: unpack_callback (unpack-trees.c:467) ==7986== by 0x4F40E5: traverse_trees (tree-walk.c:407) ==7986== by 0x4F586C: unpack_callback (unpack-trees.c:467) ==7986== by 0x4F40E5: traverse_trees (tree-walk.c:407) ==7986== by 0x4F586C: unpack_callback (unpack-trees.c:467) ==7986== by 0x4F40E5: traverse_trees (tree-walk.c:407) ==7986== by 0x4F586C: unpack_callback (unpack-trees.c:467) ==7986== by 0x4F40E5: traverse_trees (tree-walk.c:407) ==7986== ==7986== LEAK SUMMARY: ==7986== definitely lost: 2,514,117,692 bytes in 21,210,861 blocks ==7986== indirectly lost: 885,481,947 bytes in 10,165,801 blocks ==7986== possibly lost: 650,712,395 bytes in 6,014,309 blocks ==7986== still reachable: 7,734,870 bytes in 47,794 blocks ==7986== suppressed: 0 bytes in 0 blocks

This is against recent git.git (89740333e8d398f1da701e9023675321bbb9a85b). A workaround is to limit the amount of commits per cherry-pick invocation, but can we somehow fix the leaks?

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation
Next: Junio C Hamano
Message 1 of 7 in “git rev-list | git cherry-pick --stdin is leaky”
  1. Stephen BoydApr 30, 2013
  2. Junio C HamanoApr 30, 2013
  3. René ScharfeApr 30, 2013
  4. Stephen BoydMay 1, 2013
  5. Stephen BoydMay 6, 2013
  6. René ScharfeMay 9, 2013
  7. Stephen BoydMay 9, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.