{"thread":{"id":"14368","subject":"[PATCH] Correct pack memory leak causing git gc to try to exceed ulimit","startedAt":"2008-07-09T07:10:07Z","lastAt":"2008-07-10T05:37:25Z","messageCount":2,"participants":["Shawn O. Pearce","Nicolas Pitre"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"82692","messageId":"20080709071007.GA31974@spearce.org","threadId":"14368","inReplyTo":null,"subject":"[PATCH] Correct pack memory leak causing git gc to try to exceed ulimit","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-07-09T07:10:07Z","receivedAt":"2008-07-09T07:10:07Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"When recursing to unpack a delta base we must unuse_pack() so that\nthe pack window for the current object does not remain pinned in\nmemory while the delta base is itself being unpacked and materialized\nfor our use.\n\nOn a long delta chain of 50 objects we may need to access 6 different\nwindows from a very large (>3G) pack file in order to obtain all\nof the delta base content.  If the process ulimit permits us to\nmap/allocate only 1.5G we must release windows during this recursion\nto ensure we stay within the ulimit and transition memory from pack\ncache to standard malloc, or other mmap needs.\n\nInserting an unuse_pack() call prior to the recursion allows us to\navoid pinning the current window, making it available for garbage\ncollection if memory runs low.\n\nThis has been broken since at least before 1.5.1-rc1, and very\nlikely earlier than that.  Its fixed now.  :)\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n sha1_file.c |    1 +\n 1 files changed, 1 insertions(+), 0 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex d1c4060..10346b6 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1609,6 +1609,7 @@ static void *unpack_delta_entry(struct packed_git *p,\n \toff_t base_offset;\n \n \tbase_offset = get_delta_base(p, w_curs, &curpos, *type, obj_offset);\n+\tunuse_pack(w_curs);\n \tbase = cache_or_unpack_entry(p, base_offset, &base_size, type, 0);\n \tif (!base)\n \t\tdie(\"failed to read delta base object\"\n-- \n1.5.6.74.g8a5e\n"},{"id":"82778","messageId":"alpine.LFD.1.10.0807100119360.12484@xanadu.home","threadId":"14368","inReplyTo":"20080709071007.GA31974@spearce.org","subject":"Re: [PATCH] Correct pack memory leak causing git gc to try to exceed ulimit","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-07-10T05:37:25Z","receivedAt":"2008-07-10T05:37:25Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 9 Jul 2008, Shawn O. Pearce wrote:\n\n> When recursing to unpack a delta base we must unuse_pack() so that\n> the pack window for the current object does not remain pinned in\n> memory while the delta base is itself being unpacked and materialized\n> for our use.\n> \n> On a long delta chain of 50 objects we may need to access 6 different\n> windows from a very large (>3G) pack file in order to obtain all\n> of the delta base content.  If the process ulimit permits us to\n> map/allocate only 1.5G we must release windows during this recursion\n> to ensure we stay within the ulimit and transition memory from pack\n> cache to standard malloc, or other mmap needs.\n> \n> Inserting an unuse_pack() call prior to the recursion allows us to\n> avoid pinning the current window, making it available for garbage\n> collection if memory runs low.\n> \n> This has been broken since at least before 1.5.1-rc1, and very\n> likely earlier than that.  Its fixed now.  :)\n\nWell well...  I suspect this might have been the cause of our strange \nout-of-memory issue when trying to repack the gcc repository a while \nago.\n\nI updated my copy of git://git.infradead.org/gcc.git and re-attempted a \nfull repack with window=500 and depth=500, and this time it worked for \nme on a 32-bit machine!  Took about 6h30 single-threaded, and produced a \nmere 331MB pack file containing 1254664 objects across 142328 commits.\n\n\nNicolas\n"}]}