{"thread":{"id":"9010","subject":"[PATCH] reduce git-pack-objects memory usage a little more","startedAt":"2007-07-12T21:07:59Z","lastAt":"2007-07-13T03:10:51Z","messageCount":5,"participants":["Nicolas Pitre","Brian Downing","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"47209","messageId":"alpine.LFD.0.999.0707121703240.32552@xanadu.home","threadId":"9010","inReplyTo":null,"subject":"[PATCH] reduce git-pack-objects memory usage a little more","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-07-12T21:07:59Z","receivedAt":"2007-07-12T21:07:59Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"The delta depth doesn't have to be stored in the global object array \nstructure since it is only used during the deltification pass.\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n---\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex b4f3e7c..55609f3 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -25,9 +25,6 @@ git-pack-objects [{ -q | --progress | --all-progress }] [--max-pack-size=N] \\n\\\n struct object_entry {\n \tstruct pack_idx_entry idx;\n \tunsigned long size;\t/* uncompressed size */\n-\n-\tunsigned int hash;\t/* name hint hash */\n-\tunsigned int depth;\t/* delta depth */\n \tstruct packed_git *in_pack; \t/* already in pack */\n \toff_t in_pack_offset;\n \tstruct object_entry *delta;\t/* delta base object */\n@@ -37,6 +34,7 @@ struct object_entry {\n \t\t\t\t\t     */\n \tvoid *delta_data;\t/* cached delta (uncompressed) */\n \tunsigned long delta_size;\t/* delta data size (uncompressed) */\n+\tunsigned int hash;\t/* name hint hash */\n \tenum object_type type;\n \tenum object_type in_pack_type;\t/* could be delta */\n \tunsigned char in_pack_header_size;\n@@ -1270,6 +1268,7 @@ struct unpacked {\n \tstruct object_entry *entry;\n \tvoid *data;\n \tstruct delta_index *index;\n+\tunsigned depth;\n };\n \n static int delta_cacheable(struct unpacked *trg, struct unpacked *src,\n@@ -1328,7 +1327,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\treturn 0;\n \n \t/* Let's not bust the allowed depth. */\n-\tif (src_entry->depth >= max_depth)\n+\tif (src->depth >= max_depth)\n \t\treturn 0;\n \n \t/* Now some size filtering heuristics. */\n@@ -1338,9 +1337,9 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \t\tref_depth = 1;\n \t} else {\n \t\tmax_size = trg_entry->delta_size;\n-\t\tref_depth = trg_entry->depth;\n+\t\tref_depth = trg->depth;\n \t}\n-\tmax_size = max_size * (max_depth - src_entry->depth) /\n+\tmax_size = max_size * (max_depth - src->depth) /\n \t\t\t\t\t\t(max_depth - ref_depth + 1);\n \tif (max_size == 0)\n \t\treturn 0;\n@@ -1379,17 +1378,17 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \tif (trg_entry->delta_data) {\n \t\t/* Prefer only shallower same-sized deltas. */\n \t\tif (delta_size == trg_entry->delta_size &&\n-\t\t    src_entry->depth + 1 >= trg_entry->depth) {\n+\t\t    src->depth + 1 >= trg->depth) {\n \t\t\tfree(delta_buf);\n \t\t\treturn 0;\n \t\t}\n \t\tdelta_cache_size -= trg_entry->delta_size;\n \t\tfree(trg_entry->delta_data);\n+\t\ttrg_entry->delta_data = NULL;\n \t}\n-\ttrg_entry->delta_data = 0;\n \ttrg_entry->delta = src_entry;\n \ttrg_entry->delta_size = delta_size;\n-\ttrg_entry->depth = src_entry->depth + 1;\n+\ttrg->depth = src->depth + 1;\n \n \tif (delta_cacheable(src, trg, src_size, trg_size, delta_size)) {\n \t\ttrg_entry->delta_data = xrealloc(delta_buf, delta_size);\n@@ -1484,7 +1483,7 @@ static void find_deltas(struct object_entry **list, int window, int depth)\n \t\t * depth, leaving it in the window is pointless.  we\n \t\t * should evict it first.\n \t\t */\n-\t\tif (entry->delta && depth <= entry->depth)\n+\t\tif (entry->delta && depth <= n->depth)\n \t\t\tcontinue;\n \n \t\tnext:\n"},{"id":"47225","messageId":"20070713014228.GE19073@lavos.net","threadId":"9010","inReplyTo":"alpine.LFD.0.999.0707121703240.32552@xanadu.home","subject":"Re: [PATCH] reduce git-pack-objects memory usage a little more","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-13T01:42:28Z","receivedAt":"2007-07-13T01:42:28Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"On Thu, Jul 12, 2007 at 05:07:59PM -0400, Nicolas Pitre wrote:\n> The delta depth doesn't have to be stored in the global object array \n> structure since it is only used during the deltification pass.\n\nThis patch breaks pack-objects pretty horribly:\n\n:; PATH=~/src/git:$PATH /usr/bin/time ~/src/git/git-repack -a -d -f\n\n:; ls -l .git/objects/pack\ntotal 153916\n-r--r--r-- 1 bdowning bdowning   1312136 2007-07-12 20:39 pack-9ac926ee1f5810c434707d3f816f5ad2cbd14668.idx\n-r--r--r-- 1 bdowning bdowning 156130933 2007-07-12 20:39 pack-9ac926ee1f5810c434707d3f816f5ad2cbd14668.pack\n\nchain length = 1: 6182 objects\nchain length = 2: 66 objects\nchain length = 3: 27 objects\nchain length = 4: 20 objects\nchain length = 5: 15 objects\nchain length = 6: 9 objects\nchain length = 7: 5 objects\nchain length = 8: 5 objects\nchain length = 9: 6 objects\nchain length = 10: 4 objects\nchain length = 11: 6 objects\nchain length = 12: 4 objects\nchain length = 13: 3 objects\nchain length = 14: 3 objects\nchain length = 15: 2 objects\nchain length = 16: 2 objects\nchain length = 17: 1 object\nchain length = 18: 1 object\nchain length = 19: 1 object\n\nUnfortunately I didn't notice until I tried v1.5.3-rc1...\n\n-bcd\n"},{"id":"47226","messageId":"20070713014502.GF19073@lavos.net","threadId":"9010","inReplyTo":"20070713014228.GE19073@lavos.net","subject":"Re: [PATCH] reduce git-pack-objects memory usage a little more","fromName":"Brian Downing","fromEmail":"bdowning@lavos.net","sentAt":"2007-07-13T01:45:02Z","receivedAt":"2007-07-13T01:45:02Z","isPatch":true,"sender":{"key":"bdowning@lavos.net","avatar":"https://avatars.githubusercontent.com/u/366426?v=4"},"body":"On Thu, Jul 12, 2007 at 08:42:28PM -0500, Brian Downing wrote:\n> This patch breaks pack-objects pretty horribly:\n> \n> :; PATH=~/src/git:$PATH /usr/bin/time ~/src/git/git-repack -a -d -f\n\n[that's Git I'm repacking here]\n\n> :; ls -l .git/objects/pack\n> total 153916\n> -r--r--r-- 1 bdowning bdowning   1312136 2007-07-12 20:39 pack-9ac926ee1f5810c434707d3f816f5ad2cbd14668.idx\n> -r--r--r-- 1 bdowning bdowning 156130933 2007-07-12 20:39 pack-9ac926ee1f5810c434707d3f816f5ad2cbd14668.pack\n> \n> chain length = 1: 6182 objects\n> chain length = 2: 66 objects\n> chain length = 3: 27 objects\n> chain length = 4: 20 objects\n> chain length = 5: 15 objects\n> chain length = 6: 9 objects\n> chain length = 7: 5 objects\n> chain length = 8: 5 objects\n> chain length = 9: 6 objects\n> chain length = 10: 4 objects\n> chain length = 11: 6 objects\n> chain length = 12: 4 objects\n> chain length = 13: 3 objects\n> chain length = 14: 3 objects\n> chain length = 15: 2 objects\n> chain length = 16: 2 objects\n> chain length = 17: 1 object\n> chain length = 18: 1 object\n> chain length = 19: 1 object\n\n-bcd\n"},{"id":"47227","messageId":"alpine.LFD.0.999.0707122221240.32552@xanadu.home","threadId":"9010","inReplyTo":"20070713014502.GF19073@lavos.net","subject":"[PATCH] forgot to clear the depth value","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-07-13T02:27:12Z","receivedAt":"2007-07-13T02:27:12Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"Commit 5a235b5e was missing this little detail.  Otherwise your pack\nwill explode.\n\nProblem noted by Brian Downing.\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n---\n\nOn Thu, 12 Jul 2007, Brian Downing wrote:\n\n> On Thu, Jul 12, 2007 at 08:42:28PM -0500, Brian Downing wrote:\n> > This patch breaks pack-objects pretty horribly:\n> > \n> > :; PATH=~/src/git:$PATH /usr/bin/time ~/src/git/git-repack -a -d -f\n> \n> [that's Git I'm repacking here]\n> \n> > :; ls -l .git/objects/pack\n> > total 153916\n> > -r--r--r-- 1 bdowning bdowning   1312136 2007-07-12 20:39 pack-9ac926ee1f5810c434707d3f816f5ad2cbd14668.idx\n> > -r--r--r-- 1 bdowning bdowning 156130933 2007-07-12 20:39 pack-9ac926ee1f5810c434707d3f816f5ad2cbd14668.pack\n\nOooooops.\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex a43d604..5e9d1fd 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -1431,6 +1431,7 @@ static void free_unpacked(struct unpacked *n)\n \t\twindow_memory_usage -= n->entry->size;\n \t}\n \tn->entry = NULL;\n+\tn->depth = 0;\n }\n \n static void find_deltas(struct object_entry **list, int window, int depth)\n"},{"id":"47228","messageId":"7vir8proas.fsf@assigned-by-dhcp.cox.net","threadId":"9010","inReplyTo":"alpine.LFD.0.999.0707122221240.32552@xanadu.home","subject":"Re: [PATCH] forgot to clear the depth value","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-07-13T03:10:51Z","receivedAt":"2007-07-13T03:10:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Gaaah ..... thanks.\n"}]}